Skip to content

feat(tracing): make ddtrace adopt the OTel trace id in dual mode #471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NiteshDhanpal wants to merge 1 commit into
base: obs/03-temporal-otel-interceptor
Choose a base branch
from
+33 −1
Open

feat(tracing): make ddtrace adopt the OTel trace id in dual mode #471

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion src/agentex/lib/core/tracing/obs_ids.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
import os
from typing import Dict, Optional, Tuple

__all__ = ("get_obs_mode", "obs_correlation")
__all__ = ("get_obs_mode", "obs_correlation", "sync_ddtrace_to_lgtm")

DD_ONLY = "dd_only"
DUAL = "dual"
Expand Down Expand Up @@ -81,3 +81,31 @@ def obs_correlation() -> Dict[str, str]:
if not ids:
return {}
return {"obs.trace_id": ids[0], "obs.span_id": ids[1]}


def sync_ddtrace_to_lgtm() -> None:
"""In ``dual`` mode, make ddtrace adopt the active OpenTelemetry/LGTM trace
context so ddtrace-emitted spans (best-effort Datadog) share the SAME
trace_id as the OTel trace. Call at request ingress once the OTel span is
active, and after any context boundary (e.g. entering a Temporal activity).

No-op outside ``dual`` mode, or when OTel/ddtrace is unavailable or no OTel
span is active. Never raises.
"""
if get_obs_mode() != DUAL:
return
try:
from opentelemetry import trace

try:
from ddtrace.trace import Context # ddtrace 3.x
except ImportError: # pragma: no cover - older ddtrace layout
from ddtrace.context import Context # type: ignore[no-redef]
from ddtrace import tracer
except ImportError:
return

sc = trace.get_current_span().get_span_context()
if not (sc and sc.is_valid):
return
tracer.context_provider.activate(Context(trace_id=sc.trace_id, span_id=sc.span_id))

@greptile-apps greptile-apps Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Sampling priority not forwarded, ddtrace may make an independent sampling decision

Context(trace_id=sc.trace_id, span_id=sc.span_id) leaves sampling_priority unset. ddtrace will then apply its own sampler for this activated context, which can differ from the sampling decision already made by the OTel SDK for the same trace. In practice this means a span that OTel keeps may be dropped by Datadog (or vice-versa), making the two backends show different coverage for the same trace id even when DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=true guarantees matching ids. If the egp-api-backend reference sets sampling_priority=USER_KEEP (or forwards it from the W3C tracestate), doing the same here would ensure consistent sampling across both backends.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: src/agentex/lib/core/tracing/obs_ids.py
Line: 111

Comment:
**Sampling priority not forwarded, ddtrace may make an independent sampling decision**

`Context(trace_id=sc.trace_id, span_id=sc.span_id)` leaves `sampling_priority` unset. ddtrace will then apply its own sampler for this activated context, which can differ from the sampling decision already made by the OTel SDK for the same trace. In practice this means a span that OTel keeps may be dropped by Datadog (or vice-versa), making the two backends show different coverage for the same trace id even when `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=true` guarantees matching ids. If the egp-api-backend reference sets `sampling_priority=USER_KEEP` (or forwards it from the W3C `tracestate`), doing the same here would ensure consistent sampling across both backends.

How can I resolve this? If you propose a fix, please make it concise.

Fix in Cursor Fix in Claude Code Fix in Codex

4 changes: 4 additions & 0 deletions src/agentex/lib/sdk/fastacp/base/base_acp_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
from agentex.lib.environment_variables import EnvironmentVariables, refreshed_environment_variables
from agentex.types.task_message_update import TaskMessageUpdate, StreamTaskMessageFull
from agentex.types.task_message_content import TaskMessageContent
from agentex.lib.core.tracing.obs_ids import sync_ddtrace_to_lgtm as _sync_ddtrace_to_lgtm
from agentex.lib.core.tracing.span_queue import shutdown_default_span_queue
from agentex.lib.core.compat.version_guard import assert_backend_compatible
from agentex.lib.sdk.fastacp.base.constants import (
Expand Down Expand Up @@ -101,6 +102,9 @@ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
# request so business spans created here (and any downstream Temporal
# workflow started from this request) share the same observability trace.
token = _attach_incoming_trace_context(scope)
# In dual mode, make ddtrace adopt this OTel trace id so its best-effort
# Datadog spans share the same trace_id. No-op outside dual mode.
_sync_ddtrace_to_lgtm()
try:
await self.app(scope, receive, send)
finally:
Expand Down
Loading