Add head-based sampling for LLM Observability traces - #12277
Conversation
ncybul
added
tag: ai generated
ncybul
changed the title
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9f5c5c40a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ge gate Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
ncybul
changed the title
![datadog-datadog-prod-us1-2[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
A payload buffer retry removes the new sampling tags before the retry reads them. The retry then writes retain values, so the intake can keep a trace that the sampler drops.
🤖 Datadog Autotest · Commit 7a261db · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
I would recommend verifying one last time with https://app.datadoghq.com/dashboard/hr5-2cu-4fn/ai-observability?fromUser=false&refresh_mode=paused&tab_id=902bcec6-8230-4d1d-a6e1-084b24eae60e&from_ts=1787295861478&to_ts=1787900661478&live=false&tile_focus=2921063684666465 when running manual tests but LGTM!
The other thing is to note distributed tracing as a whole as a follow up feature (including sampling decision propagation) for Java.
|
Verified that sending 150 spans with 
|
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
devflow unqueued this merge request: It did not become mergeable within the expected time |
|
/code blockers |
|
View all feedbacks in Devflow UI.
Checking merge blockers for #12277...
Detected 1 merge blocker(s) to address: 🟠 Pending
|
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
devflow unqueued this merge request: It did not become mergeable within the expected time |
|
/code blockers |
|
View all feedbacks in Devflow UI.
Checking merge blockers for #12277...
Detected 1 merge blocker(s) to address: 🟠 Pending
|
|
/code blockers |
|
View all feedbacks in Devflow UI.
Checking merge blockers for #12277...
Detected 1 merge blocker(s) to address: 🟠 Pending
|
Resolves six conflicts with agent_version propagation (a9c94fe), which extended the same LLMObsContext mechanism. All resolutions keep both sides. Adopts .equals() over == for the trace-id inheritance gate in OpenAiDecorator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolves six conflicts with agent attribution (pagent_span_id/pagent_name), which extended the same LLMObsContext subtree-propagation mechanism. The reported conflicts were adjacent additions, resolved keep-both. The real break was a clean auto-merge: both sides added a 5-arg attach() overload with the same erasure, so the merged file did not compile. DDLLMObsSpan assigns its single scope from one attach() call, and that context must now carry session_id, agent_version, the sampling verdict and agent attribution together, so the two overloads are collapsed into one 7-arg overload rather than kept under separate names. Also: - Folds master's attribution inheritance inside OpenAiDecorator's existing `if (inheritable)` gate, which uses .equals() rather than master's ==, so one flag drives every inherited value. - Aligns the collapsed overload on clears-semantics for session_id, matching the 2- and 3-arg overloads. Master's null/empty session_id tests assert null and still pass. - Renames master's fiveArgAttach* tests to fullAttach*; there is no 5-arg overload any more. - Extends the stale-cross-trace forked test to assert pagent tags do not leak, matching the coverage session_id and sampling already had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@autotest review |
There was a problem hiding this comment.
More details
The sample rate flows from configuration to the root decision, child inheritance, and mapper output. The missing value in Config.toString affects diagnostics only, so it does not meet the reporting limit.
🤖 Datadog Autotest · Commit 328bce5 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
4 participants
Jira: MLOB-7815
What Does This Do
Adds head-based retention sampling for LLM Observability traces, controlled by a new
DD_LLMOBS_SAMPLE_RATE(dd.llmobs.sample.rate) config, defaulting to1.0.At the root of an LLMObs trace, the tracer makes a deterministic keep/drop decision keyed on the APM trace ID, then stamps
sampling_decisionandsample_rateinto the_ddblock of that span. The decision is carried on the LLMObs context and inherited unchanged by descendant spans, so a trace is retained or dropped as a whole.The SDK never drops a span. Every LLMObs span is still sent to the intake at 100%; the intake performs the drop. This is deliberate: token and cost metrics are computed over full volume regardless of the configured sample rate.
Auto-instrumented OpenAI spans
Manual (
DDLLMObsSpan) and auto-instrumented (OpenAiDecorator) spans are two independent creation paths, and both reach the LLMObs span mapper.OpenAiDecorator.doAfterStarttherefore inherits the verdict from the enclosing LLMObs context, the same way it already inheritssession_id. If there is no sampling decision that can be inherited (no LLMObs parent), the sampling decision is computed within the integration.The trace-id gate guards against an
LLMObsContextleaked across an async boundary outliving its trace: a verdict computed from a different trace ID says nothing about this one. Both values are stamped together or not at all, so a span can never report a decision without the rate that produced it.A span that reaches the mapper unstamped falls back to
sampling_decision=1/sample_rate=1.Motivation
Python and Node already support LLMObs head-based sampling; Java did not. Customers running high-volume LLM workloads have no way to control retention volume from the Java tracer. The deterministic algorithm matches dd-trace-py's (
SAMPLING_KNUTH_FACTOR).Distributed Tracing
The LLMObs Java SDK does not yet support distributed tracing. This is tracked in a follow-up ticket. Because of this, sampling decisions are not propagated across service boundaries.
Testing
LLMObsSamplerTestcovers the sampler in isolation: keep-everything at1.0, drop-everything at0.0, clamping of out-of-range rates (includingNaN), the formatted rate string, determinism for a given sampling ID, retention of roughly the configured fraction across0.1–0.9, and that a higher rate keeps a superset of what a lower rate keeps.DDLLMObsSpanSamplingTestcovers the manual SDK path: stamping a retained decision at the default rate, stamping a dropped decision on a root, and children inheriting the root's verdict rather than recomputing it — including a child of a dropped root staying dropped.LLMObsSpanMapperTestpins the decoded_ddmap for every combination the writer can produce — the sub-map is hand-sized msgpack, so an off-by-one corrupts the stream rather than failing loudly. That includes the unstamped fallback, which is now a defensive default rather than a path either producer takes.Auto-instrumented coverage is split across two forked classes, because the rate is read once when the sampler singleton initializes and so cannot vary within a JVM:
LlmObsContextPropagationForkedTest(renamed fromSessionIdPropagationForkedTest, since it now covers more thansession_id) runs at the default rate: a dropped verdict is inherited, a retained verdict is inherited, the trace-id gate rejects a stale cross-trace context, and — the behavior this PR adds — a span with no LLMObs parent computes its own verdict instead of arriving unstamped.LlmObsZeroSampleRateForkedTestruns under@WithConfig(key = "llmobs.sample.rate", value = "0")and asserts a parentlessopenai.requestspan is stamped dropped. This is the direct regression test: before this change such a span was retained at every configured rate.I tested an app with various sample rates set, sending 20 traces each time, to ensure that roughly
sample rate = (# traces retained / # traces sent).Sampling rate of 0
No traces were retained for this test as expected.
Sampling rate of 0.5
11 out of 20 traces survived my test:
Sampling rate of 1
All 20 traces survived the test:
Claude session:
f94e7cd8-1c7d-41b1-835f-553f557f2d04Resume:
claude --resume f94e7cd8-1c7d-41b1-835f-553f557f2d04