Stop emitting one span per SSE chunk: 361k spans in 24h, and it truncates real traces past the 1000-span cap #140

New issue

Closed

opened 2026-08-19 14:53:28 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-19 14:53:28 +00:00

Owner

What

Agent Proxy emits a POST /v1/chat/completions http send span for every server-sent-event chunk of a streamed completion. One completion produced 965 of them. Over a 24h window the proxy emitted 361,384 such spans, which is the overwhelming majority of all trace volume it produces.

Why it matters

This is not only ingest and storage cost. It destroys the debuggability of anything downstream that streams.

While diagnosing slow turns on the sirens-dowel lane on 2026-08-19, I pulled a single 183s turn trace. The trace API returned its 1000-span cap, and 965 of those 1000 rows were zero-duration chunk spans from one completion. The spans that mattered, the turn itself and its tool calls, fell outside the cap. The trace was unreadable, and the investigation had to fall back to aggregate queries instead.

Any trace containing a streamed completion is effectively opaque today. That is the exact situation where an operator most wants a trace.

Numbers

Measured from SigNoz traces, 24h window ending 2026-08-19.

POST /v1/chat/completions http send spans emitted by agent-proxy - 361,384
Chunk spans in one sampled completion - 965
Real work spans in that same trace - roughly 15, covering the model call, the proxy hop, the router, and one tool call
Signal to noise in that trace - about 1.5 percent

Proposal

Chunk-level spans carry close to zero diagnostic value individually. Each has zero duration and no attribute distinguishing it from its neighbours. Options, roughly in preference order:

Do not span individual SSE sends at all. Record the stream as attributes on the parent completion span instead: chunk count, first-token latency, total stream duration, bytes.
If per-chunk timing is genuinely wanted, put it behind a sampling or debug flag that is off by default.
At minimum, suppress the instrumentation that auto-spans http send on the streaming response path.

First-token latency is the one genuinely useful thing buried in this data, and it deserves to be a real attribute rather than something reconstructed from a thousand spans.

Reproducing

Query SigNoz traces filtered to name = 'POST /v1/chat/completions http send' grouped by service.name over 24h. Then fetch the details of any trace containing a streamed completion and observe the span composition against the 1000-span limit.

## What Agent Proxy emits a `POST /v1/chat/completions http send` span for every server-sent-event chunk of a streamed completion. One completion produced 965 of them. Over a 24h window the proxy emitted 361,384 such spans, which is the overwhelming majority of all trace volume it produces. ## Why it matters This is not only ingest and storage cost. It destroys the debuggability of anything downstream that streams. While diagnosing slow turns on the `sirens-dowel` lane on 2026-08-19, I pulled a single 183s turn trace. The trace API returned its 1000-span cap, and 965 of those 1000 rows were zero-duration chunk spans from one completion. The spans that mattered, the turn itself and its tool calls, fell outside the cap. The trace was unreadable, and the investigation had to fall back to aggregate queries instead. Any trace containing a streamed completion is effectively opaque today. That is the exact situation where an operator most wants a trace. ## Numbers Measured from SigNoz traces, 24h window ending 2026-08-19. * `POST /v1/chat/completions http send` spans emitted by `agent-proxy` - 361,384 * Chunk spans in one sampled completion - 965 * Real work spans in that same trace - roughly 15, covering the model call, the proxy hop, the router, and one tool call * Signal to noise in that trace - about 1.5 percent ## Proposal Chunk-level spans carry close to zero diagnostic value individually. Each has zero duration and no attribute distinguishing it from its neighbours. Options, roughly in preference order: * Do not span individual SSE sends at all. Record the stream as attributes on the parent completion span instead: chunk count, first-token latency, total stream duration, bytes. * If per-chunk timing is genuinely wanted, put it behind a sampling or debug flag that is off by default. * At minimum, suppress the instrumentation that auto-spans `http send` on the streaming response path. First-token latency is the one genuinely useful thing buried in this data, and it deserves to be a real attribute rather than something reconstructed from a thousand spans. ## Reproducing Query SigNoz traces filtered to `name = 'POST /v1/chat/completions http send'` grouped by `service.name` over 24h. Then fetch the details of any trace containing a streamed completion and observe the span composition against the 1000-span limit.

coilyco-ops referenced this issue from a commit

2026-08-19 15:14:13 +00:00

perf(obs): carry stream shape as attributes, not one span per SSE chunk

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-19 15:15:12 +00:00

perf(obs): carry stream shape as attributes, not one span per SSE chunk #141