perf(obs): carry stream shape as attributes, not one span per SSE chunk #141

Merged

coilysiren merged 1 commit from task/issue-140-sse-chunk-spans into main

2026-08-19 15:16:20 +00:00

coilyco-ops commented

2026-08-19 15:15:12 +00:00

Owner

Closes #140.

The problem

The ASGI instrumentation spans every send, and on the streaming path one send is one SSE frame. A streamed completion therefore cost a POST /v1/chat/completions http send span per chunk: 361,384 of them over 24h, the majority of all trace volume the proxy produced.

Ingest cost was the smaller half. The trace API caps a trace at 1000 spans, and one sampled 183s sirens-dowel turn burned 965 of that budget on chunk spans from a single completion. The turn span and its tool calls fell outside the cap and never rendered. Any trace holding a streamed completion was opaque, which is the exact trace an operator opens.

What this does

Takes the issue's first-choice option: no per-chunk spans at all, stream shape as attributes instead.

exclude_spans=["send"] on FastAPIInstrumentor.instrument_app retires the chunk spans. The knob is per-app, so the single http send span on non-streaming responses goes with them. It carried a status code the server span already reports.
Four attributes land on the request.chat span: agentproxy.stream.frames, .bytes, .duration_ms, and .first_token_ms.
Both clocks start at request receipt, not at the first upstream byte. Admission, retries, and failover are what make a turn slow, so a first-token figure that excluded them would read healthy for exactly the turn under investigation.
first_token_ms is absent, not zero, when a stream produced no content. A stream that died before generating and one that answered instantly are different events.
Totals are stamped on every exit path, so a cancelled or mid-flight-failed stream still reports what it managed to send alongside its error.type.

The dependency floor is load-bearing

exclude_spans arrived in opentelemetry-instrumentation-fastapi 0.48b0, and the floor moves from 0.46b0 to match. _instrument_fastapi swallows exceptions so observability never blocks startup, which means an older wheel would reject the keyword and silently drop all inbound tracing rather than just the send spans. The new test asserts the server span survives for that reason, not only that the chunk spans are gone.

Verification

tests/test_stream_spans.py, 5 tests. Each was mutation-checked: removing exclude_spans, dropping the attribute stamping, always-emitting first_token_ms, and skipping heartbeat frames each fail the suite.
Full suite 412 passed. just format-check, just lint, just typecheck, and pre-commit run --all-files all clean.
Docs: docs/stream-accounting.md, linked from docs/proxy.md and docs/features-landed.md.

🤖 Generated with Claude Code

Closes #140. ## The problem The ASGI instrumentation spans every send, and on the streaming path one send is one SSE frame. A streamed completion therefore cost a `POST /v1/chat/completions http send` span per chunk: **361,384** of them over 24h, the majority of all trace volume the proxy produced. Ingest cost was the smaller half. The trace API caps a trace at 1000 spans, and one sampled 183s `sirens-dowel` turn burned 965 of that budget on chunk spans from a single completion. The turn span and its tool calls fell outside the cap and never rendered. Any trace holding a streamed completion was opaque, which is the exact trace an operator opens. ## What this does Takes the issue's first-choice option: no per-chunk spans at all, stream shape as attributes instead. * `exclude_spans=["send"]` on `FastAPIInstrumentor.instrument_app` retires the chunk spans. The knob is per-app, so the single `http send` span on non-streaming responses goes with them. It carried a status code the server span already reports. * Four attributes land on the `request.chat` span: `agentproxy.stream.frames`, `.bytes`, `.duration_ms`, and `.first_token_ms`. * Both clocks start at request receipt, not at the first upstream byte. Admission, retries, and failover are what make a turn slow, so a first-token figure that excluded them would read healthy for exactly the turn under investigation. * `first_token_ms` is **absent**, not zero, when a stream produced no content. A stream that died before generating and one that answered instantly are different events. * Totals are stamped on every exit path, so a cancelled or mid-flight-failed stream still reports what it managed to send alongside its `error.type`. ## The dependency floor is load-bearing `exclude_spans` arrived in `opentelemetry-instrumentation-fastapi` 0.48b0, and the floor moves from 0.46b0 to match. `_instrument_fastapi` swallows exceptions so observability never blocks startup, which means an older wheel would reject the keyword and silently drop **all** inbound tracing rather than just the send spans. The new test asserts the server span survives for that reason, not only that the chunk spans are gone. ## Verification * `tests/test_stream_spans.py`, 5 tests. Each was mutation-checked: removing `exclude_spans`, dropping the attribute stamping, always-emitting `first_token_ms`, and skipping heartbeat frames each fail the suite. * Full suite 412 passed. `just format-check`, `just lint`, `just typecheck`, and `pre-commit run --all-files` all clean. * Docs: [`docs/stream-accounting.md`](docs/stream-accounting.md), linked from `docs/proxy.md` and `docs/features-landed.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-19 15:15:12 +00:00

perf(obs): carry stream shape as attributes, not one span per SSE chunk

ci / smoke (pull_request) Successful in 7s

Details

ci / quality (pull_request) Successful in 29s

Details

1462e6e9e1

The ASGI instrumentation spanned every send, and on the streaming path one
send is one SSE frame. A streamed completion therefore cost a
`POST /v1/chat/completions http send` span per chunk: 361,384 of them over
24h, and 965 of the 1000-span per-trace cap burned by a single completion.
The turn span and its tool calls fell outside the cap, so the traces most
worth reading were the ones that would not render.

`exclude_spans=["send"]` retires those spans. The four numbers worth keeping
move onto the `request.chat` span as `agentproxy.stream.frames`, `.bytes`,
`.duration_ms`, and `.first_token_ms`, stamped on every exit path so a
cancelled or mid-flight-failed stream still reports what it managed to send.
Both clocks start at request receipt, because admission and failover are what
make a turn slow and a first-token figure that excluded them would look
healthy for exactly the turn under investigation.

The dependency floor is load-bearing rather than cosmetic. `exclude_spans`
arrived in opentelemetry-instrumentation-fastapi 0.48b0, and
`_instrument_fastapi` swallows exceptions so observability never blocks
startup, so an older wheel would drop all inbound tracing instead of just the
send spans. The new test asserts the server span survives for that reason.

Closes #140

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-19 15:15:12 +00:00