Tag spans with backend identity and operating regime #122

Merged

coilysiren merged 1 commit from backend-regime-spans into main

2026-08-13 18:16:28 +00:00

coilyco-ops commented

2026-08-13 18:15:38 +00:00

Member

Summary

Closes #109.

litellm_request p50 3.42s against p99 233.71s was read as a heavy tail. Per #108 it is bimodal - GPU idle and GPU shared with a game. Percentiles describe one population and there were two, which is why three separate pieces of analysis reached for contention, prefill cost, and queueing before anyone asked what else was running on the box.

What changes

Backend identity. It already sat on resilience.attempt and upstream.chat, but not on request.chat - the span a latency query actually groups on - and what was there beside it was a client URL, which does not group. Dispatch now stamps the serving backend onto the result, so the identity on the request span reflects which chain entry won, not which was tried first.

Capacity state. Every backend carries a regime. PROXY_BACKEND_REGIME sets the fleet-wide value; a backend spec entry overrides it with its own "regime" field, so a shared tower and a hosted provider in one chain can report different states. Both attributes ride on request.chat, resilience.attempt, and both upstream spans, which makes p99 by backend, by regime readable.

Suggested values: idle, contended, saturated, hosted, unknown. Nothing enforces the list - it is a grouping key, and an operator needing a sixth value should use one rather than mislabel a run.

The default is `unknown`, not `idle`

The proxy cannot observe GPU occupancy today. Defaulting to idle would assert something nobody measured, and the whole point of this issue is that unmeasured states were being presented as facts.

What this does not do

Nothing derives the regime automatically. That is #108's, specifically its option 3 - capacity state from GPU utilisation via node-stats - and #109 itself flags that the local GPU host's node-stats coverage needs confirming before designing around it. Until then the value is operator-supplied, which is how #108 framed it: the state is known in advance by a human, because nobody starts a game by accident.

That is enough for the third acceptance criterion: a #107 burst run or an #81 sweep can now be shown, after the fact, to have executed against an idle backend, instead of producing a result that looks like model weakness.

How to verify

ward exec test - 3 new cases in tests/test_error_spans.py.
test_spans_carry_backend_identity_and_regime asserts request.chat carries both, which it previously did not.
test_a_backend_spec_can_declare_its_own_regime asserts per-backend override beats the fleet default.
test_regime_defaults_to_unknown_rather_than_a_guess pins the default.

Test plan

Regression tests for identity, override, and default
Existing tests still pass (331 passed)
ward exec format-check, lint, typecheck, pre-commit all clean

Risk

Low. Additive span attributes and one new optional backend-spec field. No metric labels change: llm_upstream_latency_seconds keeps logical_model and backend, because adding a label to a live histogram changes its cardinality and breaks every query already written against it.

Follow-up for Deploy

PROXY_BACKEND_REGIME wants setting per environment, and the tower backend's spec entry can carry "regime": "contended" for a known shared window. Neither is required for this to be correct.

Note

Branched off queue-wait-measures-the-wait (#120), which sits on #119 and #118. Review those first; this branch contains their commits.

## Summary Closes #109. `litellm_request` p50 **3.42s** against p99 **233.71s** was read as a heavy tail. Per #108 it is **bimodal** - GPU idle and GPU shared with a game. Percentiles describe one population and there were two, which is why three separate pieces of analysis reached for contention, prefill cost, and queueing before anyone asked what else was running on the box. ## What changes **Backend identity.** It already sat on `resilience.attempt` and `upstream.chat`, but not on `request.chat` - the span a latency query actually groups on - and what was there beside it was a client URL, which does not group. Dispatch now stamps the serving backend onto the result, so the identity on the request span reflects **which chain entry won**, not which was tried first. **Capacity state.** Every backend carries a regime. `PROXY_BACKEND_REGIME` sets the fleet-wide value; a backend spec entry overrides it with its own `"regime"` field, so a shared tower and a hosted provider in one chain can report different states. Both attributes ride on `request.chat`, `resilience.attempt`, and both upstream spans, which makes `p99 by backend, by regime` readable. Suggested values: `idle`, `contended`, `saturated`, `hosted`, `unknown`. Nothing enforces the list - it is a grouping key, and an operator needing a sixth value should use one rather than mislabel a run. ## The default is `unknown`, not `idle` The proxy cannot observe GPU occupancy today. Defaulting to `idle` would assert something nobody measured, and the whole point of this issue is that unmeasured states were being presented as facts. ## What this does not do **Nothing derives the regime automatically.** That is #108's, specifically its option 3 - capacity state from GPU utilisation via node-stats - and #109 itself flags that the local GPU host's node-stats coverage needs confirming before designing around it. Until then the value is operator-supplied, which is how #108 framed it: the state is known in advance by a human, because nobody starts a game by accident. That is enough for the third acceptance criterion: a #107 burst run or an #81 sweep can now be shown, after the fact, to have executed against an idle backend, instead of producing a result that looks like model weakness. ## How to verify 1. `ward exec test` - 3 new cases in `tests/test_error_spans.py`. 2. `test_spans_carry_backend_identity_and_regime` asserts `request.chat` carries both, which it previously did not. 3. `test_a_backend_spec_can_declare_its_own_regime` asserts per-backend override beats the fleet default. 4. `test_regime_defaults_to_unknown_rather_than_a_guess` pins the default. ## Test plan - [x] Regression tests for identity, override, and default - [x] Existing tests still pass (331 passed) - [x] `ward exec format-check`, `lint`, `typecheck`, `pre-commit` all clean ## Risk Low. Additive span attributes and one new optional backend-spec field. No metric labels change: `llm_upstream_latency_seconds` keeps `logical_model` and `backend`, because adding a label to a live histogram changes its cardinality and breaks every query already written against it. ## Follow-up for Deploy `PROXY_BACKEND_REGIME` wants setting per environment, and the tower backend's spec entry can carry `"regime": "contended"` for a known shared window. Neither is required for this to be correct. ## Note Branched off `queue-wait-measures-the-wait` (#120), which sits on #119 and #118. Review those first; this branch contains their commits.

coilyco-ops added 1 commit

2026-08-13 18:15:38 +00:00

Tag spans with backend identity and operating regime

ci / smoke (pull_request) Successful in 6s

Details

ci / quality (pull_request) Successful in 22s

Details

23d803bea2

Issue #109 opened on a 68x spread in litellm_request, p50 3.42s against p99
233.71s, read at the time as a heavy tail. Per #108 it is neither a tail nor
variance: it is bimodal, and the two modes are GPU idle and GPU shared with a
game. Percentiles describe one population and there were two, so a fully
predictable human-caused state was presented as randomness, and three separate
pieces of analysis reached for contention, prefill cost, and queueing before
anyone asked what else was running on the box.

Slicing needs two dimensions. Backend identity existed on resilience.attempt and
upstream.chat but not on request.chat, which is the span a latency query
actually groups on, and what was there beside it was a client URL, which does
not group. Dispatch now stamps the serving backend onto the result, so the
identity on the request span reflects which chain entry won rather than which
one was tried first.

Capacity state is new. Every backend carries a regime, PROXY_BACKEND_REGIME sets
the fleet-wide value, and a backend spec entry overrides it with its own regime
field so a shared tower and a hosted provider in one chain can report different
states. It rides on request.chat, resilience.attempt, and both upstream spans,
which makes p99 by backend by regime a readable number.

The default is unknown rather than idle. The proxy cannot observe GPU occupancy
today, and defaulting to idle would assert something nobody measured.

Nothing derives the regime automatically, and that half is #108's. Its option 3,
deriving capacity from GPU utilisation through the fleet's node-stats telemetry,
is what would eventually set this without a human, and that issue itself notes
the local GPU host's coverage needs confirming first. Until then the value is
operator-supplied, which is how #108 framed it: the state is known in advance by
a human, because nobody starts a game by accident.

That is enough for the last thing #109 asked for. A #107 burst run or an #81
model sweep can now be shown after the fact to have executed against an idle
backend, rather than producing a result that looks like model weakness.

llm_upstream_latency_seconds keeps its existing labels. Regime is a span
dimension only, because adding a label to a live histogram changes its
cardinality and breaks every query already written against it.

closes #109

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>