Tag spans with backend operating regime — percentiles are unreadable across a shared GPU #109

New issue

Closed

opened 2026-08-12 22:35:09 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-12 22:35:09 +00:00

Member

Problem

litellm_request over 24h on 2026-08-12: p50 3.42s, p99 233.71s. A 68× spread.

I read that as a heavy tail and guessed "contention or a stall." It was neither. Per #108 it is bimodal — two distinct operating regimes, GPU idle and GPU shared with a game, averaged into one distribution.

Percentiles are the wrong instrument for that shape. They describe a single population, and there were two. The result is that a fully predictable, human-caused state was presented as random variance, which is why three separate pieces of analysis (#162, #105, #160) reached for contention, prefill cost, and queueing before anyone thought to ask what else was running on the box.

Fix

Add a dimension that identifies the operating regime, so latency can be sliced instead of blended:

Backend identity on the span — which host and model actually served this request. Currently inferable only from a client URL, and not groupable.
Capacity state at dispatch — idle / contended / saturated, from whatever signal #108 lands on.

With those, p99 by backend, by regime is a readable number and the two modes stop hiding each other.

What this enables downstream

Alerting on regime transition rather than on a percentile threshold. A move from idle to contended is a step change and is detectable immediately; a percentile shifting inside a blended distribution is not. Relevant to coilyco-gaming/sirens-echo#190, where nothing detected a total outage.
#107's burst test becomes valid. As written it measures admission delay under concurrency — but a contended GPU would confound it completely, and without a regime tag there would be no way to tell afterwards whether the run was clean. The burst test needs this, or it needs a manually verified idle window.
#81's model sweep stops being contaminated. coilyco-gaming/sirens-echo#189 covers this: an OSS cell run on a busy GPU produces a result that looks like model weakness. A regime tag on the run makes that detectable rather than plausible.

Prerequisite worth confirming

Option 3 in #108 — deriving capacity state from GPU utilisation — needs telemetry on the local inference host. The fleet has node-stats coverage for kai-server and ser8; whether the local GPU host is covered is worth checking before designing around it.

Acceptance

Spans carry backend identity and capacity state at dispatch.
Latency can be queried per backend per regime.
A run of #107 or #81 can be shown, after the fact, to have executed against an idle backend.

#108 — the root cause and the source of the capacity signal
#107 — the burst test that needs this to be trustworthy
coilyco-gaming/sirens-echo#162 — where the bimodal distribution was first measured and misread
coilyco-gaming/sirens-echo#189 — sweep contamination
coilyco-gaming/sirens-echo#190 — the alerting gap this would help close

## Problem `litellm_request` over 24h on 2026-08-12: p50 **3.42s**, p99 **233.71s**. A 68× spread. I read that as a heavy tail and guessed "contention or a stall." It was neither. Per #108 it is **bimodal** — two distinct operating regimes, *GPU idle* and *GPU shared with a game*, averaged into one distribution. Percentiles are the wrong instrument for that shape. They describe a single population, and there were two. The result is that a **fully predictable, human-caused state was presented as random variance**, which is why three separate pieces of analysis (#162, #105, #160) reached for contention, prefill cost, and queueing before anyone thought to ask what else was running on the box. ## Fix Add a dimension that identifies the operating regime, so latency can be sliced instead of blended: - **Backend identity** on the span — which host and model actually served this request. Currently inferable only from a client URL, and not groupable. - **Capacity state at dispatch** — idle / contended / saturated, from whatever signal #108 lands on. With those, `p99 by backend, by regime` is a readable number and the two modes stop hiding each other. ## What this enables downstream - **Alerting on regime transition** rather than on a percentile threshold. A move from idle to contended is a step change and is detectable immediately; a percentile shifting inside a blended distribution is not. Relevant to `coilyco-gaming/sirens-echo#190`, where nothing detected a total outage. - **#107's burst test becomes valid.** As written it measures admission delay under concurrency — but a contended GPU would confound it completely, and without a regime tag there would be no way to tell afterwards whether the run was clean. The burst test needs this, or it needs a manually verified idle window. - **#81's model sweep stops being contaminated.** `coilyco-gaming/sirens-echo#189` covers this: an OSS cell run on a busy GPU produces a result that looks like model weakness. A regime tag on the run makes that detectable rather than plausible. ## Prerequisite worth confirming Option 3 in #108 — deriving capacity state from GPU utilisation — needs telemetry on the local inference host. The fleet has node-stats coverage for `kai-server` and `ser8`; whether the local GPU host is covered is worth checking before designing around it. ## Acceptance - Spans carry backend identity and capacity state at dispatch. - Latency can be queried per backend per regime. - A run of #107 or #81 can be shown, after the fact, to have executed against an idle backend. ## Related - #108 — the root cause and the source of the capacity signal - #107 — the burst test that needs this to be trustworthy - `coilyco-gaming/sirens-echo#162` — where the bimodal distribution was first measured and misread - `coilyco-gaming/sirens-echo#189` — sweep contamination - `coilyco-gaming/sirens-echo#190` — the alerting gap this would help close

coilyco-ops referenced this issue

2026-08-12 22:35:25 +00:00

Concurrency burst test: does agent-proxy queue or shed, and should heartbeats carry queue position #107

coilyco-ops referenced this issue

2026-08-13 17:25:38 +00:00

The context budget is 47,104 tokens against a 1M-token model, because num_ctx is a local-inference parameter applied to a hosted route #115

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 18:15:38 +00:00

Tag spans with backend identity and operating regime #122

coilyco-ops referenced this issue from a commit

2026-08-13 18:15:39 +00:00

Tag spans with backend identity and operating regime