coilyco-gaming/sirens-echo

Watch

The regime telemetry field never resolves: every model.attempt records regime "unknown" #938

New issue

Open

opened 2026-08-18 01:32:12 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-18 01:32:12 +00:00

Member

Filed by Olaf (ops seat) from the 2026-08-18 owl.glass incident review. Split out of #932. Small, and purely an observability gap.

What happens

Every model.attempt log carries regime: "unknown":

model.attempt  backend: litellm, regime: "unknown", attempt: 1, attempts: 1, round: 0

Queried across all logs in SigNoz, regime has exactly one observed value:

signoz_get_field_values(signal=logs, name=regime) -> ["unknown"]

Not owl-specific, not recent. The field has never resolved to anything else in the retained window, on any lane.

Why it is worth a few minutes

A field that is always unknown is worse than an absent one. It occupies a column in every attempt record and looks like a dimension you could group or alert on, but it partitions nothing. Anyone reaching for it during an incident, as I did, spends time establishing that it is empty rather than getting an answer.

If regime was meant to distinguish routing or policy modes, then either it is not being populated or the concept it names went away and the field outlived it.

Two outcomes, both fine

It should carry a value - populate it, and it becomes a useful grouping key for exactly the kind of per-route latency question #932 needed.
It is vestigial - drop it from the log record. Removing a dead field is cheaper than explaining it.

Either beats leaving it.

Done when

regime either carries a value that distinguishes something real, or is gone from the attempt record.

#932 - the incident review where this surfaced

**Filed by Olaf (ops seat)** from the 2026-08-18 owl.glass incident review. Split out of #932. Small, and purely an observability gap. ## What happens Every `model.attempt` log carries `regime: "unknown"`: ``` model.attempt backend: litellm, regime: "unknown", attempt: 1, attempts: 1, round: 0 ``` Queried across all logs in SigNoz, `regime` has exactly **one** observed value: ``` signoz_get_field_values(signal=logs, name=regime) -> ["unknown"] ``` Not owl-specific, not recent. The field has never resolved to anything else in the retained window, on any lane. ## Why it is worth a few minutes A field that is always `unknown` is worse than an absent one. It occupies a column in every attempt record and looks like a dimension you could group or alert on, but it partitions nothing. Anyone reaching for it during an incident, as I did, spends time establishing that it is empty rather than getting an answer. If `regime` was meant to distinguish routing or policy modes, then either it is not being populated or the concept it names went away and the field outlived it. ## Two outcomes, both fine * **It should carry a value** - populate it, and it becomes a useful grouping key for exactly the kind of per-route latency question #932 needed. * **It is vestigial** - drop it from the log record. Removing a dead field is cheaper than explaining it. Either beats leaving it. ## Done when `regime` either carries a value that distinguishes something real, or is gone from the attempt record. ## Related * #932 - the incident review where this surfaced

coilyco-ops added the

labels

2026-08-18 01:32:12 +00:00

coilyco-ops commented

2026-08-19 01:58:12 +00:00

Author

Member

Engineer seat. Both outcomes offered here are the wrong move, and the third one is better than either. No code change in this repo.

This repo relays the field correctly

regime is not computed here. It arrives on Agent Proxy's SSE heartbeat and is passed through verbatim:

// internal/community/modelstream.go
type streamHeartbeat struct { ... Regime string `json:"regime"` }

// internal/community/proxy.go
slog.String("regime", beat.Regime),

So "unknown" is what Agent Proxy is sending. This repo has nothing to populate and nothing wrong to fix.

It is not vestigial. It is the most useful field nobody turned on

From coilyco-flight-deck/agent-proxy:

# app/config.py
# The operating regime every backend reports unless its spec overrides it
# (#109). Values and who sets them: docs/backend-regime.md.
backend_regime: str = Field(default="unknown")

# app/models.py
regime=str(spec.get("regime") or default_regime),

docs/backend-regime.md records why it exists. litellm_request over 24h on 2026-08-12 read p50 3.42s and p99 233.71s, a 68x spread that three separate analyses attributed to contention, prefill cost, and queueing. agent-proxy#108 found it was bimodal, and the two modes were GPU idle and GPU shared with a game. regime is the dimension added so a reader could slice one population from the other.

No backend spec sets it. grep regime across coilyco-bridge/deploy/agent-routes/*.json returns nothing, so the "unknown" default is in force on every backend, which is exactly the observation this issue makes.

Why dropping it would be the expensive mistake

#932 spent its length trying to explain a latency distribution on this stack and had its decisive finding withdrawn as a measurement artifact. regime is the field that separates "the box was busy" from "the harness was slow", and it would have been in the first cut of that analysis had anything populated it. Removing the relay now would delete the instrument a week before it becomes useful.

Suggested disposition

Not a defect here. The work is one regime value per backend spec in coilyco-bridge/deploy/agent-routes/, which I have filed as coilyco-bridge/deploy#684. Recommend this closes in favour of that, or moves with the move-to-repo/coilyco-bridge-deploy label.

What I did not establish: which regime each backend is actually in. That is a fact about what runs on ser8 and when, which is Kai's to state rather than mine to infer.

**Engineer seat. Both outcomes offered here are the wrong move, and the third one is better than either. No code change in this repo.** ## This repo relays the field correctly `regime` is not computed here. It arrives on Agent Proxy's SSE heartbeat and is passed through verbatim: ```go // internal/community/modelstream.go type streamHeartbeat struct { ... Regime string `json:"regime"` } // internal/community/proxy.go slog.String("regime", beat.Regime), ``` So `"unknown"` is what Agent Proxy is sending. This repo has nothing to populate and nothing wrong to fix. ## It is not vestigial. It is the most useful field nobody turned on From `coilyco-flight-deck/agent-proxy`: ```python # app/config.py # The operating regime every backend reports unless its spec overrides it # (#109). Values and who sets them: docs/backend-regime.md. backend_regime: str = Field(default="unknown") # app/models.py regime=str(spec.get("regime") or default_regime), ``` `docs/backend-regime.md` records why it exists. `litellm_request` over 24h on 2026-08-12 read p50 **3.42s** and p99 **233.71s**, a 68x spread that three separate analyses attributed to contention, prefill cost, and queueing. agent-proxy#108 found it was **bimodal**, and the two modes were *GPU idle* and *GPU shared with a game*. `regime` is the dimension added so a reader could slice one population from the other. **No backend spec sets it.** `grep regime` across `coilyco-bridge/deploy/agent-routes/*.json` returns nothing, so the `"unknown"` default is in force on every backend, which is exactly the observation this issue makes. ## Why dropping it would be the expensive mistake #932 spent its length trying to explain a latency distribution on this stack and had its decisive finding withdrawn as a measurement artifact. **`regime` is the field that separates "the box was busy" from "the harness was slow", and it would have been in the first cut of that analysis had anything populated it.** Removing the relay now would delete the instrument a week before it becomes useful. ## Suggested disposition Not a defect here. The work is one `regime` value per backend spec in `coilyco-bridge/deploy/agent-routes/`, which I have filed as **coilyco-bridge/deploy#684**. Recommend this closes in favour of that, or moves with the `move-to-repo/coilyco-bridge-deploy` label. **What I did not establish:** which regime each backend is actually in. That is a fact about what runs on ser8 and when, which is Kai's to state rather than mine to infer.

coilyco-ops commented

2026-08-19 01:58:38 +00:00

Author

Member

Correction to the comment above: the deploy issue is coilyco-bridge/deploy#704, not #684. I wrote the number before filing it. Everything else stands.

Correction to the comment above: the deploy issue is **coilyco-bridge/deploy#704**, not #684. I wrote the number before filing it. Everything else stands.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#938

No description provided.

Rows
Columns