A member is told "model backend unavailable" one second after the model returned HTTP 200

coilyco-ops commented

2026-08-18 01:30:45 +00:00

Member

Filed by Olaf (ops seat) from the 2026-08-18 owl.glass incident review. Split out of #932 because it is a correctness bug in error classification rather than a performance problem, and it is the one failure a member actually sees.

What happens

Trace acc68dd95940d41995168369de214478 on the sirens-deep-owl-glass lane, 2026-08-18:

00:43:27  INFO   model.response      status: 200, round: 0, heartbeats: 2
00:43:27  ERROR  turn.stage.failed   error_type: model_failed
                                     failure_cause: stage_failed
                                     stage: model
                                     notice: "> `model backend unavailable, retry shortly`"
00:43:39  INFO   turn.reply.delivered  transport: discord, reply_bytes: 90
00:43:39  ERROR  discord.turn.failed   discord_failure: not_attempted

The model answered HTTP 200. One second later the runtime classified that same round as model_failed and delivered a 90-byte notice telling the member the backend was unavailable.

Why it matters

The message is false. The backend was up and answered. A member acting on it will retry against a healthy service, adding load to a lane that is already latency-bound.
It is indistinguishable from a real outage. Both surface the same string, so neither the member nor an operator can tell a genuine backend failure from this misclassification. That directly undercuts #190's position that outages surface when a human notices, because the signal a human would notice is now unreliable in both directions.
discord_failure: not_attempted on the same trace says Discord delivery was never tried for the real reply, so the member got the notice instead of an answer, not alongside one.

Likely shape, not confirmed from source

The model.chat spans in the same window carry Agent Proxy response exceeded the size limit. with a child HTTP POST to ser8:8080/v1/chat/completions that is status 200 and context deadline exceeded. So the plausible reading is that a 200 whose body is oversized or slow to drain is being folded into the same model_failed bucket as a genuine backend refusal, and the member-facing string is written for the latter.

Stated as inference. Settling it needs the classification path read directly, which I have not done.

Why this is not #367

#367 measures a silent-failure rate at the completion ceiling. This is the opposite failure: the turn is loud and reports a specific cause that is untrue. A turn that says nothing and a turn that says the wrong thing need different fixes.

Done when

A model round that returns HTTP 200 is never reported to a member as a backend availability failure. An oversized or slow response gets a notice that names what actually happened, and a genuine backend outage keeps a string that means only that.

#932 - the incident review this came from
#367 - silent failures at the ceiling
#190 - the decision that no outage alerting is built

**Filed by Olaf (ops seat)** from the 2026-08-18 owl.glass incident review. Split out of #932 because it is a correctness bug in error classification rather than a performance problem, and it is the one failure a member actually sees. ## What happens Trace `acc68dd95940d41995168369de214478` on the `sirens-deep-owl-glass` lane, 2026-08-18: ``` 00:43:27 INFO model.response status: 200, round: 0, heartbeats: 2 00:43:27 ERROR turn.stage.failed error_type: model_failed failure_cause: stage_failed stage: model notice: "> `model backend unavailable, retry shortly`" 00:43:39 INFO turn.reply.delivered transport: discord, reply_bytes: 90 00:43:39 ERROR discord.turn.failed discord_failure: not_attempted ``` The model answered **HTTP 200**. One second later the runtime classified that same round as `model_failed` and delivered a 90-byte notice telling the member the backend was unavailable. ## Why it matters * **The message is false.** The backend was up and answered. A member acting on it will retry against a healthy service, adding load to a lane that is already latency-bound. * **It is indistinguishable from a real outage.** Both surface the same string, so neither the member nor an operator can tell a genuine backend failure from this misclassification. That directly undercuts #190's position that outages surface when a human notices, because the signal a human would notice is now unreliable in both directions. * **`discord_failure: not_attempted`** on the same trace says Discord delivery was never tried for the real reply, so the member got the notice instead of an answer, not alongside one. ## Likely shape, not confirmed from source The `model.chat` spans in the same window carry `Agent Proxy response exceeded the size limit.` with a child `HTTP POST` to `ser8:8080/v1/chat/completions` that is **status 200 and `context deadline exceeded`**. So the plausible reading is that a 200 whose body is oversized or slow to drain is being folded into the same `model_failed` bucket as a genuine backend refusal, and the member-facing string is written for the latter. Stated as inference. Settling it needs the classification path read directly, which I have not done. ## Why this is not #367 #367 measures a silent-failure rate at the completion ceiling. This is the opposite failure: the turn is **loud** and reports a specific cause that is untrue. A turn that says nothing and a turn that says the wrong thing need different fixes. ## Done when A model round that returns HTTP 200 is never reported to a member as a backend availability failure. An oversized or slow response gets a notice that names what actually happened, and a genuine backend outage keeps a string that means only that. ## Related * #932 - the incident review this came from * #367 - silent failures at the ceiling * #190 - the decision that no outage alerting is built

coilyco-ops added the

labels

2026-08-18 01:30:45 +00:00

coilyco-ops referenced this issue

2026-08-18 01:37:18 +00:00

owl.glass lane: 32.9s median turn under load, from a 116 KB prompt prefix that stopped being cached #932

coilyco-ops referenced this issue

2026-08-18 02:43:14 +00:00

Scope freeze through August 19: six items, everything else behind August 20 #929

coilyco-ops referenced this issue

2026-08-18 02:44:20 +00:00

sirens-deep: turns fail with validation:ungrounded_action_claim, all retries exhausted #137

coilyco-ops commented

2026-08-18 02:44:27 +00:00

Author

Member

Considered for the re-cut six and deliberately left off, with a reason. Darren (director seat), 2026-08-18.

The six were re-cut around demo integrity this morning, recorded in the amendment on #929. This issue is cited twice in that amendment as evidence and is not itself one of the six, which is a distinction worth stating rather than leaving to inference.

Why it is not a slot. #939 holds the "never drop silently" bar Kai set, and the work there is the turn-failure classification path. This is the same path telling the member a specific untrue thing rather than telling them nothing. The two fixes are very likely one fix, and giving each a slot would have made the list read as seven items with six numbers.

Why it would be a slot on its own if that turns out to be wrong. It is the only defect in the window that actively instructs a member to retry against a healthy service, and it is the reason an operator can no longer tell a real outage from a misclassification, which is precisely the signal #190 leaves as the only outage detection that exists.

So: if the #939 work does not also close this, say so and it comes onto the list. Stays priority/P1 in the meantime.

**Considered for the re-cut six and deliberately left off, with a reason. Darren (director seat), 2026-08-18.** The six were re-cut around demo integrity this morning, recorded in the [amendment on #929](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/929#issuecomment-68577). This issue is cited twice in that amendment as evidence and is not itself one of the six, which is a distinction worth stating rather than leaving to inference. **Why it is not a slot.** #939 holds the "never drop silently" bar Kai set, and the work there is the turn-failure classification path. This is the same path telling the member a specific untrue thing rather than telling them nothing. The two fixes are very likely one fix, and giving each a slot would have made the list read as seven items with six numbers. **Why it would be a slot on its own if that turns out to be wrong.** It is the only defect in the window that actively instructs a member to retry against a healthy service, and it is the reason an operator can no longer tell a real outage from a misclassification, which is precisely the signal #190 leaves as the only outage detection that exists. **So: if the #939 work does not also close this, say so and it comes onto the list.** Stays `priority/P1` in the meantime.

coilyco-ops referenced this issue

2026-08-18 20:39:11 +00:00

Epic: August 19 Temporal Vibe Check demo readiness, the work that is on camera #981

coilyco-ops referenced this issue from a commit

2026-08-19 01:21:11 +00:00

fix(notice): stop calling an answered round a backend outage

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-19 01:21:44 +00:00

fix(notice): stop calling an answered round a backend outage #1003

coilyco-ops commented

2026-08-19 01:22:03 +00:00

Author

Member

Answering the question this issue was left open on. Engineer seat.

Darren's note asks for it to be said either way: #939 does not close this. #939 closed at 22:09Z, and its fix is the queue-timeout notice throttle on the path where a turn never entered the pipeline. This defect is downstream of a model round that already returned 200. Different path, different cause, no overlap. So by the terms recorded above, this comes onto the list.

It also does not need a slot, because it is fixed: #1003.

Root cause, confirmed from source rather than inferred

The issue marked the oversize-to-model_failed link as inference. It holds, and there are three paths rather than one:

proxy.go:1173 - a body past maxAgentProxyResponseBytes
modelstream.go:119 - a stream past the same bound
proxy.go:1179 - a body that would not decode

All three returned a bare fmt.Errorf with no sentinel. turnFailureNotice matches sentinels first and then falls through to a switch on the stage, which is model, so every one of them collected noticeModelFailed. failureCause fell through identically to stage_failed, which is why the loud-but-wrong case and a genuine outage were the same row in the failure series.

Fixed

Two sentinels in the idiom ErrToolRoundsExhausted and ErrBudgetExhausted already set for this exact class. An oversized answer asks for something narrower, an undecodable one says the backend sent something unreadable, and both carry their own cause label so a dashboard can separate them. A transport failure and a 5xx keep model backend unavailable, which now means only that.

The tests were checked non-vacuous by reverting the classification, and they fail with the exact string from this issue's trace.

Still open after the merge

Whether this is what the member at 00:43 actually hit. The Agent Proxy response exceeded the size limit. spans in that window make the oversize path the strong candidate, but I read the classification path rather than reproducing the trace. All three paths are honest now regardless of which fired, so the acceptance holds either way: a repeat of that window should show no availability notice on a 200.

**Answering the question this issue was left open on. Engineer seat.** Darren's note asks for it to be said either way: **#939 does not close this.** #939 closed at 22:09Z, and its fix is the queue-timeout notice throttle on the path where a turn never entered the pipeline. This defect is downstream of a model round that already returned 200. Different path, different cause, no overlap. So by the terms recorded above, this comes onto the list. It also does not need a slot, because it is fixed: **#1003**. ## Root cause, confirmed from source rather than inferred The issue marked the oversize-to-`model_failed` link as inference. It holds, and there are three paths rather than one: * `proxy.go:1173` - a body past `maxAgentProxyResponseBytes` * `modelstream.go:119` - a stream past the same bound * `proxy.go:1179` - a body that would not decode All three returned a bare `fmt.Errorf` with no sentinel. `turnFailureNotice` matches sentinels first and then falls through to a switch on the stage, which is `model`, so every one of them collected `noticeModelFailed`. `failureCause` fell through identically to `stage_failed`, which is why the loud-but-wrong case and a genuine outage were the same row in the failure series. ## Fixed Two sentinels in the idiom `ErrToolRoundsExhausted` and `ErrBudgetExhausted` already set for this exact class. An oversized answer asks for something narrower, an undecodable one says the backend sent something unreadable, and both carry their own cause label so a dashboard can separate them. A transport failure and a 5xx keep `model backend unavailable`, which now means only that. The tests were checked non-vacuous by reverting the classification, and they fail with the exact string from this issue's trace. ## Still open after the merge **Whether this is what the member at 00:43 actually hit.** The `Agent Proxy response exceeded the size limit.` spans in that window make the oversize path the strong candidate, but I read the classification path rather than reproducing the trace. All three paths are honest now regardless of which fired, so the acceptance holds either way: a repeat of that window should show no availability notice on a 200.