fix(notice): stop calling an answered round a backend outage #1003

Merged

coilysiren merged 1 commit from aos/claude/model-answered-not-unavailable into main

2026-08-19 01:31:58 +00:00

coilyco-ops commented

2026-08-19 01:21:44 +00:00

Member

Closes #933.

First, the question #933 was left open on

Darren's note there says the issue is not one of the six because the #939 work would very likely close it too, and asks for it to be said either way. Said: #939 closed at 22:09Z and does not close this. Its fix was the queue-timeout notice throttle, on the path where a turn never entered the pipeline. This defect is downstream of a model round that already returned 200. Different path, different cause, no overlap.

The defect, from the source

Three paths produced the member-facing string model backend unavailable, retry shortly on a round the backend had answered:

proxy.go:1173 - a body past maxAgentProxyResponseBytes
modelstream.go:119 - a stream past the same bound
proxy.go:1179 - a body that would not decode

All three returned a bare fmt.Errorf carrying no sentinel. turnFailureNotice matches on sentinels first and then falls through to a switch on the stage, and the stage here is model, so all three collected noticeModelFailed. failureCause fell through the same way to stage_failed.

That is exactly the trace in the issue: model.response status: 200 at 00:43:27, turn.stage.failed with stage: model one second later, and a 90-byte notice telling the member the backend was down.

The fix

Two sentinels, in the idiom ErrToolRoundsExhausted and ErrBudgetExhausted already set for the same class of problem ("the backend answered, so it must never read as an outage"):

ErrResponseTooLarge → answer too long to read, ask for something narrower. Narrowing genuinely shortens the answer, so the advice is actionable.
ErrResponseUnreadable → model backend sent an unreadable answer, retry shortly. Not the member's to fix, and retrying is honest because a malformed body is plausibly transient.

Both sit ahead of the stage switch, and both get their own failureCause label, so an oversized answer and an outage cannot collapse into one series on a dashboard. That collapse is what the issue names as the reason an operator could not tell them apart.

What deliberately did not change

A transport failure and a 5xx keep model backend unavailable. That phrase now means only "the backend did not answer", which is the second half of this issue's acceptance.
A slow drain was already correct. It wraps context.DeadlineExceeded, which is matched ahead of everything here, so it reports as a timeout. A test pins that so this change cannot quietly move it.
No routing, no roster, no model behaviour, no bounds. The bytes at which a response is refused are unchanged.

Verification

just gate passes: build, policy-check, vet, test, test-skips, pre-commit. go test -race ./... clean.

Six new tests. Checked non-vacuous by reverting the two classification cases rather than assuming, and both fail with the exact string from the issue's trace:

--- FAIL: TestAnOversizedAnswerIsNotReportedAsAnOutage
    a 200 whose body was too long still reports as "> `model backend unavailable, retry shortly`",
    which sends a member to retry against a healthy backend

The other four pin the halves that must not move: a genuine outage still says unavailable, a 502 still says unavailable, a slow answer still reports as a timeout, no notice for an answered round contains the word "unavailable", and the cause labels are all distinct.

Not established

Whether this is what the owl.glass member actually hit. The issue's trace shows Agent Proxy response exceeded the size limit. on model.chat in the same window, which is the oversize path, so it is the strong candidate. I read the classification path rather than reproducing the trace, and #933 itself marks the oversize-to-model_failed link as inference. This change makes all three paths honest regardless of which one fired.

No live verification. Acceptance is a repeat of that window showing no availability notice on a 200.

One thing for review, not part of the fix

docs/sirens-echo-delivery.md was at 7976 of its 8000-character cap, leaving 24 characters to document a new failure class. To fit, I replaced the paragraph's inline enumeration of six notice phrases with the classification rule. The enumeration duplicated notice.go, which the same paragraph already names as the home of every phrase, and it grows by one entry per condition, which is the mechanism that filled this page.

This is the second page in a row I have hit this on (sirens-echo-commands.md was at 7958/8000 on #1001). The docs count is also at 40 of 40, so a new page is not available either. Worth a decision about the band rather than another round of shaving prose.

Closes #933. ## First, the question #933 was left open on Darren's note there says the issue is not one of the six because the #939 work would very likely close it too, and asks for it to be said either way. **Said: #939 closed at 22:09Z and does not close this.** Its fix was the queue-timeout notice throttle, on the path where a turn never entered the pipeline. This defect is downstream of a model round that already returned 200. Different path, different cause, no overlap. ## The defect, from the source Three paths produced the member-facing string `model backend unavailable, retry shortly` on a round the backend had answered: * `proxy.go:1173` - a body past `maxAgentProxyResponseBytes` * `modelstream.go:119` - a stream past the same bound * `proxy.go:1179` - a body that would not decode All three returned a bare `fmt.Errorf` carrying no sentinel. `turnFailureNotice` matches on sentinels first and then falls through to a switch on the stage, and the stage here is `model`, so all three collected `noticeModelFailed`. `failureCause` fell through the same way to `stage_failed`. That is exactly the trace in the issue: `model.response status: 200` at 00:43:27, `turn.stage.failed` with `stage: model` one second later, and a 90-byte notice telling the member the backend was down. ## The fix Two sentinels, in the idiom `ErrToolRoundsExhausted` and `ErrBudgetExhausted` already set for the same class of problem ("the backend answered, so it must never read as an outage"): * `ErrResponseTooLarge` → `answer too long to read, ask for something narrower`. Narrowing genuinely shortens the answer, so the advice is actionable. * `ErrResponseUnreadable` → `model backend sent an unreadable answer, retry shortly`. Not the member's to fix, and retrying is honest because a malformed body is plausibly transient. Both sit ahead of the stage switch, and both get their own `failureCause` label, so an oversized answer and an outage cannot collapse into one series on a dashboard. That collapse is what the issue names as the reason an operator could not tell them apart. ## What deliberately did not change * **A transport failure and a 5xx keep `model backend unavailable`.** That phrase now means only "the backend did not answer", which is the second half of this issue's acceptance. * **A slow drain was already correct.** It wraps `context.DeadlineExceeded`, which is matched ahead of everything here, so it reports as a timeout. A test pins that so this change cannot quietly move it. * No routing, no roster, no model behaviour, no bounds. The bytes at which a response is refused are unchanged. ## Verification `just gate` passes: build, policy-check, vet, test, test-skips, pre-commit. `go test -race ./...` clean. Six new tests. **Checked non-vacuous by reverting the two classification cases rather than assuming**, and both fail with the exact string from the issue's trace: ``` --- FAIL: TestAnOversizedAnswerIsNotReportedAsAnOutage a 200 whose body was too long still reports as "> `model backend unavailable, retry shortly`", which sends a member to retry against a healthy backend ``` The other four pin the halves that must not move: a genuine outage still says unavailable, a 502 still says unavailable, a slow answer still reports as a timeout, no notice for an answered round contains the word "unavailable", and the cause labels are all distinct. ## Not established **Whether this is what the owl.glass member actually hit.** The issue's trace shows `Agent Proxy response exceeded the size limit.` on `model.chat` in the same window, which is the oversize path, so it is the strong candidate. I read the classification path rather than reproducing the trace, and #933 itself marks the oversize-to-`model_failed` link as inference. This change makes all three paths honest regardless of which one fired. **No live verification.** Acceptance is a repeat of that window showing no availability notice on a 200. ## One thing for review, not part of the fix `docs/sirens-echo-delivery.md` was at **7976 of its 8000-character cap**, leaving 24 characters to document a new failure class. To fit, I replaced the paragraph's inline enumeration of six notice phrases with the classification rule. The enumeration duplicated `notice.go`, which the same paragraph already names as the home of every phrase, and it grows by one entry per condition, which is the mechanism that filled this page. **This is the second page in a row I have hit this on** (`sirens-echo-commands.md` was at 7958/8000 on #1001). The docs count is also at 40 of 40, so a new page is not available either. Worth a decision about the band rather than another round of shaving prose.

coilyco-ops added 1 commit

2026-08-19 01:21:45 +00:00

fix(notice): stop calling an answered round a backend outage

ci / image-build (pull_request) Successful in 38s

Details

ci / test (pull_request) Successful in 1m5s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

3a835a3354

A model round returning HTTP 200 could be reported to a member as
"model backend unavailable, retry shortly". Three paths reached it: a
body past the read bound, a stream past the same bound, and a body that
would not decode. All three returned a bare fmt.Errorf, so every special
case in turnFailureNotice missed and the stage switch supplied the
availability phrase for the model stage.

The message was false, it sent a member to retry against a healthy
service, and it made a real outage indistinguishable from a
misclassification in both the channel and the failure series.

Two sentinels now mark them, in the idiom ErrToolRoundsExhausted and
ErrBudgetExhausted already set: an answer past the bound asks for
something narrower, and one that would not parse says the backend sent
something unreadable. Both sit ahead of the stage switch and both get
their own failure cause, so the two conditions cannot collapse into each
other on a dashboard.

A transport failure and a 5xx keep the availability phrase, which is now
the only thing that phrase means. A slow drain was already classified as
a timeout and still is.

Closes #933.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-19 01:21:45 +00:00

coilyco-ops referenced this pull request

2026-08-19 01:22:03 +00:00

A member is told "model backend unavailable" one second after the model returned HTTP 200 #933