A spent completion budget stops reading as a backend outage #807

Merged

coilysiren merged 1 commit from aos/claude/xh55-budget-spent-cause into main

2026-08-15 16:19:52 +00:00

coilyco-ops commented

2026-08-15 15:58:39 +00:00

Member

Closes #549 by taking its second acceptance branch, which needs no ceiling decision.

What was actually broken

The issue said the runner is right to report an empty-content turn as a failure, and that is correct. What was wrong is what the failure said.

formatBudgetExhausted returned a bare fmt.Errorf with no sentinel, so nothing downstream could identify the condition. failureCause could not see it and it fell through to stage_failed. The stage is model, so turnFailureNotice handed the member:

> `model backend unavailable, retry shortly`

Every model call returned 200. That notice sends an operator to a working backend and tells a member to retry a question that will fail identically. It is the same defect #258 fixed for spent tool rounds and #651 fixed for a refused reply, arriving from a third direction — and per @Quail's capture on this issue it reaches members, not only batteries.

The change

ErrBudgetExhausted sentinel, wrapped by formatBudgetExhausted so the rendered numbers survive.
budget_spent joins the failure_cause closed set.
The member reads ran out of room to answer, ask for something narrower.

rounds_spent and budget_spent stay separate causes. Both mean a ceiling this service chose ended the turn, but they are different numbers with different owners (tool_rounds against max_completion_tokens), and #367 is a ceiling decision that has to be answerable from the failure series.

failure_cause is a log attribute rather than a metric label, so the set grows without touching a cardinality bound. Its doc had listed five values and been missing reply_refused since #651; it now lists all seven.

Addressing @Angie's objection directly

the mirror of that rule is not silently reclassifying a red into a non-failure. If the model genuinely fails to answer, a battery that stays green is worse than one that goes red for the wrong reason.

Agreed, and nothing here goes green. The turn still fails, the battery still goes red, and TestAnAlwaysTruncatingProxyEndsTheTurnAsBudgetSpent asserts the error is still returned. Only what the red says changes. That is what the issue's own acceptance asked for: "characterised as its own behaviour... because a substrate event is not the agent failing a check."

What this does not do

Does not bound deliberation or raise any ceiling. #367 stays open and untouched.
Does not fix prompt-leakage, which truncates 34 times a day. @Quail was explicit about not having read that family's reasoning to the end, and a model that deliberates without converging still exhausts its budget. This changes the report, not the behaviour.
Does not resolve the model.budget.raised contradiction. I read the plumbing: finish_reason decodes correctly at proxy.go:977 and carries into truncated() intact, so the harness side is sound and @Quail's agent-proxy inference survives my reading. One correction to that handoff though: model.chat spans are marked error only on transport and decode failures, so a truncated response is a successful span. The 20-errors-against-366-truncations comparison was never a contradiction, which leaves fewer facts needing one explanation.

Verification

ward exec build, vet, test, policy-check, test-skips, and pre-commit run --all-files all pass.

Six new tests. The end-to-end one drives Complete against a proxy that only ever returns finish_reason: length with empty content, and asserts it makes exactly BudgetRaises + 1 calls before failing — so it genuinely climbs the ladder rather than passing vacuously.

Closes #549 by taking its second acceptance branch, which needs no ceiling decision. ## What was actually broken The issue said the runner is right to report an empty-content turn as a failure, and that is correct. What was wrong is **what the failure said**. `formatBudgetExhausted` returned a bare `fmt.Errorf` with no sentinel, so nothing downstream could identify the condition. `failureCause` could not see it and it fell through to `stage_failed`. The stage is `model`, so `turnFailureNotice` handed the member: ``` > `model backend unavailable, retry shortly` ``` Every model call returned 200. That notice sends an operator to a working backend and tells a member to retry a question that will fail identically. It is the same defect #258 fixed for spent tool rounds and #651 fixed for a refused reply, arriving from a third direction — and per @Quail's capture on this issue it reaches **members**, not only batteries. ## The change * `ErrBudgetExhausted` sentinel, wrapped by `formatBudgetExhausted` so the rendered numbers survive. * `budget_spent` joins the `failure_cause` closed set. * The member reads `ran out of room to answer, ask for something narrower`. `rounds_spent` and `budget_spent` stay separate causes. Both mean a ceiling this service chose ended the turn, but they are different numbers with different owners (`tool_rounds` against `max_completion_tokens`), and #367 is a ceiling decision that has to be answerable from the failure series. `failure_cause` is a log attribute rather than a metric label, so the set grows without touching a cardinality bound. Its doc had listed five values and been missing `reply_refused` since #651; it now lists all seven. ## Addressing @Angie's objection directly > the mirror of that rule is not silently reclassifying a red into a non-failure. If the model genuinely fails to answer, a battery that stays green is worse than one that goes red for the wrong reason. Agreed, and nothing here goes green. The turn still fails, the battery still goes red, and `TestAnAlwaysTruncatingProxyEndsTheTurnAsBudgetSpent` asserts the error is still returned. Only what the red *says* changes. That is what the issue's own acceptance asked for: "characterised as its own behaviour... because a substrate event is not the agent failing a check." ## What this does not do * **Does not bound deliberation or raise any ceiling.** #367 stays open and untouched. * **Does not fix `prompt-leakage`**, which truncates 34 times a day. @Quail was explicit about not having read that family's reasoning to the end, and a model that deliberates without converging still exhausts its budget. This changes the report, not the behaviour. * **Does not resolve the `model.budget.raised` contradiction.** I read the plumbing: `finish_reason` decodes correctly at `proxy.go:977` and carries into `truncated()` intact, so the harness side is sound and @Quail's agent-proxy inference survives my reading. One correction to that handoff though: `model.chat` spans are marked error only on transport and decode failures, so a truncated response is a *successful* span. The 20-errors-against-366-truncations comparison was never a contradiction, which leaves fewer facts needing one explanation. ## Verification `ward exec build`, `vet`, `test`, `policy-check`, `test-skips`, and `pre-commit run --all-files` all pass. Six new tests. The end-to-end one drives `Complete` against a proxy that only ever returns `finish_reason: length` with empty content, and asserts it makes exactly `BudgetRaises + 1` calls before failing — so it genuinely climbs the ladder rather than passing vacuously.

coilyco-ops added 1 commit

2026-08-15 15:58:39 +00:00

fix(notice): a spent completion budget stops reading as a backend outage

ci / image-build (pull_request) Successful in 25s

Details

ci / test (pull_request) Successful in 48s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

5452f76557

A reasoning model can spend its whole completion budget on reasoning_content
and return empty content with finish_reason length. The turn produced nothing,
so it fails, and reporting that as a failure is correct. What was wrong is what
the failure said.

formatBudgetExhausted returned a bare fmt.Errorf, so nothing downstream could
identify the condition. failureCause could not see it and it landed in
stage_failed; the stage is model, so turnFailureNotice handed the member
"model backend unavailable, retry shortly".

Every model call returned 200. That notice sends an operator to a working
backend and tells a member to retry a question that will fail the same way.
It is the defect #258 fixed for spent tool rounds and #651 fixed for a refused
reply, arriving from a third direction and reaching members: one capture on
#549 was a real Discord turn, not a battery case.

ErrBudgetExhausted now marks it, budget_spent counts it, and the member reads
"ran out of room to answer, ask for something narrower", which is the move that
works.

rounds_spent and budget_spent stay separate. Both mean a ceiling this service
chose ended the turn, but they are different numbers with different owners,
tool_rounds against max_completion_tokens, and #367 is a ceiling decision that
has to be answerable from the failure series.

failure_cause is a log attribute rather than a metric label, so the closed set
grows without touching a cardinality bound. Its doc listed five values and had
been missing reply_refused since #651; it now lists all seven.

WHAT THIS DOES NOT DO

It does not stop the turn failing, and it should not. A member asked a question
and got nothing, so a battery going green here would be worse than one going
red for the wrong reason. Only what the red says changes.

It does not bound deliberation or raise any ceiling. #367 stays open, and the
prompt-leakage family that truncates 34 times a day is unaffected.

Closes #549

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Angie <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>