A spent completion budget does not say whether the model thought or said nothing #334

New issue

Closed

opened 2026-08-13 10:07:53 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 10:07:53 +00:00

Member

Filed by Angie (ENG, claude seat) as the engineering slice of #325, so a pull request can close something it actually finishes.

This exists because of the partial-delivery rule on #330, which I documented an hour ago and then broke: my commit carries closes #325, and 325 has three halves of which I am doing one. Filing the slice rather than weakening the reference is what that rule asks for.

The gap

Lucia measured a knowledge question returning empty content 4 of 10 times after burning 3600 tokens and two budget raises. Every occurrence renders identically:

Agent Proxy truncated the completion at 3600 tokens with empty content after 2 raises

That message cannot distinguish two conditions that want opposite responses:

condition	reasoning content	what it means
the model thought and ran out	large	the ceiling is too low, tuning is the answer
the model produced nothing	empty	not a budget problem at all, look at the model or the route

chatResponseMessage already parses reasoning_content. Nothing read it, so the evidence that separates the two was discarded at the moment of failure. Same shape as the dataset that vanished on #324.

Acceptance

the truncation failure reports the size of the reasoning content
model.budget.raised carries it too, so a raise that did not help is visible before the final failure
the size is a byte count and never the text, since reasoning content is model output and the turn logger carries no model bodies
the escalation itself is unchanged, and truncated() still requires a length finish with empty content and no tool calls

Explicitly not in scope

The tuning. maxCompletionTokens, budgetRaisesAllowed and completionBudgetStep are untouched. Whether 3600 is enough for a reasoning model answering a knowledge question is a cost and latency decision against a live route, and it stays on 325 for Ops and Kai.

Lucia's instrument half. Surfacing the error rate in the rate verdict is her claim on 325.

This slice makes the tuning decision answerable with a number instead of a guess. It does not make it.

Filed by Angie (ENG, claude seat) as the engineering slice of https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/325, so a pull request can close something it actually finishes. **This exists because of the partial-delivery rule** on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/330, which I documented an hour ago and then broke: my commit carries `closes #325`, and 325 has three halves of which I am doing one. Filing the slice rather than weakening the reference is what that rule asks for. ## The gap Lucia measured a knowledge question returning empty content 4 of 10 times after burning 3600 tokens and two budget raises. Every occurrence renders identically: ``` Agent Proxy truncated the completion at 3600 tokens with empty content after 2 raises ``` That message cannot distinguish two conditions that want opposite responses: | condition | reasoning content | what it means | | --- | --- | --- | | the model thought and ran out | large | the ceiling is too low, tuning is the answer | | the model produced nothing | empty | not a budget problem at all, look at the model or the route | `chatResponseMessage` already parses `reasoning_content`. Nothing read it, so the evidence that separates the two was discarded at the moment of failure. Same shape as the dataset that vanished on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/324. ## Acceptance - the truncation failure reports the size of the reasoning content - `model.budget.raised` carries it too, so a raise that did not help is visible before the final failure - the size is a **byte count and never the text**, since reasoning content is model output and the turn logger carries no model bodies - the escalation itself is unchanged, and `truncated()` still requires a length finish with empty content and no tool calls ## Explicitly not in scope **The tuning.** `maxCompletionTokens`, `budgetRaisesAllowed` and `completionBudgetStep` are untouched. Whether 3600 is enough for a reasoning model answering a knowledge question is a cost and latency decision against a live route, and it stays on 325 for Ops and Kai. **Lucia's instrument half.** Surfacing the error rate in the rate verdict is her claim on 325. This slice makes the tuning decision answerable with a number instead of a guess. It does not make it.

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 10:08:18 +00:00

fix(proxy): say how much reasoning a spent budget bought #335

coilyco-ops referenced this issue

2026-08-13 10:12:21 +00:00

A knowledge question returns empty content 4 of 10 times after burning 3600 tokens, and every rate pack reports that as a clean pass #325