Reasoning output exceeds max_tokens by 2x, forcing a budget raise and a re-run of the same request #937

New issue

Open

opened 2026-08-18 01:32:02 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-18 01:32:02 +00:00

Member

Filed by Olaf (ops seat) from the 2026-08-18 owl.glass incident review. Split out of #932.

What happens

Trace db9528828e4905992a04c3d1e7f58a74 on sirens-deep-owl-glass:

00:49:23  model.request         round: 6, request_bytes: 164691, tool_count: 0
00:49:51  model.response        round: 6, status: 200
00:49:51  model.budget.raised   attempt: 1, max_tokens: 7200, reasoning_bytes: 15036
00:49:51  model.request         round: 7, request_bytes: 164691, tool_count: 0
00:50:26  model.response        round: 7, status: 200

The model emitted 15,036 bytes of reasoning against a 7,200 max_tokens ceiling. The runtime raised the budget and reissued the identical request (request_bytes 164,691 both times), costing a second full round of 35 seconds.

Why it matters

deepseek-v4-flash is a reasoning model, so reasoning tokens are the expected output shape rather than an anomaly. A ceiling set below what the model routinely produces means the overrun path is the normal path, not the exception, and every trip through it doubles the cost of that round.

On a lane where community.turn p50 is already 32.9s (#932), a wasted 35-second round is a large fraction of the budget.

Worth measuring rather than guessing: if the overrun rate is high, the ceiling is simply mis-set. If it is rare, the retry is fine and this issue closes cheaply.

What to establish first

The observed reasoning-token distribution across a real window, not one trace. p50, p90, max.
How often model.budget.raised fires as a fraction of rounds.
Where 7,200 came from. If it was picked against a non-reasoning model, it predates the current backend and nothing has revisited it.

Then set the ceiling from the measurement, the same way #431 is an open call about rate-limit tiers that were set against a ceiling measurement did not support.

Shape of a fix

Raise max_tokens above the observed envelope so the common case completes in one round.
Or carry the partial forward rather than reissuing an identical request, if the API permits continuation.

The second is better if it exists, since it does not pay for the prefix twice. The first is trivially available now.

Done when

model.budget.raised is a rare event rather than a routine one, and no round is re-run byte-identical purely to widen a ceiling.

#932 - the incident review this came from
#431 - the same class of problem on rate-limit tiers, tiers set against an unmeasured ceiling

**Filed by Olaf (ops seat)** from the 2026-08-18 owl.glass incident review. Split out of #932. ## What happens Trace `db9528828e4905992a04c3d1e7f58a74` on `sirens-deep-owl-glass`: ``` 00:49:23 model.request round: 6, request_bytes: 164691, tool_count: 0 00:49:51 model.response round: 6, status: 200 00:49:51 model.budget.raised attempt: 1, max_tokens: 7200, reasoning_bytes: 15036 00:49:51 model.request round: 7, request_bytes: 164691, tool_count: 0 00:50:26 model.response round: 7, status: 200 ``` The model emitted **15,036 bytes of reasoning against a 7,200 `max_tokens` ceiling**. The runtime raised the budget and reissued **the identical request** (`request_bytes` 164,691 both times), costing a second full round of 35 seconds. ## Why it matters `deepseek-v4-flash` is a reasoning model, so reasoning tokens are the expected output shape rather than an anomaly. A ceiling set below what the model routinely produces means the overrun path is the **normal** path, not the exception, and every trip through it doubles the cost of that round. On a lane where `community.turn` p50 is already 32.9s (#932), a wasted 35-second round is a large fraction of the budget. Worth measuring rather than guessing: if the overrun rate is high, the ceiling is simply mis-set. If it is rare, the retry is fine and this issue closes cheaply. ## What to establish first 1. **The observed reasoning-token distribution** across a real window, not one trace. p50, p90, max. 2. **How often `model.budget.raised` fires** as a fraction of rounds. 3. **Where 7,200 came from.** If it was picked against a non-reasoning model, it predates the current backend and nothing has revisited it. Then set the ceiling from the measurement, the same way #431 is an open call about rate-limit tiers that were set against a ceiling measurement did not support. ## Shape of a fix * **Raise `max_tokens` above the observed envelope** so the common case completes in one round. * **Or carry the partial forward** rather than reissuing an identical request, if the API permits continuation. The second is better if it exists, since it does not pay for the prefix twice. The first is trivially available now. ## Done when `model.budget.raised` is a rare event rather than a routine one, and no round is re-run byte-identical purely to widen a ceiling. ## Related * #932 - the incident review this came from * #431 - the same class of problem on rate-limit tiers, tiers set against an unmeasured ceiling

coilyco-ops added the

labels

2026-08-18 01:32:02 +00:00