test(budget): a round runs every call, so a turn's total is unbounded #738

Merged

coilyco-ops merged 1 commit from quail/round-fanout-unbounded into main

2026-08-13 21:34:01 +00:00

coilyco-ops commented

2026-08-13 21:31:25 +00:00

Member

Pins the factor that makes sirens-echo#635's first criterion necessary. Tests only; adds no cap.

The gap

tool_rounds: 12            bounds ROUNDS
tool_result_bytes: 16384   bounds ONE CALL
for _, call := range message.ToolCalls   <- bounds nothing

The turn ceiling is rounds x calls-per-round x 16384 and the middle term has no limit. Measured on production traffic: one trace made 25 tool calls under a 12-round limit and touched 1,055,907 bytes of tool output.

The test

A round requesting twenty calls, asserting all twenty run:

a round of 20 calls delivered 5 tool results. If a per-round cap landed,
sirens-echo#635 is being acted on and this test should assert the cap instead

That is the mutation output from capping the loop at five. Behavioural rather than a grep for absence, so it fails on the change rather than on a rename.

Why assert the defect

A cap is one of two shapes sirens-echo#635's first criterion could take, and choosing between them is not mine:

cap calls per round — cheap, blunt, refuses a legitimate wide fan-out
accumulate reinjected bytes across the turn — bounds the product, needs a number derived from the model's window

Either makes this test fail, and its message says so. The test does not presume which.

Interaction worth flagging

sirens-echo#725 lets a tool raise its own bound. Raising bounds for the nine oversized tools multiplies against this unbounded count — the second criterion can make the first worse, and nothing currently notices.

ward exec gate PASS on every step.

Pins the factor that makes sirens-echo#635's first criterion necessary. Tests only; adds no cap. ## The gap ``` tool_rounds: 12 bounds ROUNDS tool_result_bytes: 16384 bounds ONE CALL for _, call := range message.ToolCalls <- bounds nothing ``` The turn ceiling is `rounds x calls-per-round x 16384` and the middle term has no limit. Measured on production traffic: one trace made **25 tool calls under a 12-round limit** and touched 1,055,907 bytes of tool output. ## The test A round requesting twenty calls, asserting all twenty run: ``` a round of 20 calls delivered 5 tool results. If a per-round cap landed, sirens-echo#635 is being acted on and this test should assert the cap instead ``` That is the mutation output from capping the loop at five. Behavioural rather than a grep for absence, so it fails on the change rather than on a rename. ## Why assert the defect A cap is one of two shapes sirens-echo#635's first criterion could take, and choosing between them is not mine: - **cap calls per round** — cheap, blunt, refuses a legitimate wide fan-out - **accumulate reinjected bytes across the turn** — bounds the product, needs a number derived from the model's window Either makes this test fail, and its message says so. The test does not presume which. ## Interaction worth flagging sirens-echo#725 lets a tool raise its own bound. Raising bounds for the nine oversized tools multiplies against this unbounded count — **the second criterion can make the first worse**, and nothing currently notices. `ward exec gate` PASS on every step.

coilyco-ops added 1 commit

2026-08-13 21:31:25 +00:00

test(budget): a round runs every call, so a turn's total is unbounded

ci / image-build (pull_request) Successful in 20s

Details

ci / test (pull_request) Successful in 38s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

f5e5c30505

tool_rounds bounds rounds and tool_result_bytes bounds one call. The loop
between them runs whatever the model asked for, so the turn ceiling is
rounds times an unbounded call count times the per-call bound.

One production turn made 25 calls under a 12-round limit and touched a
megabyte of tool output. Asserted as it ships, so a cap is deliberate.

Refs sirens-echo#635

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 21:31:25 +00:00

coilyco-ops referenced this pull request

2026-08-13 21:33:35 +00:00

Sized for a small model: 87% of a member's Steam library was discarded into a 16 KiB cap, on a turn that used 4.8% of the context window #635

coilyco-ops merged commit 3cf57b6401 into main

2026-08-13 21:34:01 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 21:34:02 +00:00

Merge pull request 'test(budget): a round runs every call, so a turn's total is unbounded' (#738) from quail/round-fanout-unbounded into main