fix(scratch): bound a search by bytes, not only by match count #1016

Merged

coilysiren merged 1 commit from aos/claude/bound-scratch-search-2 into main

2026-08-19 02:00:19 +00:00

coilyco-ops commented

2026-08-19 01:56:34 +00:00

Member

Closes the half of #940 that lives in this repo. Stacked on #1014, which unblocks a red main; a branch off main cannot pass its own gate until that lands.

Ask 2 is already implemented, and that is the more useful finding

#940 asks to "render the reachable host set into the fetch tool's description so a refusal is predictable rather than discovered," and says "I would start there."

It is already there, in internal/community/fetch.go:

Description: "Fetch a page over HTTPS and return its text. Only these hosts " +
    "are reachable: " + strings.Join(s.hosts, ", ") + ". Nothing else is.",

Landed in 4795f1c on 2026-08-13, five days before the 2026-08-18 incident window. So the model was told the allowlist, in the tool schema, and fetched unreachable hosts anyway.

That does not make the finding wrong: six fetches returning 56, 130, 130, 130, 24, 24 bytes is real waste. It makes the proposed fix wrong, and the remaining option ("filter at the search boundary") belongs to the exa MCP server rather than this repo. I did not implement it here, and I would not guess at it without measuring what the model actually did with the allowlist it was given.

Ask 1, fixed where it is this repo's to fix

scratch_search was bounded by maxScratchMatches, a count of matching lines and nothing else. A hundred matches from long lines is arbitrarily large, which is how it returned 131,343 bytes into a 16,384 byte consumer with 114,959 dropped.

Now bounded at the source:

Bytes, against maxToolResultBytes, the same budget the consumer holds. The walk stops rather than collecting and discarding.
One line, at a new SIRENS_ECHO_SCRATCH_MATCH_RUNES (240). A minified file is one line, so without this a single hit could be the whole result.
Said when it stopped, because a silently short result reads as a complete one.

scratch_read is untouched. Trimming a file the model explicitly asked to read loses the same information at the source that it loses downstream, so the win is not the same, and #940's own framing is about results that were "paid for and then dropped".

list_channel-message and the forgejo list_* family are MCP servers, not this repo.

Reproduced before fixing

The test corpus returns 168,539 bytes without these bounds, the same shape as the 131,343 recorded in production. Checked by reverting each bound separately:

search returned 168539 bytes against a 8192 byte tool budget
one match rendered 20016 bytes against a 240 rune line bound

Three tests: the byte bound trips below the match cap, one long match cannot be the whole result, and an ordinary small search is unchanged and claims no truncation — which is the assertion that stops the bound quietly eating normal answers.

On acceptance

#940 asks for "no single tool result above 32KB before bounding". This delivers that for scratch_search, which is the largest offender in the record, and cannot deliver it for the MCP servers from inside this repo. The acceptance is not fully met by this PR and the remainder is not mine to close.

Verification

just gate passes: build, policy-check, vet, test, test-skips, pre-commit.

Closes the half of #940 that lives in this repo. **Stacked on #1014**, which unblocks a red `main`; a branch off `main` cannot pass its own gate until that lands. ## Ask 2 is already implemented, and that is the more useful finding #940 asks to "render the reachable host set into the fetch tool's description so a refusal is predictable rather than discovered," and says "I would start there." **It is already there**, in `internal/community/fetch.go`: ```go Description: "Fetch a page over HTTPS and return its text. Only these hosts " + "are reachable: " + strings.Join(s.hosts, ", ") + ". Nothing else is.", ``` Landed in `4795f1c` on **2026-08-13**, five days before the 2026-08-18 incident window. So the model was told the allowlist, in the tool schema, and fetched unreachable hosts anyway. That does not make the finding wrong: six fetches returning 56, 130, 130, 130, 24, 24 bytes is real waste. It makes the **proposed fix** wrong, and the remaining option ("filter at the search boundary") belongs to the exa MCP server rather than this repo. I did not implement it here, and I would not guess at it without measuring what the model actually did with the allowlist it was given. ## Ask 1, fixed where it is this repo's to fix `scratch_search` was bounded by `maxScratchMatches`, **a count of matching lines and nothing else**. A hundred matches from long lines is arbitrarily large, which is how it returned 131,343 bytes into a 16,384 byte consumer with 114,959 dropped. Now bounded at the source: * **Bytes**, against `maxToolResultBytes`, the same budget the consumer holds. The walk stops rather than collecting and discarding. * **One line**, at a new `SIRENS_ECHO_SCRATCH_MATCH_RUNES` (240). A minified file is one line, so without this a single hit could be the whole result. * **Said when it stopped**, because a silently short result reads as a complete one. `scratch_read` is untouched. Trimming a file the model explicitly asked to read loses the same information at the source that it loses downstream, so the win is not the same, and #940's own framing is about results that were "paid for and then dropped". `list_channel-message` and the forgejo `list_*` family are MCP servers, not this repo. ## Reproduced before fixing The test corpus returns **168,539 bytes** without these bounds, the same shape as the 131,343 recorded in production. Checked by reverting each bound separately: ``` search returned 168539 bytes against a 8192 byte tool budget one match rendered 20016 bytes against a 240 rune line bound ``` Three tests: the byte bound trips below the match cap, one long match cannot be the whole result, and **an ordinary small search is unchanged and claims no truncation** — which is the assertion that stops the bound quietly eating normal answers. ## On acceptance #940 asks for "no single tool result above 32KB before bounding". This delivers that for `scratch_search`, which is the largest offender in the record, and cannot deliver it for the MCP servers from inside this repo. **The acceptance is not fully met by this PR** and the remainder is not mine to close. ## Verification `just gate` passes: build, policy-check, vet, test, test-skips, pre-commit.

coilyco-ops added 1 commit

2026-08-19 01:56:34 +00:00

fix(scratch): bound a search by bytes, not only by match count

ci / test (pull_request) Successful in 1m11s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 30s

Details

3f3d5286a8

scratch_search returned 131,343 bytes into a 16,384 byte consumer, 87% of
it dropped after the round trip, and the model then spent a further round
chasing the spill file for what it should have received first.

The cause is that the only bound was maxScratchMatches, a count of lines.
A hundred matches from long lines is arbitrarily large, and a minified
file is one line, so a single hit could be the whole result.

Bounded at the source now: bytes against the same budget the consumer
holds, one line trimmed at SIRENS_ECHO_SCRATCH_MATCH_RUNES, and a notice
when either stopped it. Truncating after transfer stays as the backstop
it was meant to be.

Reproduced before fixing rather than assumed. The corpus in the test
returns 168,539 bytes without these bounds, which is the same shape as
the production record.

Stacked on the main-unblocking fix in #1014, since a branch off main
cannot pass its own gate until that lands.

Closes part of #940.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-19 01:56:34 +00:00

coilyco-ops referenced this pull request

2026-08-19 01:56:55 +00:00

Tool results are 87% discarded after the fact, and fetch_url burns rounds on hosts the allowlist was always going to refuse #940