Audit tool responses for caveat-last ordering: a truncating consumer destroys trailing coverage fields first, so a bounded result arrives looking complete #18

New issue

Open

opened 2026-08-15 15:50:52 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-15 15:50:52 +00:00

Member

Filed by Darren (director seat) at Kai's direction, 2026-08-15. Cross-cutting finding from coilyco-gaming/sirens-echo#449, filed on every MCP server in the portfolio because the exposure is structural rather than specific to the server that surfaced it. Kai's words: this is an issue across all her MCPs because she was not aware of it being a problem.

First, a scoping check. I routed this here because a live steam MCP surface exists in the fleet (get_owned_games, get_recently_played, get_store_search_results, get_pics_product_info, get_account_licenses) and this repo's history carries fix: wait for Steam tool-specific readiness. I did not read the tree to confirm the server lives here — this repo has no description or topics, so the routing is inference. If the Steam MCP lives elsewhere, close this and say where, and I will refile.

The failure, concretely

A community agent told a member:

Currently 0 wooden hull planks are listed for sale on the server. No active store shelves or priced trade history exists for this item.

The tool result it built that from carried its own caveat:

22405 older trades arrive as 3152 hourly rollups; party, item, store,
and unit-price views cover detailed rows only
No markets matched item='wooden hull plank' across 528 ledger rows

The surface said it searched 528 of 22,933 rows. The reply said none exists. Those are different claims and only the first is true. It cost a day of investigation into a query path that was fine.

The mechanism that generalizes

This is arithmetic, not model behaviour, which is why it reaches this repo.

A consuming harness bounds tool results at a byte cap. The consumer measured in #449 does it as a head slice — keep the front, discard the tail, append a byte-count notice:

return result[:cut] + fmt.Sprintf(truncationNotice, cut, len(result)), cut, true

The server that surfaced this serializes warnings as its last JSON key. So whenever its response exceeds the cap, the caveats are the first thing removed, deterministically. The model receives rows with no caveat and answers as though the view were complete.

Two properties make this nastier than an ordinary truncation bug:

The notice is in bytes, not in meaning. The model learns "this was cut" and never learns "the search covered 528 of 22,933 rows". Only the second changes the answer.
The cap is per-consumer. In the observed case one profile capped at 8192 and another at 16384. The same call is honest on one and silently caveat-free on the other, and nothing in either log says so.

Why this surface is a live candidate rather than a formality

A Steam library is one of the largest arrays in the portfolio, and the failure lands in the same domain as the originating bug — an agent answering a gaming question with false completeness.

get_owned_games is unbounded by account size. A large library truncated mid-array, with the count trailing, reads as a complete library. An agent then says "you do not own that game", which is the exact wrong-shaped claim above.
Store search is a window. "No results for X" from a bounded search is not the same claim as "X does not exist", and only the first is true.
get_pics_product_info returns deeply nested product metadata, which is the shape that blows a byte cap on a single item rather than on a long list.

The four invariants

Two are already written up from the other direction in coilyco-gaming/eco-app#266 and #267, worth reading before implementing.

Coverage and caveat metadata serialize first, never last. A field that explains the bound of a result must not sit where a head slice removes it. This is the new finding.
limit bounds every unbounded array, not one of them. eco-app#267 documents a tool returning ~45 KB at limit=1 because limit bounded one array of six.
Truncation always warns, naming shown-of-total. Silent truncation is prohibited. A count is not a caveat: "3152 of 22933" is actionable, "truncated" is not.
An unreadable dataset reports null, never zero. A private profile, a rate-limited call, or an expired key must not serialize the same as an account that genuinely owns nothing. Steam profile privacy makes this a routine case here rather than an edge one.

What to do here

Confirm the scoping question at the top before anything else.
Audit every tool against the four rules, sizing the largest response each can emit rather than a typical one. A thousand-game library is the worst case, not a typical one.
Fix ordering first if any coverage field trails bulk payload. In Python this is construction order, so it is cheap.

Acceptance

No tool response places coverage or caveat fields after bulk arrays.
No tool can return an unbounded collection.
Truncation emits shown-of-total, and a private or failed read is distinguishable from a real zero.
Findings recorded even where no change was needed, so the next person does not re-audit.

Source: coilyco-gaming/sirens-echo#449 and its seven-comment thread, which establishes the head-slice mechanism, the per-profile cap difference, and why a consumer-side check is hard. The consumer-side half stays open there and is not what this issue asks for. Sibling issues filed on the other portfolio MCPs; coilyco-flight-deck/mcp-beaver#68 carries the generator-level version.

**Filed by Darren (director seat) at Kai's direction, 2026-08-15.** Cross-cutting finding from coilyco-gaming/sirens-echo#449, filed on every MCP server in the portfolio because the exposure is structural rather than specific to the server that surfaced it. Kai's words: this is an issue across all her MCPs because she was not aware of it being a problem. **First, a scoping check.** I routed this here because a live `steam` MCP surface exists in the fleet (`get_owned_games`, `get_recently_played`, `get_store_search_results`, `get_pics_product_info`, `get_account_licenses`) and this repo's history carries `fix: wait for Steam tool-specific readiness`. I did **not** read the tree to confirm the server lives here — this repo has no description or topics, so the routing is inference. If the Steam MCP lives elsewhere, close this and say where, and I will refile. ## The failure, concretely A community agent told a member: > Currently 0 wooden hull planks are listed for sale on the server. No active store shelves or priced trade history exists for this item. The tool result it built that from carried its own caveat: ``` 22405 older trades arrive as 3152 hourly rollups; party, item, store, and unit-price views cover detailed rows only No markets matched item='wooden hull plank' across 528 ledger rows ``` The surface said it searched 528 of 22,933 rows. The reply said none exists. Those are different claims and only the first is true. It cost a day of investigation into a query path that was fine. ## The mechanism that generalizes This is arithmetic, not model behaviour, which is why it reaches this repo. A consuming harness bounds tool results at a byte cap. The consumer measured in #449 does it as a **head slice** — keep the front, discard the tail, append a byte-count notice: ```go return result[:cut] + fmt.Sprintf(truncationNotice, cut, len(result)), cut, true ``` The server that surfaced this serializes `warnings` as its **last** JSON key. So whenever its response exceeds the cap, the caveats are the first thing removed, deterministically. The model receives rows with no caveat and answers as though the view were complete. Two properties make this nastier than an ordinary truncation bug: * **The notice is in bytes, not in meaning.** The model learns "this was cut" and never learns "the search covered 528 of 22,933 rows". Only the second changes the answer. * **The cap is per-consumer.** In the observed case one profile capped at 8192 and another at 16384. The same call is honest on one and silently caveat-free on the other, and nothing in either log says so. ## Why this surface is a live candidate rather than a formality A Steam library is one of the largest arrays in the portfolio, and the failure lands in the same domain as the originating bug — an agent answering a gaming question with false completeness. * **`get_owned_games` is unbounded by account size.** A large library truncated mid-array, with the count trailing, reads as a complete library. An agent then says "you do not own that game", which is the exact wrong-shaped claim above. * **Store search is a window.** "No results for X" from a bounded search is not the same claim as "X does not exist", and only the first is true. * **`get_pics_product_info` returns deeply nested product metadata**, which is the shape that blows a byte cap on a single item rather than on a long list. ## The four invariants Two are already written up from the other direction in coilyco-gaming/eco-app#266 and #267, worth reading before implementing. 1. **Coverage and caveat metadata serialize first, never last.** A field that explains the bound of a result must not sit where a head slice removes it. This is the new finding. 2. **`limit` bounds every unbounded array, not one of them.** eco-app#267 documents a tool returning ~45 KB at `limit=1` because `limit` bounded one array of six. 3. **Truncation always warns, naming shown-of-total.** Silent truncation is prohibited. A count is not a caveat: "3152 of 22933" is actionable, "truncated" is not. 4. **An unreadable dataset reports null, never zero.** A private profile, a rate-limited call, or an expired key must not serialize the same as an account that genuinely owns nothing. Steam profile privacy makes this a routine case here rather than an edge one. ## What to do here * Confirm the scoping question at the top before anything else. * Audit every tool against the four rules, sizing the **largest** response each can emit rather than a typical one. A thousand-game library is the worst case, not a typical one. * Fix ordering first if any coverage field trails bulk payload. In Python this is construction order, so it is cheap. ## Acceptance * No tool response places coverage or caveat fields after bulk arrays. * No tool can return an unbounded collection. * Truncation emits shown-of-total, and a private or failed read is distinguishable from a real zero. * Findings recorded even where no change was needed, so the next person does not re-audit. --- Source: coilyco-gaming/sirens-echo#449 and its seven-comment thread, which establishes the head-slice mechanism, the per-profile cap difference, and why a consumer-side check is hard. The consumer-side half stays open there and is not what this issue asks for. Sibling issues filed on the other portfolio MCPs; coilyco-flight-deck/mcp-beaver#68 carries the generator-level version.

coilyco-ops added the

autonomy

headless

label

2026-08-15 15:51:40 +00:00

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-15 15:52:38 +00:00

A tool result that says it saw 2 percent of the data becomes a reply that says none exists #449

coilyco-ops added the

role/engineer

priority

labels