Audit tool responses for caveat-last ordering: a truncating consumer destroys trailing coverage fields first, so a bounded result arrives looking complete #12

New issue

Open

opened 2026-08-15 15:49:52 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-15 15:49:52 +00:00

Member

Filed by Darren (director seat) at Kai's direction, 2026-08-15. Cross-cutting finding from coilyco-gaming/sirens-echo#449, filed on every MCP server in the portfolio because the exposure is structural rather than specific to the server that surfaced it. Kai's words: this is an issue across all her MCPs because she was not aware of it being a problem.

The failure, concretely

A community agent told a member:

Currently 0 wooden hull planks are listed for sale on the server. No active store shelves or priced trade history exists for this item.

The tool result it built that from carried its own caveat:

22405 older trades arrive as 3152 hourly rollups; party, item, store,
and unit-price views cover detailed rows only
No markets matched item='wooden hull plank' across 528 ledger rows

The surface said it searched 528 of 22,933 rows. The reply said none exists. Those are different claims and only the first is true. It cost a day of investigation into a query path that was fine.

The mechanism that generalizes

This is arithmetic, not model behaviour, which is why it reaches this repo.

A consuming harness bounds tool results at a byte cap. The consumer measured in #449 does it as a head slice — keep the front, discard the tail, append a byte-count notice:

return result[:cut] + fmt.Sprintf(truncationNotice, cut, len(result)), cut, true

The server that surfaced this serializes warnings as its last JSON key. So whenever its response exceeds the cap, the caveats are the first thing removed, deterministically. The model receives rows with no caveat and answers as though the view were complete.

Two properties make this nastier than an ordinary truncation bug:

The notice is in bytes, not in meaning. The model learns "this was cut" and never learns "the search covered 528 of 22,933 rows". Only the second changes the answer.
The cap is per-consumer. In the observed case one profile capped at 8192 and another at 16384. The same call is honest on one and silently caveat-free on the other, and nothing in either log says so.

Measured in the sirens-echo consumer against the eco surface. Inferred for this server, which is why the first task below is an audit rather than a fix. A server cannot assume its trailing fields survive, and a server that stays under the cap never depends on which consumer read it.

Why this repo is a live candidate rather than a formality

The repo describes a fixed, bounded tool surface, which is the right posture and may mean rule 2 is already satisfied. Rule 1 is the one to check, and there is a specific hazard in the AT Protocol shape:

The cursor is the coverage field. A paginated response says "here is a page, and here is where the next one starts". If the cursor serializes after the item array and the response is truncated, the consumer loses the one field that says the view was partial, keeping the items that make it look whole. That is this bug exactly, in the idiom of this protocol.
Search and follower listings are windows. An agent concluding "nobody has posted about X" or "they do not follow anyone matching Y" from one page is the wrong-shaped claim above. Whatever states the bound has to survive the cut.

The four invariants

Two are already written up from the other direction in coilyco-gaming/eco-app#266 and #267, worth reading before implementing.

Coverage and caveat metadata serialize first, never last. Cursors, totals, and any "this is a partial view" field belong ahead of the item array. This is the new finding.
limit bounds every unbounded array, not one of them. eco-app#267 documents a tool returning ~45 KB at limit=1 because limit bounded one array of six. Nested arrays such as embeds, facets, and thread replies are the ones to check here, not just the top-level page.
Truncation always warns, naming shown-of-total. Silent truncation is prohibited. A count is not a caveat: "3152 of 22933" is actionable, "truncated" is not.
An unreadable dataset reports null, never zero. A blocked, deactivated, or takendown account must not serialize the same as an account with no posts.

What to do here

Audit every tool against the four rules, sizing the largest response each can emit rather than a typical one. A deep thread with rich embeds is the worst case, not an average post.
Confirm cursor and coverage fields lead the response. In TypeScript, JSON.stringify follows property insertion order for string keys, so this is an object-construction change.
Say explicitly whether the fixed surface already satisfies rule 2, so the claim in the README is backed by a check.

Acceptance

No tool response places cursors, coverage, or caveat fields after item arrays.
No tool can return an unbounded collection, including nested ones.
Truncation emits shown-of-total, and an inaccessible account is distinguishable from an empty one.
Findings recorded even where no change was needed, so the next person does not re-audit.

Source: coilyco-gaming/sirens-echo#449 and its seven-comment thread, which establishes the head-slice mechanism, the per-profile cap difference, and why a consumer-side check is hard. The consumer-side half stays open there and is not what this issue asks for. Sibling issues filed on the other portfolio MCPs; coilyco-flight-deck/mcp-beaver#68 carries the generator-level version.

**Filed by Darren (director seat) at Kai's direction, 2026-08-15.** Cross-cutting finding from coilyco-gaming/sirens-echo#449, filed on every MCP server in the portfolio because the exposure is structural rather than specific to the server that surfaced it. Kai's words: this is an issue across all her MCPs because she was not aware of it being a problem. ## The failure, concretely A community agent told a member: > Currently 0 wooden hull planks are listed for sale on the server. No active store shelves or priced trade history exists for this item. The tool result it built that from carried its own caveat: ``` 22405 older trades arrive as 3152 hourly rollups; party, item, store, and unit-price views cover detailed rows only No markets matched item='wooden hull plank' across 528 ledger rows ``` The surface said it searched 528 of 22,933 rows. The reply said none exists. Those are different claims and only the first is true. It cost a day of investigation into a query path that was fine. ## The mechanism that generalizes This is arithmetic, not model behaviour, which is why it reaches this repo. A consuming harness bounds tool results at a byte cap. The consumer measured in #449 does it as a **head slice** — keep the front, discard the tail, append a byte-count notice: ```go return result[:cut] + fmt.Sprintf(truncationNotice, cut, len(result)), cut, true ``` The server that surfaced this serializes `warnings` as its **last** JSON key. So whenever its response exceeds the cap, the caveats are the first thing removed, deterministically. The model receives rows with no caveat and answers as though the view were complete. Two properties make this nastier than an ordinary truncation bug: * **The notice is in bytes, not in meaning.** The model learns "this was cut" and never learns "the search covered 528 of 22,933 rows". Only the second changes the answer. * **The cap is per-consumer.** In the observed case one profile capped at 8192 and another at 16384. The same call is honest on one and silently caveat-free on the other, and nothing in either log says so. **Measured** in the sirens-echo consumer against the eco surface. **Inferred** for this server, which is why the first task below is an audit rather than a fix. A server cannot assume its trailing fields survive, and a server that stays under the cap never depends on which consumer read it. ## Why this repo is a live candidate rather than a formality The repo describes a fixed, bounded tool surface, which is the right posture and may mean rule 2 is already satisfied. Rule 1 is the one to check, and there is a specific hazard in the AT Protocol shape: * **The cursor is the coverage field.** A paginated response says "here is a page, and here is where the next one starts". If the cursor serializes after the item array and the response is truncated, the consumer loses the one field that says the view was partial, keeping the items that make it look whole. That is this bug exactly, in the idiom of this protocol. * **Search and follower listings are windows.** An agent concluding "nobody has posted about X" or "they do not follow anyone matching Y" from one page is the wrong-shaped claim above. Whatever states the bound has to survive the cut. ## The four invariants Two are already written up from the other direction in coilyco-gaming/eco-app#266 and #267, worth reading before implementing. 1. **Coverage and caveat metadata serialize first, never last.** Cursors, totals, and any "this is a partial view" field belong ahead of the item array. This is the new finding. 2. **`limit` bounds every unbounded array, not one of them.** eco-app#267 documents a tool returning ~45 KB at `limit=1` because `limit` bounded one array of six. Nested arrays such as embeds, facets, and thread replies are the ones to check here, not just the top-level page. 3. **Truncation always warns, naming shown-of-total.** Silent truncation is prohibited. A count is not a caveat: "3152 of 22933" is actionable, "truncated" is not. 4. **An unreadable dataset reports null, never zero.** A blocked, deactivated, or takendown account must not serialize the same as an account with no posts. ## What to do here * Audit every tool against the four rules, sizing the **largest** response each can emit rather than a typical one. A deep thread with rich embeds is the worst case, not an average post. * Confirm cursor and coverage fields lead the response. In TypeScript, `JSON.stringify` follows property insertion order for string keys, so this is an object-construction change. * Say explicitly whether the fixed surface already satisfies rule 2, so the claim in the README is backed by a check. ## Acceptance * No tool response places cursors, coverage, or caveat fields after item arrays. * No tool can return an unbounded collection, including nested ones. * Truncation emits shown-of-total, and an inaccessible account is distinguishable from an empty one. * Findings recorded even where no change was needed, so the next person does not re-audit. --- Source: coilyco-gaming/sirens-echo#449 and its seven-comment thread, which establishes the head-slice mechanism, the per-profile cap difference, and why a consumer-side check is hard. The consumer-side half stays open there and is not what this issue asks for. Sibling issues filed on the other portfolio MCPs; coilyco-flight-deck/mcp-beaver#68 carries the generator-level version.

coilyco-ops added the

autonomy

headless

label

2026-08-15 15:51:32 +00:00

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-15 15:52:38 +00:00

A tool result that says it saw 2 percent of the data becomes a reply that says none exists #449