Audit tool responses for caveat-last ordering: a truncating consumer destroys trailing coverage fields first, so a bounded result arrives looking complete #42

New issue

Open

opened 2026-08-15 15:50:22 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-15 15:50:22 +00:00

Member

Filed by Darren (director seat) at Kai's direction, 2026-08-15. Cross-cutting finding from coilyco-gaming/sirens-echo#449, filed on every MCP server in the portfolio because the exposure is structural rather than specific to the server that surfaced it. Kai's words: this is an issue across all her MCPs because she was not aware of it being a problem.

The failure, concretely

A community agent told a member:

Currently 0 wooden hull planks are listed for sale on the server. No active store shelves or priced trade history exists for this item.

The tool result it built that from carried its own caveat:

22405 older trades arrive as 3152 hourly rollups; party, item, store,
and unit-price views cover detailed rows only
No markets matched item='wooden hull plank' across 528 ledger rows

The surface said it searched 528 of 22,933 rows. The reply said none exists. Those are different claims and only the first is true. It cost a day of investigation into a query path that was fine.

The mechanism that generalizes

This is arithmetic, not model behaviour, which is why it reaches this repo.

A consuming harness bounds tool results at a byte cap. The consumer measured in #449 does it as a head slice — keep the front, discard the tail, append a byte-count notice:

return result[:cut] + fmt.Sprintf(truncationNotice, cut, len(result)), cut, true

The server that surfaced this serializes warnings as its last JSON key. So whenever its response exceeds the cap, the caveats are the first thing removed, deterministically. The model receives rows with no caveat and answers as though the view were complete.

Two properties make this nastier than an ordinary truncation bug:

The notice is in bytes, not in meaning. The model learns "this was cut" and never learns "the search covered 528 of 22,933 rows". Only the second changes the answer.
The cap is per-consumer. In the observed case one profile capped at 8192 and another at 16384. The same call is honest on one and silently caveat-free on the other, and nothing in either log says so.

Measured in the sirens-echo consumer against the eco surface. Inferred for this server, which is why the first task below is an audit rather than a fix. A server cannot assume its trailing fields survive, and a server that stays under the cap never depends on which consumer read it.

Why this repo is the one where it costs the most

The other servers in this sweep return facts about games, hosts, and feeds. This one returns money, and the failure mode is an agent stating a financial fact with false completeness. Concretely:

A transaction query is a date window over an unbounded ledger. "You have no subscriptions" from a truncated month is the same shape as "0 wooden hull planks are listed", with worse consequences: it is specific, confident, and a person may act on it.
A sum over a truncated array is silently wrong in a way an empty array is not. If a total is computed client-side over rows that were cut, the number is plausible and incorrect, and nothing in the reply looks partial. This is the strongest argument for rule 1 in the whole portfolio: a server-side total that survives truncation is checkable against the rows, and a total that gets cut is not.

I have not read this server's code. That is inference from the domain, and the audit settles it.

The four invariants

Two are already written up from the other direction in coilyco-gaming/eco-app#266 and #267, worth reading before implementing.

Coverage and caveat metadata serialize first, never last. Date range, row count, and any total belong ahead of the transaction array. This is the new finding.
limit bounds every unbounded array, not one of them. eco-app#267 documents a tool returning ~45 KB at limit=1 because limit bounded one array of six.
Truncation always warns, naming shown-of-total. Silent truncation is prohibited. A count is not a caveat: "3152 of 22933" is actionable, "truncated" is not.
An unreadable dataset reports null, never zero. A failed upstream call, an expired token, or an unsynced account must not serialize the same as a zero balance or an empty month. A financial zero that actually means "could not read" is the worst instance of this rule in the portfolio.

What to do here

Audit every tool against the four rules, sizing the largest response each can emit rather than a typical one. A full year of transactions is the worst case, not a week.
Fix ordering first if any coverage field trails bulk payload. In Python this is construction order, so it is cheap.
Check specifically whether any aggregate is computed over an array that can be truncated downstream, and move it ahead of the rows if so.

Acceptance

No tool response places coverage, range, or total fields after bulk arrays.
No tool can return an unbounded collection.
Truncation emits shown-of-total, and an upstream failure is distinguishable from a real zero.
Findings recorded even where no change was needed, so the next person does not re-audit.

Source: coilyco-gaming/sirens-echo#449 and its seven-comment thread, which establishes the head-slice mechanism, the per-profile cap difference, and why a consumer-side check is hard. The consumer-side half stays open there and is not what this issue asks for. Sibling issues filed on the other portfolio MCPs; coilyco-flight-deck/mcp-beaver#68 carries the generator-level version.

**Filed by Darren (director seat) at Kai's direction, 2026-08-15.** Cross-cutting finding from coilyco-gaming/sirens-echo#449, filed on every MCP server in the portfolio because the exposure is structural rather than specific to the server that surfaced it. Kai's words: this is an issue across all her MCPs because she was not aware of it being a problem. ## The failure, concretely A community agent told a member: > Currently 0 wooden hull planks are listed for sale on the server. No active store shelves or priced trade history exists for this item. The tool result it built that from carried its own caveat: ``` 22405 older trades arrive as 3152 hourly rollups; party, item, store, and unit-price views cover detailed rows only No markets matched item='wooden hull plank' across 528 ledger rows ``` The surface said it searched 528 of 22,933 rows. The reply said none exists. Those are different claims and only the first is true. It cost a day of investigation into a query path that was fine. ## The mechanism that generalizes This is arithmetic, not model behaviour, which is why it reaches this repo. A consuming harness bounds tool results at a byte cap. The consumer measured in #449 does it as a **head slice** — keep the front, discard the tail, append a byte-count notice: ```go return result[:cut] + fmt.Sprintf(truncationNotice, cut, len(result)), cut, true ``` The server that surfaced this serializes `warnings` as its **last** JSON key. So whenever its response exceeds the cap, the caveats are the first thing removed, deterministically. The model receives rows with no caveat and answers as though the view were complete. Two properties make this nastier than an ordinary truncation bug: * **The notice is in bytes, not in meaning.** The model learns "this was cut" and never learns "the search covered 528 of 22,933 rows". Only the second changes the answer. * **The cap is per-consumer.** In the observed case one profile capped at 8192 and another at 16384. The same call is honest on one and silently caveat-free on the other, and nothing in either log says so. **Measured** in the sirens-echo consumer against the eco surface. **Inferred** for this server, which is why the first task below is an audit rather than a fix. A server cannot assume its trailing fields survive, and a server that stays under the cap never depends on which consumer read it. ## Why this repo is the one where it costs the most The other servers in this sweep return facts about games, hosts, and feeds. This one returns **money**, and the failure mode is an agent stating a financial fact with false completeness. Concretely: * **A transaction query is a date window over an unbounded ledger.** "You have no subscriptions" from a truncated month is the same shape as "0 wooden hull planks are listed", with worse consequences: it is specific, confident, and a person may act on it. * **A sum over a truncated array is silently wrong** in a way an empty array is not. If a total is computed client-side over rows that were cut, the number is plausible and incorrect, and nothing in the reply looks partial. This is the strongest argument for rule 1 in the whole portfolio: a server-side total that survives truncation is checkable against the rows, and a total that gets cut is not. I have not read this server's code. That is inference from the domain, and the audit settles it. ## The four invariants Two are already written up from the other direction in coilyco-gaming/eco-app#266 and #267, worth reading before implementing. 1. **Coverage and caveat metadata serialize first, never last.** Date range, row count, and any total belong ahead of the transaction array. This is the new finding. 2. **`limit` bounds every unbounded array, not one of them.** eco-app#267 documents a tool returning ~45 KB at `limit=1` because `limit` bounded one array of six. 3. **Truncation always warns, naming shown-of-total.** Silent truncation is prohibited. A count is not a caveat: "3152 of 22933" is actionable, "truncated" is not. 4. **An unreadable dataset reports null, never zero.** A failed upstream call, an expired token, or an unsynced account must not serialize the same as a zero balance or an empty month. A financial zero that actually means "could not read" is the worst instance of this rule in the portfolio. ## What to do here * Audit every tool against the four rules, sizing the **largest** response each can emit rather than a typical one. A full year of transactions is the worst case, not a week. * Fix ordering first if any coverage field trails bulk payload. In Python this is construction order, so it is cheap. * Check specifically whether any aggregate is computed over an array that can be truncated downstream, and move it ahead of the rows if so. ## Acceptance * No tool response places coverage, range, or total fields after bulk arrays. * No tool can return an unbounded collection. * Truncation emits shown-of-total, and an upstream failure is distinguishable from a real zero. * Findings recorded even where no change was needed, so the next person does not re-audit. --- Source: coilyco-gaming/sirens-echo#449 and its seven-comment thread, which establishes the head-slice mechanism, the per-profile cap difference, and why a consumer-side check is hard. The consumer-side half stays open there and is not what this issue asks for. Sibling issues filed on the other portfolio MCPs; coilyco-flight-deck/mcp-beaver#68 carries the generator-level version.

coilyco-ops added the

autonomy

headless

label

2026-08-15 15:51:34 +00:00

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-15 15:52:38 +00:00

A tool result that says it saw 2 percent of the data becomes a reply that says none exists #449