Audit tool responses for caveat-last ordering: a truncating consumer destroys trailing coverage fields first, so a bounded result arrives looking complete #22

New issue

Open

opened 2026-08-15 15:48:55 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-15 15:48:55 +00:00

Member

Filed by Darren (director seat) at Kai's direction, 2026-08-15. Cross-cutting finding from coilyco-gaming/sirens-echo#449, filed on every MCP server in the portfolio because the exposure is structural rather than specific to the server that surfaced it. Kai's words: this is an issue across all her MCPs because she was not aware of it being a problem.

The failure, concretely

A community agent told a member:

Currently 0 wooden hull planks are listed for sale on the server. No active store shelves or priced trade history exists for this item.

The tool result it built that from carried its own caveat:

22405 older trades arrive as 3152 hourly rollups; party, item, store,
and unit-price views cover detailed rows only
No markets matched item='wooden hull plank' across 528 ledger rows

The surface said it searched 528 of 22,933 rows. The reply said none exists. Those are different claims and only the first is true. It cost a day of investigation into a query path that was fine.

The mechanism that generalizes

This is arithmetic, not model behaviour, which is why it reaches this repo.

A consuming harness bounds tool results at a byte cap. The consumer measured in #449 does it as a head slice — keep the front, discard the tail, append a byte-count notice:

return result[:cut] + fmt.Sprintf(truncationNotice, cut, len(result)), cut, true

The server that surfaced this serializes warnings as its last JSON key. So whenever its response exceeds the cap, the caveats are the first thing removed, deterministically. The model receives rows with no caveat and answers as though the view were complete.

Two properties make this nastier than an ordinary truncation bug:

The notice is in bytes, not in meaning. The model learns "this was cut" and never learns "the search covered 528 of 22,933 rows". Only the second changes the answer.
The cap is per-consumer. In the observed case one profile capped at 8192 and another at 16384. The same call is honest on one and silently caveat-free on the other, and nothing in either log says so.

Measured in the sirens-echo consumer against the eco surface. Inferred for this server, which is why the first task below is an audit rather than a fix: nobody has checked whether node-stats responses carry trailing coverage fields or overrun a consumer cap. A server cannot assume its trailing fields survive, and a server that stays under the cap never depends on which consumer read it.

Why this repo is a live candidate rather than a formality

This is a diagnostics server, and diagnostics responses are exactly the shape that overruns a byte cap: process tables, pod lists, volume and filesystem inventories, log-usage breakdowns. Several of its tools return collections whose size is a property of the host rather than of the request, which is the definition of an unbounded array. A truncated get_top_processes or get_k3s_pods that loses its "showing N of M" tail reads as a complete inventory of the host, and an agent acting on "there is no such process" is the same wrong-shaped claim as the one above.

The repo describes itself as offering bounded host introspection, so this may already be handled. The audit establishes which.

The four invariants

Two are already written up from the other direction in coilyco-gaming/eco-app#266 and #267, worth reading before implementing.

Coverage and caveat metadata serialize first, never last. A field that explains the bound of a result must not sit where a head slice removes it. This is the new finding.
limit bounds every unbounded array, not one of them. eco-app#267 documents a tool returning ~45 KB at limit=1 because limit bounded one array of six.
Truncation always warns, naming shown-of-total. Silent truncation is prohibited. A count is not a caveat: "3152 of 22933" is actionable, "truncated" is not.
An unreadable dataset reports null, never zero. A permission-denied read, an unmounted path, and a genuinely idle host must not serialize alike. This one has teeth here: an unreadable /proc entry reported as 0 is a false all-clear.

What to do here

Audit every tool against the four rules, sizing the largest response each can emit rather than a typical one.
Fix ordering first if any coverage field trails bulk payload. In Python this is construction order, so it is cheap.
Report which tools can exceed a plausible consumer cap, since that set is what rule 1 protects.

Acceptance

No tool response places coverage or caveat fields after bulk arrays.
No tool can return an unbounded collection.
Truncation emits shown-of-total, and an unreadable source is distinguishable from a measured zero.
Findings recorded even where no change was needed, so the next person does not re-audit.

Source: coilyco-gaming/sirens-echo#449 and its seven-comment thread, which establishes the head-slice mechanism, the per-profile cap difference, and why a consumer-side check is hard. The consumer-side half stays open there and is not what this issue asks for. Sibling issues filed on the other portfolio MCPs; coilyco-flight-deck/mcp-beaver#68 carries the generator-level version.

**Filed by Darren (director seat) at Kai's direction, 2026-08-15.** Cross-cutting finding from coilyco-gaming/sirens-echo#449, filed on every MCP server in the portfolio because the exposure is structural rather than specific to the server that surfaced it. Kai's words: this is an issue across all her MCPs because she was not aware of it being a problem. ## The failure, concretely A community agent told a member: > Currently 0 wooden hull planks are listed for sale on the server. No active store shelves or priced trade history exists for this item. The tool result it built that from carried its own caveat: ``` 22405 older trades arrive as 3152 hourly rollups; party, item, store, and unit-price views cover detailed rows only No markets matched item='wooden hull plank' across 528 ledger rows ``` The surface said it searched 528 of 22,933 rows. The reply said none exists. Those are different claims and only the first is true. It cost a day of investigation into a query path that was fine. ## The mechanism that generalizes This is arithmetic, not model behaviour, which is why it reaches this repo. A consuming harness bounds tool results at a byte cap. The consumer measured in #449 does it as a **head slice** — keep the front, discard the tail, append a byte-count notice: ```go return result[:cut] + fmt.Sprintf(truncationNotice, cut, len(result)), cut, true ``` The server that surfaced this serializes `warnings` as its **last** JSON key. So whenever its response exceeds the cap, the caveats are the first thing removed, deterministically. The model receives rows with no caveat and answers as though the view were complete. Two properties make this nastier than an ordinary truncation bug: * **The notice is in bytes, not in meaning.** The model learns "this was cut" and never learns "the search covered 528 of 22,933 rows". Only the second changes the answer. * **The cap is per-consumer.** In the observed case one profile capped at 8192 and another at 16384. The same call is honest on one and silently caveat-free on the other, and nothing in either log says so. **Measured** in the sirens-echo consumer against the eco surface. **Inferred** for this server, which is why the first task below is an audit rather than a fix: nobody has checked whether node-stats responses carry trailing coverage fields or overrun a consumer cap. A server cannot assume its trailing fields survive, and a server that stays under the cap never depends on which consumer read it. ## Why this repo is a live candidate rather than a formality This is a diagnostics server, and diagnostics responses are exactly the shape that overruns a byte cap: process tables, pod lists, volume and filesystem inventories, log-usage breakdowns. Several of its tools return collections whose size is a property of the host rather than of the request, which is the definition of an unbounded array. A truncated `get_top_processes` or `get_k3s_pods` that loses its "showing N of M" tail reads as a complete inventory of the host, and an agent acting on "there is no such process" is the same wrong-shaped claim as the one above. The repo describes itself as offering bounded host introspection, so this may already be handled. The audit establishes which. ## The four invariants Two are already written up from the other direction in coilyco-gaming/eco-app#266 and #267, worth reading before implementing. 1. **Coverage and caveat metadata serialize first, never last.** A field that explains the bound of a result must not sit where a head slice removes it. This is the new finding. 2. **`limit` bounds every unbounded array, not one of them.** eco-app#267 documents a tool returning ~45 KB at `limit=1` because `limit` bounded one array of six. 3. **Truncation always warns, naming shown-of-total.** Silent truncation is prohibited. A count is not a caveat: "3152 of 22933" is actionable, "truncated" is not. 4. **An unreadable dataset reports null, never zero.** A permission-denied read, an unmounted path, and a genuinely idle host must not serialize alike. This one has teeth here: an unreadable `/proc` entry reported as `0` is a false all-clear. ## What to do here * Audit every tool against the four rules, sizing the **largest** response each can emit rather than a typical one. * Fix ordering first if any coverage field trails bulk payload. In Python this is construction order, so it is cheap. * Report which tools can exceed a plausible consumer cap, since that set is what rule 1 protects. ## Acceptance * No tool response places coverage or caveat fields after bulk arrays. * No tool can return an unbounded collection. * Truncation emits shown-of-total, and an unreadable source is distinguishable from a measured zero. * Findings recorded even where no change was needed, so the next person does not re-audit. --- Source: coilyco-gaming/sirens-echo#449 and its seven-comment thread, which establishes the head-slice mechanism, the per-profile cap difference, and why a consumer-side check is hard. The consumer-side half stays open there and is not what this issue asks for. Sibling issues filed on the other portfolio MCPs; coilyco-flight-deck/mcp-beaver#68 carries the generator-level version.

coilyco-ops added the

autonomy

headless

label

2026-08-15 15:51:26 +00:00

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-15 15:52:38 +00:00

A tool result that says it saw 2 percent of the data becomes a reply that says none exists #449