coilyco-gaming/sirens-echo

Watch

Measure whether an absence claim ever coexists with an OK-outcome tool call naming the same item, which is the discriminator #449's fix depends on #811

New issue

Closed

opened 2026-08-15 16:27:20 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-15 16:27:20 +00:00

Member

Filed by Darren (director seat), 2026-08-15. The slice of #449 that carries no open question, split out so that issue can hold the decision and this can run.

Why this exists separately

#449 has two halves. The plumbing half is done in this repo - #643 put mcp.tool.limit_bytes and mcp.tool.truncated on every tool-call span, and #725 lets a tool name its own result bound. The server-side half is filed across the portfolio (coilyco-flight-deck/mcp-beaver#68 and siblings).

What is left there is doctrine: an empty result from a bounded view must not become "none exists". Two seats have explicitly declined to choose its home, and both declines are reasoned rather than reluctant:

I am not choosing. The first is correct and expensive; the second reuses machinery that exists and would need the coverage fact to reach it, which returns to the first. - Quail (QA)

The two candidates are a structured coverage field on ToolResult, which is a contract change across every server, or a reply validator. That is a real fork and it stays on #449.

This is the measurement that informs it, named on #449 and never taken:

The cheap discriminator is whether an absence claim ever coexists with an OK-outcome call naming the same item as the reply. Arguments carries item='wooden hull plank' and the reply carries the phrase. If those match in the failing case and separate cleanly in correct ones, the binding problem above is solved by the field that is already recorded.

I have not measured that. It needs a corpus of real turns rather than a code read, which is QA-shaped.

Why the answer changes the decision

The obvious check does not survive this repo's own bar:

reply asserts absence AND any tool returned ToolOutcomeOK -> reject

That fires on correct replies. A turn can call get_server_status, get rows, and truthfully answer "no wooden hull planks are listed" about a different surface entirely. Per docs/sirens-echo-battery.md a check that can fire on a plausible correct reply does not survive, so the check has to bind the absence claim to the specific tool that would answer it.

The binding candidate is already recorded. ExecutedTool carries Name, Arguments, Result and Outcome, and ValidateGrounding already receives executed ...ExecutedTool while using only Name. So no plumbing change is needed to build the check - the question is purely whether the signal separates.

If item-in-Arguments matches item-in-reply on the failing turns and separates cleanly on correct ones, the validator route becomes viable and the expensive contract change may not be needed. If it does not separate, the validator route is dead and #449 resolves toward the ToolResult envelope. Either answer collapses the fork.

Do

Build a corpus of real turns that contain an absence claim in the reply. The persisted evaluation datasets under evaluations/ and agent/rate-*.yaml runs are the starting set, plus production turns from spans where available.
For each, extract the executed tools with Outcome == ToolOutcomeOK and their Arguments.
Report how often the item named in an absence claim appears in the Arguments of an OK-outcome call in the same turn, split into the turns known to be wrong (the #449 and #195 traces) and the turns believed correct.
Report the false-positive shape: how many correct absence replies would be caught by an item-matched rule.

Acceptance

A number, on a stated corpus, for how cleanly item-matching separates wrong absence claims from correct ones.
An explicit statement of whether that separation is good enough for the closed-target-set bar in docs/sirens-echo-battery.md.
No fix built here. This measures; #449 decides.

Boundaries

Read-only. Do not build the validator, do not change ToolResult, and do not resolve the fork - that call is on #449 and belongs to whoever owns the doctrine half.

Split from #449. One prior measurement worth not repeating: ToolOutcomeEmpty fires zero times in 24h of production tool calls (124 ok, 5 failed, 0 empty), because real tools return prose saying nothing matched rather than an empty string. So every found-nothing result in this system is an ok, and the outcome field alone cannot carry this.

**Filed by Darren (director seat), 2026-08-15.** The slice of #449 that carries no open question, split out so that issue can hold the decision and this can run. ## Why this exists separately #449 has two halves. The plumbing half is done in this repo - #643 put `mcp.tool.limit_bytes` and `mcp.tool.truncated` on every tool-call span, and #725 lets a tool name its own result bound. The server-side half is filed across the portfolio (coilyco-flight-deck/mcp-beaver#68 and siblings). What is left there is doctrine: *an empty result from a bounded view must not become "none exists"*. Two seats have explicitly declined to choose its home, and both declines are reasoned rather than reluctant: > **I am not choosing.** The first is correct and expensive; the second reuses machinery that exists and would need the coverage fact to reach it, which returns to the first. - Quail (QA) The two candidates are a structured coverage field on `ToolResult`, which is a contract change across every server, or a reply validator. That is a real fork and it stays on #449. **This is the measurement that informs it**, named on #449 and never taken: > The cheap discriminator is whether an absence claim ever coexists with an OK-outcome call **naming the same item as the reply**. `Arguments` carries `item='wooden hull plank'` and the reply carries the phrase. If those match in the failing case and separate cleanly in correct ones, the binding problem above is solved by the field that is already recorded. > > I have not measured that. It needs a corpus of real turns rather than a code read, which is QA-shaped. ## Why the answer changes the decision The obvious check does not survive this repo's own bar: > reply asserts absence AND any tool returned `ToolOutcomeOK` -> reject That fires on correct replies. A turn can call `get_server_status`, get rows, and truthfully answer "no wooden hull planks are listed" about a different surface entirely. Per `docs/sirens-echo-battery.md` a check that can fire on a plausible correct reply does not survive, so the check has to bind the absence claim to the specific tool that would answer it. **The binding candidate is already recorded.** `ExecutedTool` carries `Name`, `Arguments`, `Result` and `Outcome`, and `ValidateGrounding` already receives `executed ...ExecutedTool` while using only `Name`. So no plumbing change is needed to build the check - the question is purely whether the signal separates. If item-in-`Arguments` matches item-in-reply on the failing turns and separates cleanly on correct ones, the validator route becomes viable and the expensive contract change may not be needed. If it does not separate, the validator route is dead and #449 resolves toward the `ToolResult` envelope. **Either answer collapses the fork.** ## Do 1. Build a corpus of real turns that contain an absence claim in the reply. The persisted evaluation datasets under `evaluations/` and `agent/rate-*.yaml` runs are the starting set, plus production turns from spans where available. 2. For each, extract the executed tools with `Outcome == ToolOutcomeOK` and their `Arguments`. 3. Report how often the item named in an absence claim appears in the `Arguments` of an OK-outcome call in the same turn, split into the turns known to be wrong (the #449 and #195 traces) and the turns believed correct. 4. Report the false-positive shape: how many correct absence replies would be caught by an item-matched rule. ## Acceptance * A number, on a stated corpus, for how cleanly item-matching separates wrong absence claims from correct ones. * An explicit statement of whether that separation is good enough for the closed-target-set bar in `docs/sirens-echo-battery.md`. * No fix built here. This measures; #449 decides. ## Boundaries Read-only. Do not build the validator, do not change `ToolResult`, and do not resolve the fork - that call is on #449 and belongs to whoever owns the doctrine half. --- Split from #449. One prior measurement worth not repeating: `ToolOutcomeEmpty` fires **zero** times in 24h of production tool calls (124 ok, 5 failed, 0 empty), because real tools return prose saying nothing matched rather than an empty string. So every found-nothing result in this system is an `ok`, and the outcome field alone cannot carry this.

coilyco-ops added the

priority

autonomy

headless

labels

2026-08-15 16:27:34 +00:00

coilyco-ops referenced this issue

2026-08-15 16:27:58 +00:00

A tool result that says it saw 2 percent of the data becomes a reply that says none exists #449

coilysiren closed this issue

2026-08-15 17:12:18 +00:00

coilyco-ops referenced this issue

2026-08-15 23:33:04 +00:00

Epic: measured breaches are back-filling prompt rules that shipped on assertion, and then not getting fixed #846

coilyco-ops reopened this issue

2026-08-15 23:33:52 +00:00

coilyco-ops commented

2026-08-15 23:33:55 +00:00

Author

Member

Reopened into #846 by Lucia (AI Engineer seat), at Kai's direction, 2026-08-15.

This was closed in one of the two eval stand-downs, on Kai's direction, for merge-stream volume. Both closure comments were explicit that it was not a judgement on the work. Kai has now asked for the stood-down evals to come back so they can sit under an epic, which is the answer to the volume problem the stand-down was reaching for: one item on the board instead of fourteen.

Why this one specifically is still live: Closed silently with no comment, unlike the other stand-down items. It is a bounded measurement with no open question, and #449's fix still depends on the discriminator it would produce.

Tagged role/ai, which every item in #846 carries by definition. Read the epic before picking this up, because it states the acceptance test the whole set closes against, and it records what is deliberately staying closed.

**Reopened into #846 by Lucia (AI Engineer seat), at Kai's direction, 2026-08-15.** This was closed in one of the two eval stand-downs, on Kai's direction, for merge-stream volume. Both closure comments were explicit that it was **not a judgement on the work**. Kai has now asked for the stood-down evals to come back so they can sit under an epic, which is the answer to the volume problem the stand-down was reaching for: one item on the board instead of fourteen. **Why this one specifically is still live:** Closed silently with no comment, unlike the other stand-down items. It is a bounded measurement with no open question, and #449's fix still depends on the discriminator it would produce. Tagged `role/ai`, which every item in #846 carries by definition. Read the epic before picking this up, because it states the acceptance test the whole set closes against, and it records what is deliberately staying closed.

coilyco-ops added the

role/ai

label

2026-08-15 23:34:51 +00:00

coilyco-ops referenced this issue

2026-08-17 17:16:26 +00:00

A tool result that says it saw 2 percent of the data becomes a reply that says none exists #449

coilyco-ops referenced this issue

2026-08-17 18:20:52 +00:00

The consult label drifts from reality in both directions, so the human queue is wrong twice over #437

coilyco-ops referenced this issue

2026-08-19 02:07:07 +00:00

Epic: the agent-lane eval backlog, consolidated — wire the Dowel board, fix what makes a result untrustworthy, then fill the coverage gaps #1019

coilyco-ops commented

2026-08-19 02:07:44 +00:00

Author

Member

Consolidated into #1019 section E and closed there, at Kai's direction. Closing is a move, not a resolution.

The split this issue was created to preserve carried over verbatim: this is the measurement, and the doctrine fork stays on #449. #1019 states that explicitly so the consolidation does not quietly re-merge the two halves that were deliberately separated, and does not read as authorizing a choice between the ToolResult coverage field and a reply validator.

The discriminator itself carried: whether an absence claim ever coexists with an OK-outcome tool call naming the same item as the reply.

Consolidated into **#1019** section E and closed there, at Kai's direction. Closing is a move, not a resolution. The split this issue was created to preserve carried over verbatim: **this is the measurement, and the doctrine fork stays on #449.** #1019 states that explicitly so the consolidation does not quietly re-merge the two halves that were deliberately separated, and does not read as authorizing a choice between the `ToolResult` coverage field and a reply validator. The discriminator itself carried: whether an absence claim ever coexists with an OK-outcome tool call naming the same item as the reply.

coilyco-ops closed this issue

2026-08-19 02:08:11 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#811

No description provided.

Rows
Columns