coilyco-gaming/sirens-echo

Watch

A reply check refuses the whole message, so one bad token discards every correct sentence beside it #796

New issue

Closed

opened 2026-08-15 15:22:49 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 15:22:49 +00:00

Member

Filed by Olaf (ops, claude seat), at Kai's request, split out of #794 because it survives that fix.

#794 is "the grounding check is wrong about this channel." This is "even when a check is right, refusing the entire reply is the wrong unit."

The observed cost

Trace 7ea1e319b92b0357d3e2ac71b802a66a, Sirens Deep, 2026-08-15. Prompt was "query each of your MCPs, 1 simple query each, summarize the output". Deep spent 51.5 seconds on it: 10 servers, 13 tool calls, 4 model calls. The reply was twelve blocks, one per server plus a closing line.

One block contained #general. All twelve were discarded.

The eleven survivors were correct and included a live outage the agent had correctly diagnosed:

playwright - down, same notifications/initialized: Bad Request on both attempts, so no page loaded.

That is now coilyco-flight-deck/mcp-beaver#67, an all-three-pods outage. It was found in the trace hours later rather than read in the channel, because a validation rule about a different paragraph threw the message away.

The mechanism

runReplyChecks in turnstages.go returns on the first failure:

for _, check := range checks {
    if err := check.run(); err != nil {
        return reply, check.name, err
    }
}

and the caller fails the turn. There is no partial outcome anywhere in the path: a reply is either delivered whole or replaced by > reply blocked by response check, rephrase.

What makes this worth fixing rather than accepting

The codebase already has the sentence model. decision.go:95 and decision.go:204 both split on sentenceBreak and evaluate per sentence, with a comment explaining exactly why polarity belongs to the clause that carries it. Two of the grounding rules reason at sentence granularity today and then report a single boolean.

The location is already known and thrown away. ValidateGrounding produces model invented channel #general - it has the offending token in hand. That string does not reach a span or a log either, which is #795.

So the information needed to refuse less already exists at the point of refusal. What is missing is a return type that can carry it.

What I am explicitly not deciding

This is a safety surface and a narrower refusal can be the wrong answer. Three real objections, none of which I think settles it:

Redaction can leak more than refusal. A reply with one sentence removed still tells the member what the other eleven said, and if the removed sentence was the fabrication, the surrounding text may still depend on it. A summary whose third bullet was invented is not eleven-twelfths correct.
Sentence boundaries are not claim boundaries. [.!?]+ splits prose, not reasoning. A claim can span two sentences, and refusing one half could leave a more misleading message than refusing both.
A visibly redacted reply is a new surface. "One line was removed by a policy check" invites the member to ask what it said, and that is a conversation the harness has no answer for.

Options worth weighing, roughly in increasing ambition:

Keep whole-message refusal, but tell the model what failed so the repair loop can retry with the offending clause fixed rather than rephrasing blind. Cheapest, changes no member-visible behaviour, and probably recovers most of the value here.
Refuse the clause and deliver the remainder, marked. Highest value, all three objections above apply.
Refuse whole, but keep the reply somewhere an operator can read, so a discarded outage report is not lost even if the member never sees it.

My weak preference is the first, because it is the only one that does not change what a member can see, and because a 51-second turn being retried is much cheaper than a 51-second turn being deleted.

Acceptance

Whatever lands, the failure mode this issue names should be gone: a correct sentence should not be destroyed by an unrelated sentence's failure, whether that is achieved by repair, by partial delivery, or by preservation.

**Filed by Olaf (ops, claude seat)**, at Kai's request, split out of #794 because it survives that fix. #794 is "the grounding check is wrong about this channel." This is "even when a check is **right**, refusing the entire reply is the wrong unit." ## The observed cost Trace `7ea1e319b92b0357d3e2ac71b802a66a`, Sirens Deep, 2026-08-15. Prompt was "query each of your MCPs, 1 simple query each, summarize the output". Deep spent 51.5 seconds on it: 10 servers, 13 tool calls, 4 model calls. The reply was twelve blocks, one per server plus a closing line. **One block contained `#general`. All twelve were discarded.** The eleven survivors were correct and included a live outage the agent had correctly diagnosed: > **playwright** - down, same `notifications/initialized: Bad Request` on both attempts, so no page loaded. That is now `coilyco-flight-deck/mcp-beaver#67`, an all-three-pods outage. It was found in the trace hours later rather than read in the channel, because a validation rule about a *different paragraph* threw the message away. ## The mechanism `runReplyChecks` in `turnstages.go` returns on the first failure: ```go for _, check := range checks { if err := check.run(); err != nil { return reply, check.name, err } } ``` and the caller fails the turn. There is no partial outcome anywhere in the path: a reply is either delivered whole or replaced by `> reply blocked by response check, rephrase`. ## What makes this worth fixing rather than accepting **The codebase already has the sentence model.** `decision.go:95` and `decision.go:204` both split on `sentenceBreak` and evaluate per sentence, with a comment explaining exactly why polarity belongs to the clause that carries it. Two of the grounding rules reason at sentence granularity today and then report a single boolean. **The location is already known and thrown away.** `ValidateGrounding` produces `model invented channel #general` - it has the offending token in hand. That string does not reach a span or a log either, which is #795. So the information needed to refuse less already exists at the point of refusal. What is missing is a return type that can carry it. ## What I am explicitly not deciding This is a safety surface and a narrower refusal can be the wrong answer. Three real objections, none of which I think settles it: 1. **Redaction can leak more than refusal.** A reply with one sentence removed still tells the member what the other eleven said, and if the removed sentence was the fabrication, the surrounding text may still depend on it. A summary whose third bullet was invented is not eleven-twelfths correct. 2. **Sentence boundaries are not claim boundaries.** `[.!?]+` splits prose, not reasoning. A claim can span two sentences, and refusing one half could leave a more misleading message than refusing both. 3. **A visibly redacted reply is a new surface.** "One line was removed by a policy check" invites the member to ask what it said, and that is a conversation the harness has no answer for. Options worth weighing, roughly in increasing ambition: - **Keep whole-message refusal, but tell the model what failed** so the repair loop can retry with the offending clause fixed rather than rephrasing blind. Cheapest, changes no member-visible behaviour, and probably recovers most of the value here. - **Refuse the clause and deliver the remainder**, marked. Highest value, all three objections above apply. - **Refuse whole, but keep the reply** somewhere an operator can read, so a discarded outage report is not lost even if the member never sees it. My weak preference is the first, because it is the only one that does not change what a member can see, and because a 51-second turn being retried is much cheaper than a 51-second turn being deleted. ## Acceptance Whatever lands, the failure mode this issue names should be gone: **a correct sentence should not be destroyed by an unrelated sentence's failure**, whether that is achieved by repair, by partial delivery, or by preservation.

coilyco-ops referenced this issue from a commit

2026-08-15 15:49:45 +00:00

fix(reply): repair a refused reply instead of discarding every correct line

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-15 15:50:10 +00:00

Repair a refused reply, then redact the block, before discarding the message #804

coilyco-ops commented

2026-08-15 15:50:31 +00:00

Author

Member

Taken in #804, as option one. Reading the path first changed what option one is.

The repair loop already exists, and already names the refusing check to the model (responseRepairPrompt, #549). It covers parse and response_style, because those two are duplicated inside Complete. The other five checks run in runReplyChecks after Complete returns and get no repair at all.

So the asymmetry is not that the mechanism is missing. It is that a grounding refusal ends a 51-second turn outright while a style refusal gets a retry, for no reason beyond which side of a function boundary the check sits on. ProxyClient.ValidateReply takes the harness check set, the runtime wires it to runReplyChecks, and all ten rules now reach the loop.

The verdict did not move. runReplyChecks after Complete is still the only thing that refuses, and the harness checks drop out of the loop once the repair budget is spent rather than failing there. Failing inside Complete would return ErrResponseRepairExhausted and report stage=model, which docs/sirens-echo-turn-stages.md already calls out as true about the code and false about the world. A model that will not fix the clause therefore produces the outcome it produced before this change.

On the three objections: all of them were against partial delivery, and none of them applies, because this does not deliver partially. A member still sees a whole reply or none of it.

Preservation is not implemented either, and there is a reason worth recording beyond scope. A reply refused by identifier_disclosure carries the value that check exists to keep out of logs, so storing refused replies needs its own design rather than a log line. If you want that, it is worth its own issue.

One thing the repair pass deliberately cannot do: tools are already withdrawn on repair attempts, so the model rewrites or keeps the refusal. A check refusing an unsupported claim must not become a prompt to go find support for it.

Verified against the trace's shape, and the three behavioural tests were confirmed to fail on main first.

Taken in #804, as option one. Reading the path first changed what option one is. **The repair loop already exists, and already names the refusing check to the model** (`responseRepairPrompt`, #549). It covers `parse` and `response_style`, because those two are duplicated inside `Complete`. The other five checks run in `runReplyChecks` after `Complete` returns and get no repair at all. So the asymmetry is not that the mechanism is missing. It is that a grounding refusal ends a 51-second turn outright while a style refusal gets a retry, for no reason beyond which side of a function boundary the check sits on. `ProxyClient.ValidateReply` takes the harness check set, the runtime wires it to `runReplyChecks`, and all ten rules now reach the loop. **The verdict did not move.** `runReplyChecks` after `Complete` is still the only thing that refuses, and the harness checks drop out of the loop once the repair budget is spent rather than failing there. Failing inside `Complete` would return `ErrResponseRepairExhausted` and report `stage=model`, which `docs/sirens-echo-turn-stages.md` already calls out as true about the code and false about the world. A model that will not fix the clause therefore produces the outcome it produced before this change. On the three objections: all of them were against partial delivery, and none of them applies, because this does not deliver partially. A member still sees a whole reply or none of it. Preservation is not implemented either, and there is a reason worth recording beyond scope. A reply refused by `identifier_disclosure` carries the value that check exists to keep out of logs, so storing refused replies needs its own design rather than a log line. If you want that, it is worth its own issue. One thing the repair pass deliberately cannot do: tools are already withdrawn on repair attempts, so the model rewrites or keeps the refusal. A check refusing an unsupported claim must not become a prompt to go find support for it. Verified against the trace's shape, and the three behavioural tests were confirmed to fail on `main` first.

coilyco-ops added the

priority

autonomy

async-consult

labels

2026-08-15 16:05:02 +00:00