coilyco-gaming/sirens-echo

Watch

Repair a refused reply, then redact the block, before discarding the message #804

Merged

coilysiren merged 2 commits from aos/claude/yt58-partial-refusal into main

2026-08-15 16:19:44 +00:00

coilyco-ops commented

2026-08-15 15:50:10 +00:00

Member

Closes #796. Two rungs, in the order they run.

Rung one: repair (the issue's option 1)

The issue preferred "tell the model what failed so the repair loop can retry". Reading the path first: that loop already exists, and already names the refusing check to the model (responseRepairPrompt, #549). It covered two checks, parse and response_style, because those two are duplicated inside Complete.

The other five run in runReplyChecks after Complete returns, so they had no repair. A grounding refusal ended a 51-second turn outright while a style refusal got a retry, for no reason beyond which side of a function boundary the check sat on. ProxyClient.ValidateReply takes the harness check set and the runtime wires it to runReplyChecks, so all ten rules reach the loop.

Repair is advisory. runReplyChecks after Complete is still the only thing that refuses, and the harness checks drop out of the loop once the budget is spent rather than failing there, which would report stage=model for a harness refusal.

Rung two: redaction (the issue's option 2)

Repair covers the model that will fix the block. Redaction covers the model that will not: the block goes, the rest is delivered, and a harness notice stands where it was.

- forgejo - answered, 116 open issues on sirens-echo.
- steam - answered, 412 owned titles.
- playwright - down, same notifications/initialized: Bad Request on both attempts.
> `content removed by a response check`

The unit is a block, not a sentence. A block is a paragraph or one list item. The observed reply was twelve bullets with no blank line in it, so a paragraph-only split would have found one block and saved nothing. The issue's objection 2 was right about sentences: [.!?]+ splits prose, not reasoning, and separates a claim from the qualifier bounding it. A block holds both.

Only rules a block can carry alone, by allowlist. The four grounding rules, tool-call markup, self-attributed claim. response_style and identity_claim are properties of the whole voice. parse means there was no reply to cut. identifier_disclosure looks redactable and is not: it collapses digits and spelled numbers across the whole reply, so a guarded value can span two blocks and neither fails alone. That one would have been a real hole.

Two passes, and the second is the authority. The first marks blocks failing the same rule that refused the message. The second runs every check again over the surviving prose, so what is delivered passed the full set on its own rather than merely lacking the removed block. That is what makes an imperfect first pass safe: it picks candidates and licenses nothing. A remainder that fails is refused whole, exactly as before.

The mark cannot be forged. It renders in the existing harness notice shape, whose alphabet (noticeAllowed) strips anything a model could use to imitate it. This is the first notice that stands inside a reply rather than replacing one.

On the issue's three objections

Redaction can leak more than refusal / the remainder may depend on the removed claim. Half answered. Re-validation checks rules, not reasoning, so a summary whose third bullet was invented is still not eleven-twelfths correct. The mark is the mitigation: the member is told the answer is incomplete rather than left to assume it is whole.
Sentence boundaries are not claim boundaries. Answered by not using sentences.
A visibly redacted reply is a new surface. Stands. "What was removed" has no reply here, and it was judged the smaller cost against discarding a correct outage report.

Option 3, preserving a refused reply for an operator, is still not implemented: a reply refused by identifier_disclosure carries the value that check exists to keep out of logs. Worth its own issue if wanted.

Verified

internal/community/replyrepair_test.go (4) and redaction_test.go (10), driving the observed trace's shape:

Repair: the eleven correct blocks survive; the repair prompt names the offending token; an unrepaired reply is handed back and refused under the same rule; a nil hook makes one model call and alters nothing.
Redaction: the block goes and the outage report survives; the mark is in the harness shape and stands where the block stood; a remainder that still fails any check is refused whole; two blocks on one rule both go; the cap and the whole-reply rules refuse whole; a one-block reply refuses whole; splitting round-trips losslessly across bulleted, prose, numbered, mixed and single-block replies.
End to end through runTurn: the member receives the redacted reply with the mark, and a clean turn records response.redacted.blocks: 0.

The repair tests were confirmed to fail against main first. ward exec gate green.

Notes for review

Telemetry semantics changed slightly. response.check on a delivered turn can now name a rule, meaning "this reply lost a block to it", where before a named check always meant a discarded turn. response.redacted.blocks disambiguates and is present on every turn, zero included.

Truncation can reach the mark. It is part of the answer rather than a service suffix, so a redaction in the last block can lose its mark when the reply is also being cut for length. The removal is never at risk, only the disclosure, and only on a reply already visibly truncated. Restructuring the assembly budget to protect it seemed worse than the edge.

docs/sirens-echo-notices.md is at its 80-line cap, so the "a notice can now stand inside a reply" case is documented in docs/sirens-echo-reply-redaction.md and cross-linked rather than added there. If it belongs in the notices doc, that doc needs a split first.

Not decided here: whether maxResponseRepairs should rise from 1, and whether maxRedactedBlocks (2) is the right bound. Both are tuning, and docs/sirens-echo-tuning-overrides.md treats those as deliberate.

Closes #796. Two rungs, in the order they run. ## Rung one: repair (the issue's option 1) The issue preferred "tell the model what failed so the repair loop can retry". Reading the path first: **that loop already exists, and already names the refusing check to the model** (`responseRepairPrompt`, #549). It covered two checks, `parse` and `response_style`, because those two are duplicated inside `Complete`. The other five run in `runReplyChecks` *after* `Complete` returns, so they had no repair. A grounding refusal ended a 51-second turn outright while a style refusal got a retry, for no reason beyond which side of a function boundary the check sat on. `ProxyClient.ValidateReply` takes the harness check set and the runtime wires it to `runReplyChecks`, so all ten rules reach the loop. Repair is advisory. `runReplyChecks` after `Complete` is still the only thing that refuses, and the harness checks drop out of the loop once the budget is spent rather than failing there, which would report `stage=model` for a harness refusal. ## Rung two: redaction (the issue's option 2) Repair covers the model that will fix the block. Redaction covers the model that will not: the block goes, the rest is delivered, and a harness notice stands where it was. ``` - forgejo - answered, 116 open issues on sirens-echo. - steam - answered, 412 owned titles. - playwright - down, same notifications/initialized: Bad Request on both attempts. > `content removed by a response check` ``` **The unit is a block, not a sentence.** A block is a paragraph or one list item. The observed reply was twelve bullets with no blank line in it, so a paragraph-only split would have found one block and saved nothing. The issue's objection 2 was right about sentences: `[.!?]+` splits prose, not reasoning, and separates a claim from the qualifier bounding it. A block holds both. **Only rules a block can carry alone, by allowlist.** The four grounding rules, tool-call markup, self-attributed claim. `response_style` and `identity_claim` are properties of the whole voice. `parse` means there was no reply to cut. `identifier_disclosure` looks redactable and is not: it collapses digits and spelled numbers across the whole reply, so a guarded value can span two blocks and neither fails alone. That one would have been a real hole. **Two passes, and the second is the authority.** The first marks blocks failing the same rule that refused the message. The second runs every check again over the surviving prose, so what is delivered passed the full set on its own rather than merely lacking the removed block. That is what makes an imperfect first pass safe: it picks candidates and licenses nothing. A remainder that fails is refused whole, exactly as before. **The mark cannot be forged.** It renders in the existing harness notice shape, whose alphabet (`noticeAllowed`) strips anything a model could use to imitate it. This is the first notice that stands inside a reply rather than replacing one. ## On the issue's three objections 1. **Redaction can leak more than refusal / the remainder may depend on the removed claim.** Half answered. Re-validation checks rules, not reasoning, so a summary whose third bullet was invented is still not eleven-twelfths correct. The mark is the mitigation: the member is told the answer is incomplete rather than left to assume it is whole. 2. **Sentence boundaries are not claim boundaries.** Answered by not using sentences. 3. **A visibly redacted reply is a new surface.** Stands. "What was removed" has no reply here, and it was judged the smaller cost against discarding a correct outage report. Option 3, preserving a refused reply for an operator, is still not implemented: a reply refused by `identifier_disclosure` carries the value that check exists to keep out of logs. Worth its own issue if wanted. ## Verified `internal/community/replyrepair_test.go` (4) and `redaction_test.go` (10), driving the observed trace's shape: - Repair: the eleven correct blocks survive; the repair prompt names the offending token; an unrepaired reply is handed back and refused under the same rule; a nil hook makes one model call and alters nothing. - Redaction: the block goes and the outage report survives; the mark is in the harness shape and stands where the block stood; a remainder that still fails any check is refused whole; two blocks on one rule both go; the cap and the whole-reply rules refuse whole; a one-block reply refuses whole; splitting round-trips losslessly across bulleted, prose, numbered, mixed and single-block replies. - End to end through `runTurn`: the member receives the redacted reply with the mark, and a clean turn records `response.redacted.blocks: 0`. The repair tests were confirmed to fail against `main` first. `ward exec gate` green. ## Notes for review **Telemetry semantics changed slightly.** `response.check` on a *delivered* turn can now name a rule, meaning "this reply lost a block to it", where before a named check always meant a discarded turn. `response.redacted.blocks` disambiguates and is present on every turn, zero included. **Truncation can reach the mark.** It is part of the answer rather than a service suffix, so a redaction in the last block can lose its mark when the reply is also being cut for length. The removal is never at risk, only the disclosure, and only on a reply already visibly truncated. Restructuring the assembly budget to protect it seemed worse than the edge. **`docs/sirens-echo-notices.md` is at its 80-line cap**, so the "a notice can now stand inside a reply" case is documented in `docs/sirens-echo-reply-redaction.md` and cross-linked rather than added there. If it belongs in the notices doc, that doc needs a split first. **Not decided here:** whether `maxResponseRepairs` should rise from 1, and whether `maxRedactedBlocks` (2) is the right bound. Both are tuning, and `docs/sirens-echo-tuning-overrides.md` treats those as deliberate.

coilyco-ops added 1 commit

2026-08-15 15:50:10 +00:00

fix(reply): repair a refused reply instead of discarding every correct line

ci / image-build (pull_request) Successful in 24s

Details

ci / test (pull_request) Successful in 42s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

d75c4b1eb7

A twelve-block reply lost all twelve because one block named a channel
nobody has. The eleven survivors included an outage the agent had
correctly diagnosed, which was then found in a trace hours later rather
than read in the channel. Trace 7ea1e319b92b0357d3e2ac71b802a66a.

The completion layer already had a repair loop that names the refusing
check to the model, but it covered only `parse` and `response_style`.
The other five checks ran after Complete returned and had no repair at
all, so a grounding refusal ended a 51-second turn outright. The
machinery was there and those checks were not wired into it.

ProxyClient.ValidateReply takes the harness check set and the runtime
wires it to runReplyChecks, so all ten rules reach the loop.

Repair stays advisory. runReplyChecks after Complete is still the only
thing that refuses, and the harness checks drop out of the loop once the
repair budget is spent rather than failing there, which would report
stage=model for a harness refusal. A model that will not fix the clause
produces the outcome it produced before this change.

The evaluation client keeps a nil hook, so the rate packs go on
measuring raw model behavior rather than the harness repairing it.

Closes #796

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-15 15:50:10 +00:00

coilyco-ops referenced this pull request

2026-08-15 15:50:31 +00:00

A reply check refuses the whole message, so one bad token discards every correct sentence beside it #796

coilyco-ops added 1 commit

2026-08-15 16:09:41 +00:00

feat(reply): remove the block a check refused, keep the message

ci / image-build (pull_request) Successful in 25s

Details

ci / test (pull_request) Successful in 42s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

4eb45da49c

Repair covers the case where the model will fix the block. This covers
the case where it will not: the block goes, the rest is delivered, and a
harness notice stands where it was.

A block is a paragraph or one list item. The observed reply was twelve
bullets with no blank line in it, so a paragraph-only split would have
found one block and saved nothing. A sentence was the obvious unit and
splits a claim from the qualifier that bounds it.

Only rules a block can carry alone are redactable, by allowlist: the
four grounding rules, tool-call markup, self-attributed claim. Response
style and identity claim are properties of the whole voice. Identifier
disclosure looks redactable and is not, because it collapses digits and
spelled numbers across the whole reply, so a guarded value can span two
blocks and neither fails alone.

Two passes decide, and the second is the authority. The first marks
blocks failing the same rule that refused the message. The second runs
every check again over the surviving prose, so what is delivered passed
the full set on its own rather than merely lacking the removed block.
That is what makes an imperfect first pass safe. A remainder that fails
is refused whole, exactly as before.

The mark renders in the harness notice shape, whose alphabet model prose
cannot forge. It is the first notice that stands inside a reply rather
than replacing one.

response.check names the rule a delivered reply lost a block to, so a
redaction still appears in the refusal rates, and
response.redacted.blocks carries the count on every turn, zero included.

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops changed title from ~~Repair a refused reply instead of discarding every correct line~~ to Repair a refused reply, then redact the block, before discarding the message

2026-08-15 16:10:25 +00:00