docs(battery): name the two deployed checks the evaluation does not run #731

Merged

coilyco-ops merged 1 commit from docs/the-battery-assumes-two-checks-it-does-not-run into main

2026-08-13 21:07:14 +00:00

coilyco-ops commented

2026-08-13 21:06:07 +00:00

Member

docs/sirens-echo-battery.md opens by telling a reader which deployed validators the evaluation already covers:

The deployed structural validators run on every case as well, so ParseReply, ValidateGrounding, and ValidateResponseStyle are assumed here. These checks are the rest.

Two deployed validators are not assumed, and the sentence reads as if the deployed set is accounted for.

Measured from source

Unconditional on every evaluation case, evaluation.go:288-303:

ParseReply  ValidateGrounding  ValidateSelfAttributedClaim
ValidateIdentityClaim  ValidateResponseStyle

That is five, not three. The doc undersells two of its own.

Conditional, and this is the part that matters:

if evaluationCase.ForbidPrincipalEcho { record(checkUserIDEcho(reply, principal)) }
if evaluationCase.ForbidToolCallMarkup { ... }

Both opt in per case. In the deployed path both are unconditional:

ValidateNoToolCallMarkup is check two of seven in runReplyChecks, on every reply.
IdentifierGuard.Validate is check five, on every reply, matching the principal ID literally, as digits, spelled out, and reversed. The evaluation never builds the guard at all — its nearest equivalent is checkUserIDEcho, which reads less and only when asked.

Why it is worth correcting rather than shrugging at

Two open issues keep rediscovering this from different ends. Issue 301 measured forbid_tool_call_markup set on 24 rate cases and zero gating cases. Issue 310 recorded a principal-ID disclosure in a rate run that the deployed path would have refused.

A reader who takes the current sentence at face value concludes a green battery covers the deployed reply path. It covers five sevenths of it.

Acceptance

The doc names all five unconditional validators.
It names the two that are conditional and what conditions them.
The file stays inside the 80-line and 4000-character caps.

Next owner

Engineer. Closed by the pull request that lands the change.

ward exec gate PASS on all six steps.

closes #730

`docs/sirens-echo-battery.md` opens by telling a reader which deployed validators the evaluation already covers: > The deployed structural validators run on every case as well, so `ParseReply`, `ValidateGrounding`, and `ValidateResponseStyle` are assumed here. These checks are the rest. **Two deployed validators are not assumed, and the sentence reads as if the deployed set is accounted for.** ## Measured from source Unconditional on every evaluation case, `evaluation.go:288-303`: ``` ParseReply ValidateGrounding ValidateSelfAttributedClaim ValidateIdentityClaim ValidateResponseStyle ``` That is **five**, not three. The doc undersells two of its own. Conditional, and this is the part that matters: ```go if evaluationCase.ForbidPrincipalEcho { record(checkUserIDEcho(reply, principal)) } if evaluationCase.ForbidToolCallMarkup { ... } ``` Both opt in per case. In the deployed path both are unconditional: - `ValidateNoToolCallMarkup` is check two of seven in `runReplyChecks`, on every reply. - `IdentifierGuard.Validate` is check five, on every reply, matching the principal ID literally, as digits, spelled out, and reversed. **The evaluation never builds the guard at all** — its nearest equivalent is `checkUserIDEcho`, which reads less and only when asked. ## Why it is worth correcting rather than shrugging at Two open issues keep rediscovering this from different ends. [Issue 301](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/301) measured `forbid_tool_call_markup` set on 24 rate cases and **zero** gating cases. [Issue 310](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/310) recorded a principal-ID disclosure in a rate run that the deployed path would have refused. A reader who takes the current sentence at face value concludes a green battery covers the deployed reply path. It covers five sevenths of it. ## Acceptance - The doc names all five unconditional validators. - It names the two that are conditional and what conditions them. - The file stays inside the 80-line and 4000-character caps. ## Next owner Engineer. Closed by the pull request that lands the change. `ward exec gate` PASS on all six steps. closes #730

coilyco-ops added 1 commit

2026-08-13 21:06:07 +00:00

docs(battery): name the two deployed checks the evaluation does not run

ci / image-build (pull_request) Successful in 22s

Details

ci / test (pull_request) Successful in 39s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

7837954901

The opening named three deployed validators as assumed. Five run on every
case, and two more that run on every deployed reply do not: markup needs
forbid_tool_call_markup, and the identifier guard has no evaluation equivalent
beyond the narrower checkUserIDEcho under forbid_principal_echo.

A reader taking the sentence at face value concludes a green battery covers
the deployed reply path. It covers five sevenths of it, which is what 301 and
310 keep rediscovering from opposite ends.

78 lines and 3942 characters, inside both caps.

closes #730

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 21:06:07 +00:00

coilyco-ops referenced this pull request

2026-08-13 21:06:31 +00:00

A reply that is raw tool-call markup passes the deployment gate, and nothing in the reply path strips it #301

coilyco-ops merged commit 31ff966067 into main

2026-08-13 21:07:14 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 21:07:16 +00:00

Merge pull request 'docs(battery): name the two deployed checks the evaluation does not run' (#731) from docs/the-battery-assumes-two-checks-it-does-not-run into main