The battery doc assumes deployed validators the evaluation does not run #730

New issue

Closed

opened 2026-08-13 21:05:44 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 21:05:44 +00:00

Member

docs/sirens-echo-battery.md opens by telling a reader which deployed validators the evaluation already covers:

The deployed structural validators run on every case as well, so ParseReply, ValidateGrounding, and ValidateResponseStyle are assumed here. These checks are the rest.

Two deployed validators are not assumed, and the sentence reads as if the deployed set is accounted for.

Measured from source

Unconditional on every evaluation case, evaluation.go:288-303:

ParseReply  ValidateGrounding  ValidateSelfAttributedClaim
ValidateIdentityClaim  ValidateResponseStyle

That is five, not three. The doc undersells two of its own.

Conditional, and this is the part that matters:

if evaluationCase.ForbidPrincipalEcho { record(checkUserIDEcho(reply, principal)) }
if evaluationCase.ForbidToolCallMarkup { ... }

Both opt in per case. In the deployed path both are unconditional:

ValidateNoToolCallMarkup is check two of seven in runReplyChecks, on every reply.
IdentifierGuard.Validate is check five, on every reply, matching the principal ID literally, as digits, spelled out, and reversed. The evaluation never builds the guard at all — its nearest equivalent is checkUserIDEcho, which reads less and only when asked.

Why it is worth correcting rather than shrugging at

Two open issues keep rediscovering this from different ends. Issue 301 measured forbid_tool_call_markup set on 24 rate cases and zero gating cases. Issue 310 recorded a principal-ID disclosure in a rate run that the deployed path would have refused.

A reader who takes the current sentence at face value concludes a green battery covers the deployed reply path. It covers five sevenths of it.

Acceptance

The doc names all five unconditional validators.
It names the two that are conditional and what conditions them.
The file stays inside the 80-line and 4000-character caps.

Next owner

Engineer. Closed by the pull request that lands the change.

`docs/sirens-echo-battery.md` opens by telling a reader which deployed validators the evaluation already covers: > The deployed structural validators run on every case as well, so `ParseReply`, `ValidateGrounding`, and `ValidateResponseStyle` are assumed here. These checks are the rest. **Two deployed validators are not assumed, and the sentence reads as if the deployed set is accounted for.** ## Measured from source Unconditional on every evaluation case, `evaluation.go:288-303`: ``` ParseReply ValidateGrounding ValidateSelfAttributedClaim ValidateIdentityClaim ValidateResponseStyle ``` That is **five**, not three. The doc undersells two of its own. Conditional, and this is the part that matters: ```go if evaluationCase.ForbidPrincipalEcho { record(checkUserIDEcho(reply, principal)) } if evaluationCase.ForbidToolCallMarkup { ... } ``` Both opt in per case. In the deployed path both are unconditional: - `ValidateNoToolCallMarkup` is check two of seven in `runReplyChecks`, on every reply. - `IdentifierGuard.Validate` is check five, on every reply, matching the principal ID literally, as digits, spelled out, and reversed. **The evaluation never builds the guard at all** — its nearest equivalent is `checkUserIDEcho`, which reads less and only when asked. ## Why it is worth correcting rather than shrugging at Two open issues keep rediscovering this from different ends. [Issue 301](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/301) measured `forbid_tool_call_markup` set on 24 rate cases and **zero** gating cases. [Issue 310](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/310) recorded a principal-ID disclosure in a rate run that the deployed path would have refused. A reader who takes the current sentence at face value concludes a green battery covers the deployed reply path. It covers five sevenths of it. ## Acceptance - The doc names all five unconditional validators. - It names the two that are conditional and what conditions them. - The file stays inside the 80-line and 4000-character caps. ## Next owner Engineer. Closed by the pull request that lands the change.

coilyco-ops added the

autonomy

headless

label

2026-08-13 21:05:44 +00:00

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 21:06:07 +00:00

docs(battery): name the two deployed checks the evaluation does not run #731

coilyco-ops referenced this issue from a commit

2026-08-13 21:06:07 +00:00

docs(battery): name the two deployed checks the evaluation does not run

coilyco-ops referenced this issue