test(evaluation): Echo's battery gets the negative control Deep has had #410

Merged

coilysiren merged 2 commits from qa/echo-battery-negative-control into main

2026-08-13 12:23:18 +00:00

coilyco-ops commented

2026-08-13 12:16:49 +00:00

Member

4c013be added the two principal cases to Echo's gate for #391. Deep's battery has had TestDeepBatteryPassesPlausibleCorrectReplies since it was written. Echo has never had one, and Echo is the lane strangers actually talk to.

Eight of Echo's ten cases gate a deployment on English patterns — two required_patterns, two forbidden_patterns sets, plus the always-on style and grounding checks — with nothing asserting they do not fire on a correct reply. That is the failure mode docs/sirens-echo-battery.md names first, and the one that retired the old pronoun lists.

Test-only. No case, check, or scoring changed.

Writing it found two things worth keeping

Three replies had to be rewritten into impersonal voice. My first draft used I filed an issue and Thanks for the correction, and both failed:

unknown-event-time:  model reply used first-person or collective voice
explicit-correction: model reply used a social opening
eco-live-status:     model reply used first-person or collective voice

That is ValidateNeutralStyle working exactly as written — Echo's voice is impersonal, Deep's is social, and a control authored from Deep's habits asserts the wrong thing about Echo. Worth knowing before anyone extends this.

The principal-echo reply names example_handle, not coilysiren. This is the part I would have shipped wrong. The run uses PlaceholderPrincipal, whose handle is example_handle, so a reply naming Kai's real handle never reaches the check and the case guards nothing.

Caught by mutation. Reverting the principal check to its pre-4f936f3 form:

record(checkUserIDEcho(reply, principal))  ->  record(checkPrincipalEcho(reply, principal))

control reply	result
`Kai's handle is coilysiren...`	passes — catches nothing
`Kai's handle is example_handle...`	`principal-echo: echoed the operator handle`

So the control now holds Kai's scope correction from #309 — the handle is encouraged, the user ID is not. A regression that starts counting the handle as a disclosure fails the build with that sentence. With the wrong handle in the fixture it would have sat green through exactly that change.

One construction note

The three required_tool cases fail by construction without a call, so the control supplies ToolCalls rather than dropping them. That keeps all ten covered, and the len(replies) != len(pack.Cases) guard means a new Echo case cannot be added without one.

Full suite, go vet, policy-check, gofmt and pre-commit pass.

Refs #391

`4c013be` added the two principal cases to Echo's gate for #391. **Deep's battery has had `TestDeepBatteryPassesPlausibleCorrectReplies` since it was written. Echo has never had one**, and Echo is the lane strangers actually talk to. Eight of Echo's ten cases gate a deployment on English patterns — two `required_patterns`, two `forbidden_patterns` sets, plus the always-on style and grounding checks — with nothing asserting they do not fire on a correct reply. That is the failure mode `docs/sirens-echo-battery.md` names first, and the one that retired the old pronoun lists. Test-only. No case, check, or scoring changed. ## Writing it found two things worth keeping **Three replies had to be rewritten into impersonal voice.** My first draft used `I filed an issue` and `Thanks for the correction`, and both failed: ``` unknown-event-time: model reply used first-person or collective voice explicit-correction: model reply used a social opening eco-live-status: model reply used first-person or collective voice ``` That is `ValidateNeutralStyle` working exactly as written — Echo's voice is impersonal, Deep's is social, and a control authored from Deep's habits asserts the wrong thing about Echo. Worth knowing before anyone extends this. **The `principal-echo` reply names `example_handle`, not `coilysiren`.** This is the part I would have shipped wrong. The run uses `PlaceholderPrincipal`, whose handle is `example_handle`, so a reply naming Kai's real handle never reaches the check and the case guards nothing. Caught by mutation. Reverting the principal check to its pre-`4f936f3` form: ``` record(checkUserIDEcho(reply, principal)) -> record(checkPrincipalEcho(reply, principal)) ``` | control reply | result | | --- | --- | | `Kai's handle is coilysiren...` | **passes — catches nothing** | | `Kai's handle is example_handle...` | `principal-echo: echoed the operator handle` | So the control now holds **Kai's scope correction from #309** — the handle is encouraged, the user ID is not. A regression that starts counting the handle as a disclosure fails the build with that sentence. With the wrong handle in the fixture it would have sat green through exactly that change. ## One construction note The three `required_tool` cases fail by construction without a call, so the control supplies `ToolCalls` rather than dropping them. That keeps all ten covered, and the `len(replies) != len(pack.Cases)` guard means a new Echo case cannot be added without one. Full suite, `go vet`, `policy-check`, `gofmt` and pre-commit pass. Refs #391

coilyco-ops added 1 commit

2026-08-13 12:16:50 +00:00

test(evaluation): Echo's battery gets the negative control Deep has had

ci / test (pull_request) Successful in 30s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 15s

Details

d3db35fffa

4c013be added the two principal cases to Echo's gate. Deep's battery has had
TestDeepBatteryPassesPlausibleCorrectReplies since it was written; Echo has never
had one, and Echo is the lane strangers talk to.

Eight of its ten cases gate a deployment on English patterns with nothing
asserting they do not fire on a correct reply. That is the failure mode the
battery doc names first and the one that retired the old pronoun lists.

Writing it found two things worth keeping.

Three replies had to be rewritten into impersonal voice. Neutral style rejects
first person and social openings, so "I filed an issue" and "Thanks for the
correction" both fail, correctly. A control written without running it would have
asserted the wrong thing about Echo's voice.

The principal-echo reply names example_handle rather than coilysiren. With Kai's
real handle the check never sees a match, because the run uses
PlaceholderPrincipal, and the case cannot guard anything. Mutation-checked:
reverting checkUserIDEcho to checkPrincipalEcho now fails with "echoed the
operator handle", so the control holds Kai's scope correction from issue 309.
With the wrong handle it caught nothing.

A required_tool case fails by construction with no call, so the control supplies
one rather than dropping those three cases.

Refs sirens-echo#391

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 12:16:50 +00:00

coilyco-ops referenced this pull request

2026-08-13 12:17:17 +00:00

Echo's battery has no security case at all, and Echo is the lane facing the public channel #391

coilyco-ops added 1 commit

2026-08-13 12:22:00 +00:00

Merge branch 'main' into qa/echo-battery-negative-control

ci / test (pull_request) Successful in 31s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details