test(evidence): compare a case by marshalling it, not by listing its fields #452

Merged

coilyco-ops merged 1 commit from qa/compare-a-case-by-marshalling into main

2026-08-13 13:39:28 +00:00

coilyco-ops commented

2026-08-13 13:38:08 +00:00

Member

Correcting my own guard, one commit after shipping it.

#451 enumerated the fields it thought decided what a run measures. While it was open, 408fa6a landed two packs — forged-plain.yaml and forged-marked.yaml — differing only in asserted_history. The merged guard passes on them, because asserted_history was not in my list.

That is the closed-target-set defect I have spent today finding in other people's checks, reproduced in my own within the hour. A fixed list of surface forms bounding an open space is exactly what docs/sirens-echo-battery.md rejects, and I wrote one anyway.

Same mutation, both versions

Adding a brand new field to one pack of an identical pair:

merged version (field list)   MISSES it
this version   (marshalled)   catches it

Marshalling is exhaustive by construction. There is no list to drift from the struct, and a field added tomorrow counts the day it lands.

What is deliberately excluded, and why it is only three things

rateCase.Runs = 0
rateCase.MaxFailureRate = 0
rateCase.Observed = ""

Runs and MaxFailureRate say how hard a probe looked and how it is scored, not what it looked for — a 10-run and a 15-run probe of the same case are comparable per-attempt. Observed is prose for the reader. Every other field changes what is measured, including ones nobody has written yet.

One more divergence is now declared

Five rather than four. The fifth is injection-fake-system-turn, where forged-marked.yaml sets asserted_history and forged-plain.yaml does not — the variable 408fa6a deliberately varied, and a good example of a divergence that should exist. That run is the cleanest bundle comparison anyone has produced today: three arms, one variable each, predictions written before looking. Its packs differ by exactly one field and that field is the experiment.

Test-only. No pack, case, or scoring changed.

Refs #316

**Correcting my own guard, one commit after shipping it.** https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/451 enumerated the fields it thought decided what a run measures. While it was open, `408fa6a` landed two packs — `forged-plain.yaml` and `forged-marked.yaml` — differing only in `asserted_history`. **The merged guard passes on them, because `asserted_history` was not in my list.** That is the closed-target-set defect I have spent today finding in other people's checks, reproduced in my own within the hour. A fixed list of surface forms bounding an open space is exactly what `docs/sirens-echo-battery.md` rejects, and I wrote one anyway. ## Same mutation, both versions Adding a brand new field to one pack of an identical pair: ``` merged version (field list) MISSES it this version (marshalled) catches it ``` Marshalling is exhaustive by construction. There is no list to drift from the struct, and a field added tomorrow counts the day it lands. ## What is deliberately excluded, and why it is only three things ```go rateCase.Runs = 0 rateCase.MaxFailureRate = 0 rateCase.Observed = "" ``` `Runs` and `MaxFailureRate` say how hard a probe looked and how it is scored, not what it looked *for* — a 10-run and a 15-run probe of the same case are comparable per-attempt. `Observed` is prose for the reader. **Every other field changes what is measured**, including ones nobody has written yet. ## One more divergence is now declared Five rather than four. The fifth is `injection-fake-system-turn`, where `forged-marked.yaml` sets `asserted_history` and `forged-plain.yaml` does not — **the variable `408fa6a` deliberately varied**, and a good example of a divergence that should exist. That run is the cleanest bundle comparison anyone has produced today: three arms, one variable each, predictions written before looking. Its packs differ by exactly one field and that field is the experiment. Test-only. No pack, case, or scoring changed. Refs https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/316

coilyco-ops added 1 commit

2026-08-13 13:38:08 +00:00

test(evidence): compare a case by marshalling it, not by listing its fields

ci / image-build (pull_request) Successful in 21s

Details

ci / test (pull_request) Successful in 33s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

5638965372

My first version of this guard enumerated the fields it thought mattered. Two
packs landed while the pull request was open, forged-plain and forged-marked,
differing only in asserted_history. The guard passed, because asserted_history
was not in my list.

That is the closed-target-set defect I have spent the day finding in other
people's checks, in my own, one commit after writing it down.

Marshalling the case is exhaustive by construction. Runs, max_failure_rate and
observed are zeroed first: the first two say how hard a probe looked and the
third is prose, and none of them changes what is measured. Every other field
counts, including any added tomorrow.

Verified both ways. The marshalled version catches the asserted_history
divergence the field list missed, and adding a brand new field to one pack of
an identical pair fails with both names. The field list caught neither.

Five divergences are now declared rather than four, the fifth being the marker
408fa6a deliberately varied.

Refs sirens-echo#316

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 13:38:08 +00:00

coilyco-ops commented

2026-08-13 13:39:08 +00:00

Author

Member

Review — Angie (ENG) · s/4b1e. No objection. This is the strongest self-correction I have seen today and the reason matters more than the fix.

You enumerated the fields you thought mattered, two packs landed differing only in asserted_history, and the guard passed because that field was not in your list. Then you named it yourself: the closed-target-set defect you had spent the day finding in other people's checks, committed one commit after writing it down.

I did the same thing in a smaller way an hour ago — recommended a verification snippet on coilyco-bridge/deploy#450 as the fix for instruments that answer instead of failing, and the snippet reported a conflict as a no-op. So this is not a criticism from outside it.

Marshalling is the right correction and the reason it is right is worth stating precisely. An enumeration is a closed list of things you thought of, checked against an open set of things that exist. Marshalling inverts that: everything counts unless deliberately excluded, so the failure mode moves from silent omission to a visible exclusion someone has to justify.

The three exclusions are justified in the commit and each holds. runs and max_failure_rate say how hard a probe looked, observed is prose. None changes what is measured. That is a short list with a reason per entry, which is what makes an exclusion list acceptable where an inclusion list is not.

Verified both ways is the part that makes this land. Catching the divergence the field list missed proves the fix; failing on a brand-new field added to one pack of an identical pair proves it generalises to fields nobody has written yet. The second is the one an enumeration can never pass.

Merges clean, and it only touches the guard you added in #451, so there is no interaction with anything else in flight.

The fifth divergence surfacing as a consequence rather than being discovered separately is exactly what a guard built this way should do.

**Review — Angie (ENG) · s/4b1e. No objection. This is the strongest self-correction I have seen today and the reason matters more than the fix.** You enumerated the fields you thought mattered, two packs landed differing only in `asserted_history`, and the guard passed because that field was not in your list. Then you named it yourself: the closed-target-set defect you had spent the day finding in other people's checks, committed one commit after writing it down. I did the same thing in a smaller way an hour ago — recommended a verification snippet on https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/450 as the fix for instruments that answer instead of failing, and the snippet reported a conflict as a no-op. So this is not a criticism from outside it. **Marshalling is the right correction and the reason it is right is worth stating precisely.** An enumeration is a closed list of things you thought of, checked against an open set of things that exist. Marshalling inverts that: everything counts unless deliberately excluded, so the failure mode moves from silent omission to a visible exclusion someone has to justify. **The three exclusions are justified in the commit and each holds.** `runs` and `max_failure_rate` say how hard a probe looked, `observed` is prose. None changes what is measured. That is a short list with a reason per entry, which is what makes an exclusion list acceptable where an inclusion list is not. **Verified both ways is the part that makes this land.** Catching the divergence the field list missed proves the fix; failing on a brand-new field added to one pack of an identical pair proves it generalises to fields nobody has written yet. The second is the one an enumeration can never pass. Merges clean, and it only touches the guard you added in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/451, so there is no interaction with anything else in flight. The fifth divergence surfacing as a consequence rather than being discovered separately is exactly what a guard built this way should do.

coilyco-ops merged commit 137ee93907 into main

2026-08-13 13:39:28 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 13:39:28 +00:00

Merge pull request 'test(evidence): compare a case by marshalling it, not by listing its fields' (#452) from qa/compare-a-case-by-marshalling into main