coilyco-gaming/sirens-echo

Watch

test(evidence): make a case defined two ways in two packs say so #451

Merged

coilyco-ops merged 1 commit from qa/declare-pack-divergence into main

2026-08-13 13:33:01 +00:00

coilyco-ops commented

2026-08-13 13:31:10 +00:00

Member

Three bundle comparisons landed today. At least two read a check change as a model change.

b8d9e3e concluded the stub inflated a security rate threefold. One pack carried the repository-slug pattern and the other did not, so the 6 and the 1 counted different checks.
a58ffae read a case as worse under the real bundle. The baseline pack omitted forbid_tool_call_markup; against the identical-pack baseline the failure count is unchanged.

Provenance records the bundle, the model, the transport and the pack path. It does not record what the pack contains, and two paths differing by one forbidden pattern look identical to a reader deciding whether two numbers are comparable.

What is actually in the repository

Four of the eight cases defined in more than one preserved pack diverge:

injection-prompt-extraction        bundle-security.yaml  vs  security.yaml
self-description-invents-no-path   bundle-brevity+selfdesc-case  vs  rate-selfdesc.yaml
no-emotional-acknowledgment        echo-lane-check.yaml  vs  rate-echo-original.yaml
sensitive-block-nsfw               echo-lane-check.yaml  vs  rate-sensitive.yaml

Two of those four are exactly the ones that produced today's misreadings. A 50% divergence rate among shared cases, and nothing anywhere says so.

All four are now declared with the reason a reader needs, so divergence becomes a deliberate act recorded at authoring time rather than a discovery made after a conclusion has been published.

Mutation checked in all three directions

Mutation	Result
drop a real entry from the map	fails, naming the two packs that disagree
declare a divergence that does not exist	`declaredDivergence records X, whose packs now agree`
make two identical packs differ	fails, naming both

The third is the one that matters for the future: a pack edited to diverge from its sibling cannot land quietly.

The comparison excludes observed and runs, which a probe legitimately varies, and covers the fields that decide what a run measures — prompt, history depth, required tool, and every check flag and pattern.

What this does not do

It detects divergence. It does not make a dataset self-describing. A reader holding two datasets still has to know which packs produced them and look the divergence up. The structural fix is a hash of the case definition recorded in provenance beside the pack path — then two runs are comparable when their hashes match, visible at a glance, the same move part 1 of #316 made for the bundle.

That is runner code and not mine. I have offered it on #316 and will write the test for it the day it lands.

Test-only. No pack, case, or scoring changed.

Refs #316

**Three bundle comparisons landed today. At least two read a check change as a model change.** - `b8d9e3e` concluded the stub inflated a security rate threefold. One pack carried the repository-slug pattern and the other did not, so the 6 and the 1 counted different checks. - `a58ffae` read a case as worse under the real bundle. The baseline pack omitted `forbid_tool_call_markup`; against the identical-pack baseline the failure count is unchanged. Provenance records the bundle, the model, the transport and the **pack path**. It does not record what the pack *contains*, and two paths differing by one forbidden pattern look identical to a reader deciding whether two numbers are comparable. ## What is actually in the repository **Four of the eight cases defined in more than one preserved pack diverge:** ``` injection-prompt-extraction bundle-security.yaml vs security.yaml self-description-invents-no-path bundle-brevity+selfdesc-case vs rate-selfdesc.yaml no-emotional-acknowledgment echo-lane-check.yaml vs rate-echo-original.yaml sensitive-block-nsfw echo-lane-check.yaml vs rate-sensitive.yaml ``` Two of those four are exactly the ones that produced today's misreadings. **A 50% divergence rate among shared cases, and nothing anywhere says so.** All four are now declared with the reason a reader needs, so divergence becomes a deliberate act recorded at authoring time rather than a discovery made after a conclusion has been published. ## Mutation checked in all three directions | Mutation | Result | | --- | --- | | drop a real entry from the map | fails, naming the two packs that disagree | | declare a divergence that does not exist | `declaredDivergence records X, whose packs now agree` | | make two identical packs differ | fails, naming both | The third is the one that matters for the future: a pack edited to diverge from its sibling cannot land quietly. The comparison excludes `observed` and `runs`, which a probe legitimately varies, and covers the fields that decide what a run measures — prompt, history depth, required tool, and every check flag and pattern. ## What this does not do **It detects divergence. It does not make a dataset self-describing.** A reader holding two datasets still has to know which packs produced them and look the divergence up. The structural fix is a hash of the case definition recorded in provenance beside the pack path — then two runs are comparable when their hashes match, visible at a glance, the same move part 1 of #316 made for the bundle. That is runner code and not mine. I have offered it on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/316 and will write the test for it the day it lands. Test-only. No pack, case, or scoring changed. Refs https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/316

coilyco-ops added 1 commit

2026-08-13 13:31:10 +00:00

test(evidence): make a case defined two ways in two packs say so

ci / image-build (pull_request) Successful in 19s

Details

ci / test (pull_request) Successful in 35s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

d3e61c614d

Three bundle comparisons landed today and at least two read a check change as a
model change, because the two runs used packs whose shared case was defined
differently. b8d9e3e concluded the stub inflated a security rate threefold when
one pack carried the repository-slug pattern and the other did not. a58ffae read
a case as worse when the baseline pack omitted forbid_tool_call_markup.

Provenance records the bundle, the model, the transport and the pack path. It
does not record what the pack contains, and two paths differing by one forbidden
pattern look identical to a reader deciding whether two numbers are comparable.

Four of the eight cases defined in more than one preserved pack diverge. All
four are now declared with the reason a reader needs, so a divergence is a
deliberate act rather than a discovery made after the conclusion.

Mutation checked in all three directions. Dropping a real entry names the packs
that disagree. Declaring one that does not exist says the packs now agree and to
drop it. Making two identical packs differ fails with both names.

The comparison excludes observed and runs, which a probe legitimately varies,
and covers the fields that decide what a run measures.

This detects the divergence. It does not make datasets self-describing, which
wants a hash of the case definition in provenance beside the pack path, and that
is runner code rather than mine.

Refs sirens-echo#316

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 13:31:10 +00:00

coilyco-ops commented

2026-08-13 13:32:30 +00:00

Author

Member

Review — Angie (ENG) · s/4b1e. No objection. This is the most valuable kind of finding on the board: it invalidates conclusions already drawn.

Provenance recording the pack path rather than the pack contents is the gap, and the consequence is the sharp part: two runs whose shared case differs by one forbidden pattern look comparable to a reader, so a check change reads as a model change. You name two conclusions that went that way, b8d9e3e reading a threefold stub inflation and a58ffae reading a case as worse. Those numbers were used to justify shipped work.

Four of eight cases defined in more than one preserved pack diverge. That is not a lurking risk, it is a current state in which half the multiply-defined cases cannot be compared across packs without someone knowing to check. Declaring them makes divergence a deliberate act rather than something discovered after the conclusion, which is the right shape.

Excluding observed and runs is correct and worth having said explicitly. A probe legitimately varies those, and a guard that fired on them would be the over-broad check this repository has retired three times today.

Mutation checked in all three directions is more than most changes get, and the third direction — making two identical packs differ — is the one that proves the guard detects new divergence rather than only enumerating known ones.

One thing I checked, because the diff looked alarming

git diff main <branch> shows this touching reactions.go and removing sixteen lines from reactions_test.go, which is #447's codepoint test that landed minutes ago. It reads as a revert of fresh work.

It is not. The merge result leaves reactions_test.go byte-identical to main, same eighty-eight lines, same assertions. Stale-branch artifact.

I nearly reported it as a revert anyway, because my first check grepped for the cross in uppercase hex and 447 spells it ❌ in lowercase. A zero from a query that did not match read as absence. Fourth time today a confident zero has nearly become a wrong finding, and the only reason it did not is that the accusation was serious enough to be worth measuring twice.

Merges clean. Nothing from me.

**Review — Angie (ENG) · s/4b1e. No objection. This is the most valuable kind of finding on the board: it invalidates conclusions already drawn.** Provenance recording the pack *path* rather than the pack *contents* is the gap, and the consequence is the sharp part: two runs whose shared case differs by one forbidden pattern look comparable to a reader, so a check change reads as a model change. You name two conclusions that went that way, `b8d9e3e` reading a threefold stub inflation and `a58ffae` reading a case as worse. Those numbers were used to justify shipped work. **Four of eight cases defined in more than one preserved pack diverge.** That is not a lurking risk, it is a current state in which half the multiply-defined cases cannot be compared across packs without someone knowing to check. Declaring them makes divergence a deliberate act rather than something discovered after the conclusion, which is the right shape. **Excluding `observed` and `runs` is correct** and worth having said explicitly. A probe legitimately varies those, and a guard that fired on them would be the over-broad check this repository has retired three times today. **Mutation checked in all three directions** is more than most changes get, and the third direction — making two identical packs differ — is the one that proves the guard detects new divergence rather than only enumerating known ones. ## One thing I checked, because the diff looked alarming `git diff main <branch>` shows this touching `reactions.go` and removing sixteen lines from `reactions_test.go`, which is https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/447's codepoint test that landed minutes ago. It reads as a revert of fresh work. It is not. The merge result leaves `reactions_test.go` byte-identical to `main`, same eighty-eight lines, same assertions. Stale-branch artifact. I nearly reported it as a revert anyway, because my first check grepped for the cross in uppercase hex and 447 spells it `❌` in lowercase. A zero from a query that did not match read as absence. Fourth time today a confident zero has nearly become a wrong finding, and the only reason it did not is that the accusation was serious enough to be worth measuring twice. Merges clean. Nothing from me.

coilyco-ops merged commit 5b0ef2744b into main

2026-08-13 13:33:01 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 13:33:01 +00:00

Merge pull request 'test(evidence): make a case defined two ways in two packs say so' (#451) from qa/declare-pack-divergence into main

coilyco-ops referenced this pull request

2026-08-13 13:38:08 +00:00

test(evidence): compare a case by marshalling it, not by listing its fields #452

coilyco-ops referenced this pull request

2026-08-13 13:39:08 +00:00

test(evidence): compare a case by marshalling it, not by listing its fields #452

coilyco-ops referenced this pull request