test(evidence): preserve the probe packs twelve datasets cite and cannot reach #416

Merged

coilyco-ops merged 1 commit from qa/preserve-the-probe-packs into main

2026-08-13 12:28:26 +00:00

coilyco-ops commented

2026-08-13 12:25:00 +00:00

Member

Twelve of twenty committed datasets name their pack as a file under /tmp/probe. That is scratch space. The dataset records the method, the method was not in the repository, and the numbers could be read but not re-derived.

All ten distinct packs still existed on the machine that produced them. Nothing is lost — this window was open and is now closed.

This is load-bearing evidence, not old scratch

dataset	what it is cited for
`rate-deep-security-run3-postfix.yaml`	the numbers `e7a380d` used to justify three shipped security fixes
`rate-echo-receipt-run1.yaml`	the basis of the #341 verification
`probe-prompt-leakage-15runs.yaml`	the whole of #382, currently an open gate decision
`probe-echo-security-cases.yaml`	the 36-attempt measurement behind #391

I found this trying to answer a narrow question on #310 — whether any reply still contains an 18-digit identifier after 39de9fa. I could not classify most of the corpus by era, because the datasets could not say which checkout produced them.

What is in the change

Ten packs copied to evaluations/packs/, deliberately not agent/ — policy-check and the pack-coverage gate treat everything under agent/*.yaml as a shipping pack, and these are records rather than packs the product runs.

TestEveryCitedPackIsPreserved fails when a dataset cites a pack with no copy, naming both:

a dataset cites a pack that exists nowhere in the repository, so its numbers
cannot be re-derived. Copy the pack to evaluations/packs/ and keep the
dataset's original path as the record:
  rate-deep-security-run3-postfix.yaml cites /tmp/probe/security.yaml

It also fails if no dataset cites an out-of-repo pack, so it cannot quietly stop covering anything the way the guards on #291 did.

Two deliberate choices

The datasets are not edited. Their pack: field still reads /tmp/probe/x.yaml, because that is what produced them. Rewriting it would tidy the record rather than preserve it, which is the same line Lucia held on #313 for the contaminated datasets and I held on #304 for the misattributed ones. The test resolves by basename instead.

Three packs differ from their originals by one byte — a trailing blank line the end-of-file-fixer hook removed. I checked rather than assumed: all ten parse to identical structures.

What this does not fix

The reason those runs cite /tmp at all is that they were invoked outside ward, so SIRENS_ECHO_RUNNER was never set and runner: reads unrecorded on all twelve. Only one dataset in the repository carries a real checkout SHA (rate-echo-crossmodel-run1.yaml, 949f40d). Preserving the packs makes the method recoverable; it does not make the checkout recoverable, and that half is still open. cmd/sirens-echo-eval could resolve git rev-parse --short HEAD itself when the environment variable is absent, since it runs inside the checkout — but that is production code and not mine to change.

Full suite, go vet, policy-check, gofmt and pre-commit pass.

Refs #313

**Twelve of twenty committed datasets name their pack as a file under `/tmp/probe`.** That is scratch space. The dataset records the method, the method was not in the repository, and the numbers could be read but not re-derived. **All ten distinct packs still existed on the machine that produced them. Nothing is lost — this window was open and is now closed.** ## This is load-bearing evidence, not old scratch | dataset | what it is cited for | | --- | --- | | `rate-deep-security-run3-postfix.yaml` | the numbers `e7a380d` used to justify three shipped security fixes | | `rate-echo-receipt-run1.yaml` | the basis of the #341 verification | | `probe-prompt-leakage-15runs.yaml` | the whole of #382, currently an open gate decision | | `probe-echo-security-cases.yaml` | the 36-attempt measurement behind #391 | I found this trying to answer a narrow question on #310 — whether any reply still contains an 18-digit identifier after `39de9fa`. I could not classify most of the corpus by era, because the datasets could not say which checkout produced them. ## What is in the change Ten packs copied to `evaluations/packs/`, deliberately **not** `agent/` — `policy-check` and the pack-coverage gate treat everything under `agent/*.yaml` as a shipping pack, and these are records rather than packs the product runs. `TestEveryCitedPackIsPreserved` fails when a dataset cites a pack with no copy, naming both: ``` a dataset cites a pack that exists nowhere in the repository, so its numbers cannot be re-derived. Copy the pack to evaluations/packs/ and keep the dataset's original path as the record: rate-deep-security-run3-postfix.yaml cites /tmp/probe/security.yaml ``` It also fails if **no** dataset cites an out-of-repo pack, so it cannot quietly stop covering anything the way the guards on #291 did. ## Two deliberate choices **The datasets are not edited.** Their `pack:` field still reads `/tmp/probe/x.yaml`, because that is what produced them. Rewriting it would tidy the record rather than preserve it, which is the same line Lucia held on #313 for the contaminated datasets and I held on #304 for the misattributed ones. The test resolves by basename instead. **Three packs differ from their originals by one byte** — a trailing blank line the `end-of-file-fixer` hook removed. I checked rather than assumed: **all ten parse to identical structures.** ## What this does not fix The reason those runs cite `/tmp` at all is that they were invoked outside `ward`, so `SIRENS_ECHO_RUNNER` was never set and `runner:` reads `unrecorded` on all twelve. **Only one dataset in the repository carries a real checkout SHA** (`rate-echo-crossmodel-run1.yaml`, `949f40d`). Preserving the packs makes the method recoverable; it does not make the checkout recoverable, and that half is still open. `cmd/sirens-echo-eval` could resolve `git rev-parse --short HEAD` itself when the environment variable is absent, since it runs inside the checkout — but that is production code and not mine to change. Full suite, `go vet`, `policy-check`, `gofmt` and pre-commit pass. Refs #313

coilyco-ops added 1 commit

2026-08-13 12:25:01 +00:00

test(evidence): preserve the probe packs twelve datasets cite and cannot reach

ci / test (pull_request) Successful in 32s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 17s

Details

c8e3336377

Twelve of twenty committed datasets name their pack as a file under /tmp/probe.
That is scratch. The dataset records the method and the method was not in the
repository, so the numbers could be read but not re-derived.

This is not hypothetical and it is not old evidence. rate-deep-security-run3
-postfix.yaml is what e7a380d used to justify three shipped security fixes.
rate-echo-receipt-run1.yaml is the basis of the issue 341 verification.
probe-prompt-leakage-15runs.yaml is the whole of issue 382.

All ten distinct packs still existed on the machine that produced them, so
nothing is lost yet. Copied to evaluations/packs, which keeps them out of
agent/ where policy-check and the pack-coverage gate would treat them as
shipping packs rather than as records.

The datasets are not edited. Their pack field still reads /tmp/probe/x.yaml,
because that is what produced them and rewriting it would tidy the record
rather than preserve it. The test resolves by basename instead.

Three packs differ from their originals by one byte, a trailing blank line the
end-of-file hook removed. All ten parse to identical structures, checked rather
than assumed.

TestEveryCitedPackIsPreserved fails when a dataset cites a pack with no copy,
and names both the dataset and the path. It also fails if no dataset cites an
out-of-repo pack at all, so it cannot quietly stop covering anything.

Refs sirens-echo#313

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 12:25:01 +00:00

coilyco-ops merged commit 50e7b48b2e into main

2026-08-13 12:28:26 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 12:28:27 +00:00

Merge pull request 'test(evidence): preserve the probe packs twelve datasets cite and cannot reach' (#416) from qa/preserve-the-probe-packs into main

coilyco-ops referenced this pull request

2026-08-13 12:32:50 +00:00

fix(evidence): preserve the longform pack, which reddened main on arrival #421

coilyco-ops referenced this pull request

2026-08-13 12:37:04 +00:00