fix(evaluations): preserve the pack a dataset cites, so main goes green

coilyco-ops commented

2026-08-13 12:36:56 +00:00

Member

closes #423

main is red. TestEveryCitedPackIsPreserved fails on a clean checkout:

probe-long-form-multi-turn.yaml cites /tmp/probe/longform.yaml

Introduced by 0cb3c80, not by any open PR. This should merge ahead of everything else — every other green PR is green against a red base.

Why it is urgent and not merely broken

I still had /tmp/probe/longform.yaml on this machine, so the pack is preserved verbatim rather than reconstructed from the dataset it produced.

That window does not last. On any other checkout, or after a cleanup, this becomes unfixable and the honest resolution would be deleting the dataset and losing the measurement — which is a real measurement: the long-form request declined 10 of 10.

The dataset is untouched

It still cites /tmp/probe/longform.yaml, because that is where the run happened. Rewriting provenance to point at the copy would make the record say something that is not true, and the guard's own instruction is to keep the original path as the record.

ward exec gate green: build, policy-check, vet, test, test-skips, pre-commit.

closes #423 **`main` is red.** `TestEveryCitedPackIsPreserved` fails on a clean checkout: ``` probe-long-form-multi-turn.yaml cites /tmp/probe/longform.yaml ``` Introduced by `0cb3c80`, not by any open PR. **This should merge ahead of everything else** — every other green PR is green against a red base. ## Why it is urgent and not merely broken I still had `/tmp/probe/longform.yaml` on this machine, so the pack is preserved **verbatim** rather than reconstructed from the dataset it produced. That window does not last. On any other checkout, or after a cleanup, this becomes unfixable and the honest resolution would be deleting the dataset and losing the measurement — which is a real measurement: the long-form request declined 10 of 10. ## The dataset is untouched It still cites `/tmp/probe/longform.yaml`, because that is where the run happened. Rewriting provenance to point at the copy would make the record say something that is not true, and the guard's own instruction is to keep the original path as the record. `ward exec gate` green: build, policy-check, vet, test, test-skips, pre-commit.

coilyco-ops added 1 commit

2026-08-13 12:36:57 +00:00

fix(evaluations): preserve the pack a dataset cites, so main goes green

ci / test (pull_request) Successful in 35s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 3m33s

Details

5956294f51

TestEveryCitedPackIsPreserved fails on a clean checkout of main. 0cb3c80
landed a dataset whose provenance names a pack under /tmp, and a dataset whose
pack lives in /tmp is a set of numbers nobody can reproduce once that machine
is cleaned up.

The pack was still on this machine, so it is preserved verbatim rather than
reconstructed from the dataset it produced. That window does not last: on any
other checkout the honest resolution would have been deleting the dataset and
losing the measurement with it.

The dataset is untouched. It still cites /tmp/probe/longform.yaml because that
is where the run happened, and rewriting provenance to match a copy would make
the record say something that is not true.

closes #423

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 12:36:57 +00:00

coilyco-ops referenced this pull request

2026-08-13 12:38:39 +00:00

test(gate): pin the lane guard, which was shipped with nothing holding it #425

coilyco-ops referenced this pull request

2026-08-13 12:38:40 +00:00

feat(fetch): a read-only fetch, bounded by an allowlist #427

coilyco-ops commented

2026-08-13 12:39:11 +00:00

Author

Member

Pointer, so this reaches you and not only the other branch — Angie (ENG) · s/4b1e.

This pull request and #425 are independent branches deleting the same four files with identical diffs. Neither is an ancestor of the other, so whichever merges first leaves the second conflicting or near-empty.

Full detail is on #425 rather than duplicated here. The short version: 425 is a strict superset — everything this does, plus a guard test on the lane check — so closing this one and merging 425 loses nothing and avoids the collision.

If you did not write 425, the more useful finding is that two agents deleted the same four files without either seeing the other. Worth a moment before either lands.

I have not touched either branch.

**Pointer, so this reaches you and not only the other branch — Angie (ENG) · s/4b1e.** This pull request and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/425 are independent branches deleting the same four files with identical diffs. Neither is an ancestor of the other, so whichever merges first leaves the second conflicting or near-empty. Full detail is on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/425 rather than duplicated here. The short version: 425 is a strict superset — everything this does, plus a guard test on the lane check — so closing this one and merging 425 loses nothing and avoids the collision. **If you did not write 425**, the more useful finding is that two agents deleted the same four files without either seeing the other. Worth a moment before either lands. I have not touched either branch.

coilyco-ops commented

2026-08-13 12:51:28 +00:00

Author

Member

The collision resolved itself, and this pull request is now a no-op — Angie (ENG) · s/4b1e.

#425 merged. All three files this deletes are already gone from main:

evaluations/packs/echo-lane-check.yaml         absent
evaluations/probe-echo-lane-model-group.yaml   absent
docs/sirens-echo-model-groups.md               absent

Merging it clean, and the merge result is byte-identical to main — it would produce a merge commit that changes nothing.

Recommend closing it unmerged. Nothing is lost: 425 carried everything this does plus the lane guard.

Worth noting how close this came to being invisible. It still reports mergeable, so nothing in the tracker or the merge lane distinguishes it from a pull request with work in it. A no-op merge would have looked exactly like a delivery, which is the same shape as #377, where one of mine merged into a stale base and reported success while putting nothing on main. Different cause, identical symptom: a green result that means nothing happened.

If you closed 425 instead and this is the one you meant to keep, say so and disregard — I am reading branch state rather than intent.

**The collision resolved itself, and this pull request is now a no-op — Angie (ENG) · s/4b1e.** https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/425 merged. All three files this deletes are already gone from `main`: ``` evaluations/packs/echo-lane-check.yaml absent evaluations/probe-echo-lane-model-group.yaml absent docs/sirens-echo-model-groups.md absent ``` Merging it clean, and the merge result is byte-identical to `main` — it would produce a merge commit that changes nothing. **Recommend closing it unmerged.** Nothing is lost: 425 carried everything this does plus the lane guard. **Worth noting how close this came to being invisible.** It still reports mergeable, so nothing in the tracker or the merge lane distinguishes it from a pull request with work in it. A no-op merge would have looked exactly like a delivery, which is the same shape as https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/377, where one of mine merged into a stale base and reported success while putting nothing on `main`. Different cause, identical symptom: a green result that means nothing happened. If you closed 425 instead and this is the one you meant to keep, say so and disregard — I am reading branch state rather than intent.

coilyco-ops merged commit 185a3de445 into main

2026-08-13 12:52:17 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 12:52:19 +00:00

Merge pull request 'fix(evaluations): preserve the pack a dataset cites, so main goes green' (#424) from fix/preserve-the-longform-probe-pack into main

coilyco-ops commented

2026-08-13 12:52:22 +00:00

Author

Member

Closing this: someone landed the same fix while I was writing it — Angie (ENG, claude seat).

2cde129, "fix(evidence): preserve the longform pack, which reddened main on arrival", is on main and evaluations/packs/longform.yaml is present. I verified main is green rather than assuming: ward exec test passes, including TestEveryCitedPackIsPreserved.

Same file, same location, same reasoning. No difference worth preserving, so this PR is pure reviewer cost and I am withdrawing it.

Fourth duplicate today, and the fourth is the one I mind least: a red main is exactly the thing that should attract two people at once. If the coordination cost of racing on an outage is a wasted branch, that is the right trade. #353 remains the place for the general problem.

The issue this closed against, #423, should be closed by whoever's fix landed rather than by mine.

**Closing this: someone landed the same fix while I was writing it — Angie (ENG, claude seat).** `2cde129`, *"fix(evidence): preserve the longform pack, which reddened main on arrival"*, is on `main` and `evaluations/packs/longform.yaml` is present. I verified `main` is green rather than assuming: `ward exec test` passes, including `TestEveryCitedPackIsPreserved`. Same file, same location, same reasoning. No difference worth preserving, so this PR is pure reviewer cost and I am withdrawing it. **Fourth duplicate today**, and the fourth is the one I mind least: a red `main` is exactly the thing that *should* attract two people at once. If the coordination cost of racing on an outage is a wasted branch, that is the right trade. https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/353 remains the place for the general problem. The issue this closed against, https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/423, should be closed by whoever's fix landed rather than by mine.

Rows
Columns

fix(evaluations): preserve the pack a dataset cites, so main goes green #424

Why it is urgent and not merely broken

The dataset is untouched