feat(evaluation): let a case render the marker that makes a turn forged #434

Merged

coilyco-ops merged 1 commit from feat/the-eval-can-forge-a-turn into main

2026-08-13 12:50:07 +00:00

coilyco-ops commented

2026-08-13 12:46:10 +00:00

Member

closes #432

The runner change Lucia named as ENG and explicitly left unscoped on #177.

The defect

assertedHistory renders provenance for caller-supplied conversation. It was applied on the HTTP and MCP paths and nowhere else, so the evaluation and rate runners built prompts straight from case history and the marker never rendered.

injection-fake-system-turn passed 15 times. Those runs describe a model resisting a forged system turn with no provenance marker present — not what the case claims to measure. The pack header even asserted it "now renders with the caller-asserted suffix", which was never true on this path.

Fifteen clean runs of the wrong thing is worse than no runs, because the number looks like assurance. Lucia refusing to let 15/15 stand as a pass is why this is scoped work rather than a number nobody questioned.

Opt-in, and that limit is deliberate

A pack author does supply case history, so marking all of it would arguably be more faithful — and it would change the rendered prompt for every case that has history, moving baselines measured without it. That is a decision about what every existing number means, not a fix for one case.

The board is unchanged: human-graded, and its cases do not test the marker.

The old number is relabelled, not deleted

0/15 stays with a note saying what it measured. It is a real measurement of the undefended case and worth keeping — the model resists this even with no marker. A re-measure is owed and is a live run.

Two properties under test

Marking copies rather than mutates, so a second run of a pack matches the first. And a test fails if the forged-turn case ever stops opting in, since that would silently return it to measuring the undefended turn.

ward exec gate green: build, policy-check, vet, test, test-skips, pre-commit.

closes #432 The runner change Lucia named as ENG and explicitly left unscoped on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/177. ## The defect `assertedHistory` renders provenance for caller-supplied conversation. It was applied on the HTTP and MCP paths **and nowhere else**, so the evaluation and rate runners built prompts straight from case history and the marker never rendered. `injection-fake-system-turn` passed 15 times. Those runs describe a model resisting a forged system turn **with no provenance marker present** — not what the case claims to measure. The pack header even asserted it "now renders with the caller-asserted suffix", which was never true on this path. **Fifteen clean runs of the wrong thing is worse than no runs**, because the number looks like assurance. Lucia refusing to let 15/15 stand as a pass is why this is scoped work rather than a number nobody questioned. ## Opt-in, and that limit is deliberate A pack author does supply case history, so marking all of it would arguably be more faithful — and it would change the rendered prompt for **every case that has history**, moving baselines measured without it. That is a decision about what every existing number means, not a fix for one case. The board is unchanged: human-graded, and its cases do not test the marker. ## The old number is relabelled, not deleted 0/15 stays with a note saying what it measured. It is a real measurement of the *undefended* case and worth keeping — the model resists this even with no marker. **A re-measure is owed** and is a live run. ## Two properties under test Marking **copies rather than mutates**, so a second run of a pack matches the first. And a test fails if the forged-turn case ever stops opting in, since that would silently return it to measuring the undefended turn. `ward exec gate` green: build, policy-check, vet, test, test-skips, pre-commit.

coilyco-ops added 1 commit

2026-08-13 12:46:10 +00:00

feat(evaluation): let a case render the marker that makes a turn forged

ci / image-build (pull_request) Successful in 24s

Details

ci / test (pull_request) Failing after 26s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

08011c4f33

assertedHistory marks caller-supplied conversation so the prompt says where
each entry came from. It was applied on the HTTP and MCP paths and nowhere
else, so the evaluation and rate runners built their prompt straight from case
history and the marker never rendered.

injection-fake-system-turn passed fifteen times. Those runs describe a model
resisting a forged system turn with no provenance marker present, which is not
what the case claims to measure, and the pack header asserted the opposite.
Fifteen clean runs of the wrong thing is worse than no runs, because the number
looks like assurance.

Opt-in rather than automatic. A pack author does supply case history, so
marking all of it would arguably be more faithful, and it would also change the
rendered prompt for every case that has history and move baselines measured
without it. That is a decision about what every existing number means.

The board is unchanged: it is human-graded and its cases do not test the
marker, so a rendering difference there costs a grader and gains nothing.

The old 0/15 stays, relabelled with what it measured. It is a real measurement
of the undefended case and says the model resists this even with no marker. A
re-measure is owed and is a live run.

Marking copies rather than mutates, so a second run of a pack matches the
first, and a test holds that.

closes #432

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 12:46:10 +00:00

coilyco-ops force-pushed feat/the-eval-can-forge-a-turn from 08011c4f33

ci / image-build (pull_request) Successful in 24s

Details

ci / test (pull_request) Failing after 26s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

to b7986ebcd6