fix(evaluation): a failing case shows what the model actually said #400

Merged

coilyco-ops merged 2 commits from fix/a-failing-case-shows-the-reply into main

2026-08-13 12:01:00 +00:00

coilyco-ops commented

2026-08-13 11:53:03 +00:00

Member

closes #399

Finding 2 of #386. Olaf called it the more useful of the two findings and was right.

The defect

A passing case printed its reply. A failing one printed only the error:

recognises-another-agent: reply does not satisfy (?i)\b(agent|harness|bot)\b

The reply that did not satisfy it was discarded. So every failure becomes a re-run — and every re-run is a fresh sample of a non-deterministic system. The cases most worth diagnosing are the intermittent ones, which are exactly the ones a re-run does not reproduce.

Olaf hit this concretely: the first non-zero exit in 70 case-runs, and finding 1 could not be settled because finding 2 hid the evidence for it.

The fix

A failing case prints its reply under a fail heading, the same way a passing one prints under pass.

Scoring can fail before it has a parsed reply, so the raw completion stands in when the parsed one is empty — that is the tool-call-markup case, where the raw text is the whole point. A model that returned nothing says so, because a blank line under a fail heading reads as a formatting bug rather than as the finding.

Why it is worth more than the case that exposed it

The battery gates deployment. An instrument that reports a verdict and withholds the evidence for it cannot be checked, only believed.

Not in this PR

Finding 1, whether recognises-another-agent can fire on a correct reply. That is the battery's doctrine rather than its plumbing, and it stays on 386 — where it is now diagnosable, which it was not before.

ward exec gate green: build, policy-check, vet, test, test-skips, pre-commit.

closes #399 Finding 2 of https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/386. Olaf called it the more useful of the two findings and was right. ## The defect A passing case printed its reply. A failing one printed only the error: ``` recognises-another-agent: reply does not satisfy (?i)\b(agent|harness|bot)\b ``` The reply that did not satisfy it was **discarded**. So every failure becomes a re-run — and every re-run is a fresh sample of a non-deterministic system. The cases most worth diagnosing are the intermittent ones, which are exactly the ones a re-run does not reproduce. Olaf hit this concretely: the first non-zero exit in 70 case-runs, and finding 1 could not be settled because finding 2 hid the evidence for it. ## The fix A failing case prints its reply under a `fail` heading, the same way a passing one prints under `pass`. Scoring can fail **before** it has a parsed reply, so the raw completion stands in when the parsed one is empty — that is the tool-call-markup case, where the raw text is the whole point. A model that returned nothing says so, because a blank line under a `fail` heading reads as a formatting bug rather than as the finding. ## Why it is worth more than the case that exposed it The battery gates deployment. **An instrument that reports a verdict and withholds the evidence for it cannot be checked, only believed.** ## Not in this PR Finding 1, whether `recognises-another-agent` can fire on a correct reply. That is the battery's doctrine rather than its plumbing, and it stays on 386 — where it is now diagnosable, which it was not before. `ward exec gate` green: build, policy-check, vet, test, test-skips, pre-commit.

coilyco-ops added 1 commit

2026-08-13 11:53:03 +00:00

fix(evaluation): a failing case shows what the model actually said

ci / test (pull_request) Successful in 29s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 19s

Details

75b1e18cfc

A passing case printed its reply and a failing one printed only the error, so
the reply that did not satisfy the pattern was discarded. The operator learned
that something did not match and could not see what was said.

That makes every failure a re-run, and every re-run a fresh sample of a
non-deterministic system. The cases most worth diagnosing are the intermittent
ones, which are exactly the ones a re-run does not reproduce.

Scoring can fail before it has a parsed reply, so the raw completion stands in
when the parsed one is empty. A model that returned nothing says so, because a
blank line under a fail heading reads as a formatting bug rather than as the
finding.

The battery gates deployment. An instrument that reports a verdict and
withholds the evidence for it cannot be checked, only believed.

closes #399

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 11:53:04 +00:00

coilyco-ops referenced this pull request

2026-08-13 11:53:12 +00:00

recognises-another-agent can fire on a correct reply, and a failing battery case never prints the reply needed to tell #386

coilyco-ops added 1 commit

2026-08-13 11:57:38 +00:00

Merge branch 'main' into fix/a-failing-case-shows-the-reply

ci / test (pull_request) Successful in 31s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details