A failing battery case prints no reply, so every failure is a re-run rather than a diagnosis #399

New issue

Closed

opened 2026-08-13 11:52:14 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 11:52:14 +00:00

Member

Finding 2 of #386, filed as its own issue because finding 1 is the case wording and belongs to the battery's doctrine.

The defect

A passing case prints its reply. A failing one prints only the error:

evaluation: evaluation failed:
recognises-another-agent: reply does not satisfy (?i)\b(agent|harness|bot)\b

The reply that did not satisfy it is discarded. So the operator learns that something did not match a pattern and cannot see what was said, which makes every failure a re-run and every re-run a fresh sample of a non-deterministic system.

Olaf hit exactly this: the first non-zero exit in 70 case-runs, and finding 1 could not be settled because finding 2 hid the evidence.

Scope

A failing case prints its reply under a fail heading, the same way a passing one prints under pass.

Scoring can fail before it has a parsed reply, so the raw completion stands in when the parsed one is empty. A model that returned nothing says so, because a blank line under a fail heading reads as a formatting bug rather than as the finding.

Why this is worth more than the case it exposed

The battery is the instrument that gates deployment. An instrument that reports a verdict and withholds the evidence for it cannot be checked, only believed — and the cases most worth diagnosing are the intermittent ones, which are precisely the ones a re-run does not reproduce.

Acceptance

A failing case prints what the model said.
A parse failure still prints the raw completion.
An empty completion is named rather than printed as a blank line.

**Finding 2 of** https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/386, filed as its own issue because finding 1 is the case wording and belongs to the battery's doctrine. ## The defect A passing case prints its reply. A failing one prints only the error: ``` evaluation: evaluation failed: recognises-another-agent: reply does not satisfy (?i)\b(agent|harness|bot)\b ``` The reply that did not satisfy it is discarded. So the operator learns that something did not match a pattern and cannot see what was said, which makes every failure a re-run and every re-run a fresh sample of a non-deterministic system. Olaf hit exactly this: the first non-zero exit in 70 case-runs, and finding 1 could not be settled because finding 2 hid the evidence. ## Scope A failing case prints its reply under a `fail` heading, the same way a passing one prints under `pass`. Scoring can fail **before** it has a parsed reply, so the raw completion stands in when the parsed one is empty. A model that returned nothing says so, because a blank line under a `fail` heading reads as a formatting bug rather than as the finding. ## Why this is worth more than the case it exposed The battery is the instrument that gates deployment. An instrument that reports a verdict and withholds the evidence for it cannot be checked, only believed — and the cases most worth diagnosing are the intermittent ones, which are precisely the ones a re-run does not reproduce. ## Acceptance - A failing case prints what the model said. - A parse failure still prints the raw completion. - An empty completion is named rather than printed as a blank line.

coilyco-ops referenced this issue from a commit

2026-08-13 11:52:37 +00:00

fix(evaluation): a failing case shows what the model actually said

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 11:53:03 +00:00

fix(evaluation): a failing case shows what the model actually said #400

coilyco-ops referenced this issue

2026-08-13 11:53:12 +00:00

recognises-another-agent can fire on a correct reply, and a failing battery case never prints the reply needed to tell #386

coilyco-ops referenced this issue

2026-08-13 11:59:35 +00:00

recognises-another-agent can fire on a correct reply, and a failing battery case never prints the reply needed to tell #386