test(gate): count the fail headings in output, not the format string in source #429

Merged

coilyco-ops merged 1 commit from qa/pin-the-fail-heading-count into main

2026-08-13 12:45:44 +00:00

coilyco-ops commented

2026-08-13 12:43:05 +00:00

Member

The duplicate print from #407 is fixed, and the surviving line is the failedReply(reply, result) one — the right choice, since it degrades through result.Content to (the model returned no content) rather than printing a blank line under a fail heading.

TestAFailingCasePrintsOneBlockNotTwo pins it by counting the format string in evaluation.go. That catches the duplicate that happened. It misses one spelled differently — and a differently spelled duplicate is exactly how the first one arrived.

The two prints that landed were not identical:

fmt.Fprintf(output, "%s: fail\n%s\n\n", evaluationCase.ID, reply)
fmt.Fprintf(output, "%s: fail\n%s\n\n", evaluationCase.ID, failedReply(reply, result))

Different arguments, different intent, two authors. They shared a format string only by chance, and the existing guard rests entirely on that chance.

Verified rather than argued

Re-added print	source count	this test
identical to the existing one	fails	fails
`"%s: fail\n%s\n"` — one newline fewer	passes	fails

With the second mutation in place and only the source count guarding it, the suite is green while a single failing case prints its block twice. That is the original defect, back, undetected.

Both tests stay

The source count names the mechanism and is instant. This one drives RunEvaluation and counts headings in the actual output, so it holds whatever the second print is spelled like. Neither subsumes the other.

Test-only. No production change.

Refs #407

The duplicate print from https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/407 is fixed, and the surviving line is the `failedReply(reply, result)` one — the right choice, since it degrades through `result.Content` to `(the model returned no content)` rather than printing a blank line under a `fail` heading. `TestAFailingCasePrintsOneBlockNotTwo` pins it by counting the format string in `evaluation.go`. **That catches the duplicate that happened. It misses one spelled differently — and a differently spelled duplicate is exactly how the first one arrived.** The two prints that landed were not identical: ```go fmt.Fprintf(output, "%s: fail\n%s\n\n", evaluationCase.ID, reply) fmt.Fprintf(output, "%s: fail\n%s\n\n", evaluationCase.ID, failedReply(reply, result)) ``` Different arguments, different intent, two authors. **They shared a format string only by chance**, and the existing guard rests entirely on that chance. ## Verified rather than argued | Re-added print | source count | this test | | --- | --- | --- | | identical to the existing one | **fails** | fails | | `"%s: fail\n%s\n"` — one newline fewer | passes | **fails** | With the second mutation in place and only the source count guarding it, the suite is green while a single failing case prints its block twice. That is the original defect, back, undetected. ## Both tests stay The source count names the mechanism and is instant. This one drives `RunEvaluation` and counts headings in the actual output, so it holds whatever the second print is spelled like. Neither subsumes the other. Test-only. No production change. Refs https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/407

coilyco-ops added 1 commit

2026-08-13 12:43:06 +00:00

test(gate): count the fail headings in output, not the format string in source

ci / image-build (pull_request) Successful in 21s

Details

ci / test (pull_request) Successful in 33s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

4b07328090

The duplicate print from issue 407 is fixed and pinned by
TestAFailingCasePrintsOneBlockNotTwo, which counts occurrences of the format
string in evaluation.go. That catches the duplicate that happened.

It misses one spelled differently, and a differently spelled duplicate is how
the first one arrived. The two prints that landed were not identical: one passed
reply, the other failedReply(reply, result). They shared a format string only by
chance, and the guard rests on that chance.

Verified rather than argued. Re-adding an identical print fails the source
count. Re-adding one with "%s: fail\n%s\n" does not, and the suite stays green
with a failing case printing twice.

This drives RunEvaluation and counts headings in the output, so it holds whatever
the second print is spelled like. Both tests stay: the source count names the
mechanism, this one names the symptom.

Refs sirens-echo#407

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 12:43:06 +00:00

coilyco-ops referenced this pull request

2026-08-13 12:43:29 +00:00

Every failing battery case now prints its reply twice, because two seats fixed #386 and both landed #407

coilyco-ops merged commit ca2a1b118f into main

2026-08-13 12:45:44 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 12:45:45 +00:00

Merge pull request 'test(gate): count the fail headings in output, not the format string in source' (#429) from qa/pin-the-fail-heading-count into main