grep for --- FAIL reports a package that does not compile as green #653

New issue

Closed

opened 2026-08-13 17:57:25 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 17:57:25 +00:00

Member

Filed by Angie (ENG, claude seat) after doing it to myself twice in five minutes. Small, and it is the same shape as everything else this session found.

What happened

I was mutation-testing a command and checking results with:

ward exec test 2>&1 | grep -c "\-\-\- FAIL"

It returned 0, which I read as "no failures, mutation not caught". The truth was that the package did not compile — my refactor had been reverted by a restore while the test file still referenced the new function:

cmd/sirens-echo-access-check/main_test.go:96:10: undefined: run

A compile error is not a --- FAIL line. So the grep was structurally incapable of seeing the worst outcome, and reported it as the best one.

Why it is worth recording

I concluded from that zero that a mutation had not been caught, and nearly went looking for a weakness in a test that was fine. The measurement was broken and the thing it measured was fine — which is the inverse of the usual worry and much harder to notice.

It is the same class as #592: a default state indistinguishable from a real answer. Zero failures and zero-because-nothing-ran are the same output.

What to do instead

ward exec gate and ward exec test both exit non-zero on a compile failure. The exit code is the signal; the grep is a convenience over the text. Any check written as grep -c over test output should be ward exec test with its status inspected, or at minimum grep for ok per package rather than counting failures.

Not proposing tooling

No hook can stop someone grepping. This is here because the next person to mutation-test something will reach for the same one-liner, and because I want the instance recorded next to the six others rather than remembered as a private slip.

**Filed by Angie (ENG, `claude` seat)** after doing it to myself twice in five minutes. Small, and it is the same shape as everything else this session found. ## What happened I was mutation-testing a command and checking results with: ```sh ward exec test 2>&1 | grep -c "\-\-\- FAIL" ``` It returned **0**, which I read as "no failures, mutation not caught". The truth was that the package **did not compile** — my refactor had been reverted by a restore while the test file still referenced the new function: ``` cmd/sirens-echo-access-check/main_test.go:96:10: undefined: run ``` A compile error is not a `--- FAIL` line. So the grep was structurally incapable of seeing the worst outcome, and reported it as the best one. ## Why it is worth recording I concluded from that zero that a mutation had **not** been caught, and nearly went looking for a weakness in a test that was fine. The measurement was broken and the thing it measured was fine — which is the inverse of the usual worry and much harder to notice. It is the same class as https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/592: a default state indistinguishable from a real answer. Zero failures and zero-because-nothing-ran are the same output. ## What to do instead `ward exec gate` and `ward exec test` both **exit non-zero** on a compile failure. The exit code is the signal; the grep is a convenience over the text. Any check written as `grep -c` over test output should be `ward exec test` with its status inspected, or at minimum grep for `ok ` per package rather than counting failures. ## Not proposing tooling No hook can stop someone grepping. This is here because the next person to mutation-test something will reach for the same one-liner, and because I want the instance recorded next to the six others rather than remembered as a private slip.

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 17:57:45 +00:00

test(access): prove the exit codes deploy's CI keys on #654

coilyco-ops closed this issue

2026-08-13 17:58:49 +00:00

coilyco-ops referenced this issue

2026-08-13 18:30:19 +00:00

fix(fetch): a wildcard needs a real first label, and normalises its own input #673

coilyco-ops referenced this issue

2026-08-13 19:06:58 +00:00

aosguard ops forgejo issue-comment create does not exist, and discarding its output makes an entire session of comments vanish #693

coilyco-ops referenced this issue