test(rate): pin the shared scorer's check order #250

Merged

coilysiren merged 1 commit from test/score-order into main

2026-08-13 04:20:00 +00:00

coilyco-ops commented

2026-08-13 04:18:52 +00:00

Member

Tests only, no behavior change. Follows b128603.

Why

b128603 extracted ScoreEvaluationCase so the rate runner and the gate share one set of checks. That is the right call for the reason the commit gives — a second checking system would drift, and the rate would then measure something the gate does not enforce.

I verified the extraction is faithful: same checks, same order, same error strings, and the only semantic change (returning the raw content when ParseReply fails) is discarded by the gate and needed by the rate runner.

But the refactor changed what the check order is worth. Inline in runEvaluation, the order only decided which message a failing gate printed. Shared, it decides two things:

the gate reports the first failing check
the rate runner attributes a rate to whichever check fired

Swap two checks and both runners keep working, stay green, and quietly attribute failures to the wrong cause. A rate is evidence, and evidence attributed to the wrong check is worse than no rate — it is the "measuring something other than what you think" failure that motivated sharing the scorer in the first place.

Nothing referenced ScoreEvaluationCase from a test. The gate's tests cover it indirectly, as the commit says, but indirect coverage does not pin order.

What this adds

internal/community/scoreorder_test.go:

…KeepsGateCheckOrder — five precedence boundaries: parse before all, grounding before identity, identity before style, forbidden-phrase last. Each probe reply trips more than one check, so only precedence can explain the result.
…AcceptsACleanReply — the control. Without it every precedence assertion would pass against a scorer that rejects everything.
…ReturnsTheReplyItRejected — the contract the rate runner depends on: a rejected reply is still returned so it can be persisted verbatim, including the raw content when parsing is what failed.

That last one matters because b128603 persists every reply on purpose — three first-pass findings in the source QA were defects in the check rather than in the agent, and only the text separated them.

Verified by mutation

Swapped ValidateGrounding and ValidateIdentityClaim:

--- FAIL: TestScoreEvaluationCaseKeepsGateCheckOrder
    grounding before identity: err = "model claimed to be a person",
      want it to contain "claimed an action the runtime has not performed"

Caught by the intended assertion. go vet, gofmt, full go test ./..., and pre-commit run --files all clean.

Separately, on the harness itself

I checked rate-deep against the five properties I asked for on the issue. It meets them, and adds three I did not ask for and should have — errors excluded from the denominator, replies persisted verbatim, and the gate's checks reused rather than reimplemented.

I confirmed the "never fails the build" property specifically: ci.yml never invokes it and .ward/ward.yaml marks it "Gates nothing." The verdict can fail, but only for someone who invoked the verb deliberately, which is correct for a measurement run.

Quail (QA)

Tests only, no behavior change. Follows `b128603`. ## Why `b128603` extracted `ScoreEvaluationCase` so the rate runner and the gate share one set of checks. That is the right call for the reason the commit gives — a second checking system would drift, and the rate would then measure something the gate does not enforce. I verified the extraction is faithful: same checks, same order, same error strings, and the only semantic change (returning the raw content when `ParseReply` fails) is discarded by the gate and needed by the rate runner. But the refactor **changed what the check order is worth**. Inline in `runEvaluation`, the order only decided which message a failing gate printed. Shared, it decides two things: - the gate reports the **first** failing check - the rate runner attributes a rate to **whichever check fired** Swap two checks and both runners keep working, stay green, and quietly attribute failures to the wrong cause. A rate is evidence, and evidence attributed to the wrong check is worse than no rate — it is the "measuring something other than what you think" failure that motivated sharing the scorer in the first place. Nothing referenced `ScoreEvaluationCase` from a test. The gate's tests cover it indirectly, as the commit says, but indirect coverage does not pin order. ## What this adds `internal/community/scoreorder_test.go`: - `…KeepsGateCheckOrder` — five precedence boundaries: parse before all, grounding before identity, identity before style, forbidden-phrase last. Each probe reply trips **more than one** check, so only precedence can explain the result. - `…AcceptsACleanReply` — the control. Without it every precedence assertion would pass against a scorer that rejects everything. - `…ReturnsTheReplyItRejected` — the contract the rate runner depends on: a rejected reply is still returned so it can be persisted verbatim, including the raw content when parsing is what failed. That last one matters because `b128603` persists every reply on purpose — three first-pass findings in the source QA were defects in the *check* rather than in the agent, and only the text separated them. ## Verified by mutation Swapped `ValidateGrounding` and `ValidateIdentityClaim`: ``` --- FAIL: TestScoreEvaluationCaseKeepsGateCheckOrder grounding before identity: err = "model claimed to be a person", want it to contain "claimed an action the runtime has not performed" ``` Caught by the intended assertion. `go vet`, `gofmt`, full `go test ./...`, and `pre-commit run --files` all clean. ## Separately, on the harness itself I checked `rate-deep` against the five properties I asked for on the issue. It meets them, and adds three I did not ask for and should have — errors excluded from the denominator, replies persisted verbatim, and the gate's checks reused rather than reimplemented. I confirmed the "never fails the build" property specifically: `ci.yml` never invokes it and `.ward/ward.yaml` marks it *"Gates nothing."* The verdict can fail, but only for someone who invoked the verb deliberately, which is correct for a measurement run. --- Quail (QA)

coilyco-ops added 1 commit

2026-08-13 04:18:52 +00:00

test(rate): pin the shared scorer's check order

ci / test (pull_request) Successful in 32s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 16s

Details

45e8682e4d

b128603 extracted ScoreEvaluationCase so the rate runner and the gate
apply one set of checks, which is the right call — a second checking
system would drift and the rate would then measure something the gate
does not enforce.

The extraction is faithful. Its check order is now load-bearing in a way
it was not when the code was inline: the gate reports the first failure,
and the rate runner attributes a rate to whichever check fired. Reorder
two checks and both runners keep working while quietly attributing
failures to the wrong cause.

Nothing referenced ScoreEvaluationCase from a test. These pin the order
across the parse, grounding, identity, style, and forbidden-phrase
boundaries, plus the contract the rate runner depends on: a rejected
reply is still returned so it can be recorded verbatim, including the
raw content when parsing is what failed.

Verified by swapping the grounding and identity checks, which fails with
"model claimed to be a person" where the order requires the grounding
error.

Refs: #191

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 04:18:52 +00:00