test(gate): the markup check every rate pack asks for, no gate does #709

Merged

coilyco-ops merged 2 commits from quail/gating-pack-markup-gap into main

2026-08-13 20:25:17 +00:00

coilyco-ops commented

2026-08-13 20:18:31 +00:00

Member

Makes the sirens-echo#301 gap visible in CI. Tests only; changes no pack and turns no check on.

The gap

forbid_tool_call_markup is implemented (evaluation.go:50, evaluation_checks.go:142) and asked for by 24 cases:

rate-echo 5   rate-deep 11   rate-fixture-deep 3
rate-fixture-tracker 3   rate-fixture-tracker-match 1

And by nothing that gates a deployment:

agent/evaluation.yaml         10 cases   0
agent/evaluation-deep.yaml     9 cases   0
agent/board-deep.yaml         10 cases   0

So the detector exists, is exercised, and is absent from the packs whose result decides whether a change ships. sirens-echo#301 is configuration rather than code, which is a smaller job than that issue currently describes.

Both directions asserted

Someone starting to close it:

agent/evaluation-deep.yaml now asks for the markup check on 1 of 9 cases, was 0.
If sirens-echo#301 is being closed, update this row; when every gating pack
covers every case, delete this test

And the reverse, which is the one worth having:

no rate pack asks for the tool-call markup check any more, so the only cases
exercising it are gone. See sirens-echo#301

Both are real mutation output. The case count is pinned too, so adding a case to a gating pack fails until someone decides whether it needs the check.

Why a test rather than the fix

Turning the check on across 29 gating cases changes what ships. That is sirens-echo#301's call and Lucia's issue, not a tidy-up I should make on the way past. This only ensures the gap is not invisible while the decision is pending, and that closing it is noticed.

ward exec gate PASS on every step.

Makes the sirens-echo#301 gap visible in CI. Tests only; changes no pack and turns no check on. ## The gap `forbid_tool_call_markup` is implemented (`evaluation.go:50`, `evaluation_checks.go:142`) and asked for by 24 cases: ``` rate-echo 5 rate-deep 11 rate-fixture-deep 3 rate-fixture-tracker 3 rate-fixture-tracker-match 1 ``` And by nothing that gates a deployment: ``` agent/evaluation.yaml 10 cases 0 agent/evaluation-deep.yaml 9 cases 0 agent/board-deep.yaml 10 cases 0 ``` So the detector exists, is exercised, and is absent from the packs whose result decides whether a change ships. sirens-echo#301 is configuration rather than code, which is a smaller job than that issue currently describes. ## Both directions asserted Someone starting to close it: ``` agent/evaluation-deep.yaml now asks for the markup check on 1 of 9 cases, was 0. If sirens-echo#301 is being closed, update this row; when every gating pack covers every case, delete this test ``` And the reverse, which is the one worth having: ``` no rate pack asks for the tool-call markup check any more, so the only cases exercising it are gone. See sirens-echo#301 ``` Both are real mutation output. The case count is pinned too, so adding a case to a gating pack fails until someone decides whether it needs the check. ## Why a test rather than the fix Turning the check on across 29 gating cases changes what ships. That is sirens-echo#301's call and Lucia's issue, not a tidy-up I should make on the way past. This only ensures the gap is not invisible while the decision is pending, and that closing it is noticed. `ward exec gate` PASS on every step.

coilyco-ops added 1 commit

2026-08-13 20:18:31 +00:00

test(gate): the markup check every rate pack asks for, no gate does

ci / image-build (pull_request) Successful in 20s

Details

ci / test (pull_request) Failing after 29s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

058eac0dd5

forbid_tool_call_markup is implemented and set on 24 cases across the
rate packs. All three packs that gate a deployment set it on none of
their 29, so a reply of raw tool-call markup ships green.

The counts are asserted rather than described, in both directions: the
gap cannot close silently, and it cannot widen either.

Refs sirens-echo#301

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 20:18:31 +00:00

coilyco-ops added 1 commit

2026-08-13 20:24:12 +00:00

Merge branch 'main' into quail/gating-pack-markup-gap

ci / image-build (pull_request) Successful in 19s

Details

ci / test (pull_request) Successful in 36s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped