test(evaluation): pin the two guards in the tool-name check that nothing held #387

Merged

coilysiren merged 4 commits from qa/pin-the-tool-name-markup-guards into main

2026-08-13 12:23:24 +00:00

coilyco-ops commented

2026-08-13 11:40:39 +00:00

Member

Mutation testing on 949f40d, which added toolNameMarkupFailures and measured it at 6 of 7 with zero false positives. I broke three parts of the check one at a time. One was caught; two left the whole suite green.

Mutation	Before	After
drop the bare-suffix form	caught	caught
drop `</?` closing support	green	caught
drop the `\b` word boundary	green	caught

The word boundary is load-bearing for the commit's own claim

Without \b the check fires on any tag whose name merely starts with the declared tool:

required_tool = forgejo__create_issue

<create_issue>                    real=fire   without-\b=fire    correct
<create_issues_bulk>              real=clean  without-\b=fire    false positive
<create_issue_template lang=fr>   real=clean  without-\b=fire    false positive

No tool in today's roster extends another this way, which is exactly why the gap is invisible — I checked every required_tool in agent/*.yaml and every qualified name the fixtures carry. The pin protects the property, it does not report a live defect. If a roster ever gains create_issue_comment beside create_issue, the zero-false-positive measurement quietly stops being true and nothing says so.

The optional slash is the only thing catching a closing half

</create_issue>                       real=fire   without-</?=clean
text</forgejo__create_issue>          real=fire   without-</?=clean

A reply carrying only the closing tag is a real shape here rather than a hypothetical — #357 records completions truncated mid-reply, and #382 measured that condition at roughly 13% on one prompt.

Both pins are exact

Each mutation now fails exactly one test and nothing else:

drop </? closing support  ->  TestToolNameMarkupCatchesAClosingTagOnItsOwn
drop the \b boundary      ->  TestToolNameMarkupDoesNotFireOnALongerToolName

So they cover what they claim and no more. Nothing in the check changed; this is test-only.

Full suite, go vet, policy-check, gofmt and the full pre-commit set pass.

Refs #301

Mutation testing on `949f40d`, which added `toolNameMarkupFailures` and measured it at **6 of 7 with zero false positives**. I broke three parts of the check one at a time. One was caught; two left the whole suite green. | Mutation | Before | After | | --- | --- | --- | | drop the bare-suffix form | **caught** | caught | | drop `</?` closing support | green | **caught** | | drop the `\b` word boundary | green | **caught** | ## The word boundary is load-bearing for the commit's own claim Without `\b` the check fires on any tag whose name merely *starts with* the declared tool: ``` required_tool = forgejo__create_issue <create_issue> real=fire without-\b=fire correct <create_issues_bulk> real=clean without-\b=fire false positive <create_issue_template lang=fr> real=clean without-\b=fire false positive ``` **No tool in today's roster extends another this way**, which is exactly why the gap is invisible — I checked every `required_tool` in `agent/*.yaml` and every qualified name the fixtures carry. The pin protects the property, it does not report a live defect. If a roster ever gains `create_issue_comment` beside `create_issue`, the zero-false-positive measurement quietly stops being true and nothing says so. ## The optional slash is the only thing catching a closing half ``` </create_issue> real=fire without-</?=clean text</forgejo__create_issue> real=fire without-</?=clean ``` A reply carrying only the closing tag is a real shape here rather than a hypothetical — #357 records completions truncated mid-reply, and #382 measured that condition at roughly 13% on one prompt. ## Both pins are exact Each mutation now fails **exactly one test and nothing else**: ``` drop </? closing support -> TestToolNameMarkupCatchesAClosingTagOnItsOwn drop the \b boundary -> TestToolNameMarkupDoesNotFireOnALongerToolName ``` So they cover what they claim and no more. Nothing in the check changed; this is test-only. Full suite, `go vet`, `policy-check`, `gofmt` and the full pre-commit set pass. Refs #301

coilyco-ops added 1 commit

2026-08-13 11:40:39 +00:00

test(evaluation): pin the two guards in the tool-name check that nothing held

ci / test (pull_request) Successful in 32s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 17s

Details

f36c088160

Mutation testing on 949f40d. Three parts of toolNameMarkupFailures were broken
one at a time; the bare-suffix form was caught, and two were not.

  drop the bare-suffix form   caught
  drop </? closing support    green
  drop the \b word boundary   green

The word boundary is what makes the commit's zero-false-positive measurement
true. Without it the check fires on any tag whose name merely starts with the
declared tool, so <create_issue_comment> is reported while create_issue is what
was declared. No tool in today's roster extends another this way, which is why
the gap is invisible; the pin protects the property rather than reporting a
live defect.

The optional slash is the only reason a reply carrying just the closing half is
caught at all, and a truncated reply is a real shape here per issue 357.

Each mutation now fails exactly one test and nothing else, so the pins cover
what they claim and no more.

Refs sirens-echo#301

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 11:40:40 +00:00

coilyco-ops added 1 commit

2026-08-13 11:42:42 +00:00

Merge branch 'main' into qa/pin-the-tool-name-markup-guards

ci / test (pull_request) Successful in 31s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 17s