test(grounding): pin both halves of the action-claim property #244

Merged

coilysiren merged 1 commit from test/grounding-action-claim-corpus into main

2026-08-13 04:08:02 +00:00

coilyco-ops commented

2026-08-13 03:52:46 +00:00

Member

Contributes the regression corpus offered in #243 and #241. Tests only — no behavior change.

Why

ValidateGrounding's action-claim detector is measured only against claims it should catch. The half that decides whether it is safe to widen — replies it must not touch — had no corpus. That is the half that fails members: a grounding error routes straight to failTurn (agent.go:802-815) with no repair pass, so a false positive costs the member their answer entirely.

What this adds

internal/community/groundingcorpus_test.go — one table, both halves, 18 rows.

Rows assert today's behavior (rejectedNow), not the intended behavior (shouldReject). CI therefore reports what ships rather than what is wanted, and stays green. A row whose two columns disagree is a tracked defect, and its failure message names the issue and says exactly what to edit when the fix lands.

Nine rows currently disagree:

Kind	Count	Issue	Example
false positive	7	243	`No issue has been filed for this.`
escapes	2	241	`Filed a correction for review.`

TestGroundingCorpusReportsOpenRows prints them under -v with a count, so the number is visible without reading the table.

The negations are the point

No issue has been filed for this. is rejected today. That sentence is the honest disclaimer #206, #209 and #232 are all asking the agent to produce. A detector that reads it as a claim makes the truthful answer unshippable — strictly worse than the behavior it replaced.

One more thing found while writing this

trackerWasTouched (decision.go:81) grounds a claim on any tool whose name merely contains issue. So forgejo__list_issue — a read — grounds an assertion that something was filed. actionClaimSupported is careful here, mapping filed to __create_issue specifically; the passive path is not. TestGroundingAcceptsAClaimATrackerToolSupports uses forgejo__create_issue so it does not depend on that looseness, but it is worth folding into whatever fixes 243.

Verification

go vet, gofmt, full go test ./..., and pre-commit run --files all clean. docs/sirens-echo-grounding-corpus.md documents how to read and retire a row, since the two-line comment cap keeps that out of the test file.

I did not touch decision.go. The detector is product code and both defects are reproducible, so the fix belongs to Engineering — this is the harness to fix against.

Quail (QA)

Contributes the regression corpus offered in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/243 and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/241. Tests only — no behavior change. ## Why `ValidateGrounding`'s action-claim detector is measured only against claims it should catch. The half that decides whether it is **safe to widen** — replies it must not touch — had no corpus. That is the half that fails members: a grounding error routes straight to `failTurn` (`agent.go:802-815`) with no repair pass, so a false positive costs the member their answer entirely. ## What this adds `internal/community/groundingcorpus_test.go` — one table, both halves, 18 rows. Rows assert **today's** behavior (`rejectedNow`), not the intended behavior (`shouldReject`). CI therefore reports what ships rather than what is wanted, and stays green. A row whose two columns disagree is a tracked defect, and its failure message names the issue and says exactly what to edit when the fix lands. Nine rows currently disagree: | Kind | Count | Issue | Example | | --- | --- | --- | --- | | false positive | 7 | 243 | `No issue has been filed for this.` | | escapes | 2 | 241 | `Filed a correction for review.` | `TestGroundingCorpusReportsOpenRows` prints them under `-v` with a count, so the number is visible without reading the table. ## The negations are the point `No issue has been filed for this.` is rejected today. That sentence is the honest disclaimer https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/206, https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/209 and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/232 are all asking the agent to produce. A detector that reads it as a claim makes the truthful answer unshippable — strictly worse than the behavior it replaced. ## One more thing found while writing this `trackerWasTouched` (`decision.go:81`) grounds a claim on any tool whose name merely **contains** `issue`. So `forgejo__list_issue` — a read — grounds an assertion that something was *filed*. `actionClaimSupported` is careful here, mapping `filed` to `__create_issue` specifically; the passive path is not. `TestGroundingAcceptsAClaimATrackerToolSupports` uses `forgejo__create_issue` so it does not depend on that looseness, but it is worth folding into whatever fixes 243. ## Verification `go vet`, `gofmt`, full `go test ./...`, and `pre-commit run --files` all clean. `docs/sirens-echo-grounding-corpus.md` documents how to read and retire a row, since the two-line comment cap keeps that out of the test file. I did not touch `decision.go`. The detector is product code and both defects are reproducible, so the fix belongs to Engineering — this is the harness to fix against. --- Quail (QA)

coilyco-ops added 1 commit

2026-08-13 03:52:46 +00:00

test(grounding): pin both halves of the action-claim property

ci / test (pull_request) Successful in 28s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 16s

Details

75b8c24b7e

ValidateGrounding is measured only against claims it should catch. The
half that decides whether the detector is safe to widen — replies it must
not touch — had no corpus at all.

Both halves now sit in one table. Rows assert today's behavior, so CI
reports what ships rather than what is wanted, and a row whose columns
disagree is a tracked defect with the issue named in its failure message.

Nine rows currently disagree. Seven are false positives where a correct
reply is rejected, including the negations, which matter most because a
grounding failure routes straight to failTurn with no repair pass. Two
are ungrounded claims that escape in active third-person and subjectless
voice.

Refs: #243
Refs: #241

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 03:52:46 +00:00

coilyco-ops referenced this pull request

2026-08-13 03:53:03 +00:00

passiveActionClaim fires on correct replies, including negations — "No issue has been filed" fails the turn #243

coilyco-ops referenced this pull request

2026-08-13 03:53:06 +00:00

Grounding: active third-person and subjectless action claims still escape passiveActionClaim #241