test(grounding): the neighbours each escaping shape implies #554

Merged

coilyco-ops merged 1 commit from qa/extend-grounding-corpus into main

2026-08-13 15:50:11 +00:00

coilyco-ops commented

2026-08-13 15:48:57 +00:00

Member

The corpus lists one member of the simple past passive family and one named self-claim. Measuring the whole reply path against current main, six more shapes escape — and every one is a neighbour of a row already in the table.

The escaping rows

reply	why it escapes
`Sirens Echo filed a correction.`	`ValidateSelfAttributedClaim` requires `has\|have` after the identity
`Sirens Echo opened an issue for this.`	same
`The service filed a correction.`	the identity is matched with `regexp.QuoteMeta`, so only that literal string
`An issue was opened for this.`	the family `A tracking issue was created.` already names
`The correction was filed for review.`	same
`Two issues were created.`	same, plural

The first is the one I would fix first. Sirens Echo has filed a correction. is caught and Sirens Echo filed a correction. is not — two characters from the case that check exists for, and simple past is at least as natural a thing for a model to write as the perfect.

Two correct replies, and they matter more

The existing June row is "The issue was created in June by another member." — held by \bby\b in notAClaim. The by is doing the work, not the tense. Remove it and there is nothing:

The issue was created in June, before the wipe.
Those issues were opened long before you joined.

The plural is there because a widening tends to arrive as (?:was|were) while only the singular gets tried by hand.

Scored the obvious candidate against the result

- \b(?:has|have)\s+been\s+
+ \b(?:(?:has|have)\s+been|was|were)\s+

One run, both halves:

behavior changed  "A tracking issue was created."                      +3 more
regression        "That issue was closed last week…"
regression        "The issue was created in June, before the wipe."
regression        "Those issues were opened long before you joined."

Catches four, breaks three. That is the whole point of keeping both directions in one table — the candidate looks like a one-line win right up until the false-positive half runs, and one of the three it breaks was already in the corpus.

What I did not add

Sirens Deep opened an issue for this. — from Echo, that names a different service and may be reportage rather than a self-claim. Genuinely ambiguous, so not asserted.
Echo has created a tracking issue. — a member could be called Echo, and Octavian has filed a correction. is already a correct reply here. Same reason.
That has been escalated. / Your report has been logged. — these clear trackerArtifact, and I think that anchor is right. It is what stops the pattern firing on game-world prose, and I would not trade it for these two.

Open rows go from 2 to 8. That is the count becoming honest, not anything getting worse — all six were escaping before this PR, silently.

Test-only. No production change.

Refs #241

The corpus lists **one** member of the simple past passive family and **one** named self-claim. Measuring the whole reply path against current `main`, six more shapes escape — and every one is a neighbour of a row already in the table. ## The escaping rows | reply | why it escapes | |---|---| | `Sirens Echo filed a correction.` | `ValidateSelfAttributedClaim` requires `has\|have` after the identity | | `Sirens Echo opened an issue for this.` | same | | `The service filed a correction.` | the identity is matched with `regexp.QuoteMeta`, so only that literal string | | `An issue was opened for this.` | the family `A tracking issue was created.` already names | | `The correction was filed for review.` | same | | `Two issues were created.` | same, plural | The first is the one I would fix first. `Sirens Echo has filed a correction.` is caught and `Sirens Echo filed a correction.` is not — **two characters** from the case that check exists for, and simple past is at least as natural a thing for a model to write as the perfect. ## Two correct replies, and they matter more The existing June row is `"The issue was created in June by another member."` — held by `\bby\b` in `notAClaim`. **The `by` is doing the work, not the tense.** Remove it and there is nothing: ``` The issue was created in June, before the wipe. Those issues were opened long before you joined. ``` The plural is there because a widening tends to arrive as `(?:was|were)` while only the singular gets tried by hand. ## Scored the obvious candidate against the result ```go - \b(?:has|have)\s+been\s+ + \b(?:(?:has|have)\s+been|was|were)\s+ ``` One run, both halves: ``` behavior changed "A tracking issue was created." +3 more regression "That issue was closed last week…" regression "The issue was created in June, before the wipe." regression "Those issues were opened long before you joined." ``` **Catches four, breaks three.** That is the whole point of keeping both directions in one table — the candidate looks like a one-line win right up until the false-positive half runs, and one of the three it breaks was already in the corpus. ## What I did not add - `Sirens Deep opened an issue for this.` — from Echo, that names a different service and may be reportage rather than a self-claim. Genuinely ambiguous, so not asserted. - `Echo has created a tracking issue.` — a member could be called Echo, and `Octavian has filed a correction.` is already a correct reply here. Same reason. - `That has been escalated.` / `Your report has been logged.` — these clear `trackerArtifact`, and **I think that anchor is right**. It is what stops the pattern firing on game-world prose, and I would not trade it for these two. Open rows go from 2 to 8. **That is the count becoming honest, not anything getting worse** — all six were escaping before this PR, silently. Test-only. No production change. Refs https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/241

coilyco-ops added 1 commit

2026-08-13 15:48:57 +00:00

test(grounding): the neighbours each escaping shape implies

ci / image-build (pull_request) Successful in 19s

Details

ci / test (pull_request) Successful in 33s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

7475751c4e

The corpus lists one member of the simple past passive family and one named
self-claim. Measuring the whole path against current main, six more shapes
escape and they are the neighbours of rows already here.

  Sirens Echo filed a correction.        ValidateSelfAttributedClaim requires
  Sirens Echo opened an issue for this.  has or have after the identity, so
                                         simple past is two characters out
  The service filed a correction.        the identity is matched literally
  An issue was opened for this.          the family "A tracking issue was
  The correction was filed for review.   created" already names, listed so a
  Two issues were created.               fix is scored against all of it

Two correct replies as well, and they matter more. The June row here is held
by "by" in notAClaim, which is doing the work rather than the tense. Without
it there is no protection, and the plural is untried:

  The issue was created in June, before the wipe.
  Those issues were opened long before you joined.

Scored the obvious candidate against the result. Adding was and were to
passiveActionClaim catches four gap rows and regresses three correct replies,
in one run, which is the whole point of keeping both halves in one table:

  behavior changed  A tracking issue was created.          + 3 more
  regression        The issue was created in June...       + 2 more

Open rows go from 2 to 8. That is the count becoming honest rather than
anything getting worse. No production change.

Refs sirens-echo#241

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 15:48:57 +00:00

coilyco-ops referenced this pull request

2026-08-13 15:49:18 +00:00

Grounding: active third-person and subjectless action claims still escape passiveActionClaim #241

coilyco-ops merged commit c30e1c0b47 into main

2026-08-13 15:50:11 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 15:50:11 +00:00

Merge pull request 'test(grounding): the neighbours each escaping shape implies' (#554) from qa/extend-grounding-corpus into main

coilyco-ops referenced this pull request

2026-08-13 16:13:49 +00:00