test(capability): tie capability.md's numbers to the code #248

Merged

coilysiren merged 1 commit from test/capability-doc-matches-code into main

2026-08-13 04:14:23 +00:00

coilyco-ops commented

2026-08-13 04:09:00 +00:00

Member

Tests only, no behavior change. Follows the capability work in dd76224 and f34d0b4.

Why

references/capability.md is loaded into the prompt and tells the model what the service can do. The model then repeats it to members. Every number in it — six tool rounds, the seventh failing, 1800 characters, twelve messages — is prose mirroring a value defined somewhere else.

Nothing binds them. Change maxToolRounds, and the doc keeps confidently telling the model the old ceiling, which the model states to a member as fact. That is precisely the failure class the file was written to prevent, arriving through the file itself.

What this adds

internal/community/capabilitydoc_test.go — four tests binding the doc to its sources:

Test	Binds	Source of truth
`…RealToolRoundCeiling`	"six tool rounds", "on the seventh"	`maxToolRounds`
`…RealReplyCap`	"1800"	`ParseReply` behavior at 1800 and 1801 runes
`…RealContextWindow`	"twelve recent channel messages"	every `agent/*.yaml`
`…DoesNotDenyWorkTheHarnessCanRun`	"Nothing runs between requests"	the `JobKinds` set

The reply cap is asserted through ParseReply rather than a constant, because the limit is a literal in that function and behavior is what the member actually meets.

The last one is the interesting guard: "Nothing runs between requests" is true today only because JobKinds contains just echo and ward-exec, and neither deployment configures a job store. Add a third kind that outlives a reply and the doc becomes a false denial — that test fails when someone does.

Verified by mutation

Every assertion was checked by breaking the thing it guards:

Mutation	Caught
`maxToolRounds` 6 → 5	`does not say "five tool rounds"`
`maxToolRounds` 6 → 7	plus `does not name "on the eighth"`
reply cap 1800 → 1500	`a reply of exactly 1800 runes was rejected`
`max_context_messages` 12 → 10	`does not say "ten recent channel messages"`

One of these caught nothing on the first attempt. strings.Contains(doc, "ten") passed against unmodified prose, because "ten" is a substring of "softening". Number words are now matched as phrases, and the tool-round ordinal with them. Worth stating because it is the same defect in miniature that this PR guards against — an assertion that reads as a check and verifies nothing.

Verification

go vet, gofmt, full go test ./..., and pre-commit run --files all clean.

I did not edit capability.md. I checked every claim in it against source and found no inaccuracy — the six-round ceiling, the fail-fast sequential tool loop, the 1800 cap, the twelve-message window, and the attribution of Eco watchers to the Eco application are all correct as written. This PR keeps them that way.

Quail (QA)

Tests only, no behavior change. Follows the capability work in `dd76224` and `f34d0b4`. ## Why `references/capability.md` is loaded into the prompt and tells the model what the service can do. The model then repeats it to members. Every number in it — six tool rounds, the seventh failing, 1800 characters, twelve messages — is **prose mirroring a value defined somewhere else**. Nothing binds them. Change `maxToolRounds`, and the doc keeps confidently telling the model the old ceiling, which the model states to a member as fact. That is precisely the failure class the file was written to prevent, arriving through the file itself. ## What this adds `internal/community/capabilitydoc_test.go` — four tests binding the doc to its sources: | Test | Binds | Source of truth | | --- | --- | --- | | `…RealToolRoundCeiling` | "six tool rounds", "on the seventh" | `maxToolRounds` | | `…RealReplyCap` | "1800" | `ParseReply` behavior at 1800 and 1801 runes | | `…RealContextWindow` | "twelve recent channel messages" | every `agent/*.yaml` | | `…DoesNotDenyWorkTheHarnessCanRun` | "Nothing runs between requests" | the `JobKinds` set | The reply cap is asserted through `ParseReply` rather than a constant, because the limit is a literal in that function and behavior is what the member actually meets. The last one is the interesting guard: "Nothing runs between requests" is true today only because `JobKinds` contains just `echo` and `ward-exec`, and neither deployment configures a job store. Add a third kind that outlives a reply and the doc becomes a false denial — that test fails when someone does. ## Verified by mutation Every assertion was checked by breaking the thing it guards: | Mutation | Caught | | --- | --- | | `maxToolRounds` 6 → 5 | `does not say "five tool rounds"` | | `maxToolRounds` 6 → 7 | plus `does not name "on the eighth"` | | reply cap 1800 → 1500 | `a reply of exactly 1800 runes was rejected` | | `max_context_messages` 12 → 10 | `does not say "ten recent channel messages"` | **One of these caught nothing on the first attempt.** `strings.Contains(doc, "ten")` passed against unmodified prose, because "ten" is a substring of "softening". Number words are now matched as phrases, and the tool-round ordinal with them. Worth stating because it is the same defect in miniature that this PR guards against — an assertion that reads as a check and verifies nothing. ## Verification `go vet`, `gofmt`, full `go test ./...`, and `pre-commit run --files` all clean. I did not edit `capability.md`. I checked every claim in it against source and found no inaccuracy — the six-round ceiling, the fail-fast sequential tool loop, the 1800 cap, the twelve-message window, and the attribution of Eco watchers to the Eco application are all correct as written. This PR keeps them that way. --- Quail (QA)

coilyco-ops added 1 commit

2026-08-13 04:09:01 +00:00

test(capability): tie capability.md's numbers to the code

ci / test (pull_request) Successful in 29s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 16s

Details

d0e5252c64

capability.md tells the model what the service can do, and the model
repeats it to members. Every number in it is prose mirroring a value
somewhere else, so a change to the code turns the doc into exactly the
kind of false capability claim it was written to prevent.

Four tests bind the doc to its sources: the tool-round ceiling and the
round that fails, the reply cap asserted through ParseReply rather than a
constant, the context window checked against every agent definition, and
the no-background-work claim checked against the JobKinds set.

Verified by mutation. maxToolRounds 6->5 and 6->7, the reply cap
1800->1500, and max_context_messages 12->10 each fail the intended test.

The context-window assertion caught nothing on the first attempt, because
"ten" is a substring of "softening". Number words are matched as phrases
now, and the tool-round ordinal with them.

Refs: #200
Refs: #231

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 04:09:01 +00:00

coilyco-ops referenced this pull request

2026-08-13 04:09:22 +00:00

acknowledge hallucination capability in future evaluations #231

coilyco-ops referenced this pull request

2026-08-13 04:09:33 +00:00

document and encode the limit of echo's ability to perform complex tasks #200