test(capability): guard every lane's copy, not only Echo's #261

Merged

coilysiren merged 1 commit from test/capability-docs-all into main

2026-08-13 05:07:18 +00:00

coilyco-ops commented

2026-08-13 05:05:55 +00:00

Member

Tests only. Follows 41ad239.

The gap it closes

41ad239 gave Deep its own capability.md under coilyco-general, and that is the right call — Deep's limits genuinely are not Echo's, and one shared file would have had to lie about one lane to be true about the other.

It also created exactly the drift this file exists to stop. All four number assertions bound sirens-echo-knowledge by path:

const capabilityDocPath = "../../.agents/skills/sirens-echo-knowledge/references/capability.md"

So Deep's copy shipped unguarded. Change maxToolRounds and Echo's doc fails while Deep's keeps confidently telling its model the old ceiling — which is then repeated to a member as fact. Both copies currently state six tool rounds, 1800 characters, and twelve messages, so nothing is wrong today; nothing was holding it that way.

I flagged this risk on #247 — "copying is the one I would avoid, two copies drift" — and then shipped a guard that only watched one of them. Fixing my own gap.

What changes

Every assertion now runs against every capability.md under .agents/skills, and failures name the lane:

coilyco-general does not say "six tool rounds"; maxToolRounds is 6 and the doc must match

TestCapabilityDocReachesEveryAgent needed no change — it already globbed the agent definitions, and 41ad239 correctly emptied its without map.

One phrase had to shorten

Echo says "twelve recent channel messages"; Deep says "twelve recent messages". The check now stops at "twelve recent" — still a phrase rather than a bare number word, which is what stopped "ten" matching inside "softening" when I first wrote this.

That divergence appeared within one commit of the second copy existing, which is a small illustration of why the guard needs to cover both.

Verified by mutation, against Deep's copy specifically

Deep's is the copy that was unguarded, so that is where the mutations went:

Mutation	Result
Deep's doc → `five tool rounds`	fails, naming `coilyco-general`
Deep's doc → `1500 characters`	fails, naming `coilyco-general`
`max_context_messages` 12 → 10 in code	fails twice, once per doc

Every one of these passed before this change.

go vet, gofmt, full go test ./..., and pre-commit run --files clean.

Quail (QA)

Tests only. Follows `41ad239`. ## The gap it closes `41ad239` gave Deep its own `capability.md` under `coilyco-general`, and that is the right call — Deep's limits genuinely are not Echo's, and one shared file would have had to lie about one lane to be true about the other. It also created **exactly the drift this file exists to stop.** All four number assertions bound `sirens-echo-knowledge` by path: ```go const capabilityDocPath = "../../.agents/skills/sirens-echo-knowledge/references/capability.md" ``` So Deep's copy shipped unguarded. Change `maxToolRounds` and Echo's doc fails while Deep's keeps confidently telling its model the old ceiling — which is then repeated to a member as fact. Both copies currently state six tool rounds, 1800 characters, and twelve messages, so nothing is wrong *today*; nothing was holding it that way. I flagged this risk on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/247 — *"copying is the one I would avoid, two copies drift"* — and then shipped a guard that only watched one of them. Fixing my own gap. ## What changes Every assertion now runs against every `capability.md` under `.agents/skills`, and failures name the lane: ``` coilyco-general does not say "six tool rounds"; maxToolRounds is 6 and the doc must match ``` `TestCapabilityDocReachesEveryAgent` needed no change — it already globbed the agent definitions, and `41ad239` correctly emptied its `without` map. ## One phrase had to shorten Echo says *"twelve recent **channel** messages"*; Deep says *"twelve recent messages"*. The check now stops at `"twelve recent"` — still a phrase rather than a bare number word, which is what stopped `"ten"` matching inside `"softening"` when I first wrote this. That divergence appeared within one commit of the second copy existing, which is a small illustration of why the guard needs to cover both. ## Verified by mutation, against Deep's copy specifically Deep's is the copy that was unguarded, so that is where the mutations went: | Mutation | Result | | --- | --- | | Deep's doc → `five tool rounds` | fails, naming `coilyco-general` | | Deep's doc → `1500 characters` | fails, naming `coilyco-general` | | `max_context_messages` 12 → 10 in code | fails **twice**, once per doc | Every one of these passed before this change. `go vet`, `gofmt`, full `go test ./...`, and `pre-commit run --files` clean. --- Quail (QA)

coilyco-ops added 1 commit

2026-08-13 05:05:55 +00:00

test(capability): guard every lane's copy, not only Echo's

ci / test (pull_request) Successful in 27s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 16s

Details

3ae5e3e677

41ad239 gave Deep its own capability.md under coilyco-general, which is
the right call — its limits genuinely are not Echo's, and a shared file
would have had to lie about one lane to be true about the other.

It also created the exact drift this file exists to stop. All four number
assertions bound the sirens-echo-knowledge path by name, so Deep's copy
shipped unguarded: change maxToolRounds and Echo's doc fails while Deep's
keeps confidently telling its model the old ceiling.

Every assertion now runs against every capability.md under
.agents/skills, and failures name the lane.

The context-window phrase had to shorten. Echo says "twelve recent
channel messages" and Deep says "twelve recent messages", so the check
stops at "twelve recent" — still a phrase rather than a bare number word,
which is what stopped "ten" matching inside "softening".

Verified by mutation against Deep's copy specifically, since that is the
one that was unguarded: five tool rounds fails naming coilyco-general,
1500 characters fails naming coilyco-general, and a context-window change
in code fails both docs rather than one.

Refs: #247

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 05:05:55 +00:00