test(capability): assert the limits reach every agent, not just Echo #255

Merged

coilysiren merged 1 commit from test/capability-reach into main

2026-08-13 04:39:54 +00:00

coilyco-ops commented

2026-08-13 04:38:13 +00:00

Member

Tests only. Confirms #247 by construction and guards it.

The finding

capability.md states properties of the harness, not of Echo's persona: six tool rounds, sequential and fail-fast, one reply, 1800 characters, twelve messages, and the fallibility rule. Every lane is subject to all of them.

Only one lane is told:

Lane	`local_skill_roots`	Reaches `references/capability.md`
`sirens-echo`	`sirens-echo-community`, `sirens-echo-knowledge`	yes
`sirens-deep`	`coilyco-general`	no

coilyco-general contains SKILL.md and references/guardfile.md. That is all of it. So the capability work in dd76224 and f34d0b4 — which I verified as accurate in #200 and #231 — reached Echo and not Deep.

That is worth stating plainly: Deep is the lane doing the general-purpose public work, and it is the one with no encoded limits. It confirms 247 without needing to observe another over-claim.

What this adds

TestCapabilityDocReachesEveryAgent walks every agent/*.yaml that declares skill roots and checks whether any declared root holds references/capability.md on disk.

Deep is recorded in a without map so the suite stays green today, and the row fails when Deep gains the reference — with a message naming the issue and saying to delete the entry. Same lifecycle as the grounding corpus, which Engineering has now flipped twice.

Verified in both directions

Mutation	Result
add `sirens-echo-knowledge` to Deep (the fix)	`sirens-deep.yaml now reaches capability.md; issue 247 is fixed, so drop it from the without map`
remove it from Echo (a regression)	`sirens-echo.yaml declares skill roots but none carries references/capability.md, so its model is told none of the harness limits`

Both directions matter. A test that only catches the fix would not notice a lane silently losing the reference, which is how Deep got here.

Not fixing it

Deciding what Deep loads is a product call — the two lanes have deliberately different personas and skill sets, and capability.md currently lives under sirens-echo-knowledge, so the fix is either a move, a copy, or a third shared root. That choice belongs to whoever owns the skill layout, not to me.

go vet, gofmt, full go test ./..., pre-commit run --files all clean.

Quail (QA)

Tests only. Confirms https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/247 by construction and guards it. ## The finding `capability.md` states properties of the **harness**, not of Echo's persona: six tool rounds, sequential and fail-fast, one reply, 1800 characters, twelve messages, and the fallibility rule. Every lane is subject to all of them. Only one lane is told: | Lane | `local_skill_roots` | Reaches `references/capability.md` | | --- | --- | --- | | `sirens-echo` | `sirens-echo-community`, `sirens-echo-knowledge` | **yes** | | `sirens-deep` | `coilyco-general` | **no** | `coilyco-general` contains `SKILL.md` and `references/guardfile.md`. That is all of it. So the capability work in `dd76224` and `f34d0b4` — which I verified as accurate in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/200 and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/231 — reached Echo and not Deep. That is worth stating plainly: **Deep is the lane doing the general-purpose public work**, and it is the one with no encoded limits. It confirms 247 without needing to observe another over-claim. ## What this adds `TestCapabilityDocReachesEveryAgent` walks every `agent/*.yaml` that declares skill roots and checks whether any declared root holds `references/capability.md` on disk. Deep is recorded in a `without` map so the suite stays green today, and the row fails when Deep gains the reference — with a message naming the issue and saying to delete the entry. Same lifecycle as the grounding corpus, which Engineering has now flipped twice. ## Verified in both directions | Mutation | Result | | --- | --- | | add `sirens-echo-knowledge` to Deep (the fix) | `sirens-deep.yaml now reaches capability.md; issue 247 is fixed, so drop it from the without map` | | remove it from Echo (a regression) | `sirens-echo.yaml declares skill roots but none carries references/capability.md, so its model is told none of the harness limits` | Both directions matter. A test that only catches the fix would not notice a lane silently losing the reference, which is how Deep got here. ## Not fixing it Deciding what Deep loads is a product call — the two lanes have deliberately different personas and skill sets, and `capability.md` currently lives under `sirens-echo-knowledge`, so the fix is either a move, a copy, or a third shared root. That choice belongs to whoever owns the skill layout, not to me. `go vet`, `gofmt`, full `go test ./...`, `pre-commit run --files` all clean. --- Quail (QA)

coilyco-ops added 1 commit

2026-08-13 04:38:13 +00:00

test(capability): assert the limits reach every agent, not just Echo

ci / test (pull_request) Successful in 44s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 19s

Details

93aed4af33

The limits capability.md states are properties of the harness — six tool
rounds, one reply, 1800 characters, twelve messages, fallibility. None of
them are Echo's persona, so every lane needs them.

Only one lane has them. Echo loads sirens-echo-knowledge, which carries
references/capability.md. Deep loads coilyco-general alone, which carries
SKILL.md and references/guardfile.md and nothing else, so Deep's model is
told none of the limits it is subject to.

That is the reported defect for Deep's encoded limits, confirmed by
construction rather than by observing an over-claim.

Recorded as characterization: Deep is listed in a without map so the suite
stays green, and the row fails when Deep gains the reference, naming the
issue and saying to drop the entry.

Verified both directions. Adding the knowledge root to Deep fails with
"now reaches capability.md; issue 247 is fixed". Removing it from Echo
fails with "declares skill roots but none carries references/capability.md".

Refs: #247

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 04:38:13 +00:00

coilyco-ops referenced this pull request

2026-08-13 04:38:34 +00:00

Sirens Deep has no encoded capability limits, so the same over-claiming defect can recur there #247