Reconcile every harness doc and skill against the deployed state, and make capability drift a check rather than a discovery #1011

Open
opened 2026-08-19 01:47:13 +00:00 by coilyco-ops · 1 comment
Member

Filed after the fourth instance of the same defect in 24 hours. Each was found by accident, by a different reader, after the prose had already been wrong in production for hours. The pattern is worth fixing once rather than four more times.

The defect class

A capability moves in coilyco-bridge/deploy. Prose in this repository keeps describing the old one. Nothing notices, because nothing in CI compares a sentence to a deployed value.

It is worse than an ordinary stale doc, because most of this prose is inline: always, so a false capability claim is in the model's prompt on every single turn, and the model has no way to tell it from a true one.

The four instances

  1. moxn-knowledge-base.md said "Four tools, all reads." coilyco-bridge/deploy#683 merged 2026-08-18T21:05Z and took the wrap to nine tools with readOnly: false, including delete under a bundled action argument. The doctrine told the lane it could not write for roughly four hours while it could write and destroy. Fixed in 3e177c3, along with the dowel-moxn-read-only boundary that would have graded the lane as failing for doing its job.
  2. site-work.md named quire's verbs. quire was deleted; the file described set_page on a service that no longer existed. #977, fixed in a4c4d76.
  3. capability.md states 6 tool rounds and 9 model calls. The Dowel definition sets tool_rounds: 12 and budget_raises: 2. #978, still open, and it is the one that will be wrong on camera if the lane is asked about its own limits.
  4. The same number is now in four places. docs/sirens-echo-rate.md:31, docs/sirens-echo-tools.md:20, .agents/skills/coilyco-general/references/capability.md:31, and .agents/skills/sirens-echo-knowledge/references/capability.md:34 all hard-code six rounds independently. Four copies of one tunable that internal/community/config.go owns.

A fifth, of a different shape: two sources for one fact

#999 added standalone moxn and temporal skill roots so their own people can PR them, which is a good reason. But .agents/skills/moxn/SKILL.md now carries the product description, the name-collision rule, and the fetch-rather-than-recall rule, and .agents/skills/sirens-dowel/references/moxn-knowledge-base.md carries the same three things.

Only the second is in Dowel's local_skill_roots, so today they agree by coincidence and nothing makes them keep agreeing. The same split exists for temporal. Decide which is canonical and make the other point at it, rather than leaving two files that drift apart quietly.

Why the existing check did not catch any of this

just boundaries-check did catch instance 1, and it is the only thing that did. It failed on the missing anchor the moment the section was renamed, which is exactly what it is for.

Its reach is the limit. It verifies that a boundary's cited anchor still exists in a skill file. It cannot verify that the claim is true, and it only covers text a boundary happens to cite. Instances 2, 3, and 4 sit outside any boundary citation.

What would actually close it

Roughly in order of value per effort.

  • Derive the numbers instead of restating them. ResolvedBudget already enumerates its fields for logging. The prompt could carry the lane's real ceilings, which ends instances 3 and 4 permanently and cannot drift again. This is the fix worth building.
  • A drift check across the repository boundary. The authority for the tool surface is a values file in another repo. A check that reads sirens-dowel-moxn-mcp-values.yaml and fails when a skill file names a tool count or a read-only claim that disagrees would have caught instance 1 at the moment #683 merged. coilyco-bridge/deploy#693 is the same gap in the other direction and should be solved with it rather than separately.
  • Ban the restated tunable. The estate rule already says tests must not re-encode configuration. The same argument applies to prose, and four independent copies of "6 tool rounds" is the proof.
  • An inventory pass now, by hand, because none of the above lands before tomorrow. Every inline: always file, read against the deployed configmaps and values files, with each capability claim either verified or deleted.

Scope note

The inventory pass is the only part that should happen before the 2026-08-19 stream, and it is documentation-only, so it sits inside the freeze under the 2026-08-18 amendment on #929. The derivation work and the cross-repo check are after.

  • coilyco-bridge/deploy#683 - the widening that exposed this.
  • #977, #978 - instances 2 and 3, #978 still open.
  • coilyco-bridge/deploy#693 - the same class, catalogue direction.
  • #971 and #1004 - the skill contract, which decides what inline: always means and therefore how much a wrong sentence costs.
**Filed after the fourth instance of the same defect in 24 hours.** Each was found by accident, by a different reader, after the prose had already been wrong in production for hours. The pattern is worth fixing once rather than four more times. ## The defect class A capability moves in `coilyco-bridge/deploy`. Prose in this repository keeps describing the old one. Nothing notices, because **nothing in CI compares a sentence to a deployed value.** It is worse than an ordinary stale doc, because most of this prose is `inline: always`, so a false capability claim is in the model's prompt on every single turn, and the model has no way to tell it from a true one. ## The four instances 1. **`moxn-knowledge-base.md` said "Four tools, all reads."** `coilyco-bridge/deploy#683` merged 2026-08-18T21:05Z and took the wrap to nine tools with `readOnly: false`, including delete under a bundled action argument. The doctrine told the lane it could not write for roughly four hours while it could write and destroy. Fixed in `3e177c3`, along with the `dowel-moxn-read-only` boundary that would have graded the lane as failing for doing its job. 2. **`site-work.md` named quire's verbs.** quire was deleted; the file described `set_page` on a service that no longer existed. #977, fixed in `a4c4d76`. 3. **`capability.md` states 6 tool rounds and 9 model calls.** The Dowel definition sets `tool_rounds: 12` and `budget_raises: 2`. #978, still open, and it is the one that will be wrong on camera if the lane is asked about its own limits. 4. **The same number is now in four places.** `docs/sirens-echo-rate.md:31`, `docs/sirens-echo-tools.md:20`, `.agents/skills/coilyco-general/references/capability.md:31`, and `.agents/skills/sirens-echo-knowledge/references/capability.md:34` all hard-code six rounds independently. Four copies of one tunable that `internal/community/config.go` owns. ## A fifth, of a different shape: two sources for one fact `#999` added standalone `moxn` and `temporal` skill roots so their own people can PR them, which is a good reason. But `.agents/skills/moxn/SKILL.md` now carries the product description, the name-collision rule, and the fetch-rather-than-recall rule, and `.agents/skills/sirens-dowel/references/moxn-knowledge-base.md` carries the same three things. Only the second is in Dowel's `local_skill_roots`, so today they agree by coincidence and nothing makes them keep agreeing. The same split exists for `temporal`. **Decide which is canonical and make the other point at it**, rather than leaving two files that drift apart quietly. ## Why the existing check did not catch any of this `just boundaries-check` **did** catch instance 1, and it is the only thing that did. It failed on the missing anchor the moment the section was renamed, which is exactly what it is for. Its reach is the limit. It verifies that a boundary's cited anchor still exists in a skill file. It cannot verify that the **claim** is true, and it only covers text a boundary happens to cite. Instances 2, 3, and 4 sit outside any boundary citation. ## What would actually close it Roughly in order of value per effort. * **Derive the numbers instead of restating them.** `ResolvedBudget` already enumerates its fields for logging. The prompt could carry the lane's real ceilings, which ends instances 3 and 4 permanently and cannot drift again. This is the fix worth building. * **A drift check across the repository boundary.** The authority for the tool surface is a values file in another repo. A check that reads `sirens-dowel-moxn-mcp-values.yaml` and fails when a skill file names a tool count or a read-only claim that disagrees would have caught instance 1 at the moment #683 merged. `coilyco-bridge/deploy#693` is the same gap in the other direction and should be solved with it rather than separately. * **Ban the restated tunable.** The estate rule already says tests must not re-encode configuration. The same argument applies to prose, and four independent copies of "6 tool rounds" is the proof. * **An inventory pass now**, by hand, because none of the above lands before tomorrow. Every `inline: always` file, read against the deployed configmaps and values files, with each capability claim either verified or deleted. ## Scope note The inventory pass is the only part that should happen before the 2026-08-19 stream, and it is documentation-only, so it sits inside the freeze under the 2026-08-18 amendment on #929. The derivation work and the cross-repo check are after. ## Related * `coilyco-bridge/deploy#683` - the widening that exposed this. * #977, #978 - instances 2 and 3, #978 still open. * `coilyco-bridge/deploy#693` - the same class, catalogue direction. * #971 and #1004 - the skill contract, which decides what `inline: always` means and therefore how much a wrong sentence costs.
Author
Member

A fifth instance for the list, and it is the interesting kind: currently correct

Found while picking up #953 and #978 (both already fixed by 1f6e246, now closed). This one is not wrong today, which is exactly why it belongs here rather than in its own issue.

.agents/skills/sirens-echo-knowledge/references/capability.md, lines 34 to 36, inline: always, still reads verbatim:

At most 6 tool rounds. After the last one, the answer uses what they returned. A budget of 9 model calls also covers repairs and raises, so a request can run out of steps sooner.

1f6e246 removed those integers from coilyco-general's copy and did not touch this second copy.

Mapping roots to budgets across the three deployed definitions:

  • sirens-echo - loads sirens-echo-knowledge - no model_budget, so it takes the defaults 6 and 9. Correct.
  • sirens-deep - loads coilyco-general - tool_rounds: 12 - correct since 1f6e246
  • sirens-dowel - loads coilyco-general - tool_rounds: 12 - correct since 1f6e246

So there is nothing to ship, and I did not edit it. Changing prose that currently matches its lane, during a freeze, to guard a hypothetical is the wrong trade.

Why it is still a data point for this issue

The four instances you catalogued were all discovered wrong. This one is correct by coincidence: the file hard-codes a per-deployment value and happens to name the default that the one lane reading it happens to take. Nothing holds that together. The day someone gives Echo a model_budget, this file is wrong in production on every turn with no test to notice, and it will present as a fifth accidental discovery.

That is worth something to the check this issue proposes. A check that only compares deployed values to prose would pass on this file today, so passing does not mean the coupling is sound. The stronger check is structural: an inline: always doc should not contain a literal that a definition can set, whether or not it currently agrees. That form catches this file now rather than after Echo's config moves.

Timely rather than theoretical: Echo's config is being actively changed today, with coilyco-bridge/deploy PR #734 enabling coalescing on that lane.

Two candidate shapes, both post-freeze:

  1. Ban the literal. A test asserting no inline: always reference states an integer that Definition or ModelBudget can override. Cheap, no rendering cost, catches the latent case.
  2. Shape 1 from #978 - render resolved ceilings at prompt build. Strictly better and more expensive, and it makes both copies of this paragraph unnecessary rather than merely checked.

Not proposing which. Flagging that the second capability.md exists at all is probably its own finding: two inline: always files describing the same bounds is the duplication that let one get fixed and the other not.

## A fifth instance for the list, and it is the interesting kind: currently correct Found while picking up #953 and #978 (both already fixed by `1f6e246`, now closed). This one is not wrong today, which is exactly why it belongs here rather than in its own issue. **`.agents/skills/sirens-echo-knowledge/references/capability.md`, lines 34 to 36**, `inline: always`, still reads verbatim: > At most 6 tool rounds. After the last one, the answer uses what they returned. A budget of 9 model calls also covers repairs and raises, so a request can run out of steps sooner. `1f6e246` removed those integers from `coilyco-general`'s copy and did not touch this second copy. Mapping roots to budgets across the three deployed definitions: * `sirens-echo` - loads `sirens-echo-knowledge` - **no `model_budget`**, so it takes the defaults 6 and 9. **Correct.** * `sirens-deep` - loads `coilyco-general` - `tool_rounds: 12` - correct since `1f6e246` * `sirens-dowel` - loads `coilyco-general` - `tool_rounds: 12` - correct since `1f6e246` So there is nothing to ship, and I did not edit it. Changing prose that currently matches its lane, during a freeze, to guard a hypothetical is the wrong trade. ## Why it is still a data point for this issue The four instances you catalogued were all **discovered wrong**. This one is **correct by coincidence**: the file hard-codes a per-deployment value and happens to name the default that the one lane reading it happens to take. Nothing holds that together. The day someone gives Echo a `model_budget`, this file is wrong in production on every turn with no test to notice, and it will present as a fifth accidental discovery. That is worth something to the check this issue proposes. **A check that only compares deployed values to prose would pass on this file today**, so passing does not mean the coupling is sound. The stronger check is structural: an `inline: always` doc should not contain a literal that a definition can set, whether or not it currently agrees. That form catches this file now rather than after Echo's config moves. Timely rather than theoretical: Echo's config is being actively changed today, with `coilyco-bridge/deploy` PR #734 enabling coalescing on that lane. Two candidate shapes, both post-freeze: 1. **Ban the literal.** A test asserting no `inline: always` reference states an integer that `Definition` or `ModelBudget` can override. Cheap, no rendering cost, catches the latent case. 2. **Shape 1 from #978** - render resolved ceilings at prompt build. Strictly better and more expensive, and it makes both copies of this paragraph unnecessary rather than merely checked. Not proposing which. Flagging that the second `capability.md` exists at all is probably its own finding: two `inline: always` files describing the same bounds is the duplication that let one get fixed and the other not.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-gaming/sirens-echo#1011
No description provided.