coilyco-flight-deck/agentic-os

Watch

The conformance check for #426, derived from the taxonomy rather than designed up front #1300

New issue

Open

opened 2026-08-27 03:25:21 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-27 03:25:21 +00:00

Owner

Child of #426, carrying its exit condition 2. Exit condition 3 is done: the taxonomy is written as the tooling-boundary-conformance skill in coilyco-flight-deck/agentic-os.

Filed by Angie (platform seat) after working eight of the fourteen children.

What the fourteen actually taught

The epic said to derive the check from the fixes rather than guess at the taxonomy first. Having done that, one detector dominates.

The negative control — supply a value that cannot match, and assert the result differs from the unfiltered one — independently caught three of the fourteen, in systems that share no code:

mcp-beaver#94 - an undeclared searchText was dropped and the unfiltered total returned with status: success
#391's Forgejo finding - a label filter that does not resolve returns the unfiltered issue set
sirens-echo#449 - a view that read 528 of 22,933 rows became "none exists"

Those look like three unrelated bugs and are one: a filter that cannot be honoured produces an answer indistinguishable from a real one. Everything else in the taxonomy is narrower.

That is the finding that makes a shared check worth building. Without it, a conformance check would have been eight loosely related assertions.

What a product would run

The shape the taxonomy implies, not a proposal to build blind:

Enumerate the boundaries the product declares. Each product already declares them somewhere, and the enumeration must derive from that source rather than a hand-kept list, or the check drifts into the thing it is checking.
Assert each declared boundary has a probe. A boundary with no probe is unmeasured, and unmeasured must not read as passing. sirens-echo's RateBrevity already gets this right by reporting Measured: false rather than a pass when a side scored nothing.
Run the negative control wherever the boundary bounds a set. Filters, scopes, allowlists, coverage claims.
Fail loudly. The check's own failure mode must not be the defect class it exists to catch.

Where each product's boundaries already live

Worth stating, because it decides how much of this is shared and how much is per-repo:

sirens-echo - eval/attributes.yaml declares every boundary once, and the evaluation board derives from it. This is the shape the others would be measured against.
umbra - guardfile grants and the spec-derived schema. Declared, but as a capability surface rather than a boundary list.
mcp-beaver - tool input schemas, plus withhold, restrict, reject-empty-argument, and the pins.
agent-compose - role and boundary skills, which already carry an owning side and a deferring side per boundary.

Only one of the four has a single declaration point today.

The open decision

Where the check is authored, and what "declared boundary" means for a product that has no single declaration point.

The authoring vs rollout rule says a validator is authored in agentic-os and rolled out from infrastructure. That fits if the check is one validator over a common declaration format. It does not fit if each product needs its own enumeration, in which case the shared artifact is the taxonomy skill plus a convention, and the checks are four separate per-repo tests.

I have not picked, because the answer changes what gets built and the wrong choice is expensive to undo.

Not in scope

Retrofitting a declaration format onto the three products that lack one. That is the larger half and should be a decision rather than a side effect of building a check.

Blocked on

Nothing mechanical. It needs the placement decision above, and it is genuinely last: five of the fourteen still need a human, so the taxonomy is not final.

Child of #426, carrying its exit condition 2. Exit condition 3 is done: the taxonomy is written as the `tooling-boundary-conformance` skill in `coilyco-flight-deck/agentic-os`. Filed by Angie (platform seat) after working eight of the fourteen children. ## What the fourteen actually taught The epic said to derive the check from the fixes rather than guess at the taxonomy first. Having done that, **one detector dominates**. The **negative control** — supply a value that cannot match, and assert the result differs from the unfiltered one — independently caught three of the fourteen, in systems that share no code: * `mcp-beaver#94` - an undeclared `searchText` was dropped and the unfiltered total returned with `status: success` * `#391`'s Forgejo finding - a label filter that does not resolve returns the **unfiltered** issue set * `sirens-echo#449` - a view that read 528 of 22,933 rows became "none exists" Those look like three unrelated bugs and are one: **a filter that cannot be honoured produces an answer indistinguishable from a real one.** Everything else in the taxonomy is narrower. That is the finding that makes a shared check worth building. Without it, a conformance check would have been eight loosely related assertions. ## What a product would run The shape the taxonomy implies, not a proposal to build blind: 1. **Enumerate the boundaries the product declares.** Each product already declares them somewhere, and the enumeration must derive from that source rather than a hand-kept list, or the check drifts into the thing it is checking. 2. **Assert each declared boundary has a probe.** A boundary with no probe is unmeasured, and unmeasured must not read as passing. `sirens-echo`'s `RateBrevity` already gets this right by reporting `Measured: false` rather than a pass when a side scored nothing. 3. **Run the negative control wherever the boundary bounds a set.** Filters, scopes, allowlists, coverage claims. 4. **Fail loudly.** The check's own failure mode must not be the defect class it exists to catch. ## Where each product's boundaries already live Worth stating, because it decides how much of this is shared and how much is per-repo: * **sirens-echo** - `eval/attributes.yaml` declares every boundary once, and the evaluation board derives from it. This is the shape the others would be measured against. * **umbra** - guardfile grants and the spec-derived schema. Declared, but as a capability surface rather than a boundary list. * **mcp-beaver** - tool input schemas, plus `withhold`, `restrict`, `reject-empty-argument`, and the pins. * **agent-compose** - role and boundary skills, which already carry an owning side and a deferring side per boundary. Only one of the four has a single declaration point today. ## The open decision **Where the check is authored, and what "declared boundary" means for a product that has no single declaration point.** The `authoring vs rollout` rule says a validator is authored in `agentic-os` and rolled out from `infrastructure`. That fits if the check is one validator over a common declaration format. It does not fit if each product needs its own enumeration, in which case the shared artifact is the taxonomy skill plus a convention, and the checks are four separate per-repo tests. I have not picked, because the answer changes what gets built and the wrong choice is expensive to undo. ## Not in scope Retrofitting a declaration format onto the three products that lack one. That is the larger half and should be a decision rather than a side effect of building a check. ## Blocked on Nothing mechanical. It needs the placement decision above, and it is genuinely last: five of the fourteen still need a human, so the taxonomy is not final.

coilyco-ops added the

role/platform

autonomy

headless

labels

2026-08-29 07:01:34 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#1300

No description provided.

Rows
Columns