coilyco-flight-deck/agent-compose

Watch

Coverage check: report unauthored and ungraded cases, with a switch to make it blocking #340

New issue

Open

opened 2026-08-25 17:09:40 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-25 17:09:40 +00:00

Member

Parent

#329

What to build

Close the loop the inversion exists to enable. The board is already derived from the roster, so a roster change silently changes which cases exist. Nothing currently reports that a boundary was added and its four implied cases were never written, or written and never graded.

Build a coverage check that reads the current roster, derives the implied board, and reports:

Cases implied by the roster with no authored prompt.
Authored cases with no graded record.
Graded records whose case no longer exists in the roster, meaning the evidence is now orphaned.

The switch ships in this slice, defaulted to report. #329 decision 8 sets the gate as recorded and visible now, with an explicit expectation of becoming blocking later. Building the mechanism now and flipping a default later is the whole point. Bolting a gate on afterwards is what this avoids.

The reason it stays non-blocking for now is throughput: the grader is a human by hand, the mechanical scorer was deleted rather than tuned, and a hard gate would put every roster edit behind an annotation session. That is a real cost and it is being taken deliberately rather than by default.

Acceptance criteria

A command reports unauthored, ungraded, and orphaned cases for the current roster.
It runs in CI and its output is visible on every change touching roster data.
A switch makes it exit non-zero, and it defaults to report.
Flipping the switch requires a configuration change, not a code change.
Adding a boundary to the roster and running the check reports its newly implied unauthored cases.
Removing a role reports its graded records as orphaned rather than silently dropping them.
docs/evaluation.md documents the check, the switch, and the reason the default is report.

Blocked by

Blocked by #338

## Parent #329 ## What to build Close the loop the inversion exists to enable. The board is already derived from the roster, so a roster change silently changes which cases exist. Nothing currently reports that a boundary was added and its four implied cases were never written, or written and never graded. Build a coverage check that reads the current roster, derives the implied board, and reports: * Cases implied by the roster with no authored prompt. * Authored cases with no graded record. * Graded records whose case no longer exists in the roster, meaning the evidence is now orphaned. **The switch ships in this slice, defaulted to report.** #329 decision 8 sets the gate as recorded and visible now, with an explicit expectation of becoming blocking later. Building the mechanism now and flipping a default later is the whole point. Bolting a gate on afterwards is what this avoids. The reason it stays non-blocking for now is throughput: the grader is a human by hand, the mechanical scorer was deleted rather than tuned, and a hard gate would put every roster edit behind an annotation session. That is a real cost and it is being taken deliberately rather than by default. ## Acceptance criteria - [ ] A command reports unauthored, ungraded, and orphaned cases for the current roster. - [ ] It runs in CI and its output is visible on every change touching roster data. - [ ] A switch makes it exit non-zero, and it defaults to report. - [ ] Flipping the switch requires a configuration change, not a code change. - [ ] Adding a boundary to the roster and running the check reports its newly implied unauthored cases. - [ ] Removing a role reports its graded records as orphaned rather than silently dropping them. - [ ] `docs/evaluation.md` documents the check, the switch, and the reason the default is report. ## Blocked by - Blocked by #338

coilyco-ops referenced this issue

2026-08-25 17:10:24 +00:00

Invert the stack: data, compositor, bundle, then Go. Drop KDL, move composition out of the engine, mount the roster #329

coilyco-ops added the

priority

autonomy

headless

labels

2026-08-25 17:13:52 +00:00

~~coilyco-ops referenced this issue 2026-08-25 17:14:23 +00:00~~

Make the label taxonomy declarative, then rename role/* onto the current roster #342

coilyco-ops referenced this issue

2026-08-25 17:14:40 +00:00

Invert the stack: data, compositor, bundle, then Go. Drop KDL, move composition out of the engine, mount the roster #329

coilyco-ops commented

2026-08-26 01:45:30 +00:00

Author

Member

A live instance of the orphan case this issue exists to catch, found 2026-08-25

Filed by Saiya (tpm seat). Read from the working tree at 05237fa on main, clean, not from issue text.

This issue anticipates orphaned records in the abstract:

Removing a role reports its graded records as orphaned rather than silently dropping them.

That has already happened, and nothing reported it.

The evidence

The only graded eval record in the repository is evaluations/pilot/ops-board-2026-08-12-regraded, two files, last written 2026-08-12. It grades a roster that no longer exists:

dataset.yaml carries role: ops, and the reflow renamed ops to sysadmin.
annotations.yaml carries case ids ops-fit-director and ops-fit-engineer. Both director and engineer are gone, to tpm and platform.
Both files reference the boundary as suggest-human-comms. It is now boundary-suggest-external-comms.

12c7198 feat(person)!: reflow the Core Roster to seven bonded seats landed 2026-08-22, ten days after that grading session. Every identifier above was replaced by it.

evaluations/reflow-v3/ contains exactly one file, attributes.yaml, and no graded records. So the seven-seat board is declared and ungraded.

What that means

There is currently no graded evidence against the live roster. The graded evidence that exists measures seats that were renamed or merged out of existence, and de934cc feat(evaluations): declare the seven-seat boundary board plus 78d6dbd feat(evalkit): derive the scoped boundary pair the roster already exports moved the board underneath it.

This is not an argument against the reflow. It is the exact failure mode this issue was written to surface, occurring while the mechanism that would surface it is blocked behind #338. Worth recording as the motivating instance rather than a hypothetical, since a real orphan is a better acceptance test than a synthetic one.

Consequence outside this repository

coilysiren/inbox#394 and coilysiren/inbox#338 both rest on a graded result.

The January abstract is unaffected. It commits to past-tense findings rather than inventory, and the freeze audit on inbox#394 explicitly excluded role names, counts, and case counts. A rename cannot falsify "a gate that never fails is certifying rather than measuring." This is the scenario that discipline was bought for, and it held.
The September hands-on session is the exposure, and it is smaller than it looks. coilysiren/inbox#434 needs three or four hand-picked case pairs rather than a regraded board, and the existing annotations already contain a usable disagreement case. No regrade blocks that work.

Suggested acceptance addition

Running the check against the tree as of 05237fa reports the ops-board-2026-08-12-regraded records as orphaned, and reports the reflow-v3 board as unauthored or ungraded.

A known-bad input that already exists in the repository, in the spirit of the broken-pack discriminator recorded on #240.

## A live instance of the orphan case this issue exists to catch, found 2026-08-25 Filed by Saiya (tpm seat). Read from the working tree at `05237fa` on `main`, clean, not from issue text. This issue anticipates orphaned records in the abstract: > Removing a role reports its graded records as orphaned rather than silently dropping them. That has already happened, and nothing reported it. ### The evidence **The only graded eval record in the repository is `evaluations/pilot/ops-board-2026-08-12-regraded`**, two files, last written 2026-08-12. It grades a roster that no longer exists: * `dataset.yaml` carries `role: ops`, and the reflow renamed `ops` to `sysadmin`. * `annotations.yaml` carries case ids `ops-fit-director` and `ops-fit-engineer`. Both `director` and `engineer` are gone, to `tpm` and `platform`. * Both files reference the boundary as `suggest-human-comms`. It is now `boundary-suggest-external-comms`. `12c7198 feat(person)!: reflow the Core Roster to seven bonded seats` landed 2026-08-22, ten days after that grading session. Every identifier above was replaced by it. **`evaluations/reflow-v3/` contains exactly one file, `attributes.yaml`, and no graded records.** So the seven-seat board is declared and ungraded. ### What that means There is currently **no graded evidence against the live roster**. The graded evidence that exists measures seats that were renamed or merged out of existence, and `de934cc feat(evaluations): declare the seven-seat boundary board` plus `78d6dbd feat(evalkit): derive the scoped boundary pair the roster already exports` moved the board underneath it. This is not an argument against the reflow. It is the exact failure mode this issue was written to surface, occurring while the mechanism that would surface it is blocked behind #338. Worth recording as the motivating instance rather than a hypothetical, since a real orphan is a better acceptance test than a synthetic one. ### Consequence outside this repository `coilysiren/inbox#394` and `coilysiren/inbox#338` both rest on a graded result. * **The January abstract is unaffected.** It commits to past-tense findings rather than inventory, and the freeze audit on `inbox#394` explicitly excluded role names, counts, and case counts. A rename cannot falsify "a gate that never fails is certifying rather than measuring." This is the scenario that discipline was bought for, and it held. * **The September hands-on session is the exposure**, and it is smaller than it looks. `coilysiren/inbox#434` needs three or four hand-picked case pairs rather than a regraded board, and the existing annotations already contain a usable disagreement case. No regrade blocks that work. ### Suggested acceptance addition - [ ] Running the check against the tree as of `05237fa` reports the `ops-board-2026-08-12-regraded` records as orphaned, and reports the `reflow-v3` board as unauthored or ungraded. A known-bad input that already exists in the repository, in the spirit of the broken-pack discriminator recorded on #240.

coilyco-ops referenced this issue

2026-08-26 02:45:37 +00:00

Eval prep: the measured gap on the seven-seat board, and every eval, prose, and content ticket folded into one ordering #357

coilyco-ops referenced this issue

2026-08-26 03:10:46 +00:00

Author and grade the seven-seat board, 91 cases against 13 authored prompts #318

coilyco-ops referenced this issue

2026-08-27 03:25:25 +00:00

Audience-as-grader segment: the hands-on reopen created it and nothing tracks it #367