The eval pipeline is documented one way only, and the grade-to-edit return path has no page #320

New issue

Open

opened 2026-08-22 20:32:18 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-22 20:32:18 +00:00

Member

docs/evaluation.md documents the pipeline end to end up to the taxonomy, and agentic-os/docs/aos-eval.md says plainly that taxonomy's "output is a list of things to fix rather than a score." What you do with that list is written down nowhere. There is no page for the return path.

Holding this open rather than drafting it, because the judgment half is Kai's practice and a doctrine page that anchors a rule nobody runs is worse than an absent one.

What is documented

generator -> inspect eval -> evalkit.filter -> aos-eval annotate -> aos-eval taxonomy, in docs/evaluation.md, with the verbs in the justfile and a runnable command block in agentic-os/docs/aos-eval.md. That half is in good shape.

What exists of the return path

Four fragments, located and verified, none of them doctrine and none in the evaluation page:

docs/role-boundaries.md:101 - changing a boundary body moves the pack digest for every role on either side, "retiring those results until an independently reviewed re-run." This is the only invalidation rule in the repo. It covers boundary bodies and says nothing about role charters, melds, adjacency reasons, or purposes, all of which also feed the derived board.
evaluations/retired-2026-08-06-codex/ - the single retired-* directory, created in b3be43f when the driver-plus-reviewer stack was retired under #262. That retired a whole tier of the method, not a set of results superseded by a prose edit. So the convention people cite for "where superseded records go" has exactly one instance, and it is not an instance of this case.
evaluations/pilot/ops-board-2026-08-12-regraded/ - a second pattern, and a different one: same board re-annotated, kept alongside the original with a suffix rather than retired. Two precedents, two shapes, no stated rule for which applies when.
samples.yaml header - the ops-mls-in note, where a case produced a refusal in 10 of 10 probe runs regardless of bundle, so it "measured its own framing" and the case was rewritten rather than the charter. This is the sharpest thing in the repo on fix-the-prose vs fix-the-case, and it is a comment on a data file.

docs/evaluation.md also says a cluster of undecided is item analysis, which is a loop-back instruction for exactly one tier.

Settled

The fix-the-prose vs fix-the-case call is Kai's, with an agent proposing. An agent reads the taxonomy and proposes which failure modes are charter defects and which are case defects, with evidence attached. No prose moves before Kai makes the call. This matches the three-party rule in docs/evaluation.md, where the grader seat is Kai's and no party holds two seats.

Open

The loop itself. Kai to describe, then it gets written up. Specifically what is unknown:

What a taxonomy entry turns into. An issue per failure mode, a batched charter revision, something else.
What a prose edit invalidates beyond the boundary-body rule already stated, and whether a charter or meld edit retires results the same way.
Whether a superseded run is retired to evaluations/retired-<date>-<slug>/ or regraded in place with a suffix, and what picks between them.
When a re-run is worth its inference cost against just carrying the finding.
Whether the loop is per seat, since the board is authored one seat at a time, or per whole board.

Constraint on writing it

docs/evaluation.md is at 120 of 120 lines and docs/ is at 40 of 40 pages, so this needs compression in place or a merge to free a slot. Mechanical, and mine to handle once the content is settled.

One thing to flag from the reflow

12c7198 rewrote samples.yaml from the ops slice to the sysadmin slice in place, carrying twelve prompts forward and adding one, without leaving a retired copy. Nothing was lost: samples.yaml holds authored cases rather than results, and the graded ops records under evaluations/pilot/ were not touched by that commit. But under a documented loop that rewrite might have owed a retired copy, and it is worth deciding whether it did.

Refs #318

`docs/evaluation.md` documents the pipeline end to end up to the taxonomy, and `agentic-os/docs/aos-eval.md` says plainly that taxonomy's "output is a list of things to fix rather than a score." **What you do with that list is written down nowhere.** There is no page for the return path. Holding this open rather than drafting it, because the judgment half is Kai's practice and a doctrine page that anchors a rule nobody runs is worse than an absent one. ## What is documented `generator -> inspect eval -> evalkit.filter -> aos-eval annotate -> aos-eval taxonomy`, in `docs/evaluation.md`, with the verbs in the justfile and a runnable command block in `agentic-os/docs/aos-eval.md`. That half is in good shape. ## What exists of the return path Four fragments, located and verified, none of them doctrine and none in the evaluation page: * **`docs/role-boundaries.md:101`** - changing a boundary body moves the pack digest for every role on either side, "retiring those results until an independently reviewed re-run." This is the only invalidation rule in the repo. It covers boundary bodies and says nothing about role charters, melds, adjacency reasons, or purposes, all of which also feed the derived board. * **`evaluations/retired-2026-08-06-codex/`** - the single `retired-*` directory, created in `b3be43f` when the driver-plus-reviewer stack was retired under `#262`. That retired a whole tier of the *method*, not a set of results superseded by a prose edit. So the convention people cite for "where superseded records go" has exactly one instance, and it is not an instance of this case. * **`evaluations/pilot/ops-board-2026-08-12-regraded/`** - a second pattern, and a different one: same board re-annotated, kept alongside the original with a suffix rather than retired. Two precedents, two shapes, no stated rule for which applies when. * **`samples.yaml` header** - the `ops-mls-in` note, where a case produced a refusal in 10 of 10 probe runs regardless of bundle, so it "measured its own framing" and the **case** was rewritten rather than the charter. This is the sharpest thing in the repo on fix-the-prose vs fix-the-case, and it is a comment on a data file. `docs/evaluation.md` also says a cluster of `undecided` is item analysis, which is a loop-back instruction for exactly one tier. ## Settled The fix-the-prose vs fix-the-case call is **Kai's, with an agent proposing**. An agent reads the taxonomy and proposes which failure modes are charter defects and which are case defects, with evidence attached. No prose moves before Kai makes the call. This matches the three-party rule in `docs/evaluation.md`, where the grader seat is Kai's and no party holds two seats. ## Open The loop itself. Kai to describe, then it gets written up. Specifically what is unknown: 1. What a taxonomy entry turns into. An issue per failure mode, a batched charter revision, something else. 2. What a prose edit invalidates beyond the boundary-body rule already stated, and whether a charter or meld edit retires results the same way. 3. Whether a superseded run is retired to `evaluations/retired-<date>-<slug>/` or regraded in place with a suffix, and what picks between them. 4. When a re-run is worth its inference cost against just carrying the finding. 5. Whether the loop is per seat, since the board is authored one seat at a time, or per whole board. ## Constraint on writing it `docs/evaluation.md` is at 120 of 120 lines and `docs/` is at 40 of 40 pages, so this needs compression in place or a merge to free a slot. Mechanical, and mine to handle once the content is settled. ## One thing to flag from the reflow `12c7198` rewrote `samples.yaml` from the ops slice to the sysadmin slice in place, carrying twelve prompts forward and adding one, without leaving a retired copy. Nothing was lost: `samples.yaml` holds authored cases rather than results, and the graded ops records under `evaluations/pilot/` were not touched by that commit. But under a documented loop that rewrite might have owed a retired copy, and it is worth deciding whether it did. Refs #318

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-compose#320

No description provided.

Rows
Columns