Author and grade the seven-seat board, 91 cases against 13 authored prompts #318

New issue

Open

opened 2026-08-22 20:13:24 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-22 20:13:24 +00:00

Member

Step 4 of #317. The roster landed in 12c7198, so the board now derives. Grading it does not follow automatically, and this is the part that was deferred rather than done.

What derives today

just evalkit-matrix returns 91 cases, 13 per seat: 56 boundary, 21 role-fit, 14 personality. Perfectly even across the seven seats, which the nine-seat roster never was.

The boundary tier is 28 pairs: 4 owner pairs plus the 24 non-owner pairs evaluations/reflow-v3/boundaries.yaml declares. A coverage check against the rendered person.json confirms the declared set is exactly the derived set minus the owner pairs, with no strays either way.

What is authored

samples.yaml carries 13 of the 91, the sysadmin slice. Twelve prompts carry forward from the retired nine-seat ops slice and one is new, the sysadmin-bfs scoped pair that had no analogue before. Seats are added one at a time by design, so the remaining six seats are 78 unauthored slots.

evaluations/reflow-v3/boundaries.yaml holds inside and outside target prose for all 24 non-owner boundary pairs, so the boundary tier has its targets already and needs prompts. The role-fit tier gets its descriptors from adjacency reasons and the personality tier from the meld, both derived, so those need prompts and targets.

Why this is not agent work end to end

Three parties, none holding two seats, is the rule in docs/evaluation.md. An agent may generate candidate cases working with Kai. The subject is evaluation/deepseek-v4-flash through Agent Proxy, which costs inference. The grader is Kai, by hand, and there is deliberately no mechanical scorer to stand in.

So the runnable part is just evalkit-run then just evalkit-filter, and the part that finishes it is just evalkit-annotate.

The scoped pair is the new measurement

Eight of the 24 declared pairs are scoped grants. Those are the ones worth grading first, because they measure something the previous board could not see at all: acted, but exceeded the scope. A binary boundary model scores a scoped seat as either refusing what it was granted or acting past what it was not, and calls both a pass.

Suggested order

Author the six remaining seats into samples.yaml, one seat at a time, boundary targets lifted from reflow-v3/boundaries.yaml.
Run the board unscored at five epochs, filter epoch 1 into a dataset.
Annotate by hand, role-major.
aos-eval taxonomy for the ranked failure modes.

Note on half convention

reflow-v3/boundaries.yaml was authored with in meaning "the rule fires", the reverse of evalkit.matrix and docs/evaluation.md, where in is the role acting on its own territory. 12c7198 swapped its halves and re-keyed its ids to the derived pair ids, so the two agree now. Anyone reading the file from before that commit should re-read it.

Refs #317

Step 4 of `#317`. The roster landed in `12c7198`, so the board now derives. Grading it does not follow automatically, and this is the part that was deferred rather than done. ## What derives today `just evalkit-matrix` returns **91 cases, 13 per seat**: 56 boundary, 21 role-fit, 14 personality. Perfectly even across the seven seats, which the nine-seat roster never was. The boundary tier is 28 pairs: 4 owner pairs plus the 24 non-owner pairs `evaluations/reflow-v3/boundaries.yaml` declares. A coverage check against the rendered `person.json` confirms the declared set is exactly the derived set minus the owner pairs, with no strays either way. ## What is authored `samples.yaml` carries **13** of the 91, the sysadmin slice. Twelve prompts carry forward from the retired nine-seat ops slice and one is new, the `sysadmin-bfs` scoped pair that had no analogue before. Seats are added one at a time by design, so the remaining six seats are 78 unauthored slots. `evaluations/reflow-v3/boundaries.yaml` holds `inside` and `outside` target prose for all 24 non-owner boundary pairs, so the boundary tier has its targets already and needs prompts. The role-fit tier gets its descriptors from adjacency reasons and the personality tier from the meld, both derived, so those need prompts and targets. ## Why this is not agent work end to end Three parties, none holding two seats, is the rule in [docs/evaluation.md](https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/src/branch/main/docs/evaluation.md). An agent may generate candidate cases working with Kai. The subject is `evaluation/deepseek-v4-flash` through Agent Proxy, which costs inference. **The grader is Kai, by hand**, and there is deliberately no mechanical scorer to stand in. So the runnable part is `just evalkit-run` then `just evalkit-filter`, and the part that finishes it is `just evalkit-annotate`. ## The scoped pair is the new measurement Eight of the 24 declared pairs are scoped grants. Those are the ones worth grading first, because they measure something the previous board could not see at all: *acted, but exceeded the scope*. A binary boundary model scores a scoped seat as either refusing what it was granted or acting past what it was not, and calls both a pass. ## Suggested order 1. Author the six remaining seats into `samples.yaml`, one seat at a time, boundary targets lifted from `reflow-v3/boundaries.yaml`. 2. Run the board unscored at five epochs, filter epoch 1 into a dataset. 3. Annotate by hand, role-major. 4. `aos-eval taxonomy` for the ranked failure modes. ## Note on half convention `reflow-v3/boundaries.yaml` was authored with `in` meaning "the rule fires", the reverse of `evalkit.matrix` and `docs/evaluation.md`, where `in` is the role acting on its own territory. `12c7198` swapped its halves and re-keyed its ids to the derived pair ids, so the two agree now. Anyone reading the file from before that commit should re-read it. Refs #317

coilyco-ops referenced this issue

2026-08-22 20:32:18 +00:00

The eval pipeline is documented one way only, and the grade-to-edit return path has no page #320

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-compose#318

No description provided.

Rows
Columns