Grade the board through aos-eval instead of locally #960

New issue

Closed

opened 2026-08-18 15:58:58 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-18 15:58:58 +00:00

Member

The eval docs cleanup (#957) established what is and is not shared with aos-eval today, by reading both sides rather than the descriptions.

Shared already. eval/boundaries.yaml matches aos_eval.boundaries.Boundary field for field (id, rule, inside, outside, role, origin, derived, seed), so aos-eval boundaries derive reads it as-is with no adapter.

Not shared. just board-deep emits cases: with clause, half, pair_id, history, current, target, responses. aos_eval.io.load_dataset reads the dataset: key and Sample requires id, role, test_type, prompt, target. So annotate, taxonomy, export, and pairs cannot read a board record.

The work:

Declare a sirens-echo profile YAML (test types, label sets, word caps, required fields) so the schema does not have to widen for this deployment.
Emit dataset: from internal/community/board.go with role carrying the grouping axis (the seat, where agent-compose puts a composed role), test_type naming the clause kind, and prompt rendered from history plus current. Keep clause as boundary.
Point grading at aos-eval annotate and drop the local grading path.

Do not delete scripts/boundaries.sh. Its check verifies an origin#fragment still resolves in the source it names, which aos-eval boundaries check does not do (that one compares derived slots to what a dataset authored). The list subcommand does overlap derive and can go.

Until this lands, aos-eval help and the tooling-aos-eval skill overstate the sharing. Tracked on the agentic-os side too.

The eval docs cleanup (#957) established what is and is not shared with `aos-eval` today, by reading both sides rather than the descriptions. **Shared already.** `eval/boundaries.yaml` matches `aos_eval.boundaries.Boundary` field for field (`id`, `rule`, `inside`, `outside`, `role`, `origin`, `derived`, `seed`), so `aos-eval boundaries derive` reads it as-is with no adapter. **Not shared.** `just board-deep` emits `cases:` with `clause`, `half`, `pair_id`, `history`, `current`, `target`, `responses`. `aos_eval.io.load_dataset` reads the `dataset:` key and `Sample` requires `id`, `role`, `test_type`, `prompt`, `target`. So `annotate`, `taxonomy`, `export`, and `pairs` cannot read a board record. The work: 1. Declare a sirens-echo profile YAML (test types, label sets, word caps, required fields) so the schema does not have to widen for this deployment. 2. Emit `dataset:` from `internal/community/board.go` with `role` carrying the grouping axis (the seat, where agent-compose puts a composed role), `test_type` naming the clause kind, and `prompt` rendered from `history` plus `current`. Keep `clause` as `boundary`. 3. Point grading at `aos-eval annotate` and drop the local grading path. **Do not delete `scripts/boundaries.sh`.** Its `check` verifies an `origin#fragment` still resolves in the source it names, which `aos-eval boundaries check` does not do (that one compares derived slots to what a dataset authored). The `list` subcommand does overlap `derive` and can go. Until this lands, `aos-eval help` and the `tooling-aos-eval` skill overstate the sharing. Tracked on the agentic-os side too.

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-08-18 15:59:19 +00:00

aos-eval help and its skill overstate sirens-echo adoption #1125

coilyco-ops commented

2026-08-18 16:59:24 +00:00

Author

Member

Done in #957. board-deep emits the dataset: key with each record carrying aos-eval's Sample plus its output: role holds the grouping axis, which is the clause here, boundary replaces clause, and prompt renders the turn rather than the system prompt. The epochs stay in responses, which aos-eval ignores, so the evidence does not leave the file.

eval/aos-eval-profile.yaml declares the one column, its label set, its 50-word critique cap, and the fields a boundary case cannot omit. just grade, just grade-check, and just taxonomy route to the shared CLI, with AOS_EVAL_REF pinning a tag when a run has to be reproducible.

Verified end to end rather than asserted: aos-eval validate against an emitted dataset reports 10 samples match the profile. TestBoardDatasetCarriesTheAosEvalSampleShape pins the contract so a rename on either side fails loudly.

scripts/boundaries.sh stays, as called for above. Its source-drift check answers a different question than aos-eval boundaries check, which compares derived slots to an authored dataset.

The rate runner is the one leg still local. aos-eval's RunRecord already models it, so that shape is reserved rather than missing.

Done in #957. `board-deep` emits the `dataset:` key with each record carrying aos-eval's `Sample` plus its `output`: `role` holds the grouping axis, which is the clause here, `boundary` replaces `clause`, and `prompt` renders the turn rather than the system prompt. The epochs stay in `responses`, which aos-eval ignores, so the evidence does not leave the file. `eval/aos-eval-profile.yaml` declares the one column, its label set, its 50-word critique cap, and the fields a boundary case cannot omit. `just grade`, `just grade-check`, and `just taxonomy` route to the shared CLI, with `AOS_EVAL_REF` pinning a tag when a run has to be reproducible. Verified end to end rather than asserted: `aos-eval validate` against an emitted dataset reports `10 samples match the profile`. `TestBoardDatasetCarriesTheAosEvalSampleShape` pins the contract so a rename on either side fails loudly. `scripts/boundaries.sh` stays, as called for above. Its source-drift check answers a different question than `aos-eval boundaries check`, which compares derived slots to an authored dataset. The rate runner is the one leg still local. aos-eval's `RunRecord` already models it, so that shape is reserved rather than missing.

coilyco-ops closed this issue

2026-08-18 17:00:17 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#960

No description provided.

Rows
Columns