docs(aos-eval): give the shared grading layer its own page #1124

Merged

coilysiren merged 1 commit from aos/claude/ap47 into main

2026-08-18 16:09:14 +00:00

coilyco-ops commented

2026-08-18 15:57:30 +00:00

Member

aos-eval ships from this repo and had no docs page. Its FEATURES entry pointed at the skill, so a reader with a checkout had nowhere to read why the layer is split the way it is.

Adds docs/aos-eval.md: what is shared and what stays in each runner, the pairing rule, profiles, resumable evidence-anchored grading, why export refuses instead of scrubbing, and how a third repo adopts it.

It also records what the skill and aos-eval help currently overstate. sirens-echo shares the boundary declaration shape, so boundaries derive and check read its eval/boundaries.yaml as-is, but its board dataset keys on clause/history/current where Sample requires role/test_type/prompt. So annotate, taxonomy, and export cannot read a board record yet. One full consumer and one partial one.

The docs count was at the large-band cap, so this evicts test-harness.md: it declared a test-harness-<agent> family never authored, stapled its one child inline, and covered the ad-hoc predecessor of this layer. goose-ask and aos-role-question stay in the justfile. Rehoming their notes is a follow-up.

pre-commit run --all-files passes. just test has one pre-existing failure, test_ward_specs_bundle.py, from no ward binary on this host.

🤖 Generated with Claude Code

aos-eval ships from this repo and had no docs page. Its FEATURES entry pointed at the skill, so a reader with a checkout had nowhere to read why the layer is split the way it is. Adds `docs/aos-eval.md`: what is shared and what stays in each runner, the pairing rule, profiles, resumable evidence-anchored grading, why export refuses instead of scrubbing, and how a third repo adopts it. **It also records what the skill and `aos-eval help` currently overstate.** sirens-echo shares the boundary declaration shape, so `boundaries derive` and `check` read its `eval/boundaries.yaml` as-is, but its board dataset keys on `clause`/`history`/`current` where `Sample` requires `role`/`test_type`/`prompt`. So `annotate`, `taxonomy`, and `export` cannot read a board record yet. One full consumer and one partial one. The docs count was at the large-band cap, so this evicts `test-harness.md`: it declared a `test-harness-<agent>` family never authored, stapled its one child inline, and covered the ad-hoc predecessor of this layer. `goose-ask` and `aos-role-question` stay in the justfile. Rehoming their notes is a follow-up. `pre-commit run --all-files` passes. `just test` has one pre-existing failure, `test_ward_specs_bundle.py`, from no `ward` binary on this host. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-18 15:57:30 +00:00

docs(aos-eval): give the shared grading layer its own page

ci / aos-eval-tests (pull_request) Successful in 6s

Details

ci / ward-doctor (pull_request) Successful in 15s

Details

ci / aos-cli-tests (pull_request) Successful in 22s

Details

ci / gate (pull_request) Successful in 1m19s

Details

c80290605f

aos-eval ships from this repo and is the layer agent-compose grades through,
and it had no docs page. Its FEATURES entry pointed at the skill, so a reader
with a checkout had nowhere to read why the layer is split the way it is: what
is shared, what stays in each runner, why the pair is the scoring unit, why
export refuses instead of scrubbing, and how a third repo adopts it through a
profile.

The page also records what the skill and `aos-eval help` currently overstate.
sirens-echo shares the boundary declaration shape, so `boundaries derive` and
`check` read its eval/boundaries.yaml as-is, but its board dataset keys on
clause/history/current where Sample requires role/test_type/prompt, so
annotate, taxonomy, and export cannot read a board record yet. One full
consumer and one partial one.

The docs count was at the large-band cap, so this evicts test-harness.md. That
page declared a `test-harness-<agent>` family that was never authored, stapled
its one child inline, and covered the ad-hoc predecessor of this layer:
probing an agent and model pairing before trusting it with real work.
`goose-ask` and `aos-role-question` stay in the justfile and describe
themselves. Rehoming their notes is a follow-up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-18 15:57:30 +00:00

coilyco-ops referenced this pull request

2026-08-18 15:59:19 +00:00

aos-eval help and its skill overstate sirens-echo adoption #1125

coilyco-ops referenced this pull request

2026-08-18 15:59:21 +00:00

Rehome the goose-ask and aos-role-question probe notes #1126