feat(aos-eval): a second package for the shared eval grading layer (#1104)

coilyco-ops commented

2026-08-17 03:02:51 +00:00

Member

Closes #1104.

Kai's three calls on the issue, implemented:

second package installed as a click CLI - aos-eval/ is its own project with its own pyproject.toml and dependency set (click, pydantic, PyYAML, rich). Deliberately no inspect-ai and no model client, so nothing here can pull inference into a consumer, and aos-precommit keeps its three-dependency hook environment untouched.
copy the existing scoped release trains - .forgejo/workflows/aos-eval-release.yml mirrors aos-precommit-release.yml on the aos-eval-v* prefix, path-filtered to aos-eval/**. A hook release never drags it and it never drags a hook release.
Echo adapts to agent-compose's schema - it does, without the shared library growing a branch for it. Test types, label sets, word caps, and required fields are a Profile declared in YAML. AGENT_COMPOSE is the built-in default reproducing today's constants exactly.

Both runners stay put

agent-compose's evalkit is deliberately harness-free, two inputs through Agent Proxy, which is what makes ambient contamination structurally impossible. Echo's defects are harness defects - raw tool-call markup reaching a member, grounding against a tool result, intermittency over N runs against a live MCP roster - which a two-input call cannot reach. Both designs are correct for their own problem. What moves is only the grading half.

What moved

schema.py - Sample, Response, RunRecord, Provenance, DatasetEntry, Annotation, Verdict/Fit, PairResult, annotation_order, pair_results. RunRecord and Provenance are Echo's contribution to the shared vocabulary, since without them Echo's committed datasets cannot round-trip.
boundaries.py - the declaration format and derive-and-check tool written in sirens-echo yesterday, generalized. The pairing rule now has exactly one implementation.
annotate.py - the human grading loop. Echo has a human-graded board and had no equivalent tool.
export.py - the one-way display projection and its refusal gate, for the Phoenix surface in coilyco-bridge/deploy#572.
taxonomy.py - axial coding, moved whole.
dataset.py - filter.py's runner-free half. The Inspect log reader stays in agent-compose.

matrix.py and task.py do not move. matrix.py derives its case list from agent-compose's roster Go exports and task.py is the Inspect task, so both are runner-specific.

Verification

aos-eval boundaries derive reads coilyco-gaming/sirens-echo's committed eval/boundaries.yaml unmodified and derives 56 slots from its 28 boundaries. That is the shared layer working against the repo it exists to serve, not against a fixture.

Installability was checked against the pushed branch rather than assumed: a clean venv, uv pip install "aos-eval @ git+<repo>@<branch>#subdirectory=aos-eval", then the installed binary deriving those 56 slots from Echo's declaration fetched over HTTPS.

CI run 21772 on a520cb48 - all four jobs green (aos-cli-tests, aos-eval-tests, ward-doctor, gate)
just aos-eval-test - 58 passed
just test - 540 passed
pre-commit run --all-files - clean

Two defects this PR's own history caught, both mine:

test_pull_request_ci_workflow.py failed because ci.yml gained steps promote.yml lacked, which would have let a narrower promote gate pass a red main.
The first two CI runs failed on uvx: not found - the dev-base image has uv but no uvx. The lint step is now removed rather than patched, because the gate job's pinned pre-commit ruff-check already lints aos-eval/. Verified by introducing a deliberate violation and watching that hook flag it.

Not released

No aos-eval-v* tag exists and aos-eval-release.yml is not on release. The train cuts its first tag only after this merges and promote.yml fast-forwards release with a diff touching aos-eval/**.

Worth flagging

docs/ sits at 40 of 40 for the large band, so the walkthrough went into .agents/skills/tooling-aos-eval/SKILL.md rather than a new docs/aos-eval.md, matching the existing FEATURES entries that point at skills. Say the word if you would rather displace a docs page.

Merge stays director-gated.

Generated with Claude Code

Closes #1104. Kai's three calls on the issue, implemented: * **second package installed as a click CLI** - `aos-eval/` is its own project with its own `pyproject.toml` and dependency set (`click`, `pydantic`, `PyYAML`, `rich`). Deliberately no `inspect-ai` and no model client, so nothing here can pull inference into a consumer, and `aos-precommit` keeps its three-dependency hook environment untouched. * **copy the existing scoped release trains** - `.forgejo/workflows/aos-eval-release.yml` mirrors `aos-precommit-release.yml` on the `aos-eval-v*` prefix, path-filtered to `aos-eval/**`. A hook release never drags it and it never drags a hook release. * **Echo adapts to agent-compose's schema** - it does, without the shared library growing a branch for it. Test types, label sets, word caps, and required fields are a `Profile` declared in YAML. `AGENT_COMPOSE` is the built-in default reproducing today's constants exactly. ## Both runners stay put agent-compose's evalkit is deliberately harness-free, two inputs through Agent Proxy, which is what makes ambient contamination structurally impossible. Echo's defects are harness defects - raw tool-call markup reaching a member, grounding against a tool result, intermittency over N runs against a live MCP roster - which a two-input call cannot reach. Both designs are correct for their own problem. What moves is only the grading half. ## What moved * `schema.py` - `Sample`, `Response`, `RunRecord`, `Provenance`, `DatasetEntry`, `Annotation`, `Verdict`/`Fit`, `PairResult`, `annotation_order`, `pair_results`. `RunRecord` and `Provenance` are Echo's contribution to the shared vocabulary, since without them Echo's committed datasets cannot round-trip. * `boundaries.py` - the declaration format and derive-and-check tool written in sirens-echo yesterday, generalized. The pairing rule now has exactly one implementation. * `annotate.py` - the human grading loop. Echo has a human-graded board and had no equivalent tool. * `export.py` - the one-way display projection and its refusal gate, for the Phoenix surface in coilyco-bridge/deploy#572. * `taxonomy.py` - axial coding, moved whole. * `dataset.py` - `filter.py`'s runner-free half. The Inspect log reader stays in agent-compose. `matrix.py` and `task.py` do not move. `matrix.py` derives its case list from agent-compose's roster Go exports and `task.py` is the Inspect task, so both are runner-specific. ## Verification `aos-eval boundaries derive` reads `coilyco-gaming/sirens-echo`'s committed `eval/boundaries.yaml` unmodified and derives 56 slots from its 28 boundaries. That is the shared layer working against the repo it exists to serve, not against a fixture. Installability was checked against the pushed branch rather than assumed: a clean venv, `uv pip install "aos-eval @ git+<repo>@<branch>#subdirectory=aos-eval"`, then the installed binary deriving those 56 slots from Echo's declaration fetched over HTTPS. * CI run 21772 on `a520cb48` - all four jobs green (`aos-cli-tests`, `aos-eval-tests`, `ward-doctor`, `gate`) * `just aos-eval-test` - 58 passed * `just test` - 540 passed * `pre-commit run --all-files` - clean Two defects this PR's own history caught, both mine: 1. `test_pull_request_ci_workflow.py` failed because `ci.yml` gained steps `promote.yml` lacked, which would have let a narrower promote gate pass a red main. 2. The first two CI runs failed on `uvx: not found` - the dev-base image has `uv` but no `uvx`. The lint step is now removed rather than patched, because the `gate` job's pinned pre-commit `ruff-check` already lints `aos-eval/`. Verified by introducing a deliberate violation and watching that hook flag it. ## Not released No `aos-eval-v*` tag exists and `aos-eval-release.yml` is not on `release`. The train cuts its first tag only after this merges and `promote.yml` fast-forwards `release` with a diff touching `aos-eval/**`. ## Worth flagging `docs/` sits at 40 of 40 for the large band, so the walkthrough went into `.agents/skills/tooling-aos-eval/SKILL.md` rather than a new `docs/aos-eval.md`, matching the existing FEATURES entries that point at skills. Say the word if you would rather displace a docs page. Merge stays director-gated. Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-17 03:02:51 +00:00

feat(aos-eval): a second package for the shared eval grading layer (#1104 )

ci / aos-cli-tests (pull_request) Successful in 20s

Details

ci / aos-eval-tests (pull_request) Failing after 8s

Details

ci / ward-doctor (pull_request) Successful in 14s

Details

ci / gate (pull_request) Successful in 1m19s

Details

4e3c7b3bac

agent-compose and sirens-echo were each implementing the same grading half in
two languages, and in one day both independently reinvented the rule that a
boundary pair is the scoring unit and never the half. This carves that half out
as `aos-eval`, a click CLI shipped as a second package on its own
`aos-eval-v*` train, path-filtered to `aos-eval/**` so a hook release never
drags it and its dependency set never reaches a pre-commit environment.

Neither runner moves. agent-compose's is deliberately harness-free and Echo's
defects are harness defects, so both designs are correct for their own problem
and both stay put. What moves is schema, boundary pairing, the human annotation
loop, the failure taxonomy, and the one-way display export with its refusal
gate.

The schema generalizes rather than hardcoding agent-compose's taxonomy: test
types, label sets, word caps, and required fields are a `Profile`, declared in
YAML, so Echo adopts the schema by declaring its own rather than by growing a
branch in a shared library. `aos-eval boundaries derive` reads Echo's committed
eval/boundaries.yaml unmodified and derives 56 slots from its 28 boundaries.

Kai's three calls on #1104: second package installed as a click CLI, copy the
existing scoped release trains, and Echo adapts to agent-compose's schema.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-17 03:02:51 +00:00

coilyco-ops added 1 commit

2026-08-17 03:59:00 +00:00

docs(aos-eval): install from a ref that exists, since no tag is cut yet

ci / aos-cli-tests (pull_request) Successful in 18s

Details

ci / aos-eval-tests (pull_request) Failing after 5s

Details

ci / ward-doctor (pull_request) Successful in 12s

Details

ci / gate (pull_request) Successful in 53s

Details

653c6d070a

The skill named aos-eval-v0.1.0, which the release train has not created.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops added 1 commit

2026-08-17 04:13:33 +00:00

fix(ci): drop the redundant aos-eval lint step, which needed a uvx the image lacks

ci / aos-cli-tests (pull_request) Successful in 18s

Details

ci / aos-eval-tests (pull_request) Successful in 5s

Details

ci / ward-doctor (pull_request) Successful in 13s

Details

ci / gate (pull_request) Successful in 53s

Details

a520cb48b8

The dev-base image has uv but no uvx, so the step died with exit 127 while the
tests in the same job passed. The step was duplicate coverage regardless: the
gate job's pre-commit ruff-check already lints aos-eval at a pinned rev, which
a deliberate violation confirms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilysiren merged commit 80f04eaf9b into main

2026-08-17 04:17:46 +00:00

coilysiren deleted branch aos/claude/gh56-aos-eval

2026-08-17 04:17:47 +00:00

coilysiren referenced this pull request from a commit

2026-08-17 04:17:47 +00:00

feat(aos-eval): a second package for the shared eval grading layer (#1104) (#1112)

Rows
Columns

feat(aos-eval): a second package for the shared eval grading layer (#1104) #1112

Both runners stay put

What moved

Verification

Not released

Worth flagging