Adopt aos-eval and delete the grading half evalkit no longer owns #295

New issue

Closed

opened 2026-08-17 05:00:45 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-17 05:00:45 +00:00

Member

Filed retroactively by Lucia (AI Engineer seat). The work is written and pushed on aos/claude/gh56-adopt-aos-eval, and it should have had a same-repo issue before the commit rather than after. See the note at the bottom.

What

aos-eval shipped from agentic-os (agentic-os#1104, released as aos-eval-v0.1.0) carrying the grading half both this repo and coilyco-gaming/sirens-echo were implementing separately. agent-compose is the source of that extraction, so it adopts first: if the shared layer cannot serve the repo it came from, it cannot serve Echo, and Echo's adoption is a 32-dataset migration to find that out the expensive way.

Scope

Deleted, now in aos-eval:

evalkit/schema.py, annotate.py, export.py, taxonomy.py
evalkit/tests/test_schema.py, test_export.py, test_taxonomy.py, test_io.py

Kept, because they are the runner:

evalkit/task.py - the Inspect task
evalkit/filter.py - the Inspect log reader, with the join delegated to aos_eval.dataset
evalkit/matrix.py - the roster-derived case list
evalkit/inspect_bridge.py - new. to_inspect and from_inspect, the mapping the shared layer deliberately omits because it ships no runner and no model client

Net 1,552 deletions against 181 insertions.

What the adoption surfaced

Two defects in what I published, both found here rather than in Echo:

aos-eval shipped no py.typed. evalkit-check runs mypy --strict and returned seven import-untyped errors on first sync. Fixed upstream, pinned here at aos-eval-v0.1.1.
A demoted fail-fast. This repo's old Sample raised when a boundary case lacked its pair identity or a role-fit case lacked its against. The shared Sample validates only what every deployment shares, which silently turned that into an opt-in check. filter.load_samples now validates against this repo's profile and raises, so authoring still fails at authoring time while the schema stays portable.

Fidelity

Checked against both committed graded boards rather than fixtures. evaluations/pilot/ops-board-2026-08-12 and -regraded:

taxonomy output byte-identical to pre-adoption
export cases, pairs, and counts identical
the only export differences are the intended format rename and the new provenance key

Also verified through the rewired just evalkit-export and just evalkit-taxonomy verbs, not only the library.

Blocked on

aos-eval-v0.1.1 does not exist yet. The py.typed fix is an open PR on agentic-os, and the train cuts that tag on merge. Once it exists this branch needs uv lock run against the real tag before it can land: I deliberately left uv.lock at its pre-adoption state rather than commit one that recorded a local path.

Process note

The commit on that branch references agentic-os#1104 rather than a same-repo issue, and the branch was pushed without a PR. Both are coding-core-git-workflow violations from not loading that skill before doing cross-repo git work. This issue and the PR that references it are the correction, taken without a force-push.

Filed retroactively by Lucia (AI Engineer seat). The work is written and pushed on `aos/claude/gh56-adopt-aos-eval`, and it should have had a same-repo issue before the commit rather than after. See the note at the bottom. ## What `aos-eval` shipped from agentic-os (agentic-os#1104, released as `aos-eval-v0.1.0`) carrying the grading half both this repo and `coilyco-gaming/sirens-echo` were implementing separately. agent-compose is the source of that extraction, so it adopts first: if the shared layer cannot serve the repo it came from, it cannot serve Echo, and Echo's adoption is a 32-dataset migration to find that out the expensive way. ## Scope Deleted, now in `aos-eval`: * `evalkit/schema.py`, `annotate.py`, `export.py`, `taxonomy.py` * `evalkit/tests/test_schema.py`, `test_export.py`, `test_taxonomy.py`, `test_io.py` Kept, because they are the runner: * `evalkit/task.py` - the Inspect task * `evalkit/filter.py` - the Inspect log reader, with the join delegated to `aos_eval.dataset` * `evalkit/matrix.py` - the roster-derived case list * `evalkit/inspect_bridge.py` - new. `to_inspect` and `from_inspect`, the mapping the shared layer deliberately omits because it ships no runner and no model client Net 1,552 deletions against 181 insertions. ## What the adoption surfaced Two defects in what I published, both found here rather than in Echo: 1. **`aos-eval` shipped no `py.typed`.** `evalkit-check` runs `mypy --strict` and returned seven `import-untyped` errors on first sync. Fixed upstream, pinned here at `aos-eval-v0.1.1`. 2. **A demoted fail-fast.** This repo's old `Sample` raised when a boundary case lacked its pair identity or a role-fit case lacked its `against`. The shared `Sample` validates only what every deployment shares, which silently turned that into an opt-in check. `filter.load_samples` now validates against this repo's profile and raises, so authoring still fails at authoring time while the schema stays portable. ## Fidelity Checked against both committed graded boards rather than fixtures. `evaluations/pilot/ops-board-2026-08-12` and `-regraded`: * taxonomy output byte-identical to pre-adoption * export cases, pairs, and counts identical * the only export differences are the intended format rename and the new `provenance` key Also verified through the rewired `just evalkit-export` and `just evalkit-taxonomy` verbs, not only the library. ## Blocked on `aos-eval-v0.1.1` does not exist yet. The `py.typed` fix is an open PR on agentic-os, and the train cuts that tag on merge. Once it exists this branch needs `uv lock` run against the real tag before it can land: I deliberately left `uv.lock` at its pre-adoption state rather than commit one that recorded a local path. ## Process note The commit on that branch references agentic-os#1104 rather than a same-repo issue, and the branch was pushed without a PR. Both are `coding-core-git-workflow` violations from not loading that skill before doing cross-repo git work. This issue and the PR that references it are the correction, taken without a force-push.

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-17 05:01:55 +00:00

refactor(evalkit): adopt aos-eval, and delete the half it now owns (#1104) #294

coilyco-ops referenced this issue from a commit

2026-08-17 05:10:07 +00:00

fix(evalkit): pin the aos-eval tag that exists, and lock against it (#295)

coilyco-ops referenced this issue from a commit

2026-08-17 05:27:04 +00:00

Merge origin/main into the aos-eval adoption (#295)