refactor(evalkit): adopt aos-eval, and delete the half it now owns (#1104)

coilysiren commented

2026-08-17 04:59:05 +00:00

Owner

agent-compose is the source of the extraction, so it adopts first. If the
shared layer cannot serve the repo it came from, it cannot serve Echo, and
Echo's adoption is a 32-dataset migration to find that out the expensive way.

Deleted: schema.py, annotate.py, export.py, taxonomy.py and their tests, all
now in aos-eval. Kept: the Inspect task, its log reader, and the roster-derived
case list, which are the runner and stay put. inspect_bridge.py carries
to_inspect and from_inspect, the mapping the shared layer deliberately omits
because it ships no runner.

Two defects the adoption surfaced, both in what I published:

aos-eval shipped no py.typed, so strict mypy read every import as untyped.
Fixed upstream and pinned here at aos-eval-v0.1.1.
The shared Sample validates only what every deployment shares, which silently
demoted this repo's fail-fast on a boundary case missing its pair identity or
a role-fit case missing its against. filter.load_samples now validates
against the profile and raises, so authoring still fails at authoring time.

Fidelity checked against both committed graded boards rather than fixtures:
taxonomy output is byte-identical to pre-adoption, and export cases, pairs, and
counts all match. The only export differences are the intended format rename
and the new provenance key.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Co-authored-by: Kai Siren coilysiren@gmail.com
Co-authored-by: Claude noreply@anthropic.com
Co-authored-by: Kai Siren coilysiren@gmail.com
Co-authored-by: Claude noreply@anthropic.com

agent-compose is the source of the extraction, so it adopts first. If the shared layer cannot serve the repo it came from, it cannot serve Echo, and Echo's adoption is a 32-dataset migration to find that out the expensive way. Deleted: schema.py, annotate.py, export.py, taxonomy.py and their tests, all now in aos-eval. Kept: the Inspect task, its log reader, and the roster-derived case list, which are the runner and stay put. inspect_bridge.py carries to_inspect and from_inspect, the mapping the shared layer deliberately omits because it ships no runner. Two defects the adoption surfaced, both in what I published: * aos-eval shipped no py.typed, so strict mypy read every import as untyped. Fixed upstream and pinned here at aos-eval-v0.1.1. * The shared Sample validates only what every deployment shares, which silently demoted this repo's fail-fast on a boundary case missing its pair identity or a role-fit case missing its against. filter.load_samples now validates against the profile and raises, so authoring still fails at authoring time. Fidelity checked against both committed graded boards rather than fixtures: taxonomy output is byte-identical to pre-adoption, and export cases, pairs, and counts all match. The only export differences are the intended format rename and the new provenance key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Kai Siren <coilysiren@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Kai Siren <coilysiren@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

coilysiren added 1 commit

2026-08-17 04:59:05 +00:00

refactor(evalkit): adopt aos-eval, and delete the half it now owns (#1104 )

ci / test (pull_request) Failing after 20s

Details

23488b47df

agent-compose is the source of the extraction, so it adopts first. If the
shared layer cannot serve the repo it came from, it cannot serve Echo, and
Echo's adoption is a 32-dataset migration to find that out the expensive way.

Deleted: schema.py, annotate.py, export.py, taxonomy.py and their tests, all
now in aos-eval. Kept: the Inspect task, its log reader, and the roster-derived
case list, which are the runner and stay put. inspect_bridge.py carries
to_inspect and from_inspect, the mapping the shared layer deliberately omits
because it ships no runner.

Two defects the adoption surfaced, both in what I published:

* aos-eval shipped no py.typed, so strict mypy read every import as untyped.
  Fixed upstream and pinned here at aos-eval-v0.1.1.
* The shared Sample validates only what every deployment shares, which silently
  demoted this repo's fail-fast on a boundary case missing its pair identity or
  a role-fit case missing its against. filter.load_samples now validates
  against the profile and raises, so authoring still fails at authoring time.

Fidelity checked against both committed graded boards rather than fixtures:
taxonomy output is byte-identical to pre-adoption, and export cases, pairs, and
counts all match. The only export differences are the intended format rename
and the new provenance key.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops commented

2026-08-17 05:01:55 +00:00

Member

Closes #295.

Kai opened this PR by hand because I pushed the branch and stopped, which coding-core-git-workflow calls out directly: "A branch with no PR is litter nobody reviews." I had not loaded that skill before doing cross-repo git work, which also explains why the commit references agentic-os#1104 rather than a same-repo issue. #295 is the same-repo issue, filed retroactively. Neither correction needed a force-push.

Adding the two things the commit message does not carry.

Do not merge yet

aos-eval-v0.1.1 does not exist. The py.typed fix is an open PR on agentic-os and the train cuts that tag on merge. After that this branch needs uv lock run against the real tag.

uv.lock is deliberately left at its pre-adoption state. Verifying the typing fix meant syncing against a local path override, which wrote an absolute /private/var/folders/.../aos-eval source into the lock. I caught it before pushing and reverted the file rather than ship a lock that resolves only on my machine. A stale lock is recoverable, a poisoned one is a trap for the next person.

Verification, so it is on the PR and not only in my session

Fidelity against both committed graded boards, evaluations/pilot/ops-board-2026-08-12 and -regraded:

taxonomy output byte-identical to pre-adoption
export cases, pairs, and counts identical
only differences are the intended format rename and the new provenance key

Run through the rewired just evalkit-export and just evalkit-taxonomy verbs as well as the library, so the verb rewiring is covered and not just the imports.

sh scripts/evalkit-check.sh passes ruff, format, mypy --strict, and 17 tests when resolved against the fixed package. pre-commit run --all-files is clean on the branch as pushed.

Closes #295. Kai opened this PR by hand because I pushed the branch and stopped, which `coding-core-git-workflow` calls out directly: "A branch with no PR is litter nobody reviews." I had not loaded that skill before doing cross-repo git work, which also explains why the commit references agentic-os#1104 rather than a same-repo issue. #295 is the same-repo issue, filed retroactively. Neither correction needed a force-push. Adding the two things the commit message does not carry. ## Do not merge yet `aos-eval-v0.1.1` does not exist. The `py.typed` fix is an open PR on agentic-os and the train cuts that tag on merge. After that this branch needs `uv lock` run against the real tag. `uv.lock` is deliberately left at its pre-adoption state. Verifying the typing fix meant syncing against a local path override, which wrote an absolute `/private/var/folders/.../aos-eval` source into the lock. I caught it before pushing and reverted the file rather than ship a lock that resolves only on my machine. A stale lock is recoverable, a poisoned one is a trap for the next person. ## Verification, so it is on the PR and not only in my session Fidelity against both committed graded boards, `evaluations/pilot/ops-board-2026-08-12` and `-regraded`: * taxonomy output byte-identical to pre-adoption * export cases, pairs, and counts identical * only differences are the intended format rename and the new `provenance` key Run through the rewired `just evalkit-export` and `just evalkit-taxonomy` verbs as well as the library, so the verb rewiring is covered and not just the imports. `sh scripts/evalkit-check.sh` passes ruff, format, `mypy --strict`, and 17 tests when resolved against the fixed package. `pre-commit run --all-files` is clean on the branch as pushed.

coilyco-ops added 1 commit

2026-08-17 05:10:07 +00:00

fix(evalkit): pin the aos-eval tag that exists, and lock against it (#295 )

ci / test (pull_request) Successful in 45s

Details

43ae7c1380

CI was red on a tag that will never exist. I pinned aos-eval-v0.1.1 because I
had hand-set that version in the package, but actions/tag-bump defaults to
minor and never reads the commit, so the train cut aos-eval-v0.2.0 from
v0.1.0. docs/release.md says this outright and I pinned from the pyproject
version instead of from the train.

The pyproject version lagging its tag is the documented house pattern, the same
way aos-precommit sits at 0.1.0 under aos-precommit-v0.25.0. So the package
still reports 0.1.1 and the specifier stays true. Only the ref was wrong.

uv.lock is now generated against the real tag and pins its commit, replacing
the pre-adoption lock this branch deliberately carried while the tag was
missing.

evalkit-check: ruff, format, mypy strict, and 17 tests all pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops added 1 commit

2026-08-17 05:27:04 +00:00

Merge origin/main into the aos-eval adoption (#295 )

ci / test (pull_request) Successful in 54s

Details

521041642f

main consolidated docs/ from 65 files to 40 and advanced the hook pin while
this branch was open, so every eval doc I edited was merged away underneath it.
Ten conflicts, none of them behavioural: main's changes to schema.py and
export.py were docstring pointers into renamed docs, and those two files stay
deleted because aos-eval owns them now.

The doc edits are re-applied against the files that replaced the ones I wrote
them into. The command renames land in eval-pipeline, eval-ref-papers,
eval-ref-platforms, and evaluation. The ownership split and the pipeline block
land in eval-pipeline. The fail-fast note lands in evaluation, since the page
it was written for no longer exists.

Also corrected a pre-existing line claiming Ward owns the evalkit verbs, which
the justfile has carried since 0cfe8ea.

evalkit-check and pre-commit pass on the merged tree, and taxonomy output is
still byte-identical to pre-adoption on the committed regraded board.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilysiren merged commit 8a1cf25ac9 into main

2026-08-17 05:42:11 +00:00

coilysiren referenced this pull request from a commit

2026-08-17 05:42:11 +00:00

Merge pull request 'refactor(evalkit): adopt aos-eval, and delete the half it now owns (#1104)' (#294) from aos/claude/gh56-adopt-aos-eval into main

Rows
Columns

refactor(evalkit): adopt aos-eval, and delete the half it now owns (#1104) #294

Do not merge yet

Verification, so it is on the PR and not only in my session