fix(aos-eval): ship the PEP 561 marker, so a typed consumer sees types #1113

Merged

coilysiren merged 1 commit from aos/claude/gh56-aos-eval-typed into main

2026-08-17 04:54:18 +00:00

coilyco-ops commented

2026-08-17 04:50:50 +00:00

Owner

Found by adopting aos-eval in agent-compose, which is why that repo went first.

evalkit-check runs mypy --strict. On the first sync against aos-eval-v0.1.0 it returned seven errors, all the same one:

evalkit/filter.py:15: error: Skipping analyzing "aos_eval.dataset": module is
installed, but missing library stubs or py.typed marker  [import-untyped]

The package is fully annotated and shipped no PEP 561 marker, so every consumer under strict typing reads it as untyped. That is a defect in what I published, not in the consumer.

Fix

aos_eval/py.typed added
declared under [tool.setuptools.package-data], because present in the tree is not the same as present in the built wheel
version to 0.1.1

Three tests for the shipping contract

The failure mode here is that a packaging mistake is invisible in-repo and only appears in a consumer. tests/test_packaging.py now asserts:

the marker exists next to the installed aos_eval
it is declared as package data, so it reaches the wheel
no runner (inspect-ai, openai, anthropic, litellm) has crept into the dependency set

The third is the invariant the second package exists for, and nothing was checking it.

Verification

just aos-eval-test - 61 passed
pre-commit run --all-files - clean

Ordering

agent-compose's adoption branch is written and green apart from this, and pins aos-eval-v0.1.1. It can land once this merges and the train cuts that tag.

Generated with Claude Code

Found by adopting `aos-eval` in agent-compose, which is why that repo went first. `evalkit-check` runs `mypy --strict`. On the first sync against `aos-eval-v0.1.0` it returned seven errors, all the same one: ``` evalkit/filter.py:15: error: Skipping analyzing "aos_eval.dataset": module is installed, but missing library stubs or py.typed marker [import-untyped] ``` The package is fully annotated and shipped no PEP 561 marker, so every consumer under strict typing reads it as untyped. That is a defect in what I published, not in the consumer. ## Fix * `aos_eval/py.typed` added * declared under `[tool.setuptools.package-data]`, because present in the tree is not the same as present in the built wheel * version to `0.1.1` ## Three tests for the shipping contract The failure mode here is that a packaging mistake is invisible in-repo and only appears in a consumer. `tests/test_packaging.py` now asserts: 1. the marker exists next to the installed `aos_eval` 2. it is declared as package data, so it reaches the wheel 3. no runner (`inspect-ai`, `openai`, `anthropic`, `litellm`) has crept into the dependency set The third is the invariant the second package exists for, and nothing was checking it. ## Verification * `just aos-eval-test` - 61 passed * `pre-commit run --all-files` - clean ## Ordering agent-compose's adoption branch is written and green apart from this, and pins `aos-eval-v0.1.1`. It can land once this merges and the train cuts that tag. Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-17 04:50:51 +00:00

fix(aos-eval): ship the PEP 561 marker, so a typed consumer sees types (#1104 )

ci / aos-cli-tests (pull_request) Successful in 20s

Details

ci / aos-eval-tests (pull_request) Successful in 6s

Details

ci / ward-doctor (pull_request) Successful in 13s

Details

ci / gate (pull_request) Successful in 1m6s

Details

d8e5a7b091

agent-compose is the first adopter and strict mypy immediately read every
aos_eval import as untyped, because the package shipped no py.typed. Three
packaging tests now hold the shipping contract the consumer discovers too
late: the marker exists, it is declared as package data so it reaches the
wheel, and no runner has crept into the dependency set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-17 04:50:51 +00:00