Move agent-compose's eval grading layer into AOS so sirens-echo can use it

coilyco-ops commented

2026-08-16 07:00:51 +00:00

Member

Filed by Lucia (AI Engineer seat) at Kai's direction, 2026-08-16. Kai wants agent-compose's eval machinery to live in AOS so coilyco-gaming/sirens-echo can use it.

The goal is right and the duplication is real. The shape needs a decision first, because a straight move does not give Echo what it needs.

The duplication, measured

Two independent evaluation systems, comparable in size, sharing nothing:

agent-compose  evalkit/*.py            1,339 lines + 679 test    Python, inspect-ai
               10 eval docs, 11 verbs

sirens-echo    internal/community/     1,593 lines               Go, bespoke
               evaluation.go, rate.go, board.go, evaluation_checks.go
               15 packs, 32 datasets, 10 eval docs

Live evidence from one session

Working on both repos yesterday I built, without noticing until now:

evalkit/export.py in agent-compose - projects a committed run into a display payload
eval/boundaries.yaml plus cmd/sirens-echo-boundaries in sirens-echo - declares boundaries and derives paired cases

Both are the same idea implemented twice, in two languages, in one day. Both also reinvented the pairing rule: the pair is the scoring unit, never the half. That is the concrete cost this issue exists to stop.

The part that does not move, and it is the crux

agent-compose's evalkit is deliberately harness-free. From lore and the eval design: every case is two inputs, the composed system prompt and the question, with transport through Agent Proxy. That makes ambient context contamination structurally impossible, at an accepted cost:

the board tests the composed text rather than the deployed agent

Echo's defects are harness defects. Raw tool-call markup reaching a member, grounding against a tool result, a filing rule that needs a tracker fixture to be measurable at all, intermittent behaviour over N runs against a live MCP roster. Those cannot be measured by a two-input call, and sirens-echo's rate.go and evaluation.go exist because of it.

So the runner does not move. The two repos are solving different evaluation problems and both designs are correct for their own problem.

What does move

The grading and schema layer, which is genuinely shared and currently duplicated:

schema.py - Sample, Response, DatasetEntry, Annotation, Verdict/Fit, and pair_results. sirens-echo has a parallel vocabulary and a parallel pairing rule.
annotate.py - the human grading loop. One sample per screen, one keystroke per decision, appended after every decision so an interrupted session keeps its work. Echo has a human-graded board with no equivalent tool.
export.py - the one-way display projection, including the refusal gate. Written for the Phoenix surface in coilyco-bridge/deploy#572 and just as applicable to Echo's runs.
The boundaries declaration - eval/boundaries.yaml and its derive-and-check tool, written in sirens-echo yesterday, wanted in both.
matrix.py, taxonomy.py, filter.py - probably, pending a read.

Roughly the grading half of 1,339 lines, against two runners that stay put.

The distribution problem, which is not solved

AOS already ships Python to the fleet, and sirens-echo already consumes it, so the path looks obvious. It is not.

That package is aos-precommit, named and scoped for pre-commit hooks, with dependencies PyYAML, ruamel.yaml, tomli. Adding an eval library to it would put inspect-ai and openai into every repo's pre-commit environment, fleet-wide, to run hooks that never touch a model.

So this needs one of:

A second package from the same repo, for example aos-evalkit, with its own dependency set.
A separate repo, which is a new catalog member with its own trifecta and hooks.
Vendoring, which is the duplication this issue exists to remove.

The first looks right and is a real packaging and release-train decision rather than a file move.

Also worth weighing

Echo is Go and evalkit is Python. The shared layer would be a Python library beside a Go service. That is workable for a grading TUI and an exporter, both of which are operator tools rather than runtime, but it means Echo's Go runner emits a format the Python layer reads rather than calling into it.

A schema shared across two lanes becomes a contract. Today sirens-echo can change its dataset shape freely. After this it cannot, without a coordinated release. That is the cost of the deduplication and it should be accepted deliberately.

Open decisions

Second package, separate repo, or something else. Recommendation above, weakly held.
Does Echo adopt the schema, or does the shared layer read both shapes? Adopting is cleaner and is a migration of 32 committed datasets.
Does the AOS release train carry it, and at what cadence relative to aos-precommit.

coilyco-bridge/deploy #572 - the display surface the exporter feeds.
coilyco-gaming/sirens-echo #846 - the board Echo's side serves.
coilyco-flight-deck/agent-compose #213 - the review-surface decision, which stays where it is.

**Filed by Lucia (AI Engineer seat) at Kai's direction, 2026-08-16.** Kai wants agent-compose's eval machinery to live in AOS so `coilyco-gaming/sirens-echo` can use it. The goal is right and the duplication is real. The shape needs a decision first, because a straight move does not give Echo what it needs. ## The duplication, measured Two independent evaluation systems, comparable in size, sharing nothing: ``` agent-compose evalkit/*.py 1,339 lines + 679 test Python, inspect-ai 10 eval docs, 11 verbs sirens-echo internal/community/ 1,593 lines Go, bespoke evaluation.go, rate.go, board.go, evaluation_checks.go 15 packs, 32 datasets, 10 eval docs ``` ## Live evidence from one session Working on both repos yesterday I built, without noticing until now: * `evalkit/export.py` in agent-compose - projects a committed run into a display payload * `eval/boundaries.yaml` plus `cmd/sirens-echo-boundaries` in sirens-echo - declares boundaries and derives paired cases **Both are the same idea implemented twice, in two languages, in one day.** Both also reinvented the pairing rule: the pair is the scoring unit, never the half. That is the concrete cost this issue exists to stop. ## The part that does not move, and it is the crux agent-compose's evalkit is **deliberately harness-free**. From `lore` and the eval design: every case is two inputs, the composed system prompt and the question, with transport through Agent Proxy. That makes ambient context contamination structurally impossible, at an accepted cost: > the board tests the composed text rather than the deployed agent **Echo's defects are harness defects.** Raw tool-call markup reaching a member, grounding against a tool result, a filing rule that needs a tracker fixture to be measurable at all, intermittent behaviour over N runs against a live MCP roster. Those cannot be measured by a two-input call, and sirens-echo's `rate.go` and `evaluation.go` exist because of it. So the runner does not move. The two repos are solving different evaluation problems and both designs are correct for their own problem. ## What does move The grading and schema layer, which is genuinely shared and currently duplicated: * **`schema.py`** - `Sample`, `Response`, `DatasetEntry`, `Annotation`, `Verdict`/`Fit`, and `pair_results`. sirens-echo has a parallel vocabulary and a parallel pairing rule. * **`annotate.py`** - the human grading loop. One sample per screen, one keystroke per decision, appended after every decision so an interrupted session keeps its work. Echo has a human-graded board with no equivalent tool. * **`export.py`** - the one-way display projection, including the refusal gate. Written for the Phoenix surface in `coilyco-bridge/deploy#572` and just as applicable to Echo's runs. * **The boundaries declaration** - `eval/boundaries.yaml` and its derive-and-check tool, written in sirens-echo yesterday, wanted in both. * **`matrix.py`, `taxonomy.py`, `filter.py`** - probably, pending a read. Roughly the grading half of 1,339 lines, against two runners that stay put. ## The distribution problem, which is not solved AOS already ships Python to the fleet, and sirens-echo already consumes it, so the path looks obvious. It is not. That package is **`aos-precommit`**, named and scoped for pre-commit hooks, with dependencies `PyYAML`, `ruamel.yaml`, `tomli`. Adding an eval library to it would put `inspect-ai` and `openai` into every repo's pre-commit environment, fleet-wide, to run hooks that never touch a model. So this needs one of: * **A second package** from the same repo, for example `aos-evalkit`, with its own dependency set. * **A separate repo**, which is a new catalog member with its own trifecta and hooks. * **Vendoring**, which is the duplication this issue exists to remove. The first looks right and is a real packaging and release-train decision rather than a file move. ## Also worth weighing **Echo is Go and evalkit is Python.** The shared layer would be a Python library beside a Go service. That is workable for a grading TUI and an exporter, both of which are operator tools rather than runtime, but it means Echo's Go runner emits a format the Python layer reads rather than calling into it. **A schema shared across two lanes becomes a contract.** Today sirens-echo can change its dataset shape freely. After this it cannot, without a coordinated release. That is the cost of the deduplication and it should be accepted deliberately. ## Open decisions 1. **Second package, separate repo, or something else.** Recommendation above, weakly held. 2. **Does Echo adopt the schema, or does the shared layer read both shapes?** Adopting is cleaner and is a migration of 32 committed datasets. 3. **Does the AOS release train carry it**, and at what cadence relative to `aos-precommit`. ## Related * `coilyco-bridge/deploy` #572 - the display surface the exporter feeds. * `coilyco-gaming/sirens-echo` #846 - the board Echo's side serves. * `coilyco-flight-deck/agent-compose` #213 - the review-surface decision, which stays where it is.

coilyco-ops added the

labels

2026-08-16 07:00:51 +00:00