aos-eval and agent-compose have crossed ownership: the shared layer embeds a consumer's profile, and its boundaries surface is unused by that consumer #1118

New issue

Open

opened 2026-08-17 22:14:12 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-17 22:14:12 +00:00

Owner

State of the seam between aos-eval and its first consumer. No option chosen. Read at agentic-os main and agent-compose aa276ec, 2026-08-17.

Raised because "should aos own more of agent-compose's eval Python" turned out to have the wrong sign. The volume left in agent-compose/evalkit is small and mostly correctly placed. What is actually crossed is ownership, in both directions at once.

Crossing 1: the generic layer embeds one consumer's domain constants

aos-eval/aos_eval/schema.py:109 hardcodes the consumer's profile:

AGENT_COMPOSE = Profile(
    name="agent-compose",
    test_types=(
        TestTypeSpec("boundary", "binary", 50, ("boundary", "half", "pair_id")),
        TestTypeSpec("role-fit", "binary", 50, ("against",)),
        TestTypeSpec("personality", "fit", 100),
    ),
    boundary_order=("suggest-human-comms", "modify-live-system", "seek-external-validation"),
)

Those three names are agent-compose's boundaries. schema.py:107 carries a comment explaining why the cap is 50 words in terms of the suggest-human-comms out-half, which is reasoning about a specific consumer's doctrine sitting inside the shared package.

Profile.from_dict exists at schema.py:90 for exactly this. aos_eval/schema.py:66 says so directly: "Declaring a profile is how a second repo adopts this schema without the schema growing a branch for it." The mechanism to avoid this is already built and the default consumer does not use it.

And the consumer fetches it back downward. All of agent-compose's aos_eval imports resolve to four from aos_eval.schema, one from aos_eval.io, one from aos_eval.dataset. evalkit/matrix.py:16 is from aos_eval.schema import AGENT_COMPOSE, Half. Nothing in agent-compose passes --profile or calls from_dict, checked across *.py, *.sh, and *.yaml.

Against the config-placement rule in the operating base: config lives at the lowest layer that fully determines it and is never fetched downward. agent-compose's boundary names and word caps are determined by agent-compose. They live one layer down, and agent-compose reaches down to read its own values back.

Crossing 2: the shared pairing derivation is not the one the consumer runs

aos-eval exposes boundaries as a public CLI verb (aos_eval/cli.py:164), backed by aos_eval/boundaries.py (155 lines). derive_slots at boundaries.py:118 turns a declaration of {id, rule, inside, outside} into in/out slot pairs.

agent-compose does not use it. Its justfile calls aos-eval export, aos-eval taxonomy, and aos-eval annotate, never aos-eval boundaries. It imports aos_eval.boundaries nowhere.

Instead evalkit/matrix.py:57 derives the same paired boundary slots from the roster Go exports, because the roster already carries each boundary's owner and which roles defer to it, so inside and outside are derivable rather than hand-written.

The result is two Slot dataclasses with overlapping fields (aos_eval/boundaries.py:39 and evalkit/matrix.py:24) and two implementations of pair derivation. The aos-eval README and the tooling-aos-eval skill both say the pairing rule is worth sharing because both repos reinvented it independently. It is currently implemented twice again, once in the shared layer and once in the consumer that motivated sharing it.

What is correctly placed, for the avoidance of doubt

This is not an argument to move evalkit upward.

evalkit/inspect_bridge.py (49 lines) and evalkit/task.py (58) are the runner seam. inspect_bridge's docstring states the rule: "This is the seam the shared layer deliberately does not cross. aos_eval carries no runner and no model client." Moving them up would put Inspect inside aos-eval and destroy the property that makes it shareable.
evalkit/filter.py (85) already delegates build, validate, and save_dataset upward. What remains is reading Inspect's log format plus argparse.
evalkit/matrix.py (208) interprets agent-compose's roster model: boundaries with owners, adjacency edges with reasons, melds with traits. Moving that up teaches the generic layer what a role meld is.

Options, none chosen

On crossing 1:

Move the profile down. agent-compose declares its own profile and passes --profile. aos-eval keeps Profile.from_dict and ships no named consumer. Costs a config file and a flag on every invocation, and removes the default that currently makes the tool work out of the box.
Keep a built-in default but make it generic rather than named, so the shared layer carries a shape and not suggest-human-comms.
Accept it as a deliberate reference default and document it as such, which makes aos-eval explicitly agent-compose-flavoured rather than neutral.

On crossing 2:

Give boundaries.py a roster-derived entry point that matrix.py can call, so pair derivation lives once and agent-compose keeps deriving inside and outside instead of hand-writing them.
Leave the two derivations separate and stop describing agent-compose as a consumer of the shared pairing rule. The rule it actually shares is the schema and the dataset join.
Move agent-compose to hand-written boundary declarations so it can use the existing surface, which gives up deriving the board from the roster. That property is load-bearing: the derived board picked up a personality swap in agent-compose#299 with no intervention.

Open question

Whether sirens-echo uses aos_eval.boundaries. It is not on this host, so I could not check. If it does, crossing 2 is a two-consumer divergence rather than an unused surface, and option 1 there gets stronger. If it does not, the module currently has no user at all.

agent-compose#262 - the evaluation methodology this machinery serves.
agent-compose#301 - the personality anchors, which sit in agent-compose/evaluations/ rather than in any profile.
aos-eval is pinned by agent-compose at tag aos-eval-v0.2.0, so any change here needs a tag bump on the consumer side.

State of the seam between `aos-eval` and its first consumer. No option chosen. Read at `agentic-os` main and `agent-compose` `aa276ec`, 2026-08-17. Raised because "should aos own more of agent-compose's eval Python" turned out to have the wrong sign. The volume left in `agent-compose/evalkit` is small and mostly correctly placed. What is actually crossed is ownership, in both directions at once. ## Crossing 1: the generic layer embeds one consumer's domain constants `aos-eval/aos_eval/schema.py:109` hardcodes the consumer's profile: ```python AGENT_COMPOSE = Profile( name="agent-compose", test_types=( TestTypeSpec("boundary", "binary", 50, ("boundary", "half", "pair_id")), TestTypeSpec("role-fit", "binary", 50, ("against",)), TestTypeSpec("personality", "fit", 100), ), boundary_order=("suggest-human-comms", "modify-live-system", "seek-external-validation"), ) ``` Those three names are agent-compose's boundaries. `schema.py:107` carries a comment explaining why the cap is 50 words in terms of the `suggest-human-comms` out-half, which is reasoning about a specific consumer's doctrine sitting inside the shared package. `Profile.from_dict` exists at `schema.py:90` for exactly this. `aos_eval/schema.py:66` says so directly: "Declaring a profile is how a second repo adopts this schema without the schema growing a branch for it." The mechanism to avoid this is already built and the default consumer does not use it. **And the consumer fetches it back downward.** All of agent-compose's `aos_eval` imports resolve to four from `aos_eval.schema`, one from `aos_eval.io`, one from `aos_eval.dataset`. `evalkit/matrix.py:16` is `from aos_eval.schema import AGENT_COMPOSE, Half`. Nothing in agent-compose passes `--profile` or calls `from_dict`, checked across `*.py`, `*.sh`, and `*.yaml`. Against the config-placement rule in the operating base: config lives at the lowest layer that fully determines it and is never fetched downward. agent-compose's boundary names and word caps are determined by agent-compose. They live one layer down, and agent-compose reaches down to read its own values back. ## Crossing 2: the shared pairing derivation is not the one the consumer runs `aos-eval` exposes `boundaries` as a public CLI verb (`aos_eval/cli.py:164`), backed by `aos_eval/boundaries.py` (155 lines). `derive_slots` at `boundaries.py:118` turns a declaration of `{id, rule, inside, outside}` into in/out slot pairs. agent-compose does not use it. Its justfile calls `aos-eval export`, `aos-eval taxonomy`, and `aos-eval annotate`, never `aos-eval boundaries`. It imports `aos_eval.boundaries` nowhere. Instead `evalkit/matrix.py:57` derives the same paired boundary slots from the roster Go exports, because the roster already carries each boundary's owner and which roles defer to it, so `inside` and `outside` are derivable rather than hand-written. The result is two `Slot` dataclasses with overlapping fields (`aos_eval/boundaries.py:39` and `evalkit/matrix.py:24`) and two implementations of pair derivation. The `aos-eval` README and the `tooling-aos-eval` skill both say the pairing rule is worth sharing because both repos reinvented it independently. It is currently implemented twice again, once in the shared layer and once in the consumer that motivated sharing it. ## What is correctly placed, for the avoidance of doubt This is not an argument to move `evalkit` upward. * `evalkit/inspect_bridge.py` (49 lines) and `evalkit/task.py` (58) are the runner seam. `inspect_bridge`'s docstring states the rule: "This is the seam the shared layer deliberately does not cross. `aos_eval` carries no runner and no model client." Moving them up would put Inspect inside `aos-eval` and destroy the property that makes it shareable. * `evalkit/filter.py` (85) already delegates `build`, `validate`, and `save_dataset` upward. What remains is reading Inspect's log format plus argparse. * `evalkit/matrix.py` (208) interprets agent-compose's roster model: boundaries with owners, adjacency edges with reasons, melds with traits. Moving that up teaches the generic layer what a role meld is. ## Options, none chosen **On crossing 1:** 1. Move the profile down. agent-compose declares its own profile and passes `--profile`. `aos-eval` keeps `Profile.from_dict` and ships no named consumer. Costs a config file and a flag on every invocation, and removes the default that currently makes the tool work out of the box. 2. Keep a built-in default but make it generic rather than named, so the shared layer carries a shape and not `suggest-human-comms`. 3. Accept it as a deliberate reference default and document it as such, which makes `aos-eval` explicitly agent-compose-flavoured rather than neutral. **On crossing 2:** 1. Give `boundaries.py` a roster-derived entry point that `matrix.py` can call, so pair derivation lives once and agent-compose keeps deriving `inside` and `outside` instead of hand-writing them. 2. Leave the two derivations separate and stop describing agent-compose as a consumer of the shared pairing rule. The rule it actually shares is the schema and the dataset join. 3. Move agent-compose to hand-written boundary declarations so it can use the existing surface, which gives up deriving the board from the roster. That property is load-bearing: the derived board picked up a personality swap in agent-compose#299 with no intervention. ## Open question Whether `sirens-echo` uses `aos_eval.boundaries`. It is not on this host, so I could not check. If it does, crossing 2 is a two-consumer divergence rather than an unused surface, and option 1 there gets stronger. If it does not, the module currently has no user at all. ## Related * `agent-compose#262` - the evaluation methodology this machinery serves. * `agent-compose#301` - the personality anchors, which sit in `agent-compose/evaluations/` rather than in any profile. * aos-eval is pinned by agent-compose at tag `aos-eval-v0.2.0`, so any change here needs a tag bump on the consumer side.

coilyco-ops added the

role/science

priority

labels

2026-08-25 08:11:49 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:49 +00:00

coilyco-ops added this to the Rules that do not hold: silent guards, full budgets, second writers milestone

2026-08-29 07:55:37 +00:00