aos-eval hardcodes one deployment's vocabulary in the layer that is supposed to be agnostic #1195

New issue

Closed

opened 2026-08-22 23:07:34 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-22 23:07:34 +00:00

Member

aos-eval exists so two unlike deployments share one grading layer. The Profile type is the whole mechanism: a deployment declares its own test types, label sets, and word caps, and the shared code stays out of it.

Three places break that.

The clearest one

aos_eval/taxonomy.py, assigning a failure its structural axis:

if challenge.boundary and challenge.half: return f"{challenge.boundary}:{challenge.half.value}"
if challenge.against:                     return f"role-fit:{challenge.against}"
if challenge.trait:                       return f"personality:{challenge.trait}"
return challenge.test_type

role-fit and personality are agent-compose's words. sirens-echo declares only a boundary test type, so branches two and three are dead code there. Any deployment adding a fourth kind falls through to a bare label with no axis detail, which is exactly the case the Profile indirection was built to serve.

The same three-branch chain repeats in annotate.py as three header rows and in export.py as three fields.

What it actually is

A challenge tests one entity for one attribute.

entity - what is under test. Challenge.role already meant this generically, and its own docstring said so: "A composed role in one deployment, a deployed agent in another." It was just not named for it.
attribute - what is being tested about the entity. boundary, against, and trait are three fields doing one job, and test_type already says which kind.

The collapse is lossless. axis_of becomes a derivation instead of an enumeration, and a test type this layer has never heard of still gets an axis with detail.

Scope

Challenge.role to entity. boundary + against + trait to attribute. Profile.group_order to entity_order and boundary_order to attribute_order. PairResult follows. Declaration schema to aos-eval.boundaries.v2, whose role: key becomes entity:.

Breaking, but consumers pin by tag, so nothing breaks until a pin moves.

While the field is moving: AGENT_COMPOSE.attribute_order still lists three boundary slugs that stopped existing in agent-compose@12c7198.

Downstream

agent-compose and sirens-echo both follow once the tag is cut. Tracked separately.

`aos-eval` exists so two unlike deployments share one grading layer. The `Profile` type is the whole mechanism: a deployment declares its own test types, label sets, and word caps, and the shared code stays out of it. Three places break that. ## The clearest one `aos_eval/taxonomy.py`, assigning a failure its structural axis: ```python if challenge.boundary and challenge.half: return f"{challenge.boundary}:{challenge.half.value}" if challenge.against: return f"role-fit:{challenge.against}" if challenge.trait: return f"personality:{challenge.trait}" return challenge.test_type ``` `role-fit` and `personality` are agent-compose's words. sirens-echo declares only a `boundary` test type, so branches two and three are dead code there. Any deployment adding a fourth kind falls through to a bare label with no axis detail, which is exactly the case the `Profile` indirection was built to serve. The same three-branch chain repeats in `annotate.py` as three header rows and in `export.py` as three fields. ## What it actually is A challenge tests one **entity** for one **attribute**. * **entity** - what is under test. `Challenge.role` already meant this generically, and its own docstring said so: *"A composed role in one deployment, a deployed agent in another."* It was just not named for it. * **attribute** - what is being tested about the entity. `boundary`, `against`, and `trait` are three fields doing one job, and `test_type` already says which kind. The collapse is lossless. `axis_of` becomes a derivation instead of an enumeration, and a test type this layer has never heard of still gets an axis with detail. ## Scope `Challenge.role` to `entity`. `boundary` + `against` + `trait` to `attribute`. `Profile.group_order` to `entity_order` and `boundary_order` to `attribute_order`. `PairResult` follows. Declaration schema to `aos-eval.boundaries.v2`, whose `role:` key becomes `entity:`. Breaking, but consumers pin by tag, so nothing breaks until a pin moves. While the field is moving: `AGENT_COMPOSE.attribute_order` still lists three boundary slugs that stopped existing in `agent-compose@12c7198`. ## Downstream agent-compose and sirens-echo both follow once the tag is cut. Tracked separately.

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-22 23:07:56 +00:00

feat(aos-eval)!: name the two abstractions the schema was already using #1196

coilyco-ops referenced this issue from a commit

2026-08-22 23:50:08 +00:00

refactor(aos-eval)!: finish the swap, in the module name and the roster read