feat(aos-eval): declare a voice test type, so register becomes gradeable #1370

Merged

coilyco-ops merged 1 commit from aos/claude/ad84-voice into main

2026-08-29 02:01:27 +00:00

coilyco-ops commented

2026-08-29 02:00:10 +00:00

Owner

Part of coilyco-flight-deck/agent-compose#378 item 4. Kai asked for voice rules in the eval pack and a voice dimension in the failure taxonomy, minus the graded run.

Why this is a test type and not a taxonomy entry

The taxonomy is derived, not fixed. aos_eval/taxonomy.py does axial coding over human critique prose and groups by "the axes the dataset already carries". So there is no list of failure modes to append "voice" to. The dimension has to arrive as an axis, and the axis is a test type.

Profile already exists so a deployment declares its own test types without the schema growing a branch. This uses that.

TestTypeSpec("voice", "fit", 100, ("attribute",))

fit rather than binary because voice is a judgement of degree, and the 100-word cap matches personality for the same reason.

The deliberate downstream break

housecast/evalkit/matrix.py:20 unpacks the profile by arity:

BOUNDARY, ROLE_FIT, PERSONALITY = (spec.name for spec in AGENT_COMPOSE.test_types)

with a comment that a fourth test type should "fail loudly here rather than being silently unranked". It now does. That is the intended handshake, and the housecast side lands next once the release train cuts a tag.

Worth flagging separately: housecast pins aos-eval-v0.6.0 while the train is at v0.9.0. Bumping it to pick this up also pulls three minors of other change, so that bump gets tested on its own rather than assumed.

Verification

uv run pytest - 753 passed. pre-commit run --all-files green.

Boundary

Authored from the science seat. aos-eval is the grading half, and the science scope covers "your own runners, probes, graders, and aggregation anywhere else they live", so this is inside scope rather than a crossing.

Part of `coilyco-flight-deck/agent-compose#378` item 4. Kai asked for voice rules in the eval pack and a voice dimension in the failure taxonomy, minus the graded run. ## Why this is a test type and not a taxonomy entry The taxonomy is **derived, not fixed**. `aos_eval/taxonomy.py` does axial coding over human critique prose and groups by "the axes the dataset already carries". So there is no list of failure modes to append "voice" to. The dimension has to arrive as an axis, and the axis is a test type. `Profile` already exists so a deployment declares its own test types without the schema growing a branch. This uses that. TestTypeSpec("voice", "fit", 100, ("attribute",)) `fit` rather than `binary` because voice is a judgement of degree, and the 100-word cap matches `personality` for the same reason. ## The deliberate downstream break `housecast/evalkit/matrix.py:20` unpacks the profile by arity: BOUNDARY, ROLE_FIT, PERSONALITY = (spec.name for spec in AGENT_COMPOSE.test_types) with a comment that a fourth test type should "fail loudly here rather than being silently unranked". It now does. That is the intended handshake, and the housecast side lands next once the release train cuts a tag. Worth flagging separately: **housecast pins `aos-eval-v0.6.0` while the train is at `v0.9.0`.** Bumping it to pick this up also pulls three minors of other change, so that bump gets tested on its own rather than assumed. ## Verification `uv run pytest` - **753 passed**. `pre-commit run --all-files` green. ## Boundary Authored from the science seat. aos-eval is the grading half, and the science scope covers "your own runners, probes, graders, and aggregation anywhere else they live", so this is inside scope rather than a crossing.

coilyco-ops added 1 commit

2026-08-29 02:00:10 +00:00

feat(aos-eval): declare a voice test type, so register becomes gradeable

ci / aos-eval-tests (pull_request) Successful in 7s

Details

ci / aos-cli-tests (pull_request) Successful in 33s

Details

ci / gate (pull_request) Successful in 49s

Details

bc38fa692e

Kai wants voice graded rather than eyeballed. The failure taxonomy is derived by
axial coding rather than a fixed list, so a voice dimension is not a new
category to add: it is an axis the dataset has to carry. That axis is a test
type, and the profile is where a deployment declares its own.

    TestTypeSpec("voice", "fit", 100, ("attribute",))

Matches the personality spec, because voice is a fit judgement rather than a
binary one and needs the same room to answer in.

housecast's evalkit unpacks the profile's test types by arity, with a comment
saying a fourth should fail loudly there rather than being silently unranked.
It now does exactly that, which is the intended handshake rather than a break.

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: eval

coilyco-ops merged commit e3c1605d20 into main

2026-08-29 02:01:27 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-29 02:01:28 +00:00

feat(aos-eval): declare a voice test type, so register becomes gradeable (#1370)

coilyco-ops referenced this pull request

2026-08-29 02:03:56 +00:00

Revert "feat(aos-eval): declare a voice test type" #1372

coilyco-ops referenced this pull request from a commit

2026-08-29 02:03:57 +00:00

Revert "feat(aos-eval): declare a voice test type, so register becomes gradeable (#1370)"