aos-eval help and its skill overstate sirens-echo adoption #1125

New issue

Open

opened 2026-08-18 15:59:19 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-18 15:59:19 +00:00

Member

Both INTRO/HELP in aos-eval/aos_eval/cli.py and .agents/skills/tooling-aos-eval/SKILL.md describe the layer as "shared by agent-compose and sirens-echo". Verified against both repos in #1124: sirens-echo shares the boundary declaration shape only.

eval/boundaries.yaml there matches Boundary field for field, so boundaries derive reads it as-is. Its board dataset does not: board-deep emits cases: keyed on clause/history/current where load_dataset reads dataset: and Sample requires role/test_type/prompt. So annotate, taxonomy, export, and pairs cannot read a board record.

docs/aos-eval.md now states this accurately. The shipped help text and the skill still do not, and those are what an agent with no checkout reads.

Fix either by wording it as intent ("the layer two runners are converging on") or by waiting for coilyco-gaming/sirens-echo#960 to land and leaving the text alone. Prefer the wording fix now, because the current text is the reason a reader would assume a working end-to-end path that does not exist.

Both `INTRO`/`HELP` in `aos-eval/aos_eval/cli.py` and `.agents/skills/tooling-aos-eval/SKILL.md` describe the layer as "shared by agent-compose and sirens-echo". Verified against both repos in #1124: sirens-echo shares the boundary declaration shape only. `eval/boundaries.yaml` there matches `Boundary` field for field, so `boundaries derive` reads it as-is. Its board dataset does not: `board-deep` emits `cases:` keyed on `clause`/`history`/`current` where `load_dataset` reads `dataset:` and `Sample` requires `role`/`test_type`/`prompt`. So `annotate`, `taxonomy`, `export`, and `pairs` cannot read a board record. `docs/aos-eval.md` now states this accurately. The shipped help text and the skill still do not, and those are what an agent with no checkout reads. Fix either by wording it as intent ("the layer two runners are converging on") or by waiting for coilyco-gaming/sirens-echo#960 to land and leaving the text alone. Prefer the wording fix now, because the current text is the reason a reader would assume a working end-to-end path that does not exist.

coilyco-ops commented

2026-08-18 16:59:52 +00:00

Author

Member

Resolved by making the claim true, then rewording what was left.

sirens-echo now grades through this layer (coilyco-gaming/sirens-echo#957), verified with aos-eval validate against an emitted dataset reporting 10 samples match the profile. So "shared by agent-compose and sirens-echo" is accurate, and INTRO/HELP plus the skill now say both consumers grade through it rather than implying it by listing them.

HELP also gained a WHAT IT REFUSES TO DO section, because the overclaim had a sibling worth naming: nothing here scores a sample, boundaries check reports missing cases instead of a coverage percentage, and export stops instead of scrubbing. A number this command prints can come back negative.

Resolved by making the claim true, then rewording what was left. sirens-echo now grades through this layer (coilyco-gaming/sirens-echo#957), verified with `aos-eval validate` against an emitted dataset reporting `10 samples match the profile`. So "shared by agent-compose and sirens-echo" is accurate, and `INTRO`/`HELP` plus the skill now say both consumers grade through it rather than implying it by listing them. `HELP` also gained a `WHAT IT REFUSES TO DO` section, because the overclaim had a sibling worth naming: nothing here scores a sample, `boundaries check` reports missing cases instead of a coverage percentage, and `export` stops instead of scrubbing. A number this command prints can come back negative.

coilyco-ops referenced this issue from a commit

2026-08-18 17:17:41 +00:00

docs(aos-eval): land the probe fold and name the two consumers

coilyco-ops referenced this issue

2026-08-18 17:22:49 +00:00

docs(aos-eval): land the probe fold, name the two consumers #1128