agent-compose still documents aos-eval as a live tool across 9 files, which the deletion survey missed #1416

New issue

Open

opened 2026-08-29 13:31:09 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-29 13:31:09 +00:00

Owner

Deleting aos-eval (#1374, landed as c499c377 / #1389) left agent-compose documenting it as a live tool. I did that deletion, so this is mine to report.

What #1374's survey named, and what it missed

It named one blocking consumer:

coilyco-gaming/sirens-echo calls the aos-eval CLI from its justfile and keeps eval/aos-eval-profile.yaml.

agent-compose was never surveyed, and it carries 16 references across 9 files, measured on its main at b7a14ac:

pyproject.toml                            1   prose comparison
README.md                                 1   "ships separately as aos-eval in agentic-os"
docs/evaluation.md                        4   the grading workflow, incl. `aos-eval help`
docs/FEATURES.md                          1   "grades it by hand with aos-eval"
docs/architecture.md                      1
docs/eval-references.md                   1   "the platform equivalent of aos-eval annotate"
docs/eval-ref-papers.md                   1   "`aos-eval annotate` is one case per screen"
docs/eval-ref-platforms.md                1
examples/claude-native-ui/settings.science.json  1   "theme": "custom:aos-eval"

Nothing imports it, so no build breaks. What breaks is the documentation: docs/evaluation.md describes the whole grading path through a command that no longer ships, and three eval-ref-* pages compare external tools to aos-eval annotate as if a reader could run it.

The settings.science.json hit is a UI theme string, cosmetic.

What it should say instead

#1374's own reasoning gives the answer: "housecast was always supposed to end aos-eval and had not. housecast-v0.2.0 now carries the grading half entire." So agent-compose's evaluation docs should point at housecast's evalkit, which agent-compose already vendors.

Why I am not doing it

agent-compose prose is the science seat's scope, not platform's, and rewriting an evaluation walkthrough needs to know how evalkit actually spells what aos-eval annotate did. I measured it and left the checkout untouched - no worktree remains.

The general lesson, which is the part worth keeping

A deletion's consumer survey is only as good as its search. #1374 found sirens-echo because sirens-echo called the CLI, and missed agent-compose because agent-compose only documented it. Grepping for imports and invocations finds the first class and not the second, and the second is where a stale pointer sits until someone tries to follow it.

Deleting `aos-eval` (#1374, landed as `c499c377` / #1389) left agent-compose documenting it as a live tool. I did that deletion, so this is mine to report. ## What #1374's survey named, and what it missed It named one blocking consumer: > `coilyco-gaming/sirens-echo` calls the `aos-eval` CLI from its justfile and keeps `eval/aos-eval-profile.yaml`. **agent-compose was never surveyed**, and it carries 16 references across 9 files, measured on its `main` at `b7a14ac`: ``` pyproject.toml 1 prose comparison README.md 1 "ships separately as aos-eval in agentic-os" docs/evaluation.md 4 the grading workflow, incl. `aos-eval help` docs/FEATURES.md 1 "grades it by hand with aos-eval" docs/architecture.md 1 docs/eval-references.md 1 "the platform equivalent of aos-eval annotate" docs/eval-ref-papers.md 1 "`aos-eval annotate` is one case per screen" docs/eval-ref-platforms.md 1 examples/claude-native-ui/settings.science.json 1 "theme": "custom:aos-eval" ``` Nothing imports it, so no build breaks. What breaks is the documentation: `docs/evaluation.md` describes the whole grading path through a command that no longer ships, and three `eval-ref-*` pages compare external tools to `aos-eval annotate` as if a reader could run it. The `settings.science.json` hit is a UI theme string, cosmetic. ## What it should say instead #1374's own reasoning gives the answer: "housecast was always supposed to end aos-eval and had not. `housecast-v0.2.0` now carries the grading half entire." So agent-compose's evaluation docs should point at housecast's `evalkit`, which agent-compose already vendors. ## Why I am not doing it agent-compose prose is the science seat's scope, not platform's, and rewriting an evaluation walkthrough needs to know how `evalkit` actually spells what `aos-eval annotate` did. I measured it and left the checkout untouched - no worktree remains. ## The general lesson, which is the part worth keeping A deletion's consumer survey is only as good as its search. #1374 found sirens-echo because sirens-echo *called the CLI*, and missed agent-compose because agent-compose only *documented* it. Grepping for imports and invocations finds the first class and not the second, and the second is where a stale pointer sits until someone tries to follow it.

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-08-29 13:31:11 +00:00

coilyco-ops referenced this issue

2026-08-29 13:31:25 +00:00

Delete aos-eval/ and its nine reference sites, once sirens-echo has moved to housecast grade #1374