docs(aos-eval): land the probe fold, name the two consumers #1128

Merged

coilysiren merged 1 commit from aos/claude/eval-context into main

2026-08-18 17:24:31 +00:00

coilyco-ops commented

2026-08-18 17:22:49 +00:00

Member

Three things #1124 merged an earlier commit than the branch ended on, so main has the eval page without them.

The probe layer, back from test-harness.md and closing #1126: goose-ask's transcript path, flags, and clean-tree gotcha, plus aos-role-question's mechanisms, tmpfs argument, ten-minute ceiling, and ROLE-CONFIRMED requirement. The boundary is stated once so a green probe never reads as a passed eval.
The corrected help text and skill, closing #1125. Both now say agent-compose and sirens-echo grade through this layer, which is true as of coilyco-gaming/sirens-echo#957 rather than aspirational. HELP also gained WHAT IT REFUSES TO DO: nothing here scores a sample, boundaries check reports missing cases instead of a coverage percentage, and export stops instead of scrubbing.
Cross-repo context, new here. A reader landing on this page had no way to reach either runner or the writeup, so a closing section links both consumers' eval pages and the post.

docs/*.md stays at 40 and the page at 120 of 120 lines. pre-commit run --all-files and just aos-eval-test (61 passed) both pass.

🤖 Generated with Claude Code

Three things #1124 merged an earlier commit than the branch ended on, so `main` has the eval page without them. 1. **The probe layer**, back from `test-harness.md` and closing #1126: `goose-ask`'s transcript path, flags, and clean-tree gotcha, plus `aos-role-question`'s mechanisms, tmpfs argument, ten-minute ceiling, and `ROLE-CONFIRMED` requirement. The boundary is stated once so a green probe never reads as a passed eval. 2. **The corrected help text and skill**, closing #1125. Both now say agent-compose and sirens-echo grade through this layer, which is true as of coilyco-gaming/sirens-echo#957 rather than aspirational. `HELP` also gained `WHAT IT REFUSES TO DO`: nothing here scores a sample, `boundaries check` reports missing cases instead of a coverage percentage, and `export` stops instead of scrubbing. 3. **Cross-repo context**, new here. A reader landing on this page had no way to reach either runner or the writeup, so a closing section links both consumers' eval pages and the post. `docs/*.md` stays at 40 and the page at 120 of 120 lines. `pre-commit run --all-files` and `just aos-eval-test` (61 passed) both pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-18 17:22:49 +00:00

docs(aos-eval): land the probe fold and name the two consumers

ci / aos-eval-tests (pull_request) Successful in 8s

Details

ci / aos-cli-tests (pull_request) Successful in 27s

Details

ci / ward-doctor (pull_request) Successful in 23s

Details

ci / gate (pull_request) Successful in 1m22s

Details

7f746217c9

#1124 merged an earlier commit than the branch ended on, so main has the page
without the probe layer, without the corrected help text, and without the skill
reword. This carries all three onto current main, then adds the cross-repo
context the page was missing.

The probe layer is back from test-harness.md: goose-ask's transcript path, flags,
and clean-tree gotcha, and aos-role-question's mechanisms, tmpfs argument,
ten-minute ceiling, and ROLE-CONFIRMED requirement, with the boundary stated once
so a green probe never reads as a passed eval. Closes #1126.

HELP and the skill say both consumers grade through this layer, which is true as
of coilyco-gaming/sirens-echo#957 rather than aspirational, and HELP gained a
WHAT IT REFUSES TO DO section: nothing here scores a sample, boundaries check
reports missing cases instead of a coverage percentage, and export stops instead
of scrubbing. Closes #1125.

New closing section links both consumers' eval pages and the writeup, because a
reader landing here had no way to reach either runner.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>