docs(agents): generalize evaluation inference clause (inbox#429) #1236

Merged

coilyco-ops merged 1 commit from aos/claude/yb89 into main

2026-08-25 18:44:58 +00:00

coilyco-ops commented

2026-08-25 18:38:29 +00:00

Owner

Closes the clause half of coilysiren/inbox#429.

What changes

One paragraph in AGENTS.md. Two substantive edits:

No vendor is named. The clause named a single vendor, which read as a per-vendor allowlist and invited adjudicating each new one. The admitted data class was never vendor-specific: aos-eval's export already projects challenge text and subject output to a public display target, so a provider that harvests them receives what a reader could already fetch.
The graded layer is excluded. Critiques, evidence spans, and the derived failure taxonomy now sit alongside secrets in the exclusion list. A board is an instrument later runs are compared against, so a provider that trains on those compromises the measurement rather than leaking a prompt, and the damage is silent.

The second is the one that matters once graders run on third-party inference. A remote grader sees the challenge and the subject output, and never the graded layer.

Why the reasoning is not in docs

It cites inbox#429 instead. Three homes were tried and all three breached a cap: the full clause in AGENTS.md by 219 chars, a section in docs/aos-eval.md by 20 lines, and a new docs/evaluation-egress.md by one doc against the 40-doc cap. The caps were right each time and the clause got shorter and better under them. That structural pressure is filed as coilysiren/inbox#432 rather than fixed here.

Verification

pre-commit run --all-files passes clean. AGENTS.md lands at 32,306 chars, 194 under the 32,500 cap. One line replaced, no other file touched.

Closes the clause half of coilysiren/inbox#429. ## What changes One paragraph in `AGENTS.md`. Two substantive edits: * **No vendor is named.** The clause named a single vendor, which read as a per-vendor allowlist and invited adjudicating each new one. The admitted data class was never vendor-specific: `aos-eval`'s `export` already projects challenge text and subject output to a public display target, so a provider that harvests them receives what a reader could already fetch. * **The graded layer is excluded.** Critiques, evidence spans, and the derived failure taxonomy now sit alongside secrets in the exclusion list. A board is an instrument later runs are compared against, so a provider that trains on those compromises the measurement rather than leaking a prompt, and the damage is silent. The second is the one that matters once graders run on third-party inference. A remote grader sees the challenge and the subject output, and never the graded layer. ## Why the reasoning is not in docs It cites `inbox#429` instead. Three homes were tried and all three breached a cap: the full clause in `AGENTS.md` by 219 chars, a section in `docs/aos-eval.md` by 20 lines, and a new `docs/evaluation-egress.md` by one doc against the 40-doc cap. The caps were right each time and the clause got shorter and better under them. That structural pressure is filed as coilysiren/inbox#432 rather than fixed here. ## Verification `pre-commit run --all-files` passes clean. `AGENTS.md` lands at 32,306 chars, 194 under the 32,500 cap. One line replaced, no other file touched.

coilyco-ops added 1 commit

2026-08-25 18:38:30 +00:00

docs(agents): generalize evaluation inference clause (inbox#429)

ci / aos-eval-tests (pull_request) Successful in 9s

Details

ci / ward-doctor (pull_request) Successful in 10s

Details

ci / gate (pull_request) Successful in 1m10s

Details

ci / aos-cli-tests (pull_request) Successful in 4m56s

Details

a8b6664139

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

coilyco-ops merged commit f01696eb68 into main

2026-08-25 18:44:58 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-25 18:45:00 +00:00

docs(agents): generalize evaluation inference clause (inbox#429) (#1236)

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os!1236

No description provided.

Rows
Columns