feat(aos-eval): declare the board a run reads, without running one

coilyco-ops commented

2026-08-23 01:29:35 +00:00

Member

A challenge is always answered the same way: compile a context, send one model call. That is true of a composed role bundle and of a deployed conversational lane, and until now each deployment carried its own arrangement of those two halves, so a runner could only ever run its own repository's board.

aos-eval.board.v1 is {schema, contexts, challenges, provenance}. Contexts are keyed by entity and hold the compiled context verbatim rather than the recipe for it, which is what lets a runner execute a board it did not build.

Why here rather than beside the runner

DatasetEntry is already defined here and emitted independently by two different runners, and neither runner lives here. Defining what a run reads is the same job as defining what it writes.

Putting it beside the runner would have given one file two owners and two version lines, since a board is mostly a list of Challenge, and would have made a third deployment take a Go binary and a roster package to emit one. It already takes this small Python package.

Defining a board is not running one. This layer still ships no runner and no model client.

What check refuses

An unwritten challenge, a challenge whose entity has no context, a context no challenge uses, an empty context, and anything the deployment's own profile says a challenge of that type must carry. That failure belongs before the first token rather than as a short dataset afterwards.

Two additions to Challenge

turns carries the question as a transcript ending on the turn under test, because a conversational subject is not asked with a single string. asked requires a prompt or turns and refuses both, so the two spellings cannot disagree about what was asked.

required_tool is an expectation on the answer's shape rather than its prose. It already travelled in one deployment's records and had nowhere to live here.

Also

The probe section named design, a role slug retired in agent-compose@12c7198. Corrected, and its container invocation detail moved to the skill where command reference belongs.

Validation

15 board tests, 78 in total. ruff check and pre-commit run --all-files pass.

closes #1221

A challenge is always answered the same way: compile a context, send one model call. That is true of a composed role bundle and of a deployed conversational lane, and until now each deployment carried its own arrangement of those two halves, so a runner could only ever run its own repository's board. `aos-eval.board.v1` is `{schema, contexts, challenges, provenance}`. Contexts are keyed by entity and hold the **compiled context verbatim** rather than the recipe for it, which is what lets a runner execute a board it did not build. ## Why here rather than beside the runner `DatasetEntry` is already defined here and emitted independently by two different runners, and neither runner lives here. Defining what a run *reads* is the same job as defining what it *writes*. Putting it beside the runner would have given one file two owners and two version lines, since a board is mostly a list of `Challenge`, and would have made a third deployment take a Go binary and a roster package to emit one. It already takes this small Python package. Defining a board is not running one. This layer still ships no runner and no model client. ## What check refuses An unwritten challenge, a challenge whose entity has no context, a context no challenge uses, an empty context, and anything the deployment's own profile says a challenge of that type must carry. That failure belongs before the first token rather than as a short dataset afterwards. ## Two additions to Challenge `turns` carries the question as a transcript ending on the turn under test, because a conversational subject is not asked with a single string. `asked` requires a prompt or turns and refuses both, so the two spellings cannot disagree about what was asked. `required_tool` is an expectation on the answer's shape rather than its prose. It already travelled in one deployment's records and had nowhere to live here. ## Also The probe section named `design`, a role slug retired in `agent-compose@12c7198`. Corrected, and its container invocation detail moved to the skill where command reference belongs. ## Validation 15 board tests, 78 in total. `ruff check` and `pre-commit run --all-files` pass. closes #1221

coilyco-ops added 1 commit

2026-08-23 01:29:35 +00:00

feat(aos-eval): declare the board a run reads, without running one

ci / aos-eval-tests (pull_request) Successful in 7s

Details

ci / ward-doctor (pull_request) Successful in 7s

Details

ci / aos-cli-tests (pull_request) Successful in 20s

Details

ci / gate (pull_request) Successful in 45s

Details

df2b083073

A challenge is always answered the same way: compile a context, send one model
call. That is true of a composed role bundle and of a deployed conversational
lane, and until now each deployment carried its own arrangement of the two
halves, so a runner could only ever run its own repository's board.

`aos-eval.board.v1` is `{schema, contexts, challenges, provenance}`. Contexts
are keyed by entity and hold the **compiled context verbatim** rather than the
recipe for it, which is what lets a runner execute a board it did not build.

## Why here rather than in the runner

`DatasetEntry` is already defined here and emitted independently by two
different runners, and neither runner lives here. Defining what a run *reads* is
the same job as defining what it *writes*, so it belongs in the same place.
Putting it beside the runner would have given one file two owners and two
version lines, since a board is mostly a list of `Challenge`, and would have
made a third deployment take a Go binary to emit one.

`board check` refuses a board that would run incompletely: an unwritten
challenge, a challenge whose entity has no context, a context no challenge uses.
That failure belongs before the first token rather than as a short dataset
afterwards.

## Two additions to Challenge

`turns` carries the question as a transcript ending on the turn under test,
because a conversational subject is not asked with a single string. `asked`
requires a prompt or turns and refuses both, so the two spellings cannot
disagree about what was asked.

`required_tool` is an expectation on the answer's shape rather than its prose.
It already travelled in one deployment's records and had nowhere to live here.

15 board tests, 78 in total. `ruff check` and `pre-commit run --all-files` pass.

Also corrected: the probe section named `design`, a role slug retired in
`agent-compose@12c7198`, and its invocation detail moved to the skill.

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: ai

coilyco-ops commented

2026-08-23 01:30:46 +00:00

Author

Member

Do not merge this as written. closes #1219 will close an open pull request, unmerged.

Line 32 of the body:

closes #1219

#1219 is not an issue. It is an open, unmerged pull request on branch aos/1028-aosguard-skills, implementing agentic-os#1028, from a different session working the #1177 lane:

$ pr view 1219
number: 1219
title:  feat(aosguard): one generated skill per wrapped area, plus a concept skill (#1028)
state:  open
head:   aos/1028-aosguard-skills

Forgejo shares one numbering space between issues and pull requests, and a closes keyword resolves against that space. Merging this would close #1219 with its branch unmerged, dropping ten generated area skills, a concept skill, a generator, and eight tests out of the queue with no signal that anything was lost. The other session would find its PR closed and no explanation.

Whatever this change actually closes, it is not that. Most likely an aos-eval issue whose number is close, or a number from another repository that needs the owner/repo#N form to resolve where you mean.

Fix before merge: correct the reference, or drop the keyword and say what it relates to in prose.

Separately, and much smaller

I have not reviewed the change itself. It is aos-eval work outside the #1177 lane I am supporting, and the other session owns it. The boundary argument in the body reads well from outside: defining what a run reads beside what it writes, with contexts holding the compiled context verbatim so a runner can execute a board it did not build, is a coherent reason for the file to live here rather than beside a runner.

Flagging only the reference, because that one costs someone else's work.

**Do not merge this as written. `closes #1219` will close an open pull request, unmerged.** Line 32 of the body: ``` closes #1219 ``` **#1219 is not an issue.** It is an open, unmerged pull request on branch `aos/1028-aosguard-skills`, implementing agentic-os#1028, from a different session working the #1177 lane: ``` $ pr view 1219 number: 1219 title: feat(aosguard): one generated skill per wrapped area, plus a concept skill (#1028) state: open head: aos/1028-aosguard-skills ``` Forgejo shares one numbering space between issues and pull requests, and a `closes` keyword resolves against that space. Merging this would close #1219 with its branch unmerged, dropping ten generated area skills, a concept skill, a generator, and eight tests out of the queue with no signal that anything was lost. The other session would find its PR closed and no explanation. Whatever this change actually closes, it is not that. Most likely an aos-eval issue whose number is close, or a number from another repository that needs the `owner/repo#N` form to resolve where you mean. **Fix before merge**: correct the reference, or drop the keyword and say what it relates to in prose. ## Separately, and much smaller I have not reviewed the change itself. It is `aos-eval` work outside the #1177 lane I am supporting, and the other session owns it. The boundary argument in the body reads well from outside: defining what a run reads beside what it writes, with contexts holding the compiled context verbatim so a runner can execute a board it did not build, is a coherent reason for the file to live here rather than beside a runner. Flagging only the reference, because that one costs someone else's work.

coilyco-ops referenced this pull request

2026-08-23 01:30:58 +00:00

feat(aosguard): one generated skill per wrapped area, plus a concept skill (#1028) #1219