aos-eval defines what a run writes but not what it reads, so a runner can only run its own repository's board #1221

New issue

Closed

opened 2026-08-23 01:30:08 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-23 01:30:08 +00:00

Member

DatasetEntry is defined here and emitted independently by two different runners. What a run reads has no equivalent, so each deployment carries its own arrangement of the two halves a run needs and no runner can execute another's board.

Those two halves are the same everywhere. A challenge is answered by compiling a context and sending one model call. agent-compose does it as compose --delivery compiled into a per-entity system prompt. sirens-echo does it as composedForRun then evaluationSystemPrompt then BuildTurnPrompt. Different code, identical shape.

What is missing

A declared board: {schema, contexts, challenges, provenance}, where contexts are keyed by entity and hold the compiled context verbatim rather than the recipe for it. Carrying the text rather than the recipe is what lets a runner execute a board it did not build.

It belongs here for the same reason DatasetEntry does. Defining what a run reads is the same job as defining what it writes, and neither obliges this layer to run anything. Putting it beside a runner would give one file two owners and two version lines, since a board is mostly a list of Challenge, and would make a third deployment take a Go binary to emit one.

Two gaps in Challenge it exposes

A conversational subject is asked with a transcript ending on the turn under test, not a single string. prompt cannot express that.
required_tool is an expectation on the answer's shape rather than its prose. It already travels in one deployment's records and has nowhere to live here.

Validation the board should refuse on

An unwritten challenge, a challenge whose entity has no context, a context no challenge uses, an empty context, and anything the deployment's profile says that test type must carry. Before the first token, rather than as a short dataset afterwards.

`DatasetEntry` is defined here and emitted independently by two different runners. What a run *reads* has no equivalent, so each deployment carries its own arrangement of the two halves a run needs and no runner can execute another's board. Those two halves are the same everywhere. A challenge is answered by compiling a context and sending one model call. agent-compose does it as `compose --delivery compiled` into a per-entity system prompt. sirens-echo does it as `composedForRun` then `evaluationSystemPrompt` then `BuildTurnPrompt`. Different code, identical shape. ## What is missing A declared board: `{schema, contexts, challenges, provenance}`, where contexts are keyed by entity and hold the **compiled context verbatim** rather than the recipe for it. Carrying the text rather than the recipe is what lets a runner execute a board it did not build. It belongs here for the same reason `DatasetEntry` does. Defining what a run reads is the same job as defining what it writes, and neither obliges this layer to run anything. Putting it beside a runner would give one file two owners and two version lines, since a board is mostly a list of `Challenge`, and would make a third deployment take a Go binary to emit one. ## Two gaps in Challenge it exposes * A conversational subject is asked with a transcript ending on the turn under test, not a single string. `prompt` cannot express that. * `required_tool` is an expectation on the answer's shape rather than its prose. It already travels in one deployment's records and has nowhere to live here. ## Validation the board should refuse on An unwritten challenge, a challenge whose entity has no context, a context no challenge uses, an empty context, and anything the deployment's profile says that test type must carry. Before the first token, rather than as a short dataset afterwards.

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-23 01:30:27 +00:00

feat(aos-eval): declare the board a run reads, without running one #1220