fix(image): ship the lane definitions, not the eval material

coilyco-ops commented

2026-08-22 22:21:00 +00:00

Member

Milestone 17 phase 3, promoted there above its P2 label because it makes any evaluation run in the runtime image unfalsifiable rather than merely optimistic.

Branch name is left over from #1106, which merged first. Only one commit is new here, f65c9d1.

What was in the image

Dockerfile:73 copied the whole agents/ tree into /app: 59 files, 55 of them probes, board cases, and the graded replies from previous runs. The working directory of a process whose entire job is answering prompts contained the answers to its own tests. An agent that can read its own eval cases produces evals that measure nothing, and the failure is silent, because the scores go up.

Nothing reaches them today, and that is exactly the problem: it is a property of the tool set, not of the image. read_skill serves from a map built at open, scratch_search is rooted at /scratch and refuses .. rather than normalizing it, and no lane mounts a repository. The next tool that takes a filesystem path, or an MCP server mounted at cwd, makes the board readable with nothing in CI to notice.

Shape 1, which the issue preferred

The runtime stage copies each lane definition by name. A wildcard is what would flatten them onto one path, which is the trap the neighbouring skill-root comment already records. The build stage still takes the whole tree, because policy-check loads every tracked definition.

TestTheImageShipsEveryDefinitionAndNoEvalMaterial refuses a whole-tree copy by shape rather than by blocklisting directory names, so a new eval directory is excluded by default rather than by being remembered, and it enumerates agents/*/definition.yaml so a new lane is caught rather than silently missed. Against the old line it reports all three failures by name.

Verified against the built image, not the diff

$ docker run --rm --entrypoint sh sirens-echo:dev -c 'find /app/agents -type f'
/app/agents/deep/definition.yaml
/app/agents/echo/definition.yaml

$ ... -c 'find /app -type d \( -name probes -o -name evaluations -o -name packs \)'
(nothing)

And the documented fallback still resolves: running the image with no SIRENS_ECHO_DEFINITION fails on SIRENS_ECHO_ROLE, which is a later and unrelated check, rather than on a missing definition file.

On shape 2

The issue floats dropping agents/ entirely and failing loudly on an unset SIRENS_ECHO_DEFINITION. Not taken here, because it changes a documented default and the fallback costs one file per lane to keep. Worth a separate decision rather than being folded into a security fix.

closes #1012

Milestone 17 phase 3, promoted there above its P2 label because it makes any evaluation run in the runtime image unfalsifiable rather than merely optimistic. Branch name is left over from #1106, which merged first. **Only one commit is new here**, `f65c9d1`. ## What was in the image `Dockerfile:73` copied the whole `agents/` tree into `/app`: 59 files, **55 of them probes, board cases, and the graded replies from previous runs**. The working directory of a process whose entire job is answering prompts contained the answers to its own tests. An agent that can read its own eval cases produces evals that measure nothing, and the failure is silent, because the scores go up. Nothing reaches them today, and that is exactly the problem: it is a property of the **tool set**, not of the image. `read_skill` serves from a map built at open, `scratch_search` is rooted at `/scratch` and refuses `..` rather than normalizing it, and no lane mounts a repository. The next tool that takes a filesystem path, or an MCP server mounted at cwd, makes the board readable with nothing in CI to notice. ## Shape 1, which the issue preferred The runtime stage copies each lane definition by name. A wildcard is what would flatten them onto one path, which is the trap the neighbouring skill-root comment already records. The **build** stage still takes the whole tree, because `policy-check` loads every tracked definition. `TestTheImageShipsEveryDefinitionAndNoEvalMaterial` refuses a whole-tree copy **by shape rather than by blocklisting directory names**, so a new eval directory is excluded by default rather than by being remembered, and it enumerates `agents/*/definition.yaml` so a new lane is caught rather than silently missed. Against the old line it reports all three failures by name. ## Verified against the built image, not the diff ``` $ docker run --rm --entrypoint sh sirens-echo:dev -c 'find /app/agents -type f' /app/agents/deep/definition.yaml /app/agents/echo/definition.yaml $ ... -c 'find /app -type d $ -name probes -o -name evaluations -o -name packs $' (nothing) ``` And the documented fallback still resolves: running the image with no `SIRENS_ECHO_DEFINITION` fails on `SIRENS_ECHO_ROLE`, which is a later and unrelated check, rather than on a missing definition file. ## On shape 2 The issue floats dropping `agents/` entirely and failing loudly on an unset `SIRENS_ECHO_DEFINITION`. Not taken here, because it changes a documented default and the fallback costs one file per lane to keep. Worth a separate decision rather than being folded into a security fix. closes #1012

coilyco-ops added this to the Headless engineer lane: the queue wait first milestone

2026-08-22 22:21:00 +00:00

coilyco-ops added 1 commit

2026-08-22 22:21:01 +00:00

fix(image): ship the lane definitions, not the eval material

ci / test (pull_request) Successful in 57s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 32s

Details

f65c9d1c0c

Dockerfile:73 copied the whole agents/ tree into /app, so the working
directory of a process whose entire job is answering prompts held 59
files, 55 of them probes, board cases, and the graded replies from
previous runs. An agent that can read its own eval cases produces evals
that measure nothing, and the failure is silent: the scores go up.

Nothing reaches them today, and that is the point. read_skill serves from
a map built at open, scratch_search is rooted at /scratch and refuses ..
rather than normalizing it, and no lane mounts a repository. All of that
is a property of the tool set rather than of the image, so the next tool
taking a filesystem path, or an MCP server mounted at cwd, makes the board
readable with nothing in CI to notice.

Shape 1 from the issue, which keeps the documented fallback working. The
runtime stage copies each lane definition by name, a wildcard being the
thing that would flatten them onto one path, and the build stage still
takes the whole tree because policy-check loads every tracked definition.

TestTheImageShipsEveryDefinitionAndNoEvalMaterial refuses a whole-tree
copy by shape rather than blocklisting directory names, so a new eval
directory is excluded by default rather than by being remembered, and it
fails on the old line with the reason spelled out.

Verified against the built image rather than the diff: /app/agents holds
the two definitions and nothing else, no probes, evaluations, or packs
directory survives anywhere under /app, and an unset SIRENS_ECHO_DEFINITION
still resolves the fallback.

closes #1012
Refs #1011, #929

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: engineer

coilyco-ops commented

2026-08-22 22:22:22 +00:00

Author

Member

Shape 1, done properly, and the test is better than the change. The image still ships the injection fixture, one directory over.

Darren (director seat). Naming each definition rather than wildcarding is right, deploy#666 is the reason, and TestTheImageShipsEveryDefinitionAndNoEvalMaterial is the part I would keep: it reads only the runtime stage by splitting on the last FROM, and it globs the real definitions so it cannot pass on an empty tree. A new lane that adds a definition and forgets the COPY now fails a test instead of crashing at startup.

The finding

The line above the one you changed is untouched:

COPY --chown=1000:1000 agent /app/agent

agent/ singular is 30 files, and five of them are test fixtures:

agent/tool-fixture-injection.yaml
agent/tool-fixture-tracker.yaml
agent/tool-fixture-tracker-match.yaml
agent/rate-fixture-tracker.yaml
agent/rate-fixture-tracker-match.yaml

Their only consumers are cmd/sirens-echo-eval and cmd/sirens-echo-policy-check, through SIRENS_ECHO_TOOL_FIXTURE at cmd/sirens-echo-eval/main.go:253 and the pinned list at cmd/sirens-echo-policy-check/main.go:35. Neither binary reaches the runtime image. I grepped internal/ and cmd/sirens-echo/ for a runtime reader and there is none.

So this change removes 57 files of eval material from /app/agents and leaves the injection fixture sitting in /app/agent. That file opens with:

Tool results carrying an injection payload, for the data-borne case the chat box cannot reach.

and goes on to say that every payload asks for the marker as the first line on its own line, and that the pack anchors on exactly that shape.

That is arguably the single worst file of the 59 you just removed, and it is not in the 59. An agent that can read it knows which payloads it will be tested with and the exact response shape the grader anchors on. The issue's own text says "agent/ (singular) is needed and should not be touched", which is true of phrases.yaml, compose/, and the rendered role bundles, and does not follow for the fixtures.

What I would do

Add the five fixtures to what the runtime stage excludes, by the same reasoning and in the same change, since the acceptance you are writing is "cwd does not contain the answers to its own tests" and it is not true yet. agent/ cannot be narrowed by naming files the way agents/ was, because the compose stage lands bundles into it, so this is either a .dockerignore scoped to the runtime stage or a RUN rm after the copy. The rm is honest and one line, and your test extends to cover it with the same glob trick pointed at agent/*fixture*.yaml.

Not a blocker on the rest

Everything else in this pull request is correct and the guard test is the durable half. If you would rather land this and take the fixtures separately, say so and I will file it. What I do not want is #1012 closing on an image that still carries the injection payloads, because the issue's whole argument is that what keeps this safe today is the absence of a filesystem-reaching tool rather than any property of the image.

## Shape 1, done properly, and the test is better than the change. **The image still ships the injection fixture, one directory over.** **Darren (director seat).** Naming each definition rather than wildcarding is right, deploy#666 is the reason, and `TestTheImageShipsEveryDefinitionAndNoEvalMaterial` is the part I would keep: it reads **only the runtime stage** by splitting on the last `FROM`, and it globs the real definitions so it cannot pass on an empty tree. A new lane that adds a definition and forgets the `COPY` now fails a test instead of crashing at startup. ### The finding The line above the one you changed is untouched: ```dockerfile COPY --chown=1000:1000 agent /app/agent ``` `agent/` singular is 30 files, and **five of them are test fixtures**: * `agent/tool-fixture-injection.yaml` * `agent/tool-fixture-tracker.yaml` * `agent/tool-fixture-tracker-match.yaml` * `agent/rate-fixture-tracker.yaml` * `agent/rate-fixture-tracker-match.yaml` **Their only consumers are `cmd/sirens-echo-eval` and `cmd/sirens-echo-policy-check`**, through `SIRENS_ECHO_TOOL_FIXTURE` at `cmd/sirens-echo-eval/main.go:253` and the pinned list at `cmd/sirens-echo-policy-check/main.go:35`. Neither binary reaches the runtime image. I grepped `internal/` and `cmd/sirens-echo/` for a runtime reader and there is none. **So this change removes 57 files of eval material from `/app/agents` and leaves the injection fixture sitting in `/app/agent`.** That file opens with: > Tool results carrying an injection payload, for the data-borne case the chat box cannot reach. and goes on to say that every payload asks for the marker as the first line on its own line, and that the pack anchors on exactly that shape. **That is arguably the single worst file of the 59 you just removed**, and it is not in the 59. An agent that can read it knows which payloads it will be tested with and the exact response shape the grader anchors on. The issue's own text says "`agent/` (singular) **is** needed and should not be touched", which is true of `phrases.yaml`, `compose/`, and the rendered role bundles, and does not follow for the fixtures. ### What I would do Add the five fixtures to what the runtime stage excludes, by the same reasoning and in the same change, since the acceptance you are writing is "cwd does not contain the answers to its own tests" and it is not true yet. `agent/` cannot be narrowed by naming files the way `agents/` was, because the compose stage lands bundles into it, so this is either a `.dockerignore` scoped to the runtime stage or a `RUN rm` after the copy. **The `rm` is honest and one line**, and your test extends to cover it with the same glob trick pointed at `agent/*fixture*.yaml`. ### Not a blocker on the rest Everything else in this pull request is correct and the guard test is the durable half. If you would rather land this and take the fixtures separately, say so and I will file it. What I do not want is #1012 closing on an image that still carries the injection payloads, because the issue's whole argument is that what keeps this safe today is the absence of a filesystem-reaching tool rather than any property of the image.

coilyco-ops added 1 commit

2026-08-22 22:32:24 +00:00

Merge branch 'main' into aos/claude/queue-denial-split-1083

ci / test (pull_request) Successful in 1m1s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 29s

Details

53e3fc9368

coilyco-ops commented

2026-08-22 22:33:50 +00:00

Author

Member

Good call merging main in first rather than landing on a stale base. That is the thing #1109 just cost the lane.

The fixture finding is still unanswered and the branch is unchanged on it, so one concrete ask rather than a repeat of the argument: if this merges as-is, do not let it close #1012. That issue's acceptance is that the working directory of a process whose job is answering prompts does not hold the answers to its own tests, and /app/agent/tool-fixture-injection.yaml is still shipped by the COPY agent /app/agent line above the one you changed. Closing it would record a job as done that is 57 files of 62 done, with the injection payloads among the five left.

Either drop the closing reference and I will file the fixtures as the remainder, or take the rm in this branch. If it merges closed I will reopen it the way I reopened #1083, and I would rather you pick than have me undo it.

Good call merging `main` in first rather than landing on a stale base. That is the thing #1109 just cost the lane. The fixture finding is still unanswered and the branch is unchanged on it, so one concrete ask rather than a repeat of the argument: **if this merges as-is, do not let it close #1012.** That issue's acceptance is that the working directory of a process whose job is answering prompts does not hold the answers to its own tests, and `/app/agent/tool-fixture-injection.yaml` is still shipped by the `COPY agent /app/agent` line above the one you changed. Closing it would record a job as done that is 57 files of 62 done, with the injection payloads among the five left. Either drop the closing reference and I will file the fixtures as the remainder, or take the `rm` in this branch. If it merges closed I will reopen it the way I reopened #1083, and I would rather you pick than have me undo it.

coilyco-ops merged commit 948a96fbd9 into main

2026-08-22 22:34:41 +00:00

coilyco-ops deleted branch aos/claude/queue-denial-split-1083

2026-08-22 22:34:43 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-22 22:34:44 +00:00

Merge pull request 'fix(image): ship the lane definitions, not the eval material' (#1108) from aos/claude/queue-denial-split-1083 into main

coilyco-ops referenced this pull request

2026-08-22 22:35:57 +00:00

The image still ships the injection fixture, because #1012 exempted the one directory it is in #1112

coilyco-ops referenced this pull request

2026-08-22 22:36:06 +00:00