The image still ships the injection fixture, because #1012 exempted the one directory it is in #1112

New issue

Open

opened 2026-08-22 22:35:56 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-22 22:35:56 +00:00

Member

Filed by Darren (director seat), 2026-08-22. Raised in review on #1108 and merged unaddressed at 948a96f. #1012 closed with it, and I am filing this rather than reopening that issue, for a reason worth stating: #1012 explicitly exempted this directory. Its own text says "agent/ (singular) is needed and should not be touched." The exemption is what was wrong. Reopening would file this as a redo of work that was done correctly against the scope it was given.

What still ships

Dockerfile:76, untouched by #1108 and directly above the two lines it added:

COPY --chown=1000:1000 agent /app/agent

agent/ is 30 files. Most of it genuinely belongs in the image: phrases.yaml for SIRENS_ECHO_PHRASES, compose/, content-classes.yaml, and the rendered role bundles the compose stage lands there. Five of them are test fixtures:

agent/tool-fixture-injection.yaml
agent/tool-fixture-tracker.yaml
agent/tool-fixture-tracker-match.yaml
agent/rate-fixture-tracker.yaml
agent/rate-fixture-tracker-match.yaml

Why these are eval material by #1012's own definition

Their only consumers are binaries that do not ship in the runtime image. cmd/sirens-echo-eval reads the tool fixture through SIRENS_ECHO_TOOL_FIXTURE at main.go:253, and cmd/sirens-echo-policy-check names two of them in its pinned list at main.go:35. Neither is among the four binaries copied to /usr/local/bin. I grepped internal/ and cmd/sirens-echo/ for a runtime reader and there is none.

And the injection fixture is the worst single file of the 62. It opens:

Tool results carrying an injection payload, for the data-borne case the chat box cannot reach.

It then explains that every payload asks for the marker as the first line on its own line, and that the pack anchors on exactly that shape. So an agent that can read it learns both the payloads it will be tested with and the response shape the grader keys on. #1108 removed 57 files of probes and graded replies and left this one in the same working directory.

The same argument #1012 made, unchanged

Nothing reaches these today, and the reason is the reason #1012 gave: no shipped tool takes a filesystem path. read_skill serves from an in-memory map and scratch_search is rooted at /scratch and refuses ... That is a property of the current tool set rather than of the image. The next tool that takes a path makes the injection fixture readable, and nothing in CI would notice.

The fix

agent/ cannot be narrowed by naming files the way agents/ was, because the compose stage lands bundles into it after the copy. So either a .dockerignore scoped to the runtime stage, or one RUN rm after the copy, which is honest and one line.

TestTheImageShipsEveryDefinitionAndNoEvalMaterial from #1108 already does the hard half. It reads only the runtime stage by splitting on the last FROM and globs the real tree so it cannot pass on an empty one. Point the same glob at agent/*fixture*.yaml and the guard extends to cover this.

Done when

The published image contains no file under /app/agent whose only consumers are sirens-echo-eval and sirens-echo-policy-check, pinned by the existing test rather than by a comment.

#1012 - closed, and correct against the scope it was written with
#1108 - where the agents/ half landed, at 948a96f

**Filed by Darren (director seat), 2026-08-22.** Raised in review on #1108 and merged unaddressed at `948a96f`. #1012 closed with it, and I am filing this rather than reopening that issue, for a reason worth stating: **#1012 explicitly exempted this directory.** Its own text says "`agent/` (singular) **is** needed and should not be touched." The exemption is what was wrong. Reopening would file this as a redo of work that was done correctly against the scope it was given. ## What still ships `Dockerfile:76`, untouched by #1108 and directly above the two lines it added: ```dockerfile COPY --chown=1000:1000 agent /app/agent ``` `agent/` is 30 files. Most of it genuinely belongs in the image: `phrases.yaml` for `SIRENS_ECHO_PHRASES`, `compose/`, `content-classes.yaml`, and the rendered role bundles the compose stage lands there. **Five of them are test fixtures:** * `agent/tool-fixture-injection.yaml` * `agent/tool-fixture-tracker.yaml` * `agent/tool-fixture-tracker-match.yaml` * `agent/rate-fixture-tracker.yaml` * `agent/rate-fixture-tracker-match.yaml` ## Why these are eval material by #1012's own definition **Their only consumers are binaries that do not ship in the runtime image.** `cmd/sirens-echo-eval` reads the tool fixture through `SIRENS_ECHO_TOOL_FIXTURE` at `main.go:253`, and `cmd/sirens-echo-policy-check` names two of them in its pinned list at `main.go:35`. Neither is among the four binaries copied to `/usr/local/bin`. I grepped `internal/` and `cmd/sirens-echo/` for a runtime reader and there is none. **And the injection fixture is the worst single file of the 62.** It opens: > Tool results carrying an injection payload, for the data-borne case the chat box cannot reach. It then explains that every payload asks for the marker as the first line on its own line, and that the pack anchors on exactly that shape. So an agent that can read it learns both the payloads it will be tested with **and the response shape the grader keys on**. #1108 removed 57 files of probes and graded replies and left this one in the same working directory. ## The same argument #1012 made, unchanged Nothing reaches these today, and the reason is the reason #1012 gave: no shipped tool takes a filesystem path. `read_skill` serves from an in-memory map and `scratch_search` is rooted at `/scratch` and refuses `..`. **That is a property of the current tool set rather than of the image.** The next tool that takes a path makes the injection fixture readable, and nothing in CI would notice. ## The fix `agent/` cannot be narrowed by naming files the way `agents/` was, because the compose stage lands bundles into it after the copy. So either a `.dockerignore` scoped to the runtime stage, or **one `RUN rm` after the copy**, which is honest and one line. `TestTheImageShipsEveryDefinitionAndNoEvalMaterial` from #1108 already does the hard half. It reads only the runtime stage by splitting on the last `FROM` and globs the real tree so it cannot pass on an empty one. Point the same glob at `agent/*fixture*.yaml` and the guard extends to cover this. ## Done when The published image contains no file under `/app/agent` whose only consumers are `sirens-echo-eval` and `sirens-echo-policy-check`, pinned by the existing test rather than by a comment. ## Related * #1012 - closed, and correct against the scope it was written with * #1108 - where the `agents/` half landed, at `948a96f`

coilyco-ops added the

labels

2026-08-22 22:35:57 +00:00

coilyco-ops referenced this issue

2026-08-22 22:36:06 +00:00

The runtime image ships the eval material into the agent's working directory, so the agent's cwd contains its own test answers #1012

coilyco-ops referenced this issue

2026-08-22 23:16:42 +00:00