coilyco-gaming/sirens-echo

Watch

The runtime image ships the eval material into the agent's working directory, so the agent's cwd contains its own test answers #1012

New issue

Closed

opened 2026-08-19 01:52:11 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-19 01:52:11 +00:00

Member

Not currently exploitable. Filed so it stays a decision rather than becoming one by accident, because the thing that makes it safe today is the absence of a tool rather than any bound on the image.

What is in the image

Dockerfile:63 sets WORKDIR /app, and Dockerfile:73 copies the whole agents/ tree in:

COPY --chown=1000:1000 agents /app/agents

That is 59 files, 55 of them under probes/, evaluations/, or packs/: the probe definitions, the board cases the lane is graded against, and the persisted graded replies from previous runs.

So the working directory of a process whose entire job is answering prompts contains the answers to its own tests.

What keeps it safe today, and what does not

The good news is real and was checked rather than assumed.

read_skill cannot reach it. skilltool.go builds a path-to-body map at Open and serves from memory. It never touches the filesystem at call time, and an unknown path returns a refusal listing the real ones. There is no path argument that reaches disk.
scratch_search cannot reach it. It is rooted at /scratch, a separate emptyDir. resolveUnder refuses absolute paths, refuses .. segments rather than normalizing them (the comment notes normalizing would write elsewhere while reporting the path asked for), and confine re-checks with EvalSymlinks against the nearest existing ancestor.
The lane mounts no source. sirens-dowel-values.yaml mounts three configMaps at /etc/* and one emptyDir at /scratch. Nothing mounts a repository.
cmd/, internal/, and docs/ never reach the runtime image. They stop at the build stage, so there is no Go source or docs tree in cwd either.

None of that is a property of the image. It is a property of the current tool set. The eval material is present and readable by the process; what is missing is a tool that takes a filesystem path. The next tool that does — a file reader, a repo browser, a debugging aid, an MCP server mounted at cwd — makes the board readable, and nothing in CI would notice, because nothing declares that cwd should not contain eval material.

An agent that can read its own eval cases produces evals that measure nothing, and the failure is silent: the scores go up.

What the runtime actually needs from `agents/`

Checked, so the fix does not break a lane.

No runtime consumer for probes/, evaluations/, or packs/. cmd/sirens-echo-eval is not among the binaries copied to /usr/local/bin, and the three greps for those directory names in internal/ and cmd/sirens-echo/ are all comments.
One real dependency: config.go:609 sets defaultDefinitionPath = "agents/echo/definition.yaml", the fallback when SIRENS_ECHO_DEFINITION is unset. All three deployed lanes set it explicitly to a configMap path under /etc/, so the fallback is unused in production, but it is a genuine reference and a blanket deletion would break an unset-env run.
agent/ (singular) is needed and should not be touched: SIRENS_ECHO_PHRASES points into it and the compose stage lands bundles there.

Fix shapes

Copy only the definitions. Replace the whole-tree copy with the definition.yaml files, keeping the fallback working and leaving the eval material out. Smallest change, closest to the current behaviour.
Drop agents/ from the runtime stage entirely and make the unset-SIRENS_ECHO_DEFINITION case fail loudly instead of falling back to a path that no longer ships. Arguably better, since an implicit definition is its own trap, but it changes a documented default.
Leave it and add a check that fails if a filesystem-reaching tool is registered while eval material is present in the image. Keeps the current layout and makes the hazard explicit, but it is the most machinery for the least benefit.

Shape 1 is the one I would build. Shape 2 is worth considering alongside it since the fallback is already unused everywhere it matters.

Not urgent

Nothing reaches this today and no lane is at risk before the 2026-08-19 stream. This is a Dockerfile change touching every lane's image, which is the wrong thing to land during the freeze, so it belongs after 2026-08-20.

Refs #1011, #929

Not currently exploitable. Filed so it stays a decision rather than becoming one by accident, because the thing that makes it safe today is the absence of a tool rather than any bound on the image. ## What is in the image `Dockerfile:63` sets `WORKDIR /app`, and `Dockerfile:73` copies the whole `agents/` tree in: ```dockerfile COPY --chown=1000:1000 agents /app/agents ``` That is **59 files, 55 of them under `probes/`, `evaluations/`, or `packs/`**: the probe definitions, the board cases the lane is graded against, and the persisted graded replies from previous runs. So the working directory of a process whose entire job is answering prompts contains the answers to its own tests. ## What keeps it safe today, and what does not The good news is real and was checked rather than assumed. * **`read_skill` cannot reach it.** `skilltool.go` builds a path-to-body map at `Open` and serves from memory. It never touches the filesystem at call time, and an unknown path returns a refusal listing the real ones. There is no path argument that reaches disk. * **`scratch_search` cannot reach it.** It is rooted at `/scratch`, a separate emptyDir. `resolveUnder` refuses absolute paths, **refuses `..` segments rather than normalizing them** (the comment notes normalizing would write elsewhere while reporting the path asked for), and `confine` re-checks with `EvalSymlinks` against the nearest existing ancestor. * **The lane mounts no source.** `sirens-dowel-values.yaml` mounts three configMaps at `/etc/*` and one emptyDir at `/scratch`. Nothing mounts a repository. * **`cmd/`, `internal/`, and `docs/` never reach the runtime image.** They stop at the build stage, so there is no Go source or docs tree in cwd either. **None of that is a property of the image.** It is a property of the current tool set. The eval material is present and readable by the process; what is missing is a tool that takes a filesystem path. The next tool that does — a file reader, a repo browser, a debugging aid, an MCP server mounted at cwd — makes the board readable, and nothing in CI would notice, because nothing declares that cwd should not contain eval material. An agent that can read its own eval cases produces evals that measure nothing, and the failure is silent: the scores go up. ## What the runtime actually needs from `agents/` Checked, so the fix does not break a lane. * **No runtime consumer for `probes/`, `evaluations/`, or `packs/`.** `cmd/sirens-echo-eval` is **not** among the binaries copied to `/usr/local/bin`, and the three greps for those directory names in `internal/` and `cmd/sirens-echo/` are all comments. * **One real dependency:** `config.go:609` sets `defaultDefinitionPath = "agents/echo/definition.yaml"`, the fallback when `SIRENS_ECHO_DEFINITION` is unset. All three deployed lanes set it explicitly to a configMap path under `/etc/`, so the fallback is unused in production, but it is a genuine reference and a blanket deletion would break an unset-env run. * `agent/` (singular) **is** needed and should not be touched: `SIRENS_ECHO_PHRASES` points into it and the compose stage lands bundles there. ## Fix shapes 1. **Copy only the definitions.** Replace the whole-tree copy with the `definition.yaml` files, keeping the fallback working and leaving the eval material out. Smallest change, closest to the current behaviour. 2. **Drop `agents/` from the runtime stage entirely** and make the unset-`SIRENS_ECHO_DEFINITION` case fail loudly instead of falling back to a path that no longer ships. Arguably better, since an implicit definition is its own trap, but it changes a documented default. 3. **Leave it and add a check** that fails if a filesystem-reaching tool is registered while eval material is present in the image. Keeps the current layout and makes the hazard explicit, but it is the most machinery for the least benefit. Shape 1 is the one I would build. Shape 2 is worth considering alongside it since the fallback is already unused everywhere it matters. ## Not urgent Nothing reaches this today and no lane is at risk before the 2026-08-19 stream. This is a Dockerfile change touching every lane's image, which is the wrong thing to land during the freeze, so it belongs after 2026-08-20. Refs #1011, #929

coilyco-ops added the

labels

2026-08-19 03:38:48 +00:00

coilyco-ops referenced this issue

2026-08-22 20:55:26 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops added this to the Headless engineer lane: the queue wait first milestone

2026-08-22 20:58:59 +00:00

coilyco-ops commented

2026-08-22 21:31:40 +00:00

Author

Member

Shape 1, decided. The freeze that held it has lifted, and the tree is smaller than the fix needs to be careful about.

Darren (director seat), 2026-08-22. This sits third in milestone 17, promoted above its priority/P2 label, so the engineer reaching it should find a decision rather than three options. I checked the facts before picking, since the issue's own argument is that what makes this safe is an absence rather than a bound.

Confirmed against the tree

Dockerfile still carries COPY --chown=1000:1000 agents /app/agents in the runtime stage, under WORKDIR /app.
agents/ is 59 files. 55 are under probes/, evaluations/, or packs/. The remaining four are agents/echo/definition.yaml, agents/deep/definition.yaml, and two rendered/prompt.txt snapshots.
sirens-echo-eval is not among the binaries copied into the runtime stage. Only sirens-echo, sirens-echo-compose, sirens-echo-access-check, and sirens-echo-temporal-mcp reach /usr/local/bin.
The two rendered/prompt.txt files have exactly one consumer, cmd/sirens-echo-prompt, and that binary reaches the compose stage rather than the runtime image. Nothing in the shipped image reads them.

So the runtime image's only genuine need from that tree is the two definition.yaml files, which is what shape 1 already said. Nothing I found argues against it.

The decision

Build shape 1. Replace the whole-tree copy with the two definition files. It keeps the unset-SIRENS_ECHO_DEFINITION fallback working, and it takes 57 of 59 files out of the working directory of a process whose job is answering prompts.

Shape 3 is out. A check that fires only when a filesystem-reaching tool is registered leaves the material in the image and makes the guard depend on correctly classifying every future tool. That is more machinery defending a layout nobody needs than the layout is worth.

Shape 2 is considered and declined for now, not deferred quietly. Making the unset case fail loudly is a real improvement to a real trap, and it changes a documented default in the same change that already touches every lane's image. One change, one risk. Every deployed lane sets SIRENS_ECHO_DEFINITION explicitly, so the fallback is unused where it matters and the loud failure buys nothing this week. Anyone who still wants it after shape 1 lands should say so on this issue and it becomes its own small change.

One thing to verify rather than assume while building

The rendered/prompt.txt finding above is mine and is the one part of this that shape 1 as written would silently drop. If a lane turns out to read a rendered snapshot at startup, that is a fourth file to keep, not a reason to keep the tree.

The freeze no longer applies

The issue says this belongs after 2026-08-20 because it is a Dockerfile change touching every lane's image. That date has passed, #929's revisit condition with it, and #1096 is returning the repo to the pull-request lane as I write this. Nothing is holding it.

## Shape 1, decided. The freeze that held it has lifted, and the tree is smaller than the fix needs to be careful about. **Darren (director seat), 2026-08-22.** This sits third in milestone 17, promoted above its `priority/P2` label, so the engineer reaching it should find a decision rather than three options. I checked the facts before picking, since the issue's own argument is that what makes this safe is an absence rather than a bound. ### Confirmed against the tree * `Dockerfile` still carries `COPY --chown=1000:1000 agents /app/agents` in the runtime stage, under `WORKDIR /app`. * `agents/` is **59 files**. 55 are under `probes/`, `evaluations/`, or `packs/`. The remaining four are `agents/echo/definition.yaml`, `agents/deep/definition.yaml`, and two `rendered/prompt.txt` snapshots. * `sirens-echo-eval` is **not** among the binaries copied into the runtime stage. Only `sirens-echo`, `sirens-echo-compose`, `sirens-echo-access-check`, and `sirens-echo-temporal-mcp` reach `/usr/local/bin`. * The two `rendered/prompt.txt` files have exactly one consumer, `cmd/sirens-echo-prompt`, and that binary reaches the compose stage rather than the runtime image. **Nothing in the shipped image reads them.** So the runtime image's only genuine need from that tree is the two `definition.yaml` files, which is what shape 1 already said. Nothing I found argues against it. ### The decision **Build shape 1.** Replace the whole-tree copy with the two definition files. It keeps the unset-`SIRENS_ECHO_DEFINITION` fallback working, and it takes 57 of 59 files out of the working directory of a process whose job is answering prompts. **Shape 3 is out.** A check that fires only when a filesystem-reaching tool is registered leaves the material in the image and makes the guard depend on correctly classifying every future tool. That is more machinery defending a layout nobody needs than the layout is worth. **Shape 2 is considered and declined for now**, not deferred quietly. Making the unset case fail loudly is a real improvement to a real trap, and it changes a documented default in the same change that already touches every lane's image. One change, one risk. Every deployed lane sets `SIRENS_ECHO_DEFINITION` explicitly, so the fallback is unused where it matters and the loud failure buys nothing this week. Anyone who still wants it after shape 1 lands should say so on this issue and it becomes its own small change. ### One thing to verify rather than assume while building The `rendered/prompt.txt` finding above is mine and is the one part of this that shape 1 as written would silently drop. If a lane turns out to read a rendered snapshot at startup, that is a fourth file to keep, not a reason to keep the tree. ### The freeze no longer applies The issue says this belongs after 2026-08-20 because it is a Dockerfile change touching every lane's image. That date has passed, #929's revisit condition with it, and #1096 is returning the repo to the pull-request lane as I write this. Nothing is holding it.

coilyco-ops referenced this issue

2026-08-22 21:46:08 +00:00

refactor(board)!: adopt challenge as the name for one graded question #1101

coilyco-ops referenced this issue from a commit

2026-08-22 22:20:34 +00:00

fix(image): ship the lane definitions, not the eval material

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-22 22:21:01 +00:00

fix(image): ship the lane definitions, not the eval material #1108

coilyco-ops referenced this issue

2026-08-22 22:22:22 +00:00

fix(image): ship the lane definitions, not the eval material #1108

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-22 22:33:50 +00:00

fix(image): ship the lane definitions, not the eval material #1108

coilyco-ops closed this issue

2026-08-22 22:34:43 +00:00

coilyco-ops referenced this issue

2026-08-22 22:35:57 +00:00

The image still ships the injection fixture, because #1012 exempted the one directory it is in #1112

coilyco-ops commented

2026-08-22 22:36:06 +00:00

Author

Member

Filed the remainder as #1112 rather than reopening this, and the reason matters for whoever reads both. This issue explicitly exempted agent/ singular, saying it "is needed and should not be touched", so #1108 did exactly what it was asked and did it well. The exemption is the part that was wrong: five files in that directory are fixtures whose only consumers are sirens-echo-eval and sirens-echo-policy-check, neither of which ships in the runtime image, and one of them is the injection payload set the agent is graded against. Reopening would have filed correct work as a redo.

Filed the remainder as #1112 rather than reopening this, and the reason matters for whoever reads both. **This issue explicitly exempted `agent/` singular**, saying it "is needed and should not be touched", so #1108 did exactly what it was asked and did it well. The exemption is the part that was wrong: five files in that directory are fixtures whose only consumers are `sirens-echo-eval` and `sirens-echo-policy-check`, neither of which ships in the runtime image, and one of them is the injection payload set the agent is graded against. Reopening would have filed correct work as a redo.

coilyco-ops referenced this issue

2026-08-22 23:13:57 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops referenced this issue

2026-08-22 23:16:42 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops referenced this issue

2026-08-23 00:45:41 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops referenced this issue

2026-08-23 01:14:40 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095