Nothing checks a deploy-owned definition's local_skill_roots against the image

coilyco-ops commented

2026-08-18 19:23:48 +00:00

Member

The sirens-dowel lane crashlooped for roughly 90 minutes on 2026-08-18, the day before the Temporal stream, on this:

{"level":"ERROR","msg":"startup.agent.failed",
 "error":"walk skill root .agents/skills/sirens-dowel:
          lstat .agents/skills/sirens-dowel: no such file or directory"}

coilyco-bridge/deploy#666 added .agents/skills/sirens-dowel to that lane's local_skill_roots. The root exists in this repository. The Dockerfile did not copy it, so it shipped in git and never reached /app. Fixed in #972.

The gap, which is not the Dockerfile

Two guards in this repository already cover a Dockerfile that contradicts itself, and both worked during that fix:

sirens-echo-policy-check in the build stage caught a flattening COPY .agents/skills/* in seconds, by failing on agents/echo/definition.yaml.
TestImageContextPathsStillMirrorTheDockerfile caught the paired imageContextPaths list going stale against the new COPY set.

Neither could catch the outage, and no change to either would. They read in-image definitions only, agents/echo/ and agents/deep/. The definition that named the missing root lives in deploy's ConfigMap, and nothing here reads that.

So the invariant the README states, that local_skill_roots is the one field where divergence from the image is a bug rather than a preference, is unenforced for exactly the lanes whose definitions deploy owns. Which, since deploy#577 moved definitions out, is all of them.

Shape that already exists

sirens-echo-access-check is built into the released image specifically so deploy's CI can run it against the access-policy ConfigMap before applying (#628). The same shape fits here: a verb that takes a definition and reports any local_skill_roots entry absent from /app/.agents/skills.

Run from the image, it compares the definition against the exact tree that image carries, which is the comparison that matters and the one neither side can currently make alone. deploy invokes it the way it already invokes the access check, so authoring stays here and rollout stays there.

Why the failure mode deserves a check rather than care

The README frames this risk as a quiet omission, a root added upstream that appears in no ConfigMap and is loaded by nobody. The loud half is worse and is what happened: a ConfigMap naming a root the image lacks takes the process down at startup, after a rollout, with no signal at edit time. deploy#666 was correct in intent, passed its own review, and still produced an outage, because the fact it depended on was in another repository.

The `sirens-dowel` lane crashlooped for roughly 90 minutes on 2026-08-18, the day before the Temporal stream, on this: ``` {"level":"ERROR","msg":"startup.agent.failed", "error":"walk skill root .agents/skills/sirens-dowel: lstat .agents/skills/sirens-dowel: no such file or directory"} ``` coilyco-bridge/deploy#666 added `.agents/skills/sirens-dowel` to that lane's `local_skill_roots`. The root exists in this repository. The Dockerfile did not copy it, so it shipped in git and never reached `/app`. Fixed in #972. ## The gap, which is not the Dockerfile Two guards in this repository already cover a Dockerfile that contradicts itself, and both worked during that fix: * `sirens-echo-policy-check` in the build stage caught a flattening `COPY .agents/skills/*` in seconds, by failing on `agents/echo/definition.yaml`. * `TestImageContextPathsStillMirrorTheDockerfile` caught the paired `imageContextPaths` list going stale against the new COPY set. Neither could catch the outage, and no change to either would. **They read in-image definitions only**, `agents/echo/` and `agents/deep/`. The definition that named the missing root lives in deploy's ConfigMap, and nothing here reads that. So the invariant the README states, that `local_skill_roots` is the one field where divergence from the image is a bug rather than a preference, is unenforced for exactly the lanes whose definitions deploy owns. Which, since deploy#577 moved definitions out, is all of them. ## Shape that already exists `sirens-echo-access-check` is built into the released image specifically so deploy's CI can run it against the access-policy ConfigMap before applying (#628). The same shape fits here: a verb that takes a definition and reports any `local_skill_roots` entry absent from `/app/.agents/skills`. Run from the image, it compares the definition against the exact tree that image carries, which is the comparison that matters and the one neither side can currently make alone. deploy invokes it the way it already invokes the access check, so authoring stays here and rollout stays there. ## Why the failure mode deserves a check rather than care The README frames this risk as a quiet omission, a root added upstream that appears in no ConfigMap and is loaded by nobody. The loud half is worse and is what happened: a ConfigMap naming a root the image lacks takes the process down at startup, after a rollout, with no signal at edit time. deploy#666 was correct in intent, passed its own review, and still produced an outage, because the fact it depended on was in another repository. Related: #972, coilyco-bridge/deploy#666, coilyco-bridge/deploy#577, #628.

coilyco-ops referenced this issue

2026-08-19 01:28:23 +00:00

The temporal and moxn skills from #999 are in no lane's local_skill_roots, so nothing loads them #1006

coilyco-ops added the

labels

2026-08-19 03:38:49 +00:00

coilyco-ops referenced this issue

2026-08-19 04:48:41 +00:00

Dowel's sources still describe a Moxn knowledge base it no longer has, including a live eval boundary #1054

coilyco-ops referenced this issue

2026-08-22 20:55:26 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops added this to the Headless engineer lane: the queue wait first milestone

2026-08-22 20:59:07 +00:00

coilyco-ops referenced this issue from a commit

2026-08-22 22:46:38 +00:00

feat(image): a check that reads a definition against the tree it ships with

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-22 22:46:53 +00:00

feat(image): a check that reads a definition against the tree it ships with #1116

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-22 22:58:01 +00:00

feat(image): a check that reads a definition against the tree it ships with #1116

coilyco-ops closed this issue

2026-08-22 22:58:01 +00:00

coilyco-ops commented

2026-08-22 22:58:36 +00:00

Author

Member

Closed on the capability rather than on the check, and the distinction is worth having in the record here rather than only in #1116's body.

sirens-echo-definition-check now exists and ships in the runtime image, verified from the built image against both a good definition and one naming a root the image lacks. Nothing invokes it yet. Until coilyco-bridge/deploy adds the invocation to its CI beside the access check, a definition naming a missing skill root still crashloops the pod exactly as this issue describes, because the tool that would have caught it is sitting in the image unrun.

That rollout is deploy's rather than this lane's, correctly, and it is tracked at coilyco-bridge/deploy#777 - coilyco-bridge/deploy#777.

So: this issue's ask is built, and its failure mode is still live. Anyone reading this as done should read #777 before relying on it.

Closed on the **capability** rather than on the **check**, and the distinction is worth having in the record here rather than only in #1116's body. `sirens-echo-definition-check` now exists and ships in the runtime image, verified from the built image against both a good definition and one naming a root the image lacks. **Nothing invokes it yet.** Until `coilyco-bridge/deploy` adds the invocation to its CI beside the access check, a definition naming a missing skill root still crashloops the pod exactly as this issue describes, because the tool that would have caught it is sitting in the image unrun. That rollout is deploy's rather than this lane's, correctly, and it is tracked at `coilyco-bridge/deploy#777` - https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/777. So: this issue's ask is built, and its failure mode is still live. Anyone reading this as done should read #777 before relying on it.

coilyco-ops referenced this issue

2026-08-22 23:13:57 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops referenced this issue

2026-08-22 23:16:42 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops referenced this issue

2026-08-23 00:45:41 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops referenced this issue

2026-08-23 01:14:40 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095