fix(image): ship the sirens-dowel skill root, which is crashlooping the lane

coilyco-ops commented

2026-08-18 19:15:16 +00:00

Member

The sirens-dowel lane is in CrashLoopBackOff right now and has been for at
least 90 minutes. 422 BackOff events, 22 restarts, on image d887bd8a, which
is current main. The stream is tomorrow.

The failure

The container starts and exits:

{"level":"ERROR","msg":"startup.agent.failed",
 "error":"walk skill root .agents/skills/sirens-dowel:
          lstat .agents/skills/sirens-dowel: no such file or directory"}

deploy#666 added .agents/skills/sirens-dowel to that lane's
local_skill_roots, and correctly: the root exists in this repository and
carries the lane's own SKILL.md plus site-work.md and dowel-provenance.md.

It is not in the image. The final stage copies sirens-echo-community,
sirens-echo-knowledge, coilyco-general, and coilyco-org. sirens-dowel
ships in git and never reaches /app. LoadSkillpack walks what the definition
names, lstat fails, and NewAgent returns an error before the process can
serve anything.

The README calls local_skill_roots the one field where divergence from the
image is a bug rather than a preference, and frames the risk as a quiet
omission. This is the loud half of the same defect: naming a root the image does
not carry is a startup crash.

The change

One COPY in the final stage, which is what a definition resolves against, and
one in the build stage beside the other roots so the policy check sees the same
tree.

If this needs to be faster than a build

Removing .agents/skills/sirens-dowel from local_skill_roots in
sirens-dowel-definition.yml and rolling restores the lane immediately with no
rebuild. That is the rollback, not the fix: the lane then runs without its own
skill root, which is what deploy#666 set out to give it, and what #967 is adding
content to. Merging this and letting the publish plus reconciler run is the fix.

Also wedged in that namespace, separately

Not addressed here, both visible in the same event stream:

quire-68db977c49-cmczd is ImagePullBackOff, 723 events, on image tag
quire:REPLACE_WITH_IMAGE_SHA — a literal placeholder.
ExternalSecret sirens-dowel-quire fails 32 times on
/sirens-dowel/quire-api-token, Secret does not exist.

Those are coilyco-bridge/deploy#664's wiring, and coilyco-bridge/deploy#667 and
#668 already track the token and the revert. They do not block the harness,
which is its own Deployment, but they are noise in the same namespace on demo
eve.

No test here, deliberately

The invariant worth pinning is that a definition names only roots the image
carries, and this repository cannot see deploy's definitions. Asserting the
shipped set against a list in a test would restate the Dockerfile rather than
check it, and two roots are deliberately unshipped (ops-social-discord,
repo-sirens-echo), so "every root ships" is false.

The real guard is a skill-root check deploy's CI can run against the ConfigMap
the way it already runs sirens-echo-access-check against the access policy.
Worth its own change, not this one.

go build ./... and pre-commit run --files Dockerfile pass. The image build
itself is what this PR's own image-build job proves.

**The `sirens-dowel` lane is in CrashLoopBackOff right now and has been for at least 90 minutes.** 422 BackOff events, 22 restarts, on image `d887bd8a`, which is current main. The stream is tomorrow. ## The failure The container starts and exits: ``` {"level":"ERROR","msg":"startup.agent.failed", "error":"walk skill root .agents/skills/sirens-dowel: lstat .agents/skills/sirens-dowel: no such file or directory"} ``` deploy#666 added `.agents/skills/sirens-dowel` to that lane's `local_skill_roots`, and correctly: the root exists in this repository and carries the lane's own `SKILL.md` plus `site-work.md` and `dowel-provenance.md`. **It is not in the image.** The final stage copies `sirens-echo-community`, `sirens-echo-knowledge`, `coilyco-general`, and `coilyco-org`. `sirens-dowel` ships in git and never reaches `/app`. `LoadSkillpack` walks what the definition names, `lstat` fails, and `NewAgent` returns an error before the process can serve anything. The README calls `local_skill_roots` the one field where divergence from the image is a bug rather than a preference, and frames the risk as a **quiet** omission. This is the loud half of the same defect: naming a root the image does not carry is a startup crash. ## The change One `COPY` in the final stage, which is what a definition resolves against, and one in the build stage beside the other roots so the policy check sees the same tree. ## If this needs to be faster than a build Removing `.agents/skills/sirens-dowel` from `local_skill_roots` in `sirens-dowel-definition.yml` and rolling restores the lane immediately with no rebuild. That is the rollback, not the fix: the lane then runs without its own skill root, which is what deploy#666 set out to give it, and what #967 is adding content to. Merging this and letting the publish plus reconciler run is the fix. ## Also wedged in that namespace, separately Not addressed here, both visible in the same event stream: * `quire-68db977c49-cmczd` is `ImagePullBackOff`, 723 events, on image tag `quire:REPLACE_WITH_IMAGE_SHA` — a literal placeholder. * ExternalSecret `sirens-dowel-quire` fails 32 times on `/sirens-dowel/quire-api-token`, `Secret does not exist`. Those are coilyco-bridge/deploy#664's wiring, and coilyco-bridge/deploy#667 and #668 already track the token and the revert. They do not block the harness, which is its own Deployment, but they are noise in the same namespace on demo eve. ## No test here, deliberately The invariant worth pinning is that a definition names only roots the image carries, and this repository cannot see deploy's definitions. Asserting the shipped set against a list in a test would restate the Dockerfile rather than check it, and two roots are deliberately unshipped (`ops-social-discord`, `repo-sirens-echo`), so "every root ships" is false. The real guard is a skill-root check deploy's CI can run against the ConfigMap the way it already runs `sirens-echo-access-check` against the access policy. Worth its own change, not this one. `go build ./...` and `pre-commit run --files Dockerfile` pass. The image build itself is what this PR's own `image-build` job proves.

coilyco-ops added 1 commit

2026-08-18 19:15:16 +00:00

fix(image): ship the sirens-dowel skill root

ci / image-build (pull_request) Successful in 34s

Details

ci / test (pull_request) Successful in 1m0s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

535b5945f5

The sirens-dowel lane is in CrashLoopBackOff and has been for at least 90
minutes. 422 BackOff events, 22 restarts, on image d887bd8a which is current
main. The container starts and exits:

    startup.agent.failed
    walk skill root .agents/skills/sirens-dowel:
      lstat .agents/skills/sirens-dowel: no such file or directory

deploy#666 added `.agents/skills/sirens-dowel` to that lane's
`local_skill_roots`, correctly: the root exists in this repository and carries
the lane's own SKILL.md plus site-work.md and dowel-provenance.md.

It is not in the image. The final stage copies four roots and this is not one of
them, so the tree ships in git and never reaches /app. LoadSkillpack walks what
the definition names, lstat fails, and agent construction returns an error
before the process can serve anything.

The README calls local_skill_roots the one field where divergence from the image
is a bug rather than a preference, and reads the risk as a quiet omission. This
is the loud half of the same defect: naming a root the image does not carry is a
startup crash, not a gap.

Added to the build stage beside the other roots so the policy check sees the
same tree, and to the final stage, which is what a definition actually resolves
against.

NO TEST HERE, DELIBERATELY. The invariant worth pinning is that a definition
names only roots the image carries, and this repository cannot see deploy's
definitions. Asserting the shipped set against a list in a test would restate
the Dockerfile rather than check it, and two roots are deliberately unshipped
(ops-social-discord, repo-sirens-echo) so "every root ships" is false. The real
guard is a skill-root check deploy's CI can run against a ConfigMap the way it
already runs sirens-echo-access-check against the access policy. Worth its own
change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-18 19:15:16 +00:00

coilysiren added 1 commit

2026-08-18 19:16:59 +00:00

Update Dockerfile

ci / image-build (pull_request) Failing after 27s

Details

ci / test (pull_request) Failing after 43s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

e5791d90a3

coilyco-ops added 1 commit

2026-08-18 19:19:54 +00:00

fix(image): copy the skills tree, not a wildcard over its roots

ci / image-build (pull_request) Successful in 33s

Details

ci / test (pull_request) Failing after 43s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

980e4681bb

Keeps the shape from the previous commit, which is the right one: no enumerated
list of roots, so the image cannot drift from what a definition names. Fixes how
the wildcard expands.

`COPY .agents/skills/* ./.agents/skills/` matches each root separately, and for
a directory source Docker copies the contents rather than the directory. So
every root's files merge into one directory and the root names disappear. Seven
SKILL.md files land on the same path and six of them lose.

That is worse than the bug it was fixing. `lstat .agents/skills/sirens-dowel`
still fails, so this lane keeps crashlooping, and coilyco-general and coilyco-org
stop resolving too, which takes Echo and Deep down with it.

`COPY .agents/skills ./.agents/skills` names the tree instead. The source is one
directory, its contents are the roots, and the roots keep their names.

THE GUARD IS THE POINT, not the COPY. A flattened tree is invisible at build
time and only shows up as startup.agent.failed after a rollout, which is how
this reached a running pod on demo eve. The final stage now walks the roots and
fails the build if one lost its SKILL.md, or if a SKILL.md landed at the skills
root, which is what flattening looks like. It names no root, so it stays true
when a root is added.

Not verified locally: no daemon on this host. The assertion runs in this PR's
own image-build job, so a wrong answer fails there rather than in a pod.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops added 1 commit

2026-08-18 19:22:50 +00:00

test(image): mirror the skills tree rather than four roots

ci / image-build (pull_request) Successful in 31s

Details

ci / test (pull_request) Successful in 1m1s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

08f6aea408

TestImageContextPathsStillMirrorTheDockerfile caught the other half of the COPY
change. imageContextPaths is the hand-maintained mirror of the build stage's
COPY set, and it still named the four roots individually while the Dockerfile
now names the tree, so none of the four matched a COPY source.

Collapsing them to `.agents/skills` restores the mirror and removes the same
drift the Dockerfile change removed. The list no longer enumerates roots either,
so adding one needs no edit here.

Worth noting which guard caught what. This test covers the paired list. The
build stage's own policy check is what caught the flattening, in seconds, by
failing on Echo's definition. Neither could catch the original outage, because
the root that went missing was named by a definition in deploy's ConfigMap and
no check in this repository reads that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops referenced this pull request

2026-08-18 19:23:48 +00:00

Nothing checks a deploy-owned definition's local_skill_roots against the image #973

coilysiren merged commit 1e82719e93 into main

2026-08-18 19:24:48 +00:00

coilysiren referenced this pull request from a commit

2026-08-18 19:24:49 +00:00

Merge pull request 'fix(image): ship the sirens-dowel skill root, which is crashlooping the lane' (#972) from fix/ship-the-dowel-skill-root into main

coilyco-ops referenced this pull request

2026-08-18 19:41:00 +00:00

test(prompt): derive the shipped profile set from disk #974

coilyco-ops referenced this pull request from a commit