Role snapshot drift is only caught at Dockerfile step 26, and the message names a verb that cannot run #788

New issue

Closed

opened 2026-08-14 22:59:21 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-14 22:59:21 +00:00

Member

A branch cut before a compose change fails image-build after roughly twenty seconds of building, at step 26 of 40:

Step 26/40 : RUN sirens-echo-prompt --bundles /out/bundles --check
a role's selection changed: agent/rendered/roles/ai.bundle.txt, ... qa.bundle.txt
run `ward exec role-snapshot` against baked bundles and commit the result

The tracked agent/rendered/roles/*.bundle.txt no longer match the bundles baked into the image. db90cfc changed the composed roles, so every branch based before it drifts until it merges main.

This looked like infrastructure flakiness and is not

Every image-build failure on #771, #777 and #779 was this, deterministically. It reads as flaky only because the cure is a merge rather than a retry:

#777 failed at 06:53 and 07:42, then passed at 07:50, immediately after 72d6f58 merged main in.
#779 was based on fa72b31, before the compose change, and never passed.
#786 carries the identical diff rebuilt on current main and passed first try.

I spent real time chasing a dind daemon problem that does not exist. The daemon answered in both failures I read.

The remedy the message gives does not work

ward exec role-snapshot and role-snapshot-check both read agent/bundles, which is untracked. git ls-files agent/bundles returns nothing, and on a clean checkout of main the check verb fails for an unrelated reason:

$ ward exec role-snapshot-check
read bundle directory: open agent/bundles: no such file or directory
exit status 1

The bundles exist only after stage-compose-sources.sh bakes them against the AOS catalog at /tmp/aos-catalog, which is staged inside the image build. So the check cannot simply move into the test job, and the instruction printed on failure sends a reader to a command that dies before it can help.

Worth deciding

Two directions, not exclusive:

Stage the bundles in the test job so drift fails in seconds next to the other gates, rather than after a two thirds complete image build.
Failing that, say what the reader can act on. "This branch is behind main, merge it" is the true remedy in every instance so far, and is not what the message says.

The second is cheap and removes the misdirection even if the first never happens.

A branch cut before a compose change fails `image-build` after roughly twenty seconds of building, at step 26 of 40: ``` Step 26/40 : RUN sirens-echo-prompt --bundles /out/bundles --check a role's selection changed: agent/rendered/roles/ai.bundle.txt, ... qa.bundle.txt run `ward exec role-snapshot` against baked bundles and commit the result ``` The tracked `agent/rendered/roles/*.bundle.txt` no longer match the bundles baked into the image. `db90cfc` changed the composed roles, so every branch based before it drifts until it merges `main`. ## This looked like infrastructure flakiness and is not Every `image-build` failure on #771, #777 and #779 was this, deterministically. It reads as flaky only because the cure is a merge rather than a retry: - #777 failed at 06:53 and 07:42, then passed at 07:50, immediately after `72d6f58` merged `main` in. - #779 was based on `fa72b31`, before the compose change, and never passed. - #786 carries the identical diff rebuilt on current `main` and passed first try. I spent real time chasing a dind daemon problem that does not exist. The daemon answered in both failures I read. ## The remedy the message gives does not work `ward exec role-snapshot` and `role-snapshot-check` both read `agent/bundles`, which is untracked. `git ls-files agent/bundles` returns nothing, and on a clean checkout of `main` the check verb fails for an unrelated reason: ``` $ ward exec role-snapshot-check read bundle directory: open agent/bundles: no such file or directory exit status 1 ``` The bundles exist only after `stage-compose-sources.sh` bakes them against the AOS catalog at `/tmp/aos-catalog`, which is staged inside the image build. So the check cannot simply move into the `test` job, and the instruction printed on failure sends a reader to a command that dies before it can help. ## Worth deciding Two directions, not exclusive: - Stage the bundles in the `test` job so drift fails in seconds next to the other gates, rather than after a two thirds complete image build. - Failing that, say what the reader can act on. "This branch is behind `main`, merge it" is the true remedy in every instance so far, and is not what the message says. The second is cheap and removes the misdirection even if the first never happens.

coilyco-ops added the

priority

autonomy

headless

labels

2026-08-15 15:17:26 +00:00

coilyco-ops referenced this issue from a commit

2026-08-15 15:28:49 +00:00

fix(ci): catch role snapshot drift beside the other gates, not at step 26

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-15 15:29:27 +00:00

Catch role snapshot drift beside the other gates, not at step 26 #799

coilyco-ops referenced this issue

2026-08-15 15:30:04 +00:00

ward exec compose-bundles leaves a tree that pre-commit reads as this repository's own skills #800

coilyco-ops commented

2026-08-15 15:30:26 +00:00

Author

Member

Both directions taken, in #799.

The gate moved. The test job now runs ward exec role-drift-check, which clones the catalogue at AOS_CATALOG_REF and bakes the same bundles the image build bakes, then runs the same sirens-echo-prompt --check over them. Verified locally through both paths, an AOS_CATALOG checkout and the clone fallback, and they bake identical selections. Bake plus check is ~3s with a warm Go cache.

The Dockerfile check at step 26 stays as the backstop over the bundles that actually ship.

The message now names something that runs. The merge goes first, since that was the true remedy in every instance this issue traced:

a role's selection changed: agent/rendered/roles/qa.bundle.txt
  if this branch predates a composed-sources change, merge main first:
  the record it wants is already committed there
  if the change is yours, rebake and record it:
    ward exec compose-bundles
    ward exec role-snapshot

compose-bundles is named because it is the step that creates the directory role-snapshot reads, which is the omission that sent you to a command that dies. A missing bundle directory now says which verb creates it rather than open agent/bundles: no such file or directory.

One thing the issue did not predict. The new verb bakes to a scratch directory rather than agent/bundles, because a baked bundle is a tree of skill files and documentation-layout plus dead-cross-links walk the filesystem, so the later pre-commit step reads them as this repository's own and produces 75 layout violations. That is pre-existing for ward exec compose-bundles and is now #800.

Both directions taken, in #799. **The gate moved.** The `test` job now runs `ward exec role-drift-check`, which clones the catalogue at `AOS_CATALOG_REF` and bakes the same bundles the image build bakes, then runs the same `sirens-echo-prompt --check` over them. Verified locally through both paths, an `AOS_CATALOG` checkout and the clone fallback, and they bake identical selections. Bake plus check is ~3s with a warm Go cache. The Dockerfile check at step 26 stays as the backstop over the bundles that actually ship. **The message now names something that runs.** The merge goes first, since that was the true remedy in every instance this issue traced: ``` a role's selection changed: agent/rendered/roles/qa.bundle.txt if this branch predates a composed-sources change, merge main first: the record it wants is already committed there if the change is yours, rebake and record it: ward exec compose-bundles ward exec role-snapshot ``` `compose-bundles` is named because it is the step that creates the directory `role-snapshot` reads, which is the omission that sent you to a command that dies. A missing bundle directory now says which verb creates it rather than `open agent/bundles: no such file or directory`. **One thing the issue did not predict.** The new verb bakes to a scratch directory rather than `agent/bundles`, because a baked bundle is a tree of skill files and `documentation-layout` plus `dead-cross-links` walk the filesystem, so the later `pre-commit` step reads them as this repository's own and produces 75 layout violations. That is pre-existing for `ward exec compose-bundles` and is now #800.