coilyco-flight-deck/agentic-os

Watch

CI-in-dev-base assumes runners re-pull :release, and they do not #988

New issue

Open

opened 2026-08-10 01:09:32 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-10 01:09:32 +00:00

Member

docs/ci-in-dev-base.md has every app CI run in the moving :release alias. That only works if the runner re-resolves the alias. It currently does not, so a runner can serve a months-old image indefinitely and CI silently validates the wrong world. This is the failure the convention was written to prevent, reappearing one layer down.

How it surfaced

agentic-os#986 added wasm-pack and a pinned binaryen to the Rust payload. Verified present in the published image, both architectures:

$ docker run --platform linux/amd64 ...agentic-os:release which wasm-pack wasm-opt
/usr/local/cargo/bin/wasm-pack
/usr/local/bin/wasm-opt
wasm-pack 0.15.0

galaxy-gen#76 then ran ward exec ci-setup inside container: ...agentic-os:release on the docker runner and got:

scripts/ward-command.sh: line 5: wasm-pack: command not found
exit status 127

The same commit, same verbs, same image, run locally against a freshly pulled :release passes end to end: ci-setup, lint-rust, 91/91 tests, check-js. The only difference is which copy of the alias the daemon holds.

Why it stayed invisible

Nothing before this needed a newly added tool. A stale image with the same ward, node, and cargo behaves identically to a current one, so every consumer of the convention has had this exposure and none of them could have noticed. aos's own ci / gate is in the same position today.

Fix

No pull policy is set on any runner. grep -i pull deploy/forgejo-runner*.yml in infrastructure returns nothing.

Setting force_pull: true on the runner container config is the durable fix and is cheaper than it sounds: it re-checks the manifest and pulls only changed layers, so an unchanged image costs one HEAD request rather than a full image transfer. That belongs in infrastructure as the rollout, not here, per the authoring-vs-rollout law. aos owns the convention and the doc that currently asserts something untrue about it.

Interim relief is a rollout restart of the affected runner StatefulSet, since the pods wipe /var/lib/docker on start. The nightly recycle CronJob at 09:15 UTC already does this for forgejo-runner-bridge, -flight-deck, -build-flight-deck, and -gaming, which means the staleness window is bounded at 24 hours rather than unbounded, but 24 hours is still long enough to make a green CI run meaningless after an image change.

Also worth deciding

docs/ci-in-dev-base.md should state the freshness requirement explicitly. As written it reads as though pointing at :release is sufficient, and the whole parity argument depends on a property the rollout does not currently guarantee.

Refs

agentic-os#986 - added the tools that exposed this
agentic-os#328 - the CI-in-dev-base convention
galaxy-gen#76 - the PR that hit it

`docs/ci-in-dev-base.md` has every app CI run in the moving `:release` alias. That only works if the runner re-resolves the alias. It currently does not, so a runner can serve a months-old image indefinitely and CI silently validates the wrong world. This is the failure the convention was written to prevent, reappearing one layer down. ## How it surfaced agentic-os#986 added `wasm-pack` and a pinned `binaryen` to the Rust payload. Verified present in the published image, both architectures: ``` $ docker run --platform linux/amd64 ...agentic-os:release which wasm-pack wasm-opt /usr/local/cargo/bin/wasm-pack /usr/local/bin/wasm-opt wasm-pack 0.15.0 ``` galaxy-gen#76 then ran `ward exec ci-setup` inside `container: ...agentic-os:release` on the `docker` runner and got: ``` scripts/ward-command.sh: line 5: wasm-pack: command not found exit status 127 ``` The same commit, same verbs, same image, run locally against a freshly pulled `:release` passes end to end: `ci-setup`, `lint-rust`, 91/91 tests, `check-js`. The only difference is which copy of the alias the daemon holds. ## Why it stayed invisible Nothing before this needed a newly added tool. A stale image with the same `ward`, `node`, and `cargo` behaves identically to a current one, so every consumer of the convention has had this exposure and none of them could have noticed. aos's own `ci / gate` is in the same position today. ## Fix No pull policy is set on any runner. `grep -i pull deploy/forgejo-runner*.yml` in infrastructure returns nothing. Setting `force_pull: true` on the runner container config is the durable fix and is cheaper than it sounds: it re-checks the manifest and pulls only changed layers, so an unchanged image costs one HEAD request rather than a full image transfer. That belongs in infrastructure as the rollout, not here, per the authoring-vs-rollout law. aos owns the convention and the doc that currently asserts something untrue about it. Interim relief is a `rollout restart` of the affected runner StatefulSet, since the pods wipe `/var/lib/docker` on start. The nightly recycle CronJob at 09:15 UTC already does this for `forgejo-runner-bridge`, `-flight-deck`, `-build-flight-deck`, and `-gaming`, which means the staleness window is bounded at 24 hours rather than unbounded, but 24 hours is still long enough to make a green CI run meaningless after an image change. ## Also worth deciding `docs/ci-in-dev-base.md` should state the freshness requirement explicitly. As written it reads as though pointing at `:release` is sufficient, and the whole parity argument depends on a property the rollout does not currently guarantee. ## Refs * agentic-os#986 - added the tools that exposed this * agentic-os#328 - the CI-in-dev-base convention * galaxy-gen#76 - the PR that hit it

coilyco-ops commented

2026-08-10 01:17:20 +00:00

Author

Member

Correcting the runner identification in the issue body. The mechanism is confirmed but I named the wrong cluster.

The `docker` label lives on ser8, not kai-server

grep -rn -- "--labels" deploy/ in infrastructure gives the real map:

deploy/forgejo-runners-ser8/general-gaming/kustomization.yaml - --labels "docker:docker://node:20-bookworm"
same for general-flight-deck and general-bridge

The kai-server StatefulSets I originally pointed at (forgejo-runner-gaming, -flight-deck, -bridge) are all 0/0, scaled to zero. Every live runner pod in kai-server's forgejo namespace is a deploy:host or tap-writer:host scoped runner. forgejo-runner-deploy-galaxy-gen advertises only deploy:host, so it never serves runs-on: docker.

A rollout restart of statefulset/forgejo-runner-gaming on kai-server therefore reports partitioned roll out complete: 0 new pods have been updated and does nothing. The correct target is statefulset/forgejo-runner-ser8-gaming in namespace forgejo on ser8.

The no-pull behaviour, from the job log

Run 133's "Set up job" step shows the container coming up with no pull anywhere in the sequence:

🚀  Start image=forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:release
  🐳  docker create image=...agentic-os:release platform=linux/amd64 ...
  🐳  docker run    image=...agentic-os:release platform=linux/amd64 ...

docker create straight to docker run. act_runner never re-resolves the alias, which is container.force_pull: false. No pull policy is set in the ser8 runner config either.

Staleness is bounded, but by ser8's own recycle

deploy/forgejo-runners-ser8/recycle/kustomization.yaml restarts forgejo-runner-ser8-gaming, -flight-deck, -canary-flight-deck, and -bridge on the shared 15 9 * * * schedule. So the window is 24 hours, not unbounded, which is what I said originally, but it is ser8's recycle doing it rather than the kai-server one I was reading at the time.

24 hours is still long enough that a green CI run immediately after an image change means nothing, which is the substance of this issue and is unchanged.

Fix location

force_pull: true belongs in the ser8 runner config under deploy/forgejo-runners-ser8/, not the kai-server manifests I referenced. Everything else in the issue body stands.

Correcting the runner identification in the issue body. The mechanism is confirmed but I named the wrong cluster. ## The `docker` label lives on ser8, not kai-server `grep -rn -- "--labels" deploy/` in infrastructure gives the real map: * `deploy/forgejo-runners-ser8/general-gaming/kustomization.yaml` - `--labels "docker:docker://node:20-bookworm"` * same for `general-flight-deck` and `general-bridge` The kai-server StatefulSets I originally pointed at (`forgejo-runner-gaming`, `-flight-deck`, `-bridge`) are all **0/0**, scaled to zero. Every live runner pod in kai-server's `forgejo` namespace is a `deploy:host` or `tap-writer:host` scoped runner. `forgejo-runner-deploy-galaxy-gen` advertises only `deploy:host`, so it never serves `runs-on: docker`. A `rollout restart` of `statefulset/forgejo-runner-gaming` on kai-server therefore reports `partitioned roll out complete: 0 new pods have been updated` and does nothing. The correct target is `statefulset/forgejo-runner-ser8-gaming` in namespace `forgejo` on **ser8**. ## The no-pull behaviour, from the job log Run 133's "Set up job" step shows the container coming up with no pull anywhere in the sequence: ``` 🚀 Start image=forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:release 🐳 docker create image=...agentic-os:release platform=linux/amd64 ... 🐳 docker run image=...agentic-os:release platform=linux/amd64 ... ``` `docker create` straight to `docker run`. act_runner never re-resolves the alias, which is `container.force_pull: false`. No pull policy is set in the ser8 runner config either. ## Staleness is bounded, but by ser8's own recycle `deploy/forgejo-runners-ser8/recycle/kustomization.yaml` restarts `forgejo-runner-ser8-gaming`, `-flight-deck`, `-canary-flight-deck`, and `-bridge` on the shared `15 9 * * *` schedule. So the window is 24 hours, not unbounded, which is what I said originally, but it is ser8's recycle doing it rather than the kai-server one I was reading at the time. 24 hours is still long enough that a green CI run immediately after an image change means nothing, which is the substance of this issue and is unchanged. ## Fix location `force_pull: true` belongs in the ser8 runner config under `deploy/forgejo-runners-ser8/`, not the kai-server manifests I referenced. Everything else in the issue body stands.

coilyco-ops added the

labels

2026-08-17 02:35:47 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#988

No description provided.

Rows
Columns