publish-dev-base red on main: stale WARD_VERSION, swagger-lock vs ward doctor, dep-bump gate deadlock

coilyco-ops commented

2026-07-10 18:49:44 +00:00

Owner

publish-dev-base is red on every main push, and the dep-bump loop that should prevent it is gate-blocked

Found while landing aos#450 (codex pin bump). The publish-image path in release.yml has been failing on main (runs 986/jobs/2 at d5079cc, and the push before it), so landed Dockerfile bumps - including aos#450's CODEX_VERSION 0.144.1 - never reach the published :latest. Three stacked causes, in dependency order:

stale ward pin - docker/dev-base/core/Dockerfile pins ARG WARD_VERSION=0.567.0 while ward is at v0.596.0. The in-image CLIGUARD_NO_SANDBOX=1 ward doctor gate fails at 0.567.0 because that ward predates the split .ward bundle layout (it wants ward-kdl.defaults.kdl / ward-kdl.fleet.kdl / ward-kdl.forgejo.guardfile.kdl).
swagger lock vs ward doctor - bumping WARD_VERSION alone cannot go green either. aos#435 removed forgejo.swagger.lock.json from the repo and from the ward-specs bundle tarball, but every current ward release still hard-requires it in a bundle config source (bundleForgejoSpecLockPath in cmd/ward/configsource.go, verified failing on v0.584.0 and a source-built v0.596.0, and still present at ward main HEAD). Doctor fails ops bundle: read spec lock. Either ward drops or soft-fails the lock requirement for bundles (then bump WARD_VERSION here), or the bundle restores the lock. Needs a ward-side decision - this half likely wants a ward issue.
dep-bump is deadlocked by its own gate - scripts/dep-bump.py apply rewrites only Dockerfile ARG lines, while tests/test_dev_base_image.py hard-pins every managed ARG default as a string literal (test_split_tiers_keep_their_managed_arg_defaults, plus ARG WARD_VERSION=0.567.0 in test_core_tier_keeps_the_hidden_ward_builder_stage). The daily auto-bump commit therefore always fails the pytest gate it runs before pushing, and every pin silently drifts - exactly the incident class the auto-bump was built to close (docs/dev-base-auto-bump.md, agentic-os#272). Fix by making apply rewrite the test pins in the same commit, or by deriving the test expectations from the Dockerfiles instead of literals.

Acceptance

publish-dev-base goes green on a main push, and the built :latest ships codex 0.144.1 (or newer) - completing aos#450's acceptance.
The daily dep-bump run can land a pin bump end to end without a human syncing tests.

Refs: aos#450, aos#435, docs/dev-base-auto-bump.md, docker/dev-base/core/Dockerfile, scripts/dep-bump.py, tests/test_dev_base_image.py.

## publish-dev-base is red on every main push, and the dep-bump loop that should prevent it is gate-blocked Found while landing aos#450 (codex pin bump). The `publish-image` path in [`release.yml`](../src/branch/main/.forgejo/workflows/release.yml) has been failing on main (runs 986/jobs/2 at d5079cc, and the push before it), so landed Dockerfile bumps - including aos#450's `CODEX_VERSION` 0.144.1 - never reach the published `:latest`. Three stacked causes, in dependency order: * stale ward pin - `docker/dev-base/core/Dockerfile` pins `ARG WARD_VERSION=0.567.0` while ward is at v0.596.0. The in-image `CLIGUARD_NO_SANDBOX=1 ward doctor` gate fails at 0.567.0 because that ward predates the split `.ward` bundle layout (it wants `ward-kdl.defaults.kdl` / `ward-kdl.fleet.kdl` / `ward-kdl.forgejo.guardfile.kdl`). * swagger lock vs ward doctor - bumping `WARD_VERSION` alone cannot go green either. aos#435 removed `forgejo.swagger.lock.json` from the repo and from the ward-specs bundle tarball, but every current ward release still hard-requires it in a bundle config source (`bundleForgejoSpecLockPath` in `cmd/ward/configsource.go`, verified failing on v0.584.0 and a source-built v0.596.0, and still present at ward main HEAD). Doctor fails `ops bundle: read spec lock`. Either ward drops or soft-fails the lock requirement for bundles (then bump `WARD_VERSION` here), or the bundle restores the lock. Needs a ward-side decision - this half likely wants a ward issue. * dep-bump is deadlocked by its own gate - `scripts/dep-bump.py apply` rewrites only Dockerfile `ARG` lines, while `tests/test_dev_base_image.py` hard-pins every managed `ARG` default as a string literal (`test_split_tiers_keep_their_managed_arg_defaults`, plus `ARG WARD_VERSION=0.567.0` in `test_core_tier_keeps_the_hidden_ward_builder_stage`). The daily auto-bump commit therefore always fails the pytest gate it runs before pushing, and every pin silently drifts - exactly the incident class the auto-bump was built to close (docs/dev-base-auto-bump.md, agentic-os#272). Fix by making `apply` rewrite the test pins in the same commit, or by deriving the test expectations from the Dockerfiles instead of literals. ## Acceptance * `publish-dev-base` goes green on a main push, and the built `:latest` ships codex 0.144.1 (or newer) - completing aos#450's acceptance. * The daily dep-bump run can land a pin bump end to end without a human syncing tests. Refs: aos#450, aos#435, docs/dev-base-auto-bump.md, `docker/dev-base/core/Dockerfile`, `scripts/dep-bump.py`, `tests/test_dev_base_image.py`.

coilyco-ops commented

2026-07-10 18:55:02 +00:00

Author

Owner

Prevention for this class filed as #454: run the dev-base docker build + in-image ward doctor on PRs as a build-only validation step, so a stale WARD_VERSION / swagger-lock desync / pin drift fails the PR instead of going red on main. This issue stays scoped to making publish-dev-base green now; #454 shifts the check left so it does not recur.

Prevention for this class filed as #454: run the dev-base docker build + in-image `ward doctor` on PRs as a build-only validation step, so a stale WARD_VERSION / swagger-lock desync / pin drift fails the PR instead of going red on main. This issue stays scoped to making `publish-dev-base` green now; #454 shifts the check left so it does not recur.

coilyco-ops added the

labels

2026-07-10 19:23:24 +00:00

coilyco-ops commented

2026-07-10 19:23:24 +00:00

Author

Owner

WARD-TRIAGE: #452 P3 consult

triage details

method: tooling-issue-prioritization
P0 candidate: no
score: 3
mode: consult
confidence: no
tier: P3
labels: P3 consult
reason: Red CI blocking published :latest is the clear next thing, but the gate-blocked dep-bump deadlock needs a design call and the fix touches the externally-visible publish path.

— Claude (she/her), via ward agent

WARD-TRIAGE: #452 P3 consult <details><summary>triage details</summary> method: tooling-issue-prioritization P0 candidate: no score: 3 mode: consult confidence: no tier: P3 labels: P3 consult reason: Red CI blocking published :latest is the clear next thing, but the gate-blocked dep-bump deadlock needs a design call and the fix touches the externally-visible publish path. </details>  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue

2026-07-10 22:07:42 +00:00

Require the ci / build-dev-base status context in branch protection #457

coilyco-ops commented

2026-07-10 22:20:25 +00:00

Author

Owner

Fresh data from the #454 run (claude-linux-6dfbe4beef5b-da08-she-her): PR #456 now builds the dev-base family on every PR through a shared actions/dev-base-build composite, and its first run reproduced this incident pre-merge, so the prevention half is in place. Updated version matrix, each entry verified by running that ward's doctor against the current .ward/ bundle (source-built where needed):

v0.567.0 (current pin) - fails wanting the pre-reshape ward-kdl.* filenames, as diagnosed above.
v0.582.0-v0.604.0 - read the reshaped names but hard-require forgejo.swagger.lock.json (removed by aos#435). Verified on a source-built v0.604.0: FAIL ops bundle: read spec lock.
v0.605.0+ - newly require a ward.bundle.kdl ops manifest first (ops { forgejo "guardfile.forgejo.kdl" }, parsed fail-closed in cmd/ward/configsource.go), and with that manifest present still hard-require the swagger lock. Verified on v0.605.0.
ward main HEAD (815cb40) - bundleForgejoSpecLockPath = "forgejo.swagger.lock.json" is still read unconditionally by buildForgejoOps, so no unreleased fix is waiting either.

Conclusion unchanged but sharper: bumping WARD_VERSION alone cannot go green at any released tag, and the ward-side half maps to ward#1064 (bundle loading independent of consumer KDL filenames, with ward#1080/ward#1082 pushing the same direction). The run deliberately did not add ward.bundle.kdl to .ward/ since that is the file-shape churn ward#1080 complains about. Once ward ships the lock-independence, the aos-side close-out is the pin bump plus the test_dev_base_image.py literal sync (bullet 3 above), and PR #456's ci / build-dev-base check will confirm the fix on its own PR.

Fresh data from the #454 run (claude-linux-6dfbe4beef5b-da08-she-her): PR #456 now builds the dev-base family on every PR through a shared `actions/dev-base-build` composite, and its first run reproduced this incident pre-merge, so the prevention half is in place. Updated version matrix, each entry verified by running that ward's doctor against the current `.ward/` bundle (source-built where needed): * v0.567.0 (current pin) - fails wanting the pre-reshape `ward-kdl.*` filenames, as diagnosed above. * v0.582.0-v0.604.0 - read the reshaped names but hard-require `forgejo.swagger.lock.json` (removed by aos#435). Verified on a source-built v0.604.0: `FAIL ops bundle: read spec lock`. * v0.605.0+ - newly require a `ward.bundle.kdl` ops manifest first (`ops { forgejo "guardfile.forgejo.kdl" }`, parsed fail-closed in `cmd/ward/configsource.go`), and with that manifest present still hard-require the swagger lock. Verified on v0.605.0. * ward main HEAD (815cb40) - `bundleForgejoSpecLockPath = "forgejo.swagger.lock.json"` is still read unconditionally by `buildForgejoOps`, so no unreleased fix is waiting either. Conclusion unchanged but sharper: bumping `WARD_VERSION` alone cannot go green at any released tag, and the ward-side half maps to ward#1064 (bundle loading independent of consumer KDL filenames, with ward#1080/ward#1082 pushing the same direction). The run deliberately did not add `ward.bundle.kdl` to `.ward/` since that is the file-shape churn ward#1080 complains about. Once ward ships the lock-independence, the aos-side close-out is the pin bump plus the `test_dev_base_image.py` literal sync (bullet 3 above), and PR #456's `ci / build-dev-base` check will confirm the fix on its own PR.

coilyco-ops commented

2026-07-11 23:37:41 +00:00

Author

Owner

Re-verified 2026-07-11 after the pytest gate was unblocked (aos#471)

The gate half of this is now fixed and merged: ci / gate, mirror-to-github, release / test, and plan-release are all green on main (run 1055). publish-dev-base therefore runs now instead of being skipped, which exposes the remaining half.

The swagger-lock cause listed above is resolved - .ward/forgejo.swagger.lock.json is present and ward doctor no longer complains about it. The live failure set is different, and I pulled the actual step log (run 1055, job 2, step 7) to confirm:

core build, Dockerfile:182  ->  CLIGUARD_NO_SANDBOX=1 ward doctor
FAIL smart defaults: read ward-kdl.defaults.kdl: no such file or directory
FAIL fleet:          read ward-kdl.fleet.kdl: no such file or directory
FAIL ops bundle:     read ward-kdl.forgejo.guardfile.kdl: no such file or directory
ward doctor: 3 check(s) failed

That is the stale WARD_VERSION=0.567.0 pin, exactly as this issue says: 0.567.0 predates the bundle rename (8a04260).

But bumping the pin alone is still not enough

I ran ward doctor from v0.626.0 and from ward main HEAD against the current bundle. Both give the same three different failures:

FAIL smart defaults: repos body: unknown node "burndown" (want repo-authority; fail-closed)
FAIL fleet: role "engineer" body: unknown node "guardfile" (want guardfiles | agent; fail-closed)
FAIL ops bundle: guardfile base-url "git.example.com/api/v1" still looks like a placeholder
PASS exec bundle

Per Kai, aos is the correct state and ward/cli-guard adapt to it. So all three are ward-side gaps:

burndown - ward has no burndown anywhere in its source; repos {} accepts only repo-authority (cmd/ward/smartdefaults.go). Tracked as ward#1105.
singular guardfile - cli-guard main already supports this (case "guardfile" in pkg/fleetconfig, added by f6063aa, 2026-07-10). ward just pins an older cli-guard (v0.85.1-0.20260709002015). Blocker: cli-guard removed the legacy cli/dispatch package in 05e8b78 (2026-07-09), before the guardfile support landed, and ward still imports it. So ward cannot reach a cli-guard with guardfile support without first migrating off cli/dispatch. There is no tag with both. Filed separately.
placeholder base-url - ops.forgejo.kdl and guardfile.forgejo.read.kdl intentionally carry git.example.com / example* / /example/forgejo/api-token (their comments say the real values belong to the selected bundle, not the baked default). ward's doctor placeholder check has to stop rejecting them.

Order to green

cli-guard dispatch migration in ward -> bump cli-guard -> teach ward burndown -> relax the doctor placeholder check -> release ward -> bump ARG WARD_VERSION here and the hard-pinned literal in tests/test_dev_base_image.py (the dep-bump deadlock this issue already calls out).

Until then the published image stays at v0.245.0 (ward v0.529.0 inside). It is pullable and ward does run in it - just stale.

## Re-verified 2026-07-11 after the pytest gate was unblocked (aos#471) The gate half of this is now fixed and merged: `ci / gate`, `mirror-to-github`, `release / test`, and `plan-release` are **all green on main** (run 1055). `publish-dev-base` therefore *runs* now instead of being skipped, which exposes the remaining half. **The swagger-lock cause listed above is resolved** - `.ward/forgejo.swagger.lock.json` is present and `ward doctor` no longer complains about it. The live failure set is different, and I pulled the actual step log (run 1055, job 2, step 7) to confirm: ``` core build, Dockerfile:182 -> CLIGUARD_NO_SANDBOX=1 ward doctor FAIL smart defaults: read ward-kdl.defaults.kdl: no such file or directory FAIL fleet: read ward-kdl.fleet.kdl: no such file or directory FAIL ops bundle: read ward-kdl.forgejo.guardfile.kdl: no such file or directory ward doctor: 3 check(s) failed ``` That is the stale `WARD_VERSION=0.567.0` pin, exactly as this issue says: 0.567.0 predates the bundle rename (8a04260). ### But bumping the pin alone is still not enough I ran `ward doctor` from **v0.626.0** and from **ward main HEAD** against the current bundle. Both give the same three *different* failures: ``` FAIL smart defaults: repos body: unknown node "burndown" (want repo-authority; fail-closed) FAIL fleet: role "engineer" body: unknown node "guardfile" (want guardfiles | agent; fail-closed) FAIL ops bundle: guardfile base-url "git.example.com/api/v1" still looks like a placeholder PASS exec bundle ``` Per Kai, **aos is the correct state and ward/cli-guard adapt to it.** So all three are ward-side gaps: 1. **burndown** - ward has no `burndown` anywhere in its source; `repos {}` accepts only `repo-authority` (`cmd/ward/smartdefaults.go`). Tracked as ward#1105. 2. **singular `guardfile`** - cli-guard **main already supports this** (`case "guardfile"` in `pkg/fleetconfig`, added by `f6063aa`, 2026-07-10). ward just pins an older cli-guard (`v0.85.1-0.20260709002015`). **Blocker:** cli-guard removed the legacy `cli/dispatch` package in `05e8b78` (2026-07-09), *before* the guardfile support landed, and ward still imports it. So ward cannot reach a cli-guard with guardfile support without first migrating off `cli/dispatch`. There is no tag with both. Filed separately. 3. **placeholder base-url** - `ops.forgejo.kdl` and `guardfile.forgejo.read.kdl` intentionally carry `git.example.com` / `example*` / `/example/forgejo/api-token` (their comments say the real values belong to the selected bundle, not the baked default). ward's doctor placeholder check has to stop rejecting them. ### Order to green cli-guard dispatch migration in ward -> bump cli-guard -> teach ward `burndown` -> relax the doctor placeholder check -> release ward -> bump `ARG WARD_VERSION` here **and** the hard-pinned literal in `tests/test_dev_base_image.py` (the dep-bump deadlock this issue already calls out). Until then the published image stays at v0.245.0 (ward v0.529.0 inside). It is pullable and ward does run in it - just stale.

coilysiren referenced this issue from a commit

2026-07-12 05:44:19 +00:00

fix(launch): live file:// ward bundle, canonical workflow spellings, ward 0.629.0

coilysiren referenced this issue

2026-07-12 05:45:01 +00:00

fix(launch): live file:// ward bundle, canonical workflow spellings, ward 0.629.0 #474

coilysiren referenced this issue

2026-07-12 06:00:07 +00:00

feat(release): two-stage release - gate-green main promotes to a queued release branch #475

coilysiren commented

2026-07-12 06:46:14 +00:00

Owner

State after the 2026-07-12 stability session

ARG WARD_VERSION is at 0.629.0 (aos#474), and ward doctor passes against the current .ward bundle - verified locally and by release run 1063 (manual) going green end to end.
The dep-bump deadlock (bullet 3) is closed: tests/test_dev_base_image.py now derives managed-ARG values from the Dockerfiles (names stay the contract), so the daily auto-bump no longer fails its own pytest gate (aos#474).
The remaining publish-dev-base "red" pattern turned out to be Forgejo''s per-branch auto-cancellation on push bursts (see #477), not a build failure - the two-stage release (#475) removes that class.

Release run 1078 (a push-triggered main release carrying today''s batch) is in flight; if it lands green, the original acceptance ("publish-dev-base goes green on a main push") is met and this can close.

## State after the 2026-07-12 stability session * `ARG WARD_VERSION` is at **0.629.0** (aos#474), and `ward doctor` passes against the current `.ward` bundle - verified locally and by release run 1063 (manual) going green end to end. * The **dep-bump deadlock (bullet 3) is closed**: `tests/test_dev_base_image.py` now derives managed-ARG values from the Dockerfiles (names stay the contract), so the daily auto-bump no longer fails its own pytest gate (aos#474). * The remaining publish-dev-base "red" pattern turned out to be Forgejo''s per-branch auto-cancellation on push bursts (see #477), not a build failure - the two-stage release (#475) removes that class. Release run 1078 (a push-triggered main release carrying today''s batch) is in flight; if it lands green, the original acceptance ("publish-dev-base goes green on a main push") is met and this can close.

coilysiren referenced this issue from coilyco-flight-deck/ward

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283

~~coilyco-ops referenced this issue 2026-07-18 04:27:26 +00:00~~

Consume Ward release artifacts in dev-base and workflow bootstrap #606

coilyco-ops commented

2026-07-22 08:14:56 +00:00

Author

Owner

The agent is closing this incident as resolved. Current main, release, and tag v0.255.0 all point to c90cbde. CI run 1564, promotion run 1566, and the complete dev-base publish retry run 1568 succeeded. The dependency-bump test deadlock was also removed as recorded earlier in this thread. Prevention work remains separately tracked in #454 and #457.\n\nPublish evidence: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/1568

The agent is closing this incident as resolved. Current main, release, and tag v0.255.0 all point to c90cbde. CI run 1564, promotion run 1566, and the complete dev-base publish retry run 1568 succeeded. The dependency-bump test deadlock was also removed as recorded earlier in this thread. Prevention work remains separately tracked in #454 and #457.\n\nPublish evidence: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/1568

coilyco-ops closed this issue

2026-07-22 08:15:16 +00:00

Rows
Columns

publish-dev-base red on main: stale WARD_VERSION, swagger-lock vs ward doctor, dep-bump gate deadlock #452

publish-dev-base is red on every main push, and the dep-bump loop that should prevent it is gate-blocked

Acceptance

Re-verified 2026-07-11 after the pytest gate was unblocked (aos#471)

But bumping the pin alone is still not enough

Order to green

State after the 2026-07-12 stability session