coilyco-flight-deck/agentic-os

Fork 0

Releases are blocked: a dotnet image-push flake skips the release job, so no tag and no ward-specs bundle (stuck at v0.248.0) #490

New issue

Open

opened 2026-07-13 04:07:26 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-13 04:07:26 +00:00

Member

aos is not releasing right now

Latest tag is v0.248.0. Main's HEAD (c75de6b) is untagged and unreleased. v0.249.0 was computed but never created.

Run 1127 (release.yml, push to release) failed in publish-dev-base, step "Build and push (multi-arch)":

#13 ERROR: error writing layer blob: unknown
#11 ERROR: failed to push forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:lang-dotnet-v0.249.0: unknown
ERROR: failed to build: failed to solve: failed to push ...:lang-dotnet-v0.249.0: unknown

The real defect: the tag is gated behind the image push

The job graph hard-chains them:

plan-release computes the tag.
publish-dev-base - needs: plan-release.
release - needs: [plan-release, publish-dev-base].

So release, the job that creates the git tag, cuts the Forgejo release, and publishes the ward-specs bundle as a pinned, checksummed release asset, is skipped whenever the image publish fails. Those artifacts are cheap and deterministic. The thing gating them is the single most fragile step in the repo: a 7-tier, 2-arch container build-and-push (core, lang-node, lang-go, lang-dotnet, ops, agent, full) running under a 120-minute timeout with no retry. One transient registry blob error from the largest tier (dotnet) silently costs the release.

This is backwards. An image-registry flake should cost a re-push of an image, not the version tag and the spec bundle that ward consumes.

Ask

1. Decouple the release artifacts from the image publish. Let release proceed on plan-release alone, so the tag, the Forgejo release, and the ward-specs bundle asset land regardless of whether an image tier flaked. Publishing images is a consequence of the tag, not a precondition for it. If some ordering must be preserved (for example the release notes want to reference pullable image tags), invert it: publish images after the tag, and let that job fail independently and loudly.

2. Retry the image push. A multi-GB multi-arch push to a self-hosted registry will flake. Add bounded retries around the push in scripts/dev-base-build.py (or the workflow step), per tier, so a single bad blob write does not fail the run. Prefer retrying only the failed tier over rebuilding all seven.

3. Fail loudly, not silently. The Telegram alert exists but this still reached a state where main sat unreleased and nothing obviously surfaced it. Confirm the alert actually fired for run 1127. If it did not, fix that too, because a blocked release pipeline that nobody notices is how v0.248.0 becomes stale.

4. Land the missing release. Once fixed, get c75de6b (or whatever main is by then) tagged and released so the dev-base family and the ward-specs bundle are published from current main.

Contributing factor worth checking

kai-server is at 87% disk (63 GB free of 515 GB). The Forgejo package registry stores these blobs, and the dev-base family is large and multi-arch. error writing layer blob: unknown is consistent with storage pressure. Check whether the registry is up against a quota or the disk is squeezing it, and whether old dev-base tags are being retained forever. If every release keeps 7 tiers x 2 arches indefinitely, add a retention policy - that is a slow-motion outage otherwise.

Context: this pipeline is new and has been red repeatedly

The last several main commits are all release-pipeline surgery, and main went fully red twice during it (runs 7426-7428 and 7435-7437 had ci.yml, promote.yml, and mirror-to-github.yml all three failing on the same push). It is green on ci/promote/mirror now, but release.yml is still broken. Treat hardening this pipeline as the goal, not just clearing the one red run.

Done condition

release.yml is green on a push to release, main's HEAD carries a version tag, the ward-specs bundle asset is published for it, and a forced failure of one image tier no longer prevents the tag and bundle from landing.

## aos is not releasing right now Latest tag is **`v0.248.0`**. Main's HEAD (`c75de6b`) is **untagged and unreleased**. `v0.249.0` was computed but never created. Run [1127](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/1127) (`release.yml`, push to `release`) failed in `publish-dev-base`, step "Build and push (multi-arch)": ```text #13 ERROR: error writing layer blob: unknown #11 ERROR: failed to push forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:lang-dotnet-v0.249.0: unknown ERROR: failed to build: failed to solve: failed to push ...:lang-dotnet-v0.249.0: unknown ``` ## The real defect: the tag is gated behind the image push The job graph hard-chains them: * `plan-release` computes the tag. * `publish-dev-base` - `needs: plan-release`. * `release` - **`needs: [plan-release, publish-dev-base]`**. So `release`, the job that creates the git tag, cuts the Forgejo release, **and publishes the ward-specs bundle as a pinned, checksummed release asset**, is skipped whenever the image publish fails. Those artifacts are cheap and deterministic. The thing gating them is the single most fragile step in the repo: a **7-tier, 2-arch container build-and-push** (`core`, `lang-node`, `lang-go`, `lang-dotnet`, `ops`, `agent`, `full`) running under a 120-minute timeout **with no retry**. One transient registry blob error from the largest tier (dotnet) silently costs the release. This is backwards. An image-registry flake should cost a re-push of an image, not the version tag and the spec bundle that ward consumes. ## Ask **1. Decouple the release artifacts from the image publish.** Let `release` proceed on `plan-release` alone, so the tag, the Forgejo release, and the ward-specs bundle asset land regardless of whether an image tier flaked. Publishing images is a consequence of the tag, not a precondition for it. If some ordering must be preserved (for example the release notes want to reference pullable image tags), invert it: publish images *after* the tag, and let that job fail independently and loudly. **2. Retry the image push.** A multi-GB multi-arch push to a self-hosted registry will flake. Add bounded retries around the push in `scripts/dev-base-build.py` (or the workflow step), per tier, so a single bad blob write does not fail the run. Prefer retrying only the failed tier over rebuilding all seven. **3. Fail loudly, not silently.** The Telegram alert exists but this still reached a state where main sat unreleased and nothing obviously surfaced it. Confirm the alert actually fired for run 1127. If it did not, fix that too, because a blocked release pipeline that nobody notices is how `v0.248.0` becomes stale. **4. Land the missing release.** Once fixed, get `c75de6b` (or whatever main is by then) tagged and released so the dev-base family and the ward-specs bundle are published from current main. ## Contributing factor worth checking **kai-server is at 87% disk** (63 GB free of 515 GB). The Forgejo package registry stores these blobs, and the dev-base family is large and multi-arch. `error writing layer blob: unknown` is consistent with storage pressure. Check whether the registry is up against a quota or the disk is squeezing it, and whether old dev-base tags are being retained forever. If every release keeps 7 tiers x 2 arches indefinitely, add a retention policy - that is a slow-motion outage otherwise. ## Context: this pipeline is new and has been red repeatedly The last several main commits are all release-pipeline surgery, and main went fully red twice during it (runs 7426-7428 and 7435-7437 had `ci.yml`, `promote.yml`, and `mirror-to-github.yml` **all three** failing on the same push). It is green on ci/promote/mirror now, but `release.yml` is still broken. Treat hardening this pipeline as the goal, not just clearing the one red run. ## Done condition `release.yml` is green on a push to `release`, main's HEAD carries a version tag, the ward-specs bundle asset is published for it, and a forced failure of one image tier no longer prevents the tag and bundle from landing.

coilyco-ops commented

2026-07-13 04:07:50 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-claude-agentic-os-490 on host KAI-DESKTOP-TOWER.

Reserved by ward agent --harness claude (reserved 2026-07-13T04:07:47Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/agentic-os#490 · branch issue-490 · harness claude · workflow pull-request-and-merge
Run: engineer-claude-agentic-os-490 · ward v0.633.0 · dispatched 2026-07-13T04:07:47Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.633.0).

— Claude (she/her), via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-claude-agentic-os-490` on host `KAI-DESKTOP-TOWER`. Reserved by `ward agent --harness claude` (reserved 2026-07-13T04:07:47Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/agentic-os#490` · branch `issue-490` · harness `claude` · workflow `pull-request-and-merge` - **Run:** `engineer-claude-agentic-os-490` · ward `v0.633.0` · dispatched `2026-07-13T04:07:47Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.633.0). </details> </details>  — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-13 04:14:04 +00:00

Author

Member

Clarification for the record (does not change the asks): the release cadence is healthy, not stale. v0.244 through v0.248 all landed 2026-07-10 to 2026-07-12, and v0.248.0 is roughly an hour old. aos publishes from the release branch (promote.yml moves main to release, release.yml triggers on push to release), so "main is untagged" in the body is incidental - release and main are both at c75de6b (ahead=0, behind=0), and the tag is cut on release.

So this is one fresh failed release, not a chronically broken pipeline. Run 1127 on the release branch lost v0.249.0 to a single transient error writing layer blob on the dotnet tier. The three commits since v0.248.0 (c3b07e3, 0354ab3, c75de6b) are the currently-unreleased ones.

That sharpens the diagnosis rather than softening it: a pipeline that released five times in three days and then dropped a release to one flaky blob write is precisely the case for decoupling release from publish-dev-base and retrying the push per tier. Fix the coupling and the retry, not a phantom staleness problem.

Clarification for the record (does not change the asks): the release cadence is **healthy**, not stale. v0.244 through v0.248 all landed 2026-07-10 to 2026-07-12, and v0.248.0 is roughly an hour old. aos publishes from the **`release` branch** (promote.yml moves main to `release`, release.yml triggers on push to `release`), so "main is untagged" in the body is incidental - `release` and `main` are both at c75de6b (ahead=0, behind=0), and the tag is cut on `release`. So this is **one fresh failed release, not a chronically broken pipeline**. Run 1127 on the `release` branch lost v0.249.0 to a single transient `error writing layer blob` on the dotnet tier. The three commits since v0.248.0 (c3b07e3, 0354ab3, c75de6b) are the currently-unreleased ones. That sharpens the diagnosis rather than softening it: a pipeline that released five times in three days and then dropped a release to one flaky blob write is precisely the case for decoupling `release` from `publish-dev-base` and retrying the push per tier. Fix the coupling and the retry, not a phantom staleness problem.

coilyco-ops referenced this issue

2026-07-13 04:16:31 +00:00

Finish the dev-base image split: linear tier DAG, unsplit CI jobs, and a build cache thrown away every run #491

coilyco-ops referenced this issue from a commit

2026-07-13 04:22:44 +00:00

fix(release): land the tag and ward-specs bundle even when an image tier flakes

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-13 04:23:00 +00:00

fix(release): land the tag and ward-specs bundle even when an image tier flakes #492

coilyco-ops referenced this issue

2026-07-13 04:23:16 +00:00

dev-base registry retention: kai-server at 87% disk, dev-base tags likely retained forever #493