fix(release): land the tag and ward-specs bundle even when an image tier flakes

coilyco-ops commented

2026-07-13 04:23:00 +00:00

Member

What

aos stopped releasing: run 1127's publish-dev-base flaked on one registry blob write for the lang-dotnet tier, and because release needed publish-dev-base, the v0.249.0 tag, the Forgejo release, and the ward-specs bundle were all skipped. The Telegram alert for that failure also died - one un-retried TLS handshake timeout - so main sat unreleased at v0.248.0 with no signal.

Changes

decouple - release now needs plan-release alone, so the tag, the Forgejo release, and the ward-specs bundle land regardless of the image publish. publish-dev-base chains off release instead (images are a consequence of the tag, not a precondition) and fails independently and loudly. New tests/test_release_workflow_decoupling.py pins the job graph.
retry the push - scripts/dev-base-build.py retries each pushed tier (3 attempts, 15s/60s backoff), so a single blob-write flake re-pushes one tier instead of costing the run. Already-pushed layers and the registry buildcache make a retry mostly re-push, not rebuild. Local builds stay fail-fast. Covered by new tests.
fail loudly - the alert for run 1127 DID fire but died on one urlopen call (_ssl.c:975: The handshake operation timed out). The composite actions/telegram-alert and all nine inline workflow copies (ci, promote, mirror x2, dep-bump, freshness, release x3) now retry the send (3 attempts, 10s/30s backoff, 30s timeout). Covered by new tests.
docs - release.md, dev-base-image.md, telegram-ci-alerts.md updated for the inverted ordering.

Landing the missing release

Merging this PR pushes main forward, promote.yml fast-forwards release, and the fixed pipeline cuts the next tag plus the ward-specs bundle from plan-release alone - even if the dotnet tier flakes again. That satisfies the issue's done condition structurally: a forced failure of one image tier can no longer prevent the tag and bundle.

Contributing factor (follow-up, not in this PR)

kai-server is at 87% disk (63.5 GB free of 515 GB, verified via node stats). The packages API returns 403 for the container token, so dev-base tag retention could not be enumerated from here. A follow-up issue covers checking registry retention and adding a cleanup policy.

closes #490

ward.workflow: pull-request-and-merge

## What aos stopped releasing: run 1127's `publish-dev-base` flaked on one registry blob write for the `lang-dotnet` tier, and because `release` needed `publish-dev-base`, the `v0.249.0` tag, the Forgejo release, and the ward-specs bundle were all skipped. The Telegram alert for that failure also died - one un-retried TLS handshake timeout - so main sat unreleased at `v0.248.0` with no signal. ## Changes * decouple - `release` now needs `plan-release` alone, so the tag, the Forgejo release, and the ward-specs bundle land regardless of the image publish. `publish-dev-base` chains off `release` instead (images are a consequence of the tag, not a precondition) and fails independently and loudly. New `tests/test_release_workflow_decoupling.py` pins the job graph. * retry the push - `scripts/dev-base-build.py` retries each pushed tier (3 attempts, 15s/60s backoff), so a single blob-write flake re-pushes one tier instead of costing the run. Already-pushed layers and the registry buildcache make a retry mostly re-push, not rebuild. Local builds stay fail-fast. Covered by new tests. * fail loudly - the alert for run 1127 DID fire but died on one `urlopen` call (`_ssl.c:975: The handshake operation timed out`). The composite `actions/telegram-alert` and all nine inline workflow copies (ci, promote, mirror x2, dep-bump, freshness, release x3) now retry the send (3 attempts, 10s/30s backoff, 30s timeout). Covered by new tests. * docs - release.md, dev-base-image.md, telegram-ci-alerts.md updated for the inverted ordering. ## Landing the missing release Merging this PR pushes main forward, promote.yml fast-forwards `release`, and the fixed pipeline cuts the next tag plus the ward-specs bundle from `plan-release` alone - even if the dotnet tier flakes again. That satisfies the issue's done condition structurally: a forced failure of one image tier can no longer prevent the tag and bundle. ## Contributing factor (follow-up, not in this PR) kai-server is at 87% disk (63.5 GB free of 515 GB, verified via node stats). The packages API returns 403 for the container token, so dev-base tag retention could not be enumerated from here. A follow-up issue covers checking registry retention and adding a cleanup policy. closes #490 ward.workflow: pull-request-and-merge

coilyco-ops added 1 commit

2026-07-13 04:23:00 +00:00

fix(release): land the tag and ward-specs bundle even when an image tier flakes

ci / gate (pull_request) Successful in 57s

Details

b9a9fd07f3

The release job now needs plan-release alone, so the git tag, the
Forgejo release, and the ward-specs bundle land regardless of the
image publish. publish-dev-base chains off the tag instead - images
are a consequence of the tag, not a precondition - and fails
independently and loudly. Each pushed tier in dev-base-build.py
retries with backoff, so a single registry blob-write flake re-pushes
one tier instead of costing the run. Every Telegram failure alert
(the composite action and all inline workflow copies) retries its
send: run 1127's alert died on one un-retried TLS handshake timeout,
so the blocked release surfaced nowhere.

closes #490

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

coilyco-ops referenced this pull request

2026-07-13 04:23:16 +00:00

dev-base registry retention: kai-server at 87% disk, dev-base tags likely retained forever #493

coilysiren commented

2026-07-13 05:28:14 +00:00

Owner

almost certainly it will be easier just to rewrite this PR from scratch

coilyco-ops commented

2026-07-13 07:14:42 +00:00

Author

Member

Heads-up from Kai's desktop session: this branch is now mergeable: false against main.

main landed 27eccc2 and e51de73 tonight: the ward-specs tar step in release.yml switched to a wildcard plus deny-list (the enumerated list had silently dropped the role-bound guardfile.aws.engineer.kdl), ci.yml grew a standalone ward-doctor job, and ci/promote moved off the retired :latest container to :release (its ward v0.529.0 predates the doctor verb). A rebase needs to keep the deny-list tar form.
Fresh evidence for this PR's motivation: release run 1148 died in job setup on a github.com egress timeout fetching actions/checkout (infrastructure#553 class), and run 1149 died on failed to push ...lang-go-v0.250.0: unknown (the #490 blob-write class). Two consecutive runs, two different single-job flakes, zero tags - the decoupling and per-tier retries here are the structural fix. Run 1150 is a manual re-dispatch in the meantime.

Heads-up from Kai's desktop session: this branch is now `mergeable: false` against main. * main landed `27eccc2` and `e51de73` tonight: the ward-specs tar step in release.yml switched to a wildcard plus deny-list (the enumerated list had silently dropped the role-bound `guardfile.aws.engineer.kdl`), ci.yml grew a standalone `ward-doctor` job, and ci/promote moved off the retired `:latest` container to `:release` (its ward v0.529.0 predates the doctor verb). A rebase needs to keep the deny-list tar form. * Fresh evidence for this PR's motivation: release run 1148 died in job setup on a github.com egress timeout fetching actions/checkout (infrastructure#553 class), and run 1149 died on `failed to push ...lang-go-v0.250.0: unknown` (the #490 blob-write class). Two consecutive runs, two different single-job flakes, zero tags - the decoupling and per-tier retries here are the structural fix. Run 1150 is a manual re-dispatch in the meantime.

coilysiren commented

2026-07-13 21:07:19 +00:00

Owner

remake this PR from scratch

coilysiren closed this pull request

2026-07-13 21:07:28 +00:00