Package retention keeps only 9 agentic-os image versions, so every exact-version CI pin dies within about a day #828

New issue

Closed

opened 2026-08-14 06:25:38 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-14 06:25:38 +00:00

Member

Any workflow that pins agentic-os:vX.Y.Z instead of the moving :release alias stops working within roughly a day, because retention prunes the tag out from under it. Two open pull requests are already dead on this, and a third was closed partly because of it.

Measured

The coilyco-flight-deck/agentic-os container package carries 350 versions. Filtering to semver tags leaves exactly nine:

v0.272.0 v0.271.0 v0.270.0 v0.269.0 v0.268.0 v0.267.0 v0.266.0 v0.265.0 v0.264.0

Everything else is latest, release, buildcache, per-language draft-<sha> tags, and content-addressed digests.

Direct registry probe:

GET /v2/coilyco-flight-deck/agentic-os/manifests/v0.242.0  -> 404
GET /v2/coilyco-flight-deck/agentic-os/manifests/v0.195.0  -> 404
GET /v2/coilyco-flight-deck/agentic-os/manifests/release   -> 200

The window is nine versions wide and agentic-os moved v0.264.0 to v0.272.0 inside today, so a pin has on the order of a day before it 404s.

What it is currently breaking

infrastructure#799 (safe Sentry DSN rotation) - Scan for secrets job 27827 pulls agentic-os:v0.242.0 and dies in 1s with Error response from daemon: manifest unknown. Both its checks fail this way. The change itself was never evaluated.
coilysiren/lore#14 (Forgejo hygiene workflow) - pins agentic-os:v0.195.0. It has not failed on this yet only because its run was cancelled first. It will 404 the moment it runs.
agentic-os-hardware#93 - pinned the same v0.195.0. Closed earlier today for other reasons, and this is a second one.

The convention is already established everywhere else. Eleven workflows across infrastructure, sirens-echo, and galaxy-gen all use agentic-os:release, and none of them have this failure mode.

Why this reads as a defect rather than a preference

A pin that resolves at authoring time and 404s a day later fails in the worst way: not at review, but later, on an unrelated run, with an error that points at the registry rather than at the pin. The author gets no signal that they picked an expiring reference.

Fix, in the order I would do it

Change the two live pins to :release. That unblocks infrastructure#799 and lore#14 with a one-line edit each.
Add a validator that rejects an exact-version agentic-os: reference in a workflow container: field, so the next one fails at commit time rather than a day later in CI. The catalog suite in agentic-os is the natural home, since the rule is about how a workflow addresses the shared image.
Decide whether nine is the intended retention depth. If pinning a specific version is meant to be supported at all, nine releases is far too shallow given the current release rate. If it is not meant to be supported, item 2 is the whole fix and the depth is fine.

Related: #600 and #733 cover package retention policy generally. This is the concrete consumer-side breakage that retention causes, which neither of those records.

Found while re-reading every open pull request in the fleet.

Any workflow that pins `agentic-os:vX.Y.Z` instead of the moving `:release` alias stops working within roughly a day, because retention prunes the tag out from under it. Two open pull requests are already dead on this, and a third was closed partly because of it. ## Measured The `coilyco-flight-deck/agentic-os` container package carries 350 versions. Filtering to semver tags leaves exactly nine: ``` v0.272.0 v0.271.0 v0.270.0 v0.269.0 v0.268.0 v0.267.0 v0.266.0 v0.265.0 v0.264.0 ``` Everything else is `latest`, `release`, `buildcache`, per-language `draft-<sha>` tags, and content-addressed digests. Direct registry probe: ``` GET /v2/coilyco-flight-deck/agentic-os/manifests/v0.242.0 -> 404 GET /v2/coilyco-flight-deck/agentic-os/manifests/v0.195.0 -> 404 GET /v2/coilyco-flight-deck/agentic-os/manifests/release -> 200 ``` The window is nine versions wide and agentic-os moved v0.264.0 to v0.272.0 inside today, so a pin has on the order of a day before it 404s. ## What it is currently breaking * `infrastructure#799` (safe Sentry DSN rotation) - `Scan for secrets` job 27827 pulls `agentic-os:v0.242.0` and dies in 1s with `Error response from daemon: manifest unknown`. Both its checks fail this way. The change itself was never evaluated. * `coilysiren/lore#14` (Forgejo hygiene workflow) - pins `agentic-os:v0.195.0`. It has not failed on this yet only because its run was cancelled first. It will 404 the moment it runs. * `agentic-os-hardware#93` - pinned the same `v0.195.0`. Closed earlier today for other reasons, and this is a second one. The convention is already established everywhere else. Eleven workflows across infrastructure, sirens-echo, and galaxy-gen all use `agentic-os:release`, and none of them have this failure mode. ## Why this reads as a defect rather than a preference A pin that resolves at authoring time and 404s a day later fails in the worst way: not at review, but later, on an unrelated run, with an error that points at the registry rather than at the pin. The author gets no signal that they picked an expiring reference. ## Fix, in the order I would do it 1. Change the two live pins to `:release`. That unblocks `infrastructure#799` and `lore#14` with a one-line edit each. 2. Add a validator that rejects an exact-version `agentic-os:` reference in a workflow `container:` field, so the next one fails at commit time rather than a day later in CI. The catalog suite in agentic-os is the natural home, since the rule is about how a workflow addresses the shared image. 3. Decide whether nine is the intended retention depth. If pinning a specific version is meant to be supported at all, nine releases is far too shallow given the current release rate. If it is not meant to be supported, item 2 is the whole fix and the depth is fine. Related: #600 and #733 cover package retention policy generally. This is the concrete consumer-side breakage that retention causes, which neither of those records. Found while re-reading every open pull request in the fleet.

coilyco-ops referenced this issue

2026-08-14 06:26:02 +00:00

Runner 625b878f has no reachable Docker daemon, and it is the same runner behind #824 #829

coilyco-ops referenced this issue from a commit

2026-08-14 06:27:26 +00:00

fix(ci): track agentic-os:release instead of a pinned version