dev-base registry retention: kai-server at 87% disk, dev-base tags likely retained forever #493

New issue

Open

opened 2026-07-13 04:23:16 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-13 04:23:16 +00:00

Member

Split out of #490 (the dotnet image-push flake that blocked releases). The pipeline fix landed via PR #492; this issue tracks the storage-pressure contributing factor.

Evidence

kai-server root disk is at 87% (63.5 GB free of 515 GB), verified via node stats on 2026-07-13. error writing layer blob: unknown from run 1127 is consistent with storage pressure on the Forgejo package registry.
The dev-base family publishes 7 tiers x 2 arches per release, plus per-tier buildcache tags, one vX.Y.Z set per minor bump - which is every push to main. If old versions are retained forever this is a slow-motion outage.
The container package listing could not be enumerated from the agent container: GET /api/v1/packages/coilyco-flight-deck?type=container returns 403 for the run token, so current retention depth is unverified.

Ask

Enumerate the agentic-os container package versions (needs a token with read:package, or the admin UI) and measure how much of the 425 GB used is registry blobs.
Add a retention policy: Forgejo has built-in package cleanup rules (admin -> packages) that can keep the last N versions per tag pattern - prefer that over a hand-rolled deletion job. Keep release, buildcache, and the last handful of v* sets; everything older is re-buildable from the Dockerfiles.
Confirm garbage collection actually reclaims blob storage after version deletion (forgejo doctor / cron cleanup_packages).

Split out of #490 (the dotnet image-push flake that blocked releases). The pipeline fix landed via PR #492; this issue tracks the storage-pressure contributing factor. ## Evidence * kai-server root disk is at **87%** (63.5 GB free of 515 GB), verified via node stats on 2026-07-13. `error writing layer blob: unknown` from run 1127 is consistent with storage pressure on the Forgejo package registry. * The dev-base family publishes **7 tiers x 2 arches** per release, plus per-tier `buildcache` tags, one `vX.Y.Z` set per minor bump - which is every push to main. If old versions are retained forever this is a slow-motion outage. * The container package listing could not be enumerated from the agent container: `GET /api/v1/packages/coilyco-flight-deck?type=container` returns 403 for the run token, so current retention depth is unverified. ## Ask 1. Enumerate the `agentic-os` container package versions (needs a token with `read:package`, or the admin UI) and measure how much of the 425 GB used is registry blobs. 2. Add a retention policy: Forgejo has built-in package cleanup rules (admin -> packages) that can keep the last N versions per tag pattern - prefer that over a hand-rolled deletion job. Keep `release`, `buildcache`, and the last handful of `v*` sets; everything older is re-buildable from the Dockerfiles. 3. Confirm garbage collection actually reclaims blob storage after version deletion (forgejo doctor / cron `cleanup_packages`).

coilyco-ops added the

consult

labels

2026-07-14 07:44:26 +00:00