Bound dev-base package churn before it fills kai-server again #739

New issue

Closed

opened 2026-07-25 05:55:08 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-25 05:55:08 +00:00

Member

Problem

The dev-base publisher regenerated roughly 79.8 GiB of Forgejo package blobs and 688 package versions in about one day after the 2026-07-23 emergency purge. On the next evening, kai-server root usage had returned to 89.2% and the Forgejo package store had reached 110.9 GiB.

The pipeline intentionally publishes commit-scoped draft-* images, per-tier cache refs, OCI digest versions, and the moving release aliases. Infrastructure#600 owns server-side retention. This issue owns reducing and bounding what the publisher creates before retention runs.

Related live incident report: infrastructure#642. Related publisher incidents: #692 and #721.

Work

The agent measures physical and logical registry growth per promoted commit, tier, draft tag, digest, and build-cache ref.
The agent identifies which writes are required for release recovery and which are replaceable transient state.
The agent keeps immutable release tags and the moving release alias usable.
The agent makes transient draft and cache publication overwrite, expire, or clean up through supported Forgejo mechanisms without direct filesystem deletion.
The agent prevents concurrent tier publication from multiplying equivalent blobs or stranded manifests.
The agent adds workflow evidence that reports package bytes and versions produced by one publish.
The agent documents the boundary between publisher cleanup here and registry-wide retention in infrastructure#600.

Acceptance

A representative warm publish and one cold-cache publish complete successfully.
The published draft remains promotable through the established release path.
Current release consumers still pull successfully.
The measured worst-case package growth over one 24-hour retention window cannot consume the root-filesystem headroom between the 71% recovery baseline and the 85% pressure threshold.
Repeated publishes reach a stable bounded footprint instead of accumulating roughly 80 GiB per day.
Repository validation passes, and an ops-capable run records the live Forgejo measurements without using repeated pushes as probes.

## Problem The dev-base publisher regenerated roughly 79.8 GiB of Forgejo package blobs and 688 package versions in about one day after the 2026-07-23 emergency purge. On the next evening, kai-server root usage had returned to 89.2% and the Forgejo package store had reached 110.9 GiB. The pipeline intentionally publishes commit-scoped `draft-*` images, per-tier cache refs, OCI digest versions, and the moving release aliases. Infrastructure#600 owns server-side retention. This issue owns reducing and bounding what the publisher creates before retention runs. Related live incident report: infrastructure#642. Related publisher incidents: #692 and #721. ## Work * The agent measures physical and logical registry growth per promoted commit, tier, draft tag, digest, and build-cache ref. * The agent identifies which writes are required for release recovery and which are replaceable transient state. * The agent keeps immutable release tags and the moving `release` alias usable. * The agent makes transient draft and cache publication overwrite, expire, or clean up through supported Forgejo mechanisms without direct filesystem deletion. * The agent prevents concurrent tier publication from multiplying equivalent blobs or stranded manifests. * The agent adds workflow evidence that reports package bytes and versions produced by one publish. * The agent documents the boundary between publisher cleanup here and registry-wide retention in infrastructure#600. ## Acceptance * A representative warm publish and one cold-cache publish complete successfully. * The published draft remains promotable through the established release path. * Current `release` consumers still pull successfully. * The measured worst-case package growth over one 24-hour retention window cannot consume the root-filesystem headroom between the 71% recovery baseline and the 85% pressure threshold. * Repeated publishes reach a stable bounded footprint instead of accumulating roughly 80 GiB per day. * Repository validation passes, and an ops-capable run records the live Forgejo measurements without using repeated pushes as probes.

coilyco-ops referenced this issue from coilyco-flight-deck/infrastructure

2026-07-25 06:18:04 +00:00

Reverse #543 and migrate dev-base consumers to the single :release tag #643

coilyco-ops referenced this issue

2026-07-25 06:23:18 +00:00

Interactive: promote a self-bootstrapping dev-base image to release #721

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-29 03:44:56 +00:00