Define and automate Forgejo package and attachment retention after the disk-pressure purge

coilyco-ops commented

2026-07-23 09:41:38 +00:00

Member

Context

The 2026-07-23 Forgejo DiskPressure incident required an operator-approved emergency purge. The inventory found 1,149 package versions totaling 170,495,076,359 logical bytes and 4,462 issue, comment, and release attachments totaling 51,871,068,871 bytes older than seven days. The purge removed or verified absent every targeted object with zero failures, Forgejo cleanup completed, and the root filesystem recovered to 56% utilization.

The seven-day cutoff was an emergency recovery choice, not a durable asset policy.

Related incident and prior storage work: #596, #488, and #490.

Policy decisions

Kai defines separate retention rules for:

Package versions, including protected versions and minimum versions retained per package.
Release assets, including whether tagged releases receive longer or indefinite retention.
Issue and comment attachments.
Legal, audit, or project-specific holds.
User-visible expectations before automated deletion.

Implementation

The implementation uses Forgejo APIs or supported cleanup mechanisms, never direct filesystem deletion.
A dry run reports object counts and logical bytes by class and owner before deletion.
Scheduled cleanup is bounded, auditable, and safe to retry.
Disk thresholds can trigger warning and escalation before Forgejo reaches DiskPressure.
Operators can pause cleanup and protect explicit exceptions.
Runbooks document emergency and routine paths separately.

Done when

Kai approves a documented retention policy for each asset class.
Infrastructure enforces the policy automatically with dry-run evidence and audit output.
Monitoring warns early enough to avoid another Forgejo outage.
A non-destructive validation demonstrates the policy against current Forgejo data.

## Context The 2026-07-23 Forgejo DiskPressure incident required an operator-approved emergency purge. The inventory found 1,149 package versions totaling 170,495,076,359 logical bytes and 4,462 issue, comment, and release attachments totaling 51,871,068,871 bytes older than seven days. The purge removed or verified absent every targeted object with zero failures, Forgejo cleanup completed, and the root filesystem recovered to 56% utilization. The seven-day cutoff was an emergency recovery choice, not a durable asset policy. Related incident and prior storage work: #596, #488, and #490. ## Policy decisions Kai defines separate retention rules for: * Package versions, including protected versions and minimum versions retained per package. * Release assets, including whether tagged releases receive longer or indefinite retention. * Issue and comment attachments. * Legal, audit, or project-specific holds. * User-visible expectations before automated deletion. ## Implementation * The implementation uses Forgejo APIs or supported cleanup mechanisms, never direct filesystem deletion. * A dry run reports object counts and logical bytes by class and owner before deletion. * Scheduled cleanup is bounded, auditable, and safe to retry. * Disk thresholds can trigger warning and escalation before Forgejo reaches DiskPressure. * Operators can pause cleanup and protect explicit exceptions. * Runbooks document emergency and routine paths separately. ## Done when * Kai approves a documented retention policy for each asset class. * Infrastructure enforces the policy automatically with dry-run evidence and audit output. * Monitoring warns early enough to avoid another Forgejo outage. * A non-destructive validation demonstrates the policy against current Forgejo data.

coilyco-ops added the

labels

2026-07-23 09:41:38 +00:00

coilyco-ops referenced this issue

2026-07-24 04:18:58 +00:00

Set measured ephemeral-storage budgets for Forgejo and PostgreSQL #616

coilyco-ops referenced this issue

2026-07-24 04:18:58 +00:00

Alert when the Forgejo runner recycle stops completing #617

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-24 06:20:54 +00:00

Add guarded Forgejo Actions artifact API operations #697

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-24 06:47:54 +00:00

dev-base registry retention: kai-server at 87% disk, dev-base tags likely retained forever #493

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-24 06:48:05 +00:00

Add Forgejo package cleanup coverage for dev-base draft tags #534

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-24 06:48:14 +00:00

Releases are blocked: a dotnet image-push flake skips the release job, so no tag and no ward-specs bundle (stuck at v0.248.0) #490

coilyco-ops commented

2026-07-25 05:56:26 +00:00

Author

Member

2026-07-24 retention evidence

Infrastructure#642 records the next-day recurrence and the completed 24-hour emergency purge.

Kai explicitly directed the operator to drop every package version older than 24 hours for this incident and accepted the recovery cost. The operator removed 1,072 user-visible versions through the Forgejo API, ran the supported cleanup_packages task, and verified that zero user-visible versions and zero unreferenced blobs older than the cutoff remain.

Implementation evidence for this issue:

A midnight-only cleanup can retain nearly two days of writes, which is unsafe at the observed growth rate.
The internal container _upload sentinel returns 404 through the ordinary package-version route. Forgejo's supported cleanup task must own its expired files.
The publisher regenerated roughly 79.8 GiB and 688 versions in about one day. Agentic-os#739 owns reducing that producer-side churn.
The dry run must distinguish logical referenced bytes from physically reclaimable deduplicated blobs.
Attachments were outside this purge and still need a separate durable policy decision.

Today's 24-hour instruction is authoritative for the emergency recovery. This issue still owns whether Kai adopts it as the unattended permanent package policy and how exceptions, cadence, and audit output work.

## 2026-07-24 retention evidence Infrastructure#642 records the next-day recurrence and the completed 24-hour emergency purge. Kai explicitly directed the operator to drop every package version older than 24 hours for this incident and accepted the recovery cost. The operator removed 1,072 user-visible versions through the Forgejo API, ran the supported `cleanup_packages` task, and verified that zero user-visible versions and zero unreferenced blobs older than the cutoff remain. Implementation evidence for this issue: * A midnight-only cleanup can retain nearly two days of writes, which is unsafe at the observed growth rate. * The internal container `_upload` sentinel returns 404 through the ordinary package-version route. Forgejo's supported cleanup task must own its expired files. * The publisher regenerated roughly 79.8 GiB and 688 versions in about one day. Agentic-os#739 owns reducing that producer-side churn. * The dry run must distinguish logical referenced bytes from physically reclaimable deduplicated blobs. * Attachments were outside this purge and still need a separate durable policy decision. Today's 24-hour instruction is authoritative for the emergency recovery. This issue still owns whether Kai adopts it as the unattended permanent package policy and how exceptions, cadence, and audit output work.

coilyco-ops referenced this issue

2026-07-25 06:18:04 +00:00

Reverse #543 and migrate dev-base consumers to the single :release tag #643

coilyco-ops referenced this issue

2026-07-26 05:55:54 +00:00

Make Forgejo runner scratch classification and recycle busy-aware #650

coilyco-ops added

and removed

labels

2026-07-29 06:27:51 +00:00

coilyco-ops commented

2026-08-01 16:45:55 +00:00

Author

Member

2026-08-01 retention evidence: the package store is 120 GB referenced and only 149 MB expired-unreferenced, so garbage collection alone cannot recover the node. coilyco-flight-deck/container owns 117 GB. The agentic-os package alone owns 116 GB across 749 external tags. The owner has 823 container versions, 820 older than 24 hours, and no package cleanup rules.\n\nThe proposed supported Forgejo rule for coilyco-flight-deck/container is enabled=true, keep_count=50, keep_pattern=(?:latest|release|buildcache|lang-(?:node|go|dotnet|rust|python)-(?:release|buildcache)), remove_days=0, remove_pattern=.+, match_full_name=false. Forgejo 15 restricts remove_days to 0, 7, 14, 30, 60, 90, or 180. Seven days is unsafe at the observed growth rate. Keep-count 50 leaves smaller packages unchanged and bounds agentic-os while preserving moving release and build-cache aliases.\n\nNo rule has been enabled and no package version has been deleted. The current wall is a fresh Forgejo backup plus Kai's explicit destructive approval after preview.

coilyco-ops commented

2026-08-01 16:55:30 +00:00

Author

Member

Guarded immediate-relief tooling landed on main in 0d1b5a0 (infrastructure#719). The script previews coilyco-flight-deck/container/agentic-os retention, preserves the newest 50 external versions plus latest/release/buildcache and every specialist release/buildcache alias, and requires an exact confirmation plus a successful restic backup no more than four hours old before using Forgejo package DELETE endpoints. It then launches the supported cleanup_packages task. No package version has been deleted yet. The next checkpoint is an attended live preview, followed by explicit approval and a fresh backup before apply. Permanent built-in cleanup-rule configuration and attachment retention remain open here.

Guarded immediate-relief tooling landed on main in 0d1b5a0 (infrastructure#719). The script previews coilyco-flight-deck/container/agentic-os retention, preserves the newest 50 external versions plus latest/release/buildcache and every specialist release/buildcache alias, and requires an exact confirmation plus a successful restic backup no more than four hours old before using Forgejo package DELETE endpoints. It then launches the supported cleanup_packages task. No package version has been deleted yet. The next checkpoint is an attended live preview, followed by explicit approval and a fresh backup before apply. Permanent built-in cleanup-rule configuration and attachment retention remain open here.

coilyco-ops added

and removed

labels

2026-08-05 10:25:59 +00:00

coilyco-ops referenced this issue

2026-08-14 06:25:38 +00:00

Package retention keeps only 9 agentic-os image versions, so every exact-version CI pin dies within about a day #828