Incident report: third Forgejo package-pressure recurrence and 24-hour purge #642

New issue

Closed

opened 2026-07-25 05:55:38 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-25 05:55:38 +00:00

Member

Summary

kai-server reached Forgejo-driven root-disk pressure for the third consecutive pressure event. The 2026-07-24 evening inspection found root usage at 89.2% with about 51.7 GiB available. The Forgejo package store had regrown to 110.9 GiB after the prior emergency recovery.

Since infrastructure#600 was opened after the previous purge, Forgejo had accumulated roughly 79.8 GiB of new package blobs and 688 package versions. Kai explicitly authorized deletion of every package version older than 24 hours and accepted the recovery cost. The operator did not create a new backup before deletion. The unresolved backup gap remains infrastructure#599.

Related incidents: #596 and #619.

Frozen scope

The operator froze the cutoff at 2026-07-23 22:25:55 PDT before the first DELETE.

1,073 package-version database rows matched the cutoff.
13,971 package-file references matched the cutoff.
The matching versions represented about 500 GiB of logical referenced bytes because Forgejo deduplicates blobs.
The dry run estimated 92.6 GiB of physically reclaimable package blobs.
The operator excluded attachments, repositories, LFS, releases, runner scratch, logs, and game data from this purge.

Response

The operator deleted package versions through DELETE /api/v1/packages/{owner}/{type}/{name}/{version} with the Forgejo site-admin credential resolved through the guarded operator path.
Forgejo returned 1,072 successful HTTP 204 responses and one idempotent HTTP 404.
The remaining database row was the internal container _upload sentinel. Forgejo v15.0.2 does not expose that internal version through the ordinary package-delete route.
The operator triggered the supported cleanup_packages task through the admin cron API.
Forgejo removed expired internal upload files and old unreferenced package blobs. Forgejo retained recent upload files and unreferenced blobs younger than 24 hours.
The operator performed no direct filesystem or database deletion and restarted no service.

Verified outcome

Root usage fell from 89.2% to 71%.
Available root space rose from about 51.7 GiB to 133.2 GiB.
The Forgejo package store fell from 110.9 GiB to 29.2 GiB, reclaiming about 81.7 GiB physically.
Zero user-visible package versions older than the frozen cutoff remain.
Zero unreferenced package blobs older than the frozen cutoff remain.
Forgejo retained 72 unreferenced blobs totaling about 240 MiB because they were younger than 24 hours.
Forgejo retained 830 recent files in the internal _upload sentinel.
Root inode usage remained healthy at 13%.
Forgejo, PostgreSQL, the package registry, and the runner pool remained available after recovery.

Findings

The recurrence is byte growth, not inode exhaustion.
A 24-hour server-side retention window is necessary but insufficient while the publisher can regenerate roughly 80 GiB inside one window.
Forgejo package cleanup must treat internal _upload state through the supported cleanup task, not the ordinary package-version API.
The backup recovery and restore-drill gap materially increases the cost of every emergency purge.
Ward upgraded from 0.800.0 to 0.805.0 during the incident and removed the old operator surface before native Aguard was available on this workstation. The operator completed the guarded credential lookup with the still-installed 0.800.0 binary. Infrastructure#635 owns the native Aguard rollout.

Follow-ups

infrastructure#599 restores and drills Forgejo backup coverage.
infrastructure#600 records Kai's 24-hour package-retention decision and automates supported cleanup with dry-run evidence.
infrastructure#619 owns recurring node-pressure classification and live watcher convergence.
infrastructure#635 completes native Aguard availability on operator hosts.
agentic-os#739 measures and bounds publisher-side dev-base package churn.

Status

The incident is recovered. Root usage is below the configured 85% pressure threshold. The recurrence risk remains open in the linked follow-ups.

## Summary kai-server reached Forgejo-driven root-disk pressure for the third consecutive pressure event. The 2026-07-24 evening inspection found root usage at 89.2% with about 51.7 GiB available. The Forgejo package store had regrown to 110.9 GiB after the prior emergency recovery. Since infrastructure#600 was opened after the previous purge, Forgejo had accumulated roughly 79.8 GiB of new package blobs and 688 package versions. Kai explicitly authorized deletion of every package version older than 24 hours and accepted the recovery cost. The operator did not create a new backup before deletion. The unresolved backup gap remains infrastructure#599. Related incidents: #596 and #619. ## Frozen scope The operator froze the cutoff at 2026-07-23 22:25:55 PDT before the first DELETE. * 1,073 package-version database rows matched the cutoff. * 13,971 package-file references matched the cutoff. * The matching versions represented about 500 GiB of logical referenced bytes because Forgejo deduplicates blobs. * The dry run estimated 92.6 GiB of physically reclaimable package blobs. * The operator excluded attachments, repositories, LFS, releases, runner scratch, logs, and game data from this purge. ## Response * The operator deleted package versions through `DELETE /api/v1/packages/{owner}/{type}/{name}/{version}` with the Forgejo site-admin credential resolved through the guarded operator path. * Forgejo returned 1,072 successful HTTP 204 responses and one idempotent HTTP 404. * The remaining database row was the internal container `_upload` sentinel. Forgejo v15.0.2 does not expose that internal version through the ordinary package-delete route. * The operator triggered the supported `cleanup_packages` task through the admin cron API. * Forgejo removed expired internal upload files and old unreferenced package blobs. Forgejo retained recent upload files and unreferenced blobs younger than 24 hours. * The operator performed no direct filesystem or database deletion and restarted no service. ## Verified outcome * Root usage fell from 89.2% to 71%. * Available root space rose from about 51.7 GiB to 133.2 GiB. * The Forgejo package store fell from 110.9 GiB to 29.2 GiB, reclaiming about 81.7 GiB physically. * Zero user-visible package versions older than the frozen cutoff remain. * Zero unreferenced package blobs older than the frozen cutoff remain. * Forgejo retained 72 unreferenced blobs totaling about 240 MiB because they were younger than 24 hours. * Forgejo retained 830 recent files in the internal `_upload` sentinel. * Root inode usage remained healthy at 13%. * Forgejo, PostgreSQL, the package registry, and the runner pool remained available after recovery. ## Findings * The recurrence is byte growth, not inode exhaustion. * A 24-hour server-side retention window is necessary but insufficient while the publisher can regenerate roughly 80 GiB inside one window. * Forgejo package cleanup must treat internal `_upload` state through the supported cleanup task, not the ordinary package-version API. * The backup recovery and restore-drill gap materially increases the cost of every emergency purge. * Ward upgraded from 0.800.0 to 0.805.0 during the incident and removed the old operator surface before native Aguard was available on this workstation. The operator completed the guarded credential lookup with the still-installed 0.800.0 binary. Infrastructure#635 owns the native Aguard rollout. ## Follow-ups * infrastructure#599 restores and drills Forgejo backup coverage. * infrastructure#600 records Kai's 24-hour package-retention decision and automates supported cleanup with dry-run evidence. * infrastructure#619 owns recurring node-pressure classification and live watcher convergence. * infrastructure#635 completes native Aguard availability on operator hosts. * agentic-os#739 measures and bounds publisher-side dev-base package churn. ## Status The incident is recovered. Root usage is below the configured 85% pressure threshold. The recurrence risk remains open in the linked follow-ups.

coilyco-ops closed this issue

2026-07-25 05:56:30 +00:00

coilyco-ops referenced this issue

2026-07-25 06:18:04 +00:00

Reverse #543 and migrate dev-base consumers to the single :release tag #643

coilyco-ops referenced this issue

2026-07-26 05:55:54 +00:00

Make Forgejo runner scratch classification and recycle busy-aware #650