Attended: classify recurring kai-server disk pressure #619

New issue

Closed

opened 2026-07-24 04:26:14 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-07-24 04:26:14 +00:00

Owner

Evidence:

Disk pressure transitioned 30 times over 10 days.
Kubelet recorded more than 2,800 failed image-GC attempts because almost nothing was eligible for collection.
Forgejo, its database, ingress endpoint, registry, and runner pool are healthy again.
infrastructure#618 repaired the detector and passed both post-push workflows.

Attended completion:

Converge the repaired watcher role onto ser8 and kai-server.
Run the detailed read-only classifier from docs/kai-disk-pressure-watch.md.
Choose any cleanup only after mapping the measured owner and confirming recoverability under docs/kai-disk-pressure-drill.md.
Verify root pressure is below 85% and affected service data is intact.

No cleanup, prune, PVC mutation, or live Ansible converge was attempted during the incident response.

The 2026-07-23 Forgejo outage was an ephemeral-storage eviction during kai-server root filesystem pressure. Forgejo recovered after kubelet cleared the disk-pressure taint, but the underlying node remains at 89.7% used with about 50 GiB free. Evidence: * Disk pressure transitioned 30 times over 10 days. * Kubelet recorded more than 2,800 failed image-GC attempts because almost nothing was eligible for collection. * Forgejo, its database, ingress endpoint, registry, and runner pool are healthy again. * infrastructure#618 repaired the detector and passed both post-push workflows. Attended completion: * Converge the repaired watcher role onto ser8 and kai-server. * Run the detailed read-only classifier from docs/kai-disk-pressure-watch.md. * Choose any cleanup only after mapping the measured owner and confirming recoverability under docs/kai-disk-pressure-drill.md. * Verify root pressure is below 85% and affected service data is intact. No cleanup, prune, PVC mutation, or live Ansible converge was attempted during the incident response.

coilysiren added the

autonomy

live-collab

label

2026-07-24 04:26:21 +00:00

coilyco-ops referenced this issue

2026-07-25 05:55:38 +00:00

Incident report: third Forgejo package-pressure recurrence and 24-hour purge #642

coilyco-ops commented

2026-07-25 05:56:27 +00:00

Member

Attended recovery completed for the package-registry owner

Infrastructure#642 contains the full incident report.

Before: root 89.2% used, about 51.7 GiB available, Forgejo packages 110.9 GiB.
Action: the operator deleted the frozen set of user-visible package versions older than 24 hours through Forgejo's API and ran the supported package cleanup task.
After: root 71% used, 133.2 GiB available, Forgejo packages 29.2 GiB, inodes 13%.
Verification: zero user-visible versions and zero unreferenced blobs older than the cutoff remain.

The classification identified package-registry churn as the acute owner for this recurrence. The issue should remain open for watcher convergence and broader recurring-pressure work. Infrastructure#600 owns server-side retention, and agentic-os#739 owns the roughly 80 GiB per day publisher growth.

## Attended recovery completed for the package-registry owner Infrastructure#642 contains the full incident report. * Before: root 89.2% used, about 51.7 GiB available, Forgejo packages 110.9 GiB. * Action: the operator deleted the frozen set of user-visible package versions older than 24 hours through Forgejo's API and ran the supported package cleanup task. * After: root 71% used, 133.2 GiB available, Forgejo packages 29.2 GiB, inodes 13%. * Verification: zero user-visible versions and zero unreferenced blobs older than the cutoff remain. The classification identified package-registry churn as the acute owner for this recurrence. The issue should remain open for watcher convergence and broader recurring-pressure work. Infrastructure#600 owns server-side retention, and agentic-os#739 owns the roughly 80 GiB per day publisher growth.

coilyco-ops referenced this issue

2026-07-26 05:55:54 +00:00

Make Forgejo runner scratch classification and recycle busy-aware #650

coilyco-ops added the

priority

label

2026-07-29 03:46:27 +00:00

coilyco-ops closed this issue

2026-07-29 06:04:42 +00:00