fix: recover stuck local-path PV deletion #588

New issue

Closed

opened 2026-07-23 04:35:44 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-23 04:35:44 +00:00

Owner

Problem

The cluster has three local-path PersistentVolumes stuck in Released with a Delete reclaim policy. Two belong to retired Forgejo runner claims and one belongs to a retired OpenClaw claim.

The local-path provisioner has emitted more than a thousand VolumeFailedDelete events over about 12 days because its cleanup helper pods are reported as not found. The failed cleanup leaves stale volume records and may leave backing directories on the node.

Scope

Confirm that each former claim and workload is retired.
Confirm that no retained state or backup is required.
Diagnose why the local-path cleanup helper lifecycle loses the helper pod.
Remove orphaned backing directories only after the safety checks pass.
Reconcile or remove the three stuck PV objects.
Prevent the same helper-pod failure from recurring.

Acceptance

The three released PVs no longer exist.
The node has no orphaned backing directories for those retired claims.
New local-path volume deletion completes without VolumeFailedDelete.
The cleanup procedure is documented without recording opaque volume identifiers.

## Problem The cluster has three local-path PersistentVolumes stuck in `Released` with a `Delete` reclaim policy. Two belong to retired Forgejo runner claims and one belongs to a retired OpenClaw claim. The local-path provisioner has emitted more than a thousand `VolumeFailedDelete` events over about 12 days because its cleanup helper pods are reported as not found. The failed cleanup leaves stale volume records and may leave backing directories on the node. ## Scope * Confirm that each former claim and workload is retired. * Confirm that no retained state or backup is required. * Diagnose why the local-path cleanup helper lifecycle loses the helper pod. * Remove orphaned backing directories only after the safety checks pass. * Reconcile or remove the three stuck PV objects. * Prevent the same helper-pod failure from recurring. ## Acceptance * The three released PVs no longer exist. * The node has no orphaned backing directories for those retired claims. * New local-path volume deletion completes without `VolumeFailedDelete`. * The cleanup procedure is documented without recording opaque volume identifiers.

coilyco-ops referenced this issue

2026-07-23 04:36:22 +00:00

kai-server unhealthy: forgejo bounce, runner crashloop, missing /github/pat, signoz/metrics 500s #308

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-07-29 03:46:25 +00:00

coilyco-ops commented

2026-08-25 06:09:58 +00:00

Author

Owner

Resolved on 2026-08-25. Same three PVs this describes.

pvc-4ad1d2a2…  forgejo/data-forgejo-runner-0
pvc-c1e38068…  forgejo/data-forgejo-runner-1
pvc-e56c913b…  openclaw/openclaw-home-pvc

The event count had reached 2,629 each before deletion, up from the number recorded here, so this was still accumulating.

Verified safe first: the live data-forgejo-runner-0 and -1 claims are Bound to different, newer PVs, the openclaw namespace no longer exists, and none of the three data directories were on disk. That last one needed the privileged node-stats volume scan, because /var/lib/rancher/k3s/storage is 0700 root and a test -d as kai returns a false negative either way.

Patched persistentVolumeReclaimPolicy to Retain so the provisioner stopped retrying a delete that could never succeed, then removed the three PV objects.

After: VolumeFailedDelete generation stopped at the moment of deletion, 41 PVs Bound, 0 Released, all runner PVCs still Bound.

Worth recording for the backlog

This defect had five open issues covering it: this one, #810 or #588 (whichever you are not reading), #863, #901, and part of #846. I closed #863 and #901 earlier today without checking for older duplicates, which is exactly the failure a dedup pass would have caught. Closing the rest now.

Resolved on 2026-08-25. Same three PVs this describes. ``` pvc-4ad1d2a2… forgejo/data-forgejo-runner-0 pvc-c1e38068… forgejo/data-forgejo-runner-1 pvc-e56c913b… openclaw/openclaw-home-pvc ``` The event count had reached **2,629 each** before deletion, up from the number recorded here, so this was still accumulating. Verified safe first: the live `data-forgejo-runner-0` and `-1` claims are Bound to different, newer PVs, the `openclaw` namespace no longer exists, and none of the three data directories were on disk. That last one needed the privileged node-stats volume scan, because `/var/lib/rancher/k3s/storage` is `0700 root` and a `test -d` as `kai` returns a false negative either way. Patched `persistentVolumeReclaimPolicy` to `Retain` so the provisioner stopped retrying a delete that could never succeed, then removed the three PV objects. After: `VolumeFailedDelete` generation stopped at the moment of deletion, 41 PVs Bound, 0 Released, all runner PVCs still Bound. ## Worth recording for the backlog This defect had **five** open issues covering it: this one, #810 or #588 (whichever you are not reading), #863, #901, and part of #846. I closed #863 and #901 earlier today without checking for older duplicates, which is exactly the failure a dedup pass would have caught. Closing the rest now.

coilyco-ops closed this issue

2026-08-25 06:10:00 +00:00

coilyco-ops referenced this issue

2026-08-25 06:10:03 +00:00

Three PVs retry deletion 1473 times each after their data is already gone #810

coilyco-ops referenced this issue

2026-08-25 06:10:23 +00:00

The openclaw namespace is completely empty and has been for ~70 days #812

coilyco-ops referenced this issue

2026-08-25 06:10:26 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846