local-path provisioner is stuck in a VolumeFailedDelete loop on three orphaned PVs, 2,014 events each #863

New issue

Closed

opened 2026-08-18 20:15:38 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-18 20:15:38 +00:00

Owner

Found during a fleet health sweep on 2026-08-18.

Observed

The local-path provisioner on kai-server is in a permanent retry loop against three PersistentVolumes it cannot delete:

PersistentVolume default/pvc-4ad1d2a2-82dd-4637-9129-e8d61712c26a   VolumeFailedDelete   count 2014
PersistentVolume default/pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f  VolumeFailedDelete   count 2014
PersistentVolume default/pvc-e56c913b-a8c9-4b01-ba03-c6dd2d7ca579  VolumeFailedDelete   count 2014

Each with the same message shape:

failed to delete volume pvc-<uid>: failed to delete volume pvc-<uid>:
pods "helper-pod-delete-pvc-<uid>" not found

Source is rancher.io/local-path_local-path-provisioner-774c6665dc-s8m8t.

Reading

The provisioner spawns a helper pod to remove the on-disk directory, and the helper pod is not there when it looks. The PVs stay in Released or Failed and it retries forever. 2,014 events each and climbing.

This is plausibly connected to kube-system/local-path-provisioner sitting at 39 restarts, though that is over a 492d lifetime, so the rate is low and the link is inference rather than established.

Two costs: standing event noise that buries real warnings, and backing directories under the local-path root that are never reclaimed, which matters on a node that has had disk-pressure history.

Suggested

Identify the backing directories for the three PVs, confirm nothing needs the data, remove the directories on the node, then clear the PV finalizers so the objects can go. Worth checking whether the helper-pod path is broken generally, since if it is, the next PVC deletion adds a fourth.

Found during a fleet health sweep on 2026-08-18. ## Observed The local-path provisioner on kai-server is in a permanent retry loop against three PersistentVolumes it cannot delete: ``` PersistentVolume default/pvc-4ad1d2a2-82dd-4637-9129-e8d61712c26a VolumeFailedDelete count 2014 PersistentVolume default/pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f VolumeFailedDelete count 2014 PersistentVolume default/pvc-e56c913b-a8c9-4b01-ba03-c6dd2d7ca579 VolumeFailedDelete count 2014 ``` Each with the same message shape: ``` failed to delete volume pvc-<uid>: failed to delete volume pvc-<uid>: pods "helper-pod-delete-pvc-<uid>" not found ``` Source is `rancher.io/local-path_local-path-provisioner-774c6665dc-s8m8t`. ## Reading The provisioner spawns a helper pod to remove the on-disk directory, and the helper pod is not there when it looks. The PVs stay in `Released` or `Failed` and it retries forever. 2,014 events each and climbing. This is plausibly connected to `kube-system/local-path-provisioner` sitting at 39 restarts, though that is over a 492d lifetime, so the rate is low and the link is inference rather than established. Two costs: standing event noise that buries real warnings, and backing directories under the local-path root that are never reclaimed, which matters on a node that has had disk-pressure history. ## Suggested Identify the backing directories for the three PVs, confirm nothing needs the data, remove the directories on the node, then clear the PV finalizers so the objects can go. Worth checking whether the helper-pod path is broken generally, since if it is, the next PVC deletion adds a fourth.

coilyco-ops referenced this issue

2026-08-19 11:42:26 +00:00

kai-server: root filesystem chronically at the 85% critical line - plan a disk capacity increase #870

coilyco-ops commented

2026-08-25 05:45:35 +00:00

Author

Owner

Cleared. VolumeFailedDelete has stopped entirely.

What they were

Three PVs, Released for 82 days, reclaim=Delete, finalizer kubernetes.io/pv-protection:

pvc-4ad1d2a2...  forgejo/data-forgejo-runner-0
pvc-c1e38068...  forgejo/data-forgejo-runner-1
pvc-e56c913b...  openclaw/openclaw-home-pvc

The event count had climbed to 2,624 each by the time I looked, up from the 2,014 in the original title, so this was still actively accumulating.

Verified safe before deleting

data-forgejo-runner-0 and -1 still exist and are Bound, but to different PVs (pvc-435457c7, pvc-eb193ec4, both 73d). The stuck pair are the older volumes from before the runner StatefulSet was recreated. Nothing references them.
The openclaw namespace does not exist at all.
None of the three data directories exist on disk. Confirmed through the privileged node-stats volume scan rather than a shell check, because /var/lib/rancher/k3s/storage is 0700 root and a test -d as kai returns a false negative. The scan also reports zero unattributed paths.

So this was pure API-object cleanup. No data at risk and no disk to reclaim: the provisioner had been retrying deletes of directories that were already gone, which is why it produced only noise.

What was done

Patched persistentVolumeReclaimPolicy to Retain first, so the provisioner stopped attempting the delete that could never succeed, then deleted the three PV objects.

After

kubectl get events -A --field-selector reason=VolumeFailedDelete   ->  (nothing)
kubectl get pv                                                     ->  41 Bound, 0 Released
kubectl get pvc -n forgejo | grep runner                           ->  all Bound

Roughly 7,900 warning events per cycle stop being generated. Closing.

Cleared. `VolumeFailedDelete` has stopped entirely. ## What they were Three PVs, Released for 82 days, `reclaim=Delete`, finalizer `kubernetes.io/pv-protection`: ``` pvc-4ad1d2a2... forgejo/data-forgejo-runner-0 pvc-c1e38068... forgejo/data-forgejo-runner-1 pvc-e56c913b... openclaw/openclaw-home-pvc ``` The event count had climbed to **2,624 each** by the time I looked, up from the 2,014 in the original title, so this was still actively accumulating. ## Verified safe before deleting * `data-forgejo-runner-0` and `-1` **still exist and are Bound**, but to different PVs (`pvc-435457c7`, `pvc-eb193ec4`, both 73d). The stuck pair are the older volumes from before the runner StatefulSet was recreated. Nothing references them. * The `openclaw` namespace **does not exist** at all. * None of the three data directories exist on disk. Confirmed through the privileged node-stats volume scan rather than a shell check, because `/var/lib/rancher/k3s/storage` is `0700 root` and a `test -d` as `kai` returns a false negative. The scan also reports zero unattributed paths. So this was pure API-object cleanup. No data at risk and no disk to reclaim: the provisioner had been retrying deletes of directories that were already gone, which is why it produced only noise. ## What was done Patched `persistentVolumeReclaimPolicy` to `Retain` first, so the provisioner stopped attempting the delete that could never succeed, then deleted the three PV objects. ## After ``` kubectl get events -A --field-selector reason=VolumeFailedDelete -> (nothing) kubectl get pv -> 41 Bound, 0 Released kubectl get pvc -n forgejo | grep runner -> all Bound ``` Roughly 7,900 warning events per cycle stop being generated. Closing.

coilyco-ops closed this issue

2026-08-25 05:45:36 +00:00

coilyco-ops referenced this issue

2026-08-25 05:46:42 +00:00

kai-server: root filesystem chronically at the 85% critical line - plan a disk capacity increase #870

coilyco-ops referenced this issue

2026-08-25 05:46:44 +00:00

ser8: root filesystem climbing ~10% -> ~54% over 30 days - attribute the growth #871

coilyco-ops referenced this issue

2026-08-25 05:46:45 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846

coilyco-ops referenced this issue