coilyco-flight-deck/infrastructure

Watch

Three local-path PVs stuck Released for ~80 days, their delete helper pod never found, ~7,200 warning events #901

New issue

Closed

opened 2026-08-22 22:05:25 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-22 22:05:25 +00:00

Owner

Found on 2026-08-22 while decommissioning the retired null-db stack (coilyco-bridge/deploy#677). Filed because it is adjacent, unowned, and I am not fixing it.

Observed

Three PersistentVolumes are stuck in Released and their reclaim never completes:

NAME                                       CAP   RECLAIM   STATUS     CLAIM                            AGE
pvc-4ad1d2a2-82dd-4637-9129-e8d61712c26a   1Gi   Delete    Released   forgejo/data-forgejo-runner-0    80d
pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f   1Gi   Delete    Released   forgejo/data-forgejo-runner-1    80d
pvc-e56c913b-a8c9-4b01-ba03-c6dd2d7ca579   1Gi   Delete    Released   openclaw/openclaw-home-pvc       79d

Each re-errors continuously from local-path-provisioner:

reason:  VolumeFailedDelete
message: failed to delete volume pvc-c1e38068-...: failed to delete volume
         pvc-c1e38068-...: pods "helper-pod-delete-pvc-c1e38068-..." not found
count:   2402
source:  rancher.io/local-path_local-path-provisioner-774c6665dc-s8m8t

Roughly 2,400 events apiece, so about 7,200 accumulated warnings. With deploy#677's orphaned ExternalSecrets now cleared and null-db gone, these are the largest remaining source of standing Warning noise on the cluster.

Mechanism

local-path-provisioner reclaims a volume by spawning a short-lived helper-pod-delete-<pv> that wipes the backing directory, then removes the PV. The helper pod is not being found, so the provisioner never gets its completion signal, never deletes the PV, and retries on its backoff forever.

Not established from here: whether the helper pod fails to schedule, is created and reaped before the provisioner observes it, or is blocked by something in the local-path-storage config. That distinction is the fix, and it needs provisioner logs, which I did not read.

Why it is not merely noise

The claims are dead. data-forgejo-runner-0 and -1 belong to forgejo-runner, a StatefulSet that is currently at 0 replicas and has been scaled down since well before the demo lockdown. openclaw has no pods in the cluster at all. So nothing is going to reattach these.

Two costs while they sit there:

Disk is not reclaimed. The backing directories under /var/lib/rancher/k3s/storage/ are still occupied. Only ~3 GiB here, so this is not the main disk story, but it is 3 GiB that the cluster believes it has released and has not.
The retry loop is masking. A genuine VolumeFailedDelete on a volume someone cares about would land in an event stream already carrying 7,200 identical warnings, which is exactly the condition under which it gets missed.

Contrast: the reclaim path does work

Worth recording, because it narrows the cause. The null-db PV deleted today (pvc-ae288d3a-..., 5Gi, local-path, Delete) went Bound to Released to fully removed within about a minute, with no VolumeFailedDelete and no stuck directory. Verified by get pv returning NotFound and by the node's filesystem pressure dropping.

So local-path reclaim is healthy in general today. Whatever broke these three was specific to them, or specific to conditions around 79 to 80 days ago. Both remaining claims trace to workloads that were scaled to zero or removed rather than deleted cleanly, which is a plausible shared trigger worth checking first.

Suggested

Read local-path-provisioner logs for one of the three PV names to establish why the helper pod is missing rather than guessing between the three candidates above.
If the directories are confirmed dead, clear the stuck PVs. Removing the kubernetes.io/pv-protection finalizer is the blunt path and leaves the on-disk directory behind, so the directory wants removing too, in that order.
Decide whether openclaw should still have a namespace at all. It has a PVC claim and no workloads.

coilyco-bridge/deploy#677 - the orphaned-ExternalSecret sweep this was found alongside, now nearly closed
coilyco-flight-deck/agentic-os#1060 - dev-base release blocked with kai-server root at 87% critical. These 3 GiB will not resolve that, but the two share a disk-accounting story and are worth reading together

Found on 2026-08-22 while decommissioning the retired `null-db` stack (`coilyco-bridge/deploy#677`). Filed because it is adjacent, unowned, and I am not fixing it. ## Observed Three PersistentVolumes are stuck in `Released` and their reclaim never completes: ``` NAME CAP RECLAIM STATUS CLAIM AGE pvc-4ad1d2a2-82dd-4637-9129-e8d61712c26a 1Gi Delete Released forgejo/data-forgejo-runner-0 80d pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f 1Gi Delete Released forgejo/data-forgejo-runner-1 80d pvc-e56c913b-a8c9-4b01-ba03-c6dd2d7ca579 1Gi Delete Released openclaw/openclaw-home-pvc 79d ``` Each re-errors continuously from `local-path-provisioner`: ``` reason: VolumeFailedDelete message: failed to delete volume pvc-c1e38068-...: failed to delete volume pvc-c1e38068-...: pods "helper-pod-delete-pvc-c1e38068-..." not found count: 2402 source: rancher.io/local-path_local-path-provisioner-774c6665dc-s8m8t ``` **Roughly 2,400 events apiece, so about 7,200 accumulated warnings.** With `deploy#677`'s orphaned ExternalSecrets now cleared and `null-db` gone, these are the largest remaining source of standing Warning noise on the cluster. ## Mechanism `local-path-provisioner` reclaims a volume by spawning a short-lived `helper-pod-delete-<pv>` that wipes the backing directory, then removes the PV. The helper pod is not being found, so the provisioner never gets its completion signal, never deletes the PV, and retries on its backoff forever. Not established from here: whether the helper pod fails to schedule, is created and reaped before the provisioner observes it, or is blocked by something in the `local-path-storage` config. That distinction is the fix, and it needs provisioner logs, which I did not read. ## Why it is not merely noise The claims are dead. `data-forgejo-runner-0` and `-1` belong to `forgejo-runner`, a StatefulSet that is currently at **0 replicas** and has been scaled down since well before the demo lockdown. `openclaw` has no pods in the cluster at all. So nothing is going to reattach these. **Two costs while they sit there:** 1. **Disk is not reclaimed.** The backing directories under `/var/lib/rancher/k3s/storage/` are still occupied. Only ~3 GiB here, so this is not the main disk story, but it is 3 GiB that the cluster believes it has released and has not. 2. **The retry loop is masking.** A genuine `VolumeFailedDelete` on a volume someone cares about would land in an event stream already carrying 7,200 identical warnings, which is exactly the condition under which it gets missed. ## Contrast: the reclaim path does work Worth recording, because it narrows the cause. The `null-db` PV deleted today (`pvc-ae288d3a-...`, 5Gi, local-path, `Delete`) went `Bound` to `Released` to fully removed within about a minute, with no `VolumeFailedDelete` and no stuck directory. Verified by `get pv` returning `NotFound` and by the node's filesystem pressure dropping. **So local-path reclaim is healthy in general today.** Whatever broke these three was specific to them, or specific to conditions around 79 to 80 days ago. Both remaining claims trace to workloads that were scaled to zero or removed rather than deleted cleanly, which is a plausible shared trigger worth checking first. ## Suggested 1. Read `local-path-provisioner` logs for one of the three PV names to establish why the helper pod is missing rather than guessing between the three candidates above. 2. If the directories are confirmed dead, clear the stuck PVs. Removing the `kubernetes.io/pv-protection` finalizer is the blunt path and leaves the on-disk directory behind, so the directory wants removing too, in that order. 3. Decide whether `openclaw` should still have a namespace at all. It has a PVC claim and no workloads. ## Related * `coilyco-bridge/deploy#677` - the orphaned-ExternalSecret sweep this was found alongside, now nearly closed * `coilyco-flight-deck/agentic-os#1060` - dev-base release blocked with kai-server root at 87% critical. These 3 GiB will not resolve that, but the two share a disk-accounting story and are worth reading together

coilyco-ops commented

2026-08-25 05:45:37 +00:00

Author

Owner

Cleared. VolumeFailedDelete has stopped entirely.

What they were

Three PVs, Released for 82 days, reclaim=Delete, finalizer kubernetes.io/pv-protection:

pvc-4ad1d2a2...  forgejo/data-forgejo-runner-0
pvc-c1e38068...  forgejo/data-forgejo-runner-1
pvc-e56c913b...  openclaw/openclaw-home-pvc

The event count had climbed to 2,624 each by the time I looked, up from the 2,014 in the original title, so this was still actively accumulating.

Verified safe before deleting

data-forgejo-runner-0 and -1 still exist and are Bound, but to different PVs (pvc-435457c7, pvc-eb193ec4, both 73d). The stuck pair are the older volumes from before the runner StatefulSet was recreated. Nothing references them.
The openclaw namespace does not exist at all.
None of the three data directories exist on disk. Confirmed through the privileged node-stats volume scan rather than a shell check, because /var/lib/rancher/k3s/storage is 0700 root and a test -d as kai returns a false negative. The scan also reports zero unattributed paths.

So this was pure API-object cleanup. No data at risk and no disk to reclaim: the provisioner had been retrying deletes of directories that were already gone, which is why it produced only noise.

What was done

Patched persistentVolumeReclaimPolicy to Retain first, so the provisioner stopped attempting the delete that could never succeed, then deleted the three PV objects.

After

kubectl get events -A --field-selector reason=VolumeFailedDelete   ->  (nothing)
kubectl get pv                                                     ->  41 Bound, 0 Released
kubectl get pvc -n forgejo | grep runner                           ->  all Bound

Roughly 7,900 warning events per cycle stop being generated. Closing.

Cleared. `VolumeFailedDelete` has stopped entirely. ## What they were Three PVs, Released for 82 days, `reclaim=Delete`, finalizer `kubernetes.io/pv-protection`: ``` pvc-4ad1d2a2... forgejo/data-forgejo-runner-0 pvc-c1e38068... forgejo/data-forgejo-runner-1 pvc-e56c913b... openclaw/openclaw-home-pvc ``` The event count had climbed to **2,624 each** by the time I looked, up from the 2,014 in the original title, so this was still actively accumulating. ## Verified safe before deleting * `data-forgejo-runner-0` and `-1` **still exist and are Bound**, but to different PVs (`pvc-435457c7`, `pvc-eb193ec4`, both 73d). The stuck pair are the older volumes from before the runner StatefulSet was recreated. Nothing references them. * The `openclaw` namespace **does not exist** at all. * None of the three data directories exist on disk. Confirmed through the privileged node-stats volume scan rather than a shell check, because `/var/lib/rancher/k3s/storage` is `0700 root` and a `test -d` as `kai` returns a false negative. The scan also reports zero unattributed paths. So this was pure API-object cleanup. No data at risk and no disk to reclaim: the provisioner had been retrying deletes of directories that were already gone, which is why it produced only noise. ## What was done Patched `persistentVolumeReclaimPolicy` to `Retain` first, so the provisioner stopped attempting the delete that could never succeed, then deleted the three PV objects. ## After ``` kubectl get events -A --field-selector reason=VolumeFailedDelete -> (nothing) kubectl get pv -> 41 Bound, 0 Released kubectl get pvc -n forgejo | grep runner -> all Bound ``` Roughly 7,900 warning events per cycle stop being generated. Closing.

coilyco-ops closed this issue

2026-08-25 05:45:38 +00:00

coilyco-ops referenced this issue

2026-08-25 05:46:42 +00:00

kai-server: root filesystem chronically at the 85% critical line - plan a disk capacity increase #870

coilyco-ops referenced this issue

2026-08-25 05:46:44 +00:00

ser8: root filesystem climbing ~10% -> ~54% over 30 days - attribute the growth #871

coilyco-ops referenced this issue

2026-08-25 05:46:45 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846

coilyco-ops referenced this issue