coilyco-flight-deck/infrastructure

Watch

Three PVs retry deletion 1473 times each after their data is already gone #810

New issue

Closed

opened 2026-08-13 06:32:42 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-13 06:32:42 +00:00

Owner

What this is

Three PersistentVolumes on kai-server are stuck in a delete loop, each with 1473 VolumeFailedDelete events:

failed to delete volume pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f: failed to delete volume
pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f: pods "helper-pod-delete-pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f" not found

Affected PVs:

pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f
pvc-e56c913b-a8c9-4b01-ba03-c6dd2d7ca579
pvc-4ad1d2a2-82dd-4637-9129-e8d61712c26a

Source is rancher.io/local-path_local-path-provisioner-774c6665dc-s8m8t. The provisioner creates a short-lived helper pod to rm -rf the backing directory, and it cannot find the helper pod it expects.

The data is already gone

This is the reassuring part, and it changes the priority. A get_k3s_volume_usage scan of /var/lib/rancher/k3s/storage on kai-server returned:

45 volumes, none of them these three
unattributed_paths: [], unattributed_path_count: 0, with discovery_truncated: false

Every directory under the storage root maps to a known, live PVC. There is no orphaned data for these three, so this is not a disk leak — it is three PV API objects retrying a cleanup that has nothing left to clean.

Caveat: the usage scan itself timed out (timed_out: true, volume_results_truncated: true), so per-volume sizes are lower bounds. Directory discovery completed, which is what the above conclusion rests on.

Actual cost

~4400 warning events, regenerating indefinitely. Combined with the DNS warnings on the same node, ordinary event output is mostly noise, which is how a real warning goes unread.
Three PVs that will never leave the cluster on their own.
Whatever the retry loop costs the provisioner, which is small but not zero and runs forever.

What I did not determine

Which workloads these belonged to. The PVs are in default and the PVCs are gone, so the association is lost. Given the untracked null-* and docker-registry objects in default covered in #809, these are plausibly from the same era of removed workloads, but I have not established that.
How long this has run. 1473 retries with no visible retry interval in the event data. If the provisioner backs off, this could be months.
Why the helper pod is missing. Candidates: it was created and reaped before the provisioner looked, the provisioner lost a watch, or its namespace/RBAC changed under it. Distinguishing them needs local-path-provisioner logs, which I did not pull.

Suggested resolution

Once the three PVs are confirmed to correspond to nothing wanted, clearing the kubernetes.io/pv-protection finalizer lets them go. That is a deliberate operator action on objects with no data behind them, not something to do from an investigation, and it should be paired with a look at the provisioner logs so the loop does not simply recur on the next deletion.

Acceptance

The three PVs are gone, or the reason they must stay is recorded.
The next PV deletion on kai-server completes, i.e. the helper-pod failure was understood rather than worked around by finalizer removal.

Notes

No changes made.

Next owner

Ops.

## What this is Three PersistentVolumes on kai-server are stuck in a delete loop, each with **1473** `VolumeFailedDelete` events: ``` failed to delete volume pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f: failed to delete volume pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f: pods "helper-pod-delete-pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f" not found ``` Affected PVs: - `pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f` - `pvc-e56c913b-a8c9-4b01-ba03-c6dd2d7ca579` - `pvc-4ad1d2a2-82dd-4637-9129-e8d61712c26a` Source is `rancher.io/local-path_local-path-provisioner-774c6665dc-s8m8t`. The provisioner creates a short-lived helper pod to `rm -rf` the backing directory, and it cannot find the helper pod it expects. ## The data is already gone This is the reassuring part, and it changes the priority. A `get_k3s_volume_usage` scan of `/var/lib/rancher/k3s/storage` on kai-server returned: - **45 volumes**, none of them these three - `unattributed_paths: []`, `unattributed_path_count: 0`, with `discovery_truncated: false` Every directory under the storage root maps to a known, live PVC. There is no orphaned data for these three, so **this is not a disk leak** — it is three PV API objects retrying a cleanup that has nothing left to clean. Caveat: the usage scan itself timed out (`timed_out: true`, `volume_results_truncated: true`), so per-volume *sizes* are lower bounds. Directory *discovery* completed, which is what the above conclusion rests on. ## Actual cost - ~4400 warning events, regenerating indefinitely. Combined with the DNS warnings on the same node, ordinary event output is mostly noise, which is how a real warning goes unread. - Three PVs that will never leave the cluster on their own. - Whatever the retry loop costs the provisioner, which is small but not zero and runs forever. ## What I did not determine - **Which workloads these belonged to.** The PVs are in `default` and the PVCs are gone, so the association is lost. Given the untracked `null-*` and `docker-registry` objects in `default` covered in https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/809, these are plausibly from the same era of removed workloads, but I have not established that. - **How long this has run.** 1473 retries with no visible retry interval in the event data. If the provisioner backs off, this could be months. - **Why the helper pod is missing.** Candidates: it was created and reaped before the provisioner looked, the provisioner lost a watch, or its namespace/RBAC changed under it. Distinguishing them needs local-path-provisioner logs, which I did not pull. ## Suggested resolution Once the three PVs are confirmed to correspond to nothing wanted, clearing the `kubernetes.io/pv-protection` finalizer lets them go. That is a deliberate operator action on objects with no data behind them, not something to do from an investigation, and it should be paired with a look at the provisioner logs so the loop does not simply recur on the next deletion. ## Acceptance - The three PVs are gone, or the reason they must stay is recorded. - The next PV deletion on kai-server completes, i.e. the helper-pod failure was understood rather than worked around by finalizer removal. ## Notes No changes made. ## Next owner Ops.

coilyco-ops commented

2026-08-13 06:50:07 +00:00

Author

Owner

Correction: the owning claims are recorded, and I said they were not

The body says "the PVCs are gone, so the association is lost" and speculates the PVs relate to the null-* era. Both wrong. kubectl get pv carries the claim reference:

PV	Claim	Status	Age
`pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f`	`forgejo/data-forgejo-runner-1`	Released	70d
`pvc-e56c913b-a8c9-4b01-ba03-c6dd2d7ca579`	`openclaw/openclaw-home-pvc`	Released	70d
`pvc-4ad1d2a2-82dd-4637-9129-e8d61712c26a`	`forgejo/data-forgejo-runner-0`	Released	70d

All three are Released with Delete reclaim policy, which is why the provisioner keeps trying.

What that tells us

Two are superseded runner volumes. forgejo/data-forgejo-runner-0 and -1 are live, bound PVCs today — both appear in the current volume scan holding ~1.7GB each. These 70-day-old PVs are earlier generations from before the runner StatefulSet was recreated. New PVCs took the same names, bound to new PVs, and the old ones were left Released.

One belongs to a namespace with nothing in it. openclaw/openclaw-home-pvc — the openclaw namespace shows in the volume scan with 1 volume, 0 bytes, 0 pods. That looks like a fully decommissioned workload whose PV never finished leaving.

This also drops the guess linking them to the null-* objects in #809. Different origin — StatefulSet recreation and a decommission, not the untracked-default-namespace set. The two issues are independent.

What still holds

The core finding is unchanged and now better supported: three Released PVs with Delete policy, no backing data on disk, retrying a helper-pod deletion forever. Identifying them makes the cleanup decision easier rather than harder — none of the three has a live consumer, and the two runner volumes are provably superseded by bound PVCs of the same name.

The openclaw namespace is worth a separate look. An empty namespace holding a released PVC is either an incomplete decommission or something that was never finished.

## Correction: the owning claims are recorded, and I said they were not The body says *"the PVCs are gone, so the association is lost"* and speculates the PVs relate to the `null-*` era. Both wrong. `kubectl get pv` carries the claim reference: | PV | Claim | Status | Age | | --- | --- | --- | --- | | `pvc-c1e38068-b66e-4c5e-aae7-1866cb1a2f7f` | `forgejo/data-forgejo-runner-1` | Released | 70d | | `pvc-e56c913b-a8c9-4b01-ba03-c6dd2d7ca579` | `openclaw/openclaw-home-pvc` | Released | 70d | | `pvc-4ad1d2a2-82dd-4637-9129-e8d61712c26a` | `forgejo/data-forgejo-runner-0` | Released | 70d | All three are `Released` with `Delete` reclaim policy, which is why the provisioner keeps trying. ### What that tells us **Two are superseded runner volumes.** `forgejo/data-forgejo-runner-0` and `-1` are *live, bound PVCs today* — both appear in the current volume scan holding ~1.7GB each. These 70-day-old PVs are earlier generations from before the runner StatefulSet was recreated. New PVCs took the same names, bound to new PVs, and the old ones were left `Released`. **One belongs to a namespace with nothing in it.** `openclaw/openclaw-home-pvc` — the `openclaw` namespace shows in the volume scan with 1 volume, **0 bytes, 0 pods**. That looks like a fully decommissioned workload whose PV never finished leaving. This also drops the guess linking them to the `null-*` objects in https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/809. Different origin — StatefulSet recreation and a decommission, not the untracked-`default`-namespace set. The two issues are independent. ### What still holds The core finding is unchanged and now better supported: three `Released` PVs with `Delete` policy, no backing data on disk, retrying a helper-pod deletion forever. Identifying them makes the cleanup decision easier rather than harder — none of the three has a live consumer, and the two runner volumes are provably superseded by bound PVCs of the same name. The `openclaw` namespace is worth a separate look. An empty namespace holding a released PVC is either an incomplete decommission or something that was never finished.

coilyco-ops referenced this issue

2026-08-13 07:07:40 +00:00

The openclaw namespace is completely empty and has been for ~70 days #812

coilyco-ops referenced this issue

2026-08-25 06:09:58 +00:00

fix: recover stuck local-path PV deletion #588

coilyco-ops commented

2026-08-25 06:10:03 +00:00

Author

Owner

Resolved on 2026-08-25. Same three PVs this describes.

pvc-4ad1d2a2…  forgejo/data-forgejo-runner-0
pvc-c1e38068…  forgejo/data-forgejo-runner-1
pvc-e56c913b…  openclaw/openclaw-home-pvc

The event count had reached 2,629 each before deletion, up from the number recorded here, so this was still accumulating.

Verified safe first: the live data-forgejo-runner-0 and -1 claims are Bound to different, newer PVs, the openclaw namespace no longer exists, and none of the three data directories were on disk. That last one needed the privileged node-stats volume scan, because /var/lib/rancher/k3s/storage is 0700 root and a test -d as kai returns a false negative either way.

Patched persistentVolumeReclaimPolicy to Retain so the provisioner stopped retrying a delete that could never succeed, then removed the three PV objects.

After: VolumeFailedDelete generation stopped at the moment of deletion, 41 PVs Bound, 0 Released, all runner PVCs still Bound.

Worth recording for the backlog

This defect had five open issues covering it: this one, #810 or #588 (whichever you are not reading), #863, #901, and part of #846. I closed #863 and #901 earlier today without checking for older duplicates, which is exactly the failure a dedup pass would have caught. Closing the rest now.

Resolved on 2026-08-25. Same three PVs this describes. ``` pvc-4ad1d2a2… forgejo/data-forgejo-runner-0 pvc-c1e38068… forgejo/data-forgejo-runner-1 pvc-e56c913b… openclaw/openclaw-home-pvc ``` The event count had reached **2,629 each** before deletion, up from the number recorded here, so this was still accumulating. Verified safe first: the live `data-forgejo-runner-0` and `-1` claims are Bound to different, newer PVs, the `openclaw` namespace no longer exists, and none of the three data directories were on disk. That last one needed the privileged node-stats volume scan, because `/var/lib/rancher/k3s/storage` is `0700 root` and a `test -d` as `kai` returns a false negative either way. Patched `persistentVolumeReclaimPolicy` to `Retain` so the provisioner stopped retrying a delete that could never succeed, then removed the three PV objects. After: `VolumeFailedDelete` generation stopped at the moment of deletion, 41 PVs Bound, 0 Released, all runner PVCs still Bound. ## Worth recording for the backlog This defect had **five** open issues covering it: this one, #810 or #588 (whichever you are not reading), #863, #901, and part of #846. I closed #863 and #901 earlier today without checking for older duplicates, which is exactly the failure a dedup pass would have caught. Closing the rest now.

coilyco-ops closed this issue

2026-08-25 06:10:04 +00:00

coilyco-ops referenced this issue

2026-08-25 06:10:23 +00:00

The openclaw namespace is completely empty and has been for ~70 days #812

coilyco-ops referenced this issue

2026-08-25 06:10:26 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846