fix(storage): stop the wide usage profiles from OOMKilling the container #27

Merged

coilysiren merged 1 commit from aos/claude/uq98-scan-memory into main

2026-08-17 19:28:04 +00:00

coilyco-ops commented

2026-08-17 18:35:27 +00:00

Owner

Closes part of #26 (root causes A, B, and C).

Root cause A - the OOM is structural, not a budget mismatch

_scan_tree kept every scanned file inode in its dedup set, so the set grew with the tree rather than with hard links. Measured locally: 162 MB per million (dev, ino) tuples, so the configured 5,000,000-entry budget needs roughly 810 MB for the dedup set alone, before any traversal state, against a 512Mi limit. That is why root, var, and var-lib died at four minutes every time with exit 137 while the bounded profiles finished fine.

The fix removes the term rather than re-tuning around it:

Track inodes only when st_nlink > 1. Only a multiply-linked file can be reached twice, and same-filesystem bind and subtree mounts are already excluded by _mount_exclusions. This is the line du draws and it leaves the documented hard-link contract in host-storage-snapshot.md unchanged. Live evidence: the current k3s-storage snapshot reports deduplicated_entries: 202 across 344,464 entries, all of it in the Docker layer store.
Pack the key into one int - 114 MB per million rather than 162 MB, collision-free via a 64-bit shift.
Carry the traversal stack as str rather than Path - 153 vs 362 bytes per entry.
Report hardlinked_inodes_tracked so the remaining cost is observable rather than inferred.

Config and limit now agree because the scan fits, so no deploy change is proposed for the limit. If a live root scan still dies, that is new evidence and a different fix.

Root cause C - the pod-ephemeral domain

Adds a pod-ephemeral profile at /var/lib/kubelet/pods. Mount exclusion already drops the local-path PVCs kubelet bind-mounts into that tree, so the profile does not re-count k3s-storage bytes - pinned by a new test. Nothing is needed in deploy: it sets no NODE_STATS_HOST_USAGE_PROFILES override, so the new default ships with the image.

Root cause B - please read before merging

I did not add /var/lib/kubelet to NODE_STATS_READABLE_ROOTS, and I recommend against it. That allowlist gates read_text_head as well as stat_path through the shared _resolve_readable. /var/lib/kubelet/pods/*/volumes/kubernetes.io~secret/ and ~projected/ hold decrypted Kubernetes Secrets and service-account tokens as plaintext files, so granting the root would make every mounted secret on the node readable through an MCP tool call. That contradicts this repo's own AGENTS.md safety rule.

The pod-ephemeral profile answers what the incident actually needed - byte attribution under that tree - without any file-read grant. If path-level stat there is still wanted, the clean shape is a separate stat-only allowlist rather than widening this one, and that is worth its own issue.

Verification

ward exec lint, ward exec test (46 passed, 2 new), and ward exec precommit are all clean.

Opened as a PR rather than pushed to main: the merge-remote-main workflow applies, but the direct push to main was refused by the harness, so this branch is the durable checkpoint.

🤖 Generated with Claude Code

Closes part of #26 (root causes A, B, and C). ## Root cause A - the OOM is structural, not a budget mismatch `_scan_tree` kept **every** scanned file inode in its dedup set, so the set grew with the tree rather than with hard links. Measured locally: **162 MB per million `(dev, ino)` tuples**, so the configured 5,000,000-entry budget needs roughly **810 MB** for the dedup set alone, before any traversal state, against a 512Mi limit. That is why `root`, `var`, and `var-lib` died at four minutes every time with exit 137 while the bounded profiles finished fine. The fix removes the term rather than re-tuning around it: * **Track inodes only when `st_nlink > 1`.** Only a multiply-linked file can be reached twice, and same-filesystem bind and subtree mounts are already excluded by `_mount_exclusions`. This is the line `du` draws and it leaves the documented hard-link contract in `host-storage-snapshot.md` unchanged. Live evidence: the current `k3s-storage` snapshot reports `deduplicated_entries: 202` across 344,464 entries, all of it in the Docker layer store. * **Pack the key into one int** - 114 MB per million rather than 162 MB, collision-free via a 64-bit shift. * **Carry the traversal stack as `str` rather than `Path`** - 153 vs 362 bytes per entry. * **Report `hardlinked_inodes_tracked`** so the remaining cost is observable rather than inferred. Config and limit now agree because the scan fits, so no `deploy` change is proposed for the limit. If a live `root` scan still dies, that is new evidence and a different fix. ## Root cause C - the pod-ephemeral domain Adds a `pod-ephemeral` profile at `/var/lib/kubelet/pods`. Mount exclusion already drops the local-path PVCs kubelet bind-mounts into that tree, so the profile does not re-count `k3s-storage` bytes - pinned by a new test. Nothing is needed in `deploy`: it sets no `NODE_STATS_HOST_USAGE_PROFILES` override, so the new default ships with the image. ## Root cause B - please read before merging **I did not add `/var/lib/kubelet` to `NODE_STATS_READABLE_ROOTS`, and I recommend against it.** That allowlist gates `read_text_head` as well as `stat_path` through the shared `_resolve_readable`. `/var/lib/kubelet/pods/*/volumes/kubernetes.io~secret/` and `~projected/` hold decrypted Kubernetes Secrets and service-account tokens as plaintext files, so granting the root would make every mounted secret on the node readable through an MCP tool call. That contradicts this repo's own `AGENTS.md` safety rule. The `pod-ephemeral` profile answers what the incident actually needed - byte attribution under that tree - without any file-read grant. If path-level `stat` there is still wanted, the clean shape is a separate stat-only allowlist rather than widening this one, and that is worth its own issue. ## Verification `ward exec lint`, `ward exec test` (46 passed, 2 new), and `ward exec precommit` are all clean. Opened as a PR rather than pushed to `main`: the `merge-remote-main` workflow applies, but the direct push to `main` was refused by the harness, so this branch is the durable checkpoint. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-17 18:35:27 +00:00

fix(storage): stop the wide usage profiles from OOMKilling the container 8e4abfdb61

The root, var, and var-lib profiles never returned on kai-server. The container
died at four minutes every time with exit 137 against its 512Mi limit, and the
restart timestamps matched the scan starts exactly.

_scan_tree kept every scanned file inode in a dedup set, so the set grew with
the tree rather than with hard links. Measured at 162 MB per million (dev, ino)
tuples, the configured 5,000,000-entry budget alone needs about 810 MB before
any traversal state. The config promised a scan the container could not survive.

Only a multiply-linked file can be reached twice, and same-filesystem bind and
subtree mounts are already excluded upstream, so track inodes only when
st_nlink > 1. That is the line du draws, and it leaves the documented
hard-link contract unchanged. Pack the key into one int (114 MB per million
rather than 162 MB) and carry the traversal stack as str rather than Path
(153 vs 362 bytes per entry). Report hardlinked_inodes_tracked so the remaining
cost is observable rather than inferred.

Add a pod-ephemeral profile at /var/lib/kubelet/pods. The 2026-08-17 incident
lived in 62 GiB of emptyDir there and no configured domain could see it. Mount
exclusion already drops the local-path PVCs kubelet bind-mounts into that tree,
so the profile does not re-count k3s-storage bytes.

Refs coilyco-flight-deck/node-stats-mcp#26

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-17 18:35:27 +00:00

coilyco-ops referenced this pull request

2026-08-17 18:36:02 +00:00

Disk-pressure investigation could not run unattended: every filesystem read fell back to manual operator commands #26