coilyco-flight-deck/node-stats-mcp
Watch
2
0
Code Issues 3 Pull requests Projects Releases Packages 1 Wiki Activity Actions

Disk-pressure investigation could not run unattended: every filesystem read fell back to manual operator commands #26

New issue
Closed
opened 2026-08-17 06:30:23 +00:00 by coilyco-ops · 4 comments
coilyco-ops commented 2026-08-17 06:30:23 +00:00
Owner
Copy link

Context

During a disk-pressure investigation on kai-server (2026-08-17, root hit 90.4%), the agent had to ask Kai to hand-run eight read-only filesystem commands in her terminal. Every one of them was an ordinary read that the tooling is supposed to cover. This cost real operator time in the middle of a live incident.

Relief itself succeeded (90.4% to 83.6%, 35.3 GB reclaimed by restarting two runner pods to release emptyDir scratch). This issue is only about why the diagnosis needed a human at the keyboard.

Every manual command that was requested

  1. bash infrastructure/scripts/forgejo-storage-measure.sh - guarded surface exposes no kubectl exec, and the skill explicitly routes this to an attended operator.
  2. sudo du -sh /var/lib/rancher/k3s/storage/pvc-e0167dde-..._forgejo_forgejo-data/data/attachments - no attachments domain in the bounded report, and stat_path refused the path.
  3. sudo du -x --max-depth=1 / - node-stats root profile never completes.
  4. sudo du -xh --max-depth=2 /var --exclude='/var/lib/kubelet/pods' - same.
  5. sudo du -xh --max-depth=1 /var/lib/kubelet - same, plus path outside the readable-root allowlist.
  6. sudo du -xh --max-depth=1 /var/lib/kubelet/pods --exclude='*kubernetes.io~local-volume*' - same.
  7. kubectl -n forgejo rollout restart statefulset/forgejo-runner-deploy-sirens-discord-ops - blocked by the Claude Code classifier despite aosguard granting the rollout verb.
  8. kubectl -n forgejo rollout restart statefulset/forgejo-runner-deploy-ward-mcp - same.

Also offered but not run, for the same class of reason: kubectl delete pvc docker-lib-forgejo-runner-build-0 (aosguard exposes no delete verb) and two read-only psql queries against the attachment table (no exec).

Root causes

A. node-stats-mcp is OOMKilled on the wide profiles

The root, var, and var-lib profiles never return. The container dies partway through, every time.

Last State:   Terminated
Reason:       OOMKilled
Exit Code:    137
Limits:       memory: 512Mi
Requests:     memory: 128Mi

Restart timestamps matched the scan restarts exactly (05:00:54, 05:07:04, 05:10:51), each surviving about four minutes. The root profile is configured for max_entries: 5000000; the tree holds millions of inodes and the scan accumulates them in memory. 512Mi does not fit that budget.

Bounded profiles are fine - k3s-storage completed 908,948 entries in 139s. The failure is specific to the wide profiles and it is deterministic, not flaky.

Fix is either raising the limit to fit the configured budget or lowering the wide profiles' entry cap so the two agree. Right now the config promises a scan the container cannot survive.

B. readable-root allowlist is too narrow

stat_path rejected /var/lib/kubelet and /usr/share/ollama/.ollama/models as outside the allowlist. /var/lib/kubelet turned out to hold the actual cause of the incident.

C. No pod-ephemeral domain in the bounded report - this is the big one

kai-disk-pressure-report covers containerd, local-path PVCs, logs, journald, game saves, and kai home. Those summed to 287 GB against 456 GB used.

The gap was pod ephemeral storage under /var/lib/kubelet/pods: 62 GiB of emptyDir and writable container layers, invisible to every domain. That is where the incident actually lived, and releasing it is what resolved it.

The tool built to find disk pressure could not see the thing causing the disk pressure. A pod_ephemeral_bytes domain would have made this a two-minute diagnosis.

D. No attachments split in forgejo-storage-measure.sh

The script measures packages exhaustively (blob references, ownership, age buckets, cleanup rules) but reports data/attachments as a bare du line. Attachments turned out to be 38 GB with no ownership attribution available. It needs the same treatment: a query against attachment joined to repository, split by release vs issue, with age buckets.

E. Bind-mount double counting has no guardrail

du -x does not exclude kubelet's bind mounts of local-path PVCs, because they are the same filesystem. A plain du -x / reported 567 GiB on a filesystem holding 401 GiB. The MCP dedupes and reports deduplicated_entries; raw du does not, and the runbook does not warn about it. Anyone reading that output at face value goes hunting for 166 GiB that does not exist. Nearly happened here.

F. Harness and policy disagree on rollout restart

aosguard grants rollout, and the Claude Code classifier blocked it anyway. Either the action is permitted or it is not, and right now the agent gets to the point of executing before finding out.

G. aosguard exposes no delete verb

May well be intentional. Worth stating explicitly in the runbook so an agent proposes the operator path from the start instead of discovering it mid-incident.

Proposed work

  • Raise node-stats-mcp memory limit to fit the configured scan budget, or lower max_entries on the wide profiles so config and limit agree (deploy)
  • Add /var/lib/kubelet to the readable-root allowlist (node-stats-mcp)
  • Add a pod_ephemeral_bytes domain to kai-disk-pressure-report, excluding kubernetes.io~local-volume bind mounts (infrastructure)
  • Add an attachments section to forgejo-storage-measure.sh with DB-backed ownership (infrastructure)
  • Document bind-mount double counting in the disk-pressure skill, with the correct --exclude incantation (agentic-os-kai)
  • Reconcile the classifier and aosguard on rollout restart
  • State the no-delete boundary in the skill so the operator handoff is proposed upfront

Separate follow-ups from the same incident

Filed here for traceability, not part of this issue's scope:

  • Missing coilyco.io/runner-scratch: recycle label on forgejo-runner-deploy-sirens-discord-ops and forgejo-runner-tap-writer-scoped. The unlabelled runner reached 24 GiB of scratch over 3d8h while its recycled peers sat at 4.6 to 9 GiB after 13h. The reclaimed space will rebuild within days without this. Two-line change in deploy.
  • Zero Forgejo package cleanup rules. The package store is 74 GB, grew entirely within 24 days, and holds 578 retained container versions of agentic-os. Roughly 3 GB/day with nothing retiring anything.
## Context During a disk-pressure investigation on kai-server (2026-08-17, root hit 90.4%), the agent had to ask Kai to hand-run eight read-only filesystem commands in her terminal. Every one of them was an ordinary read that the tooling is supposed to cover. This cost real operator time in the middle of a live incident. Relief itself succeeded (90.4% to 83.6%, 35.3 GB reclaimed by restarting two runner pods to release `emptyDir` scratch). This issue is only about why the diagnosis needed a human at the keyboard. ## Every manual command that was requested 1. `bash infrastructure/scripts/forgejo-storage-measure.sh` - guarded surface exposes no `kubectl exec`, and the skill explicitly routes this to an attended operator. 2. `sudo du -sh /var/lib/rancher/k3s/storage/pvc-e0167dde-..._forgejo_forgejo-data/data/attachments` - no attachments domain in the bounded report, and `stat_path` refused the path. 3. `sudo du -x --max-depth=1 /` - node-stats `root` profile never completes. 4. `sudo du -xh --max-depth=2 /var --exclude='/var/lib/kubelet/pods'` - same. 5. `sudo du -xh --max-depth=1 /var/lib/kubelet` - same, plus path outside the readable-root allowlist. 6. `sudo du -xh --max-depth=1 /var/lib/kubelet/pods --exclude='*kubernetes.io~local-volume*'` - same. 7. `kubectl -n forgejo rollout restart statefulset/forgejo-runner-deploy-sirens-discord-ops` - blocked by the Claude Code classifier despite aosguard granting the `rollout` verb. 8. `kubectl -n forgejo rollout restart statefulset/forgejo-runner-deploy-ward-mcp` - same. Also offered but not run, for the same class of reason: `kubectl delete pvc docker-lib-forgejo-runner-build-0` (aosguard exposes no `delete` verb) and two read-only `psql` queries against the `attachment` table (no `exec`). ## Root causes ### A. node-stats-mcp is OOMKilled on the wide profiles The `root`, `var`, and `var-lib` profiles never return. The container dies partway through, every time. ``` Last State: Terminated Reason: OOMKilled Exit Code: 137 Limits: memory: 512Mi Requests: memory: 128Mi ``` Restart timestamps matched the scan restarts exactly (05:00:54, 05:07:04, 05:10:51), each surviving about four minutes. The `root` profile is configured for `max_entries: 5000000`; the tree holds millions of inodes and the scan accumulates them in memory. 512Mi does not fit that budget. Bounded profiles are fine - `k3s-storage` completed 908,948 entries in 139s. The failure is specific to the wide profiles and it is deterministic, not flaky. Fix is either raising the limit to fit the configured budget or lowering the wide profiles' entry cap so the two agree. Right now the config promises a scan the container cannot survive. ### B. readable-root allowlist is too narrow `stat_path` rejected `/var/lib/kubelet` and `/usr/share/ollama/.ollama/models` as outside the allowlist. `/var/lib/kubelet` turned out to hold the actual cause of the incident. ### C. No pod-ephemeral domain in the bounded report - this is the big one `kai-disk-pressure-report` covers containerd, local-path PVCs, logs, journald, game saves, and kai home. Those summed to 287 GB against 456 GB used. The gap was pod ephemeral storage under `/var/lib/kubelet/pods`: **62 GiB of `emptyDir` and writable container layers, invisible to every domain**. That is where the incident actually lived, and releasing it is what resolved it. The tool built to find disk pressure could not see the thing causing the disk pressure. A `pod_ephemeral_bytes` domain would have made this a two-minute diagnosis. ### D. No attachments split in forgejo-storage-measure.sh The script measures packages exhaustively (blob references, ownership, age buckets, cleanup rules) but reports `data/attachments` as a bare `du` line. Attachments turned out to be 38 GB with no ownership attribution available. It needs the same treatment: a query against `attachment` joined to `repository`, split by release vs issue, with age buckets. ### E. Bind-mount double counting has no guardrail `du -x` does not exclude kubelet's bind mounts of local-path PVCs, because they are the same filesystem. A plain `du -x /` reported 567 GiB on a filesystem holding 401 GiB. The MCP dedupes and reports `deduplicated_entries`; raw `du` does not, and the runbook does not warn about it. Anyone reading that output at face value goes hunting for 166 GiB that does not exist. Nearly happened here. ### F. Harness and policy disagree on rollout restart aosguard grants `rollout`, and the Claude Code classifier blocked it anyway. Either the action is permitted or it is not, and right now the agent gets to the point of executing before finding out. ### G. aosguard exposes no delete verb May well be intentional. Worth stating explicitly in the runbook so an agent proposes the operator path from the start instead of discovering it mid-incident. ## Proposed work - [ ] Raise `node-stats-mcp` memory limit to fit the configured scan budget, or lower `max_entries` on the wide profiles so config and limit agree (`deploy`) - [ ] Add `/var/lib/kubelet` to the readable-root allowlist (`node-stats-mcp`) - [ ] Add a `pod_ephemeral_bytes` domain to `kai-disk-pressure-report`, excluding `kubernetes.io~local-volume` bind mounts (`infrastructure`) - [ ] Add an attachments section to `forgejo-storage-measure.sh` with DB-backed ownership (`infrastructure`) - [ ] Document bind-mount double counting in the disk-pressure skill, with the correct `--exclude` incantation (`agentic-os-kai`) - [ ] Reconcile the classifier and aosguard on `rollout restart` - [ ] State the no-`delete` boundary in the skill so the operator handoff is proposed upfront ## Separate follow-ups from the same incident Filed here for traceability, not part of this issue's scope: - Missing `coilyco.io/runner-scratch: recycle` label on `forgejo-runner-deploy-sirens-discord-ops` and `forgejo-runner-tap-writer-scoped`. The unlabelled runner reached 24 GiB of scratch over 3d8h while its recycled peers sat at 4.6 to 9 GiB after 13h. The reclaimed space will rebuild within days without this. Two-line change in `deploy`. - Zero Forgejo package cleanup rules. The package store is 74 GB, grew entirely within 24 days, and holds 578 retained container versions of `agentic-os`. Roughly 3 GB/day with nothing retiring anything.
coilyco-ops commented 2026-08-17 18:36:02 +00:00
Author
Owner
Copy link

Two PRs cover the checkbox list. Both are open for review, neither is merged.

  • node-stats-mcp PR #27 - root causes A, B, and C.
  • infrastructure PR #859 - root causes C, D, E, F, and G.

A - the OOM is structural, not a budget mismatch

_scan_tree kept every scanned file inode in its dedup set, so the set grew with the tree rather than with hard links. Measured: 162 MB per million (dev, ino) tuples, so the 5,000,000-entry budget needs about 810 MB for that set alone, before any traversal state, against a 512Mi limit.

The fix removes the term instead of re-tuning around it. Only a multiply-linked file can be reached twice, and same-filesystem bind and subtree mounts are already excluded upstream, so inodes are tracked only when st_nlink > 1. That is the line du draws and it leaves the documented hard-link contract unchanged. Live evidence for how little that costs: the current k3s-storage snapshot reports deduplicated_entries: 202 across 344,464 entries, all of it in the Docker layer store. The packed inode key and a str traversal stack take the rest.

So neither proposed lever is needed: the limit stays 512Mi and max_entries stays 5,000,000, because the scan now fits. If a live root scan still dies, that is new evidence and a different fix.

B - I did not add /var/lib/kubelet to the readable roots, and I recommend against it

NODE_STATS_READABLE_ROOTS gates read_text_head as well as stat_path, through the shared _resolve_readable. /var/lib/kubelet/pods/*/volumes/kubernetes.io~secret/ and ~projected/ hold decrypted Kubernetes Secrets and service-account tokens as plaintext files, so granting that root would make every mounted secret on the node readable through an MCP tool call. That contradicts this repo's own AGENTS.md safety rule.

The pod-ephemeral profile answers what the incident actually needed - byte attribution under that tree - with no file-read grant. If path-level stat there is still wanted, the clean shape is a separate stat-only allowlist rather than widening this one. Worth its own issue if you want it.

C - covered on both sides

node-stats-mcp gains a pod-ephemeral profile at /var/lib/kubelet/pods; mount exclusion already drops the bind-mounted local-path PVCs, pinned by a new test. No deploy change: it sets no NODE_STATS_HOST_USAGE_PROFILES override, so the new default ships with the image. kai-disk-pressure-report gains pod_ephemeral_bytes with the *kubernetes.io~local-volume* and *volume-subpaths* excludes, and --detailed resolves each pod UID to its namespace and pod name.

D, E, G - as specified

Attachment reports join attachment to repository, split by release, issue, and unattached, with 7, 30, and 90-day age buckets, plus orphaned-row and on-disk file counts to reconcile against each other. Column names were checked against Forgejo's models/repo/attachment.go rather than recalled. A new raw-du-guardrails.md reference carries the 567-GiB-on-a-401-GiB-filesystem story, the correct exclude incantation, and the no-delete / no-exec boundaries.

F - the premise is wrong, and the correction is the useful part

There is no classifier disagreeing with aosguard. Bash(kubectl *) is a deliberate fleet-wide deny, installed by agentic-os/scripts/apply-base-claude-settings.py on the stated ground that live-infrastructure CLIs belong to a guarded verb. aosguard ops kubectl rollout restart does not match that rule and was available the whole time. The incident agent reached for bare kubectl and read a designed deny as a surprise. That is documented now, so the next agent picks the guarded verb first.

Actually changing the deny would be a live-system permission decision, so it stays yours rather than mine.

Not touched

The two follow-ups in the issue body are still open: the missing coilyco.io/runner-scratch: recycle label on two runners, and the zero Forgejo package cleanup rules. Both are deploy and Forgejo-admin work, out of this issue's stated scope.

Two PRs cover the checkbox list. Both are open for review, neither is merged. * `node-stats-mcp` PR [#27](https://forgejo.coilysiren.me/coilyco-flight-deck/node-stats-mcp/pulls/27) - root causes A, B, and C. * `infrastructure` PR [#859](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/859) - root causes C, D, E, F, and G. ## A - the OOM is structural, not a budget mismatch `_scan_tree` kept **every** scanned file inode in its dedup set, so the set grew with the tree rather than with hard links. Measured: **162 MB per million `(dev, ino)` tuples**, so the 5,000,000-entry budget needs about **810 MB** for that set alone, before any traversal state, against a 512Mi limit. The fix removes the term instead of re-tuning around it. Only a multiply-linked file can be reached twice, and same-filesystem bind and subtree mounts are already excluded upstream, so inodes are tracked only when `st_nlink > 1`. That is the line `du` draws and it leaves the documented hard-link contract unchanged. Live evidence for how little that costs: the current `k3s-storage` snapshot reports `deduplicated_entries: 202` across 344,464 entries, all of it in the Docker layer store. The packed inode key and a `str` traversal stack take the rest. So neither proposed lever is needed: the limit stays 512Mi and `max_entries` stays 5,000,000, because the scan now fits. If a live `root` scan still dies, that is new evidence and a different fix. ## B - I did not add `/var/lib/kubelet` to the readable roots, and I recommend against it `NODE_STATS_READABLE_ROOTS` gates `read_text_head` as well as `stat_path`, through the shared `_resolve_readable`. `/var/lib/kubelet/pods/*/volumes/kubernetes.io~secret/` and `~projected/` hold decrypted Kubernetes Secrets and service-account tokens as plaintext files, so granting that root would make every mounted secret on the node readable through an MCP tool call. That contradicts this repo's own `AGENTS.md` safety rule. The `pod-ephemeral` profile answers what the incident actually needed - byte attribution under that tree - with no file-read grant. If path-level `stat` there is still wanted, the clean shape is a separate stat-only allowlist rather than widening this one. Worth its own issue if you want it. ## C - covered on both sides `node-stats-mcp` gains a `pod-ephemeral` profile at `/var/lib/kubelet/pods`; mount exclusion already drops the bind-mounted local-path PVCs, pinned by a new test. No `deploy` change: it sets no `NODE_STATS_HOST_USAGE_PROFILES` override, so the new default ships with the image. `kai-disk-pressure-report` gains `pod_ephemeral_bytes` with the `*kubernetes.io~local-volume*` and `*volume-subpaths*` excludes, and `--detailed` resolves each pod UID to its namespace and pod name. ## D, E, G - as specified Attachment reports join `attachment` to `repository`, split by release, issue, and unattached, with 7, 30, and 90-day age buckets, plus orphaned-row and on-disk file counts to reconcile against each other. Column names were checked against Forgejo's `models/repo/attachment.go` rather than recalled. A new `raw-du-guardrails.md` reference carries the 567-GiB-on-a-401-GiB-filesystem story, the correct exclude incantation, and the no-`delete` / no-`exec` boundaries. ## F - the premise is wrong, and the correction is the useful part There is no classifier disagreeing with aosguard. `Bash(kubectl *)` is a deliberate fleet-wide **deny**, installed by `agentic-os/scripts/apply-base-claude-settings.py` on the stated ground that live-infrastructure CLIs belong to a guarded verb. `aosguard ops kubectl rollout restart` does not match that rule and was available the whole time. The incident agent reached for bare `kubectl` and read a designed deny as a surprise. That is documented now, so the next agent picks the guarded verb first. Actually changing the deny would be a live-system permission decision, so it stays yours rather than mine. ## Not touched The two follow-ups in the issue body are still open: the missing `coilyco.io/runner-scratch: recycle` label on two runners, and the zero Forgejo package cleanup rules. Both are `deploy` and Forgejo-admin work, out of this issue's stated scope.
coilyco-ops commented 2026-08-17 19:45:33 +00:00
Author
Owner
Copy link

Correcting my own work on item D. Kai's read is right: adding attachment queries to forgejo-storage-measure.sh improved what a human sees when they run it and did nothing about the fact that a human has to run it. That script needs kubectl exec, and no amount of extra SQL in it changes that. Measured against this issue's title, that part was worth close to nothing.

The reason it stayed manual turned out to be smaller than "no exec". Forgejo's data sits on a local-path PVC on the host, in the same tree the MCP already walks and already totals at 123 GB. The only thing standing between an agent and the data / git / packages split was reporting granularity: scan_usage_profile reported immediate children with recursive totals and nothing below that, so a per-level du was the only way down. _scan_pressure_children has the same one-level shape.

So max_depth is now on UsageProfile, and k3s-storage and pod-ephemeral ship at depth 3. That reaches claim -> data -> attachments and pod -> volumes -> volume type, with no pod access.

Depth changes reporting, never the walk. _scan_node reuses the same flat _scan_tree for everything below the reported levels and absorbs its totals, so the numbers cannot drift between views. Verified on a fixture:

depth=1 total=757760 entries=9    depth=3 total=757760 entries=9
  pvc-forgejo = 757760              pvc-forgejo = 757760
                                      pvc-forgejo/data = 503808
                                        pvc-forgejo/data/attachments = 401408
                                        pvc-forgejo/data/lfs = 102400
                                      pvc-forgejo/packages = 200704

A regression test scans one tree at both depths and asserts identical bytes and entry counts, because a depth that moved a byte would make both views untrustworthy.

The eight commands, honestly

  • 1 forgejo-storage-measure.sh - byte half unattended, ownership half still attended. Which repository, which release or issue, and which age bucket own the bytes lives only in PostgreSQL. The attachment SQL is now correctly framed as the attended half rather than as a fix.
  • 2 du -sh .../data/attachments - unattended, k3s-storage depth 3.
  • 3 du -x --max-depth=1 / - unattended, the root profile completes now.
  • 4 du -xh --max-depth=2 /var - unattended by chaining var into var-lib.
  • 5 du -xh --max-depth=1 /var/lib/kubelet - unattended, var-lib plus pod-ephemeral.
  • 6 du -xh --max-depth=1 /var/lib/kubelet/pods --exclude=... - unattended, pod-ephemeral with mount exclusion doing the dedup.
  • 7, 8 kubectl rollout restart - operator, by design. A deliberate fleet-wide deny, not a gap.

Five of eight fully unattended, one halved, two operator-only on purpose.

The remaining honest gap is attachment and package ownership. Forgejo's API exposes per-release and per-issue assets, so an aosguard verb could reconstruct it without exec, at the cost of walking repositories. That is agentic-os work and a different issue if you want it.

Both PRs updated in place: node-stats-mcp#27, infrastructure#859. 47 tests pass, lint and precommit clean on both.

Correcting my own work on item D. Kai's read is right: adding attachment queries to `forgejo-storage-measure.sh` improved what a human sees when they run it and did nothing about the fact that a human has to run it. That script needs `kubectl exec`, and no amount of extra SQL in it changes that. Measured against this issue's title, that part was worth close to nothing. The reason it stayed manual turned out to be smaller than "no `exec`". Forgejo's data sits on a **local-path PVC on the host**, in the same tree the MCP already walks and already totals at 123 GB. The only thing standing between an agent and the `data` / `git` / `packages` split was **reporting granularity**: `scan_usage_profile` reported immediate children with recursive totals and nothing below that, so a per-level `du` was the only way down. `_scan_pressure_children` has the same one-level shape. So `max_depth` is now on `UsageProfile`, and `k3s-storage` and `pod-ephemeral` ship at depth 3. That reaches claim -> `data` -> `attachments` and pod -> `volumes` -> volume type, with no pod access. Depth changes reporting, never the walk. `_scan_node` reuses the same flat `_scan_tree` for everything below the reported levels and absorbs its totals, so the numbers cannot drift between views. Verified on a fixture: ``` depth=1 total=757760 entries=9 depth=3 total=757760 entries=9 pvc-forgejo = 757760 pvc-forgejo = 757760 pvc-forgejo/data = 503808 pvc-forgejo/data/attachments = 401408 pvc-forgejo/data/lfs = 102400 pvc-forgejo/packages = 200704 ``` A regression test scans one tree at both depths and asserts identical bytes and entry counts, because a depth that moved a byte would make both views untrustworthy. ## The eight commands, honestly * 1 `forgejo-storage-measure.sh` - **byte half unattended**, ownership half still attended. Which repository, which release or issue, and which age bucket own the bytes lives only in PostgreSQL. The attachment SQL is now correctly framed as the attended half rather than as a fix. * 2 `du -sh .../data/attachments` - **unattended**, `k3s-storage` depth 3. * 3 `du -x --max-depth=1 /` - **unattended**, the `root` profile completes now. * 4 `du -xh --max-depth=2 /var` - **unattended** by chaining `var` into `var-lib`. * 5 `du -xh --max-depth=1 /var/lib/kubelet` - **unattended**, `var-lib` plus `pod-ephemeral`. * 6 `du -xh --max-depth=1 /var/lib/kubelet/pods --exclude=...` - **unattended**, `pod-ephemeral` with mount exclusion doing the dedup. * 7, 8 `kubectl rollout restart` - **operator, by design**. A deliberate fleet-wide deny, not a gap. Five of eight fully unattended, one halved, two operator-only on purpose. The remaining honest gap is attachment and package **ownership**. Forgejo's API exposes per-release and per-issue assets, so an aosguard verb could reconstruct it without `exec`, at the cost of walking repositories. That is agentic-os work and a different issue if you want it. Both PRs updated in place: [node-stats-mcp#27](https://forgejo.coilysiren.me/coilyco-flight-deck/node-stats-mcp/pulls/27), [infrastructure#859](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/859). 47 tests pass, lint and precommit clean on both.
coilyco-ops commented 2026-08-17 21:42:46 +00:00
Author
Owner
Copy link

Both merges rolled to kai-server, so the claims are now measured rather than argued.

A - the root profile completes

status=complete  complete=true  totals_are_lower_bounds=false
truncated=false  timed_out=false  permission_errors=0  scan_errors=0
duration=102s    entries_scanned=2,683,740
hardlinked_inodes_tracked=109,290   deduplicated_entries=280,503

It used to die at roughly four minutes with exit 137. It now finishes in 102 seconds and reports a complete total rather than a lower bound.

The dedup set holds 109,290 inodes against 2,683,740 entries scanned - four percent of what it used to carry. At the old 162 MB per million that set alone would have been about 435 MB on this tree, against a 512Mi limit, which is the whole story of the OOM. Container RSS mid-scan measured 292 MB, 57 percent of the limit.

deduplicated_entries=280,503 against 109,290 tracked inodes means each hard-linked inode was reached about 3.5 times and counted once. The dedup is doing more work than before, not less, so the st_nlink > 1 narrowing cost no accuracy.

C and the depth work - the Forgejo split, unattended

One tool call, 27 seconds, no kubectl exec:

forgejo-data                    124.3 GB
  packages                       80.6 GB   (257 shards)
  data                           40.0 GB
    attachments                  40.0 GB   <- was `sudo du -sh` by hand
    avatars                        2.0 MB
    repo-avatars                  65.5 KB
  repo-archive                     1.9 GB

Attachments are 40.0 GB, up from the 38 GB in this issue's body, and data/ is essentially nothing but attachments. Packages are 80.6 GB, up from 74 GB, which tracks the "roughly 3 GB/day with nothing retiring anything" note.

A gap this verification exposed

The root breakdown's top children are:

224.35 GB  /var
 84.94 GB  /home
 32.00 GB  /swapfile
 10.48 GB  /usr
  6.62 GB  /srv

/swapfile is 32 GB and no domain in kai-disk-pressure-report names it. It is the third-largest consumer of the root filesystem, it is permanently allocated, and it is exactly the same class of blind spot as the pod_ephemeral_bytes gap this issue was filed about: a large, real, unattributed consumer that the bounded report silently omits from its sum. Anyone reconciling domains against df is off by 32 GB before they start.

Worth its own issue in infrastructure rather than reopening scope here.

Both merges rolled to kai-server, so the claims are now measured rather than argued. ## A - the `root` profile completes ``` status=complete complete=true totals_are_lower_bounds=false truncated=false timed_out=false permission_errors=0 scan_errors=0 duration=102s entries_scanned=2,683,740 hardlinked_inodes_tracked=109,290 deduplicated_entries=280,503 ``` It used to die at roughly four minutes with exit 137. It now finishes in **102 seconds** and reports a complete total rather than a lower bound. The dedup set holds **109,290** inodes against 2,683,740 entries scanned - four percent of what it used to carry. At the old 162 MB per million that set alone would have been about **435 MB** on this tree, against a 512Mi limit, which is the whole story of the OOM. Container RSS mid-scan measured **292 MB**, 57 percent of the limit. `deduplicated_entries=280,503` against 109,290 tracked inodes means each hard-linked inode was reached about 3.5 times and counted once. The dedup is doing more work than before, not less, so the `st_nlink > 1` narrowing cost no accuracy. ## C and the depth work - the Forgejo split, unattended One tool call, 27 seconds, no `kubectl exec`: ``` forgejo-data 124.3 GB packages 80.6 GB (257 shards) data 40.0 GB attachments 40.0 GB <- was `sudo du -sh` by hand avatars 2.0 MB repo-avatars 65.5 KB repo-archive 1.9 GB ``` Attachments are **40.0 GB**, up from the 38 GB in this issue's body, and `data/` is essentially nothing but attachments. Packages are **80.6 GB**, up from 74 GB, which tracks the "roughly 3 GB/day with nothing retiring anything" note. ## A gap this verification exposed The `root` breakdown's top children are: ``` 224.35 GB /var 84.94 GB /home 32.00 GB /swapfile 10.48 GB /usr 6.62 GB /srv ``` **`/swapfile` is 32 GB and no domain in `kai-disk-pressure-report` names it.** It is the third-largest consumer of the root filesystem, it is permanently allocated, and it is exactly the same class of blind spot as the `pod_ephemeral_bytes` gap this issue was filed about: a large, real, unattributed consumer that the bounded report silently omits from its sum. Anyone reconciling domains against `df` is off by 32 GB before they start. Worth its own issue in `infrastructure` rather than reopening scope here.
coilyco-ops commented 2026-08-27 03:18:55 +00:00
Author
Owner
Copy link

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901).

Closing as complete. Root causes A through G landed across node-stats-mcp#27 and infrastructure#859, and the verification comment measures the results. The unattributed /swapfile gap was explicitly deferred to its own infrastructure issue.

This was priority/P3, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless.

If this is live work, reopen it. The whole set is recoverable with state:closed label:burndown-2026-08.

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901). Closing as complete. Root causes A through G landed across node-stats-mcp#27 and infrastructure#859, and the verification comment measures the results. The unattributed /swapfile gap was explicitly deferred to its own infrastructure issue. This was `priority/P3`, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless. If this is live work, reopen it. The whole set is recoverable with `state:closed label:burndown-2026-08`.
coilyco-ops 2026-08-27 03:18:56 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
aos/claude/uq98-usage-depth
aos/claude/uq98-scan-memory
aos/claude/mg96-fm
aos/claude/mg96
No results found.
Labels
Clear labels   
burndown-2026-08
Closed in the 2026-08-26 backlog burn-down. Reopen freely: state:closed label:burndown-2026-08 recovers the whole set.

  
autonomy
async-consult
 A human needs to consult on the issue to upgrade it to headless

  
autonomy
epic
 This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  
autonomy
headless
 The agent can perform the work on its own

  
autonomy
live-collab
 The agent and the human need to work together in realtime

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
priority
P0
 priority tier

  
priority
P1
 priority tier

  
priority
P2
 priority tier

  
priority
P3
 priority tier

  
priority
P4
 priority tier

  
qa-fixture
Disposable issue admitted to the bounded Ward QA verification lane.

  
role/advocate
requires work from the Developer Advocate seat

  
role/director
requires work from the Portfolio Director seat

  
role/exec
requires work from the exec role

  
role/frontend
requires work from the Frontend Engineer seat

  
role/gamedev
requires work from the Game Developer seat

  
role/human
requires a person, and specifically not an agent seat

  
role/platform
requires work from the Platform Engineer seat

  
role/qa
requires work from the QA role

  
role/science
requires work from the Applied Scientist seat

  
role/sysadmin
requires work from the Systems Administrator seat

  
state
ambient
 ambient and ephemeral work, held as a maintained document rather than a queue

No labels
burndown-2026-08
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/advocate
role/director
role/exec
role/frontend
role/gamedev
role/human
role/platform
role/qa
role/science
role/sysadmin
state
ambient
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-flight-deck/node-stats-mcp#26
No description provided.