feat(disk-pressure): measure pod ephemeral bytes and attachment ownership #859

Merged

coilysiren merged 2 commits from aos/claude/uq98-disk-domains into main

2026-08-17 19:25:41 +00:00

coilyco-ops commented

2026-08-17 18:34:45 +00:00

Owner

Closes part of coilyco-flight-deck/node-stats-mcp#26.

What the incident showed

The 2026-08-17 kai-server disk-pressure investigation lived in 62 GiB of emptyDir under /var/lib/kubelet/pods, which no ownership domain in the bounded report could see, and in 38 GB of Forgejo attachments the measurement script reported as one bare du line.

Changes

pod_ephemeral_bytes domain in kai-disk-pressure-report. Kubelet bind-mounts local-path PVCs into the pod tree on the same filesystem, so du -x walks them again and the bytes land in two domains. The walk excludes *kubernetes.io~local-volume* and *volume-subpaths* so the domains stay additive. --detailed resolves each pod UID to its namespace and pod name.
Attachment reports in forgejo-storage-measure.sh: bytes split by release, issue, and unattached with 7/30/90-day age buckets; the 50 largest owners by repository and kind; attachments whose repository row is gone; and an on-disk file count to reconcile against those row counts. Column names verified against Forgejo's models/repo/attachment.go.
New raw-du-guardrails.md reference covering why a plain du -x / reported 567 GiB on a 401 GiB filesystem, the exclude incantation that corrects it, and the aosguard no-delete / no-exec boundaries so the operator handoff is proposed upfront rather than discovered mid-incident.
Correction on root cause F. The blocked kubectl rollout restart was not a classifier disagreeing with aosguard. Bash(kubectl *) is a deliberate fleet-wide deny installed by agentic-os/scripts/apply-base-claude-settings.py, on the stated ground that live-infrastructure CLIs belong to a guarded verb. aosguard ops kubectl rollout restart was available the whole time.

Verification

ward exec disk-pressure-watch-test passes with a new assertion pinning the exclude contract, and asserting no other domain carries an exclude. pre-commit run --all-files is clean.

The recycle-report half of that test needs GNU date -d, so it only completes on Linux. That is pre-existing and unrelated; it was verified here against a GNU date shim.

🤖 Generated with Claude Code

Closes part of coilyco-flight-deck/node-stats-mcp#26. ## What the incident showed The 2026-08-17 kai-server disk-pressure investigation lived in 62 GiB of `emptyDir` under `/var/lib/kubelet/pods`, which no ownership domain in the bounded report could see, and in 38 GB of Forgejo attachments the measurement script reported as one bare `du` line. ## Changes * **`pod_ephemeral_bytes` domain** in `kai-disk-pressure-report`. Kubelet bind-mounts local-path PVCs into the pod tree on the same filesystem, so `du -x` walks them again and the bytes land in two domains. The walk excludes `*kubernetes.io~local-volume*` and `*volume-subpaths*` so the domains stay additive. `--detailed` resolves each pod UID to its namespace and pod name. * **Attachment reports** in `forgejo-storage-measure.sh`: bytes split by release, issue, and unattached with 7/30/90-day age buckets; the 50 largest owners by repository and kind; attachments whose repository row is gone; and an on-disk file count to reconcile against those row counts. Column names verified against Forgejo's `models/repo/attachment.go`. * **New `raw-du-guardrails.md` reference** covering why a plain `du -x /` reported 567 GiB on a 401 GiB filesystem, the exclude incantation that corrects it, and the aosguard no-`delete` / no-`exec` boundaries so the operator handoff is proposed upfront rather than discovered mid-incident. * **Correction on root cause F.** The blocked `kubectl rollout restart` was not a classifier disagreeing with aosguard. `Bash(kubectl *)` is a deliberate fleet-wide deny installed by `agentic-os/scripts/apply-base-claude-settings.py`, on the stated ground that live-infrastructure CLIs belong to a guarded verb. `aosguard ops kubectl rollout restart` was available the whole time. ## Verification `ward exec disk-pressure-watch-test` passes with a new assertion pinning the exclude contract, and asserting no other domain carries an exclude. `pre-commit run --all-files` is clean. The recycle-report half of that test needs GNU `date -d`, so it only completes on Linux. That is pre-existing and unrelated; it was verified here against a GNU `date` shim. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 2 commits

2026-08-17 18:34:45 +00:00

feat(disk-pressure): measure pod ephemeral bytes and attachment ownership 23f1c85579

The 2026-08-17 kai-server disk-pressure incident lived in 62 GiB of emptyDir
under /var/lib/kubelet/pods, which no ownership domain in the bounded report
could see, and in 38 GB of Forgejo attachments the measurement script reported
as one bare du line.

Add a pod_ephemeral_bytes domain that excludes the local-path PVCs kubelet
bind-mounts into the same tree on the same filesystem, so the domains stay
additive, and resolve each pod UID to its namespace and name in --detailed.

Add attachment reports to forgejo-storage-measure.sh: bytes split by release,
issue, and unattached with 7/30/90-day age buckets, the 50 largest owners by
repository and kind, attachments whose repository row is gone, and an on-disk
file count to reconcile against those row counts.

Add a raw-du-guardrails reference covering why du -x reported 567 GiB on a
401 GiB filesystem, the exclude incantation that corrects it, and the aosguard
no-delete and no-exec boundaries so the operator handoff is proposed upfront.

Refs coilyco-flight-deck/node-stats-mcp#26

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

docs(disk-pressure): name the real kubectl deny instead of a harness surprise

TruffleHog / Scan for secrets (pull_request) Successful in 5s

Details

CI / lint (pull_request) Successful in 41s

Details

94c2f732a1

The incident report read the blocked `kubectl rollout restart` as a classifier
disagreeing with aosguard. It is not. `Bash(kubectl *)` is a deliberate
fleet-wide deny installed by agentic-os/scripts/apply-base-claude-settings.py,
on the stated ground that live-infrastructure CLIs belong to a guarded verb,
and `aosguard ops kubectl rollout restart` was available the whole time.

Refs coilyco-flight-deck/node-stats-mcp#26

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-17 18:34:45 +00:00

coilyco-ops referenced this pull request from coilyco-flight-deck/node-stats-mcp

2026-08-17 18:36:02 +00:00

Disk-pressure investigation could not run unattended: every filesystem read fell back to manual operator commands #26