feat(o11y): add kai disk pressure watcher

coilyco-ops commented

2026-07-23 06:50:20 +00:00

Member

Adds a ser8-owned, read-only kai-server root filesystem watcher at the 85% critical threshold and repairs the runner recycle job so its disposable DinD scratch can actually be reset. Cleanup remains attended and owner-scoped: nothing here prunes images, volumes, logs, databases, or game data.

Adds rootfs alerting, bounded ownership classification, and a ranked recovery drill
Aligns rootfs telemetry with the 85% threshold
Replaces the recycle job's invalid shell invocation with bounded native kubectl calls for the two runner StatefulSets

Tests: ward exec disk-pressure-watch-test; ward exec forgejo-runner-recycle-test; ward exec pre-commit. Live convergence and node-stats verification require an attended operator.

closes #596

Adds a ser8-owned, read-only kai-server root filesystem watcher at the 85% critical threshold and repairs the runner recycle job so its disposable DinD scratch can actually be reset. Cleanup remains attended and owner-scoped: nothing here prunes images, volumes, logs, databases, or game data. - Adds rootfs alerting, bounded ownership classification, and a ranked recovery drill - Aligns rootfs telemetry with the 85% threshold - Replaces the recycle job's invalid shell invocation with bounded native kubectl calls for the two runner StatefulSets Tests: `ward exec disk-pressure-watch-test`; `ward exec forgejo-runner-recycle-test`; `ward exec pre-commit`. Live convergence and node-stats verification require an attended operator. closes #596

coilyco-ops added 1 commit

2026-07-23 06:50:20 +00:00

feat(o11y): add kai disk pressure watcher

CI / lint (pull_request) Successful in 45s

Details

TruffleHog / Scan for secrets (pull_request) Successful in 14s

Details

6d80e8254c