Alert on leading-edge memory exhaustion (swap drain + PSI) before livelock blackout #184
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
kai-server livelocked from memory exhaustion two nights running (2026-05-29 ~06:22 and 2026-05-30 ~06:51). Root cause: a 3am
brew upgrade(viacoily-update.service) spawned a swarm of ~16cc1pluscompilers (~1GB RSS each) building a formula from source, which drove anon memory to ~30.7GB on a 31GB-usable box with only 2GB swap. Swap hit 0, the kernel entered unbounded__alloc_pages_slowpathreclaim, and the box thrashed for ~3 hours (one OOM stack trace took 03:14 to 06:51 to finish printing) before a power-cycle recovered it ~07:00.Key constraint: in-host telemetry goes dark during the livelock
node_exporteron kai-server has a hard data gap from ~03:15 to ~07:10 PDT on 2026-05-30 - the exact livelock window. SwapFree was observably draining (0.72 -> 0.23 GiB) right up to 03:15, then total blackout until reboot. You can only ever see up to the cliff, never through it. So post-hoc logging cannot diagnose the terminal phase. The only useful signal is an alert that fires on the leading edge (03:00-03:13 ramp) while the box can still emit a notification.Ask
Add a vmalert rule (or equivalent) on data that already exists in vmsingle from node_exporter:
node_memory_SwapFree_bytes{instance="kai-server"}low for N minutes (e.g. < 15% of SwapTotal).rate(node_pressure_memory_waiting_seconds_total{instance="kai-server"}[1m])rising (PSI memory stall = reclaim thrash starting).node_memory_MemAvailable_bytescollapsing.Route to a push notification (Discord / the existing alert path) so the spiral is caught before the blackout.
Notes
node_memory_Swap{Total,Free,Cached}_bytesandnode_pressure_memory_{waiting,stalled}_seconds_totalfor kai-server. No new exporter needed.coily-updatecap (separate issue) and the heartbeat-ingestion fix (separate issue).Found during the 2026-05-30 crash investigation.