coilyco-flight-deck/infrastructure

Watch

Alert on leading-edge memory exhaustion (swap drain + PSI) before livelock blackout #184

New issue

Closed

opened 2026-05-30 17:53:52 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-30 17:53:52 +00:00

Owner

Problem

kai-server livelocked from memory exhaustion two nights running (2026-05-29 ~06:22 and 2026-05-30 ~06:51). Root cause: a 3am brew upgrade (via coily-update.service) spawned a swarm of ~16 cc1plus compilers (~1GB RSS each) building a formula from source, which drove anon memory to ~30.7GB on a 31GB-usable box with only 2GB swap. Swap hit 0, the kernel entered unbounded __alloc_pages_slowpath reclaim, and the box thrashed for ~3 hours (one OOM stack trace took 03:14 to 06:51 to finish printing) before a power-cycle recovered it ~07:00.

Key constraint: in-host telemetry goes dark during the livelock

node_exporter on kai-server has a hard data gap from ~03:15 to ~07:10 PDT on 2026-05-30 - the exact livelock window. SwapFree was observably draining (0.72 -> 0.23 GiB) right up to 03:15, then total blackout until reboot. You can only ever see up to the cliff, never through it. So post-hoc logging cannot diagnose the terminal phase. The only useful signal is an alert that fires on the leading edge (03:00-03:13 ramp) while the box can still emit a notification.

Ask

Add a vmalert rule (or equivalent) on data that already exists in vmsingle from node_exporter:

node_memory_SwapFree_bytes{instance="kai-server"} low for N minutes (e.g. < 15% of SwapTotal).
rate(node_pressure_memory_waiting_seconds_total{instance="kai-server"}[1m]) rising (PSI memory stall = reclaim thrash starting).
Optionally node_memory_MemAvailable_bytes collapsing.

Route to a push notification (Discord / the existing alert path) so the spiral is caught before the blackout.

Notes

node_exporter already exposes node_memory_Swap{Total,Free,Cached}_bytes and node_pressure_memory_{waiting,stalled}_seconds_total for kai-server. No new exporter needed.
Pairs with the coily-update cap (separate issue) and the heartbeat-ingestion fix (separate issue).

Found during the 2026-05-30 crash investigation.

## Problem kai-server livelocked from memory exhaustion two nights running (2026-05-29 ~06:22 and 2026-05-30 ~06:51). Root cause: a 3am `brew upgrade` (via `coily-update.service`) spawned a swarm of ~16 `cc1plus` compilers (~1GB RSS each) building a formula from source, which drove anon memory to ~30.7GB on a 31GB-usable box with only 2GB swap. Swap hit 0, the kernel entered unbounded `__alloc_pages_slowpath` reclaim, and the box thrashed for ~3 hours (one OOM stack trace took 03:14 to 06:51 to finish printing) before a power-cycle recovered it ~07:00. ## Key constraint: in-host telemetry goes dark during the livelock `node_exporter` on kai-server has a hard data gap from ~03:15 to ~07:10 PDT on 2026-05-30 - the exact livelock window. SwapFree was observably draining (0.72 -> 0.23 GiB) right up to 03:15, then total blackout until reboot. You can only ever see *up to* the cliff, never through it. So post-hoc logging cannot diagnose the terminal phase. The only useful signal is an alert that fires on the **leading edge** (03:00-03:13 ramp) while the box can still emit a notification. ## Ask Add a vmalert rule (or equivalent) on data that already exists in vmsingle from node_exporter: - `node_memory_SwapFree_bytes{instance="kai-server"}` low for N minutes (e.g. < 15% of SwapTotal). - `rate(node_pressure_memory_waiting_seconds_total{instance="kai-server"}[1m])` rising (PSI memory stall = reclaim thrash starting). - Optionally `node_memory_MemAvailable_bytes` collapsing. Route to a push notification (Discord / the existing alert path) so the spiral is caught before the blackout. ## Notes - node_exporter already exposes `node_memory_Swap{Total,Free,Cached}_bytes` and `node_pressure_memory_{waiting,stalled}_seconds_total` for kai-server. No new exporter needed. - Pairs with the `coily-update` cap (separate issue) and the heartbeat-ingestion fix (separate issue). Found during the 2026-05-30 crash investigation.

coilysiren added the

priority

label

2026-06-04 08:16:53 +00:00

coilysiren referenced this issue

2026-07-04 10:38:26 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #470

coilysiren referenced this issue

2026-07-04 10:38:33 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #471

coilyco-ops added

and removed

labels

2026-07-09 21:46:40 +00:00

coilyco-ops removed the

labels

2026-07-16 01:06:19 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-07-29 03:45:41 +00:00

coilyco-ops added the

role/sysadmin

label

2026-08-25 08:02:43 +00:00

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops

2026-08-28 21:41:08 +00:00

closed this issue
added the
burndown-2026-08
label

coilyco-ops commented

2026-08-28 21:41:37 +00:00

Owner

Closed in the 2026-08-28 burn-down (#981) on age and dormancy. That was wrong, and I already held the evidence contradicting it when I made the call.

From get_node_pressure_stalls on kai-server earlier today, cumulative since boot 49 days ago:

oom_kill    13,769
pswpin      11,511,212
pswpout     20,435,028
pgmajfault  29,464,508
pgsteal_direct     498,167,041
pgscan_direct      512,421,316

13,769 OOM kills and 20.4 million swap-out pages. Direct reclaim is running at roughly a billion pages scanned and stolen. That is the memory-pressure regime this issue was filed about, still operating, on the same host.

Current instantaneous PSI is calm (memory some avg300: 0), so the box is not in a spiral right now. But this issue is not about the instantaneous state, it is about the leading edge going undetected, and its central observation stands untouched: node_exporter went dark for the entire livelock window, so post-hoc telemetry can never diagnose the terminal phase. The alert has to fire on the ramp or not at all.

The dependency has also improved since filing. #680 records the kai-server observability stack scaled to zero with ser8 now canonical, so the vmalert rule this asks for would target ser8 rather than a local vmsingle. That is a change of target, not a reason the work is obsolete.

Recommend reopening. state:closed label:burndown-2026-08 recovers it, and it is worth doing individually rather than waiting for a bulk reversal.

Why I got it wrong

I sorted the stale batch by age, priority and comment count, and closed on those signals without re-reading the bodies against evidence I had gathered hours earlier in the same session. The measurement was in my hands and I did not connect it. Same failure as the two merge errors recorded in #981: judging an issue by its metadata rather than by what is currently true about the system it describes.

## I closed this as stale and I should not have. Recommend reopening. Closed in the 2026-08-28 burn-down (#981) on age and dormancy. That was wrong, and I already held the evidence contradicting it when I made the call. From `get_node_pressure_stalls` on kai-server earlier today, cumulative since boot 49 days ago: ``` oom_kill 13,769 pswpin 11,511,212 pswpout 20,435,028 pgmajfault 29,464,508 pgsteal_direct 498,167,041 pgscan_direct 512,421,316 ``` **13,769 OOM kills and 20.4 million swap-out pages.** Direct reclaim is running at roughly a billion pages scanned and stolen. That is the memory-pressure regime this issue was filed about, still operating, on the same host. Current instantaneous PSI is calm (`memory some avg300: 0`), so the box is not in a spiral right now. But this issue is not about the instantaneous state, it is about the **leading edge going undetected**, and its central observation stands untouched: node_exporter went dark for the entire livelock window, so post-hoc telemetry can never diagnose the terminal phase. The alert has to fire on the ramp or not at all. The dependency has also improved since filing. #680 records the kai-server observability stack scaled to zero with ser8 now canonical, so the vmalert rule this asks for would target ser8 rather than a local vmsingle. That is a change of target, not a reason the work is obsolete. **Recommend reopening.** `state:closed label:burndown-2026-08` recovers it, and it is worth doing individually rather than waiting for a bulk reversal. ### Why I got it wrong I sorted the stale batch by age, priority and comment count, and closed on those signals without re-reading the bodies against evidence I had gathered hours earlier in the same session. The measurement was in my hands and I did not connect it. Same failure as the two merge errors recorded in #981: judging an issue by its metadata rather than by what is currently true about the system it describes.

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#184

No description provided.

Rows
Columns