coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

Orphan-reaper for kai-server: reap PPID-1 container payloads holding stale locks #316

New issue
Open
opened 2026-06-12 11:20:02 +00:00 by coilysiren · 0 comments
coilysiren commented 2026-06-12 11:20:02 +00:00
Owner
Copy link

Mitigation 3 from #311. The disease is deterministic so the cure scripts cleanly: find PPID-1 processes whose cgroup maps to no live pod, confirm they hold the contested lock, kill them, delete the crashlooped pods. Two deployment shapes that leave k3s config untouched: a privileged hostPID DaemonSet shipped as a manifest from this repo, or a standalone orphan-reaper.service systemd unit ordered After=k3s.service (separate unit, not a drop-in). Durable-state workloads (VictoriaMetrics, ClickHouse) need this; scratch-state workloads are covered by the init-wipe ticket instead.

Mitigation 3 from #311. The disease is deterministic so the cure scripts cleanly: find PPID-1 processes whose cgroup maps to no live pod, confirm they hold the contested lock, kill them, delete the crashlooped pods. Two deployment shapes that leave k3s config untouched: a privileged hostPID DaemonSet shipped as a manifest from this repo, or a standalone orphan-reaper.service systemd unit ordered After=k3s.service (separate unit, not a drop-in). Durable-state workloads (VictoriaMetrics, ClickHouse) need this; scratch-state workloads are covered by the init-wipe ticket instead.
coilysiren added
P2
 and removed
P1
 labels 2026-06-17 08:39:25 +00:00
coilyco-ops added
P2
 and removed
P3
 labels 2026-07-10 09:00:11 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#316
No description provided.