coilyco-flight-deck/infrastructure
Watch
2
0
Code Issues 111 Pull requests Projects Releases Packages Wiki Activity Actions

Orphan-reaper for kai-server: reap PPID-1 container payloads holding stale locks #316

New issue
Closed
opened 2026-06-12 11:20:02 +00:00 by coilysiren · 0 comments
coilysiren commented 2026-06-12 11:20:02 +00:00
Owner
Copy link

Mitigation 3 from #311. The disease is deterministic so the cure scripts cleanly: find PPID-1 processes whose cgroup maps to no live pod, confirm they hold the contested lock, kill them, delete the crashlooped pods. Two deployment shapes that leave k3s config untouched: a privileged hostPID DaemonSet shipped as a manifest from this repo, or a standalone orphan-reaper.service systemd unit ordered After=k3s.service (separate unit, not a drop-in). Durable-state workloads (VictoriaMetrics, ClickHouse) need this; scratch-state workloads are covered by the init-wipe ticket instead.

Mitigation 3 from #311. The disease is deterministic so the cure scripts cleanly: find PPID-1 processes whose cgroup maps to no live pod, confirm they hold the contested lock, kill them, delete the crashlooped pods. Two deployment shapes that leave k3s config untouched: a privileged hostPID DaemonSet shipped as a manifest from this repo, or a standalone orphan-reaper.service systemd unit ordered After=k3s.service (separate unit, not a drop-in). Durable-state workloads (VictoriaMetrics, ClickHouse) need this; scratch-state workloads are covered by the init-wipe ticket instead.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
aos/claude/uv47-state-ambient-taxonomy
eval/ansible-slug-tidy
eval/label-advocate
aos/claude/mt75-kubectl-context-callsites
eval/slug-pass-labels
aos/claude/mt75-coredns-hairpin
eval/label-devrel-relations
eval/label-platform-engineer
eval/label-descriptions-retitle
eval/label-descriptions-roster
aos/claude/wz68-mirror-push
aos/claude/wz68-mirror-watch
aos/claude/wz68-bot-identity
aos/claude/wz68-bot-handle
aos/claude/wz68
aos/claude/ff54-retire-coilyco-ai-hook
aos/claude/ff54-retire-issue-refs
aos/claude/ue86
aos/claude/yq86
aos/claude/ser8-image-build-929
ops/zombie-retire-secret-strays
ops/917-alert-unit-name
ops/917-restic-failure-alert
ops/914-drop-comfyui-cname
ops/914-drop-stale-dns-instruction
ops/retire-coilysiren-me-mail
ops/911-dns-mail-auth
aos/claude/fj88
ops/393-aos-pin-rollout
ops/708-o11y-telegram-per-host-cache
chore/claude-permissions-allow-all
fix/comfyui-preflight-whatif
fix/comfyui-supervisor-job-object
fix/forgejo-creds-windows-ssm
fix/comfyui-orphan-on-refused-restart
fix/windows-projects-root
fix/comfyui-restart-race
windows-tasks-s4u
windows-attribution-bash-shim-python
fix/comfyui-whatif-verification
windows-terminal-kai-tower-3026
fix/setup-git-lfs-windows-path
chore/drop-ward-yaml
fix/terraform-docs-action-arg
fix/mint-key-orphan
feat/public-asset-host
aos/claude/ka54-failclosed
aos/claude/ka54-cleanup
aos/claude/ka54
aos/claude/uq98-unattended-forgejo
aos/claude/uq98-disk-domains
aos/claude/xk58
aos/claude/aw85-bands
aos/claude/ym96-docs-band
aos/claude/wt57-alerts-to-mapper
aos/claude/xu59-role-trailer
aos/claude/ym96-verb-convert
aos/claude/ym96-token-cleanup
aos/claude/ym96-label-sync
aos/claude/wt57-atlas-cleanup
fix/mirror-unskip-sirens-echo
aos/claude/uz86-session-name
retire-code-fleet-watch-830
codex/alacritty-link-rollout
chore/mcp-beaver-fleet-watch
aos/claude/attribution-interpreter
aos/claude/mg96-fm
aos/claude/js44
coilyco-ops/deprovision-tangled-knot
aos/claude/mg96
aos/claude/zv96
outdated-branch-815
aos/claude/agent-proxy-publisher-runner
aos/claude/git-attribution-pin-interpreter
aos/claude/restore-tag-quarantine
aos/claude/mirror-workflows-permission
aos/claude/forgejo-16
aos/claude/dind-mtu
aos/claude/precommit-egress-proxy
aos/claude/qa57-branch-protection
chore/egress-proxy-open
chore/egress-allow-code-forgejo
aos/claude/qa57
chore/aos-precommit-v0.18-bump
aos/claude/ck67
infra/claude/ser8-runner-force-pull
aos/claude/runner-scratch-10gi
aos/claude/runner-recycle-selector
aos/claude/xk86
aos/codex/20260806t002312z-7943-00371332
codex/sirens-echo-deepseek-tailnet-20260805
codex/daily-aos-sync
issue-756-remove-role-provider-symlink
aos/codex/sirens-echo-runner-rename
aos/codex/20260731t171534z-9817-e7959fba
issue-697-aos-check-config
issue-694-aos-convergence
ops/remove-mac-models-launchd-20260729
ops/remove-mac-models-20260729
tailscale-proxy-converge-self-heal
issue-587
backup/issue-785-first-person
issue-785-first-person
ops/galaxy-gen-registry-rotate
ops/website-deploy-runner
codex/history-derived-skill-20260728
ops/k3s-pod-capacity-config
ops/publisher-runners
codex/code-fleet-watch-v2
codex/code-fleet-watch
recovery/2026-07-28-triaged-branch-archive
recovery/2026-07-27-local-work
recovery/machine-exit-20260727
issue-653-registry-retirement
issue-593
fix/aws-iam-ssm-import
feat/warded-ssm-identity
issue-500
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
burndown-2026-08
Closed in the 2026-08-26 backlog burn-down. Reopen freely: state:closed label:burndown-2026-08 recovers the whole set.

  
autonomy
async-consult
 A human needs to consult on the issue to upgrade it to headless

  
autonomy
epic
 This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  
autonomy
headless
 The agent can perform the work on its own

  
autonomy
live-collab
 The agent and the human need to work together in realtime

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
priority
P0
 priority tier

  
priority
P1
 priority tier

  
priority
P2
 priority tier

  
priority
P3
 priority tier

  
priority
P4
 priority tier

  
qa-fixture
Disposable issue admitted to the bounded Ward QA verification lane.

  
role/advocate
requires work from the Developer Advocate seat

  
role/director
requires work from the Portfolio Director seat

  
role/exec
requires work from the exec role

  
role/frontend
requires work from the Frontend Engineer seat

  
role/gamedev
requires work from the Game Developer seat

  
role/human
requires a person, and specifically not an agent seat

  
role/platform
requires work from the Platform Engineer seat

  
role/qa
requires work from the QA role

  
role/science
requires work from the Applied Scientist seat

  
role/sysadmin
requires work from the Systems Administrator seat

  
state
ambient
 ambient and ephemeral work, held as a maintained document rather than a queue

No labels
burndown-2026-06
burndown-2026-08
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/advocate
role/director
role/exec
role/frontend
role/gamedev
role/human
role/platform
role/qa
role/science
role/sysadmin
state
ambient
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-flight-deck/infrastructure#316
No description provided.