coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

ser8 read-only watcher: alert on the lock-class crashloop signature on kai-server #315

New issue
Closed
opened 2026-06-12 11:20:02 +00:00 by coilysiren · 1 comment
coilysiren commented 2026-06-12 11:20:02 +00:00
Owner
Copy link

Mitigation 2 from #311. The o11y outage ran 45 days because the alerting stack was the casualty. ser8 holds a read-only kubeconfig for kai-server and sits in a different failure domain, so it watches for the deterministic signature: recent k3s restart plus CrashLoopBackOff plus a lock-class log error (flock.lock, boltdb open, address already in use, Another server instance). The alert carries the diagnosis and reap commands, not a generic pod-unhealthy. Interim shape: a small poller until the full o11y stack on ser8 takes over the job.

Mitigation 2 from #311. The o11y outage ran 45 days because the alerting stack was the casualty. ser8 holds a read-only kubeconfig for kai-server and sits in a different failure domain, so it watches for the deterministic signature: recent k3s restart plus CrashLoopBackOff plus a lock-class log error (flock.lock, boltdb open, address already in use, Another server instance). The alert carries the diagnosis and reap commands, not a generic pod-unhealthy. Interim shape: a small poller until the full o11y stack on ser8 takes over the job.
coilysiren commented 2026-06-17 08:22:32 +00:00
Author
Owner
Copy link

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.
coilysiren 2026-06-17 08:22:32 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#315
No description provided.