coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

kai-server unhealthy: forgejo bounce, runner crashloop, missing /github/pat, signoz/metrics 500s #308

New issue
Open
opened 2026-06-12 10:18:13 +00:00 by coilysiren · 0 comments
coilysiren commented 2026-06-12 10:18:13 +00:00
Owner
Copy link

Observed 2026-06-12 ~03:15 PT while verifying GitHub mirrors (agentic-os-kai#662). Forgejo answered 502 then refused connections for several minutes, then recovered. Cluster state afterward:

  • forgejo-db-0 at 18 restarts and forgejo app pod at 10, both restarted in the same minute - looks like a node-level event (memory pressure?), worth checking node conditions and dmesg
  • forgejo-runner-0 and forgejo-runner-1 in CrashLoopBackOff (runner + dind containers), ~66m old
  • external-secrets UpdateFailed everywhere: SSM key /github/pat does not exist (github-pat, docker-registry ExternalSecrets failing in several namespaces)
  • signoz-0 and metrics-server readiness probes failing with 500, chi-signoz-clickhouse BackOff
  • three persistentvolumes stuck in VolumeFailedDelete (helper pods missing)

Possibly aggravated by today's fleet-wide mirror pushes hammering Forgejo, but the runner crashloop and missing /github/pat predate the bounce.

Observed 2026-06-12 ~03:15 PT while verifying GitHub mirrors (agentic-os-kai#662). Forgejo answered 502 then refused connections for several minutes, then recovered. Cluster state afterward: * forgejo-db-0 at 18 restarts and forgejo app pod at 10, both restarted in the same minute - looks like a node-level event (memory pressure?), worth checking node conditions and dmesg * forgejo-runner-0 and forgejo-runner-1 in CrashLoopBackOff (runner + dind containers), ~66m old * external-secrets UpdateFailed everywhere: SSM key /github/pat does not exist (github-pat, docker-registry ExternalSecrets failing in several namespaces) * signoz-0 and metrics-server readiness probes failing with 500, chi-signoz-clickhouse BackOff * three persistentvolumes stuck in VolumeFailedDelete (helper pods missing) Possibly aggravated by today's fleet-wide mirror pushes hammering Forgejo, but the runner crashloop and missing /github/pat predate the bounce.
coilysiren added
P1
 and removed
P0
 labels 2026-06-17 08:39:27 +00:00
coilyco-ops added
P2
 and removed
P0
 labels 2026-07-09 21:57:13 +00:00
coilyco-ops added
P1
 and removed
P2
 labels 2026-07-10 09:00:13 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#308
No description provided.