kai-server unhealthy: forgejo bounce, runner crashloop, missing /github/pat, signoz/metrics 500s #308

New issue

Closed

opened 2026-06-12 10:18:13 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-06-12 10:18:13 +00:00

Owner

Observed 2026-06-12 ~03:15 PT while verifying GitHub mirrors (agentic-os-kai#662). Forgejo answered 502 then refused connections for several minutes, then recovered. Cluster state afterward:

forgejo-db-0 at 18 restarts and forgejo app pod at 10, both restarted in the same minute - looks like a node-level event (memory pressure?), worth checking node conditions and dmesg
forgejo-runner-0 and forgejo-runner-1 in CrashLoopBackOff (runner + dind containers), ~66m old
external-secrets UpdateFailed everywhere: SSM key /github/pat does not exist (github-pat, docker-registry ExternalSecrets failing in several namespaces)
signoz-0 and metrics-server readiness probes failing with 500, chi-signoz-clickhouse BackOff
three persistentvolumes stuck in VolumeFailedDelete (helper pods missing)

Possibly aggravated by today's fleet-wide mirror pushes hammering Forgejo, but the runner crashloop and missing /github/pat predate the bounce.

Observed 2026-06-12 ~03:15 PT while verifying GitHub mirrors (agentic-os-kai#662). Forgejo answered 502 then refused connections for several minutes, then recovered. Cluster state afterward: * forgejo-db-0 at 18 restarts and forgejo app pod at 10, both restarted in the same minute - looks like a node-level event (memory pressure?), worth checking node conditions and dmesg * forgejo-runner-0 and forgejo-runner-1 in CrashLoopBackOff (runner + dind containers), ~66m old * external-secrets UpdateFailed everywhere: SSM key /github/pat does not exist (github-pat, docker-registry ExternalSecrets failing in several namespaces) * signoz-0 and metrics-server readiness probes failing with 500, chi-signoz-clickhouse BackOff * three persistentvolumes stuck in VolumeFailedDelete (helper pods missing) Possibly aggravated by today's fleet-wide mirror pushes hammering Forgejo, but the runner crashloop and missing /github/pat predate the bounce.

coilysiren referenced this issue

2026-06-12 11:13:08 +00:00

k3s.service pkill -9 containerd-shim orphans container payloads on every restart, causing recurring lock-blocked crashloops (CI, o11y) #311

coilysiren added the

priority

autonomy

async-consult

labels

2026-06-17 07:06:40 +00:00

coilyco-ops added

priority

and removed

priority

labels

2026-07-09 21:57:13 +00:00

coilyco-ops removed the

autonomy

async-consult

label

2026-07-16 01:08:58 +00:00

coilyco-ops referenced this issue

2026-07-23 04:35:46 +00:00

reconcile: retire or replace stale /github/pat ExternalSecrets #589

coilyco-ops commented

2026-07-23 04:36:21 +00:00

Owner

The 2026-07-22 follow-up split this broad incident into focused tracks:

Node DiskPressure and Forgejo eviction: #560
Broken runner recycle CronJob: #587
Stuck local-path PV deletion: #588
Missing /github/pat ExternalSecrets: #589
Recurring stateful-service probe timeouts: #592

Current workloads recovered after DiskPressure cleared. These focused issues now own the remaining durable faults.

The 2026-07-22 follow-up split this broad incident into focused tracks: * Node DiskPressure and Forgejo eviction: #560 * Broken runner recycle CronJob: #587 * Stuck local-path PV deletion: #588 * Missing `/github/pat` ExternalSecrets: #589 * Recurring stateful-service probe timeouts: #592 Current workloads recovered after DiskPressure cleared. These focused issues now own the remaining durable faults.

coilyco-ops added

and removed

labels

2026-07-29 03:45:38 +00:00

coilyco-ops closed this issue

2026-08-05 21:23:25 +00:00