investigate: recurring probe timeouts across stateful services #592

New issue

Closed

opened 2026-07-23 04:35:50 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-23 04:35:50 +00:00

Member

Problem

The node has recorded recurring readiness and liveness probe timeouts for about 12 days across unrelated stateful workloads, including Forgejo database, GlitchTip Valkey, SigNoz Zookeeper, Eco GNOME database, and Open WebUI.

The affected workloads are currently Ready after the 2026-07-22 recovery, so this issue tracks chronic latency rather than a current outage. The shared node and overlapping DiskPressure history suggest host I/O or runtime latency should be measured before anyone widens probe thresholds.

Scope

Correlate probe failures with rootfs usage, disk latency, containerd activity, memory pressure, and runner load.
Measure endpoint latency from inside the affected pods during healthy and degraded periods.
Separate shared node-level causes from workload-specific probe defects.
Remediate storage or runtime contention when it is the cause.
Tune a probe only when measurements show the workload is healthy but the probe budget is too narrow.

Acceptance

The investigation records a supported root cause for each affected workload class.
The owning fix removes the shared bottleneck or corrects the specific probe.
The cluster completes a representative observation window without recurring timeout warnings.

Related to the node-pressure evidence in #560.

## Problem The node has recorded recurring readiness and liveness probe timeouts for about 12 days across unrelated stateful workloads, including Forgejo database, GlitchTip Valkey, SigNoz Zookeeper, Eco GNOME database, and Open WebUI. The affected workloads are currently Ready after the 2026-07-22 recovery, so this issue tracks chronic latency rather than a current outage. The shared node and overlapping DiskPressure history suggest host I/O or runtime latency should be measured before anyone widens probe thresholds. ## Scope * Correlate probe failures with rootfs usage, disk latency, containerd activity, memory pressure, and runner load. * Measure endpoint latency from inside the affected pods during healthy and degraded periods. * Separate shared node-level causes from workload-specific probe defects. * Remediate storage or runtime contention when it is the cause. * Tune a probe only when measurements show the workload is healthy but the probe budget is too narrow. ## Acceptance * The investigation records a supported root cause for each affected workload class. * The owning fix removes the shared bottleneck or corrects the specific probe. * The cluster completes a representative observation window without recurring timeout warnings. Related to the node-pressure evidence in #560.

coilyco-ops referenced this issue

2026-07-23 04:36:22 +00:00

kai-server unhealthy: forgejo bounce, runner crashloop, missing /github/pat, signoz/metrics 500s #308

coilyco-ops referenced this issue

2026-07-23 06:35:47 +00:00

Incident: kai-server root filesystem is at 94.1% pressure #596

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-29 03:46:27 +00:00