Unexplained: CI lost its path to Forgejo at 17:42Z on 2026-08-28 and recovered on its own at 18:15Z #982

New issue

Open

opened 2026-08-29 00:15:57 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-29 00:15:57 +00:00

Owner

Consolidating the 2026-08-28 CI outage, because its cause was never found and the evidence is currently spread across two closed issues, a burn-down record, and one session's working memory.

The hairpin fix that came out of this is not the explanation. It landed 23 minutes after CI had already recovered. So if this recurs, the removed hairpin is no longer available as an answer and this list is where the next person should start.

Timeline, all measured

08-27 04:22Z   coilyco-flight-deck/ward archived
08-28 06:47Z   run 4185  success
08-28 06:49Z   run 4186  success
08-28 09:15Z   ser8 runner recycle fires (cron 15 9, UTC on ser8)
08-28 09:17Z   ser8 runner pods up
08-28 16:15Z   kai-server runner recycle fires (cron 15 9, LOCAL = 16:15Z)
08-28 16:17Z   kai-server runner pods up
08-28 17:41:16Z node-stats-exporter OOMKilled on ser8
08-28 17:43:09Z run 4187 starts -> FAILS
08-28 17:48:27Z run 4188 starts -> FAILS
08-28 17:54Z   gate: first 133s connect timeout to forgejo.coilysiren.me:443
08-28 17:56Z   ward-doctor curl gets a real HTTP 404 from the same host
08-28 18:01Z   gate: third 133s connect timeout
08-28 18:15:12Z run 4189  SUCCESS   <- recovery, unexplained
08-28 18:26-28Z runs 4190, 4191, 4192  success
08-28 18:38Z   ser8 CoreDNS split-horizon applied
08-28 18:42Z   kai-server CoreDNS split-horizon applied

Both ends are unexplained. Nothing is known to have changed at 17:42Z, and nothing is known to have changed at 18:15Z.

The strongest clue

At 17:56Z, in the middle of the failure window, ward-doctor issued a curl to forgejo.coilysiren.me and got back a real HTTP 404. A 404 is an application response. The host was up, TLS completed, Traefik routed, and Forgejo answered.

At the same time gate was taking 133 seconds to a connect timeout against the same hostname from the same cluster.

So this was never "Forgejo is down" or "the path is broken." It was a path that worked for some flows and not others, concurrently. Any explanation has to account for that asymmetry, and most of the obvious ones do not.

Ruled out, with the evidence

Runner recycle - ser8 pods came up 09:17Z, kai-server 16:17Z. Both hours before the break, and green runs happened after both
Stale DNS - the Route53 A record is 99.110.50.213, which matched the live WAN address measured against checkip.amazonaws.com the same afternoon
kai-server resource exhaustion - CPU 7.4 percent, memory 43.7 percent, PSI low, no active pressure
Forgejo being down - the 17:56Z 404 above
Ward archival - happened 37 hours earlier, and its release assets returned 200 unauthenticated throughout
The resolve_release_ref fallback - fires only after a network failure. It converted an unreadable error into a 404 and cost an hour of misdirection, but it is a diagnostic defect, not a cause
The 17:41:16Z OOM on ser8 - one minute before the break, and I do not believe it is related. It was a cgroup-limit kill on a container with a 128Mi ceiling while the node had 52.3 GB free and PSI memory flat at zero. A container hitting its own limit does not disrupt other pods' networking. Recorded because the coincidence is close enough that someone will find it again

Standing hypotheses, none confirmed

A. Gateway conntrack or NAT state exhaustion. Runner traffic was double-NATed, masqueraded by flannel and hairpinned again at the router. That path degrades under concurrency rather than failing cleanly, and it self-recovers when state ages out. Fits the asymmetry, the duration, and the spontaneous recovery. Not measurable - no access to the router, and no counters were captured at the time.

B. Transient upstream or WAN event. Would produce the same shape and also self-recover. Not measurable retroactively.

C. Concurrency threshold. Roughly 25 runner pods across both clusters long-poll the task queue continuously. A burst may have pushed the hairpin past a limit. Partially testable in principle, except klipper-lb SNATs every client to one pod IP so the Traefik access log cannot attribute load by source.

D. Something local to ser8. The failing runners were ser8's flight-deck pool, and ser8's own OOM lands inside the window. No mechanism found, kept only because the coincidence is unexplained.

A and C are the same family and are the most likely. Neither is provable from what was captured.

Why this is worth keeping open

The hairpin is gone. Both clusters now resolve forgejo.coilysiren.me in-cluster to the tailnet address, landed in #976, and #912 is closed. So the leading hypothesis is no longer reachable as an explanation for a recurrence. If checkout starts timing out again, hypotheses A and C are off the table and something new is happening.

Nothing was instrumented to catch this next time. That is the real gap.

Done-condition

Either the 17:42Z trigger is identified, or instrumentation exists that would identify an equivalent event. The second is the achievable one:

Gateway conntrack and NAT table counters, sampled and retained. This is the measurement whose absence blocked hypothesis A
Per-source attribution at the ingress, which today is impossible because klipper-lb collapses every client to one address. Replacing klipper-lb or fronting Traefik with PROXY protocol
A recorded probe from inside a runner pod, so "the path is degraded" is a datapoint rather than an inference from job failures

coilyco-flight-deck/agentic-os#1344 - closed. The incident as filed, with three of my own corrections recorded in its comments
#912 - closed. The hairpin, fixed
#976 - the declarative landing of the fix
coilyco-bridge/deploy#436 - the ser8 OOM at 17:41:16Z, with the same coincidence recorded from that side

Consolidating the 2026-08-28 CI outage, because its cause was never found and the evidence is currently spread across two closed issues, a burn-down record, and one session's working memory. **The hairpin fix that came out of this is not the explanation.** It landed 23 minutes after CI had already recovered. So if this recurs, the removed hairpin is no longer available as an answer and this list is where the next person should start. ## Timeline, all measured ``` 08-27 04:22Z coilyco-flight-deck/ward archived 08-28 06:47Z run 4185 success 08-28 06:49Z run 4186 success 08-28 09:15Z ser8 runner recycle fires (cron 15 9, UTC on ser8) 08-28 09:17Z ser8 runner pods up 08-28 16:15Z kai-server runner recycle fires (cron 15 9, LOCAL = 16:15Z) 08-28 16:17Z kai-server runner pods up 08-28 17:41:16Z node-stats-exporter OOMKilled on ser8 08-28 17:43:09Z run 4187 starts -> FAILS 08-28 17:48:27Z run 4188 starts -> FAILS 08-28 17:54Z gate: first 133s connect timeout to forgejo.coilysiren.me:443 08-28 17:56Z ward-doctor curl gets a real HTTP 404 from the same host 08-28 18:01Z gate: third 133s connect timeout 08-28 18:15:12Z run 4189 SUCCESS <- recovery, unexplained 08-28 18:26-28Z runs 4190, 4191, 4192 success 08-28 18:38Z ser8 CoreDNS split-horizon applied 08-28 18:42Z kai-server CoreDNS split-horizon applied ``` **Both ends are unexplained.** Nothing is known to have changed at 17:42Z, and nothing is known to have changed at 18:15Z. ## The strongest clue At **17:56Z**, in the middle of the failure window, `ward-doctor` issued a curl to `forgejo.coilysiren.me` and got back a real **HTTP 404**. A 404 is an application response. The host was up, TLS completed, Traefik routed, and Forgejo answered. At the same time `gate` was taking **133 seconds to a connect timeout** against the same hostname from the same cluster. So this was never "Forgejo is down" or "the path is broken." It was a path that worked for some flows and not others, concurrently. Any explanation has to account for that asymmetry, and most of the obvious ones do not. ## Ruled out, with the evidence * **Runner recycle** - ser8 pods came up 09:17Z, kai-server 16:17Z. Both hours before the break, and green runs happened after both * **Stale DNS** - the Route53 A record is `99.110.50.213`, which matched the live WAN address measured against `checkip.amazonaws.com` the same afternoon * **kai-server resource exhaustion** - CPU 7.4 percent, memory 43.7 percent, PSI low, no active pressure * **Forgejo being down** - the 17:56Z 404 above * **Ward archival** - happened 37 hours earlier, and its release assets returned 200 unauthenticated throughout * **The `resolve_release_ref` fallback** - fires only *after* a network failure. It converted an unreadable error into a 404 and cost an hour of misdirection, but it is a diagnostic defect, not a cause * **The 17:41:16Z OOM on ser8** - one minute before the break, and I do not believe it is related. It was a cgroup-limit kill on a container with a 128Mi ceiling while the node had 52.3 GB free and PSI memory flat at zero. A container hitting its own limit does not disrupt other pods' networking. Recorded because the coincidence is close enough that someone will find it again ## Standing hypotheses, none confirmed **A. Gateway conntrack or NAT state exhaustion.** Runner traffic was double-NATed, masqueraded by flannel and hairpinned again at the router. That path degrades under concurrency rather than failing cleanly, and it self-recovers when state ages out. Fits the asymmetry, the duration, and the spontaneous recovery. **Not measurable** - no access to the router, and no counters were captured at the time. **B. Transient upstream or WAN event.** Would produce the same shape and also self-recover. Not measurable retroactively. **C. Concurrency threshold.** Roughly 25 runner pods across both clusters long-poll the task queue continuously. A burst may have pushed the hairpin past a limit. Partially testable in principle, except klipper-lb SNATs every client to one pod IP so the Traefik access log cannot attribute load by source. **D. Something local to ser8.** The failing runners were ser8's flight-deck pool, and ser8's own OOM lands inside the window. No mechanism found, kept only because the coincidence is unexplained. A and C are the same family and are the most likely. Neither is provable from what was captured. ## Why this is worth keeping open The hairpin is gone. Both clusters now resolve `forgejo.coilysiren.me` in-cluster to the tailnet address, landed in #976, and #912 is closed. So **the leading hypothesis is no longer reachable as an explanation for a recurrence.** If checkout starts timing out again, hypotheses A and C are off the table and something new is happening. Nothing was instrumented to catch this next time. That is the real gap. ## Done-condition Either the 17:42Z trigger is identified, or instrumentation exists that would identify an equivalent event. The second is the achievable one: * Gateway conntrack and NAT table counters, sampled and retained. This is the measurement whose absence blocked hypothesis A * Per-source attribution at the ingress, which today is impossible because klipper-lb collapses every client to one address. Replacing klipper-lb or fronting Traefik with PROXY protocol * A recorded probe from inside a runner pod, so "the path is degraded" is a datapoint rather than an inference from job failures ## Related * coilyco-flight-deck/agentic-os#1344 - closed. The incident as filed, with three of my own corrections recorded in its comments * #912 - closed. The hairpin, fixed * #976 - the declarative landing of the fix * coilyco-bridge/deploy#436 - the ser8 OOM at 17:41:16Z, with the same coincidence recorded from that side

coilyco-ops added this to the CI and repo hygiene milestone

2026-08-29 06:10:25 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#982

No description provided.

Rows
Columns