CoreDNS SERVFAIL window took out CI, Flux, and external-secrets (08:29 to 08:54 UTC, 2026-08-25) #926

New issue

Open

opened 2026-08-25 09:10:16 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-25 09:10:16 +00:00

Owner

What failed

Three coilysiren/website Actions runs on the same push, each dying at the identical line:

Error response from daemon: Get "https://forgejo.coilysiren.me/v2/": dial tcp: lookup forgejo.coilysiren.me on 10.43.0.10:53: server misbehaving

run 581, job test (id 42788), workflow mirror-to-github.yml
run 582, job publish (id 42791), workflow publish-image.yml
run 583, job Scan for secrets (id 42792), workflow trufflehog.yml

All three call docker login forgejo.coilysiren.me through scripts/ci/run-in-container.sh or scripts/publish-image.sh, so one DNS failure took all three. Note the split inside a single job: actions/checkout succeeded because it uses forgejo.forgejo.svc.cluster.local, and the image pull failed because the registry constant is the public hostname. The git path has no external-DNS dependency and the container path does.

Same SERVFAIL, same resolver, other consumers:

external-secrets in authelia and sirens-deep: Post "https://ssm.us-east-1.amazonaws.com/": dial tcp: lookup ssm.us-east-1.amazonaws.com on 10.43.0.10:53: server misbehaving at 08:53:49 through 08:54:05.
flux-system GitRepository flux-system and bridge-deploy: unable to list remote for 'https://forgejo.coilysiren.me/...' with a context deadline, 08:52:44 and 08:53:16.

Timeline

08:29:18-08:29:43 - a simultaneous readiness-probe wave: coredns /ready, traefik /ping, registry /v2/, both external-secrets webhooks, and metrics-server returning 500. traefik also logged UnAvailableLoadBalancer: There are no available nodes for LoadBalancer.
08:30:10 - tailscale status --peers=false readiness probes timed out in lunch-money-mcp and sirens-echo.
08:52-08:54 - the SERVFAIL errors above.
08:54:05 - last observed SERVFAIL.
09:09:22 - verified recovered. An in-cluster pod (bluesky-mcp) resolved and reached the public Bluesky API successfully.

No pod is crash-looping now. CoreDNS has 8 restarts over 82 days, none recent. Node memory sits at 40 percent with no memory PSI, and Ready/MemoryPressure/PIDPressure are all healthy.

Standing config smell

The kubelet has been truncating the node's nameserver list continuously, 31,357 occurrences on the CoreDNS pod alone:

DNSConfigForming: Nameserver limits were exceeded, some nameservers have been omitted,
the applied nameserver line is: 192.168.0.1 2600:1700:4268:78e0::1 100.100.100.100

So CoreDNS's own upstream set is the LAN router, an IPv6 router address, and Tailscale MagicDNS. Tailscale was itself timing out at 08:30, which leaves the router as the only healthy upstream during the window. That is inference from the event correlation rather than something confirmed from CoreDNS logs, and CoreDNS logs from the window would settle it.

Worth considering

Trim the node's resolv.conf to three usable nameservers so kubelet stops silently choosing which three survive.
Give CoreDNS an explicit forward upstream set rather than inheriting the truncated host list.
Consider a CoreDNS hosts or rewrite entry pinning forgejo.coilysiren.me to the in-cluster ingress, so registry pulls and Flux stop depending on upstream resolution for a name the cluster already serves. The public name has to stay for TLS, so this maps the name rather than replacing it.
The three website runs need re-running once someone is happy the cause is understood. I did not re-run them: publish-image.yml pushes to the registry, and that is not a read.

Root cause of the 08:29 wave itself is not established. I only had read-only node and Kubernetes API access, and pinning it needs host logs from that window.

Cluster DNS on kai-server returned SERVFAIL for external names for roughly 25 minutes. Everything that needed to resolve a name outside the cluster failed in that window. It has since recovered on its own, so this is a hardening and root-cause issue rather than an active outage. ## What failed Three `coilysiren/website` Actions runs on the same push, each dying at the identical line: ``` Error response from daemon: Get "https://forgejo.coilysiren.me/v2/": dial tcp: lookup forgejo.coilysiren.me on 10.43.0.10:53: server misbehaving ``` * run 581, job `test` (id 42788), workflow `mirror-to-github.yml` * run 582, job `publish` (id 42791), workflow `publish-image.yml` * run 583, job `Scan for secrets` (id 42792), workflow `trufflehog.yml` All three call `docker login forgejo.coilysiren.me` through `scripts/ci/run-in-container.sh` or `scripts/publish-image.sh`, so one DNS failure took all three. Note the split inside a single job: `actions/checkout` succeeded because it uses `forgejo.forgejo.svc.cluster.local`, and the image pull failed because the registry constant is the public hostname. The git path has no external-DNS dependency and the container path does. Same SERVFAIL, same resolver, other consumers: * `external-secrets` in `authelia` and `sirens-deep`: `Post "https://ssm.us-east-1.amazonaws.com/": dial tcp: lookup ssm.us-east-1.amazonaws.com on 10.43.0.10:53: server misbehaving` at 08:53:49 through 08:54:05. * `flux-system` GitRepository `flux-system` and `bridge-deploy`: `unable to list remote for 'https://forgejo.coilysiren.me/...'` with a context deadline, 08:52:44 and 08:53:16. ## Timeline * 08:29:18-08:29:43 - a simultaneous readiness-probe wave: `coredns` `/ready`, `traefik` `/ping`, `registry` `/v2/`, both `external-secrets` webhooks, and `metrics-server` returning 500. `traefik` also logged `UnAvailableLoadBalancer: There are no available nodes for LoadBalancer`. * 08:30:10 - `tailscale status --peers=false` readiness probes timed out in `lunch-money-mcp` and `sirens-echo`. * 08:52-08:54 - the SERVFAIL errors above. * 08:54:05 - last observed SERVFAIL. * 09:09:22 - verified recovered. An in-cluster pod (`bluesky-mcp`) resolved and reached the public Bluesky API successfully. No pod is crash-looping now. CoreDNS has 8 restarts over 82 days, none recent. Node memory sits at 40 percent with no memory PSI, and Ready/MemoryPressure/PIDPressure are all healthy. ## Standing config smell The kubelet has been truncating the node's nameserver list continuously, 31,357 occurrences on the CoreDNS pod alone: ``` DNSConfigForming: Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.0.1 2600:1700:4268:78e0::1 100.100.100.100 ``` So CoreDNS's own upstream set is the LAN router, an IPv6 router address, and Tailscale MagicDNS. Tailscale was itself timing out at 08:30, which leaves the router as the only healthy upstream during the window. That is inference from the event correlation rather than something confirmed from CoreDNS logs, and CoreDNS logs from the window would settle it. ## Worth considering * Trim the node's `resolv.conf` to three usable nameservers so kubelet stops silently choosing which three survive. * Give CoreDNS an explicit `forward` upstream set rather than inheriting the truncated host list. * Consider a CoreDNS `hosts` or `rewrite` entry pinning `forgejo.coilysiren.me` to the in-cluster ingress, so registry pulls and Flux stop depending on upstream resolution for a name the cluster already serves. The public name has to stay for TLS, so this maps the name rather than replacing it. * The three website runs need re-running once someone is happy the cause is understood. I did not re-run them: `publish-image.yml` pushes to the registry, and that is not a read. Root cause of the 08:29 wave itself is not established. I only had read-only node and Kubernetes API access, and pinning it needs host logs from that window.

coilyco-ops commented

2026-08-25 09:13:43 +00:00

Author

Owner

Prior art I should have found before filing. This is not a new discovery, it is the failure the existing thread predicted and could not prove.

#811 (2026-08-13) - both k3s nodes drop a nameserver from every pod. States plainly: "I have not shown this causes a failure ... probably benign, unproven". Names the open question as which nameserver is dropped, unanswerable from the event alone.
#864 (2026-08-18) - same truncation on kai-server, filed "so it stops being rediscovered". Reaches the same verdict, "almost certainly cosmetic".
#688 (2026-07-29) - the operator task to apply and verify the k3s pod resolver. Open since July.
#923 (2026-08-25, ~3 hours before this outage) - why #688 never happened. Privileged Ansible on kai-server needs a human at a sudo prompt, so authored fixes sit unapplied while the repository looks like it shipped. It cites the pod resolver as a confirmed instance and records dig @192.168.0.194 returning connection-refused on the host.

The fix is written. ansible/roles/k3s-config sets resolv-conf to /etc/rancher/k3s/resolv.conf with the single nameserver 192.168.0.194, and installs 20-k3s-pod-dns.conf so systemd-resolved listens there. docs/ansible-k3s.md explains the reasoning: pods reach the host's split-DNS engine rather than copying a truncated upstream list. None of it is on kai-server.

The applied nameserver line in this outage is the untrimmed one, which confirms the role is still unapplied.

What today adds, stated no more strongly than the evidence supports: the truncated set is the LAN gateway, an IPv6 gateway address, and Tailscale MagicDNS, and Tailscale probes were timing out at 08:30, minutes before CoreDNS began returning SERVFAIL. That is consistent with #811's unproven failure mode and does not establish it. CoreDNS logs from the window would.

So this issue is better read as the first observed cost of #923 rather than a separate DNS problem. If someone wants to fold it into #811, that is reasonable, though the incident timeline is worth keeping somewhere.

Prior art I should have found before filing. This is not a new discovery, it is the failure the existing thread predicted and could not prove. * #811 (2026-08-13) - both k3s nodes drop a nameserver from every pod. States plainly: "I have not shown this causes a failure ... probably benign, unproven". Names the open question as which nameserver is dropped, unanswerable from the event alone. * #864 (2026-08-18) - same truncation on kai-server, filed "so it stops being rediscovered". Reaches the same verdict, "almost certainly cosmetic". * #688 (2026-07-29) - the operator task to apply and verify the k3s pod resolver. Open since July. * #923 (2026-08-25, ~3 hours before this outage) - why #688 never happened. Privileged Ansible on kai-server needs a human at a sudo prompt, so authored fixes sit unapplied while the repository looks like it shipped. It cites the pod resolver as a confirmed instance and records `dig @192.168.0.194` returning connection-refused on the host. The fix is written. `ansible/roles/k3s-config` sets `resolv-conf` to `/etc/rancher/k3s/resolv.conf` with the single nameserver `192.168.0.194`, and installs `20-k3s-pod-dns.conf` so systemd-resolved listens there. `docs/ansible-k3s.md` explains the reasoning: pods reach the host's split-DNS engine rather than copying a truncated upstream list. None of it is on kai-server. The applied nameserver line in this outage is the untrimmed one, which confirms the role is still unapplied. What today adds, stated no more strongly than the evidence supports: the truncated set is the LAN gateway, an IPv6 gateway address, and Tailscale MagicDNS, and Tailscale probes were timing out at 08:30, minutes before CoreDNS began returning SERVFAIL. That is consistent with #811's unproven failure mode and does not establish it. CoreDNS logs from the window would. So this issue is better read as the first observed cost of #923 rather than a separate DNS problem. If someone wants to fold it into #811, that is reasonable, though the incident timeline is worth keeping somewhere.

coilyco-ops referenced this issue

2026-08-25 09:13:55 +00:00

Both k3s nodes silently drop a nameserver from every pod, ~65k warnings and counting #811

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:51 +00:00

coilyco-ops added this to the Credentials and access milestone

2026-08-29 06:09:55 +00:00