DNSConfigForming nameserver-limit warnings are the bulk of standing event noise on kai-server

coilyco-ops commented

2026-08-18 20:15:40 +00:00

Owner

Found during a fleet health sweep on 2026-08-18. Low severity, filed so it stops being rediscovered.

Observed

Several pods on kai-server emit DNSConfigForming warnings continuously:

Nameserver limits were exceeded, some nameservers have been omitted,
the applied nameserver line is: 192.168.0.1 2600:1700:4268:78e0::1 100.100.100.100

Counts at time of sweep:

kube-system/coredns-94b8d5df-tr5fs - 23,855
observability/node-exporter-prometheus-node-exporter-qt58z - 21,167
fleet-reachability/gatus-786db97fc-t26qr - 21,126

Reading

The node's resolv.conf carries more than the three nameservers Kubernetes will propagate, so kubelet truncates and warns every time it forms a pod DNS config. The three that survive are the LAN gateway, its IPv6 counterpart, and the Tailscale MagicDNS resolver, which is the set actually wanted.

So the warning is almost certainly cosmetic, and name resolution works. The cost is volume: these three sources alone are roughly 66,000 events, which is most of the standing warning noise on the cluster and makes event streams hard to read during a real incident.

Not established: whether any resolver that matters is among the omitted ones. Worth one look at the node's resolv.conf to confirm the truncation is dropping only redundant entries.

Suggested

Trim the host resolv.conf to three or fewer nameservers through the owning Ansible role, or set an explicit resolvConf for kubelet pointing at a trimmed file. Either removes the warning at the source rather than muting it.

Found during a fleet health sweep on 2026-08-18. Low severity, filed so it stops being rediscovered. ## Observed Several pods on kai-server emit `DNSConfigForming` warnings continuously: ``` Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.0.1 2600:1700:4268:78e0::1 100.100.100.100 ``` Counts at time of sweep: * `kube-system/coredns-94b8d5df-tr5fs` - 23,855 * `observability/node-exporter-prometheus-node-exporter-qt58z` - 21,167 * `fleet-reachability/gatus-786db97fc-t26qr` - 21,126 ## Reading The node's resolv.conf carries more than the three nameservers Kubernetes will propagate, so kubelet truncates and warns every time it forms a pod DNS config. The three that survive are the LAN gateway, its IPv6 counterpart, and the Tailscale MagicDNS resolver, which is the set actually wanted. So the warning is almost certainly cosmetic, and name resolution works. The cost is volume: these three sources alone are roughly 66,000 events, which is most of the standing warning noise on the cluster and makes event streams hard to read during a real incident. Not established: whether any resolver that matters is among the omitted ones. Worth one look at the node's resolv.conf to confirm the truncation is dropping only redundant entries. ## Suggested Trim the host resolv.conf to three or fewer nameservers through the owning Ansible role, or set an explicit `resolvConf` for kubelet pointing at a trimmed file. Either removes the warning at the source rather than muting it.

coilyco-ops referenced this issue

2026-08-19 11:42:26 +00:00

kai-server: root filesystem chronically at the 85% critical line - plan a disk capacity increase #870

coilyco-ops commented

2026-08-25 05:46:01 +00:00

Author

Owner

Diagnosed to root cause. The fix already exists in this repo and has never been converged onto kai-server. Not applying it: it needs an attended sudo password and a k3s restart.

Root cause

/etc/resolv.conf on kai-server is the systemd-resolved stub with a single nameserver, which is not what kubelet reads. With systemd-resolved active, k3s selects /run/systemd/resolve/resolv.conf, and that file carries four nameservers:

192.168.0.1        LAN router, v4
2600:…             ISP, v6
100.100.100.100    Tailscale MagicDNS, v4
fd7a:…             Tailscale, v6

Kubernetes caps a pod's resolv.conf at three. Kubelet drops one and emits DNSConfigForming every time, which is why the count is what it is: 31,146 on coredns, 28,437 on node-exporter, 2,632 on gatus.

Worth noting the cluster runs disable-ipv6: true, so two of those four slots are spent on resolvers pods cannot use anyway.

The fix is already written

ansible/roles/k3s-config handles this and the design is good. k3s_pod_resolver_nameservers defaults to a single entry, 192.168.0.194, which is kai-server's own LAN address. The role sets DNSStubListenerExtra=192.168.0.194 so systemd-resolved listens there, then points pods at that one nameserver. Resolved keeps doing the split-DNS routing upstream, so pods get full resolution from a single entry and the three-nameserver limit stops being reachable at all.

There is even an assert that k3s_pod_resolver_nameservers holds between one and three entries.

Why it is not in effect

None of it has been applied:

/etc/rancher/k3s/config.yaml   ->  no resolv-conf: key
/etc/rancher/k3s/resolv.conf   ->  No such file or directory
dig @192.168.0.194             ->  connection refused

The live config.yaml does carry the current kubelet-arg block, so it is not wholly stale. The resolv-conf half specifically never landed.

This is the same shape as the restic failure-alert work in #921: kai-server's privileged Ansible roles fail unattended with Missing sudo password, so authored fixes sit unapplied. That pattern is worth its own issue if it keeps recurring.

Applying it is safe, but attended

The handler order is correct, and I checked it rather than assuming. Ansible runs handlers in definition order:

Reload k3s unit
Restart systemd-resolved
Verify the pod DNS listener - wait_for port 53 on each configured nameserver, 10s timeout
Restart k3s

So resolved comes up and is verified listening before k3s is touched. If the listener does not appear, the run aborts before the k3s restart rather than leaving pods pointed at a dead resolver. That is the guard this needs.

There is also a k3s-config-stage tag that skips the k3s restart, so the resolved half can land and be verified independently first.

just ansible-sync apply fleet privileged tags=k3s-config-stage hosts=kai-server ask_pass
dig +short @192.168.0.194 forgejo.coilysiren.me A
just ansible-sync apply fleet privileged tags=k3s-config hosts=kai-server ask_pass

The second command restarts k3s, which briefly bounces the control plane. That plus the sudo prompt is why this is Kai's to run rather than mine.

Diagnosed to root cause. **The fix already exists in this repo and has never been converged onto kai-server.** Not applying it: it needs an attended sudo password and a k3s restart. ## Root cause `/etc/resolv.conf` on kai-server is the systemd-resolved stub with a single nameserver, which is not what kubelet reads. With systemd-resolved active, k3s selects `/run/systemd/resolve/resolv.conf`, and that file carries **four** nameservers: ``` 192.168.0.1 LAN router, v4 2600:… ISP, v6 100.100.100.100 Tailscale MagicDNS, v4 fd7a:… Tailscale, v6 ``` Kubernetes caps a pod's resolv.conf at three. Kubelet drops one and emits `DNSConfigForming` every time, which is why the count is what it is: 31,146 on coredns, 28,437 on node-exporter, 2,632 on gatus. Worth noting the cluster runs `disable-ipv6: true`, so two of those four slots are spent on resolvers pods cannot use anyway. ## The fix is already written `ansible/roles/k3s-config` handles this and the design is good. `k3s_pod_resolver_nameservers` defaults to a single entry, `192.168.0.194`, which is **kai-server's own LAN address**. The role sets `DNSStubListenerExtra=192.168.0.194` so systemd-resolved listens there, then points pods at that one nameserver. Resolved keeps doing the split-DNS routing upstream, so pods get full resolution from a single entry and the three-nameserver limit stops being reachable at all. There is even an assert that `k3s_pod_resolver_nameservers` holds between one and three entries. ## Why it is not in effect None of it has been applied: ``` /etc/rancher/k3s/config.yaml -> no resolv-conf: key /etc/rancher/k3s/resolv.conf -> No such file or directory dig @192.168.0.194 -> connection refused ``` The live `config.yaml` does carry the current `kubelet-arg` block, so it is not wholly stale. The `resolv-conf` half specifically never landed. This is the same shape as the restic failure-alert work in #921: kai-server's privileged Ansible roles fail unattended with `Missing sudo password`, so authored fixes sit unapplied. That pattern is worth its own issue if it keeps recurring. ## Applying it is safe, but attended The handler order is correct, and I checked it rather than assuming. Ansible runs handlers in definition order: 1. `Reload k3s unit` 2. `Restart systemd-resolved` 3. `Verify the pod DNS listener` - `wait_for` port 53 on each configured nameserver, 10s timeout 4. `Restart k3s` So resolved comes up and is **verified listening** before k3s is touched. If the listener does not appear, the run aborts before the k3s restart rather than leaving pods pointed at a dead resolver. That is the guard this needs. There is also a `k3s-config-stage` tag that skips the k3s restart, so the resolved half can land and be verified independently first. ``` just ansible-sync apply fleet privileged tags=k3s-config-stage hosts=kai-server ask_pass dig +short @192.168.0.194 forgejo.coilysiren.me A just ansible-sync apply fleet privileged tags=k3s-config hosts=kai-server ask_pass ``` The second command restarts k3s, which briefly bounces the control plane. That plus the sudo prompt is why this is Kai's to run rather than mine.

coilyco-ops referenced this issue

2026-08-25 05:47:05 +00:00

restic: finish the failure-alert rollout on kai-server, and test the restore path #917

coilyco-ops referenced this issue

2026-08-25 05:47:29 +00:00

kai-server privileged Ansible cannot converge unattended, and authored fixes are silently sitting unapplied #923

coilyco-ops referenced this issue

2026-08-25 06:11:09 +00:00

Both k3s nodes silently drop a nameserver from every pod, ~65k warnings and counting #811

coilyco-ops referenced this issue

2026-08-25 06:11:12 +00:00

interactive: apply and verify k3s pod resolver #688

coilyco-ops added the

role/sysadmin

priority

P2

labels

2026-08-25 08:02:08 +00:00

coilyco-ops referenced this issue

2026-08-25 09:13:43 +00:00

CoreDNS SERVFAIL window took out CI, Flux, and external-secrets (08:29 to 08:54 UTC, 2026-08-25) #926

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:58 +00:00

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops commented

2026-08-28 21:40:12 +00:00

Author

Owner

Duplicate of #811, which survives.

Read both before choosing. Same root cause, same DNSConfigForming truncation, but #811 is a strict superset:

This issue covers kai-server. #811 covers both nodes, with per-node applied nameserver lines
#811 has the finding this one lacks: the two nodes order their resolvers differently. ser8 puts Tailscale MagicDNS first, kai-server puts it third, and kai-server workloads address ser8 by tailnet name. That is a potential latency and failure-order concern rather than pure noise
#811 has acceptance criteria covering deliberate per-node ordering, not just removing the warning

Both issues independently reached the same honest conclusion, that the truncation is probably benign but unproven because neither could read the node's resolv.conf.

Confirmed still true today: /etc/resolv.conf remains outside the node-stats MCP readable-root allowlist on both nodes, so the open question in both issues, which nameserver is being dropped, is still unanswerable without a shell on the host.

Part of the 2026-08-28 burn-down, rationale in #981. Tagged burndown-2026-08 for recovery.

Duplicate of **#811**, which survives. Read both before choosing. Same root cause, same `DNSConfigForming` truncation, but #811 is a strict superset: * This issue covers kai-server. #811 covers **both nodes**, with per-node applied nameserver lines * #811 has the finding this one lacks: the two nodes order their resolvers **differently**. ser8 puts Tailscale MagicDNS first, kai-server puts it **third**, and kai-server workloads address ser8 by tailnet name. That is a potential latency and failure-order concern rather than pure noise * #811 has acceptance criteria covering deliberate per-node ordering, not just removing the warning Both issues independently reached the same honest conclusion, that the truncation is probably benign but unproven because neither could read the node's `resolv.conf`. Confirmed still true today: `/etc/resolv.conf` remains outside the node-stats MCP readable-root allowlist on **both** nodes, so the open question in both issues, which nameserver is being dropped, is still unanswerable without a shell on the host. Part of the 2026-08-28 burn-down, rationale in #981. Tagged `burndown-2026-08` for recovery.

coilyco-ops

2026-08-28 21:40:16 +00:00

closed this issue
added the
burndown-2026-08
label

Rows
Columns

DNSConfigForming nameserver-limit warnings are the bulk of standing event noise on kai-server #864

Observed

Reading

Suggested

Root cause

The fix is already written

Why it is not in effect

Applying it is safe, but attended