Both k3s nodes silently drop a nameserver from every pod, ~65k warnings and counting #811

New issue

Open

opened 2026-08-13 06:33:02 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 06:33:02 +00:00

Member

What this is

Every pod scheduled on either k3s node gets a truncated DNS config, and the kubelet warns about it each time:

DNSConfigForming — Nameserver limits were exceeded, some nameservers have been omitted

Kubernetes hard-caps a pod's resolv.conf at 3 nameservers. Both hosts supply more, so one or more is dropped from every pod on the node.

The two nodes are configured differently

Node	Applied nameserver line	Sample counts
kai-server	`192.168.0.1`, `2600:1700:4268:78e0::1`, `100.100.100.100`	coredns 17411, gatus 14671, node-exporter 14711
ser8	`100.100.100.100`, `fd7a:115c:a1e0::53`, `75.75.75.75`	coredns 32500, gatus 32475

Roughly 65,000 warning events across the two nodes for the observed pods alone, and every pod on each node contributes its own counter.

The ordering divergence is the part worth attention:

ser8 puts Tailscale MagicDNS first (100.100.100.100, plus its IPv6 form fd7a:115c:a1e0::53), with Comcast (75.75.75.75) third.
kai-server puts the LAN gateway first and MagicDNS third — so a tailnet name only resolves there after two other resolvers have been tried and failed.

That matters because kai-server workloads address ser8 by tailnet name. Sirens Deep, for instance, is configured with AGENT_PROXY_URL: http://ser8:8080 and OTEL_EXPORTER_OTLP_ENDPOINT: http://ser8:30418. Those resolutions go through a resolver list where MagicDNS is last.

What I am explicitly not claiming

I have not shown this causes a failure. MagicDNS survives truncation on both nodes, so the names that matter are still resolvable. Resolver order plus ndots and search-domain handling determine real behavior, and I could not read either host's /etc/resolv.conf — it is outside the node-stats readable-root allowlist.
I do not know which nameserver is being dropped. The event reports what was applied, never what was discarded. If the fourth entry is a redundant ISP resolver this is cosmetic; if it is something load-bearing it is not. That is the actual open question, and it cannot be answered from the event.
I have not measured DNS latency on either node.

So the honest severity is: probably benign, unproven, and generating a very large amount of noise.

Why it is still worth fixing

The noise is the concrete cost. DNSConfigForming is the single highest-count event on both nodes and it crowds out everything else — during an unrelated investigation this morning it was the top line in every event query on both nodes. coilyco-gaming/sirens-echo#190 makes the case that undetected silence is expensive here; an event stream that is ~90% one repeating benign warning is the same problem approached from the other side.

Fixing it is also cheap: trim each host's resolver list to three, and the warning stops permanently on both nodes.

Acceptance

Each host supplies at most 3 nameservers, so no pod's DNS config is silently truncated.
The resolver order is a deliberate choice per node rather than an artifact, with tailnet resolution ranked according to how much each node depends on tailnet names.
The two nodes' orderings are either consistent or the difference is recorded with a reason.

Notes

No changes made. This is host-level resolv.conf convergence, so it belongs to Ansible rather than to any cluster manifest.

Next owner

Ops.

## What this is Every pod scheduled on either k3s node gets a truncated DNS config, and the kubelet warns about it each time: > `DNSConfigForming` — Nameserver limits were exceeded, some nameservers have been omitted Kubernetes hard-caps a pod's `resolv.conf` at **3 nameservers**. Both hosts supply more, so one or more is dropped from every pod on the node. ## The two nodes are configured differently | Node | Applied nameserver line | Sample counts | | --- | --- | --- | | kai-server | `192.168.0.1`, `2600:1700:4268:78e0::1`, `100.100.100.100` | coredns 17411, gatus 14671, node-exporter 14711 | | ser8 | `100.100.100.100`, `fd7a:115c:a1e0::53`, `75.75.75.75` | coredns 32500, gatus 32475 | Roughly **65,000 warning events** across the two nodes for the observed pods alone, and every pod on each node contributes its own counter. The ordering divergence is the part worth attention: - **ser8** puts Tailscale MagicDNS first (`100.100.100.100`, plus its IPv6 form `fd7a:115c:a1e0::53`), with Comcast (`75.75.75.75`) third. - **kai-server** puts the LAN gateway first and MagicDNS **third** — so a tailnet name only resolves there after two other resolvers have been tried and failed. That matters because kai-server workloads address ser8 by tailnet name. Sirens Deep, for instance, is configured with `AGENT_PROXY_URL: http://ser8:8080` and `OTEL_EXPORTER_OTLP_ENDPOINT: http://ser8:30418`. Those resolutions go through a resolver list where MagicDNS is last. ## What I am explicitly not claiming - **I have not shown this causes a failure.** MagicDNS survives truncation on both nodes, so the names that matter are still resolvable. Resolver *order* plus `ndots` and search-domain handling determine real behavior, and I could not read either host's `/etc/resolv.conf` — it is outside the node-stats readable-root allowlist. - **I do not know which nameserver is being dropped.** The event reports what was applied, never what was discarded. If the fourth entry is a redundant ISP resolver this is cosmetic; if it is something load-bearing it is not. **That is the actual open question**, and it cannot be answered from the event. - I have not measured DNS latency on either node. So the honest severity is: probably benign, unproven, and generating a very large amount of noise. ## Why it is still worth fixing The noise is the concrete cost. `DNSConfigForming` is the single highest-count event on both nodes and it crowds out everything else — during an unrelated investigation this morning it was the top line in every event query on both nodes. https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/190 makes the case that undetected silence is expensive here; an event stream that is ~90% one repeating benign warning is the same problem approached from the other side. Fixing it is also cheap: trim each host's resolver list to three, and the warning stops permanently on both nodes. ## Acceptance - Each host supplies at most 3 nameservers, so no pod's DNS config is silently truncated. - The resolver order is a deliberate choice per node rather than an artifact, with tailnet resolution ranked according to how much each node depends on tailnet names. - The two nodes' orderings are either consistent or the difference is recorded with a reason. ## Notes No changes made. This is host-level `resolv.conf` convergence, so it belongs to Ansible rather than to any cluster manifest. ## Next owner Ops.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#811

No description provided.

Rows
Columns