interactive: apply and verify k3s pod resolver #688

New issue

Open

opened 2026-07-29 12:51:00 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-29 12:51:00 +00:00

Owner

Origin: coilyco-flight-deck/infrastructure#591
Candidate: e992407d0d

The sealed engineer landed the tracked host convergence but cannot apply or observe kai-server. An operator with live access must collect the acceptance evidence.

Run ward exec ansible-sync check hosts=kai-server tags=k3s-config and review the host diff.
Run the same command without check in an attended maintenance window. Expect systemd-resolved to restart before k3s and the port 53 listener assertion to pass.
Confirm /etc/rancher/k3s/resolv.conf contains one nameserver, /etc/rancher/k3s/config.yaml pins resolv-conf to it, and kubelet configz reports that managed path.
Obtain explicit approval before any diagnostic pod creation or CoreDNS rollout. Then confirm a newly created pod has no fresh DNSConfigForming event.
From CoreDNS and a representative application pod, verify internet, LAN, and tailnet lookups, using established names available to the operator.
After the next reboot and Ansible converge, repeat the file, listener, event, and lookup checks.

Record commands and bounded non-secret output here. Close only when all evidence is present.

— Codex

Origin: coilyco-flight-deck/infrastructure#591 Candidate: e992407d0d25e018600a0f9162221b74e3b37ebb The sealed engineer landed the tracked host convergence but cannot apply or observe kai-server. An operator with live access must collect the acceptance evidence. 1. Run `ward exec ansible-sync check hosts=kai-server tags=k3s-config` and review the host diff. 2. Run the same command without `check` in an attended maintenance window. Expect systemd-resolved to restart before k3s and the port 53 listener assertion to pass. 3. Confirm `/etc/rancher/k3s/resolv.conf` contains one nameserver, `/etc/rancher/k3s/config.yaml` pins `resolv-conf` to it, and kubelet configz reports that managed path. 4. Obtain explicit approval before any diagnostic pod creation or CoreDNS rollout. Then confirm a newly created pod has no fresh `DNSConfigForming` event. 5. From CoreDNS and a representative application pod, verify internet, LAN, and tailnet lookups, using established names available to the operator. 6. After the next reboot and Ansible converge, repeat the file, listener, event, and lookup checks. Record commands and bounded non-secret output here. Close only when all evidence is present. — Codex

coilyco-ops added the

autonomy

live-collab

label

2026-07-29 12:51:01 +00:00

coilyco-ops referenced this issue

2026-07-29 12:53:35 +00:00

fix: stop cluster-wide DNSConfigForming warnings #591

coilyco-ops added the

priority

label

2026-08-05 10:25:58 +00:00

coilyco-ops referenced this issue

2026-08-25 06:11:09 +00:00

Both k3s nodes silently drop a nameserver from every pod, ~65k warnings and counting #811

coilyco-ops commented

2026-08-25 06:11:12 +00:00

Author

Owner

Still the right apply step, and now fully specified. Diagnosis on #864, correct scope on #811 (it is both nodes: kai-server has 4 nameservers, ser8 has 6, and neither has resolv-conf configured).

Sequence, staged so the resolved half is proven before anything restarts k3s:

just ansible-sync apply fleet privileged tags=k3s-config-stage hosts=ser8
dig +short @<ser8-lan-ip> forgejo.coilysiren.me A
just ansible-sync apply fleet privileged tags=k3s-config hosts=ser8

ser8 first because it has passwordless sudo, so it runs unattended and proves the change on the node that is not carrying Forgejo. kai-server then needs ask_pass per #923.

Note k3s_pod_resolver_nameservers defaults to 192.168.0.194, which is kai-server's LAN address. ser8 needs its own value or a host-var, otherwise ser8's pods would be pointed at kai-server's stub listener. Worth checking before the ser8 run rather than after.

Still the right apply step, and now fully specified. Diagnosis on #864, correct scope on #811 (it is **both** nodes: kai-server has 4 nameservers, ser8 has 6, and neither has `resolv-conf` configured). Sequence, staged so the resolved half is proven before anything restarts k3s: ``` just ansible-sync apply fleet privileged tags=k3s-config-stage hosts=ser8 dig +short @<ser8-lan-ip> forgejo.coilysiren.me A just ansible-sync apply fleet privileged tags=k3s-config hosts=ser8 ``` ser8 first because it has passwordless sudo, so it runs unattended and proves the change on the node that is not carrying Forgejo. kai-server then needs `ask_pass` per #923. Note `k3s_pod_resolver_nameservers` defaults to `192.168.0.194`, which is **kai-server's** LAN address. ser8 needs its own value or a host-var, otherwise ser8's pods would be pointed at kai-server's stub listener. Worth checking before the ser8 run rather than after.

coilyco-ops added the

role/sysadmin

label

2026-08-25 08:02:18 +00:00

coilyco-ops referenced this issue

2026-08-25 09:13:43 +00:00

CoreDNS SERVFAIL window took out CI, Flux, and external-secrets (08:29 to 08:54 UTC, 2026-08-25) #926

coilyco-ops referenced this issue

2026-08-25 09:13:55 +00:00

Both k3s nodes silently drop a nameserver from every pod, ~65k warnings and counting #811

coilyco-ops added this to the Networking and DNS milestone

2026-08-29 06:09:37 +00:00