kubectl logs/exec 502 against tailnet-joined nodes #10

New issue

Closed

opened 2026-05-23 20:54:26 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-23 20:54:26 +00:00

Owner

Originally filed by @coilysiren on 2026-05-22T13:55:43Z - https://github.com/coilysiren/infrastructure/issues/279

Symptom - kubectl logs <pod> and kubectl exec against pods on kai-macbook-pro-vm return 502 Bad Gateway: proxy error from 127.0.0.1:6443 while dialing 100.96.209.24:10250.

Ruled out

kubelet is listening on *:10250
the VM reaches its own kubelet on the tailnet IP: https://100.96.209.24:10250/healthz returns 401, so the endpoint is up
no firewall: ufw is inactive in the VM
the kubelet serving cert SAN includes 100.96.209.24
tailnet path is healthy: tailscale ping kai-server returns a direct pong at 24ms

Likely root - either the k3s egress-selector-mode=agent konnectivity tunnel, or a kai-server-initiated tailnet path asymmetry (the VM is double-NATed behind the Mac). The apiserver-to-kubelet direction specifically.

Context - kai-macbook-pro-vm is a Lima VM on the Mac, joined over Tailscale, intentionally cordoned. Pod scheduling and local container execution work (verified via crictl logs on the node). Only the apiserver-proxied logs/exec path is affected. The kai-desktop-tower-wsl node likely has the same class of issue, worse: its InternalIP 172.27.244.126 is a WSL NAT address that is not control-plane-routable at all.

Found while standing up the Mac as a learning node.

_Originally filed by @coilysiren on 2026-05-22T13:55:43Z - [https://github.com/coilysiren/infrastructure/issues/279](https://github.com/coilysiren/infrastructure/issues/279)_ **Symptom** - `kubectl logs <pod>` and `kubectl exec` against pods on `kai-macbook-pro-vm` return `502 Bad Gateway`: `proxy error from 127.0.0.1:6443 while dialing 100.96.209.24:10250`. **Ruled out** - kubelet is listening on `*:10250` - the VM reaches its own kubelet on the tailnet IP: `https://100.96.209.24:10250/healthz` returns 401, so the endpoint is up - no firewall: `ufw` is inactive in the VM - the kubelet serving cert SAN includes `100.96.209.24` - tailnet path is healthy: `tailscale ping kai-server` returns a direct pong at 24ms **Likely root** - either the k3s `egress-selector-mode=agent` konnectivity tunnel, or a kai-server-initiated tailnet path asymmetry (the VM is double-NATed behind the Mac). The apiserver-to-kubelet direction specifically. **Context** - `kai-macbook-pro-vm` is a Lima VM on the Mac, joined over Tailscale, intentionally cordoned. Pod scheduling and local container execution work (verified via `crictl logs` on the node). Only the apiserver-proxied logs/exec path is affected. The `kai-desktop-tower-wsl` node likely has the same class of issue, worse: its InternalIP `172.27.244.126` is a WSL NAT address that is not control-plane-routable at all. **Found while** standing up the Mac as a learning node.

coilysiren referenced this issue

2026-05-28 00:21:03 +00:00

Revoke GH Actions tailnet credentials after rip-out sweep #160

coilysiren added the

priority

label

2026-06-04 08:16:57 +00:00

coilysiren referenced this issue

2026-06-07 03:46:41 +00:00

Full-mesh reachability healthcheck (N×N connectivity matrix) across the fleet #255

coilysiren commented

2026-06-17 08:23:00 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:23:01 +00:00