#251 - kai-server SSH sessions hang / drop - can't reliably get on-box - coilyco-flight-deck/infrastructure

coilysiren commented

2026-06-06 23:06:30 +00:00

Owner

SSH into kai-server is unreliable: interactive sessions frequently hang (freeze with no prompt / mid-session stall) and often drop the connection outright ("kicked out"). This blocks hands-on work at the box.

Why this matters now

The k3s runner outage (#215) and the tap-writer crashloop (#249 / PR #250) both need on-box recovery - apiserver is reachable for read-only coily ops kubectl, but the actual fixes (node recovery, PVC recreate, k3s kubectl apply + pod delete) require getting onto kai-server. Right now that path itself is the blocker, so it gets its own ticket and o2r channel before any on-box debugging.

Symptom

ssh kai@kai-server hangs (connects then freezes, or stalls before a prompt).
Established sessions get dropped after a while ("kicks me out").
Frequent / intermittent, not a hard down - sometimes it works.

Possibly relevant context

kai-server was hard power-cycled 2026-06-05 to clear a wedged k3s control plane (see FQT7 / #215).
Two cluster nodes (kai-macbook-pro-vm, kai-desktop-tower-wsl) are NotReady; only kai-server is Ready.
The wedged/recovering k3s may be putting the box under CPU / memory / IO pressure, which can starve sshd and stall interactive sessions.

Investigation directions (for the on-box / ansible-debug effort)

Transport: direct ssh vs Tailscale SSH - does one hang and the other not? ssh -vvv to see where it stalls (auth, channel open, shell).
Keepalive: ServerAliveInterval / ClientAliveInterval / ClientAliveCountMax - tune so a stalled link is detected and the session survives transient blips instead of silently freezing.
Box health: uptime / load avg, free -m, dmesg/OOM-killer, journalctl -u sshd, journalctl -u tailscaled, IO wait - is sshd being starved by k3s?
sshd limits: MaxStartups, MaxSessions, LoginGraceTime - whether new sessions are being refused/queued under load.
Tailnet stability: packet loss / latency to the box (the o2r ansible-debug ping/traceroute probes).

Coordination

o2r channel created for this (id + link to be added below). Diagnosis and on-box findings get posted there; this issue is the durable tracker.

SSH into kai-server is unreliable: interactive sessions frequently hang (freeze with no prompt / mid-session stall) and often drop the connection outright ("kicked out"). This blocks hands-on work at the box. ## Why this matters now The k3s runner outage (#215) and the tap-writer crashloop (#249 / PR #250) both need on-box recovery - apiserver is reachable for read-only `coily ops kubectl`, but the actual fixes (node recovery, PVC recreate, `k3s kubectl apply` + pod delete) require getting onto kai-server. Right now that path itself is the blocker, so it gets its own ticket and o2r channel before any on-box debugging. ## Symptom - `ssh kai@kai-server` hangs (connects then freezes, or stalls before a prompt). - Established sessions get dropped after a while ("kicks me out"). - Frequent / intermittent, not a hard down - sometimes it works. ## Possibly relevant context - kai-server was hard power-cycled 2026-06-05 to clear a wedged k3s control plane (see FQT7 / #215). - Two cluster nodes (`kai-macbook-pro-vm`, `kai-desktop-tower-wsl`) are `NotReady`; only kai-server is `Ready`. - The wedged/recovering k3s may be putting the box under CPU / memory / IO pressure, which can starve sshd and stall interactive sessions. ## Investigation directions (for the on-box / ansible-debug effort) - **Transport:** direct ssh vs Tailscale SSH - does one hang and the other not? `ssh -vvv` to see where it stalls (auth, channel open, shell). - **Keepalive:** `ServerAliveInterval` / `ClientAliveInterval` / `ClientAliveCountMax` - tune so a stalled link is detected and the session survives transient blips instead of silently freezing. - **Box health:** `uptime` / load avg, `free -m`, `dmesg`/OOM-killer, `journalctl -u sshd`, `journalctl -u tailscaled`, IO wait - is sshd being starved by k3s? - **sshd limits:** `MaxStartups`, `MaxSessions`, `LoginGraceTime` - whether new sessions are being refused/queued under load. - **Tailnet stability:** packet loss / latency to the box (the o2r ansible-debug ping/traceroute probes). ## Coordination o2r channel created for this (id + link to be added below). Diagnosis and on-box findings get posted there; this issue is the durable tracker.

coilysiren commented

2026-06-06 23:07:14 +00:00

Author

Owner

o2r coordination channel for this: HDBF - http://api/agent-channel/HDBF (charter posted as event 127). On-box findings and transport/keepalive diagnosis go there; this issue stays the durable tracker.

o2r coordination channel for this: **HDBF** - `http://api/agent-channel/HDBF` (charter posted as event 127). On-box findings and transport/keepalive diagnosis go there; this issue stays the durable tracker.

coilysiren referenced this issue

2026-06-07 18:54:03 +00:00

Land lan-hosts role: LAN-pin kai-server to dodge the flaky tailnet path #266

coilysiren referenced this issue

2026-06-07 18:54:11 +00:00

Land lan-hosts role: LAN-pin kai-server to dodge the flaky tailnet path #267

coilysiren referenced this issue from a commit

2026-06-07 18:55:31 +00:00

feat(ansible): add inert lan-hosts role (LAN-pin kai-server, off by default)

coilysiren referenced this issue

2026-06-12 06:33:56 +00:00

ssh-config alias kai-server -> kai-server.local on home LAN (supersedes inert lan-hosts /etc/hosts pin, #266) #294