coilyco-flight-deck/infrastructure

Watch

kai-desktop-tower: 2:50am Windows maintenance reboot drops both tailnet nodes, no remote recovery (~19h offline) #257

New issue

Closed

opened 2026-06-07 05:00:12 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-06-07 05:00:12 +00:00

Owner

Summary

kai-desktop-tower dropped off the tailnet at approximately 2:50am and stayed dark for ~19h with no remote recovery path. The Xfinity app still showed the box "online" at the link layer, but both of its tailnet nodes were gone and nothing on it was reachable from any vantage point. The 2:50am timing places this squarely in the Windows automatic-maintenance / Update window, and the most likely root cause is a maintenance reboot that hung (configuring-updates / recovery prompt) so tailscale never came back.

Evidence

Both tailnet nodes went offline at the same instant. From ser8 (tailscale status): kai-desktop-tower (Windows) and kai-desktop-tower-wsl (WSL/k3s node) both last seen ~19h ago, identical timestamp. One service crashing cannot take out two independent tailscale installs at once. This is a whole-machine event.
Link layer up, everything else down. Xfinity reported the device online (NIC/radio associated), which rules out a full power-off, but it was unreachable on the tailnet and exposed no service on the local segment (probed from ser8 over IPv6: no RDP / SMB / SSH / Ollama / WinRM).
WSL node down is consistent with a reboot - WSL2 does not auto-start its distro/sshd, so the WSL tailnet node dropping is expected after any Windows restart.
Rest of the fleet healthy throughout (kai-server, tailnet, ser8).

Why there was no remote recovery

Every lever was unavailable at once:

Not on the Kasa strip - no remote hard power-cycle (unlike kai-server). See #256.
No reachable service surface - tailscale was the only inbound door, and it was down. LAN/segment offers nothing to log into.
WiFi, not Ethernet - so Wake-on-LAN is not an option (WoWLAN does not reliably wake a desktop from sleep, and not at all from a hung/soft-off state).

Net result: the only fix was a physical power-button cycle at the desk.

Proposed fixes (remote-recoverability cluster)

In rough priority:

Wire the tower to Ethernet. More stable than WiFi for an always-on host, and it enables real Wake-on-LAN so a same-segment box (ser8) can wake it next time.
Move the tower onto the Kasa HS300. Two empty energized outlets (slots 2/3) are free. Gives a remote hard power-cycle for the "hung, not asleep" case that WoL cannot solve. Overlaps with #256 (BIOS Restore-on-AC-Power-Loss is the other half of that path).
Stop it sleeping and tame the update window. Power plan -> never sleep. Set Windows active hours so the automatic-maintenance reboot cannot strand the box at 2:50am.
Tailscale watchdog + heartbeat. A scheduled task that restarts tailscaled after N minutes offline, plus an alert, so an offline node pages instead of going silent for 19h. Pairs with #242 (WSL always-on hardening).

#256 - Restore-on-AC-Power-Loss BIOS (the no-remote-power-cycle gap this incident hit).
#242 - always-on hardening for the kai-desktop-tower-wsl k3s node.

Acceptance

Tower has a remote recovery path that does not require a walk to the desk (Ethernet+WoL for sleep, Kasa strip for hung).
A 2:50am-style drop produces an alert rather than 19h of silence.

## Summary `kai-desktop-tower` dropped off the tailnet at approximately **2:50am** and stayed dark for ~19h with **no remote recovery path**. The Xfinity app still showed the box "online" at the link layer, but both of its tailnet nodes were gone and nothing on it was reachable from any vantage point. The 2:50am timing places this squarely in the **Windows automatic-maintenance / Update window**, and the most likely root cause is a **maintenance reboot that hung** (configuring-updates / recovery prompt) so tailscale never came back. ## Evidence - **Both tailnet nodes went offline at the same instant.** From `ser8` (`tailscale status`): `kai-desktop-tower` (Windows) and `kai-desktop-tower-wsl` (WSL/k3s node) both **last seen ~19h ago**, identical timestamp. One service crashing cannot take out two independent tailscale installs at once. This is a whole-machine event. - **Link layer up, everything else down.** Xfinity reported the device online (NIC/radio associated), which rules out a full power-off, but it was unreachable on the tailnet and exposed no service on the local segment (probed from `ser8` over IPv6: no RDP / SMB / SSH / Ollama / WinRM). - **WSL node down is consistent with a reboot** - WSL2 does not auto-start its distro/sshd, so the WSL tailnet node dropping is expected after any Windows restart. - Rest of the fleet healthy throughout (`kai-server`, tailnet, `ser8`). ## Why there was no remote recovery Every lever was unavailable at once: * **Not on the Kasa strip** - no remote hard power-cycle (unlike `kai-server`). See #256. * **No reachable service surface** - tailscale was the only inbound door, and it was down. LAN/segment offers nothing to log into. * **WiFi, not Ethernet** - so Wake-on-LAN is not an option (WoWLAN does not reliably wake a desktop from sleep, and not at all from a hung/soft-off state). Net result: the only fix was a physical power-button cycle at the desk. ## Proposed fixes (remote-recoverability cluster) In rough priority: 1. **Wire the tower to Ethernet.** More stable than WiFi for an always-on host, and it enables real Wake-on-LAN so a same-segment box (`ser8`) can wake it next time. 2. **Move the tower onto the Kasa HS300.** Two empty energized outlets (slots 2/3) are free. Gives a remote hard power-cycle for the "hung, not asleep" case that WoL cannot solve. Overlaps with #256 (BIOS Restore-on-AC-Power-Loss is the other half of that path). 3. **Stop it sleeping and tame the update window.** Power plan -> never sleep. Set Windows active hours so the automatic-maintenance reboot cannot strand the box at 2:50am. 4. **Tailscale watchdog + heartbeat.** A scheduled task that restarts `tailscaled` after N minutes offline, plus an alert, so an offline node pages instead of going silent for 19h. Pairs with #242 (WSL always-on hardening). ## Related - #256 - Restore-on-AC-Power-Loss BIOS (the no-remote-power-cycle gap this incident hit). - #242 - always-on hardening for the `kai-desktop-tower-wsl` k3s node. ## Acceptance - Tower has a remote recovery path that does not require a walk to the desk (Ethernet+WoL for sleep, Kasa strip for hung). - A 2:50am-style drop produces an alert rather than 19h of silence.

coilysiren added the

priority

label

2026-06-17 07:05:42 +00:00

coilysiren commented

2026-06-17 08:22:35 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:22:35 +00:00

closed this issue
added the
burndown-2026-06
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#257

No description provided.

Rows
Columns