coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

forgejo runner pinned to offline kai-desktop-tower-wsl - docker-label CI queue starves #312

New issue
Open
opened 2026-06-12 11:15:30 +00:00 by coilysiren · 1 comment
coilysiren commented 2026-06-12 11:15:30 +00:00
Owner
Copy link

eco-app CI run #3 (and likely all docker-label jobs) sits in "Waiting to run" indefinitely. Tailscale shows the runner host offline:

  • kai-desktop-tower-wsl - offline, last seen 6 days ago
  • kai-desktop-tower - offline, last seen 6 days ago
  • kai-tower-3026 / kai-tower-3026-wsl - active

The forgejo runner StatefulSet is deliberately pinned to kai-desktop-tower-wsl (per docs/k3s-deploy-notes.md, to keep DinD bridge churn off kai-server). With the tower replaced by kai-tower-3026, the pin is stale: runner pods cannot schedule and the queue starves.

Fix path:

  1. Join kai-tower-3026-wsl to the k3s cluster (if not already) and repin the runner StatefulSet's nodeSelector to it.
  2. Clear any stale local-path PVCs from the old node so the StatefulSet can recreate them (known trap in the deploy notes triage tree).
  3. Update docs/k3s-deploy-notes.md mentions of kai-desktop-tower-wsl.

Found while shipping the first eco-app rollout (2026-06-12); the rollout proceeded by building the image on kai-server directly.

eco-app CI run #3 (and likely all docker-label jobs) sits in "Waiting to run" indefinitely. Tailscale shows the runner host offline: * `kai-desktop-tower-wsl` - offline, last seen 6 days ago * `kai-desktop-tower` - offline, last seen 6 days ago * `kai-tower-3026` / `kai-tower-3026-wsl` - active The forgejo runner StatefulSet is deliberately pinned to `kai-desktop-tower-wsl` (per docs/k3s-deploy-notes.md, to keep DinD bridge churn off kai-server). With the tower replaced by kai-tower-3026, the pin is stale: runner pods cannot schedule and the queue starves. Fix path: 1. Join `kai-tower-3026-wsl` to the k3s cluster (if not already) and repin the runner StatefulSet's nodeSelector to it. 2. Clear any stale `local-path` PVCs from the old node so the StatefulSet can recreate them (known trap in the deploy notes triage tree). 3. Update docs/k3s-deploy-notes.md mentions of kai-desktop-tower-wsl. Found while shipping the first eco-app rollout (2026-06-12); the rollout proceeded by building the image on kai-server directly.
coilysiren commented 2026-06-12 11:21:46 +00:00
Author
Owner
Copy link

Update from the eco-app rollout (same day): the queue drained itself before any repin. kubectl get nodes during the rollout showed the cluster is now single-node (kai-server only) - the tower node is not in the cluster at all - and forgejo-runner-0 / forgejo-runner-1 were Running on kai-server, ~7 minutes old with 3 restarts, right before eco-app CI run #3 went green and published the image.

So the original framing (stale nodeSelector pin to kai-desktop-tower-wsl) does not match the live cluster: either the pin was already removed, or the StatefulSet tolerates kai-server now. What remains of this issue:

  1. eco-app run #3 sat queued ~25+ minutes before the runner pods (re)started - find what restarted them and whether pickup latency like that will recur.
  2. The runners on kai-server is exactly what docs/k3s-deploy-notes.md warns about (DinD bridge churn racing kube-proxy on kai-server). Either re-home them to kai-tower-3026-wsl by joining it to the cluster, or consciously accept the kai-server placement and update the notes.
  3. Update the kai-desktop-tower-wsl references in docs/k3s-deploy-notes.md to the 3026 tower either way.
Update from the eco-app rollout (same day): the queue drained itself before any repin. `kubectl get nodes` during the rollout showed the cluster is now **single-node (kai-server only)** - the tower node is not in the cluster at all - and `forgejo-runner-0` / `forgejo-runner-1` were Running **on kai-server**, ~7 minutes old with 3 restarts, right before eco-app CI run #3 went green and published the image. So the original framing (stale nodeSelector pin to kai-desktop-tower-wsl) does not match the live cluster: either the pin was already removed, or the StatefulSet tolerates kai-server now. What remains of this issue: 1. eco-app run #3 sat queued ~25+ minutes before the runner pods (re)started - find what restarted them and whether pickup latency like that will recur. 2. The runners on kai-server is exactly what docs/k3s-deploy-notes.md warns about (DinD bridge churn racing kube-proxy on kai-server). Either re-home them to `kai-tower-3026-wsl` by joining it to the cluster, or consciously accept the kai-server placement and update the notes. 3. Update the kai-desktop-tower-wsl references in docs/k3s-deploy-notes.md to the 3026 tower either way.
coilyco-ops added
P3
 and removed
P2
 labels 2026-07-10 09:00:12 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#312
No description provided.