fix(forgejo-runner): set workingDir /data on tap-writer runner #250

Closed

coilysiren wants to merge 0 commits from fix/tap-writer-workingdir into main

coilysiren commented

2026-06-06 22:42:03 +00:00

Owner

Add workingDir: /data to the tap-writer runner container so the daemon finds its registration file.

Problem

forgejo-runner-tap-writer-0 has been in CrashLoopBackOff since #195 was applied - runner container exit code 99, ~202 restarts over 17h. The register init container completes fine (the runner shows registered in Forgejo, id=7, label tap-writer), but the daemon container exits instantly.

Root cause

The runner container had no workingDir, and its shell command never cd /data before exec forgejo-runner daemon. The init container does cd /data before writing /data/.runner. So the daemon launched from /, could not find /data/.runner, and exited 99. The sibling deploy/forgejo-runner.yml sets workingDir: /data and stays up - this restores parity.

(Pod logs are unreadable from the Mac - coily ops kubectl logs hits a node-level Unable to open log: Permission denied - so the cause was confirmed by diffing against the working sibling manifest, not from the log. High confidence, but worth a glance at the daemon log once it's reapplied.)

Apply (on kai-server, after merge)

sudo k3s kubectl apply -f deploy/forgejo-runner-tap-writer.yml
sudo k3s kubectl delete pod -n forgejo forgejo-runner-tap-writer-0

Confirm it reaches 1/1 Running.

Context

Surfaced while checking the tap-writer runner for ward#61 (which moved the homebrew tap bump to runs-on: tap-writer). Separate fault from the regular-runner outage in #215. Diagnosis posted to o2r channel FQT7.

closes #249

🤖 Generated with Claude Code

Add `workingDir: /data` to the tap-writer runner container so the daemon finds its registration file. ## Problem `forgejo-runner-tap-writer-0` has been in `CrashLoopBackOff` since #195 was applied - runner container exit code **99**, ~202 restarts over 17h. The `register` init container completes fine (the runner shows registered in Forgejo, `id=7`, label `tap-writer`), but the daemon container exits instantly. ## Root cause The runner container had no `workingDir`, and its shell command never `cd /data` before `exec forgejo-runner daemon`. The init container does `cd /data` before writing `/data/.runner`. So the daemon launched from `/`, could not find `/data/.runner`, and exited 99. The sibling `deploy/forgejo-runner.yml` sets `workingDir: /data` and stays up - this restores parity. (Pod logs are unreadable from the Mac - `coily ops kubectl logs` hits a node-level `Unable to open log: Permission denied` - so the cause was confirmed by diffing against the working sibling manifest, not from the log. High confidence, but worth a glance at the daemon log once it's reapplied.) ## Apply (on kai-server, after merge) ``` sudo k3s kubectl apply -f deploy/forgejo-runner-tap-writer.yml sudo k3s kubectl delete pod -n forgejo forgejo-runner-tap-writer-0 ``` Confirm it reaches `1/1 Running`. ## Context Surfaced while checking the tap-writer runner for ward#61 (which moved the homebrew tap bump to `runs-on: tap-writer`). Separate fault from the regular-runner outage in #215. Diagnosis posted to o2r channel FQT7. closes #249 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilysiren added 1 commit

2026-06-06 22:42:03 +00:00

fix(forgejo-runner): set workingDir /data on tap-writer runner

CI / lint (pull_request) Has been cancelled

Details

TruffleHog / Scan for secrets (pull_request) Has been cancelled

Details

23289ee9c2

The tap-writer runner daemon container had no workingDir and never cd'd
into /data, so forgejo-runner daemon launched from / and could not find
the /data/.runner registration file the init container writes there. It
exited 99 on every start (CrashLoopBackOff, ~202 restarts/17h) while
still showing registered in Forgejo (the init container succeeds).

Add workingDir: /data to match the working sibling deploy/forgejo-runner.yml.

Surfaced while checking the runner for ward#61 (tap bump now targets
runs-on: tap-writer). Separate from the regular-runner outage in #215.

closes #249

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coilysiren referenced this pull request

2026-06-06 22:42:23 +00:00

forgejo-actions runners wedged in Terminating - no CI on main pushes #215

coilysiren referenced this pull request

2026-06-06 23:06:30 +00:00

kai-server SSH sessions hang / drop - can't reliably get on-box #251

coilysiren added 3 commits

2026-06-07 18:55:31 +00:00

feat(ansible): fail ansible-freshen on uncommitted/stash/unmerged + blocked pulls, report repos needing push f6f9afbaa4

The git-role sweep surfaced repos needing attention but never failed the
run, and stashes / unmerged local branches were flagged yet excluded from
action_required. Add a play-end freshness gate that fails on those plus
blocked (non-ff/rebase-conflict) pulls, and a separate informational
needs_push surface for repos that just need a push.

closes #252

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(k3s): worker-join script for ser8 cross-tailnet cluster join e34f7d2b11

Adds scripts/k3s-join-worker.sh to join a host (ser8 first) to kai-server's
k3s cluster as a worker over the tailscale0 flannel plane. Pins the agent to
the server k3s version and reads the node-token over ssh so it never lands in
chat or a checked-in file. Plain-worker-now, warm-standby-promotion-later.

closes #258

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): add inert lan-hosts role (LAN-pin kai-server, off by default)

CI / lint (pull_request) Has been cancelled

Details

TruffleHog / Scan for secrets (pull_request) Has been cancelled

Details

2076d523e6

Mac-side workaround for the flaky tailnet path to kai-server (HDBF / #251):
pin fleet hostnames to their LAN IPs in /etc/hosts via a managed blockinfile,
gated on being on the home /24 so a freshen off-LAN self-cleans. Wired into
freshen.yml but INERT - lan_hosts_enabled defaults false, writes nothing until
flipped, pending an impl decision (/etc/hosts vs ssh-config alias).

Refs #251

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coilysiren referenced this pull request

2026-06-12 06:33:56 +00:00

ssh-config alias kai-server -> kai-server.local on home LAN (supersedes inert lan-hosts /etc/hosts pin, #266) #294

coilysiren closed this pull request

2026-06-12 06:34:01 +00:00