tap-writer runner CrashLoopBackOff (exit 99) - missing workingDir: /data #249

New issue

Closed

opened 2026-06-06 22:40:13 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-06 22:40:13 +00:00

Owner

The dedicated tap-writer Forgejo runner (deploy/forgejo-runner-tap-writer.yml, added in #194/#195) never comes up. Its runner container is in CrashLoopBackOff, exit code 99, ~202 restarts over 17h.

Observed

$ coily ops kubectl get pods -n forgejo
forgejo-runner-tap-writer-0   0/1   CrashLoopBackOff   202 (...)   17h

describe shows the register init container completed (exit 0) - so the runner is registered in Forgejo (admin runners API: id=7, name forgejo-runner-tap-writer-0, label tap-writer). Only the daemon container fails, instantly (Started == Finished), with exit 99.

Pod logs are unreadable from the Mac (coily ops kubectl logs returns a node-level Unable to open log: Permission denied), so the cause is from the manifest, not the log.

Root cause

The runner container has no workingDir, and its shell command never cd /data before exec forgejo-runner daemon. The register init container does cd /data before writing /data/.runner. So the daemon launches from /, cannot find /data/.runner, and exits 99.

The sibling deploy/forgejo-runner.yml runner container sets workingDir: /data and stays up - that is the missing line.

Fix

Add workingDir: /data to the tap-writer runner container, matching the sibling. PR forthcoming.

Apply (on kai-server, after the fix lands)

sudo k3s kubectl apply -f deploy/forgejo-runner-tap-writer.yml
sudo k3s kubectl delete pod -n forgejo forgejo-runner-tap-writer-0

Then confirm forgejo-runner-tap-writer-0 reaches 1/1 Running.

Context

Surfaced while checking the tap-writer runner for ward#61 (which moved the homebrew tap bump to runs-on: tap-writer). ward#61 is correct but inert until this runner is healthy. Separate from the regular-runner outage tracked in #215. Diagnosis posted to o2r channel FQT7.

The dedicated tap-writer Forgejo runner (`deploy/forgejo-runner-tap-writer.yml`, added in #194/#195) never comes up. Its `runner` container is in `CrashLoopBackOff`, exit code **99**, ~202 restarts over 17h. ## Observed ``` $ coily ops kubectl get pods -n forgejo forgejo-runner-tap-writer-0 0/1 CrashLoopBackOff 202 (...) 17h ``` `describe` shows the `register` init container completed (exit 0) - so the runner **is** registered in Forgejo (admin runners API: `id=7`, name `forgejo-runner-tap-writer-0`, label `tap-writer`). Only the daemon container fails, instantly (Started == Finished), with exit 99. Pod logs are unreadable from the Mac (`coily ops kubectl logs` returns a node-level `Unable to open log: Permission denied`), so the cause is from the manifest, not the log. ## Root cause The `runner` container has no `workingDir`, and its shell command never `cd /data` before `exec forgejo-runner daemon`. The `register` init container **does** `cd /data` before writing `/data/.runner`. So the daemon launches from `/`, cannot find `/data/.runner`, and exits 99. The sibling `deploy/forgejo-runner.yml` runner container sets `workingDir: /data` and stays up - that is the missing line. ## Fix Add `workingDir: /data` to the tap-writer runner container, matching the sibling. PR forthcoming. ## Apply (on kai-server, after the fix lands) ``` sudo k3s kubectl apply -f deploy/forgejo-runner-tap-writer.yml sudo k3s kubectl delete pod -n forgejo forgejo-runner-tap-writer-0 ``` Then confirm `forgejo-runner-tap-writer-0` reaches `1/1 Running`. ## Context Surfaced while checking the tap-writer runner for ward#61 (which moved the homebrew tap bump to `runs-on: tap-writer`). ward#61 is correct but inert until this runner is healthy. Separate from the regular-runner outage tracked in #215. Diagnosis posted to o2r channel FQT7.

coilysiren referenced this issue from a commit

2026-06-06 22:41:49 +00:00

fix(forgejo-runner): set workingDir /data on tap-writer runner

coilysiren referenced this issue from a pull request that will close it,

2026-06-06 22:42:03 +00:00

fix(forgejo-runner): set workingDir /data on tap-writer runner #250

coilysiren referenced this issue

2026-06-06 22:42:23 +00:00

forgejo-actions runners wedged in Terminating - no CI on main pushes #215

coilysiren referenced this issue

2026-06-06 23:06:30 +00:00

kai-server SSH sessions hang / drop - can't reliably get on-box #251