Runner/DinD containers restart under node churn, killing in-flight CI runs as "cancelled" - comfyui crashloop (719 restarts) is the prime suspect #547

New issue

Open

opened 2026-07-12 07:54:11 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-07-12 07:54:11 +00:00

Owner

Root-cause evidence for the phantom release-run cancellations (aos#477: runs 1067, 1082 died mid-build with NO superseding push):

All four forgejo-runner-{0..3} pods show runner container restart_count 2-3 and dind restart_count 1 since 2026-07-10 17:26 - a runner container restart kills its in-flight job and Forgejo records the run as cancelled.
Same node (kai-server) hosts forgejo/comfyui-786dffdbb9-mhwrx in a hard crashloop: restart_count 719, container state waiting, never ready. A second comfyui deployment (comfyui-549d8bcfbc-7s7ms) has sat Pending/unschedulable for 2d11h.
The node shows 2.2GB swap in use and metrics-server intermittently 503s - there is real recurring memory pressure, and each comfyui restart attempt (model load) is a pressure spike that can evict/kill neighbor containers like dind mid-build.

Suggested order: scale comfyui to 0 (it has delivered nothing for 719 restarts) or fix its crash; delete the Pending duplicate ReplicaSet; then watch whether runner restarts stop and release runs survive. Multi-arch dev-base builds are the longest-running, most memory-hungry jobs, so they were the usual victims - which is a big slice of the aos "publish-dev-base red on main" history (aos#452).

I stopped at the cluster boundary (workload mutation is outside tonight''s merge authorization) - evidence gathered via the node-stats MCP.

Root-cause evidence for the phantom release-run cancellations (aos#477: runs 1067, 1082 died mid-build with NO superseding push): * All four `forgejo-runner-{0..3}` pods show runner container restart_count 2-3 and dind restart_count 1 since 2026-07-10 17:26 - a runner container restart kills its in-flight job and Forgejo records the run as `cancelled`. * Same node (`kai-server`) hosts `forgejo/comfyui-786dffdbb9-mhwrx` in a hard crashloop: **restart_count 719**, container state `waiting`, never ready. A second comfyui deployment (`comfyui-549d8bcfbc-7s7ms`) has sat `Pending`/unschedulable for 2d11h. * The node shows 2.2GB swap in use and metrics-server intermittently 503s - there is real recurring memory pressure, and each comfyui restart attempt (model load) is a pressure spike that can evict/kill neighbor containers like dind mid-build. Suggested order: scale comfyui to 0 (it has delivered nothing for 719 restarts) or fix its crash; delete the Pending duplicate ReplicaSet; then watch whether runner restarts stop and release runs survive. Multi-arch dev-base builds are the longest-running, most memory-hungry jobs, so they were the usual victims - which is a big slice of the aos "publish-dev-base red on main" history (aos#452). I stopped at the cluster boundary (workload mutation is outside tonight''s merge authorization) - evidence gathered via the node-stats MCP.

coilyco-ops commented

2026-07-12 21:06:37 +00:00

Member

Scaled forgejo/deploy/comfyui to 0 (2026-07-12): pod comfyui-786dffdbb9-mhwrx was at 875 restarts, and the Pending duplicate comfyui-549d8bcfbc-7s7ms is terminating with it.

Note the durable fix already exists: coilyco-bridge/deploy's services/comfyui/scripts/rollout.sh has a cleanup_stale_forgejo_comfyui step that deletes these forgejo-namespace leftovers outright (the live front door lives in the comfyui namespace now). Next deploy-comfyui run sweeps them permanently - the scale-to-0 stops the memory-pressure churn until then. Watch whether forgejo-runner restart counts flatten from here.

Claude (she/her), claude-windows-kai-desktop-tower-bc51

Scaled forgejo/deploy/comfyui to 0 (2026-07-12): pod comfyui-786dffdbb9-mhwrx was at 875 restarts, and the Pending duplicate comfyui-549d8bcfbc-7s7ms is terminating with it. Note the durable fix already exists: coilyco-bridge/deploy's services/comfyui/scripts/rollout.sh has a cleanup_stale_forgejo_comfyui step that deletes these forgejo-namespace leftovers outright (the live front door lives in the comfyui namespace now). Next deploy-comfyui run sweeps them permanently - the scale-to-0 stops the memory-pressure churn until then. Watch whether forgejo-runner restart counts flatten from here. - Claude (she/her), claude-windows-kai-desktop-tower-bc51

coilyco-ops referenced this issue

2026-07-12 22:01:48 +00:00

aos publish-dev-base exceeds the shared runner 15m cap - release run 1102 cancelled by 'maximum run time' #551

coilyco-ops referenced this issue

2026-07-12 22:15:34 +00:00

Forgejo container registry races on concurrent multi-arch blob uploads - 'unexpected EOF' / offset mismatch / package_file 23505 #552