release run 1067 cancelled on main with no superseding push - find the canceller #477

New issue

Open

opened 2026-07-12 06:20:25 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-12 06:20:25 +00:00

Owner

Release run 1067 (push-triggered by the #474 merge, 2026-07-12) went cancelled mid-publish with NO newer main push. The manual bump dispatch that evening (run 1063) had the same pattern the other way: auto run 1062 cancelled, manual dispatch succeeded.

The two-stage flow (aos#475 / ward#1117, concurrency queue with cancel-in-progress: false) should eliminate the overlap-and-cancel class, but an unexplained cancellation with no superseding trigger is a different animal - possibly Forgejo auto-cancellation keyed too broadly (cross-branch?), runner eviction, or the cleanup cron. Correlate run 1067 timestamps with the Forgejo instance logs. If it stays unexplained after two-stage lands, close as superseded.

Release run 1067 (push-triggered by the #474 merge, 2026-07-12) went `cancelled` mid-publish with NO newer main push. The manual bump dispatch that evening (run 1063) had the same pattern the other way: auto run 1062 cancelled, manual dispatch succeeded. The two-stage flow (aos#475 / ward#1117, concurrency queue with cancel-in-progress: false) should eliminate the overlap-and-cancel class, but an unexplained cancellation with no superseding trigger is a different animal - possibly Forgejo auto-cancellation keyed too broadly (cross-branch?), runner eviction, or the cleanup cron. Correlate run 1067 timestamps with the Forgejo instance logs. If it stays unexplained after two-stage lands, close as superseded.

coilysiren commented

2026-07-12 06:45:35 +00:00

Author

Owner

Solved by observation: run 1073 (release.yml on the aos#478 merge) was cancelled the moment the aos#479 merge pushed - Forgejo''s default per-workflow-per-branch auto-cancellation supersedes the in-flight run on every new push. 1067 died the same way. This is precisely the overlap-and-cancel class the two-stage flow (aos#475, ward#1117) removes: release.yml moves to the release branch under concurrency: cancel-in-progress: false, and promotion collapses push bursts before they reach the release pipeline. Closing once aos#475 lands and a queued release survives a main-push burst.

Solved by observation: run 1073 (release.yml on the aos#478 merge) was cancelled the moment the aos#479 merge pushed - Forgejo''s default per-workflow-per-branch auto-cancellation supersedes the in-flight run on every new push. 1067 died the same way. This is precisely the overlap-and-cancel class the two-stage flow (aos#475, ward#1117) removes: release.yml moves to the `release` branch under `concurrency: cancel-in-progress: false`, and promotion collapses push bursts before they reach the release pipeline. Closing once aos#475 lands and a queued release survives a main-push burst.

coilysiren referenced this issue

2026-07-12 06:46:14 +00:00

publish-dev-base red on main: stale WARD_VERSION, swagger-lock vs ward doctor, dep-bump gate deadlock #452

coilysiren commented

2026-07-12 07:54:11 +00:00

Author

Owner

Second no-push cancellation: run 1082 died 16 min into publish-dev-base (created 07:10, cancelled 07:27, no subsequent main push). Root cause found on the cluster: runner/dind container restarts under node churn - see infrastructure#547 (comfyui crashloop at 719 restarts on the same node). The push-burst auto-cancel explanation covers 1073 only; the runner-restart class covers 1067/1082.