fix: settle and announce the jobs a restart drops #911

Merged

coilysiren merged 1 commit from issue-878-settle-dropped-jobs into main

2026-08-17 17:53:33 +00:00

coilyco-ops commented

2026-08-17 17:39:56 +00:00

Member

Settles the jobs a restart drops, and announces every job restart recovery settles.

What changed

SettleDroppedJobs and DroppedJobIDs move the queued group to failed under dropped by a restart. RecoverStrandedJobs keeps its running and cancelling scope and both now share one settleInterrupted helper.
Agent.recoverJobs runs two passes and notifies each settled job. Correcting the record tells nobody, because a Discord requester never reads one, and the stranded jobs were being settled silently for the same reason.

The decisions the issue asked for

No JobStore query by state. Both durable stores already expose All() []Job and recoverJobs already asserts for it, so a fourth interface method PostgresJobStore would have to answer buys nothing. Option 1 stays open behind this.
Settle rather than requeue. Requeueing is the larger change, it needs Effects to become load-bearing, and #430 may replace it outright. Settling is the honest record and does not block it.

Tests

jobrestart_test.go. A dropped job settles and its requester is told, a stranded job's requester is told, a terminal job is neither rewritten nor announced, and the two groups list apart.

TestARestartDropsWhatWasQueued from #824 is untouched and still passes: it pins Start requeueing nothing, and this lives in recoverJobs.

Mutation: removing the dropped pass fails TestARestartSettlesAndAnnouncesWhatWasQueued. Verified.

just gate PASS.

closes #878

Settles the jobs a restart drops, and announces every job restart recovery settles. ## What changed * `SettleDroppedJobs` and `DroppedJobIDs` move the `queued` group to `failed` under `dropped by a restart`. `RecoverStrandedJobs` keeps its `running` and `cancelling` scope and both now share one `settleInterrupted` helper. * `Agent.recoverJobs` runs two passes and notifies each settled job. Correcting the record tells nobody, because a Discord requester never reads one, and the stranded jobs were being settled silently for the same reason. ## The decisions the issue asked for * **No `JobStore` query by state.** Both durable stores already expose `All() []Job` and `recoverJobs` already asserts for it, so a fourth interface method `PostgresJobStore` would have to answer buys nothing. Option 1 stays open behind this. * **Settle rather than requeue.** Requeueing is the larger change, it needs `Effects` to become load-bearing, and #430 may replace it outright. Settling is the honest record and does not block it. ## Tests `jobrestart_test.go`. A dropped job settles and its requester is told, a stranded job's requester is told, a terminal job is neither rewritten nor announced, and the two groups list apart. `TestARestartDropsWhatWasQueued` from #824 is untouched and still passes: it pins `Start` requeueing nothing, and this lives in `recoverJobs`. Mutation: removing the dropped pass fails `TestARestartSettlesAndAnnouncesWhatWasQueued`. Verified. `just gate` PASS. closes #878

coilyco-ops added 1 commit

2026-08-17 17:39:56 +00:00

fix: settle and announce the jobs a restart drops

ci / image-build (pull_request) Successful in 40s

Details

ci / test (pull_request) Successful in 1m3s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

54127c6cd3

A job that was accepted, durable, and queued when the process stopped was
never picked up again and its requester was told nothing. Start builds an
empty queue, enqueue has one caller in Submit, and nothing reads the store,
so the accurate queued record was a permanently pending one.

Recovery now settles that group to failed under "dropped by a restart", and
announces every job it settles. The stranded running and cancelling jobs
RecoverStrandedJobs already handled were settled silently, which is the same
defect one state over, so they are announced too.

No JobStore query by state: both durable stores already expose All(), which
is what recoverJobs asserts for. Requeueing stays open behind this.

closes #878

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-17 17:39:56 +00:00

coilyco-ops referenced this pull request

2026-08-17 17:40:12 +00:00

A queued job is dropped without notice on restart, because nothing requeues it and the store cannot list by state #878