Make dispatch reservations issue-backed for multi-node ward #1205

New issue

Closed

opened 2026-07-13 22:21:37 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-13 22:21:37 +00:00

Member

Ward dispatch is moving toward a multi-node model, so dispatch state must not be canonically owned by one host's Docker daemon or ~/.ward state.

Problem:

issue reservation and dispatch state currently split across local reservation sentinels, Docker container labels, dispatch logs, and issue comments
backlog guards and reservation checks rely on host-local state, which fails when multiple dispatch nodes can see the same issue
clearing issue comments does not fully clear stale local host state, which has already caused hard director stops and stale capacity blocks

Required direction:

make the issue thread the canonical reservation / dispatch-state source
local state may exist only as a cache, similar in spirit to gitcache, and must be rebuildable from issue state
backlog guard decisions must come from canonical issue-backed state, not local Docker state
reservation acquire/release must write the issue-backed state first, then update any local cache
stale local cache must never block dispatch when issue state says no active reservation exists
multi-node dispatch must be able to converge from issue state after a process restart or host change

Prime targets:

agent_reserve.go local sentinels and launch locks
backlog guard paths that look for running Docker containers by ward.repo / ward.issue
dispatch list / capacity paths that blend local reservations, dispatch logs, and Docker labels
cleanup/reap behavior that currently leaves stale host-local records capable of blocking new work

Acceptance criteria:

a stale Docker container or stale ~/.ward reservation file cannot reserve an issue by itself
reservations are visible and clearable through issue state
local caches are documented as caches, not authority
unit tests cover at least: no issue reservation means no block despite stale local cache, active issue reservation blocks, released issue reservation unblocks, and two-node cache divergence converges on issue state
docs or code comments identify the canonical dispatch-state owner

This is architecture work, not just a TTL mitigation.

Ward dispatch is moving toward a multi-node model, so dispatch state must not be canonically owned by one host's Docker daemon or ~/.ward state. Problem: - issue reservation and dispatch state currently split across local reservation sentinels, Docker container labels, dispatch logs, and issue comments - backlog guards and reservation checks rely on host-local state, which fails when multiple dispatch nodes can see the same issue - clearing issue comments does not fully clear stale local host state, which has already caused hard director stops and stale capacity blocks Required direction: - make the issue thread the canonical reservation / dispatch-state source - local state may exist only as a cache, similar in spirit to gitcache, and must be rebuildable from issue state - backlog guard decisions must come from canonical issue-backed state, not local Docker state - reservation acquire/release must write the issue-backed state first, then update any local cache - stale local cache must never block dispatch when issue state says no active reservation exists - multi-node dispatch must be able to converge from issue state after a process restart or host change Prime targets: - agent_reserve.go local sentinels and launch locks - backlog guard paths that look for running Docker containers by ward.repo / ward.issue - dispatch list / capacity paths that blend local reservations, dispatch logs, and Docker labels - cleanup/reap behavior that currently leaves stale host-local records capable of blocking new work Acceptance criteria: - a stale Docker container or stale ~/.ward reservation file cannot reserve an issue by itself - reservations are visible and clearable through issue state - local caches are documented as caches, not authority - unit tests cover at least: no issue reservation means no block despite stale local cache, active issue reservation blocks, released issue reservation unblocks, and two-node cache divergence converges on issue state - docs or code comments identify the canonical dispatch-state owner This is architecture work, not just a TTL mitigation.

coilyco-ops referenced this issue

2026-07-13 22:21:45 +00:00

Remove ward docker exec #1206

coilyco-ops referenced this issue

2026-07-13 22:25:43 +00:00

Do not leave transient reservation dispatch comments on issues #1207

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-13 22:56:34 +00:00

Make dispatch reservations issue-backed for multi-node ward #1212

coilyco-ops commented

2026-07-13 22:59:18 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-request-and-merge; review summary: skipped (temporary ward default pending brokered QA)
retrospective: steady. The authority swap stayed local to the reservation paths.
confidence: high
surprises: Forgejo CI took a few minutes to settle.
follow-ups: none

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped (temporary ward default pending brokered QA) retrospective: steady. The authority swap stayed local to the reservation paths. confidence: high surprises: Forgejo CI took a few minutes to settle. follow-ups: none </details>