director: heartbeat livelocks in a permanent DISPATCH: none hold on stale infra-failure history (no live forge-health probe) #528

New issue

Closed

opened 2026-07-02 09:21:23 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-02 09:21:23 +00:00

Owner

Symptom

The ward agent director heartbeat can enter a permanent hold: it keeps
choosing DISPATCH: none tick after tick and never drains the backlog, even
after the condition that first justified holding has cleared.

Observed live: four consecutive engineer dispatches (#470, #481, #482, #441)
all failed at the same pre-flight step, forgejo: get issue, with transient
502 bad gateway from Forgejo. The director correctly read that as "the
resolve pipeline looks broken, hold." Forgejo then recovered - all four
ward ops forgejo issue get ... --output json calls return 200 now - but the
director stayed held across every subsequent tick with 29 issues queued and 10
free slots.

Root cause: a livelock on stale outcome history

directorDecidePrompt (cmd/ward/agent_director_heartbeat.go:300) hands the
LLM three inputs - QUEUED, IN FLIGHT, RECENT OUTCOMES - and invites it to hold
"if recent runs are failing in a way that suggests waiting." It passes no
live forge-health signal. That creates a self-perpetuating loop:

A transient upstream blip (502) parks a few dispatches in RECENT OUTCOMES.
The LLM sees the failing streak and holds.
Holding dispatches nothing, so no new outcome is produced.
The old failures stay the newest signal in RECENT OUTCOMES.
Next tick reads the same failures and holds again - forever.

The forge is never re-probed, so recovery is invisible to the decision. The
mechanical layer already does the right thing on the next attempt (v0.314.0
defers dispatch-time infra failures instead of parking them failed; v0.315.0
self-heals legacy dispatch-error strandings, ward#527) - but the LLM judgment
layer on top never makes an attempt, so those fixes never fire.

Want

Break the livelock. The decision must not hold indefinitely on a failure
signal that may already be stale. Options for the implementer to weigh:

Live health probe (preferred). Before building the prompt, run one cheap
forge liveness check (a single issue get on a known issue, or the Forgejo
version endpoint) and pass "forge health: ok | degraded" into the prompt.
When health is ok, a past infra-failure streak is not grounds to hold.
Age out / classify the infra-failure signal. Distinguish infra failures
(502, dispatch-error) from substantive engineer failures. Do not let an
infra-failure streak justify a hold for more than a bounded number of ticks -
after the bound, force one probe-dispatch of the top candidate regardless,
since the defer/self-heal path makes a retry cheap and safe.
Livelock guard. If the only reason to hold is infra failures AND nothing
has dispatched for K ticks, dispatch one candidate to refresh the signal.

Prefer (1) or a combination - the invariant is that a recovered forge always
becomes visible to the next hold/dispatch decision without needing a human to
step in.

Acceptance

After a transient forge outage recovers, the director resumes dispatching on
its own within a bounded number of ticks, with no human nudge.
The hold decision keys off a live health signal (or a bounded infra-failure
age-out), not solely the historical RECENT OUTCOMES window.
A unit test drives the livelock scenario: seed a failing infra-outcome
streak + healthy forge and assert the director dispatches rather than holding
indefinitely. Extend agent_director_heartbeat_test.go.

Notes

Discovered from the director's read-only surface session while the heartbeat
was stuck holding on the #470/#481/#482/#441 502 streak.
Related but distinct: #497 (single dispatch's transient resolve failure
surfaces as bare exit status 3, no cause, no retry). This issue is about the
aggregate hold decision livelocking, not one dispatch's error text.

## Symptom The `ward agent director` heartbeat can enter a **permanent hold**: it keeps choosing `DISPATCH: none` tick after tick and never drains the backlog, even after the condition that first justified holding has cleared. Observed live: four consecutive engineer dispatches (#470, #481, #482, #441) all failed at the same pre-flight step, `forgejo: get issue`, with transient `502 bad gateway` from Forgejo. The director correctly read that as "the resolve pipeline looks broken, hold." Forgejo then recovered - all four `ward ops forgejo issue get ... --output json` calls return 200 now - but the director stayed held across every subsequent tick with 29 issues queued and 10 free slots. ## Root cause: a livelock on stale outcome history `directorDecidePrompt` (`cmd/ward/agent_director_heartbeat.go:300`) hands the LLM three inputs - QUEUED, IN FLIGHT, RECENT OUTCOMES - and invites it to hold "if recent runs are failing in a way that suggests waiting." It passes **no live forge-health signal**. That creates a self-perpetuating loop: 1. A transient upstream blip (502) parks a few dispatches in RECENT OUTCOMES. 2. The LLM sees the failing streak and holds. 3. Holding dispatches nothing, so no new outcome is produced. 4. The old failures stay the newest signal in RECENT OUTCOMES. 5. Next tick reads the same failures and holds again - forever. The forge is never re-probed, so recovery is invisible to the decision. The mechanical layer already does the right thing on the next attempt (v0.314.0 defers dispatch-time infra failures instead of parking them failed; v0.315.0 self-heals legacy dispatch-error strandings, ward#527) - but the LLM judgment layer on top never *makes* an attempt, so those fixes never fire. ## Want Break the livelock. The decision must not hold indefinitely on a failure signal that may already be stale. Options for the implementer to weigh: 1. **Live health probe (preferred).** Before building the prompt, run one cheap forge liveness check (a single `issue get` on a known issue, or the Forgejo version endpoint) and pass "forge health: ok | degraded" into the prompt. When health is ok, a past infra-failure streak is not grounds to hold. 2. **Age out / classify the infra-failure signal.** Distinguish infra failures (502, dispatch-error) from substantive engineer failures. Do not let an infra-failure streak justify a hold for more than a bounded number of ticks - after the bound, force one probe-dispatch of the top candidate regardless, since the defer/self-heal path makes a retry cheap and safe. 3. **Livelock guard.** If the only reason to hold is infra failures AND nothing has dispatched for K ticks, dispatch one candidate to refresh the signal. Prefer (1) or a combination - the invariant is that a recovered forge always becomes visible to the next hold/dispatch decision without needing a human to step in. ## Acceptance - After a transient forge outage recovers, the director resumes dispatching on its own within a bounded number of ticks, with no human nudge. - The hold decision keys off a live health signal (or a bounded infra-failure age-out), not solely the historical RECENT OUTCOMES window. - A unit test drives the livelock scenario: seed a failing infra-outcome streak + healthy forge and assert the director dispatches rather than holding indefinitely. Extend `agent_director_heartbeat_test.go`. ## Notes - Discovered from the director's read-only surface session while the heartbeat was stuck holding on the #470/#481/#482/#441 502 streak. - Related but distinct: #497 (single dispatch's transient resolve failure surfaces as bare `exit status 3`, no cause, no retry). This issue is about the aggregate hold decision livelocking, not one dispatch's error text.

coilyco-ops commented

2026-07-02 09:21:53 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-528 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T09:21:51Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-528` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T09:21:51Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-02 09:37:08 +00:00

fix(director): break the stale-infra hold livelock with a live forge-health probe

coilysiren closed this issue

2026-07-02 09:37:08 +00:00

coilysiren commented

2026-07-02 09:37:31 +00:00

Author

Owner

WARD-OUTCOME: done - live forge-health probe + deterministic livelock guard, landed on main (b48573e), closes #528.

The shape landed close to option 1 + option 3 combined, which felt like the honest answer: the prompt now carries a FORGE HEALTH line from a per-tick probe (a real issue get on the top candidate - the exact read that 502s), so the LLM can recover on its own, but I didn't want the invariant to depend on the LLM cooperating. So directorLivelockGuard is the deterministic backstop: a held decision on a live-ok forge with a pure-infra failing streak force-dispatches the single top candidate. The ward#524 defer path already made that retry cheap, which is what made a "just dispatch one and see" guard safe rather than reckless.

What fought back: nothing in the logic (the directorBackend seam + fakeDirector made the livelock scenario a clean unit test), but the pre-commit doc/comment caps did - the 2-line/90-char comment rule and the 4000-char cap on agent-director.md took a few rounds of trimming, and the durable prose ended up in agent-director-dispatch.md where it belongs anyway.

Confidence: high on the guard and classifier (well covered by tests). One rough edge worth a note: the probe is Forgejo-only (reuses hostForgejoClient, matching the existing director poll), so a GitHub-targeted director gets unknown health and never fires the guard - safe, but it means the livelock fix doesn't cover a GitHub director. The infra/substantive text classifier is also heuristic (regex on outcome text); I deliberately excluded exited-no-outcome from "infra" so the guard never burns real runs re-dispatching a genuinely-crashing engineer. Both are reasonable follow-ups if the director grows a GitHub backend.

WARD-OUTCOME: done - live forge-health probe + deterministic livelock guard, landed on main (b48573e), closes #528. The shape landed close to option 1 + option 3 combined, which felt like the honest answer: the prompt now carries a `FORGE HEALTH` line from a per-tick probe (a real `issue get` on the top candidate - the exact read that 502s), so the LLM *can* recover on its own, but I didn't want the invariant to depend on the LLM cooperating. So `directorLivelockGuard` is the deterministic backstop: a held decision on a live-ok forge with a pure-infra failing streak force-dispatches the single top candidate. The ward#524 defer path already made that retry cheap, which is what made a "just dispatch one and see" guard safe rather than reckless. What fought back: nothing in the logic (the `directorBackend` seam + `fakeDirector` made the livelock scenario a clean unit test), but the pre-commit doc/comment caps did - the 2-line/90-char comment rule and the 4000-char cap on `agent-director.md` took a few rounds of trimming, and the durable prose ended up in `agent-director-dispatch.md` where it belongs anyway. Confidence: high on the guard and classifier (well covered by tests). One rough edge worth a note: the probe is Forgejo-only (reuses `hostForgejoClient`, matching the existing director poll), so a GitHub-targeted director gets `unknown` health and never fires the guard - safe, but it means the livelock fix doesn't cover a GitHub director. The infra/substantive text classifier is also heuristic (regex on outcome text); I deliberately excluded `exited-no-outcome` from "infra" so the guard never burns real runs re-dispatching a genuinely-crashing engineer. Both are reasonable follow-ups if the director grows a GitHub backend.