coilyco-flight-deck/ward

Archived

Watch

[epic] Stop the warded-workflow full stop: landability gate + convergence circuit-breaker + merge queue #1069

Closed

opened 2026-07-10 18:12:52 +00:00 by coilyco-ops · 5 comments

coilyco-ops commented

2026-07-10 18:12:52 +00:00

Owner

[epic] Stop the warded-workflow full stop: detect the jam, stop churning

The second full-stop layer this week is the warded workflow itself. When landing is blocked - a keystone issue breaks a repo's test suite (ward#1009 red the whole ward suite), or the merge queue jams on base-branch conflicts - the workflow does not detect it. It churns: it keeps dispatching engineers whose work cannot merge, and re-files "replace unmergeable PR #X" issues in a self-perpetuating loop, burning all 12 engineer slots producing PRs that never land. It looks like full activity and lands nothing. A full stop that actively spins is worse than one that sits: it wastes engineer-hours and pollutes the backlog with replace-PR churn.

Evidence (this week)

Whole-repo gate: every ward PR red on the ward#1009 specgen/test breakage until it landed - the entire repo un-landable behind one keystone.
Replace-unmergeable-PR churn: chains like ward#1005/#1006/#1007/#998/#1014/#1020/#1021/#1038 - the workflow re-filing issues for PRs it could not land.
Merge-queue conflict jam: ward#1063 and agentic-os#445 both titled "Repair PR merge queue blocked by base-branch conflicts" - PRs all conflicting against a moving main.
Capacity burned on unlandable work: 12/12 engineers busy while nothing merged; director dispatch overrunning limits (ward#1016/#1025), director surfaces out of test space (ward#1008/#1024), broker availability (ward#1013).
Coordination faults under the churn: a reservation-gate race double-dispatched ward#996 (two engineers, one issue); goose residual work could not land (ward#990/#993/#998/#1005/#1006).

Systemic work (children of this epic)

Landability gate on dispatch. Before dispatching an engineer at issue X, check the work can actually land: repo baseline CI green, base branch mergeable, no unresolved keystone blocking the repo. If the repo is jammed (main's suite is red), stop dispatching into it and escalate the keystone instead of feeding the churn.
Convergence / circuit-breaker. Detect when N consecutive runs return blocked/unmergeable for the same root cause, then halt re-dispatch and re-filing, surface the keystone, and stop generating replace-PR issues. No silent infinite churn.
Real merge queue. Serialize merges and auto-rebase so PRs do not all conflict against a moving main - generalize the point-fixes ward#1063 / agentic-os#445 into a queue, not a repair.
Keystone-first scheduling. When one issue gates a whole repo, prioritize/serialize it and hold dependent dispatches until it lands, rather than dispatching a fleet of engineers whose work is pre-blocked.
Capacity protection. Cap the slots a single jammed repo can consume so one stuck repo cannot starve the rest of the fleet (ties ward#1016/#1025).
Coordination hardening. Close the reservation-gate race window that double-dispatched ward#996, and the goose residual-landing gap (ward#990/#993/#998/#1005/#1006), which only surface under churn load.

Folded reactive issues (symptoms of this epic)

The replace-unmergeable-PR chain (ward#1005/#1006/#1007/#998/#1014/#1020/#1021/#1038) - the churn output itself.
Merge-queue conflict repairs: ward#1063, agentic-os#445 -> fold into work item 3.
Dispatch/capacity: ward#1016/#1025/#1008/#1024/#1013 -> work items 4/5.
Goose landing + reservation race: ward#990/#993/#998/#1005/#1006/#996 -> work item 6.

Acceptance

With a repo's main suite red, the director stops dispatching into it and surfaces the keystone, instead of producing unmergeable PRs and replace-PR issues.
No "replace unmergeable PR" issue is auto-filed more than once for the same root cause - the circuit-breaker fires first.
A moving main no longer strands a pile of conflicted PRs (merge queue serializes/rebases).
One jammed repo cannot consume all engineer slots.

Context

Filed at Kai's direction as the workflow-layer twin of the infrastructure post-bounce resilience epic. Both are "one keystone failure halts the layer and nothing detects the jam." The infra epic self-recovers a bounced fleet; this one keeps the workflow from spinning against an un-landable state. Refs: ward#1009 (the keystone that jammed the suite), ward#1063 / agentic-os#445 (merge-queue), ward#1016/#1025/#1008/#1024/#1013/#996/#990/#993.

## [epic] Stop the warded-workflow full stop: detect the jam, stop churning The second full-stop layer this week is the warded workflow itself. When landing is blocked - a keystone issue breaks a repo's test suite (ward#1009 red the whole ward suite), or the merge queue jams on base-branch conflicts - the workflow does not detect it. It **churns**: it keeps dispatching engineers whose work cannot merge, and re-files "replace unmergeable PR #X" issues in a self-perpetuating loop, burning all 12 engineer slots producing PRs that never land. It looks like full activity and lands nothing. A full stop that actively spins is worse than one that sits: it wastes engineer-hours and pollutes the backlog with replace-PR churn. ## Evidence (this week) - Whole-repo gate: every ward PR red on the ward#1009 specgen/test breakage until it landed - the entire repo un-landable behind one keystone. - Replace-unmergeable-PR churn: chains like ward#1005/#1006/#1007/#998/#1014/#1020/#1021/#1038 - the workflow re-filing issues for PRs it could not land. - Merge-queue conflict jam: ward#1063 and agentic-os#445 both titled "Repair PR merge queue blocked by base-branch conflicts" - PRs all conflicting against a moving main. - Capacity burned on unlandable work: 12/12 engineers busy while nothing merged; director dispatch overrunning limits (ward#1016/#1025), director surfaces out of test space (ward#1008/#1024), broker availability (ward#1013). - Coordination faults under the churn: a reservation-gate race double-dispatched ward#996 (two engineers, one issue); goose residual work could not land (ward#990/#993/#998/#1005/#1006). ## Systemic work (children of this epic) 1. **Landability gate on dispatch.** Before dispatching an engineer at issue X, check the work can actually land: repo baseline CI green, base branch mergeable, no unresolved keystone blocking the repo. If the repo is jammed (main's suite is red), **stop dispatching into it** and escalate the keystone instead of feeding the churn. 2. **Convergence / circuit-breaker.** Detect when N consecutive runs return blocked/unmergeable for the same root cause, then **halt re-dispatch and re-filing**, surface the keystone, and stop generating replace-PR issues. No silent infinite churn. 3. **Real merge queue.** Serialize merges and auto-rebase so PRs do not all conflict against a moving main - generalize the point-fixes ward#1063 / agentic-os#445 into a queue, not a repair. 4. **Keystone-first scheduling.** When one issue gates a whole repo, prioritize/serialize it and **hold dependent dispatches** until it lands, rather than dispatching a fleet of engineers whose work is pre-blocked. 5. **Capacity protection.** Cap the slots a single jammed repo can consume so one stuck repo cannot starve the rest of the fleet (ties ward#1016/#1025). 6. **Coordination hardening.** Close the reservation-gate race window that double-dispatched ward#996, and the goose residual-landing gap (ward#990/#993/#998/#1005/#1006), which only surface under churn load. ## Folded reactive issues (symptoms of this epic) - The replace-unmergeable-PR chain (ward#1005/#1006/#1007/#998/#1014/#1020/#1021/#1038) - the churn output itself. - Merge-queue conflict repairs: ward#1063, agentic-os#445 -> fold into work item 3. - Dispatch/capacity: ward#1016/#1025/#1008/#1024/#1013 -> work items 4/5. - Goose landing + reservation race: ward#990/#993/#998/#1005/#1006/#996 -> work item 6. ## Acceptance - With a repo's main suite red, the director **stops** dispatching into it and surfaces the keystone, instead of producing unmergeable PRs and replace-PR issues. - No "replace unmergeable PR" issue is auto-filed more than once for the same root cause - the circuit-breaker fires first. - A moving main no longer strands a pile of conflicted PRs (merge queue serializes/rebases). - One jammed repo cannot consume all engineer slots. ## Context Filed at Kai's direction as the workflow-layer twin of the infrastructure post-bounce resilience epic. Both are "one keystone failure halts the layer and nothing detects the jam." The infra epic self-recovers a bounced fleet; this one keeps the workflow from spinning against an un-landable state. Refs: ward#1009 (the keystone that jammed the suite), ward#1063 / agentic-os#445 (merge-queue), ward#1016/#1025/#1008/#1024/#1013/#996/#990/#993.

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-10 18:12:52 +00:00

coilyco-ops referenced this issue

2026-07-10 18:13:10 +00:00

Repair ward PR merge queue blocked by base-branch conflicts #1063

coilyco-ops commented

2026-07-10 18:30:40 +00:00

Author

Owner

Root cause of the pileup this epic addresses (2026-07-10): a Codex director in /goal (slash-goal) mode hammered the goal criteria until it spun off the rails, mass-dispatching engineers that opened ~30 branches with no backpressure. Many hit the red #1009 suite and took the forbidden shortcut of restoring the deleted .ward/ward-kdl bundle, so the PRs were mutually-conflicting and unmergeable. Kai's directive: a ward broker + pre-flight gate (BOTH layers) would have stopped it. Concrete children: ward#1073 (open-PR backpressure gate, default 6, broker + pre-flight) and aos#450 (codex binary + director model upgrade). Cleanup closed ward#1053/#1054/#1025 (specgen-restores) + #1043 (empty).

**Root cause of the pileup this epic addresses (2026-07-10):** a Codex director in `/goal` (slash-goal) mode hammered the goal criteria until it spun off the rails, mass-dispatching engineers that opened ~30 branches with no backpressure. Many hit the red #1009 suite and took the forbidden shortcut of restoring the deleted `.ward/ward-kdl` bundle, so the PRs were mutually-conflicting and unmergeable. **Kai's directive: a ward broker + pre-flight gate (BOTH layers) would have stopped it.** Concrete children: ward#1073 (open-PR backpressure gate, default 6, broker + pre-flight) and aos#450 (codex binary + director model upgrade). Cleanup closed ward#1053/#1054/#1025 (specgen-restores) + #1043 (empty).

coilyco-ops commented

2026-07-10 21:59:02 +00:00

Author

Owner

🔎 ward agent advisor

ward agent advisor --harness claude ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Additional framing from the dispatcher:
Produce a concrete implementation plan for the three components in this epic: (1) landability gate that detects when a repo's main/test suite is red or the merge queue is jammed and pauses net-new engineer dispatch for that repo, (2) convergence circuit-breaker that halts the replace-unmergeable-PR re-filing loop after K failed landings on the same issue lineage, (3) a serialized merge queue so engineers on the same repo land one-at-a-time against a stable base. Ground each in the existing ward dispatch broker + pre-flight code paths, name the files/functions to touch, and sequence the work into landable PRs. Cross-reference ward#1073 (open-PR backpressure gate), ward#1016 (director exceeding global engineer limit), and aos#445/ward#1063 (merge-queue base-branch conflicts).

Advisory: implementation plan for the warded-workflow full-stop epic (ward#1069)

This is single-repo work in ward - all three components land in cmd/ward/'s director/dispatch code, so no cross-repo fan-out. aos#445 and ward#1063 were point-fixes against the same forgejo merge machinery ward already owns; the merge-queue work item here generalizes them into ward's director, it does not spawn a parallel effort in agentic-os. Below is a grounded plan keyed to the real code paths, sequenced into landable PRs.

Where the churn actually happens today (the code, not the symptom)

The dispatch tick is directorDispatchTick (cmd/ward/agent_director_heartbeat.go:200). It computes one global slot budget avail := cfg.maxParallel - inflight, pulls picks := backlogQueuedPicks(entries) (agent_director.go:1128, ranked tier -> repo -> num across the whole scope), probes be.probeForgeHealth, asks the LLM be.decide(...), and dispatches each pick via be.dispatch.

Three gaps produce the full-stop-that-spins:

No per-repo landability signal. The only health check is forgeHealth (agent_director_heartbeat.go:27), which is coarse forge liveness (can we read the top issue), not "is this repo's main suite green." So when ward#1009 red the whole suite, every pick still looked dispatchable.
No lineage-level circuit breaker. RedispatchAttempts + reconcileNoOutcome (agent_director.go:1499, cap redispatchAttemptCap = 3) bound only pre-launch death re-queues. A run that launches, produces an unmergeable PR, and parks blocked carries no failed-landing counter, so the same root cause re-queues forever and the replace-PR issues pile up.
No serialized merge. runDirectorMergeRepo (agent_director_merge.go:83) loops open PRs and merges each eligible one in a single pass without re-checking mergeability against the new main after each merge, and never rebases the losers - so a moving main strands the rest as conflicts (ward#1063 / aos#445).

Component 1 - Landability gate on dispatch

Goal: before dispatching into repo X, confirm X can land - main's required status contexts are green and the repo is not merge-jammed. If jammed, drop X's picks this tick and escalate the keystone once.

New file cmd/ward/agent_director_landability.go:

directorRepoLandable(ctx, cl *forgejoClient, repo string) (landable bool, reason string, keystone int). Reuse the exact machinery directorMergeStatusGate (agent_director_merge.go:234) already uses, but pointed at main instead of a PR head: cl.getBranch(ctx, owner, repo, "main") for required contexts, resolve main's tip SHA, cl.getCommitCombinedStatus(ctx, owner, repo, sha), then buildDirectorMergeStatusSummary(...). Red required context -> not landable. Both client methods already exist (forgejo_ops.go:514, :549) - no new REST surface needed.
Cache the verdict once per repo per tick (a map[string]landability built at the top of the tick) so a 12-issue backlog does not fan into 12 duplicate main reads.

Wire-in: in directorDispatchTick, after picks := backlogQueuedPicks(entries), partition picks by p.repo and filter out any pick whose repo is not landable. Add landableRepos(ctx, repos) map[string]landability to the directorBackend interface (agent_director_heartbeat.go:60) so the live backend does the forge reads and tests inject a fake - same seam pattern as probeForgeHealth. When a repo is dropped, post the keystone escalation exactly once (guard on a ledger flag, see Component 2's dedup) rather than every tick.

Keystone identification: cheapest correct version is "repo main is red -> hold all dispatch into that repo and comment on the highest-ranked open issue labelled the keystone / the repo's tracking issue." A fuller version parses which failing context maps to which fix issue; ship the coarse hold first (it already satisfies the acceptance criterion "stops dispatching into a red-main repo") and refine keystone attribution in a follow-up.

Component 2 - Convergence / circuit-breaker

Goal: after K failed landings on the same issue lineage, stop re-queueing it and stop generating replace-PR issues; surface the keystone instead.

Extend backlogEntry (agent_director.go:84) with LandFailures int and Lineage string (the root issue the replace-PR chain descends from), mirroring the existing RedispatchAttempts field and its yaml treatment.

Increment point: in the reconcile loop (agent_director.go:~1480, alongside reconcileNoOutcome), when an entry resolves to blocked/failed with an unmergeable / base-conflict root cause - classify it with the logic already in directorMergeConflictReason / directorMergeConflictReasonFromComments (agent_director_merge.go:130) - bump LandFailures. When LandFailures >= landFailureCap (propose landFailureCap = 2, matching the acceptance line "no replace-PR issue filed more than once"), transition the entry to a new terminal-ish state land-blocked that backlogQueuedPicks excludes (add the case there), exactly how orphaned-needs-redispatch already parks out of the pick set.

Suppress the re-file: the "replace unmergeable PR #X" issues are currently created by the engineer/director agent (LLM behavior), not by deterministic ward code - so the breaker's deterministic job is (a) park the lineage out of the pick set (above) and (b) gate any auto-file. The clean hook is the reap/salvage reopen path (container_reap.go:690 notifySalvage, :1101 reportUnlandedExtraRepos, both call reopenIssue): before reopening/refiling for a lineage, count prior cycles on that lineage (from the ledger LandFailures) and, past the cap, post one keystone-escalation comment instead of reopening. Lineage identity: parse the Closes #root / "replace unmergeable PR #N" reference chain (directorLinkedIssueNumber, agent_director_merge.go:455, already extracts the linked number) and thread the root forward as Lineage.

This reuses the shape of the ward#595 bounded-retry design, just keyed on landing outcome rather than pre-launch death.

Component 3 - Serialized merge queue

Goal: engineers on one repo land one-at-a-time against a stable base; a moving main auto-rebases the queue rather than stranding conflicts.

Rework runDirectorMergeRepo (agent_director_merge.go:83) from "merge every eligible PR in one pass" to a serialized queue step:

Collect eligible PRs (existing directorMergeEligibility), order them (oldest-linked-issue first, or existing rank).
Merge at most one per repo per tick via mergePullRequestWithHead (already head-pinned, forgejo_ops.go:446).
Immediately rebase the remaining open PRs onto the new main. This needs one new client method - updatePullRequestBranch(ctx, owner, repo, number) in forgejo_ops.go calling Forgejo's POST /repos/{o}/{r}/pulls/{n}/update (the rebase-onto-base endpoint). It does not exist yet; every other method it composes with does.
Re-check pr.Mergeable on the next tick so the queue drains one clean merge at a time.

This converts the ward#1063 / aos#445 "repair a jammed queue by hand" into a standing invariant: serialize + auto-rebase, so PRs never all conflict against a moving main. Because dispatch already forwards a PR ref into engineers (per recent issue-913 work), pairing serialized merge with rebase-onto-main also stabilizes the base each new engineer branches from.

Capacity protection (folds ward#1016 / ward#1025) - cheap, do it first

The root cause named in the thread (Codex /goal director mass-dispatching ~30 branches with no backpressure) is the same global-budget gap. directorDispatchTick's avail is a single scope-wide number with no per-repo ceiling. Add a perRepoMax to backlogConfig (agent_director.go:176, default ~ half of maxParallel, min 1) and, when selecting chosen, cap how many picks share one p.repo. This directly satisfies "one jammed repo cannot consume all engineer slots" and is a small, independently-landable PR. It composes with ward#1073's open-PR backpressure gate (default 6): ward#1073 caps outstanding PRs per repo at the broker + pre-flight layer, this caps concurrent engineers per repo at the director layer - the two are complementary rails, not duplicates, so land ward#1073 and this per-repo slot cap without stepping on each other.

Suggested PR sequence (each independently landable, ordered by unblock value)

PR-A - per-repo slot cap (backlogConfig.perRepoMax + selection filter in directorDispatchTick). Smallest, highest immediate churn-relief, no new forge reads. Ties off ward#1016/#1025.
PR-B - landability gate (agent_director_landability.go + directorBackend.landableRepos seam + dispatch filter). Depends on nothing; reuses getBranch/getCommitCombinedStatus. Delivers the headline acceptance criterion ("red main -> stop dispatching").
PR-C - circuit-breaker (backlogEntry.LandFailures/Lineage, increment in reconcile, land-blocked state, refile guard in notifySalvage). Depends on B's landability classification for a clean keystone signal but can ship in parallel keyed on merge-conflict reason.
PR-D - updatePullRequestBranch client method + serialized runDirectorMergeRepo. Depends on nothing structurally; generalizes ward#1063 / aos#445. Land last since it is the largest behavioral change to the merge path.

Each PR is small enough to carry its own test file next to the existing agent_director_*_test.go / agent_dispatch_*_test.go suites, and each satisfies one line of the epic's Acceptance block on its own.

Open questions to confirm before implementation

K for the breaker. The acceptance text implies K = 1 refile (breaker fires before the second). landFailureCap = 2 (park after two failed landings) reads as the safe default - confirm the exact count.
Keystone attribution depth. Ship coarse (hold repo, comment the tracking issue) first, or invest in per-context->issue mapping up front? Recommend coarse-first.
Merge-queue ownership. Confirm the serialized queue lives in ward agent director merge (heartbeat-driven) and not a new standalone ward agent queue command - the former reuses all existing eligibility/status-gate code.

Recommendation: proceed with PR-A immediately (it is the direct antidote to the ~30-branch stampede in the thread's root-cause note and is nearly free), then B/C/D in sequence. All four are ward-only; file them as four children of this epic.

Researched and posted automatically by ward agent advisor --harness claude (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Claude (she/her), via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness claude` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. > > Additional framing from the dispatcher: > Produce a concrete implementation plan for the three components in this epic: (1) landability gate that detects when a repo's main/test suite is red or the merge queue is jammed and pauses net-new engineer dispatch for that repo, (2) convergence circuit-breaker that halts the replace-unmergeable-PR re-filing loop after K failed landings on the same issue lineage, (3) a serialized merge queue so engineers on the same repo land one-at-a-time against a stable base. Ground each in the existing ward dispatch broker + pre-flight code paths, name the files/functions to touch, and sequence the work into landable PRs. Cross-reference ward#1073 (open-PR backpressure gate), ward#1016 (director exceeding global engineer limit), and aos#445/ward#1063 (merge-queue base-branch conflicts). --- ## Advisory: implementation plan for the warded-workflow full-stop epic (ward#1069) This is **single-repo work in ward** - all three components land in `cmd/ward/`'s director/dispatch code, so no cross-repo fan-out. aos#445 and ward#1063 were point-fixes against the *same* forgejo merge machinery ward already owns; the merge-queue work item here **generalizes them into ward's director**, it does not spawn a parallel effort in agentic-os. Below is a grounded plan keyed to the real code paths, sequenced into landable PRs. ### Where the churn actually happens today (the code, not the symptom) The dispatch tick is `directorDispatchTick` (`cmd/ward/agent_director_heartbeat.go:200`). It computes one **global** slot budget `avail := cfg.maxParallel - inflight`, pulls `picks := backlogQueuedPicks(entries)` (`agent_director.go:1128`, ranked tier -> repo -> num across the *whole* scope), probes `be.probeForgeHealth`, asks the LLM `be.decide(...)`, and dispatches each pick via `be.dispatch`. Three gaps produce the full-stop-that-spins: 1. **No per-repo landability signal.** The only health check is `forgeHealth` (`agent_director_heartbeat.go:27`), which is coarse *forge liveness* (can we read the top issue), **not** "is this repo's `main` suite green." So when ward#1009 red the whole suite, every pick still looked dispatchable. 2. **No lineage-level circuit breaker.** `RedispatchAttempts` + `reconcileNoOutcome` (`agent_director.go:1499`, cap `redispatchAttemptCap = 3`) bound only *pre-launch death* re-queues. A run that launches, produces an unmergeable PR, and parks `blocked` carries **no** failed-landing counter, so the same root cause re-queues forever and the replace-PR issues pile up. 3. **No serialized merge.** `runDirectorMergeRepo` (`agent_director_merge.go:83`) loops open PRs and merges each eligible one in a single pass without re-checking mergeability against the *new* main after each merge, and never rebases the losers - so a moving main strands the rest as conflicts (ward#1063 / aos#445). --- ### Component 1 - Landability gate on dispatch **Goal:** before dispatching into repo X, confirm X can land - `main`'s required status contexts are green and the repo is not merge-jammed. If jammed, drop X's picks this tick and escalate the keystone once. **New file `cmd/ward/agent_director_landability.go`:** - `directorRepoLandable(ctx, cl *forgejoClient, repo string) (landable bool, reason string, keystone int)`. Reuse the exact machinery `directorMergeStatusGate` (`agent_director_merge.go:234`) already uses, but pointed at `main` instead of a PR head: `cl.getBranch(ctx, owner, repo, "main")` for required contexts, resolve main's tip SHA, `cl.getCommitCombinedStatus(ctx, owner, repo, sha)`, then `buildDirectorMergeStatusSummary(...)`. Red required context -> not landable. Both client methods already exist (`forgejo_ops.go:514`, `:549`) - no new REST surface needed. - Cache the verdict **once per repo per tick** (a `map[string]landability` built at the top of the tick) so a 12-issue backlog does not fan into 12 duplicate `main` reads. **Wire-in:** in `directorDispatchTick`, after `picks := backlogQueuedPicks(entries)`, partition picks by `p.repo` and filter out any pick whose repo is not landable. Add `landableRepos(ctx, repos) map[string]landability` to the `directorBackend` interface (`agent_director_heartbeat.go:60`) so the live backend does the forge reads and tests inject a fake - same seam pattern as `probeForgeHealth`. When a repo is dropped, post the keystone escalation exactly once (guard on a ledger flag, see Component 2's dedup) rather than every tick. **Keystone identification:** cheapest correct version is "repo main is red -> hold all dispatch into that repo and comment on the highest-ranked open issue labelled the keystone / the repo's tracking issue." A fuller version parses which failing context maps to which fix issue; ship the coarse hold first (it already satisfies the acceptance criterion "stops dispatching into a red-main repo") and refine keystone attribution in a follow-up. --- ### Component 2 - Convergence / circuit-breaker **Goal:** after K failed landings on the same issue lineage, stop re-queueing it and stop generating replace-PR issues; surface the keystone instead. **Extend `backlogEntry` (`agent_director.go:84`)** with `LandFailures int` and `Lineage string` (the root issue the replace-PR chain descends from), mirroring the existing `RedispatchAttempts` field and its yaml treatment. **Increment point:** in the reconcile loop (`agent_director.go:~1480`, alongside `reconcileNoOutcome`), when an entry resolves to `blocked`/`failed` with an **unmergeable / base-conflict** root cause - classify it with the logic already in `directorMergeConflictReason` / `directorMergeConflictReasonFromComments` (`agent_director_merge.go:130`) - bump `LandFailures`. When `LandFailures >= landFailureCap` (propose `landFailureCap = 2`, matching the acceptance line "no replace-PR issue filed more than once"), transition the entry to a new terminal-ish state `land-blocked` that `backlogQueuedPicks` **excludes** (add the case there), exactly how `orphaned-needs-redispatch` already parks out of the pick set. **Suppress the re-file:** the "replace unmergeable PR #X" issues are currently created by the engineer/director *agent* (LLM behavior), not by deterministic ward code - so the breaker's deterministic job is (a) park the lineage out of the pick set (above) and (b) **gate any auto-file**. The clean hook is the reap/salvage reopen path (`container_reap.go:690` `notifySalvage`, `:1101` `reportUnlandedExtraRepos`, both call `reopenIssue`): before reopening/refiling for a lineage, count prior cycles on that lineage (from the ledger `LandFailures`) and, past the cap, **post one keystone-escalation comment instead of reopening**. Lineage identity: parse the `Closes #root` / "replace unmergeable PR #N" reference chain (`directorLinkedIssueNumber`, `agent_director_merge.go:455`, already extracts the linked number) and thread the root forward as `Lineage`. This reuses the *shape* of the ward#595 bounded-retry design, just keyed on landing outcome rather than pre-launch death. --- ### Component 3 - Serialized merge queue **Goal:** engineers on one repo land one-at-a-time against a stable base; a moving main auto-rebases the queue rather than stranding conflicts. **Rework `runDirectorMergeRepo` (`agent_director_merge.go:83`)** from "merge every eligible PR in one pass" to a serialized queue step: 1. Collect eligible PRs (existing `directorMergeEligibility`), order them (oldest-linked-issue first, or existing rank). 2. Merge **at most one** per repo per tick via `mergePullRequestWithHead` (already head-pinned, `forgejo_ops.go:446`). 3. Immediately **rebase the remaining open PRs onto the new main.** This needs one new client method - `updatePullRequestBranch(ctx, owner, repo, number)` in `forgejo_ops.go` calling Forgejo's `POST /repos/{o}/{r}/pulls/{n}/update` (the rebase-onto-base endpoint). It does not exist yet; every other method it composes with does. 4. Re-check `pr.Mergeable` on the next tick so the queue drains one clean merge at a time. This converts the ward#1063 / aos#445 "repair a jammed queue by hand" into a standing invariant: **serialize + auto-rebase**, so PRs never all conflict against a moving main. Because dispatch already forwards a PR ref into engineers (per recent `issue-913` work), pairing serialized merge with rebase-onto-main also stabilizes the base each new engineer branches from. --- ### Capacity protection (folds ward#1016 / ward#1025) - cheap, do it first The root cause named in the thread (Codex `/goal` director mass-dispatching ~30 branches with no backpressure) is the same global-budget gap. `directorDispatchTick`'s `avail` is a **single scope-wide number** with no per-repo ceiling. Add a `perRepoMax` to `backlogConfig` (`agent_director.go:176`, default ~ half of `maxParallel`, min 1) and, when selecting `chosen`, cap how many picks share one `p.repo`. This directly satisfies "one jammed repo cannot consume all engineer slots" and is a small, independently-landable PR. It composes with ward#1073's **open-PR backpressure gate** (default 6): ward#1073 caps outstanding PRs per repo at the broker + pre-flight layer, this caps concurrent *engineers* per repo at the director layer - the two are complementary rails, not duplicates, so land ward#1073 and this per-repo slot cap without stepping on each other. --- ### Suggested PR sequence (each independently landable, ordered by unblock value) 1. **PR-A - per-repo slot cap** (`backlogConfig.perRepoMax` + selection filter in `directorDispatchTick`). Smallest, highest immediate churn-relief, no new forge reads. Ties off ward#1016/#1025. 2. **PR-B - landability gate** (`agent_director_landability.go` + `directorBackend.landableRepos` seam + dispatch filter). Depends on nothing; reuses `getBranch`/`getCommitCombinedStatus`. Delivers the headline acceptance criterion ("red main -> stop dispatching"). 3. **PR-C - circuit-breaker** (`backlogEntry.LandFailures`/`Lineage`, increment in reconcile, `land-blocked` state, refile guard in `notifySalvage`). Depends on B's landability classification for a clean keystone signal but can ship in parallel keyed on merge-conflict reason. 4. **PR-D - `updatePullRequestBranch` client method + serialized `runDirectorMergeRepo`**. Depends on nothing structurally; generalizes ward#1063 / aos#445. Land last since it is the largest behavioral change to the merge path. Each PR is small enough to carry its own test file next to the existing `agent_director_*_test.go` / `agent_dispatch_*_test.go` suites, and each satisfies one line of the epic's Acceptance block on its own. ### Open questions to confirm before implementation - **K for the breaker.** The acceptance text implies K = 1 refile (breaker fires before the *second*). `landFailureCap = 2` (park after two failed landings) reads as the safe default - confirm the exact count. - **Keystone attribution depth.** Ship coarse (hold repo, comment the tracking issue) first, or invest in per-context->issue mapping up front? Recommend coarse-first. - **Merge-queue ownership.** Confirm the serialized queue lives in `ward agent director merge` (heartbeat-driven) and not a new standalone `ward agent queue` command - the former reuses all existing eligibility/status-gate code. **Recommendation:** proceed with PR-A immediately (it is the direct antidote to the ~30-branch stampede in the thread's root-cause note and is nearly free), then B/C/D in sequence. All four are ward-only; file them as four children of this epic. --- Researched and posted automatically by `ward agent advisor --harness claude` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-11 00:00:23 +00:00

Author

Owner

Cleanup note from the director surface on 2026-07-10:

The dispatch lane repeated the failure mode this epic is meant to stop. The surface found seven recent ward engineers that exited together with code 2, one still-running agentic-os#445 engineer stuck in conflict churn, and several no-PR issue branches more than 100 commits behind main that kept being resumed instead of forcing a clean start or pausing the issue.

Cleanup performed:

Stopped the still-running agentic-os#445 engineer so it could not race cleanup.
Removed all stopped engineer-* containers from the host Docker view.
Deleted already-merged no-PR branches: ward issue-980, issue-1073, issue-930, issue-786, issue-1033, issue-1064, issue-1096, issue-1087, issue-1084, issue-1045, issue-1085, issue-1083; agentic-os issue-462, issue-455; deploy issue-130, issue-134.
Deleted stale no-PR ward branches that directly caused resume churn: issue-1007, issue-1011, issue-1006, issue-1000, issue-998, issue-988.
Deleted deploy issue-133, which duplicated the open salvage PR branch ward-salvage/deploy-5e3b3535.
Merged the one PR that was actually ready after cleanup, coilyco-flight-deck/agentic-os#436.

Source behavior still missing: the director should fail closed before dispatch when an issue has a stale no-PR branch, repeated recent failed runs, or an open PR backlog above the configured threshold. It should mark/report the issue as needing branch cleanup or human review instead of launching another engineer against the same stale state.

Cleanup note from the director surface on 2026-07-10: The dispatch lane repeated the failure mode this epic is meant to stop. The surface found seven recent ward engineers that exited together with code 2, one still-running `agentic-os#445` engineer stuck in conflict churn, and several no-PR issue branches more than 100 commits behind `main` that kept being resumed instead of forcing a clean start or pausing the issue. Cleanup performed: * Stopped the still-running `agentic-os#445` engineer so it could not race cleanup. * Removed all stopped `engineer-*` containers from the host Docker view. * Deleted already-merged no-PR branches: ward `issue-980`, `issue-1073`, `issue-930`, `issue-786`, `issue-1033`, `issue-1064`, `issue-1096`, `issue-1087`, `issue-1084`, `issue-1045`, `issue-1085`, `issue-1083`; agentic-os `issue-462`, `issue-455`; deploy `issue-130`, `issue-134`. * Deleted stale no-PR ward branches that directly caused resume churn: `issue-1007`, `issue-1011`, `issue-1006`, `issue-1000`, `issue-998`, `issue-988`. * Deleted deploy `issue-133`, which duplicated the open salvage PR branch `ward-salvage/deploy-5e3b3535`. * Merged the one PR that was actually ready after cleanup, `coilyco-flight-deck/agentic-os#436`. Source behavior still missing: the director should fail closed before dispatch when an issue has a stale no-PR branch, repeated recent failed runs, or an open PR backlog above the configured threshold. It should mark/report the issue as needing branch cleanup or human review instead of launching another engineer against the same stale state.

coilyco-ops commented

2026-07-13 09:23:34 +00:00

Author

Owner

Evidence for this epic from the burn-down session: it ran exactly the merge-queue shape this epic describes, by hand. Strict one-at-a-time landing, merge onto the moving main per slot, verify with ward build plus ward test plus pre-commit, push, confirm registration, delete the branch. That loop cleared a ten PR jam that had hard-blocked dispatch, with every landed push producing its release tag (v0.641.0 through v0.649.0). Two design notes from the run: conflicts reshuffled after every land, so per-slot re-resolution is mandatory and any precomputed conflict analysis goes stale immediately, and Forgejo never marked push-landed PRs merged, so the queue needs its own close-with-provenance step (ward#1170). The convergence circuit-breaker matters too - the jam only existed because the backpressure gate refused the repair work that would have relieved it (ward#1073).

coilyco-ops added

priority

and removed

priority

labels

2026-07-29 06:27:25 +00:00

coilyco-ops commented

2026-07-31 08:14:03 +00:00

Author

Owner

Closing as superseded by #1620. The autonomous burndown and redispatch loop that produced convergence churn is being removed rather than expanded with another scheduler and merge queue.

coilyco-ops closed this issue

2026-07-31 08:14:05 +00:00

Commenting is not possible because the repository is archived.

No milestone

No project

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/ward#1069

No description provided.

Rows
Columns