coilyco-flight-deck/ward

Fork 0

REGRESSION: reaper salvages + reopens runs whose work already landed (provenance-file residual + gate ignores run-owned-landed) #662

New issue

Closed

opened 2026-07-08 04:23:54 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-07-08 04:23:54 +00:00

Member

Regression: the reaper salvages + reopens runs whose work already landed

Severity: high. Every dispatched run this session (#653, #654, #660 x2) landed its closes #N commit on main yet the reaper salvaged residual and reopened the issue with run-owned landed: yes + missing same-repo closing reference. Four+ spurious ward-salvage/* branches in one hour, each containing only a junk residual commit.

Correction to this issue's original diagnosis

The first version of #662 blamed Forgejo's passive commit-keyword close racing on already-seen commits. That was wrong. #653 was reaper-reopened just like #654/#660 (run-owned landed: yes, salvage ward-salvage/ward-d9e51395). The close never depended on Forgejo's scan - the reaper actively reopens landed work. Rewritten with the real root cause below.

Root cause (two parts, both in `cmd/ward/container_reap.go`)

1. The reaper manufactures a residual commit every run. Each run writes .ward-run-provenance.json into the work tree (read back at container_reap.go:189), untracked and never gitignored. captureAndCommitResidualRepo (:302) runs git add -A (:305) which stages that file, then commits it (:311) as ward-container: residual ... work with no closes #N trailer. So after the run's real work lands on main, the reaper itself creates a trailing origin/main..HEAD commit carrying no closing reference. Every reap diagnostic shows exactly this: working tree: 1 dirty path(s) -> ?? .ward-run-provenance.json, and HEAD is NOT yet on origin/main.

2. The closing-reference gate ignores the landed verdict it just computed. At :197-202:

landed := r.runProvenanceLanded(ctx, work, prov, env.Issue)   // TRUE: closes #N is on origin/main
if env.Issue != 0 && !r.issueClosingReferencePresent(ctx, work, env.Issue) {
    return r.salvage(... reapDecision{Gate: "missing same-repo closing reference", Landed: landed})
}

issueClosingReferencePresent checks only origin/main..HEAD (:678) - the trailing provenance commit from part 1 - so it returns false, and the gate salvages + reopens without checking landed. landed is threaded only into the diagnostics struct (Landed: landed), never into the branch condition, despite the :195 comment claiming the verdict is "reused by every gate below." When landed is true, closes #N is at/before origin/main so this gate is guaranteed to fire falsely.

Fix (both)

(a) Stop manufacturing the residual. Exclude .ward-run-provenance.json from the residual snapshot - gitignore it, write it outside the work tree, or skip it in captureAndCommitResidualRepo. With it gone, residual == 0 and :173 ("nothing to reap: tree clean, HEAD on origin/main") early-returns before any gate.
(b) Honor landed in the gate. Guard :198 with !landed: if env.Issue != 0 && !landed && !r.issueClosingReferencePresent(...). A run whose closing commit already reached main must never be salvaged + reopened for a missing closing-ref in trailing residual. This is the fundamental fix - any legitimate trailing residual (e.g. an integrate-main merge commit) would otherwise re-trigger even after (a). Confirm executeReap (:217, called with landed) also does not reopen when landed is true.

(a) removes the current trigger, (b) is the correctness invariant. Land both.

Propagation caveat

The reaper runs as the pinned ward version inside each container (v0.439.0 today). Landing this fix on main does not stop the bleeding for in-flight runs until a ward version carrying the fix is released and used by new containers. So a run dispatched to fix this will itself land its code but still be spuriously reaped by the old reaper - the code lands, the salvage+reopen noise continues until the fixed version propagates.

Cleanup (separate from the fix)

The spurious ward-salvage/* branches (each just the provenance-file residual, no lost work) and the falsely-reopened issues need sweeping. Landed issues re-closed by hand: #653, #654, #660. Branch deletion is a human/operator action, not done from this read-only surface.

Filed from the read-only director surface (she/her).

## Regression: the reaper salvages + reopens runs whose work already landed **Severity: high.** Every dispatched run this session (#653, #654, #660 x2) **landed its `closes #N` commit on `main`** yet the reaper salvaged residual and **reopened** the issue with `run-owned landed: yes` + `missing same-repo closing reference`. Four+ spurious `ward-salvage/*` branches in one hour, each containing only a junk residual commit. ## Correction to this issue's original diagnosis The first version of #662 blamed Forgejo's passive commit-keyword close racing on already-seen commits. **That was wrong.** #653 was reaper-reopened just like #654/#660 (`run-owned landed: yes`, salvage `ward-salvage/ward-d9e51395`). The close never depended on Forgejo's scan - the **reaper actively reopens** landed work. Rewritten with the real root cause below. ## Root cause (two parts, both in `cmd/ward/container_reap.go`) **1. The reaper manufactures a residual commit every run.** Each run writes `.ward-run-provenance.json` into the work tree (read back at `container_reap.go:189`), untracked and never gitignored. `captureAndCommitResidualRepo` (`:302`) runs `git add -A` (`:305`) which stages that file, then commits it (`:311`) as `ward-container: residual ... work` with **no `closes #N` trailer**. So after the run's real work lands on `main`, the reaper itself creates a trailing `origin/main..HEAD` commit carrying no closing reference. Every reap diagnostic shows exactly this: `working tree: 1 dirty path(s)` -> `?? .ward-run-provenance.json`, and `HEAD is NOT yet on origin/main`. **2. The closing-reference gate ignores the `landed` verdict it just computed.** At `:197-202`: ```go landed := r.runProvenanceLanded(ctx, work, prov, env.Issue) // TRUE: closes #N is on origin/main if env.Issue != 0 && !r.issueClosingReferencePresent(ctx, work, env.Issue) { return r.salvage(... reapDecision{Gate: "missing same-repo closing reference", Landed: landed}) } ``` `issueClosingReferencePresent` checks only `origin/main..HEAD` (`:678`) - the trailing provenance commit from part 1 - so it returns false, and the gate salvages + reopens **without checking `landed`**. `landed` is threaded only into the diagnostics struct (`Landed: landed`), never into the branch condition, despite the `:195` comment claiming the verdict is "reused by every gate below." When `landed` is true, `closes #N` is at/before `origin/main` so this gate is **guaranteed to fire falsely**. ## Fix (both) * **(a) Stop manufacturing the residual.** Exclude `.ward-run-provenance.json` from the residual snapshot - gitignore it, write it outside the work tree, or skip it in `captureAndCommitResidualRepo`. With it gone, `residual == 0` and `:173` ("nothing to reap: tree clean, HEAD on origin/main") early-returns before any gate. * **(b) Honor `landed` in the gate.** Guard `:198` with `!landed`: `if env.Issue != 0 && !landed && !r.issueClosingReferencePresent(...)`. A run whose closing commit already reached `main` must never be salvaged + reopened for a missing closing-ref in trailing residual. This is the fundamental fix - any legitimate trailing residual (e.g. an integrate-main merge commit) would otherwise re-trigger even after (a). Confirm `executeReap` (`:217`, called with `landed`) also does not reopen when `landed` is true. (a) removes the current trigger, (b) is the correctness invariant. Land both. ## Propagation caveat The reaper runs as the **pinned** ward version inside each container (v0.439.0 today). Landing this fix on `main` does not stop the bleeding for in-flight runs until a ward version carrying the fix is released and used by new containers. So a run dispatched to fix this will itself land its code but still be spuriously reaped by the old reaper - the code lands, the salvage+reopen noise continues until the fixed version propagates. ## Cleanup (separate from the fix) The spurious `ward-salvage/*` branches (each just the provenance-file residual, no lost work) and the falsely-reopened issues need sweeping. Landed issues re-closed by hand: #653, #654, #660. Branch deletion is a human/operator action, not done from this read-only surface. Filed from the read-only director surface (she/her).

coilyco-ops added the

label

2026-07-08 04:23:54 +00:00

coilyco-ops changed title from ~~Landed issue not auto-closed when the closing commit reaches main via an integration/merge (Forgejo keyword-close no-ops on already-seen commits)~~ to REGRESSION: reaper salvages + reopens runs whose work already landed (provenance-file residual + gate ignores run-owned-landed)

2026-07-08 04:45:48 +00:00

coilyco-ops added the

headless

label

2026-07-08 04:45:49 +00:00

coilyco-ops referenced this issue

2026-07-08 04:45:49 +00:00

Config-driven ward: WARD_CONFIG_REF grammar + the shared syncGitRef TTL resolver (#650-2) #654

coilyco-ops referenced this issue

2026-07-08 04:45:49 +00:00

ward agent: accept both --harness and --agent for the harness pick (retire --driver to deprecated alias) #660

coilyco-ops commented

2026-07-08 04:46:29 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-662 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-08T04:46:24Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#662 · branch issue-662 · driver claude · workflow direct-main
Run: engineer-claude-ward-662 · ward v0.439.0 · dispatched 2026-07-08T04:46:24Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Issue body as seeded:

## Regression: the reaper salvages + reopens runs whose work already landed

**Severity: high.** Every dispatched run this session (#653, #654, #660 x2) **landed its `closes #N` commit on `main`** yet the reaper salvaged residual and **reopened** the issue with `run-owned landed: yes` + `missing same-repo closing reference`. Four+ spurious `ward-salvage/*` branches in one hour, each containing only a junk residual commit.

## Correction to this issue's original diagnosis

The first version of #662 blamed Forgejo's passive commit-keyword close racing on already-seen commits. **That was wrong.** #653 was reaper-reopened just like #654/#660 (`run-owned landed: yes`, salvage `ward-salvage/ward-d9e51395`). The close never depended on Forgejo's scan - the **reaper actively reopens** landed work. Rewritten with the real root cause below.

## Root cause (two parts, both in `cmd/ward/container_reap.go`)

**1. The reaper manufactures a residual commit every run.** Each run writes `.ward-run-provenance.json` into the work tree (read back at `container_reap.go:189`), untracked and never gitignored. `captureAndCommitResidualRepo` (`:302`) runs `git add -A` (`:305`) which stages that file, then commits it (`:311`) as `ward-container: residual ... work` with **no `closes #N` trailer**. So after the run's real work lands on `main`, the reaper itself creates a trailing `origin/main..HEAD` commit carrying no closing reference. Every reap diagnostic shows exactly this: `working tree: 1 dirty path(s)` -> `?? .ward-run-provenance.json`, and `HEAD is NOT yet on origin/main`.

**2. The closing-reference gate ignores the `landed` verdict it just computed.** At `:197-202`:

` ` `go
landed := r.runProvenanceLanded(ctx, work, prov, env.Issue)   // TRUE: closes #N is on origin/main
if env.Issue != 0 && !r.issueClosingReferencePresent(ctx, work, env.Issue) {
    return r.salvage(... reapDecision{Gate: "missing same-repo closing reference", Landed: landed})
}
` ` `

`issueClosingReferencePresent` ch

… (truncated to 2000 chars; full body is on this issue)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.439.0).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-662` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-08T04:46:24Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#662` · branch `issue-662` · driver `claude` · workflow `direct-main` - **Run:** `engineer-claude-ward-662` · ward `v0.439.0` · dispatched `2026-07-08T04:46:24Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). **Issue body as seeded:** ``` ## Regression: the reaper salvages + reopens runs whose work already landed **Severity: high.** Every dispatched run this session (#653, #654, #660 x2) **landed its `closes #N` commit on `main`** yet the reaper salvaged residual and **reopened** the issue with `run-owned landed: yes` + `missing same-repo closing reference`. Four+ spurious `ward-salvage/*` branches in one hour, each containing only a junk residual commit. ## Correction to this issue's original diagnosis The first version of #662 blamed Forgejo's passive commit-keyword close racing on already-seen commits. **That was wrong.** #653 was reaper-reopened just like #654/#660 (`run-owned landed: yes`, salvage `ward-salvage/ward-d9e51395`). The close never depended on Forgejo's scan - the **reaper actively reopens** landed work. Rewritten with the real root cause below. ## Root cause (two parts, both in `cmd/ward/container_reap.go`) **1. The reaper manufactures a residual commit every run.** Each run writes `.ward-run-provenance.json` into the work tree (read back at `container_reap.go:189`), untracked and never gitignored. `captureAndCommitResidualRepo` (`:302`) runs `git add -A` (`:305`) which stages that file, then commits it (`:311`) as `ward-container: residual ... work` with **no `closes #N` trailer**. So after the run's real work lands on `main`, the reaper itself creates a trailing `origin/main..HEAD` commit carrying no closing reference. Every reap diagnostic shows exactly this: `working tree: 1 dirty path(s)` -> `?? .ward-run-provenance.json`, and `HEAD is NOT yet on origin/main`. **2. The closing-reference gate ignores the `landed` verdict it just computed.** At `:197-202`: ` ` `go landed := r.runProvenanceLanded(ctx, work, prov, env.Issue) // TRUE: closes #N is on origin/main if env.Issue != 0 && !r.issueClosingReferencePresent(ctx, work, env.Issue) { return r.salvage(... reapDecision{Gate: "missing same-repo closing reference", Landed: landed}) } ` ` ` `issueClosingReferencePresent` ch ``` … (truncated to 2000 chars; full body is on this issue) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.439.0). </details>  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue

2026-07-08 04:49:15 +00:00

[epic] Config-driven ward: one precompiled binary, personal config resolved live from a git ref (supersedes the build-variant matrix) #650

coilyco-ops commented

2026-07-08 07:53:49 +00:00

Author

Member

🔒 Reserved by ward agent --harness codex — container engineer-codex-ward-662 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-08T07:53:47Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#662 · branch issue-662 · harness codex · workflow direct-main
Run: engineer-codex-ward-662 · ward v0.443.0 · dispatched 2026-07-08T07:53:47Z
Comment thread: 0 included in the pre-flight read, 1 stripped (ward's own automated comments).
- stripped: @coilyco-ops (2026-07-08T04:46:29Z)

Issue body as seeded:

## Regression: the reaper salvages + reopens runs whose work already landed

**Severity: high.** Every dispatched run this session (#653, #654, #660 x2) **landed its `closes #N` commit on `main`** yet the reaper salvaged residual and **reopened** the issue with `run-owned landed: yes` + `missing same-repo closing reference`. Four+ spurious `ward-salvage/*` branches in one hour, each containing only a junk residual commit.

## Correction to this issue's original diagnosis

The first version of #662 blamed Forgejo's passive commit-keyword close racing on already-seen commits. **That was wrong.** #653 was reaper-reopened just like #654/#660 (`run-owned landed: yes`, salvage `ward-salvage/ward-d9e51395`). The close never depended on Forgejo's scan - the **reaper actively reopens** landed work. Rewritten with the real root cause below.

## Root cause (two parts, both in `cmd/ward/container_reap.go`)

**1. The reaper manufactures a residual commit every run.** Each run writes `.ward-run-provenance.json` into the work tree (read back at `container_reap.go:189`), untracked and never gitignored. `captureAndCommitResidualRepo` (`:302`) runs `git add -A` (`:305`) which stages that file, then commits it (`:311`) as `ward-container: residual ... work` with **no `closes #N` trailer**. So after the run's real work lands on `main`, the reaper itself creates a trailing `origin/main..HEAD` commit carrying no closing reference. Every reap diagnostic shows exactly this: `working tree: 1 dirty path(s)` -> `?? .ward-run-provenance.json`, and `HEAD is NOT yet on origin/main`.

**2. The closing-reference gate ignores the `landed` verdict it just computed.** At `:197-202`:

` ` `go
landed := r.runProvenanceLanded(ctx, work, prov, env.Issue)   // TRUE: closes #N is on origin/main
if env.Issue != 0 && !r.issueClosingReferencePresent(ctx, work, env.Issue) {
    return r.salvage(... reapDecision{Gate: "missing same-repo closing reference", Landed: landed})
}
` ` `

`issueClosingReferencePresent` ch

… (truncated to 2000 chars; full body is on this issue)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.443.0).

— Codex, via ward agent

🔒 Reserved by `ward agent --harness codex` — container `engineer-codex-ward-662` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-08T07:53:47Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#662` · branch `issue-662` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-ward-662` · ward `v0.443.0` · dispatched `2026-07-08T07:53:47Z` - **Comment thread:** 0 included in the pre-flight read, 1 stripped (ward's own automated comments). - stripped: @coilyco-ops (2026-07-08T04:46:29Z) **Issue body as seeded:** ``` ## Regression: the reaper salvages + reopens runs whose work already landed **Severity: high.** Every dispatched run this session (#653, #654, #660 x2) **landed its `closes #N` commit on `main`** yet the reaper salvaged residual and **reopened** the issue with `run-owned landed: yes` + `missing same-repo closing reference`. Four+ spurious `ward-salvage/*` branches in one hour, each containing only a junk residual commit. ## Correction to this issue's original diagnosis The first version of #662 blamed Forgejo's passive commit-keyword close racing on already-seen commits. **That was wrong.** #653 was reaper-reopened just like #654/#660 (`run-owned landed: yes`, salvage `ward-salvage/ward-d9e51395`). The close never depended on Forgejo's scan - the **reaper actively reopens** landed work. Rewritten with the real root cause below. ## Root cause (two parts, both in `cmd/ward/container_reap.go`) **1. The reaper manufactures a residual commit every run.** Each run writes `.ward-run-provenance.json` into the work tree (read back at `container_reap.go:189`), untracked and never gitignored. `captureAndCommitResidualRepo` (`:302`) runs `git add -A` (`:305`) which stages that file, then commits it (`:311`) as `ward-container: residual ... work` with **no `closes #N` trailer**. So after the run's real work lands on `main`, the reaper itself creates a trailing `origin/main..HEAD` commit carrying no closing reference. Every reap diagnostic shows exactly this: `working tree: 1 dirty path(s)` -> `?? .ward-run-provenance.json`, and `HEAD is NOT yet on origin/main`. **2. The closing-reference gate ignores the `landed` verdict it just computed.** At `:197-202`: ` ` `go landed := r.runProvenanceLanded(ctx, work, prov, env.Issue) // TRUE: closes #N is on origin/main if env.Issue != 0 && !r.issueClosingReferencePresent(ctx, work, env.Issue) { return r.salvage(... reapDecision{Gate: "missing same-repo closing reference", Landed: landed}) } ` ` ` `issueClosingReferencePresent` ch ``` … (truncated to 2000 chars; full body is on this issue) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.443.0). </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-08 08:00:10 +00:00

fix(reaper): ignore provenance artifact in residual reap

coilyco-ops closed this issue

2026-07-08 08:00:10 +00:00

coilyco-ops commented

2026-07-08 08:00:54 +00:00

Author

Member

WARD-OUTCOME: done - reaper now ignores the provenance artifact and skips the false close-ref salvage path when the run already landed.

This was straightforward once the failure was reproduced in a landed-run fixture. The only surprise was that the provenance artifact fix had to be in the reap code path itself, not repo ignore rules. Confidence is high.

PR-BODY-NOTE: ADVISORY-ONLY REVIEW: no heterogeneous reviewer family was available besides the worker (codex), so the adversarial panel could not run and did NOT gate this diff. Dropped: opencode (unavailable: opencode not on PATH); codex (worker's own family - never reviews its own diff). A human should review this change with the extra scrutiny an unrun panel would have applied.

WARD-OUTCOME: done - reaper now ignores the provenance artifact and skips the false close-ref salvage path when the run already landed. This was straightforward once the failure was reproduced in a landed-run fixture. The only surprise was that the provenance artifact fix had to be in the reap code path itself, not repo ignore rules. Confidence is high. PR-BODY-NOTE: ADVISORY-ONLY REVIEW: no heterogeneous reviewer family was available besides the worker (codex), so the adversarial panel could not run and did NOT gate this diff. Dropped: opencode (unavailable: opencode not on PATH); codex (worker's own family - never reviews its own diff). A human should review this change with the extra scrutiny an unrun panel would have applied.