Document manual stale reservation cleanup #1198

New issue

Closed

opened 2026-07-13 21:21:18 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-13 21:21:18 +00:00

Member

Document the manual stale reservation cleanup workaround.

Context:

Ward can get into a state where ward agent list shows non-visible container starting records that still count toward repo/global capacity.
Until #1191 and #1196 fix the architecture and override behavior, operators need a documented manual recovery path.
The durable reservation marker is a Forgejo issue comment containing ward-agent-reservation / WARD-RESERVATION.
This surface did not expose issue-comment edit, only targeted issue-comment delete.

Document:

How to identify stale reservations:
- ward agent list --json
- look for phase: container starting, empty started_at, expired or implausibly old budget_remaining
- verify whether the issue already has WARD-OUTCOME: submitted / other terminal evidence before clearing
How to find the reservation comment ID:
- ward ops forgejo issue-comment list <owner> <repo> <issue> --query '[?contains(body, "ward-agent-reservation")].{id:id,created_at:created_at}' --output json
How to clear it when deletion is the only exposed mutator:
- ward ops forgejo issue-comment delete <owner> <repo> <comment-id>
Safety notes:
- delete only the targeted reservation comment, never bulk-delete comments
- do not delete a reservation for a visible/running engineer
- if a comment edit verb is available in another surface, editing out the marker should also work and is less destructive
- after cleanup, run ward agent list and retry dispatch
Caveat from the 2026-07-13 incident: deleting reservation comments reduced active count but did not clear every displayed container starting record, so host/broker state may still need #1191/#1196. The workaround is still the first recovery step because the issue comments are the durable lock ward advertises.

Acceptance:

The workaround is documented in the ward operator docs near dispatch/reap/list troubleshooting.
Help text or docs point from the repo-limit error to the documented recovery path.
The docs distinguish issue-comment cleanup from host-side file deletion.

Document the manual stale reservation cleanup workaround. Context: - Ward can get into a state where `ward agent list` shows non-visible `container starting` records that still count toward repo/global capacity. - Until #1191 and #1196 fix the architecture and override behavior, operators need a documented manual recovery path. - The durable reservation marker is a Forgejo issue comment containing `ward-agent-reservation` / `WARD-RESERVATION`. - This surface did not expose issue-comment edit, only targeted issue-comment delete. Document: - How to identify stale reservations: - `ward agent list --json` - look for `phase: container starting`, empty `started_at`, expired or implausibly old `budget_remaining` - verify whether the issue already has `WARD-OUTCOME: submitted` / other terminal evidence before clearing - How to find the reservation comment ID: - `ward ops forgejo issue-comment list <owner> <repo> <issue> --query '[?contains(body, `"ward-agent-reservation"`)].{id:id,created_at:created_at}' --output json` - How to clear it when deletion is the only exposed mutator: - `ward ops forgejo issue-comment delete <owner> <repo> <comment-id>` - Safety notes: - delete only the targeted reservation comment, never bulk-delete comments - do not delete a reservation for a visible/running engineer - if a comment edit verb is available in another surface, editing out the marker should also work and is less destructive - after cleanup, run `ward agent list` and retry dispatch - Caveat from the 2026-07-13 incident: deleting reservation comments reduced active count but did not clear every displayed `container starting` record, so host/broker state may still need #1191/#1196. The workaround is still the first recovery step because the issue comments are the durable lock ward advertises. Acceptance: - The workaround is documented in the ward operator docs near dispatch/reap/list troubleshooting. - Help text or docs point from the repo-limit error to the documented recovery path. - The docs distinguish issue-comment cleanup from host-side file deletion.

coilyco-ops referenced this issue

2026-07-13 21:47:02 +00:00

Make stop --print honest for ghost launch records #1200

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-13 23:56:59 +00:00

Document manual stale reservation cleanup #1218

coilyco-ops commented

2026-07-13 23:58:49 +00:00

Author

Member

WARDED_WORKFLOW repair-request

PR 1218 is blocked by director review: #1218 (comment)

Repair scope:

Keep the existing PR branch issue-1198 and update PR 1218. Do not open a duplicate PR.
Fix the stale-reservation docs so terminal-but-stale issues are explicitly in scope. A terminal issue or PR-link WARD-OUTCOME is evidence that cleanup may be safe after verifying no visible engineer is running, not evidence that cleanup is irrelevant.
Include the host-side reservation file cleanup path for .ward/agent-reservations/<owner>-<repo>-issue-<n>.json and .lock, with the same safety checks.
Do not recommend WARD-OUTCOME: submitted as the durable terminal marker. Use WARD-OUTCOME: <fully qualified pull request link> examples.
Fix the git diff --check blank-line-at-EOF failure.
Re-run the ward test command and pre-commit before updating the PR.

WARDED_WORKFLOW repair-request PR 1218 is blocked by director review: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1218#issuecomment-32554 Repair scope: - Keep the existing PR branch `issue-1198` and update PR 1218. Do not open a duplicate PR. - Fix the stale-reservation docs so terminal-but-stale issues are explicitly in scope. A terminal issue or PR-link `WARD-OUTCOME` is evidence that cleanup may be safe after verifying no visible engineer is running, not evidence that cleanup is irrelevant. - Include the host-side reservation file cleanup path for `.ward/agent-reservations/<owner>-<repo>-issue-<n>.json` and `.lock`, with the same safety checks. - Do not recommend `WARD-OUTCOME: submitted` as the durable terminal marker. Use `WARD-OUTCOME: <fully qualified pull request link>` examples. - Fix the `git diff --check` blank-line-at-EOF failure. - Re-run the ward test command and pre-commit before updating the PR.