coilyco-flight-deck/ward

Fork 0

ward agent engineer: reservation (local hold + forge road-block comment) not rolled back when docker run fails #570

New issue

Closed

opened 2026-07-03 17:15:17 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-07-03 17:15:17 +00:00

Member

Symptom

When a ward agent engineer dispatch fails at docker run (see companion env-file issue #569), it leaves both halves of the reservation orphaned:

Local hold - ~/.ward/agent-reservations/<owner>-<repo>-issue-<N>.json (2h TTL).
Forge road-block - a ward-agent-reservation comment on the issue posted by coilyco-ops, claiming container engineer-claude-infrastructure-452 ... is carrying this issue.

Neither is true after the failure - no container was ever created. A human reading the issue sees a live-run road-block that does not exist, and a re-run needs --force (or a 2h wait) even though nothing is actually holding the slot.

Root cause

The reservation is acquired (local file written + forge comment posted) before docker run, with no rollback on the launch failing. The happy path presumably releases on container exit, but the docker run error path returns without releasing either half.

Repro

ward agent engineer coilyco-flight-deck/infrastructure#452 --workflow patch-only on a host where the env-file bug fires - observe the leftover reservation file and the issuecomment on the issue after exit 125. (For 452 the leftover comment was issuecomment-21221; the local file has been manually cleared.)

Fix direction

Wrap the launch so a failed docker run (or any failure after reservation acquire and before the container is confirmed up) releases both the local hold and the forge comment - a defer-with-rollback that is disarmed only once the container is confirmed running. Bonus: the forge comment currently has no ward delete path (ward ops forgejo issue-comment exposes only list), so cleanup is manual - a delete surface would help.

## Symptom When a `ward agent engineer` dispatch fails at `docker run` (see companion env-file issue #569), it leaves both halves of the reservation orphaned: 1. Local hold - `~/.ward/agent-reservations/<owner>-<repo>-issue-<N>.json` (2h TTL). 2. Forge road-block - a `ward-agent-reservation` comment on the issue posted by `coilyco-ops`, claiming `container engineer-claude-infrastructure-452 ... is carrying this issue`. Neither is true after the failure - no container was ever created. A human reading the issue sees a live-run road-block that does not exist, and a re-run needs `--force` (or a 2h wait) even though nothing is actually holding the slot. ## Root cause The reservation is acquired (local file written + forge comment posted) *before* `docker run`, with no rollback on the launch failing. The happy path presumably releases on container exit, but the `docker run` error path returns without releasing either half. ## Repro `ward agent engineer coilyco-flight-deck/infrastructure#452 --workflow patch-only` on a host where the env-file bug fires - observe the leftover reservation file and the `issuecomment` on the issue after exit 125. (For 452 the leftover comment was `issuecomment-21221`; the local file has been manually cleared.) ## Fix direction Wrap the launch so a failed `docker run` (or any failure after reservation acquire and before the container is confirmed up) releases both the local hold and the forge comment - a defer-with-rollback that is disarmed only once the container is confirmed running. Bonus: the forge comment currently has no ward delete path (ward ops forgejo issue-comment exposes only `list`), so cleanup is manual - a delete surface would help.

coilyco-ops added the

headless

labels

2026-07-03 17:30:40 +00:00

coilyco-ops commented

2026-07-03 17:41:15 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-570 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T17:41:09Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-570` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T17:41:09Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-03 17:50:28 +00:00

fix(agent): roll reservation back when a launch dies before the container is up

coilysiren closed this issue

2026-07-03 17:50:28 +00:00

coilysiren commented

2026-07-03 17:51:01 +00:00

Owner

WARD-OUTCOME: done - launch failures now roll back both reservation halves (local sentinel + forge road-block) instead of orphaning them.

Honest retro: this one was cleaner than I expected going in. The reservation subsystem already had every piece I needed - a release-marker convention (ward#264) the reaper uses on pre-launch death, a reservationReleaseCommentBody, and freshReservationComment logic that already honors a release stamped at/after a reservation. So the fix was mostly plumbing: have reserveIssue hand back a both-halves release, and arm a defer-with-rollback in launchAgentContainer that disarms the moment createAgentContainer confirms the container up. I leaned on the existing release-comment path rather than adding a hard delete, which kept the blast radius small and matched the reaper exactly.

What fought back: nothing conceptually, but the code-comment discipline hook did - the 2-line/90-char cap made me rewrite every doc comment two or three times to land the ward-ref inside the budget. Minor, but it ate a few cycles.

Confidence is high on the core logic (unit-tested the release path across success, comment-failure, unsupported-unlock, and soft-unlock branches; full suite + pre-commit green). The one gap I could not unit-test cheaply is the launchAgentContainer wiring itself - hostForgeClient builds a real forge client, so the arm/disarm sequencing is covered by reading, not a test. It is a small, linear defer so I am comfortable, but an integration test there would be the obvious follow-up.

Follow-up worth filing: the issue bonus - a real ward ops forgejo issue-comment delete surface. I deliberately did not add it (needs a ward-kdl guardfile surface, out of scope for the rollback fix), but cleanup of a stray reservation comment is still manual today. The release-comment approach makes it a non-blocker, not a non-issue.

WARD-OUTCOME: done - launch failures now roll back both reservation halves (local sentinel + forge road-block) instead of orphaning them. Honest retro: this one was cleaner than I expected going in. The reservation subsystem already had every piece I needed - a release-marker convention (ward#264) the reaper uses on pre-launch death, a `reservationReleaseCommentBody`, and `freshReservationComment` logic that already honors a release stamped at/after a reservation. So the fix was mostly plumbing: have `reserveIssue` hand back a both-halves release, and arm a defer-with-rollback in `launchAgentContainer` that disarms the moment `createAgentContainer` confirms the container up. I leaned on the existing release-comment path rather than adding a hard delete, which kept the blast radius small and matched the reaper exactly. What fought back: nothing conceptually, but the code-comment discipline hook did - the 2-line/90-char cap made me rewrite every doc comment two or three times to land the ward-ref inside the budget. Minor, but it ate a few cycles. Confidence is high on the core logic (unit-tested the release path across success, comment-failure, unsupported-unlock, and soft-unlock branches; full suite + pre-commit green). The one gap I could not unit-test cheaply is the `launchAgentContainer` wiring itself - `hostForgeClient` builds a real forge client, so the arm/disarm sequencing is covered by reading, not a test. It is a small, linear defer so I am comfortable, but an integration test there would be the obvious follow-up. Follow-up worth filing: the issue bonus - a real `ward ops forgejo issue-comment` delete surface. I deliberately did not add it (needs a ward-kdl guardfile surface, out of scope for the rollback fix), but cleanup of a stray reservation comment is still manual today. The release-comment approach makes it a non-blocker, not a non-issue.

coilysiren referenced this issue

2026-07-03 17:54:25 +00:00

warded advisor dispatch 125s on host docker run (research mode) while engineer runs succeed same session #577

coilyco-ops commented

2026-07-04 08:35:24 +00:00

Author

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-570 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-04T08:35:18Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-570` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-04T08:35:18Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-04 08:51:25 +00:00

feat(ops-forgejo): add issue-comment delete leaf for reservation cleanup