Make the reservation folder disposable and safe to clear wholesale

coilyco-ops commented

2026-07-14 01:31:07 +00:00

Member

Humans are going to recover from stale local reservation state by deleting the entire reservation folder, not by surgically deleting one JSON/lock pair per issue. Ward needs to architect for that reality instead of relying on a fragile per-issue manual cleanup path.

Context from the 2026-07-14 director run:

ward agent list --json kept reporting ghost rows for ward issues 1171, 1194, and 1196.
Docker had no matching ward engineer containers.
Forgejo issue threads had durable terminal WARDED_WORKFLOW outcomes and no active reservation comments.
ward agent dispatch-health --repo coilyco-flight-deck/ward --json reported running=0 and backpressure=false.
Dispatching the repair PR for ward#1196 / PR #1222 still failed with repo engineer limit is reached for coilyco-flight-deck/ward: 3 active engineer(s) (limit 3), even with --override-reservation --override-capacity --skip-preflight.
Kai cleared the entire host reservation folder as the practical workaround.

Problem:

The documented/operator mental model says “clear the stale reservation state,” but the ergonomic human action is “delete the whole reservations directory.”
If ward stores multiple kinds of state in that folder, the folder must be designed so wholesale deletion is safe, expected, and recoverable.
Canonical dispatch/reservation state cannot live only in local files if multi-node dispatch is coming. Local files may be cache, not authority.

Required design:

Treat the reservation directory as disposable cache. On missing directory or missing files, ward should recreate the directory and continue from canonical issue/thread/PR/container state.
Keep any non-disposable state out of that directory.
Make stale local reservation rows self-heal from durable issue outcomes and Docker reality.
Add a first-class cleanup command if operator intervention is still needed, but do not make the safe workaround require knowing individual file names.
Document the host-side emergency workaround as clearing the reservation directory, with a warning about what is safe because the directory is cache-only.

Acceptance:

Deleting the whole reservation folder does not corrupt dispatch state or lose canonical information.
After folder deletion, ward agent list and repo capacity gates no longer count old ghost rows.
The repair path for stale prelaunch ghosts works without requiring per-issue JSON/lock deletion.
Docs name the whole-folder cleanup path explicitly.

ward#1174
ward#1196
ward#1222

Humans are going to recover from stale local reservation state by deleting the entire reservation folder, not by surgically deleting one JSON/lock pair per issue. Ward needs to architect for that reality instead of relying on a fragile per-issue manual cleanup path. Context from the 2026-07-14 director run: - `ward agent list --json` kept reporting ghost rows for ward issues 1171, 1194, and 1196. - Docker had no matching ward engineer containers. - Forgejo issue threads had durable terminal `WARDED_WORKFLOW` outcomes and no active reservation comments. - `ward agent dispatch-health --repo coilyco-flight-deck/ward --json` reported `running=0` and `backpressure=false`. - Dispatching the repair PR for ward#1196 / PR #1222 still failed with `repo engineer limit is reached for coilyco-flight-deck/ward: 3 active engineer(s) (limit 3)`, even with `--override-reservation --override-capacity --skip-preflight`. - Kai cleared the entire host reservation folder as the practical workaround. Problem: - The documented/operator mental model says “clear the stale reservation state,” but the ergonomic human action is “delete the whole reservations directory.” - If ward stores multiple kinds of state in that folder, the folder must be designed so wholesale deletion is safe, expected, and recoverable. - Canonical dispatch/reservation state cannot live only in local files if multi-node dispatch is coming. Local files may be cache, not authority. Required design: - Treat the reservation directory as disposable cache. On missing directory or missing files, ward should recreate the directory and continue from canonical issue/thread/PR/container state. - Keep any non-disposable state out of that directory. - Make stale local reservation rows self-heal from durable issue outcomes and Docker reality. - Add a first-class cleanup command if operator intervention is still needed, but do not make the safe workaround require knowing individual file names. - Document the host-side emergency workaround as clearing the reservation directory, with a warning about what is safe because the directory is cache-only. Acceptance: - Deleting the whole reservation folder does not corrupt dispatch state or lose canonical information. - After folder deletion, `ward agent list` and repo capacity gates no longer count old ghost rows. - The repair path for stale prelaunch ghosts works without requiring per-issue JSON/lock deletion. - Docs name the whole-folder cleanup path explicitly. Related: - ward#1174 - ward#1196 - ward#1222

coilyco-ops added the

P1

headless

labels

2026-07-14 06:50:00 +00:00

coilyco-ops commented

2026-07-14 07:29:51 +00:00

Author

Member

WARDED_WORKFLOW: blocked 🛑

details

workflow: pull-request-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default

Implemented the reservation cache as disposable local state, added whole-folder clear, and taught ward agent list to prune stale cache rows from durable issue state and live Docker state.

Validation:

pre-commit run --all-files passed.
Targeted go test ./cmd/ward ... passed.
Full ward exec test is blocked by an unrelated syncGitRef failure in the existing suite.

Retrospective: the cache split was straightforward, but the full-suite failure is outside the reservation path and still needs follow-up.
Confidence: medium.
Follow-up: stabilize the existing syncGitRef test path, then rerun ward exec test.

WARDED_WORKFLOW: blocked 🛑 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default Implemented the reservation cache as disposable local state, added whole-folder clear, and taught `ward agent list` to prune stale cache rows from durable issue state and live Docker state. Validation: - `pre-commit run --all-files` passed. - Targeted `go test ./cmd/ward ...` passed. - Full `ward exec test` is blocked by an unrelated `syncGitRef` failure in the existing suite. Retrospective: the cache split was straightforward, but the full-suite failure is outside the reservation path and still needs follow-up. Confidence: medium. Follow-up: stabilize the existing `syncGitRef` test path, then rerun `ward exec test`.  </details>

coilyco-ops commented

2026-07-14 12:12:20 +00:00

Author

Member

Director burn-down update from 2026-07-14: PR #1236 remains open and nonmergeable. ward ops forgejo pr update coilyco-flight-deck ward 1236 returned 409 Conflict with merge failed because of conflict, so this branch needs a conflict repair against current main before director merge can proceed. New engineer dispatch is currently blocked by agentic-os#532 because the host cannot launch agentic-os-full:latest.

Director burn-down update from 2026-07-14: PR #1236 remains open and nonmergeable. `ward ops forgejo pr update coilyco-flight-deck ward 1236` returned 409 Conflict with `merge failed because of conflict`, so this branch needs a conflict repair against current main before director merge can proceed. New engineer dispatch is currently blocked by agentic-os#532 because the host cannot launch `agentic-os-full:latest`.

coilysiren referenced this issue

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283

~~coilyco-ops referenced this issue 2026-07-14 19:10:24 +00:00~~

Make the reservation folder disposable and safe to clear wholesale #1293

coilyco-ops commented

2026-07-14 19:14:16 +00:00

Author

Member

WARDED_WORKFLOW: blocked 🛑

details

workflow: pull-request-and-merge; review summary: review gate intentionally skipped by the temporary ward default pending brokered QA

Forgejo Actions run #2052 failed in the repo's test job after go vet and before golangci-lint. The run page exposed step-level failure only, so I could not pull a lower-level log from the API surface here.

Retrospective: the reservation-cache change itself felt straightforward. The blocker is outside the reservation path and looks like the existing go test track, not the new cleanup surface.

Confidence: medium.
Surprises: the Forgejo run view gave me the failing step list but not the raw log body through the available API surface.
Follow-up: resolve the failing go test path or the runner environment, then rerun the PR workflow.

WARDED_WORKFLOW: blocked 🛑 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: review gate intentionally skipped by the temporary ward default pending brokered QA Forgejo Actions run #2052 failed in the repo's `test` job after `go vet` and before `golangci-lint`. The run page exposed step-level failure only, so I could not pull a lower-level log from the API surface here. Retrospective: the reservation-cache change itself felt straightforward. The blocker is outside the reservation path and looks like the existing `go test` track, not the new cleanup surface. Confidence: medium. Surprises: the Forgejo run view gave me the failing step list but not the raw log body through the available API surface. Follow-up: resolve the failing `go test` path or the runner environment, then rerun the PR workflow. </details>

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-15 01:10:54 +00:00

Make the reservation folder disposable and safe to clear wholesale #1236

coilyco-ops commented

2026-07-15 01:21:46 +00:00

Author

Member

WARDED_WORKFLOW: blocked 🛑

details

workflow: pull-request-and-merge; review summary: review gate intentionally skipped by the temporary ward default pending brokered QA

The reservation cache work is validated locally, and PR #1236 is green, but Forgejo's merge endpoint is returning 405 after repeated settle attempts. That blocks landing even though the branch itself is clean.

Validation:

ward exec test passed.
pre-commit run --all-files passed.

Retrospective: the implementation felt straightforward. The remaining blocker is merge-queue pressure, not the reservation cache path.
Confidence: medium.
Surprise: Forgejo's merge queue can still refuse a green PR with a transient 405, and the available API surface does not expose a rerun path.
Follow-up: retry the merge later or have the director sweep the PR once the queue clears.

WARDED_WORKFLOW: blocked 🛑 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: review gate intentionally skipped by the temporary ward default pending brokered QA The reservation cache work is validated locally, and PR #1236 is green, but Forgejo's merge endpoint is returning 405 after repeated settle attempts. That blocks landing even though the branch itself is clean. Validation: - `ward exec test` passed. - `pre-commit run --all-files` passed. Retrospective: the implementation felt straightforward. The remaining blocker is merge-queue pressure, not the reservation cache path. Confidence: medium. Surprise: Forgejo's merge queue can still refuse a green PR with a transient 405, and the available API surface does not expose a rerun path. Follow-up: retry the merge later or have the director sweep the PR once the queue clears.  </details>

coilyco-ops commented

2026-07-15 03:28:46 +00:00

Author

Member

WARDED_WORKFLOW: blocked 🛑

details

workflow: pull-request-and-merge; review summary: review gate intentionally skipped by the temporary ward default pending brokered QA

The branch is green locally and the PR CI passed, but Forgejo's merge endpoint is still returning 405 "Please try again later" after repeated settle attempts.

Retrospective: the reservation-cache work itself is in good shape. The remaining blocker is Forgejo merge-queue pressure, not the code path.
Confidence: medium.
Follow-up: retry the merge once the Forgejo queue clears.

WARDED_WORKFLOW: blocked 🛑 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: review gate intentionally skipped by the temporary ward default pending brokered QA The branch is green locally and the PR CI passed, but Forgejo's merge endpoint is still returning 405 "Please try again later" after repeated settle attempts. Retrospective: the reservation-cache work itself is in good shape. The remaining blocker is Forgejo merge-queue pressure, not the code path. Confidence: medium. Follow-up: retry the merge once the Forgejo queue clears.  </details>

coilyco-ops added this to the warded control plane coherence milestone

2026-07-15 05:14:52 +00:00

coilyco-ops added the

coherence-core

label

2026-07-15 05:28:20 +00:00

coilyco-ops commented

2026-07-15 06:55:18 +00:00

Author

Member

Release triage note from Kai on 2026-07-15: headless carry is valid. The desired invariant is that the reservation folder is disposable: clearing it wholesale must not corrupt live dispatch because state is reconstructable or stale-detectable.

coilyco-ops commented

2026-07-15 07:08:46 +00:00

Author

Member

P1 clarity note from Kai on 2026-07-15: prior work appears to have died in PR #1236. First step is to check current main and PR history to confirm whether the reservation-cache implementation landed out of band. If not, lift the prior implementation shape rather than rediscovering it from scratch.

coilyco-ops commented

2026-07-15 07:16:34 +00:00

Author

Member

WARD-TRIAGE: warded control plane coherence milestone

This issue is part of the warded control plane coherence sprint. The release thesis is to make warded feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run.

For this sprint, headless means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

WARD-TRIAGE: warded control plane coherence milestone This issue is part of the `warded control plane coherence` sprint. The release thesis is to make `warded` feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run. For this sprint, `headless` means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

coilyco-ops commented

2026-07-15 07:16:47 +00:00

Author

Member

WARD-TRIAGE: coherence-core label

This issue is part of the coherence-core review set for the warded control plane coherence sprint. Treat it as release-spine work: it should directly improve the safety, capacity, configuration coherence, observability, or recovery loop that makes director burndown trustworthy.

Core does not mean broader scope. Keep the implementation narrow, verify current main before reusing prior failed PRs, and split any design fork or lower-confidence follow-up out of the core path.

WARD-TRIAGE: coherence-core label This issue is part of the `coherence-core` review set for the `warded control plane coherence` sprint. Treat it as release-spine work: it should directly improve the safety, capacity, configuration coherence, observability, or recovery loop that makes director burndown trustworthy. Core does not mean broader scope. Keep the implementation narrow, verify current `main` before reusing prior failed PRs, and split any design fork or lower-confidence follow-up out of the core path.

coilyco-ops commented

2026-07-15 07:48:11 +00:00

Author

Member

WARDED_WORKFLOW: reservation-released

release details

Run never started. ward container reap released container engineer-codex-ward-1224 (--harness codex): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the launch intent it took is retracted. Nothing is running on this issue. It needs re-dispatch. A ward agent director re-queues it automatically. A manual ward agent retry no longer needs --override-reservation.

— Codex, via ward agent

WARDED_WORKFLOW: reservation-released <details><summary>release details</summary> Run never started. `ward container reap` released container `engineer-codex-ward-1224` (`--harness codex`): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the launch intent it took is retracted. Nothing is running on this issue. It needs re-dispatch. A `ward agent director` re-queues it automatically. A manual `ward agent` retry no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-15 07:51:29 +00:00

Prevent replacement PR bodies from closing their own PR number instead of the carried issue #1344

coilyco-ops commented

2026-07-15 07:52:12 +00:00

Author

Member

QA closure note from the operations director on 2026-07-15.

I verified PR #1293 against this issue's acceptance criteria on current main:

The reservation directory is treated as disposable cache and recreated on demand (cmd/ward/agent_list.go, readAgentReservationCacheEntries).
Stale local reservation rows are pruned when durable issue-thread state does not hold a fresh reservation (TestAgentListPrunesStaleReservationCacheEntry).
Whole-folder cleanup exists as ward agent reservations clear and recreates the directory empty (cmd/ward/agent_reservation_cache.go, TestClearAgentReservationCacheDirRecreatesDirectory).
Missing reservation-cache directory recovery is covered (TestAgentListRecreatesMissingReservationCacheDir).
Docs explicitly name ~/.ward/agent-reservations as cache-only and describe whole-folder cleanup (docs/agent-reservation-cache.md, docs/agent-ops.md, docs/troubleshooting.md).

Verification run in a scratch clone of current main:

go test ./cmd/ward -run 'TestAgentListPrunesStaleReservationCacheEntry|TestClearAgentReservationCacheDirRecreatesDirectory|TestAgentListRecreatesMissingReservationCacheDir'

Result: passed.

The issue remained open because PR #1293's body used closes #1293 instead of closes #1224. I filed a follow-up to prevent that close-trailer defect from recurring and dispatched an engineer there. Closing this issue as implemented by PR #1293.

QA closure note from the operations director on 2026-07-15. I verified PR #1293 against this issue's acceptance criteria on current `main`: - The reservation directory is treated as disposable cache and recreated on demand (`cmd/ward/agent_list.go`, `readAgentReservationCacheEntries`). - Stale local reservation rows are pruned when durable issue-thread state does not hold a fresh reservation (`TestAgentListPrunesStaleReservationCacheEntry`). - Whole-folder cleanup exists as `ward agent reservations clear` and recreates the directory empty (`cmd/ward/agent_reservation_cache.go`, `TestClearAgentReservationCacheDirRecreatesDirectory`). - Missing reservation-cache directory recovery is covered (`TestAgentListRecreatesMissingReservationCacheDir`). - Docs explicitly name `~/.ward/agent-reservations` as cache-only and describe whole-folder cleanup (`docs/agent-reservation-cache.md`, `docs/agent-ops.md`, `docs/troubleshooting.md`). Verification run in a scratch clone of current `main`: ```bash go test ./cmd/ward -run 'TestAgentListPrunesStaleReservationCacheEntry|TestClearAgentReservationCacheDirRecreatesDirectory|TestAgentListRecreatesMissingReservationCacheDir' ``` Result: passed. The issue remained open because PR #1293's body used `closes #1293` instead of `closes #1224`. I filed a follow-up to prevent that close-trailer defect from recurring and dispatched an engineer there. Closing this issue as implemented by PR #1293.

coilyco-ops closed this issue

2026-07-15 07:52:15 +00:00

Rows
Columns

Make the reservation folder disposable and safe to clear wholesale #1224