coilyco-flight-deck/ward

Watch

Ghost launch records consume repo engineer capacity #1502

New issue

Closed

opened 2026-07-22 08:53:08 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-22 08:53:08 +00:00

Member

Failed pre-agent launches are leaving cleanup-needed records that count against the per-repo engineer limit even after the Docker containers have exited. This blocked a follow-up dispatch on 2026-07-22 with: repo engineer limit is reached for coilyco-flight-deck/ward: 3 active engineer(s) (limit 3), while ward agent list --repo coilyco-flight-deck/ward simultaneously reported active engineer launches (0/12, 12 slots free) + 4 cleanup-needed records.\n\nObserved records at the time:\n\n- engineer-goose-ward-1500 for #1500\n- engineer-goose-ward-1499 for #1499\n- engineer-codex-ward-1498 for #1498\n- engineer-codex-ward-1497 for #1497\n\nThe Docker containers for those attempts had already exited. Earlier ward agent stop also rejected the records as ghost launch records and pointed at stale reservation cleanup / future stale-prelaunch override.\n\nExpected behavior:\n\n- Repo capacity should count real running launch attempts, not exited ghost records.\n- A failed-before-start launch should transition to a durable failed state and free repo capacity.\n- The cleanup path should provide a director-safe command that clears these stale holds without requiring manual Docker or cache surgery.\n- ward agent list, capacity checks, and dispatch-health should agree on the state.\n\nThis compounds the harness prompt launch failure tracked at #1499 and currently blocks dispatching follow-up work such as #1501.

Failed pre-agent launches are leaving cleanup-needed records that count against the per-repo engineer limit even after the Docker containers have exited. This blocked a follow-up dispatch on 2026-07-22 with: `repo engineer limit is reached for coilyco-flight-deck/ward: 3 active engineer(s) (limit 3)`, while `ward agent list --repo coilyco-flight-deck/ward` simultaneously reported `active engineer launches (0/12, 12 slots free) + 4 cleanup-needed records`.\n\nObserved records at the time:\n\n- `engineer-goose-ward-1500` for https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1500\n- `engineer-goose-ward-1499` for https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1499\n- `engineer-codex-ward-1498` for https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1498\n- `engineer-codex-ward-1497` for https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1497\n\nThe Docker containers for those attempts had already exited. Earlier `ward agent stop` also rejected the records as ghost launch records and pointed at stale reservation cleanup / future stale-prelaunch override.\n\nExpected behavior:\n\n- Repo capacity should count real running launch attempts, not exited ghost records.\n- A failed-before-start launch should transition to a durable failed state and free repo capacity.\n- The cleanup path should provide a director-safe command that clears these stale holds without requiring manual Docker or cache surgery.\n- `ward agent list`, capacity checks, and dispatch-health should agree on the state.\n\nThis compounds the harness prompt launch failure tracked at https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1499 and currently blocks dispatching follow-up work such as https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1501.

coilyco-ops commented

2026-07-22 08:54:16 +00:00

Author

Member

Additional observation: after filing this issue, a retry dispatch for #1501 used --override-reservation --override-capacity with --harness opencode, but the broker still rejected it as repo engineer limit is reached for coilyco-flight-deck/ward: 4 active engineer(s) (limit 3); pass --override-capacity to exceed real running capacity. The local command included the override flag, so either the override is not surviving this broker path consistently or the repo-limit check is not honoring it in this state.

Additional observation: after filing this issue, a retry dispatch for https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1501 used `--override-reservation --override-capacity` with `--harness opencode`, but the broker still rejected it as `repo engineer limit is reached for coilyco-flight-deck/ward: 4 active engineer(s) (limit 3); pass --override-capacity to exceed real running capacity`. The local command included the override flag, so either the override is not surviving this broker path consistently or the repo-limit check is not honoring it in this state.

coilyco-ops referenced this issue

2026-07-22 08:57:32 +00:00

Opencode smoke test probes Ollama tags against an OpenAI-compatible endpoint #1503

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-22 08:57:51 +00:00

Sync aosh-selected local harness models into the Ward bundle #617

coilyco-ops commented

2026-07-22 08:58:19 +00:00

Author

Member

Additional repeat on 2026-07-22: a dispatch for #1503 used WARD_SMOKE_TEST_SKIP=1, --harness opencode, --override-reservation, and --override-capacity, but the broker still refused it with repo engineer limit is reached for coilyco-flight-deck/ward: 4 active engineer(s) (limit 3); pass --override-capacity to exceed real running capacity. This is the second observed refusal where the operator command already carried the requested override flag.

Additional repeat on 2026-07-22: a dispatch for https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1503 used `WARD_SMOKE_TEST_SKIP=1`, `--harness opencode`, `--override-reservation`, and `--override-capacity`, but the broker still refused it with `repo engineer limit is reached for coilyco-flight-deck/ward: 4 active engineer(s) (limit 3); pass --override-capacity to exceed real running capacity`. This is the second observed refusal where the operator command already carried the requested override flag.

coilyco-ops referenced this issue

2026-07-22 08:59:26 +00:00

Remove embedded local harness model defaults #1504

coilyco-ops commented

2026-07-22 08:59:54 +00:00

Author

Member

Third repeat on 2026-07-22: dispatch for #1504 used WARD_SMOKE_TEST_SKIP=1, --harness opencode, --override-reservation, and --override-capacity, but the broker refused it before launch with repo engineer limit is reached for coilyco-flight-deck/ward: 4 active engineer(s) (limit 3); pass --override-capacity to exceed real running capacity. The Ward repo remains unable to accept new sibling dispatches from this director surface until the ghost capacity state is cleared or the override path is fixed.

Third repeat on 2026-07-22: dispatch for https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1504 used `WARD_SMOKE_TEST_SKIP=1`, `--harness opencode`, `--override-reservation`, and `--override-capacity`, but the broker refused it before launch with `repo engineer limit is reached for coilyco-flight-deck/ward: 4 active engineer(s) (limit 3); pass --override-capacity to exceed real running capacity`. The Ward repo remains unable to accept new sibling dispatches from this director surface until the ghost capacity state is cleared or the override path is fixed.

coilyco-ops referenced this issue

2026-07-22 09:01:32 +00:00

Brokered dispatch does not carry WARD_SMOKE_TEST_SKIP into the container #1505

coilyco-ops referenced this issue from a commit

2026-07-22 10:28:55 +00:00

fix(agent): reconcile stale launch capacity (closes #1502)

coilyco-ops closed this issue

2026-07-22 10:28:55 +00:00

coilyco-ops commented

2026-07-28 08:58:58 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run never started. ward container reap released container engineer-opencode-ward-1502 (--harness opencode): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the launch intent it took is retracted. Nothing is running on this issue. It needs re-dispatch. A ward agent director re-queues it automatically. A manual ward agent retry no longer needs --override-reservation.

— Qwen, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run never started. `ward container reap` released container `engineer-opencode-ward-1502` (`--harness opencode`): it exited without launching the agent (smoke-test death, ward#222/#264/#595), so it did no work and the launch intent it took is retracted. Nothing is running on this issue. It needs re-dispatch. A `ward agent director` re-queues it automatically. A manual `ward agent` retry no longer needs `--override-reservation`. </details>  — Qwen, via `ward agent`

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/ward#1502

No description provided.

Rows
Columns