coilyco-flight-deck/ward

Watch

Make engineer recovery forge-outage-safe: durable rescue bundles and one-command resume #1515

New issue

Closed

opened 2026-07-23 04:36:58 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-23 04:36:58 +00:00

Member

Incident

During the 2026-07-23 Forgejo disk-pressure outage, eight engineers exited while the forge returned HTTP 503. Recovery required a director to:

Identify exited containers from broker state and logs.
Copy each repository .git directory out of the stopped containers before cleanup.
Build and verify portable Git bundles for ten commits across primary and additional repositories.
Copy the bundles into host-owned storage.
Separate valid residual work from an unrelated 43-file deletion set in an additional repo.
Clear stale reservation assumptions and redispatch continuation runs with hand-written bundle paths and commit-specific instructions.

The reaper created useful residual commits for ward#1470 and ward#993, but those commits still lived only inside disposable containers. ward agent pr recover covers a remote PR state. It does not recover issue work that never reached the forge.

Related incidents and recovered lanes: #993, #1455, #1470, #1512, eco-app#112, eco-app#130, eco-app#132, and eco-app#144.

Outcome

Ward automatically preserves every issue run that has local work but cannot complete its forge landing, and a later operator can resume that exact work with one Ward command. Recovery must not require direct Docker access, manual docker cp, hand-built bundles, or reconstructing commit ids from console logs.

Durable rescue artifact

Before Ward removes an engineer container, the host/reaper must create a self-contained rescue directory when any primary or granted additional repo contains unlanded commits or residual changes.

Use a host-owned path such as ~/.ward/rescues/<run-id>/ and include:

One verified Git bundle per affected repository.
A machine-readable manifest containing issue ref, run id, repository, branch, base commit, rescued head, bundle checksum, original landing workflow, terminal failure class, validation evidence, and whether the work was committed normally or by the residual backstop.
A concise file-change inventory for human review.
Separate artifact and outcome state for each additional repository.

The artifact must survive container removal and broker restart. Creation must be idempotent. Ward must not copy working-tree credentials, opaque identifiers, ignored secret files, or arbitrary container state into the rescue directory. Reuse the existing residual commit and safety gates rather than archiving the whole workspace.

Unexpected broad deletions, generated binaries, or additional-repo changes that are not tied to the issue must be marked quarantined in the manifest. Ward must never treat a quarantined residual as automatically landable.

Document retention and an explicit prune operation. Ordinary reaping must not silently remove a rescue artifact.

Recovery command

Add an issue-level recovery surface, for example:

ward agent recover coilyco-flight-deck/ward#1470

The command must:

Find the newest applicable rescue manifest for the exact issue and repository.
Show a read-only recovery plan before mutation, including rescued commits, prerequisites, quarantined repos, and current remote state.
Refuse when a live engineer or non-stale reservation still owns the issue unless the existing explicit override contract is used.
Start or prepare a continuation on current main that consumes the rescued bundle instead of recomputing the implementation.
Preserve existing remote branches or PRs when they already contain the rescued commit.
Require explicit review of quarantined artifacts rather than blind cherry-pick.
Record the rescue artifact id in the new reservation and final workflow evidence.
Be safe to retry after another transient forge outage.

Keep this distinct from ward agent pr recover, which diagnoses a closed-unmerged remote PR. The new command recovers local issue work that may have no remote branch or PR.

Acceptance

A test simulates a forge 503 after an engineer creates a normal local commit. Container teardown leaves a verified host rescue bundle and manifest, then removes the container without losing the commit.
A test covers dirty residual work that the reaper commits before rescue.
A test covers two repositories in one run and keeps their artifacts and landing states separate.
A test injects unrelated broad deletions and proves Ward marks that repository quarantined and does not auto-land it.
A recovery test starts from a fresh clone, consumes the saved bundle, and reaches the normal branch/PR workflow without manual commit-id or path instructions.
Stale reservations are reconciled, while live ownership still fails closed.
Secrets and ignored files never enter the artifact.
ward agent list and logs show whether a run is rescued, where its manifest lives, and whether recovery was consumed.
Documentation includes the outage workflow, retention, and prune behavior.
Ward validation gates pass.

Coordination

Audit current main and #993 before changing the residual/reaper path. Reuse its harness-neutral diagnostics and fail-closed rules rather than creating a competing salvage mechanism.

## Incident During the 2026-07-23 Forgejo disk-pressure outage, eight engineers exited while the forge returned HTTP 503. Recovery required a director to: 1. Identify exited containers from broker state and logs. 2. Copy each repository `.git` directory out of the stopped containers before cleanup. 3. Build and verify portable Git bundles for ten commits across primary and additional repositories. 4. Copy the bundles into host-owned storage. 5. Separate valid residual work from an unrelated 43-file deletion set in an additional repo. 6. Clear stale reservation assumptions and redispatch continuation runs with hand-written bundle paths and commit-specific instructions. The reaper created useful residual commits for ward#1470 and ward#993, but those commits still lived only inside disposable containers. `ward agent pr recover` covers a remote PR state. It does not recover issue work that never reached the forge. Related incidents and recovered lanes: #993, #1455, #1470, #1512, eco-app#112, eco-app#130, eco-app#132, and eco-app#144. ## Outcome Ward automatically preserves every issue run that has local work but cannot complete its forge landing, and a later operator can resume that exact work with one Ward command. Recovery must not require direct Docker access, manual `docker cp`, hand-built bundles, or reconstructing commit ids from console logs. ## Durable rescue artifact Before Ward removes an engineer container, the host/reaper must create a self-contained rescue directory when any primary or granted additional repo contains unlanded commits or residual changes. Use a host-owned path such as `~/.ward/rescues/<run-id>/` and include: - One verified Git bundle per affected repository. - A machine-readable manifest containing issue ref, run id, repository, branch, base commit, rescued head, bundle checksum, original landing workflow, terminal failure class, validation evidence, and whether the work was committed normally or by the residual backstop. - A concise file-change inventory for human review. - Separate artifact and outcome state for each additional repository. The artifact must survive container removal and broker restart. Creation must be idempotent. Ward must not copy working-tree credentials, opaque identifiers, ignored secret files, or arbitrary container state into the rescue directory. Reuse the existing residual commit and safety gates rather than archiving the whole workspace. Unexpected broad deletions, generated binaries, or additional-repo changes that are not tied to the issue must be marked quarantined in the manifest. Ward must never treat a quarantined residual as automatically landable. Document retention and an explicit prune operation. Ordinary reaping must not silently remove a rescue artifact. ## Recovery command Add an issue-level recovery surface, for example: ```text ward agent recover coilyco-flight-deck/ward#1470 ``` The command must: - Find the newest applicable rescue manifest for the exact issue and repository. - Show a read-only recovery plan before mutation, including rescued commits, prerequisites, quarantined repos, and current remote state. - Refuse when a live engineer or non-stale reservation still owns the issue unless the existing explicit override contract is used. - Start or prepare a continuation on current main that consumes the rescued bundle instead of recomputing the implementation. - Preserve existing remote branches or PRs when they already contain the rescued commit. - Require explicit review of quarantined artifacts rather than blind cherry-pick. - Record the rescue artifact id in the new reservation and final workflow evidence. - Be safe to retry after another transient forge outage. Keep this distinct from `ward agent pr recover`, which diagnoses a closed-unmerged remote PR. The new command recovers local issue work that may have no remote branch or PR. ## Acceptance - A test simulates a forge 503 after an engineer creates a normal local commit. Container teardown leaves a verified host rescue bundle and manifest, then removes the container without losing the commit. - A test covers dirty residual work that the reaper commits before rescue. - A test covers two repositories in one run and keeps their artifacts and landing states separate. - A test injects unrelated broad deletions and proves Ward marks that repository quarantined and does not auto-land it. - A recovery test starts from a fresh clone, consumes the saved bundle, and reaches the normal branch/PR workflow without manual commit-id or path instructions. - Stale reservations are reconciled, while live ownership still fails closed. - Secrets and ignored files never enter the artifact. - `ward agent list` and logs show whether a run is rescued, where its manifest lives, and whether recovery was consumed. - Documentation includes the outage workflow, retention, and prune behavior. - Ward validation gates pass. ## Coordination Audit current main and #993 before changing the residual/reaper path. Reuse its harness-neutral diagnostics and fail-closed rules rather than creating a competing salvage mechanism.

coilyco-ops referenced this issue from a commit

2026-07-23 04:49:51 +00:00

feat(agent): preserve forge-outage rescue bundles

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-23 04:50:04 +00:00

Make engineer recovery forge-outage-safe #1518

coilyco-ops commented

2026-07-23 04:51:27 +00:00

Author

Member

WARDED_WORKFLOW: blocked 🛑

details

manual close/update snapshot at 2026-07-23T04:50:04Z is newer than the latest ward acknowledgement at 2026-07-23T04:37:22Z

This action is blocked until the feedback is visibly acknowledged.

— Codex, via ward agent

WARDED_WORKFLOW: blocked 🛑 <details><summary>details</summary> manual close/update snapshot at 2026-07-23T04:50:04Z is newer than the latest ward acknowledgement at 2026-07-23T04:37:22Z This action is blocked until the feedback is visibly acknowledged. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-23 04:51:56 +00:00

Author

Member

WARDED_WORKFLOW: blocked 🛑

details

human comment by @coilyco-ops at 2026-07-23T04:51:52Z is newer than the latest ward acknowledgement at 2026-07-23T04:51:28Z

This action is blocked until the feedback is visibly acknowledged.

— Codex, via ward agent

WARDED_WORKFLOW: blocked 🛑 <details><summary>details</summary> human comment by @coilyco-ops at 2026-07-23T04:51:52Z is newer than the latest ward acknowledgement at 2026-07-23T04:51:28Z This action is blocked until the feedback is visibly acknowledged. </details>  — Codex, via `ward agent`

coilyco-ops closed this issue

2026-07-23 04:52:22 +00:00

coilyco-ops commented

2026-07-23 04:52:32 +00:00

Author

Member

WARDED_WORKFLOW: #1518

details

review summary: skipped (the in-container review gate was intentionally skipped because the temporary ward default is pending brokered QA)

workflow: pull-request-and-merge; review summary: skipped (temporary ward default pending brokered QA)

The implementation felt like extending the right seam: the existing harness-neutral residual backstop stays authoritative, while host teardown makes its committed Git state durable. Confidence: medium-high. Surprise: the merge feedback guard required an explicit machine-readable acknowledgement of the newly-created PR description. Follow-up: exercise the host Docker-copy path in an integration fixture.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1518 <details><summary>details</summary> review summary: skipped (the in-container review gate was intentionally skipped because the temporary ward default is pending brokered QA) workflow: pull-request-and-merge; review summary: skipped (temporary ward default pending brokered QA) The implementation felt like extending the right seam: the existing harness-neutral residual backstop stays authoritative, while host teardown makes its committed Git state durable. Confidence: medium-high. Surprise: the merge feedback guard required an explicit machine-readable acknowledgement of the newly-created PR description. Follow-up: exercise the host Docker-copy path in an integration fixture. </details>

coilyco-ops commented

2026-07-23 04:52:40 +00:00

Author

Member

WARDED_WORKFLOW: reservation-released

release details

Run finished with WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1518.

ward container reap released container engineer-codex-ward-1515 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

Outcome summary: #1518

— Codex, via ward agent

WARDED_WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1518`. `ward container reap` released container `engineer-codex-ward-1515` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. **Outcome summary:** https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1518 </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-23 06:51:42 +00:00

Make long-lived director Forgejo authentication recoverable and persist pending briefs #1530

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/ward#1515

No description provided.

Rows
Columns