coilyco-flight-deck/ward

Watch

Make long-lived director Forgejo authentication recoverable and persist pending briefs #1530

New issue

Closed

opened 2026-07-23 06:51:41 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-23 06:51:41 +00:00

Member

Incident

On 2026-07-23, a long-lived read-only director experienced repeated failure of its injected Forgejo credential:

Forgejo returned 401 Unauthorized: access token does not exist.
Authentication failed around 05:04:13, recovered without restarting by 05:04:42, and worked through at least 06:12:17.
Authentication failed again from 06:29:38 onward.
The public Forgejo version endpoint remained healthy.
Authenticated reads, writes, and freeform issue creation all failed.
Brokered listing and independently launched issue-bound runs continued working.
Capture-and-dispatch briefs were stranded inside the ephemeral director because they could not be persisted as issues.

Closed #1521 restored brokered Forgejo access after the Go-bootstrap migration and requested rotation coverage. This later incident proves that invalid-after-launch credential behavior and capture durability still need an explicit end-to-end contract. Closed #1515 provides related durable rescue patterns for engineer Git work, but does not preserve unfiled director briefs.

Required behavior

Detect an invalid or revoked Forgejo credential promptly and surface a clear, visible degraded-auth state. Distinguish authentication failure from Forgejo availability, authorization denial, and network failure.
Define secure refresh or replacement behavior for long-lived directors. The dropped director must acquire current authority through the root broker or be recycled deliberately without reading, printing, or storing the secret.
Fail loudly when capture-dependent workflows cannot persist work. Do not let a director claim capture or dispatch succeeded when issue creation failed.
Durably retain each pending issue brief outside the ephemeral director until it is persisted.
Replay each retained brief exactly once after authentication recovers, with a stable idempotency identity and crash-safe state transitions.
Resume dispatch only after the corresponding issue identity is durable. Retries must not create duplicate issues or duplicate sibling runs.
Expose pending, replayed, and terminally failed capture state through an operator-readable Ward surface and logs.
Add coverage for a credential that is valid at launch and becomes invalid later, including repeated invalid and recovered periods.

Security constraints

Do not reuse engineer push credentials.
Do not expose token values in environment dumps, argv, logs, audit records, pending-brief storage, or agent-visible files.
Do not weaken per-run credential isolation, the read-only clone boundary, broker role authorization, or repository scope.
Do not turn transient authentication failure into broader standing authority.

Acceptance scenarios

Token A works at launch, becomes invalid, and token B becomes current. The director detects the failure, broker-mediated recovery restores allowed operations, and the agent never handles either secret.
Issue creation fails during the invalid window. The full brief survives director/container loss and is replayed once after recovery.
A crash occurs between remote issue creation and local acknowledgement. Recovery reconciles by idempotency identity and does not create a second issue.
Dispatch occurs once and only after the issue reference is durably recorded.
Authenticated reads, writes, and capture-dependent workflow status fail visibly while the public health endpoint remains green.
Engineer push credentials and non-director credential behavior remain isolated and unchanged.

## Incident On 2026-07-23, a long-lived read-only director experienced repeated failure of its injected Forgejo credential: - Forgejo returned `401 Unauthorized: access token does not exist`. - Authentication failed around 05:04:13, recovered without restarting by 05:04:42, and worked through at least 06:12:17. - Authentication failed again from 06:29:38 onward. - The public Forgejo version endpoint remained healthy. - Authenticated reads, writes, and freeform issue creation all failed. - Brokered listing and independently launched issue-bound runs continued working. - Capture-and-dispatch briefs were stranded inside the ephemeral director because they could not be persisted as issues. Closed #1521 restored brokered Forgejo access after the Go-bootstrap migration and requested rotation coverage. This later incident proves that invalid-after-launch credential behavior and capture durability still need an explicit end-to-end contract. Closed #1515 provides related durable rescue patterns for engineer Git work, but does not preserve unfiled director briefs. ## Required behavior - Detect an invalid or revoked Forgejo credential promptly and surface a clear, visible degraded-auth state. Distinguish authentication failure from Forgejo availability, authorization denial, and network failure. - Define secure refresh or replacement behavior for long-lived directors. The dropped director must acquire current authority through the root broker or be recycled deliberately without reading, printing, or storing the secret. - Fail loudly when capture-dependent workflows cannot persist work. Do not let a director claim capture or dispatch succeeded when issue creation failed. - Durably retain each pending issue brief outside the ephemeral director until it is persisted. - Replay each retained brief exactly once after authentication recovers, with a stable idempotency identity and crash-safe state transitions. - Resume dispatch only after the corresponding issue identity is durable. Retries must not create duplicate issues or duplicate sibling runs. - Expose pending, replayed, and terminally failed capture state through an operator-readable Ward surface and logs. - Add coverage for a credential that is valid at launch and becomes invalid later, including repeated invalid and recovered periods. ## Security constraints - Do not reuse engineer push credentials. - Do not expose token values in environment dumps, argv, logs, audit records, pending-brief storage, or agent-visible files. - Do not weaken per-run credential isolation, the read-only clone boundary, broker role authorization, or repository scope. - Do not turn transient authentication failure into broader standing authority. ## Acceptance scenarios 1. Token A works at launch, becomes invalid, and token B becomes current. The director detects the failure, broker-mediated recovery restores allowed operations, and the agent never handles either secret. 2. Issue creation fails during the invalid window. The full brief survives director/container loss and is replayed once after recovery. 3. A crash occurs between remote issue creation and local acknowledgement. Recovery reconciles by idempotency identity and does not create a second issue. 4. Dispatch occurs once and only after the issue reference is durably recorded. 5. Authenticated reads, writes, and capture-dependent workflow status fail visibly while the public health endpoint remains green. 6. Engineer push credentials and non-director credential behavior remain isolated and unchanged.

coilyco-ops commented

2026-07-23 06:52:29 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1530 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-23T06:52:28Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1530 · branch issue-1530 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1530 · ward v0.793.0 · dispatched 2026-07-23T06:52:09Z
Reservation: held
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.793.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1530` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-23T06:52:28Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1530` · branch `issue-1530` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1530` · ward `v0.793.0` · dispatched `2026-07-23T06:52:09Z` - **Reservation:** held - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.793.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-28 02:25:02 +00:00

Stabilize Ward for safe autonomous burndown and verified releases #1574

coilyco-ops referenced this issue

2026-07-28 02:47:45 +00:00

Stabilize Ward for safe autonomous burndown and verified releases #1574

coilyco-ops added the

autonomy

live-collab

label

2026-07-28 03:23:31 +00:00

coilyco-ops commented

2026-07-28 03:29:02 +00:00

Author

Member

Disposition checkpoint for parent #1574. The 2026-07-23 reservation is stale, and the current broker reports no running engineer or launch intent for this issue. The issue remains valid and is now interactive because its credential-recovery acceptance requires operator evidence. A director read-only surface must run ward agent stop coilyco-flight-deck/ward#1530 to clear the stale issue reservation. Redispatch should wait until the exposed Forgejo bot token is rotated under coilyco-bridge/agentic-os-kai#776. Ward transcript hardening is tracked separately at #1582.

Disposition checkpoint for parent #1574. The 2026-07-23 reservation is stale, and the current broker reports no running engineer or launch intent for this issue. The issue remains valid and is now interactive because its credential-recovery acceptance requires operator evidence. A director read-only surface must run ward agent stop coilyco-flight-deck/ward#1530 to clear the stale issue reservation. Redispatch should wait until the exposed Forgejo bot token is rotated under https://forgejo.coilysiren.me/coilyco-bridge/agentic-os-kai/issues/776. Ward transcript hardening is tracked separately at #1582.

coilyco-ops referenced this issue

2026-07-28 20:05:30 +00:00

Keep Forgejo credentials behind the native director broker #1612

coilyco-ops added the

priority

label

2026-07-29 03:47:17 +00:00

coilyco-ops closed this issue

2026-07-29 06:04:43 +00:00