coilyco-flight-deck/ward

Fork 0

Dispatch UX: make a failed run self-documenting on the tracker (verbose reservation comment + enriched release comment + docker-log backstop) #609

New issue

Closed

opened 2026-07-05 02:37:43 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-05 02:37:43 +00:00

Owner

Problem

When a dispatched container aborts early - most sharply at the in-container auth smoke test (ward#222) when host credentials are stale - the operator cannot tell what the run was carrying or why it died without opening raw host docker logs and already knowing which issue the container held.

Two tracker signals exist today but are thin:

The reservation comment (posted on GO) says who is carrying the issue, but not the prompt the engineer was seeded with.
The reaper's reservation-released comment says only "exited without launching the agent (smoke-test death, ward#222/#264)" - it does not name which gate failed (auth vs uv/bootstrap), the error, or how to recover.

Live example (2026-07-05): a wave across coilyco-gaming/eco-app, coilyco-flight-deck/agentic-os, and coilyco-bridge/deploy all died at the auth smoke test after a host credential went stale. The director session initially guessed the cause wrongly; the operator found the truth only by reading docker logs.

Key ordering fact

The reservation comment is posted before the auth-smoke gate (observed on agentic-os#327: reserved 02:13:22, smoke-death release 02:13:34). So a reservation comment carrying the seed context is visible even when the run dies at the gate - making the issue thread, not docker logs, the right primary surface.

Design (three surfaces, each its own job)

1. Reservation comment = WHAT (primary). Fold the per-run seed context into the reservation comment inside a collapsed <details> block (keeps re-dispatches from walling the thread). Include only the dynamic bytes:

the issue body as actually seeded, and which thread comments were included vs stripped (ward strips its own automated comments),
resolved refs (owner/repo#N, target branch), driver/model, run id, dispatch timestamp.

Do not paste the static container doctrine / seed boilerplate - it is identical every run. Reference it by version/hash instead. Scrub secrets to a higher bar than a log: this comment is durable and semi-public.

2. Reservation-released comment = WHY + RECOVER. Enrich the reaper's release comment beyond "smoke-test death": name the specific gate that failed (auth vs uv/bootstrap vs ollama-probe), include the actual error line, and give the recovery step (for auth: "refresh host claude credentials, then re-dispatch"). This is the diagnostic the operator currently goes to docker logs for.

3. Docker-log echo = BACKSTOP. Echo the same dynamic context to the container's stdout at startup, before any gate, for aborts that happen before even the reservation comment posts (image pull, container init). Delimited, greppable banner. Lower priority than 1 and 2, which cover the common case on the tracker.

Acceptance

A run that dies at the auth smoke test leaves, on the issue thread: a reservation comment with the folded seed context, and a release comment naming the gate + error + recovery. No docker logs required to diagnose.
The folded block is collapsed by default and carries only dynamic per-run bytes, not the static doctrine. Secrets are not present.
A pre-reservation abort still leaves the full context in docker logs.

Origin

coilyco-gaming/eco-app director session, 2026-07-05. Kai, after reading host docker logs to find an opaque auth-smoke failure: first "echo the pre-flight text into the docker logs," then "make the reservation comment much more verbose - include the entire pre-flight prompt?" This issue unifies both into the three-surface design above. Related: ward#222 (auth smoke gate), ward#264, the reservation/reap comment machinery.

## Problem When a dispatched container aborts early - most sharply at the **in-container auth smoke test** ([ward#222](https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/222)) when host credentials are stale - the operator cannot tell **what** the run was carrying or **why** it died without opening raw host docker logs and already knowing which issue the container held. Two tracker signals exist today but are thin: * The **reservation** comment (posted on GO) says who is carrying the issue, but not the prompt the engineer was seeded with. * The reaper's **reservation-released** comment says only "exited without launching the agent (smoke-test death, ward#222/#264)" - it does not name **which** gate failed (auth vs uv/bootstrap), the error, or how to recover. Live example (2026-07-05): a wave across coilyco-gaming/eco-app, coilyco-flight-deck/agentic-os, and coilyco-bridge/deploy all died at the auth smoke test after a host credential went stale. The director session initially guessed the cause wrongly; the operator found the truth only by reading docker logs. ## Key ordering fact The **reservation comment is posted before the auth-smoke gate** (observed on agentic-os#327: reserved `02:13:22`, smoke-death release `02:13:34`). So a reservation comment carrying the seed context is visible **even when the run dies at the gate** - making the issue thread, not docker logs, the right primary surface. ## Design (three surfaces, each its own job) **1. Reservation comment = WHAT (primary).** Fold the per-run seed context into the reservation comment inside a **collapsed `<details>` block** (keeps re-dispatches from walling the thread). Include only the **dynamic** bytes: * the issue body **as actually seeded**, and which thread comments were **included vs stripped** (ward strips its own automated comments), * resolved refs (owner/repo#N, target branch), driver/model, run id, dispatch timestamp. Do **not** paste the static container doctrine / seed boilerplate - it is identical every run. Reference it by version/hash instead. **Scrub secrets** to a higher bar than a log: this comment is durable and semi-public. **2. Reservation-released comment = WHY + RECOVER.** Enrich the reaper's release comment beyond "smoke-test death": name the **specific gate** that failed (auth vs uv/bootstrap vs ollama-probe), include the actual error line, and give the recovery step (for auth: "refresh host claude credentials, then re-dispatch"). This is the diagnostic the operator currently goes to docker logs for. **3. Docker-log echo = BACKSTOP.** Echo the same dynamic context to the container's stdout at startup, **before any gate**, for aborts that happen **before even the reservation comment posts** (image pull, container init). Delimited, greppable banner. Lower priority than 1 and 2, which cover the common case on the tracker. ## Acceptance * A run that dies at the auth smoke test leaves, **on the issue thread**: a reservation comment with the folded seed context, and a release comment naming the gate + error + recovery. No docker logs required to diagnose. * The folded block is collapsed by default and carries only dynamic per-run bytes, not the static doctrine. Secrets are not present. * A pre-reservation abort still leaves the full context in docker logs. ## Origin coilyco-gaming/eco-app director session, 2026-07-05. Kai, after reading host docker logs to find an opaque auth-smoke failure: first "echo the pre-flight text into the docker logs," then "make the reservation comment much more verbose - include the entire pre-flight prompt?" This issue unifies both into the three-surface design above. Related: [ward#222](https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/222) (auth smoke gate), [ward#264](https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/264), the reservation/reap comment machinery.

coilysiren changed title from ~~Dispatch UX: echo full preflight context (issue text + composed prompt) into the container docker logs before the auth smoke gate~~ to Dispatch UX: make a failed run self-documenting on the tracker (verbose reservation comment + enriched release comment + docker-log backstop)

2026-07-05 02:42:01 +00:00

coilyco-ops commented

2026-07-05 02:52:55 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-609 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-05T02:52:46Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-609` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-05T02:52:46Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-05 03:15:55 +00:00

feat(agent): make a failed run self-documenting on the tracker (closes #609)

coilysiren closed this issue

2026-07-05 03:15:55 +00:00

coilysiren commented

2026-07-05 03:16:30 +00:00

Author

Owner

WARD-OUTCOME: done - three-surface self-documenting dispatch landed on main (reservation seed context, gate-named release comment, docker-log echo).

Retrospective: this one was mostly about plumbing context to where it was already almost available, which is the satisfying kind of change. The key ordering fact from the issue (reservation comment posts before the auth gate) held up in the code exactly as described, so surface 1 was a clean fold into reservationCommentBody once I threaded a resolvedWork through launchAgentContainer instead of the loose ref/title/seed triple.

What fought back a little: the gate-failure handoff from the bash entrypoint to the Go reaper. They're separate processes, so I went with a small file (WARD_GATE_FAILURE_FILE) written by die() off a WARD_GATE_PHASE marker, rather than trying to smuggle it through env across the EXIT trap. It's simple and testable, and it survives the subshell-y spots in the entrypoint. The code-comments pre-commit hook (2-line contiguous cap + 90-char cap) took a few rounds to satisfy - my instinct writes 4-line doctrine comments, and this repo wants them terse with detail pushed to docs.

Confidence: high on the rendering/parsing logic (unit tests exercise every surface and the strip-partition mirrors the pre-flight predicate exactly). Medium on the live end-to-end, since I couldn't drive a real container + stale-cred death here - that path only exists with Docker + Forgejo.

Rough edges / follow-ups worth filing:

The experimental Go bootstrap path (WARD_USE_GO_BOOTSTRAP=1) never sets WARD_AGENT_LAUNCHED, so its reaper always reads Launched=false. I added gate-recording parity there, but the underlying missing flag is a pre-existing latent bug that would make a successful Go-path run post a spurious generic release comment. Not in scope here, but worth a ticket.
The seeded-body fence-neutralization and 2000-char cap are pragmatic; a body containing </details> before the fenced block is the only residual way to break the collapse, and it's cosmetic.

WARD-OUTCOME: done - three-surface self-documenting dispatch landed on main (reservation seed context, gate-named release comment, docker-log echo). Retrospective: this one was mostly about plumbing context to where it was already *almost* available, which is the satisfying kind of change. The key ordering fact from the issue (reservation comment posts before the auth gate) held up in the code exactly as described, so surface 1 was a clean fold into `reservationCommentBody` once I threaded a `resolvedWork` through `launchAgentContainer` instead of the loose ref/title/seed triple. What fought back a little: the gate-failure handoff from the bash entrypoint to the Go reaper. They're separate processes, so I went with a small file (`WARD_GATE_FAILURE_FILE`) written by `die()` off a `WARD_GATE_PHASE` marker, rather than trying to smuggle it through env across the EXIT trap. It's simple and testable, and it survives the subshell-y spots in the entrypoint. The `code-comments` pre-commit hook (2-line contiguous cap + 90-char cap) took a few rounds to satisfy - my instinct writes 4-line doctrine comments, and this repo wants them terse with detail pushed to docs. Confidence: high on the rendering/parsing logic (unit tests exercise every surface and the strip-partition mirrors the pre-flight predicate exactly). Medium on the live end-to-end, since I couldn't drive a real container + stale-cred death here - that path only exists with Docker + Forgejo. Rough edges / follow-ups worth filing: - The experimental Go bootstrap path (`WARD_USE_GO_BOOTSTRAP=1`) never sets `WARD_AGENT_LAUNCHED`, so its reaper always reads `Launched=false`. I added gate-recording parity there, but the underlying missing flag is a pre-existing latent bug that would make a *successful* Go-path run post a spurious generic release comment. Not in scope here, but worth a ticket. - The seeded-body fence-neutralization and 2000-char cap are pragmatic; a body containing `</details>` before the fenced block is the only residual way to break the collapse, and it's cosmetic.

coilysiren referenced this issue

2026-07-05 04:02:09 +00:00

Unify run outcomes: one run-lifecycle event schema, rendered to both SigNoz (deploy#23) and the forgejo comment (#609) #610

coilyco-ops referenced this issue

2026-07-08 22:19:00 +00:00

Make agent workflow defaults configurable and default engineers to PRs #707