Persist broker dispatch logs as first-class agent-log artifacts

coilyco-ops commented

2026-07-14 07:59:33 +00:00

Member

Problem

Broker-dispatch activity should be persisted as first-class agent-log artifacts under ~/.ward/agent-logs/, with the same discoverability and retention expectations as in-container agent runs.

A brokered dispatch can currently fail in the host-side wrapper while the requesting surface remains alive. When that happens, the useful evidence is easy to miss because it is not tied into the normal agent-log shape or lookup path for the run.

Incident Evidence

During the 2026-07-14 director incident:

The read-only director surface stayed alive, but its advertised dispatch broker stopped accepting connections.
A host wrapper log for a brokered engineer launch captured the useful error: Docker refused the launch because the target engineer container name was already in use.
The requesting director later only saw broker connection failures, not the original Docker name-conflict reason.
The audit row for the brokered launch path recorded exit status 125, but did not include the stderr tail that would have explained the Docker refusal.

Proposed Fix

Make each broker-dispatch request produce a stable artifact under ~/.ward/agent-logs/, for example under a broker/ or dispatch/ namespace that is included by the same listing and lookup commands used for in-container runs.

The artifact should include:

Request metadata: requester role, target role, issue/ref when present, harness, repo, timestamps, outcome, and error class.
Redacted argv and stderr/stdout tails for the host wrapper.
The broker request id or another stable join key that ties audit rows, wrapper logs, and any container-side drain logs together.
A redacted, user-facing summary file that is safe to paste into an issue.
Clear handling for launch-before-container cases, where no in-container console log can exist.

Acceptance Criteria

A failed brokered Docker launch leaves a discoverable artifact under ~/.ward/agent-logs/.
ward agent logs or the nearest existing log-listing command can find that artifact by issue/ref, role, or recent time.
The artifact explains launch-before-container failures without requiring direct filesystem archaeology.
The implementation reuses the same redaction/public-safety path used for existing agent logs.

ward#1077
ward#1079
ward#1240

## Problem Broker-dispatch activity should be persisted as first-class agent-log artifacts under `~/.ward/agent-logs/`, with the same discoverability and retention expectations as in-container agent runs. A brokered dispatch can currently fail in the host-side wrapper while the requesting surface remains alive. When that happens, the useful evidence is easy to miss because it is not tied into the normal agent-log shape or lookup path for the run. ## Incident Evidence During the 2026-07-14 director incident: * The read-only director surface stayed alive, but its advertised dispatch broker stopped accepting connections. * A host wrapper log for a brokered engineer launch captured the useful error: Docker refused the launch because the target engineer container name was already in use. * The requesting director later only saw broker connection failures, not the original Docker name-conflict reason. * The audit row for the brokered launch path recorded `exit status 125`, but did not include the stderr tail that would have explained the Docker refusal. ## Proposed Fix Make each broker-dispatch request produce a stable artifact under `~/.ward/agent-logs/`, for example under a `broker/` or `dispatch/` namespace that is included by the same listing and lookup commands used for in-container runs. The artifact should include: * Request metadata: requester role, target role, issue/ref when present, harness, repo, timestamps, outcome, and error class. * Redacted argv and stderr/stdout tails for the host wrapper. * The broker request id or another stable join key that ties audit rows, wrapper logs, and any container-side drain logs together. * A redacted, user-facing summary file that is safe to paste into an issue. * Clear handling for launch-before-container cases, where no in-container console log can exist. ## Acceptance Criteria * A failed brokered Docker launch leaves a discoverable artifact under `~/.ward/agent-logs/`. * `ward agent logs` or the nearest existing log-listing command can find that artifact by issue/ref, role, or recent time. * The artifact explains launch-before-container failures without requiring direct filesystem archaeology. * The implementation reuses the same redaction/public-safety path used for existing agent logs. ## Related * ward#1077 * ward#1079 * ward#1240

coilysiren referenced this issue

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283

coilyco-ops added the

headless

P2

labels

2026-07-15 04:51:18 +00:00

coilyco-ops added this to the warded control plane coherence milestone

2026-07-15 05:14:51 +00:00

coilyco-ops added the

coherence-core

label

2026-07-15 05:28:21 +00:00

coilyco-ops commented

2026-07-15 06:55:00 +00:00

Author

Member

Release triage note from Kai on 2026-07-15: headless carry is valid. Scope is first-class broker dispatch log artifacts: discoverable, redacted, linked to issue/ref/time, and useful when stdout/stderr or container output is otherwise missing.

coilyco-ops commented

2026-07-15 07:16:36 +00:00

Author

Member

WARD-TRIAGE: warded control plane coherence milestone

This issue is part of the warded control plane coherence sprint. The release thesis is to make warded feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run.

For this sprint, headless means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

WARD-TRIAGE: warded control plane coherence milestone This issue is part of the `warded control plane coherence` sprint. The release thesis is to make `warded` feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run. For this sprint, `headless` means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

coilyco-ops commented

2026-07-15 07:16:48 +00:00

Author

Member

WARD-TRIAGE: coherence-core label

This issue is part of the coherence-core review set for the warded control plane coherence sprint. Treat it as release-spine work: it should directly improve the safety, capacity, configuration coherence, observability, or recovery loop that makes director burndown trustworthy.

Core does not mean broader scope. Keep the implementation narrow, verify current main before reusing prior failed PRs, and split any design fork or lower-confidence follow-up out of the core path.

WARD-TRIAGE: coherence-core label This issue is part of the `coherence-core` review set for the `warded control plane coherence` sprint. Treat it as release-spine work: it should directly improve the safety, capacity, configuration coherence, observability, or recovery loop that makes director burndown trustworthy. Core does not mean broader scope. Keep the implementation narrow, verify current `main` before reusing prior failed PRs, and split any design fork or lower-confidence follow-up out of the core path.

coilyco-ops commented

2026-07-15 08:12:45 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1242 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-15T08:12:44Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1242 · branch issue-1242 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1242 · ward v0.710.0 · dispatched 2026-07-15T08:12:38Z
Comment thread: 3 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-15T06:55:00Z), @coilyco-ops (2026-07-15T07:16:36Z), @coilyco-ops (2026-07-15T07:16:48Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1242` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-15T08:12:44Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1242` · branch `issue-1242` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1242` · ward `v0.710.0` · dispatched `2026-07-15T08:12:38Z` - **Comment thread:** 3 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-15T06:55:00Z), @coilyco-ops (2026-07-15T07:16:36Z), @coilyco-ops (2026-07-15T07:16:48Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-15 08:30:38 +00:00

Record partial launches when a running engineer lacks an issue reservation marker #1353

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-15 08:35:53 +00:00

Persist broker dispatch artifacts #1357

coilyco-ops closed this issue

2026-07-15 08:37:19 +00:00

coilyco-ops commented

2026-07-15 08:40:33 +00:00

Author

Member

QA verification after merged PR #1357 on 2026-07-15.

Checked current origin/main (369495a36e83272141e0b25879c44d0cf8196d3a) against this issue acceptance:

Broker dispatch requests now create directory-backed artifacts under ~/.ward/agent-logs/dispatch/ with console.log, meta.json, summary.md, and redacted siblings.
Metadata includes request id, requester/target role, ref, repo, issue, harness, timestamps, outcome, error class, redacted argv, log path, and summary path.
Failed launch-before-container paths finalize artifacts with failed-before-container and preserve the host wrapper output tail after redaction.
ward agent logs resolves dispatch artifacts by issue/ref and by role after live/drained container logs are unavailable.
Docs now point operators at ~/.ward/agent-logs/dispatch/ and the shared logs lookup path.

Focused verification run:

go test ./cmd/ward -run 'DispatchArtifact|DispatchBroker|AgentLogs|BrokerDispatch|LaunchBeforeContainer|Redact|agent logs' -count=1

Result: passed. This satisfies #1242; no replacement issue needed.

QA verification after merged PR #1357 on 2026-07-15. Checked current `origin/main` (`369495a36e83272141e0b25879c44d0cf8196d3a`) against this issue acceptance: - Broker dispatch requests now create directory-backed artifacts under `~/.ward/agent-logs/dispatch/` with `console.log`, `meta.json`, `summary.md`, and redacted siblings. - Metadata includes request id, requester/target role, ref, repo, issue, harness, timestamps, outcome, error class, redacted argv, log path, and summary path. - Failed launch-before-container paths finalize artifacts with `failed-before-container` and preserve the host wrapper output tail after redaction. - `ward agent logs` resolves dispatch artifacts by issue/ref and by role after live/drained container logs are unavailable. - Docs now point operators at `~/.ward/agent-logs/dispatch/` and the shared logs lookup path. Focused verification run: ```bash go test ./cmd/ward -run 'DispatchArtifact|DispatchBroker|AgentLogs|BrokerDispatch|LaunchBeforeContainer|Redact|agent logs' -count=1 ``` Result: passed. This satisfies #1242; no replacement issue needed.

coilyco-ops referenced this issue from a commit

2026-07-15 08:42:10 +00:00

Merge issue-1242: Persist broker dispatch artifacts (closes #1242)

coilyco-ops commented

2026-07-15 08:42:50 +00:00

Author

Member

WARDED_WORKFLOW: #1357

details

workflow: pull-request-and-merge; review summary: skipped in-container because the temporary ward default pending brokered QA

Implemented first-class broker dispatch artifacts under ~/.ward/agent-logs/dispatch/ with redacted summaries, metadata, and lookup support by issue/ref, role, and recency. The change felt straightforward once the existing agent-log shape and dispatch wrappers were read together.

Confidence: high. Surprise: the broker path needed compatibility handling for the legacy flat log files so lookups stayed stable during the transition. Follow-up: none unless a downstream consumer wants a narrower listing filter.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1357 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped in-container because the temporary ward default pending brokered QA Implemented first-class broker dispatch artifacts under `~/.ward/agent-logs/dispatch/` with redacted summaries, metadata, and lookup support by issue/ref, role, and recency. The change felt straightforward once the existing agent-log shape and dispatch wrappers were read together. Confidence: high. Surprise: the broker path needed compatibility handling for the legacy flat log files so lookups stayed stable during the transition. Follow-up: none unless a downstream consumer wants a narrower listing filter. </details>

coilyco-ops referenced this issue

2026-07-15 08:46:44 +00:00

Prevent pull-request-and-merge workers from re-merging an already merged PR branch #1358

Rows
Columns

Persist broker dispatch logs as first-class agent-log artifacts #1242

Problem

Incident Evidence

Proposed Fix

Acceptance Criteria

Related