Emit structured agent run summaries in meta.json

coilyco-ops commented

2026-07-14 07:11:05 +00:00

Member

Problem

A review of warded agent log dumps showed that meta.json currently carries run identity plus a coarse outcome, while the facts needed for review and evals are spread across console.log, transcript.jsonl, reaper text, WARD-OUTCOME comments, PR state, branch state, and CI status.

That makes later analysis depend on scraping noisy terminal output. It also makes coarse labels like ward-salvage and unknown misleading, because some runs with those labels are actually green PRs, landed work, or prelaunch failures with no transcript.

Proposed change

Have Ward emit a versioned, structured end-of-run overview in meta.json. Keep the current identity fields stable, then add a structured summary block with raw and normalized outcomes.

Candidate fields:

schema_version
workflow
started_at, ended_at, duration_seconds
raw_reap_outcome
normalized_outcome
outcome_confidence
artifacts.console_log, artifacts.transcript
git.head, git.pushed_branch, git.pushed_main, git.pr
reap.started, reap.preserved_branch, reap.standalone_salvage_issue, reap.reason
signals.ward_outcome_comment, signals.checks_green, signals.transcript_present

Optionally append one human-facing footer to console.log:

WARD-RUN-SUMMARY: outcome=<normalized_outcome> meta=meta.json transcript=<path>

The footer should be a convenience for tail, not the primary machine interface.

Acceptance criteria

meta.json is sufficient for fleet review without scraping ANSI console logs.
Raw reaper classification and normalized outcome are both preserved.
Existing consumers that read the current top-level fields keep working.
The schema is documented and versioned.
Log drain copies the final meta.json atomically with the log artifacts.
Tests cover landed-main, PR-green, blocked or failed, preserved salvage branch, and prelaunch or probe failure with no transcript.

ward#1119 is a prompt eval framework. This issue supplies the structured run artifact that evals and fleet review can consume.

### Problem A review of warded agent log dumps showed that `meta.json` currently carries run identity plus a coarse outcome, while the facts needed for review and evals are spread across `console.log`, `transcript.jsonl`, reaper text, `WARD-OUTCOME` comments, PR state, branch state, and CI status. That makes later analysis depend on scraping noisy terminal output. It also makes coarse labels like `ward-salvage` and `unknown` misleading, because some runs with those labels are actually green PRs, landed work, or prelaunch failures with no transcript. ### Proposed change Have Ward emit a versioned, structured end-of-run overview in `meta.json`. Keep the current identity fields stable, then add a structured summary block with raw and normalized outcomes. Candidate fields: - `schema_version` - `workflow` - `started_at`, `ended_at`, `duration_seconds` - `raw_reap_outcome` - `normalized_outcome` - `outcome_confidence` - `artifacts.console_log`, `artifacts.transcript` - `git.head`, `git.pushed_branch`, `git.pushed_main`, `git.pr` - `reap.started`, `reap.preserved_branch`, `reap.standalone_salvage_issue`, `reap.reason` - `signals.ward_outcome_comment`, `signals.checks_green`, `signals.transcript_present` Optionally append one human-facing footer to `console.log`: ```text WARD-RUN-SUMMARY: outcome=<normalized_outcome> meta=meta.json transcript=<path> ``` The footer should be a convenience for `tail`, not the primary machine interface. ### Acceptance criteria - `meta.json` is sufficient for fleet review without scraping ANSI console logs. - Raw reaper classification and normalized outcome are both preserved. - Existing consumers that read the current top-level fields keep working. - The schema is documented and versioned. - Log drain copies the final `meta.json` atomically with the log artifacts. - Tests cover landed-main, PR-green, blocked or failed, preserved salvage branch, and prelaunch or probe failure with no transcript. ### Related - `ward#1119` is a prompt eval framework. This issue supplies the structured run artifact that evals and fleet review can consume.

coilyco-ops added the

headless

P2

labels

2026-07-14 07:15:20 +00:00

coilysiren referenced this issue

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283

coilyco-ops added this to the warded control plane coherence milestone

2026-07-15 05:14:52 +00:00

coilyco-ops added the

coherence-core

label

2026-07-15 05:28:22 +00:00

coilyco-ops commented

2026-07-15 06:55:18 +00:00

Author

Member

Release triage note from Kai on 2026-07-15: headless carry is valid. Each agent run should leave a compact structured meta.json summary with outcome, role, issue/PR refs, timing, selected harness/model, and paths to the important artifacts.

Release triage note from Kai on 2026-07-15: headless carry is valid. Each agent run should leave a compact structured `meta.json` summary with outcome, role, issue/PR refs, timing, selected harness/model, and paths to the important artifacts.

coilyco-ops commented

2026-07-15 07:16:35 +00:00

Author

Member

WARD-TRIAGE: warded control plane coherence milestone

This issue is part of the warded control plane coherence sprint. The release thesis is to make warded feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run.

For this sprint, headless means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

WARD-TRIAGE: warded control plane coherence milestone This issue is part of the `warded control plane coherence` sprint. The release thesis is to make `warded` feel like one dependable control plane for agent work: higher safe parallelism, coherent config defaults, reliable broker/container behavior, human-feedback gates, and enough structured evidence for the next actor after a paused or failed run. For this sprint, `headless` means an engineer should be able to carry the issue from current issue context to a merged change without new human decisions. If the issue discovers a missing decision, split or demote the unclear part instead of guessing.

coilyco-ops commented

2026-07-15 07:16:48 +00:00

Author

Member

WARD-TRIAGE: coherence-core label

This issue is part of the coherence-core review set for the warded control plane coherence sprint. Treat it as release-spine work: it should directly improve the safety, capacity, configuration coherence, observability, or recovery loop that makes director burndown trustworthy.

Core does not mean broader scope. Keep the implementation narrow, verify current main before reusing prior failed PRs, and split any design fork or lower-confidence follow-up out of the core path.

WARD-TRIAGE: coherence-core label This issue is part of the `coherence-core` review set for the `warded control plane coherence` sprint. Treat it as release-spine work: it should directly improve the safety, capacity, configuration coherence, observability, or recovery loop that makes director burndown trustworthy. Core does not mean broader scope. Keep the implementation narrow, verify current `main` before reusing prior failed PRs, and split any design fork or lower-confidence follow-up out of the core path.

coilyco-ops commented

2026-07-15 08:38:08 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1234 on host 74a70515a491.

Accepted by ward agent --harness codex (reserved 2026-07-15T08:38:08Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1234 · branch issue-1234 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1234 · ward v0.710.0 · dispatched 2026-07-15T08:38:05Z
Comment thread: 3 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-15T06:55:18Z), @coilyco-ops (2026-07-15T07:16:35Z), @coilyco-ops (2026-07-15T07:16:48Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1234` on host `74a70515a491`. Accepted by `ward agent --harness codex` (reserved 2026-07-15T08:38:08Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1234` · branch `issue-1234` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1234` · ward `v0.710.0` · dispatched `2026-07-15T08:38:05Z` - **Comment thread:** 3 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-15T06:55:18Z), @coilyco-ops (2026-07-15T07:16:35Z), @coilyco-ops (2026-07-15T07:16:48Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-15 08:53:19 +00:00

Author

Member

\nWARDED_WORKFLOW: dispatch-deferred\n\n

launch failure details

\n\nRun never started. The launch accepted this reservation, then failed before container creation because was unavailable remotely and no local image existed. Docker has no container.\n\nThe hold is released for redispatch from a fresh director after the active agentic-os release/image publication completes. No implementation work was performed.\n\n

\n\n\n— Codex, via ward agent: the startup-role roster (4 roles)

warded engineer - Implements a ticket end to end.
capabilities: read + engineering
modes: A ref carries that issue detached, fire-and-forget. Freeform text files an issue first, then carries it. Detached-only - interactive work funnels to the director. Role overlays: claude{model=claude-fable-5, reasoning-effort=medium}; codex{model=gpt-5.4-mini, reasoning-effort=medium}.
docs: docs/agent-engineer.md

warded director - Opens the read-only director surface. Autonomous burndown is opt-in.
capabilities: read + project-management
modes: Attached read-only control surface over a repo's backlog (--repo scope). Use --burndown or --drain for the autonomous heartbeat. Role overlays: claude{model=claude-opus-4-8[1m], reasoning-effort=high}; codex{model=gpt-5.5, reasoning-effort=high}.
docs: docs/agent-director.md

warded advisor - Answers without writing code.
capabilities: read
modes: A ref researches the issue and posts the answer as a comment. Freeform text answers inline. Role overlays: claude{model=claude-opus-4-8[1m], reasoning-effort=high}; codex{model=gpt-5.5, reasoning-effort=high}.
docs: docs/agent-advisor.md

warded qa - Inspects a candidate and posts a structured verdict comment.
capabilities: read
modes: A ref inspects the issue, branch, pull request, and checks, then posts a structured QA verdict comment. Freeform mode is not exposed.
docs: docs/agent-qa.md

Pick a role above, or hand a bare ref straight to the engineer:
warded # e.g. warded #98 or owner/repo#N - the engineer default
warded ... # send in a specific role from the roster above
ward agent roster # reprint this roster on its own

\nWARDED_WORKFLOW: dispatch-deferred\n\n<details><summary>launch failure details</summary>\n\nRun never started. The launch accepted this reservation, then failed before container creation because was unavailable remotely and no local image existed. Docker has no container.\n\nThe hold is released for redispatch from a fresh director after the active agentic-os release/image publication completes. No implementation work was performed.\n\n</details>\n\n\n— Codex, via ward agent: the startup-role roster (4 roles) warded engineer - Implements a ticket end to end. capabilities: read + engineering modes: A ref carries that issue detached, fire-and-forget. Freeform text files an issue first, then carries it. Detached-only - interactive work funnels to the director. Role overlays: claude{model=claude-fable-5, reasoning-effort=medium}; codex{model=gpt-5.4-mini, reasoning-effort=medium}. docs: docs/agent-engineer.md warded director - Opens the read-only director surface. Autonomous burndown is opt-in. capabilities: read + project-management modes: Attached read-only control surface over a repo's backlog (`--repo` scope). Use `--burndown` or `--drain` for the autonomous heartbeat. Role overlays: claude{model=claude-opus-4-8[1m], reasoning-effort=high}; codex{model=gpt-5.5, reasoning-effort=high}. docs: docs/agent-director.md warded advisor - Answers without writing code. capabilities: read modes: A ref researches the issue and posts the answer as a comment. Freeform text answers inline. Role overlays: claude{model=claude-opus-4-8[1m], reasoning-effort=high}; codex{model=gpt-5.5, reasoning-effort=high}. docs: docs/agent-advisor.md warded qa - Inspects a candidate and posts a structured verdict comment. capabilities: read modes: A ref inspects the issue, branch, pull request, and checks, then posts a structured QA verdict comment. Freeform mode is not exposed. docs: docs/agent-qa.md Pick a role above, or hand a bare ref straight to the engineer: warded <ref> # e.g. warded #98 or owner/repo#N - the engineer default warded <role> ... # send in a specific role from the roster above ward agent roster # reprint this roster on its own

coilyco-ops commented

2026-07-15 08:53:56 +00:00

Author

Member

WARD-DIRECTOR: correction for the dispatch-deferred marker above

The machine-readable WARDED_WORKFLOW: dispatch-deferred marker above is authoritative. Its detail text was mangled by shell interpolation.

Exact evidence:

Command: warded engineer coilyco-flight-deck/ward#1234 --harness codex
Failure occurred before container creation.
forgejo.coilysiren.me/coilyco-flight-deck/agentic-os-full:latest was unavailable remotely and no local image existed.
Docker has no engineer-codex-ward-1234 container.
No implementation work was performed.

The reservation is released for redispatch from a fresh director after the active agentic-os release/image publication completes.

WARD-DIRECTOR: correction for the dispatch-deferred marker above The machine-readable `WARDED_WORKFLOW: dispatch-deferred` marker above is authoritative. Its detail text was mangled by shell interpolation. Exact evidence: - Command: `warded engineer coilyco-flight-deck/ward#1234 --harness codex` - Failure occurred before container creation. - `forgejo.coilysiren.me/coilyco-flight-deck/agentic-os-full:latest` was unavailable remotely and no local image existed. - Docker has no `engineer-codex-ward-1234` container. - No implementation work was performed. The reservation is released for redispatch from a fresh director after the active agentic-os release/image publication completes.

coilyco-ops commented

2026-07-15 08:55:31 +00:00

Author

Member

WARDED_WORKFLOW: dispatch-deferred

launch failure details

Run never started. warded engineer coilyco-flight-deck/ward#1234 --harness codex failed before container creation because forgejo.coilysiren.me/coilyco-flight-deck/agentic-os-full:latest was unavailable remotely and no local image existed. Docker has no engineer-codex-ward-1234 container. No implementation work was performed.

The reservation is released for redispatch from a fresh director after the active agentic-os release/image publication completes.

— Codex, via ward agent

WARDED_WORKFLOW: dispatch-deferred <details><summary>launch failure details</summary> Run never started. `warded engineer coilyco-flight-deck/ward#1234 --harness codex` failed before container creation because `forgejo.coilysiren.me/coilyco-flight-deck/agentic-os-full:latest` was unavailable remotely and no local image existed. Docker has no `engineer-codex-ward-1234` container. No implementation work was performed. The reservation is released for redispatch from a fresh director after the active agentic-os release/image publication completes. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-15 09:02:50 +00:00

Author

Member

WARDED_WORKFLOW: dispatch-deferred

reservation-collision details

This forwarded dispatch collided with a reservation another run still holds.

Attempted harness: codex
Attempted run: ward agent engineer coilyco-flight-deck/ward#1234 --harness codex --ward-version v0.710.0 --workflow pull-request-and-merge
Host log: /Users/kai/.ward/agent-logs/dispatch/20260715T090230Z-director-codex-md65-coilyco-flight-deck-ward-1234.log
Collision: ward agent engineer --harness codex: issue coilyco-flight-deck/ward#1234 is already reserved remotely (by @coilyco-ops at 2026-07-15T08:38:08Z); wait for it to finish or pass --override-reservation to override

Nothing new is running for this dispatch, and the live run's hold is untouched. Retry: a ward agent director heartbeat sweeps this marker and redispatches once the hold releases - the run's terminal WARDED_WORKFLOW outcome supersedes its reservation (ward#1149). A manual retry can pass --override-reservation if the collision is genuinely stale.

— Codex, via ward agent

WARDED_WORKFLOW: dispatch-deferred <details><summary>reservation-collision details</summary> This forwarded dispatch collided with a reservation another run still holds. Attempted harness: `codex` Attempted run: `ward agent engineer coilyco-flight-deck/ward#1234 --harness codex --ward-version v0.710.0 --workflow pull-request-and-merge` Host log: `/Users/kai/.ward/agent-logs/dispatch/20260715T090230Z-director-codex-md65-coilyco-flight-deck-ward-1234.log` Collision: `ward agent engineer --harness codex: issue coilyco-flight-deck/ward#1234 is already reserved remotely (by @coilyco-ops at 2026-07-15T08:38:08Z); wait for it to finish or pass --override-reservation to override` Nothing new is running for this dispatch, and the live run's hold is untouched. Retry: a `ward agent director` heartbeat sweeps this marker and redispatches once the hold releases - the run's terminal `WARDED_WORKFLOW` outcome supersedes its reservation (ward#1149). A manual retry can pass `--override-reservation` if the collision is genuinely stale. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-15 09:03:12 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-ward-1234 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-15T09:03:11Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1234 · branch issue-1234 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-ward-1234 · ward v0.710.0 · dispatched 2026-07-15T09:03:04Z
Comment thread: 7 included in the pre-flight read, 1 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-15T06:55:18Z), @coilyco-ops (2026-07-15T07:16:35Z), @coilyco-ops (2026-07-15T07:16:48Z), @coilyco-ops (2026-07-15T08:53:19Z), @coilyco-ops (2026-07-15T08:53:56Z), @coilyco-ops (2026-07-15T08:55:31Z), @coilyco-ops (2026-07-15T09:02:50Z)
- stripped: @coilyco-ops (2026-07-15T08:38:08Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-ward-1234` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-15T09:03:11Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1234` · branch `issue-1234` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-ward-1234` · ward `v0.710.0` · dispatched `2026-07-15T09:03:04Z` - **Comment thread:** 7 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-15T06:55:18Z), @coilyco-ops (2026-07-15T07:16:35Z), @coilyco-ops (2026-07-15T07:16:48Z), @coilyco-ops (2026-07-15T08:53:19Z), @coilyco-ops (2026-07-15T08:53:56Z), @coilyco-ops (2026-07-15T08:55:31Z), @coilyco-ops (2026-07-15T09:02:50Z) - stripped: @coilyco-ops (2026-07-15T08:38:08Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.710.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-15 09:23:05 +00:00

Emit structured agent run summaries in meta.json #1366

coilyco-ops closed this issue

2026-07-15 09:24:47 +00:00

coilyco-ops commented

2026-07-15 09:25:07 +00:00

Author

Member

WARDED_WORKFLOW: #1366

details

workflow: pull-request-and-merge; review summary: review gate intentionally skipped because the temporary ward default pending brokered QA.

Implementation felt narrow and mechanical: the structured summary was added at drain time without disturbing the existing top-level meta fields, and the atomically written artifacts stayed aligned with the log footer.

confidence: high

surprises: none beyond the expected lint pressure around the outcome normalization helpers.

follow-ups: none.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1366 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: review gate intentionally skipped because the temporary ward default pending brokered QA. Implementation felt narrow and mechanical: the structured summary was added at drain time without disturbing the existing top-level meta fields, and the atomically written artifacts stayed aligned with the log footer. confidence: high surprises: none beyond the expected lint pressure around the outcome normalization helpers. follow-ups: none. </details>

coilyco-ops commented

2026-07-15 09:31:38 +00:00

Author

Member

QA pass after merge #1366:

Verified the merged log-drain code preserves existing top-level meta.json fields and adds a versioned summary block with raw and normalized outcomes, workflow, timing, artifact paths, git/reap details, and outcome signals.
Verified meta.json is still written through the atomic JSON path and the drained console.log gets the tail-friendly WARD-RUN-SUMMARY footer.
Verified docs/agent-observability.md documents the summary block and footer.
Focused merged-main test passed:

go test ./cmd/ward -run 'Test(NormalizeRunSummaryOutcome|WriteDiskArtifactsAddsRunSummaryFooter|PickMetaEnvAllowlistOnly|BuildRunMetaRecordsOOMKilled|CollectFrictionEvents|ClassifyReapOutcome)$' -count=1

Hosted post-merge evidence is green as well: promote run 8411 / #2170 succeeded and release run 8412 / #2171 succeeded.

QA pass after merge #1366: - Verified the merged log-drain code preserves existing top-level `meta.json` fields and adds a versioned `summary` block with raw and normalized outcomes, workflow, timing, artifact paths, git/reap details, and outcome signals. - Verified `meta.json` is still written through the atomic JSON path and the drained `console.log` gets the tail-friendly `WARD-RUN-SUMMARY` footer. - Verified docs/agent-observability.md documents the summary block and footer. - Focused merged-main test passed: ```text go test ./cmd/ward -run 'Test(NormalizeRunSummaryOutcome|WriteDiskArtifactsAddsRunSummaryFooter|PickMetaEnvAllowlistOnly|BuildRunMetaRecordsOOMKilled|CollectFrictionEvents|ClassifyReapOutcome)$' -count=1 ``` Hosted post-merge evidence is green as well: promote run 8411 / #2170 succeeded and release run 8412 / #2171 succeeded.

Rows
Columns

Emit structured agent run summaries in meta.json #1234

Problem

Proposed change

Acceptance criteria

Related