Emit skill-use telemetry from reaped agent runs #873

New issue

Open

opened 2026-07-09 22:04:28 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-09 22:04:28 +00:00

Member

Follow-up from the ChatGPT dashboard skill-use graph observation.

Goal:

Ward should emit a stable, machine-readable skill-use artifact as part of the agent run drain/reap archive so downstream telemetry consumers do not scrape freeform console logs.

Why ward owns the producer:

Ward owns the agent containers, run metadata, transcript drain, and reap archive.
Existing drains already preserve console.log, transcript.jsonl, transcript.redacted.jsonl, and meta.json.
Skill use is a run-level fact that belongs beside the reaped run metadata.

Proposed first slice:

Parse skill-use events from the redacted transcript or another existing safe run artifact after drain.
Emit a small JSON artifact such as skill-usage.json beside meta.json.
Include stable dimensions: run id/container, role, harness, repo, issue ref, workflow, ward version, skill name, count, first_seen, last_seen.
Keep raw prompt/body content out of the artifact.
Update docs for the drain artifact contract.
Add tests with representative Codex/Claude transcript snippets.

Acceptance:

A completed warded run leaves a machine-readable skill-use summary in the drained archive when skills are observed.
The summary is absent or empty when no skill use is observed.
No secret or prompt-body content is copied into the summary.
Tests cover at least one transcript shape used by the current harnesses.
Run ward exec test and ward exec lint.

Downstream consumer: coilyco-flight-deck/agent-proxy should ingest this for dashboard skill-use graphs.

Follow-up from the ChatGPT dashboard skill-use graph observation. Goal: Ward should emit a stable, machine-readable skill-use artifact as part of the agent run drain/reap archive so downstream telemetry consumers do not scrape freeform console logs. Why ward owns the producer: - Ward owns the agent containers, run metadata, transcript drain, and reap archive. - Existing drains already preserve console.log, transcript.jsonl, transcript.redacted.jsonl, and meta.json. - Skill use is a run-level fact that belongs beside the reaped run metadata. Proposed first slice: - Parse skill-use events from the redacted transcript or another existing safe run artifact after drain. - Emit a small JSON artifact such as skill-usage.json beside meta.json. - Include stable dimensions: run id/container, role, harness, repo, issue ref, workflow, ward version, skill name, count, first_seen, last_seen. - Keep raw prompt/body content out of the artifact. - Update docs for the drain artifact contract. - Add tests with representative Codex/Claude transcript snippets. Acceptance: - A completed warded run leaves a machine-readable skill-use summary in the drained archive when skills are observed. - The summary is absent or empty when no skill use is observed. - No secret or prompt-body content is copied into the summary. - Tests cover at least one transcript shape used by the current harnesses. - Run ward exec test and ward exec lint. Downstream consumer: coilyco-flight-deck/agent-proxy should ingest this for dashboard skill-use graphs.

coilyco-ops added the

headless

labels

2026-07-09 22:04:28 +00:00

coilyco-ops referenced this issue from coilyco-flight-deck/agent-proxy

2026-07-09 22:04:40 +00:00

Ingest ward reap skill-use telemetry for dashboard graphs #39

coilyco-ops commented

2026-07-09 22:04:55 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-873 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T22:04:53Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#873 · branch issue-873 · harness codex · workflow direct-to-main
Run: engineer-codex-ward-873 · ward v0.522.0 · dispatched 2026-07-09T22:04:53Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.522.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-873` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T22:04:53Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#873` · branch `issue-873` · harness `codex` · workflow `direct-to-main` - **Run:** `engineer-codex-ward-873` · ward `v0.522.0` · dispatched `2026-07-09T22:04:53Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.522.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops added

and removed

labels

2026-07-10 09:00:35 +00:00

coilyco-ops added

consult

and removed

headless

labels

2026-07-15 05:34:59 +00:00