coilyco-flight-deck/ward

Archived

Watch

Emit skill-use telemetry from reaped agent runs #873

Closed

opened 2026-07-09 22:04:28 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-09 22:04:28 +00:00

Owner

Follow-up from the ChatGPT dashboard skill-use graph observation.

Goal:

Ward should emit a stable, machine-readable skill-use artifact as part of the agent run drain/reap archive so downstream telemetry consumers do not scrape freeform console logs.

Why ward owns the producer:

Ward owns the agent containers, run metadata, transcript drain, and reap archive.
Existing drains already preserve console.log, transcript.jsonl, transcript.redacted.jsonl, and meta.json.
Skill use is a run-level fact that belongs beside the reaped run metadata.

Proposed first slice:

Parse skill-use events from the redacted transcript or another existing safe run artifact after drain.
Emit a small JSON artifact such as skill-usage.json beside meta.json.
Include stable dimensions: run id/container, role, harness, repo, issue ref, workflow, ward version, skill name, count, first_seen, last_seen.
Keep raw prompt/body content out of the artifact.
Update docs for the drain artifact contract.
Add tests with representative Codex/Claude transcript snippets.

Acceptance:

A completed warded run leaves a machine-readable skill-use summary in the drained archive when skills are observed.
The summary is absent or empty when no skill use is observed.
No secret or prompt-body content is copied into the summary.
Tests cover at least one transcript shape used by the current harnesses.
Run ward exec test and ward exec lint.

Downstream consumer: coilyco-flight-deck/agent-proxy should ingest this for dashboard skill-use graphs.

Follow-up from the ChatGPT dashboard skill-use graph observation. Goal: Ward should emit a stable, machine-readable skill-use artifact as part of the agent run drain/reap archive so downstream telemetry consumers do not scrape freeform console logs. Why ward owns the producer: - Ward owns the agent containers, run metadata, transcript drain, and reap archive. - Existing drains already preserve console.log, transcript.jsonl, transcript.redacted.jsonl, and meta.json. - Skill use is a run-level fact that belongs beside the reaped run metadata. Proposed first slice: - Parse skill-use events from the redacted transcript or another existing safe run artifact after drain. - Emit a small JSON artifact such as skill-usage.json beside meta.json. - Include stable dimensions: run id/container, role, harness, repo, issue ref, workflow, ward version, skill name, count, first_seen, last_seen. - Keep raw prompt/body content out of the artifact. - Update docs for the drain artifact contract. - Add tests with representative Codex/Claude transcript snippets. Acceptance: - A completed warded run leaves a machine-readable skill-use summary in the drained archive when skills are observed. - The summary is absent or empty when no skill use is observed. - No secret or prompt-body content is copied into the summary. - Tests cover at least one transcript shape used by the current harnesses. - Run ward exec test and ward exec lint. Downstream consumer: coilyco-flight-deck/agent-proxy should ingest this for dashboard skill-use graphs.

coilyco-ops added the

autonomy

headless

priority

labels

2026-07-09 22:04:28 +00:00

coilyco-ops referenced this issue from coilyco-flight-deck/agent-proxy

2026-07-09 22:04:40 +00:00

Ingest ward reap skill-use telemetry for dashboard graphs #39

coilyco-ops added

priority

and removed

priority

labels

2026-07-10 09:00:35 +00:00

coilyco-ops added

and removed

labels

2026-07-15 05:34:59 +00:00

coilyco-ops referenced this issue from a commit

2026-07-29 16:22:20 +00:00

feat(agent): emit drained skill-use telemetry

coilyco-ops closed this issue

2026-07-29 16:22:20 +00:00

coilyco-ops commented

2026-07-29 16:33:35 +00:00

Author

Owner

WARD-WORKFLOW: done ✅

details

Landed secret-free skill-usage.json drain telemetry for Claude and Codex at feature commit b70197b0, merge d68eb40c; ward exec test, lint, vet, and pre-commit passed. Forgejo promote/release are green; published v0.858.0.

workflow: merge-remote-main; review summary: skipped — the in-container review gate was intentionally skipped because implementation and QA are role-separated; QA is a separate, opt-in exact-commit verification role bound to exact candidate commit b70197b0.

Felt: clean fit in the drain pipeline; Codex path-based observation was the subtle edge. Confidence: high. Surprise: the published ARM64 asset checksum passed, but this container keeps /usr/local/bin/ward root-owned and /opt/ward read-only, so host binary refresh remains. Follow-ups: agent-proxy ingestion; optional exact-commit QA.

— Codex, via ward agent

WARD-WORKFLOW: done ✅ <details><summary>details</summary> Landed secret-free `skill-usage.json` drain telemetry for Claude and Codex at feature commit `b70197b0`, merge `d68eb40c`; `ward exec test`, `lint`, `vet`, and pre-commit passed. Forgejo promote/release are green; published `v0.858.0`. workflow: merge-remote-main; review summary: skipped — the in-container review gate was intentionally skipped because implementation and QA are role-separated; QA is a separate, opt-in exact-commit verification role bound to exact candidate commit `b70197b0`. Felt: clean fit in the drain pipeline; Codex path-based observation was the subtle edge. Confidence: high. Surprise: the published ARM64 asset checksum passed, but this container keeps `/usr/local/bin/ward` root-owned and `/opt/ward` read-only, so host binary refresh remains. Follow-ups: agent-proxy ingestion; optional exact-commit QA.  — Codex, via `ward agent` </details>

coilyco-ops commented

2026-07-29 16:33:53 +00:00

Author

Owner

WARD-WORKFLOW: reservation-released

release details

Run finished with WARD-WORKFLOW: done ✅.

ward container reap released container engineer-codex-ward-873 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARD-WORKFLOW: done ✅`. `ward container reap` released container `engineer-codex-ward-873` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

Commenting is not possible because the repository is archived.

No milestone

No project

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/ward#873

No description provided.

Rows
Columns