coilyco-flight-deck/ward

Fork 0

Agent-run observability: drain container logs+transcript on reap, ship redacted envelopes to SigNoz #363

New issue

Closed

opened 2026-06-26 02:05:59 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-06-26 02:05:59 +00:00

Owner

Problem

A headless ward agent container's output is effectively unqueryable after the run. Today:

Container stdout (stream-json piped to concise progress lines) + stderr land only in Docker's default json-file driver, reachable via docker logs <name>, kept by the keep-10 sweep (container.go), then gone.
The claude transcript (~/.claude/projects/**/*.jsonl under /home/ubuntu) - the genuinely valuable artifact (tool calls, args, results) - dies with the container and is never exported.

There is no way to ask "what did run X touch", "which runs failed at the merge step", or "p50 time-to-merge" after the fact.

Chosen architecture (Kai picked this)

Host-native drain + redacted SigNoz envelopes. Two slices, both built here. Reuse three patterns that already exist in the fleet:

claude-drain (agentic-os/scripts/claude_transcript.py) already sweeps ~/.claude/projects/**/*.jsonl into a host cache dir - precedent for transcript export.
The ser8 watchtower (deploy/services/ser8-observability/host/ser8-watchtower.*) - the "write to host, survive a sick cluster" host-native pattern.
Dozzle - the already-deployed host-native container-log UI (ansible role in infrastructure).

Slice 1 - host-native drain on reap (always on)

In the reaper (container_reap.go / container_reap_compute.go), before the keep-10 sweep docker rms the container, drain:

docker logs <name> (full stdout+stderr)
the transcript jsonl

to a host path. Mirror the ~/.ward/audit/<slug>.jsonl convention - suggest ~/.ward/agent-logs/<slug>/ holding console.log + transcript.jsonl + a small meta.json (issue ref, role, driver, branch, outcome: pushed-to-main vs ward-salvage). This is local, raw, ages out on its own rotation, and never leaves the host. Point Dozzle / grep / jq at it.

Ordering is load-bearing: the drain must run before any docker rm of that container, and before the reaper's own teardown. Confirm against container-cleanup.md ("why not --rm" - --rm takes the log with it) and container-reap.md.

Slice 2 - redacted envelope stream to SigNoz (export defaults OFF)

An extractor turns the transcript into one envelope per tool call - NO bodies. Fields: tool name, redacted args, cwd, duration, exit/pass-fail, lifecycle step (clone / implement / merge / push), files touched, plus run-level dims (issue ref, container slug, role, driver, outcome). Ship envelopes to the fleet SigNoz OTLP endpoint (signoz-otel-collector...:4318 HTTP, cross-cluster ser8:4318) as structured logs.

This slice defines a first-of-pattern schema. Ground it against the field contract in deploy/services/ser8-observability/docs/log-schema.md (bounded indexed attributes, unbounded ids stay in body) and the o11y overview in deploy/docs/o11y.md.

The crux - redaction must happen at extraction, before export

The fleet's existing secret-redaction (the Warp regex list in agentic-os/docs/features-shell-secrets.md covering AWS / GitHub / Anthropic / Slack / JWT / etc.) scrubs the terminal surface, not the transcript jsonl. And SigNoz has no ingest redaction processor - redaction is enforced upstream of the sink, never at it. So:

Tool results / bodies carry the highest cred risk (a Read of a .env, a command echoing a token). Drop them entirely from the envelope stream - envelopes are call-metadata only.
Apply the Warp redaction regex list to the args before they go into an envelope.
The OTLP export defaults OFF behind an opt-in (e.g. WARD_AGENT_TELEMETRY=1). The extractor and shipper are built and unit-tested, but nothing flows into the 90-day indexed store until Kai has reviewed the redaction. The host-drain (slice 1) is always-on; only the external export is gated.

Downstream - FILE these as follow-ups (do not carry them)

This run is ward-only. Before exit, file two linked follow-up issues (you have FORGEJO access in-container):

coilyco-bridge/deploy - SigNoz ingest pipeline for the envelope stream (deploy/services/signoz-pipelines/), a dashboard, and a log-schema.md entry for the new fields. Blocked on this issue's envelope schema + Kai's redaction review.
coilyco-flight-deck/infrastructure - point the existing Dozzle ansible role at ~/.ward/agent-logs/ (or document the grep/jq query path), and note that the stale roles/claude-hooks/templates/o2r-nudge.sh.j2 hook is unrelated cleanup.

Read before the first edit (front-load)

ward/docs/container-cleanup.md, container-reap.md, audit.md (the ~/.ward/ layout), container.md
ward/cmd/ward/container_reap.go, container_reap_compute.go, container.go (keep-10 sweep), containerassets/entrypoint.sh
agentic-os/scripts/claude_transcript.py (claude-drain precedent)
agentic-os/docs/features-shell-secrets.md (the redaction regex list to reuse)
deploy/docs/o11y.md, deploy/services/ser8-observability/docs/log-schema.md (the schema contract)

Done means

Reaper drains console + transcript to ~/.ward/agent-logs/<slug>/ before keep-10, always on, with a test asserting drain precedes docker rm.
Envelope extractor + redaction + OTLP shipper built and unit-tested, export default-OFF.
ward/docs/FEATURES.md updated; a docs/<feature>.md walkthrough added and linked.
Two downstream follow-up issues filed (deploy, infrastructure).
Landed on main, green.

## Problem A headless `ward agent` container's output is effectively unqueryable after the run. Today: - Container stdout (stream-json piped to concise progress lines) + stderr land **only** in Docker's default `json-file` driver, reachable via `docker logs <name>`, kept by the **keep-10** sweep (`container.go`), then gone. - The claude **transcript** (`~/.claude/projects/**/*.jsonl` under `/home/ubuntu`) - the genuinely valuable artifact (tool calls, args, results) - **dies with the container** and is never exported. There is no way to ask "what did run X touch", "which runs failed at the merge step", or "p50 time-to-merge" after the fact. ## Chosen architecture (Kai picked this) **Host-native drain + redacted SigNoz envelopes.** Two slices, both built here. Reuse three patterns that already exist in the fleet: - `claude-drain` (`agentic-os/scripts/claude_transcript.py`) already sweeps `~/.claude/projects/**/*.jsonl` into a host cache dir - precedent for transcript export. - The ser8 **watchtower** (`deploy/services/ser8-observability/host/ser8-watchtower.*`) - the "write to host, survive a sick cluster" host-native pattern. - **Dozzle** - the already-deployed host-native container-log UI (ansible role in `infrastructure`). ### Slice 1 - host-native drain on reap (always on) In the reaper (`container_reap.go` / `container_reap_compute.go`), before the keep-10 sweep `docker rm`s the container, drain: - `docker logs <name>` (full stdout+stderr) - the transcript jsonl to a host path. Mirror the `~/.ward/audit/<slug>.jsonl` convention - suggest `~/.ward/agent-logs/<slug>/` holding `console.log` + `transcript.jsonl` + a small `meta.json` (issue ref, role, driver, branch, outcome: pushed-to-main vs ward-salvage). This is local, raw, ages out on its own rotation, and never leaves the host. Point Dozzle / `grep` / `jq` at it. **Ordering is load-bearing:** the drain must run before any `docker rm` of that container, and before the reaper's own teardown. Confirm against `container-cleanup.md` ("why not --rm" - `--rm` takes the log with it) and `container-reap.md`. ### Slice 2 - redacted envelope stream to SigNoz (export defaults OFF) An extractor turns the transcript into one **envelope per tool call** - NO bodies. Fields: tool name, **redacted** args, cwd, duration, exit/pass-fail, lifecycle step (clone / implement / merge / push), files touched, plus run-level dims (issue ref, container slug, role, driver, outcome). Ship envelopes to the fleet SigNoz OTLP endpoint (`signoz-otel-collector...:4318` HTTP, cross-cluster `ser8:4318`) as structured logs. **This slice defines a first-of-pattern schema. Ground it** against the field contract in `deploy/services/ser8-observability/docs/log-schema.md` (bounded indexed attributes, unbounded ids stay in body) and the o11y overview in `deploy/docs/o11y.md`. ## The crux - redaction must happen at extraction, before export The fleet's existing secret-redaction (the Warp regex list in `agentic-os/docs/features-shell-secrets.md` covering AWS / GitHub / Anthropic / Slack / JWT / etc.) scrubs the **terminal surface, not the transcript jsonl**. And SigNoz has **no ingest redaction processor** - redaction is enforced upstream of the sink, never at it. So: - Tool **results / bodies** carry the highest cred risk (a Read of a `.env`, a command echoing a token). **Drop them entirely** from the envelope stream - envelopes are call-metadata only. - Apply the Warp redaction regex list to the **args** before they go into an envelope. - **The OTLP export defaults OFF** behind an opt-in (e.g. `WARD_AGENT_TELEMETRY=1`). The extractor and shipper are built and unit-tested, but nothing flows into the 90-day indexed store until Kai has reviewed the redaction. The host-drain (slice 1) is always-on; only the external export is gated. ## Downstream - FILE these as follow-ups (do not carry them) This run is ward-only. Before exit, file two linked follow-up issues (you have FORGEJO access in-container): - **`coilyco-bridge/deploy`** - SigNoz ingest pipeline for the envelope stream (`deploy/services/signoz-pipelines/`), a dashboard, and a `log-schema.md` entry for the new fields. Blocked on this issue's envelope schema + Kai's redaction review. - **`coilyco-flight-deck/infrastructure`** - point the existing Dozzle ansible role at `~/.ward/agent-logs/` (or document the `grep`/`jq` query path), and note that the stale `roles/claude-hooks/templates/o2r-nudge.sh.j2` hook is unrelated cleanup. ## Read before the first edit (front-load) - `ward/docs/container-cleanup.md`, `container-reap.md`, `audit.md` (the `~/.ward/` layout), `container.md` - `ward/cmd/ward/container_reap.go`, `container_reap_compute.go`, `container.go` (keep-10 sweep), `containerassets/entrypoint.sh` - `agentic-os/scripts/claude_transcript.py` (claude-drain precedent) - `agentic-os/docs/features-shell-secrets.md` (the redaction regex list to reuse) - `deploy/docs/o11y.md`, `deploy/services/ser8-observability/docs/log-schema.md` (the schema contract) ## Done means - Reaper drains console + transcript to `~/.ward/agent-logs/<slug>/` before keep-10, always on, with a test asserting drain precedes `docker rm`. - Envelope extractor + redaction + OTLP shipper built and unit-tested, export default-OFF. - `ward/docs/FEATURES.md` updated; a `docs/<feature>.md` walkthrough added and linked. - Two downstream follow-up issues filed (deploy, infrastructure). - Landed on `main`, green.

coilysiren commented

2026-06-26 02:06:22 +00:00

Author

Owner

🔒 Reserved by ward agent --driver claude — container ward-ward-issue-363-claude-6747ed9c on host ceb793dcd76d is carrying this issue (reserved 2026-06-26T02:06:21Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `ward-ward-issue-363-claude-6747ed9c` on host `ceb793dcd76d` is carrying this issue (reserved 2026-06-26T02:06:21Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-06-26 02:29:35 +00:00

feat(agent): drain container logs+transcript on reap, redacted SigNoz envelopes (ward#363)

coilysiren closed this issue

2026-06-26 02:29:35 +00:00

coilysiren referenced this issue from coilyco-flight-deck/infrastructure

2026-06-26 02:30:23 +00:00

Point Dozzle (or document grep/jq path) at ~/.ward/agent-logs/ for ward agent-run drains (ward#363) #413

coilysiren commented

2026-06-26 02:30:58 +00:00

Author

Owner

The load-bearing surprise was that the issue's prose says "in the reaper," but the reaper runs inside the container with no docker socket - it can never call docker logs. So slice 1 had to be host-side, folded into the keep-10 sweep right before the docker rm. Once that clicked, the ordering-is-the-contract framing made a pure sweepActions planner the obvious testable seam. The other genuinely security-shaped bit was meta.json: docker inspect exposes the whole Config.Env, including the --env-file secrets, so the allowlist isn't a nicety - it's the boundary, and it has a test that asserts the token never escapes.

What fought back was mostly the gate, not the code. FEATURES.md was sitting exactly at the 4000-char cap, so a new bullet meant clawing back space from existing ones; the walkthrough doc and every comment block got trimmed hard to fit the 80-line / 2-line-comment rules. This container also shipped without trufflehog or golangci-lint, so I fetched the trufflehog binary to clear the commit hook rather than reach for --no-verify.

Confidence: high on the pure, tested logic - drain ordering, the redaction list, body-drop, the env allowlist, envelope extraction all have unit tests. Lower on the side-effecting docker path (logs/cp/inspect), which I couldn't integration-test with no docker daemon in here; it's reasoned through but unexercised. Rough edges worth a follow-up: the in-container transcript path is hardcoded to /home/ubuntu/.claude/projects, and the docker-drain path deserves an integration test on a host that has a daemon. The export stays default-off by design until Kai reviews the redaction.

WARD-OUTCOME: done - host-native drain (console+transcript+meta) before keep-10 rm, plus a default-off redacted OTLP envelope shipper; landed on main, downstream issues filed (deploy#23, infrastructure#413). The load-bearing surprise was that the issue's prose says "in the reaper," but the reaper runs *inside* the container with no docker socket - it can never call `docker logs`. So slice 1 had to be host-side, folded into the keep-10 sweep right before the `docker rm`. Once that clicked, the ordering-is-the-contract framing made a pure `sweepActions` planner the obvious testable seam. The other genuinely security-shaped bit was meta.json: `docker inspect` exposes the whole `Config.Env`, including the `--env-file` secrets, so the allowlist isn't a nicety - it's the boundary, and it has a test that asserts the token never escapes. What fought back was mostly the gate, not the code. FEATURES.md was sitting *exactly* at the 4000-char cap, so a new bullet meant clawing back space from existing ones; the walkthrough doc and every comment block got trimmed hard to fit the 80-line / 2-line-comment rules. This container also shipped without trufflehog or golangci-lint, so I fetched the trufflehog binary to clear the commit hook rather than reach for --no-verify. Confidence: high on the pure, tested logic - drain ordering, the redaction list, body-drop, the env allowlist, envelope extraction all have unit tests. Lower on the side-effecting docker path (logs/cp/inspect), which I couldn't integration-test with no docker daemon in here; it's reasoned through but unexercised. Rough edges worth a follow-up: the in-container transcript path is hardcoded to `/home/ubuntu/.claude/projects`, and the docker-drain path deserves an integration test on a host that has a daemon. The export stays default-off by design until Kai reviews the redaction.

coilysiren referenced this issue

2026-07-05 04:02:09 +00:00

Unify run outcomes: one run-lifecycle event schema, rendered to both SigNoz (deploy#23) and the forgejo comment (#609) #610