coilyco-flight-deck/agentic-os

Fork 0

feat(o11y): drain Claude Code transcripts for tool-use failures (is_error) into the failure-record buffer #249

New issue

Closed

opened 2026-06-18 07:22:16 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-06-18 07:22:16 +00:00

Owner

Problem

Claude Code agent tool-use failures need to land in the same failure-record buffer as Goose (issue A), so the o11y feed is harness-complete.

Key finding (empirical, this session)

Claude Code hooks cannot see tool failures, verified with a throwaway PreToolUse+PostToolUse probe:

PostToolUse does not fire on failure - it is success-only. A failing Bash (exit 7), a no-match grep, and a missing-file Read each produced only a PreToolUse row and no PostToolUse.
The Bash payload has no exit_code - tool_response is {stdout, stderr, interrupted, isImage, noOutputExpected}. No exit code, no success flag.
Some failures fire no hook at all - a failed Edit (old_string not found) produced zero events, not even PreToolUse. Client-validated tool errors bypass the hook pipeline.

The real source is the session transcript JSONL at ~/.claude/projects/<munged-cwd>/<session-id>.jsonl. Every tool result carries a uniform is_error: true flag across all tools, and the Bash exit code is embedded in the error text (Exit code 7). The probe confirmed all four failures (including the Edit that fired no hook) appear there - 4 errors of 21 tool results.

Scope

Transcript drainer - sweep ~/.claude/projects/**/*.jsonl, track a per-file watermark (byte offset or last-seen record id), extract records where is_error == true (across user-role tool_result content blocks), including subagent/sidechain transcripts.
Map each error to a failure-record (schema v1 from issue A): harness="claude", source="claude_transcript", failure_class derived (parse Exit code N, <tool_use_error>, file-not-found, MCP error), tool = the tool name, exit_code parsed when present, stderr_excerpt/detail from the error content, fingerprint over the same fields.
Append to the same per-repo buffer ~/.cache/agentic-os/tool-failures/<repo-slug>.jsonl. No network here (issue C ships it).
Classifier - drop expected non-zero (grep/rg no-match, test, diff, ... || true) so genuine failures are not buried. The buffer may keep everything; emission (issue C) gates on genuine.

Trigger

A SessionEnd (or Stop) ward hook fires the drain for that session, with a periodic watermark sweep as backstop. No live tailing - failure telemetry tolerates minute latency, and the watermark sweep is simpler and fully decoupled from the hot path. (Depends on ward gaining a non-PreToolUse hook surface.)

Done condition

A drain run converts new is_error transcript records into failure-records in the buffer, idempotently (watermark-respecting).
Subagent transcripts and MCP tool errors covered (verify their is_error shape on existing on-disk transcripts first).
Expected-non-zero classifier in place and unit-tested.

Depends on

Issue A (failure-record schema v1 + buffer path).

## Problem Claude Code agent tool-use failures need to land in the same failure-record buffer as Goose (issue A), so the o11y feed is harness-complete. ## Key finding (empirical, this session) Claude Code **hooks cannot see tool failures**, verified with a throwaway PreToolUse+PostToolUse probe: - **PostToolUse does not fire on failure** - it is success-only. A failing `Bash` (exit 7), a no-match `grep`, and a missing-file `Read` each produced only a PreToolUse row and no PostToolUse. - **The Bash payload has no `exit_code`** - `tool_response` is `{stdout, stderr, interrupted, isImage, noOutputExpected}`. No exit code, no success flag. - **Some failures fire no hook at all** - a failed `Edit` (old_string not found) produced zero events, not even PreToolUse. Client-validated tool errors bypass the hook pipeline. The real source is the **session transcript JSONL** at `~/.claude/projects/<munged-cwd>/<session-id>.jsonl`. Every tool result carries a uniform `is_error: true` flag across all tools, and the Bash exit code is embedded in the error text (`Exit code 7`). The probe confirmed all four failures (including the Edit that fired no hook) appear there - 4 errors of 21 tool results. ## Scope 1. **Transcript drainer** - sweep `~/.claude/projects/**/*.jsonl`, track a per-file watermark (byte offset or last-seen record id), extract records where `is_error == true` (across `user`-role `tool_result` content blocks), including subagent/sidechain transcripts. 2. **Map each error to a failure-record** (schema v1 from issue A): `harness="claude"`, `source="claude_transcript"`, `failure_class` derived (parse `Exit code N`, `<tool_use_error>`, file-not-found, MCP error), `tool` = the tool name, `exit_code` parsed when present, `stderr_excerpt`/`detail` from the error content, `fingerprint` over the same fields. 3. **Append to the same per-repo buffer** `~/.cache/agentic-os/tool-failures/<repo-slug>.jsonl`. No network here (issue C ships it). 4. **Classifier** - drop expected non-zero (`grep`/`rg` no-match, `test`, `diff`, `... || true`) so genuine failures are not buried. The buffer may keep everything; emission (issue C) gates on genuine. ## Trigger A `SessionEnd` (or `Stop`) ward hook fires the drain for that session, with a periodic watermark sweep as backstop. No live tailing - failure telemetry tolerates minute latency, and the watermark sweep is simpler and fully decoupled from the hot path. (Depends on ward gaining a non-PreToolUse hook surface.) ## Done condition - A drain run converts new `is_error` transcript records into failure-records in the buffer, idempotently (watermark-respecting). - Subagent transcripts and MCP tool errors covered (verify their `is_error` shape on existing on-disk transcripts first). - Expected-non-zero classifier in place and unit-tested. ## Depends on - Issue A (failure-record schema v1 + buffer path).

coilyco-ops added the

interactive

labels

2026-06-24 06:13:08 +00:00

coilyco-ops added

headless

and removed

interactive

labels

2026-06-25 09:10:17 +00:00

coilyco-ops commented

2026-06-25 09:51:18 +00:00

Member

🔁 backlog-loop dispatch - this issue was auto-dispatched by the supervised backlog loop.

When you finish, your final issue comment must start with exactly one of:

WARD-OUTCOME: done - landed/merged, nothing more needed.
WARD-OUTCOME: blocked - <the specific decision or information you need from a human>
WARD-OUTCOME: failed - <why, briefly>

Put your candid retrospective on the line(s) below it. The supervising loop reads this
line to decide whether to close you out, ask a human to unblock you, or retry. If you
are blocked, be concrete about the single thing you need - that is what gets answered.

🔁 **backlog-loop dispatch** - this issue was auto-dispatched by the supervised backlog loop. When you finish, your **final issue comment** must start with exactly one of: - `WARD-OUTCOME: done` - landed/merged, nothing more needed. - `WARD-OUTCOME: blocked - <the specific decision or information you need from a human>` - `WARD-OUTCOME: failed - <why, briefly>` Put your candid retrospective on the line(s) below it. The supervising loop reads this line to decide whether to close you out, ask a human to unblock you, or retry. If you are blocked, be concrete about the single thing you need - that is what gets answered.

coilyco-ops commented

2026-06-25 09:51:22 +00:00

Member

🔒 Reserved by ward agent --driver claude — container ward-agentic-os-issue-249-claude-555d9c90 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-06-25T09:51:20Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `ward-agentic-os-issue-249-claude-555d9c90` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-06-25T09:51:20Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-06-25 10:00:39 +00:00

feat(o11y): drain Claude Code transcripts for tool-use failures into the failure buffer

coilysiren closed this issue

2026-06-25 10:00:39 +00:00

coilysiren commented

2026-06-25 10:00:59 +00:00

Author

Owner

WARD-OUTCOME: done

Smoother than I expected, mostly because issue A had already laid the schema-v1 rails - I got to mirror Goose's failure-record contract rather than invent one, and reuse its _stderr_signature so fingerprints collapse the same way across harnesses.

The transcript shape fought back a little. The tool name isn't on the error record, it's on the preceding assistant tool_use block, so the drain has to carry a tool_use_id -> name/input map as it scans. And the watermark has a subtlety: I parse the whole file to build that map but only emit errors past the byte offset, otherwise a re-sweep would either miss tool names or double-write. My first truncation test was wrong (same-length rewrite never tripped the reset) - the test caught my own sloppiness before I shipped it.

Nice validation moment: the dry-run against real on-disk transcripts caught a hook-invisible Edit failure from earlier in this very session - exactly the class of failure the issue says hooks can't see. So the premise held up empirically, not just on paper.

Confidence is high on the drainer and classifier (24 focused tests, 456 in the suite green). Two honest rough edges worth a follow-up: (1) the SessionEnd/Stop hook wiring is still deferred - it depends on ward gaining a non-PreToolUse hook surface, so today this is a periodic sweep only; (2) I only had is_error: false records on disk to test the real MCP-error and Bash-exit-code shapes against - those paths are covered by synthetic fixtures built from the issue's empirical findings, not yet a live failing MCP call. Worth a quick confirmation pass once genuine failures accumulate in a buffer.

WARD-OUTCOME: done Smoother than I expected, mostly because issue A had already laid the schema-v1 rails - I got to mirror Goose's failure-record contract rather than invent one, and reuse its `_stderr_signature` so fingerprints collapse the same way across harnesses. The transcript shape fought back a little. The tool *name* isn't on the error record, it's on the preceding assistant `tool_use` block, so the drain has to carry a `tool_use_id -> name/input` map as it scans. And the watermark has a subtlety: I parse the whole file to build that map but only emit errors past the byte offset, otherwise a re-sweep would either miss tool names or double-write. My first truncation test was wrong (same-length rewrite never tripped the reset) - the test caught my own sloppiness before I shipped it. Nice validation moment: the dry-run against real on-disk transcripts caught a hook-invisible `Edit` failure from earlier in this very session - exactly the class of failure the issue says hooks can't see. So the premise held up empirically, not just on paper. Confidence is high on the drainer and classifier (24 focused tests, 456 in the suite green). Two honest rough edges worth a follow-up: (1) the `SessionEnd`/`Stop` hook wiring is still deferred - it depends on ward gaining a non-PreToolUse hook surface, so today this is a periodic sweep only; (2) I only had `is_error: false` records on disk to test the *real* MCP-error and Bash-exit-code shapes against - those paths are covered by synthetic fixtures built from the issue's empirical findings, not yet a live failing MCP call. Worth a quick confirmation pass once genuine failures accumulate in a buffer.

coilysiren referenced this issue

2026-07-04 09:17:56 +00:00

feat(o11y): create GlitchTip project + populate /sentry-dsn/tool-failures; decide failure-record producer re-homing #326

coilysiren referenced this issue

2026-07-04 09:18:19 +00:00

feat(o11y): shared GlitchTip shipper - classify, fingerprint, drain failure-record JSONL to GlitchTip (DSN from SSM) #250