feat(goose): instrument goose_json.ask() failures + triage failure counter (failure-record schema v1) #248

New issue

Closed

opened 2026-06-18 07:22:10 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-18 07:22:10 +00:00

Owner

Problem

Goose's tool-use error rate is the motivating pain, and it is currently unmeasurable by construction. Two layers of masking:

Layer 1 - the wrapper discards the evidence. scripts/goose_json.py ask() is the single boundary every triage call passes through. It runs:

out = subprocess.run(GOOSE_BASE + ["--recipe", path],
                     capture_output=True, text=True,
                     timeout=timeout).stdout

It captures .stderr and .returncode and throws both away, keeping only .stdout. Every distinct failure (provider 5xx, rate-limit, panic, malformed envelope, schema-invalid reply, timeout) collapses into one undifferentiated return None. GOOSE_BASE also passes --no-session, so none of it lands in ~/.local/share/goose/sessions/sessions.db either.

Layer 2 - triage launders the None into a plausible default. scripts/goose-triage.py fail-softs every None: "unscored -> default 30", MODE_DEFAULT = "consult", runoff failure keeps issue order. A failed model call surfaces as a wrong-but-valid label, not a failure. The cached report's failed: 0 measures label-write success, not Goose-call success.

Why this issue is the foundation

Goose here is already 100% ward-routed (every call is a ward-exec'd subprocess), and the failure data is already in hand at one Python function. The fix is to stop discarding .returncode / .stderr. This issue also defines the failure-record schema v1 that the Claude-transcript drain (issue B) and the GlitchTip shipper (issue C) both consume.

Scope

Instrument goose_json.ask(). Keep .returncode and .stderr. Classify each failure into the taxonomy already latent in the code:
- timeout (TimeoutExpired)
- nonzero_exit (returncode != 0 - currently ignored; likely where provider / rate-limit / panic errors hide in Goose's stderr)
- bad_envelope (JSONDecodeError on the envelope)
- no_schema_valid_message (envelope parsed, nothing satisfied the schema)
- exhausted_retries (all N attempts failed)
Append a failure-record (one JSON line per genuine failure) to a per-repo buffer at ~/.cache/agentic-os/tool-failures/<repo-slug>.jsonl. No network in this issue - local buffer only.
Stop the silent laundering in goose-triage. The fail-soft defaults stay as resilience, but the substitution must be counted: add a goose_failures block to the report, per pass (P0-confirm / urgency / runoff / mode), separate from the label-write failed.

failure-record schema v1 (proposed)

One JSON object per line:

ts               int     unix seconds
harness          str     "goose"
source           str     "goose_json.ask"
repo             str     repo slug
failure_class    str     one of the taxonomy above
schema_title     str     recipe/schema title (the goose "tool" analog)
exit_code        int?    Goose process returncode (null on timeout)
attempt          int     which retry (0-based)
stderr_excerpt   str     truncated Goose stderr (the discarded evidence)
detail           str     short message
fingerprint      str     hash of (harness, failure_class, schema_title, stderr-signature)

fingerprint collapses a flood of identical failures into one counted bucket - the field that finally answers "which failure mode dominates."

Done condition

Every ask() failure path writes one classified failure-record to the buffer.
goose-triage report carries a real goose_failures count per pass.
Existing behavior (fail-soft defaults, exit codes, stdout contract) unchanged.
ward exec test green; the failure-record schema documented in docs/goose-triage.md.

Out of scope (separate issues)

Claude Code transcript drain (issue B).
Shipping the buffer to GlitchTip (issue C).
sessions.db drain for the interactive Goose slice.

Origin: design session 2026-06-18. Hooks were empirically ruled out as a failure source (Claude Code PostToolUse does not fire on failure and carries no exit code); the harness-specific sources are the transcript (Claude) and the wrapper boundary (Goose).

## Problem Goose's tool-use error rate is the motivating pain, and it is currently **unmeasurable by construction**. Two layers of masking: **Layer 1 - the wrapper discards the evidence.** `scripts/goose_json.py` `ask()` is the single boundary every triage call passes through. It runs: ```python out = subprocess.run(GOOSE_BASE + ["--recipe", path], capture_output=True, text=True, timeout=timeout).stdout ``` It captures `.stderr` and `.returncode` and **throws both away**, keeping only `.stdout`. Every distinct failure (provider 5xx, rate-limit, panic, malformed envelope, schema-invalid reply, timeout) collapses into one undifferentiated `return None`. `GOOSE_BASE` also passes `--no-session`, so none of it lands in `~/.local/share/goose/sessions/sessions.db` either. **Layer 2 - triage launders the `None` into a plausible default.** `scripts/goose-triage.py` fail-softs every `None`: `"unscored -> default 30"`, `MODE_DEFAULT = "consult"`, runoff failure keeps issue order. A failed model call surfaces as a wrong-but-valid label, not a failure. The cached report's `failed: 0` measures label-write success, not Goose-call success. ## Why this issue is the foundation Goose here is already 100% ward-routed (every call is a ward-exec'd subprocess), and the failure data is already in hand at one Python function. The fix is to stop discarding `.returncode` / `.stderr`. This issue also **defines the failure-record schema v1** that the Claude-transcript drain (issue B) and the GlitchTip shipper (issue C) both consume. ## Scope 1. **Instrument `goose_json.ask()`.** Keep `.returncode` and `.stderr`. Classify each failure into the taxonomy already latent in the code: - `timeout` (`TimeoutExpired`) - `nonzero_exit` (returncode != 0 - currently ignored; likely where provider / rate-limit / panic errors hide in Goose's stderr) - `bad_envelope` (`JSONDecodeError` on the envelope) - `no_schema_valid_message` (envelope parsed, nothing satisfied the schema) - `exhausted_retries` (all N attempts failed) 2. **Append a failure-record** (one JSON line per genuine failure) to a per-repo buffer at `~/.cache/agentic-os/tool-failures/<repo-slug>.jsonl`. No network in this issue - local buffer only. 3. **Stop the silent laundering in `goose-triage`.** The fail-soft defaults stay as resilience, but the substitution must be counted: add a `goose_failures` block to the report, per pass (P0-confirm / urgency / runoff / mode), separate from the label-write `failed`. ## failure-record schema v1 (proposed) One JSON object per line: ``` ts int unix seconds harness str "goose" source str "goose_json.ask" repo str repo slug failure_class str one of the taxonomy above schema_title str recipe/schema title (the goose "tool" analog) exit_code int? Goose process returncode (null on timeout) attempt int which retry (0-based) stderr_excerpt str truncated Goose stderr (the discarded evidence) detail str short message fingerprint str hash of (harness, failure_class, schema_title, stderr-signature) ``` `fingerprint` collapses a flood of identical failures into one counted bucket - the field that finally answers "which failure mode dominates." ## Done condition - Every `ask()` failure path writes one classified failure-record to the buffer. - `goose-triage` report carries a real `goose_failures` count per pass. - Existing behavior (fail-soft defaults, exit codes, stdout contract) unchanged. - `ward exec test` green; the failure-record schema documented in `docs/goose-triage.md`. ## Out of scope (separate issues) - Claude Code transcript drain (issue B). - Shipping the buffer to GlitchTip (issue C). - `sessions.db` drain for the interactive Goose slice. Origin: design session 2026-06-18. Hooks were empirically ruled out as a failure source (Claude Code PostToolUse does not fire on failure and carries no exit code); the harness-specific sources are the transcript (Claude) and the wrapper boundary (Goose).

coilysiren referenced this issue from a commit

2026-06-18 07:32:54 +00:00

feat(goose): instrument goose_json.ask() failures + per-pass triage failure counter

coilysiren closed this issue

2026-06-18 07:32:54 +00:00

coilysiren referenced this issue

2026-07-04 09:17:56 +00:00

feat(o11y): create GlitchTip project + populate /sentry-dsn/tool-failures; decide failure-record producer re-homing #326

coilysiren referenced this issue

2026-07-04 09:18:19 +00:00

feat(o11y): shared GlitchTip shipper - classify, fingerprint, drain failure-record JSONL to GlitchTip (DSN from SSM) #250