coilyco-flight-deck/agentic-os

Fork 0

feat(o11y): create GlitchTip project + populate /sentry-dsn/tool-failures; decide failure-record producer re-homing #326

New issue

Closed

opened 2026-07-04 09:17:56 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-04 09:17:56 +00:00

Owner

Follow-up from #250 (the GlitchTip shipper landed).

The one externally-visible step (needs Kai)

The shipper (ward exec ship-tool-failures) is DSN-pluggable and fail-soft: it no-ops and the buffer keeps accumulating until a DSN exists. To turn it on:

Create a GlitchTip project for the shared tool-use failure feed.
Put its DSN in SSM at /sentry-dsn/tool-failures (the path recorded in SSM.md and read by the shipper; override with TOOL_FAILURES_DSN_SSM_PATH if a different path is preferred). Following the observability skill's /sentry-dsn/<project> convention.

Once the param exists, a timer or SessionEnd hook running the verb drains the buffer, fingerprint-grouped.

The producer question (raised by the build)

#250 was designed to drain the failure-record buffer that #248 (Goose goose_json.ask) and #249 (Claude transcript drain) populate. Both producers were removed from aos by commit 03d1525 ("aos execed agent commands are last era", 2026-06-26), so nothing in-tree currently writes ~/.cache/agentic-os/tool-failures/<repo>.jsonl.

The shipper is producer-agnostic by design and reads the exact schema-v1 contract (recovered from the deleted docs/goose-failure-records.md / docs/claude-transcript-drain.md), so it works the moment any producer writes records again. But the failure-record chain now needs a decision: re-home the producers (e.g. into ward's embedded agent fleet), or retire the whole chain (which would make this shipper vestigial). This issue tracks that call.

Filed by the #250 run.

Follow-up from #250 (the GlitchTip shipper landed). ## The one externally-visible step (needs Kai) The shipper (`ward exec ship-tool-failures`) is DSN-pluggable and fail-soft: it no-ops and the buffer keeps accumulating until a DSN exists. To turn it on: 1. Create a GlitchTip project for the shared tool-use failure feed. 2. Put its DSN in SSM at `/sentry-dsn/tool-failures` (the path recorded in `SSM.md` and read by the shipper; override with `TOOL_FAILURES_DSN_SSM_PATH` if a different path is preferred). Following the observability skill's `/sentry-dsn/<project>` convention. Once the param exists, a timer or SessionEnd hook running the verb drains the buffer, fingerprint-grouped. ## The producer question (raised by the build) #250 was designed to drain the failure-record buffer that #248 (Goose `goose_json.ask`) and #249 (Claude transcript drain) populate. Both producers were **removed** from aos by commit `03d1525` ("aos execed agent commands are last era", 2026-06-26), so nothing in-tree currently writes `~/.cache/agentic-os/tool-failures/<repo>.jsonl`. The shipper is producer-agnostic by design and reads the exact schema-v1 contract (recovered from the deleted `docs/goose-failure-records.md` / `docs/claude-transcript-drain.md`), so it works the moment any producer writes records again. But the failure-record chain now needs a decision: re-home the producers (e.g. into ward's embedded agent fleet), or retire the whole chain (which would make this shipper vestigial). This issue tracks that call. Filed by the #250 run.

coilysiren referenced this issue

2026-07-04 09:18:19 +00:00

feat(o11y): shared GlitchTip shipper - classify, fingerprint, drain failure-record JSONL to GlitchTip (DSN from SSM) #250

coilyco-ops added the

consult

labels

2026-07-04 10:26:05 +00:00

coilyco-ops referenced this issue from coilyco-flight-deck/ward

2026-07-09 21:37:28 +00:00

feat(o11y): re-home tool-failure producers into ward's embedded agent fleet #871

coilyco-ops referenced this issue

2026-07-09 21:37:29 +00:00

docs(o11y): retarget tool-failure shipper docs around ward-owned producers #393

coilyco-ops commented

2026-07-09 21:37:31 +00:00

Member

🔎 ward agent advisor - cross-repo fan-out

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Recommendation

Keep the tool-failure chain, but do not re-home the producers back into agentic-os. The right split is:

ward owns producers because the live agent harnesses and host-side observability now live there.
agentic-os keeps the shared shipper for now because it is already landed, tested, producer-agnostic, and decoupled from any one harness.
Create the GlitchTip project and /sentry-dsn/tool-failures only once the first ward producer is actively landing, unless Kai wants to pre-provision a dark project early.

Why this is the right call

Two facts in the repo history matter:

On 2026-06-26, commit 03d1525 intentionally removed the old in-tree producer surfaces from agentic-os (goose_json.ask, Claude transcript drain, backlog-loop/triage stack). Reintroducing producers there would recreate a surface the repo explicitly retired.
On 2026-07-04, commit e386997 added the GlitchTip shipper as a producer-agnostic drain of ~/.cache/agentic-os/tool-failures/<repo>.jsonl, with SSM-backed DSN resolution, genuine-failure gating, fingerprint grouping, and watermark-based idempotency.

That leaves a clean architectural answer: the shipper can stay where it is, while the current harness owner writes records again.

Evidence for `ward` as the producer owner

ward is already the repo that owns the modern agent runtime and observability surface:

It has the embedded fleet and launches Claude/Codex/Goose from there.
It already has host-side agent-run observability (docs/agent-observability.md, cmd/ward/agent_log_drain.go), including transcript draining from containerized Claude runs.
It already has a Sentry/GlitchTip DSN pattern of its own (WARD_SENTRY_DSN, resolved from /sentry-dsn/ward per cmd/ward/errorreport.go).

That is a much better fit than putting harness-specific failure production back into agentic-os, which now reads as a shared utilities/catalog repo, not the active runtime.

Best near-term shape

1. First producer in `ward`: Claude transcript-based drain

This should be the first implementation, not Goose.

Reasons:

The deleted docs/claude-transcript-drain.md already captured the important design finding: Claude hooks do not reliably see all tool failures, while the transcript JSONL does.
ward already drains Claude transcripts host-side after runs, so the raw source is already in the runtime that matters.
Claude is the highest-confidence path to restoring signal without reviving old agentic-os exec surfaces.

2. Preserve the existing schema-v1 contract

The deleted producer docs and the current shipper still line up on one shared contract:

buffer location: ~/.cache/agentic-os/tool-failures/<repo>.jsonl
load-bearing fields: fingerprint, failure_class, harness, repo, optional expected
shipper behavior: drop expected, group by fingerprint, advance watermarks only after accepted lines

That contract is worth keeping because it lets ward add producers without rewriting the shipping side.

3. Treat Goose as follow-up, not day-one scope

The old Goose producer was specifically tied to the removed goose_json.ask() path in agentic-os. ward still supports Goose as a harness, but that is no longer the same seam. Recreate signal for Claude first, then decide whether Goose needs:

a harness-local failure recorder in ward, or
no dedicated producer if its use is now too marginal to justify the extra surface.

Options considered

Option A. Re-home producers into `ward`, keep shipper in `agentic-os`

Recommended.

Why:

Matches current repo boundaries.
Preserves the already-landed and tested shipper.
Minimizes churn and restores signal fastest.

Option B. Move the entire chain into `ward`

Viable later, but not the best immediate move.

Why not first:

It duplicates or relocates code that just landed on 2026-07-04.
It delays getting real events flowing.
The current shipper is already generic enough to serve ward producers.

Option C. Retire the chain entirely

Not recommended.

Why:

It strands a fresh, tested shipper.
It gives up the original observability goal at exactly the moment the harnesses became more centralized and thus easier to instrument in one place.
Nothing in the current repo says the need disappeared, only that the old producer location disappeared.

Operational step: the GlitchTip project and DSN

Creating the project and storing /sentry-dsn/tool-failures is still the right convention. It follows the existing /sentry-dsn/<project> pattern already documented in agentic-os/SSM.md.

My recommendation is:

Do create a dedicated tool-failures GlitchTip project.
Do store its DSN at /sentry-dsn/tool-failures.
Time that step with the first ward producer landing so the project does not sit dark indefinitely.

If Kai prefers to pre-provision it now, that is low-risk because the shipper is fail-soft and no producer means no traffic. It is just lower signal than sequencing it alongside the first producer.

Open questions to settle in implementation

Should ward write directly into the existing ~/.cache/agentic-os/tool-failures/ buffer path, or should that path become a neutral ward-owned location later with a compatibility bridge first?
Does the schema-v1 contract stay documented in agentic-os, or does ward become the canonical owner once the first producer lands there?
For Claude events, where is the right dedupe boundary to avoid double-reporting if ward later grows both transcript and hook-based capture?
How much producer-side redaction is required before writing stderr_excerpt-style fields into the local buffer, given ward already has stronger observability scrubbing rules than the old agentic-os docs assumed?

Risks

Boundary drift: if ward writes records but agentic-os remains the undocumented schema owner, the contract can fork silently.
Duplicate events: a future transcript drain plus hook capture can report the same failure twice unless event identity is explicit.
Secret leakage in failure detail: the shipper groups correctly, but producer-side excerpts still need scrubbing discipline before they ever hit disk.
Dark project risk: creating the DSN/project before any producer exists is harmless but can look like the feature is on when no signal source exists.

Bottom line

The issue should be resolved as "keep the chain, move production to ward, keep the current shipper, and activate the DSN when the first ward producer is landing."

That preserves the July 4 shipper work, respects the June 26 runtime cleanup, and puts the instrumentation where the agent fleet actually lives now.

The work spans multiple repos, so it was filed as 2 tracked issue(s), in dependency order:

coilyco-flight-deck/ward#871 - feat(o11y): re-home tool-failure producers into ward's embedded agent fleet
coilyco-flight-deck/agentic-os#393 - docs(o11y): retarget tool-failure shipper docs around ward-owned producers

Researched and posted automatically by ward agent advisor --harness codex (ward#424). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor - cross-repo fan-out `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. ## Recommendation Keep the tool-failure chain, but **do not re-home the producers back into `agentic-os`**. The right split is: - **`ward` owns producers** because the live agent harnesses and host-side observability now live there. - **`agentic-os` keeps the shared shipper for now** because it is already landed, tested, producer-agnostic, and decoupled from any one harness. - **Create the GlitchTip project and `/sentry-dsn/tool-failures` only once the first `ward` producer is actively landing**, unless Kai wants to pre-provision a dark project early. ## Why this is the right call Two facts in the repo history matter: - On **2026-06-26**, commit `03d1525` intentionally removed the old in-tree producer surfaces from `agentic-os` (`goose_json.ask`, Claude transcript drain, backlog-loop/triage stack). Reintroducing producers there would recreate a surface the repo explicitly retired. - On **2026-07-04**, commit `e386997` added the GlitchTip shipper as a **producer-agnostic** drain of `~/.cache/agentic-os/tool-failures/<repo>.jsonl`, with SSM-backed DSN resolution, genuine-failure gating, fingerprint grouping, and watermark-based idempotency. That leaves a clean architectural answer: the shipper can stay where it is, while the **current** harness owner writes records again. ## Evidence for `ward` as the producer owner `ward` is already the repo that owns the modern agent runtime and observability surface: - It has the **embedded fleet** and launches Claude/Codex/Goose from there. - It already has **host-side agent-run observability** (`docs/agent-observability.md`, `cmd/ward/agent_log_drain.go`), including transcript draining from containerized Claude runs. - It already has a **Sentry/GlitchTip DSN pattern** of its own (`WARD_SENTRY_DSN`, resolved from `/sentry-dsn/ward` per `cmd/ward/errorreport.go`). That is a much better fit than putting harness-specific failure production back into `agentic-os`, which now reads as a shared utilities/catalog repo, not the active runtime. ## Best near-term shape ### 1. First producer in `ward`: Claude transcript-based drain This should be the first implementation, not Goose. Reasons: - The deleted `docs/claude-transcript-drain.md` already captured the important design finding: **Claude hooks do not reliably see all tool failures**, while the transcript JSONL does. - `ward` already drains Claude transcripts host-side after runs, so the raw source is already in the runtime that matters. - Claude is the highest-confidence path to restoring signal without reviving old `agentic-os` exec surfaces. ### 2. Preserve the existing schema-v1 contract The deleted producer docs and the current shipper still line up on one shared contract: - buffer location: `~/.cache/agentic-os/tool-failures/<repo>.jsonl` - load-bearing fields: `fingerprint`, `failure_class`, `harness`, `repo`, optional `expected` - shipper behavior: drop `expected`, group by `fingerprint`, advance watermarks only after accepted lines That contract is worth keeping because it lets `ward` add producers without rewriting the shipping side. ### 3. Treat Goose as follow-up, not day-one scope The old Goose producer was specifically tied to the removed `goose_json.ask()` path in `agentic-os`. `ward` still supports Goose as a harness, but that is no longer the same seam. Recreate signal for Claude first, then decide whether Goose needs: - a harness-local failure recorder in `ward`, or - no dedicated producer if its use is now too marginal to justify the extra surface. ## Options considered ### Option A. Re-home producers into `ward`, keep shipper in `agentic-os` **Recommended.** Why: - Matches current repo boundaries. - Preserves the already-landed and tested shipper. - Minimizes churn and restores signal fastest. ### Option B. Move the entire chain into `ward` Viable later, but not the best immediate move. Why not first: - It duplicates or relocates code that just landed on **2026-07-04**. - It delays getting real events flowing. - The current shipper is already generic enough to serve `ward` producers. ### Option C. Retire the chain entirely Not recommended. Why: - It strands a fresh, tested shipper. - It gives up the original observability goal at exactly the moment the harnesses became more centralized and thus easier to instrument in one place. - Nothing in the current repo says the need disappeared, only that the old producer location disappeared. ## Operational step: the GlitchTip project and DSN Creating the project and storing `/sentry-dsn/tool-failures` is still the right convention. It follows the existing `/sentry-dsn/<project>` pattern already documented in `agentic-os/SSM.md`. My recommendation is: - **Do create a dedicated `tool-failures` GlitchTip project.** - **Do store its DSN at `/sentry-dsn/tool-failures`.** - **Time that step with the first `ward` producer landing** so the project does not sit dark indefinitely. If Kai prefers to pre-provision it now, that is low-risk because the shipper is fail-soft and no producer means no traffic. It is just lower signal than sequencing it alongside the first producer. ## Open questions to settle in implementation - Should `ward` write directly into the existing `~/.cache/agentic-os/tool-failures/` buffer path, or should that path become a neutral `ward`-owned location later with a compatibility bridge first? - Does the `schema-v1` contract stay documented in `agentic-os`, or does `ward` become the canonical owner once the first producer lands there? - For Claude events, where is the right dedupe boundary to avoid double-reporting if `ward` later grows both transcript and hook-based capture? - How much producer-side redaction is required before writing `stderr_excerpt`-style fields into the local buffer, given `ward` already has stronger observability scrubbing rules than the old `agentic-os` docs assumed? ## Risks - **Boundary drift**: if `ward` writes records but `agentic-os` remains the undocumented schema owner, the contract can fork silently. - **Duplicate events**: a future transcript drain plus hook capture can report the same failure twice unless event identity is explicit. - **Secret leakage in failure detail**: the shipper groups correctly, but producer-side excerpts still need scrubbing discipline before they ever hit disk. - **Dark project risk**: creating the DSN/project before any producer exists is harmless but can look like the feature is on when no signal source exists. ## Bottom line The issue should be resolved as **"keep the chain, move production to `ward`, keep the current shipper, and activate the DSN when the first `ward` producer is landing."** That preserves the July 4 shipper work, respects the June 26 runtime cleanup, and puts the instrumentation where the agent fleet actually lives now. --- The work spans multiple repos, so it was filed as 2 tracked issue(s), in dependency order: 1. [coilyco-flight-deck/ward#871](https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/871) - feat(o11y): re-home tool-failure producers into ward's embedded agent fleet 2. [coilyco-flight-deck/agentic-os#393](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/393) - docs(o11y): retarget tool-failure shipper docs around ward-owned producers --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#424). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilyco-ops commented

2026-07-09 21:38:52 +00:00

Member

Triage pass on 2026-07-09: closing this parent after advisor resolution.

Decision: keep the chain, move producer ownership to ward, keep the current AOS shipper for now, and create the GlitchTip project plus /sentry-dsn/tool-failures when the first ward producer is landing.

Follow-ups now track the remaining work:

coilyco-flight-deck/ward#871 - re-home tool-failure producers into ward.
agentic-os#393 - retarget AOS shipper docs after the ward producer contract settles.

Triage pass on 2026-07-09: closing this parent after advisor resolution. Decision: keep the chain, move producer ownership to ward, keep the current AOS shipper for now, and create the GlitchTip project plus /sentry-dsn/tool-failures when the first ward producer is landing. Follow-ups now track the remaining work: * coilyco-flight-deck/ward#871 - re-home tool-failure producers into ward. * agentic-os#393 - retarget AOS shipper docs after the ward producer contract settles.

coilyco-ops closed this issue

2026-07-09 21:39:00 +00:00

No labels

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

coilyco-flight-deck/agentic-os#326

No description provided.

Rows
Columns