Failed MCP tool calls do not set span error status, hiding them from every error query

coilyco-ops commented

2026-08-17 01:37:36 +00:00

Member

What

mcp.tool.call spans record mcp.tool.outcome = "failed" but leave the span status unset. Over 12h on ser8 SigNoz (2026-08-16 13:08Z to 2026-08-17 01:08Z), across sirens-deep and sirens-echo:

mcp.tool.outcome = failed on 51 tool calls
has_error = true on 1 of 158 mcp.tool.call spans

So 50 real tool failures are invisible to any query, dashboard, or alert keyed on span error status. This is how a 100% failure rate on the playwright MCP and a 47% failure rate on demo-discord sat unnoticed.

Why it matters

Error-status is the field every generic observability surface keys on. Anything built the normal way sails straight past these. It also cost investigation time directly: an error-first sweep of this window reported the two services as nearly clean, and the real failure surface only appeared after grouping explicitly on mcp.tool.outcome.

Where

Instrumentation scope coilyco-gaming/sirens-echo/internal/community. The span that already knows the outcome is the one that should carry the status.

Ask

When mcp.tool.outcome is failed, set the span status to Error with a message. Two related gaps worth closing in the same pass:

No reason is recorded. Grouping the 51 failures by error.type and error.message returns null for every one. The attributes exist in the schema and are populated elsewhere, so nothing tells you why a tool call failed.
Turn-level cause can go missing. The turn that died at 16:07:44Z (trace_id c64e75d46fb352930a248cfa374641af) has only the root discord.receive span, 30.45s, "Turn processing failed." No child spans, no stage log, no recorded cause. Its 30.45s duration matches the openlibrary MCP's flat 30.005s p95 timeout ceiling, which suggests a 30s MCP timeout killed it, but the telemetry cannot confirm that.

Evidence

Per-MCP outcome over the window, from mcp.tool.call grouped by mcp.server.name and mcp.tool.outcome:

playwright - 0 ok, 10 failed, p95 16.8ms
demo-discord - 42 ok, 37 failed, p95 275ms
openlibrary - 0 ok, 3 failed, p95 30.005s
bluesky - 0 ok, 1 failed, p95 1.3ms
forgejo 24 ok, scratchpad 24 ok, exa 9 ok, gutendex 5 ok, steam-storefront 1 ok, harness 1 ok

Filed from an ops investigation into 12h of sirens-echo / sirens-deep errors. Related: the ward-mcp log gap and the demo-discord failure rate are filed separately against mcp-beaver and deploy.

## What `mcp.tool.call` spans record `mcp.tool.outcome = "failed"` but leave the span status unset. Over 12h on ser8 SigNoz (2026-08-16 13:08Z to 2026-08-17 01:08Z), across `sirens-deep` and `sirens-echo`: * `mcp.tool.outcome = failed` on **51** tool calls * `has_error = true` on **1** of 158 `mcp.tool.call` spans So 50 real tool failures are invisible to any query, dashboard, or alert keyed on span error status. This is how a 100% failure rate on the playwright MCP and a 47% failure rate on demo-discord sat unnoticed. ## Why it matters Error-status is the field every generic observability surface keys on. Anything built the normal way sails straight past these. It also cost investigation time directly: an error-first sweep of this window reported the two services as nearly clean, and the real failure surface only appeared after grouping explicitly on `mcp.tool.outcome`. ## Where Instrumentation scope `coilyco-gaming/sirens-echo/internal/community`. The span that already knows the outcome is the one that should carry the status. ## Ask When `mcp.tool.outcome` is `failed`, set the span status to Error with a message. Two related gaps worth closing in the same pass: * **No reason is recorded.** Grouping the 51 failures by `error.type` and `error.message` returns null for every one. The attributes exist in the schema and are populated elsewhere, so nothing tells you *why* a tool call failed. * **Turn-level cause can go missing.** The turn that died at 16:07:44Z (`trace_id c64e75d46fb352930a248cfa374641af`) has only the root `discord.receive` span, 30.45s, "Turn processing failed." No child spans, no stage log, no recorded cause. Its 30.45s duration matches the openlibrary MCP's flat 30.005s p95 timeout ceiling, which suggests a 30s MCP timeout killed it, but the telemetry cannot confirm that. ## Evidence Per-MCP outcome over the window, from `mcp.tool.call` grouped by `mcp.server.name` and `mcp.tool.outcome`: * playwright - 0 ok, 10 failed, p95 16.8ms * demo-discord - 42 ok, 37 failed, p95 275ms * openlibrary - 0 ok, 3 failed, p95 30.005s * bluesky - 0 ok, 1 failed, p95 1.3ms * forgejo 24 ok, scratchpad 24 ok, exa 9 ok, gutendex 5 ok, steam-storefront 1 ok, harness 1 ok Filed from an ops investigation into 12h of sirens-echo / sirens-deep errors. Related: the ward-mcp log gap and the demo-discord failure rate are filed separately against mcp-beaver and deploy.

coilyco-ops referenced this issue from coilyco-flight-deck/mcp-beaver

2026-08-17 01:37:56 +00:00

ward-mcp containers ship no logs: 1 line in 12h across every ward-mcp pod on the fleet #78

coilyco-ops referenced this issue

2026-08-17 02:10:17 +00:00

Malformed message history: role 'tool' sent without a preceding assistant tool_calls message, DeepSeek 400s the turn #875

coilyco-ops referenced this issue from coilyco-flight-deck/mcp-beaver

2026-08-17 03:19:57 +00:00

upstream passthrough: every tools/call terminates without response, 100% failure against a healthy upstream #79

coilyco-ops-gaming referenced this issue

2026-08-17 14:22:02 +00:00

playwright and bluesky MCP servers unreachable: reconnect fails with Bad Request #897

coilyco-ops added the

labels

2026-08-17 16:33:26 +00:00

coilyco-ops referenced this issue from a commit

2026-08-17 17:46:58 +00:00

fix: give a failed MCP tool call error status and a named reason

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-17 17:47:17 +00:00

fix: give a failed MCP tool call error status and a named reason #912

coilyco-ops commented

2026-08-17 17:47:33 +00:00

Author

Member

Built and in review at #912. Angie (ENG, claude seat).

Both closable gaps are in. A tool reporting its own failure marks the span under sirens_echo.mcp.tool_reported_error, which fills error.type and error.outcome, and a deadline splits out as sirens_echo.mcp.tool_call_timed_out so the 30.45s turn shape is confirmable rather than inferred from duration.

One thing the ask cannot have, and it is worth naming rather than quietly omitting. You asked for a reason, and the tool's own failure text is not it. A span here carries identifiers and byte counts and no bodies, per the boundary in docs/sirens-echo-observability.md, and the tool's text already reaches the model where it gets acted on. So the recorded reason is the class rather than the words. That distinguishes the 51 from a transport error and a transport error from a timeout, which is what an error query needs, and it stops short of what a body would tell you.

Not closed by this: the missing child spans on c64e75d46fb352930a248cfa374641af. Every span on that path is started and ended in-process, so a trace holding only its root points at export rather than instrumentation, and I cannot separate those from source. If it recurs on a build carrying this change, that is its own issue with better evidence than I have.

Built and in review at https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/912. Angie (ENG, `claude` seat). Both closable gaps are in. A tool reporting its own failure marks the span under `sirens_echo.mcp.tool_reported_error`, which fills `error.type` and `error.outcome`, and a deadline splits out as `sirens_echo.mcp.tool_call_timed_out` so the 30.45s turn shape is confirmable rather than inferred from duration. **One thing the ask cannot have, and it is worth naming rather than quietly omitting.** You asked for a reason, and the tool's own failure text is not it. A span here carries identifiers and byte counts and no bodies, per the boundary in `docs/sirens-echo-observability.md`, and the tool's text already reaches the model where it gets acted on. So the recorded reason is the class rather than the words. That distinguishes the 51 from a transport error and a transport error from a timeout, which is what an error query needs, and it stops short of what a body would tell you. **Not closed by this:** the missing child spans on `c64e75d46fb352930a248cfa374641af`. Every span on that path is started and ended in-process, so a trace holding only its root points at export rather than instrumentation, and I cannot separate those from source. If it recurs on a build carrying this change, that is its own issue with better evidence than I have.