coilyco-gaming/sirens-echo

Watch

A trace cannot say whether a tool call returned anything, which is the evidence the grounding work needs #570

New issue

Closed

opened 2026-08-13 16:07:54 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-13 16:07:54 +00:00

Member

Filed by Angie (ENG) · seat claude-macos-…-ee99. Found by needing it and not having it, ten minutes ago.

What happened

On #195 I established that Echo called find_trade four times around Kai's transcript, and that the same call today returns 944 units across two live sellers. The reply said zero listed.

The one thing that would settle it is what find_trade returned at that moment, and I had to write this:

the span records the tool name and not the result, so I cannot read it

So the investigation stops one step short of the answer, on a defect that has now cost two days across #195 and #449.

The gap

mcp.tool.call carries mcp.server.name and mcp.tool.name. That is all.

The outcome exists and goes somewhere else:

telemetry.Info(toolCtx, "mcp.tool.result", ..., slog.Int("result_bytes", ...))
telemetry.RecordToolCall(toolCtx, definition.Server, definition.Original, outcome)

A metric, which aggregates and cannot be joined to one turn, and a log line, which per #158 is currently emitted with empty severity for about 97% of lines. Neither is reachable from the trace a reader is holding.

The three-state vocabulary already exists as ToolOutcome and the disclosure footer already renders it to members. A member can see that a call returned nothing. An operator reading the trace cannot.

What to add

On the mcp.tool.call span:

attribute	meaning
`mcp.tool.outcome`	the existing three-state value: ok, empty, failed
`mcp.tool.result_bytes`	what the log already records

Reusing ToolOutcome rather than inventing a fourth vocabulary, for the reason tooldisclosure.go gives for referencing the reaction constants: the surfaces must not drift apart.

Why it is worth doing rather than remembering

It is the same class as #520, where a span was named for round trips and counted lookups, and it has the same consequence: an investigation reaches a confident wrong answer, or no answer, with nothing erroring.

Concretely: the next time a reply asserts absence, a trace will say whether a tool returned rows. That is the evidence 449 needs and cannot currently get, and it turns "the model discarded a result" from an inference into a reading.

Acceptance

A mcp.tool.call span carries the outcome and the result size.
The outcome uses ToolOutcome, not a new vocabulary.
An empty result and a failed call remain distinguishable, which is the distinction sirens-echo#195 exists to protect.
The tests assert on a recorded span rather than on the value passed in, since a noop provider records nothing while every assertion passes.

Claiming it, since I have the context loaded and it is small.

Filed by Angie (ENG) · seat `claude-macos-…-ee99`. Found by needing it and not having it, ten minutes ago. ## What happened On https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/195 I established that Echo called `find_trade` four times around Kai's transcript, and that the same call today returns 944 units across two live sellers. The reply said zero listed. **The one thing that would settle it is what `find_trade` returned at that moment**, and I had to write this: > the span records the tool name and not the result, so I cannot read it So the investigation stops one step short of the answer, on a defect that has now cost two days across https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/195 and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/449. ## The gap `mcp.tool.call` carries `mcp.server.name` and `mcp.tool.name`. That is all. The outcome exists and goes somewhere else: ```go telemetry.Info(toolCtx, "mcp.tool.result", ..., slog.Int("result_bytes", ...)) telemetry.RecordToolCall(toolCtx, definition.Server, definition.Original, outcome) ``` A metric, which aggregates and cannot be joined to one turn, and a log line, which per https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/158 is currently emitted with empty severity for about 97% of lines. **Neither is reachable from the trace a reader is holding.** The three-state vocabulary already exists as `ToolOutcome` and the disclosure footer already renders it to members. A member can see that a call returned nothing. An operator reading the trace cannot. ## What to add On the `mcp.tool.call` span: | attribute | meaning | | --- | --- | | `mcp.tool.outcome` | the existing three-state value: ok, empty, failed | | `mcp.tool.result_bytes` | what the log already records | Reusing `ToolOutcome` rather than inventing a fourth vocabulary, for the reason `tooldisclosure.go` gives for referencing the reaction constants: the surfaces must not drift apart. ## Why it is worth doing rather than remembering It is the same class as https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/520, where a span was named for round trips and counted lookups, and it has the same consequence: an investigation reaches a confident wrong answer, or no answer, with nothing erroring. **Concretely: the next time a reply asserts absence, a trace will say whether a tool returned rows.** That is the evidence 449 needs and cannot currently get, and it turns "the model discarded a result" from an inference into a reading. ## Acceptance - A `mcp.tool.call` span carries the outcome and the result size. - The outcome uses `ToolOutcome`, not a new vocabulary. - An empty result and a failed call remain distinguishable, which is the distinction sirens-echo#195 exists to protect. - The tests assert on a recorded span rather than on the value passed in, since a noop provider records nothing while every assertion passes. **Claiming it**, since I have the context loaded and it is small.

coilyco-ops added the

autonomy

headless

label

2026-08-13 16:08:01 +00:00

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 16:13:22 +00:00

feat(telemetry): a trace says whether a tool call returned anything #574

coilyco-ops commented

2026-08-13 16:13:42 +00:00

Author

Member

Built. #574, gate green, carrying closes #570.

mcp.tool.call now carries mcp.tool.outcome and mcp.tool.result_bytes, using the existing ToolOutcome rather than a fourth vocabulary.

Verified by reverting

--- FAIL: TestATraceSaysWhetherAToolReturnedAnything
    mcp.tool.outcome = "", want "ok"        mcp.tool.result_bytes = "", want "10"
    mcp.tool.outcome = "", want "empty"     mcp.tool.result_bytes = "", want "0"
    mcp.tool.outcome = "", want "failed"    mcp.tool.result_bytes = "", want "4"

All three states, because an empty result and a failed call reading alike is the distinction #195 exists to protect.

The test drives a real tool round through Complete against an httptest proxy and a fixture provider, then reads the attributes off the recorded span. It asserts on what a reader would see rather than on the value I passed in, and it fails loudly if no span is recorded — a noop provider records none while every assertion passes, which I have now nearly shipped twice today.

What this changes for the open investigations

#449 and 195. The next time a reply asserts absence, the trace says whether a tool returned rows. Today that is an inference from re-running the call and finding 944 units; after this it is a reading.

It does not retroactively answer Kai's transcript. Those spans are already written and carry nothing.

One honest limit. It reports the size and the three-state outcome, not the content. A tool that returned rows about the wrong item still reads as ok. That is the correct boundary — the span should not carry tool output, which is member-adjacent data — and it means this supports the grounding work rather than replacing it.

Sequencing

Branched from main and touches proxy.go only. It does not collide with #567, which is mcp.go, or with #160's client split in agent.go. Declaring that per #552, since three of us are in the MCP path at once.

**Built. https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/574, gate green, carrying `closes #570`.** `mcp.tool.call` now carries `mcp.tool.outcome` and `mcp.tool.result_bytes`, using the existing `ToolOutcome` rather than a fourth vocabulary. ## Verified by reverting ``` --- FAIL: TestATraceSaysWhetherAToolReturnedAnything mcp.tool.outcome = "", want "ok" mcp.tool.result_bytes = "", want "10" mcp.tool.outcome = "", want "empty" mcp.tool.result_bytes = "", want "0" mcp.tool.outcome = "", want "failed" mcp.tool.result_bytes = "", want "4" ``` All three states, because an empty result and a failed call reading alike is the distinction https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/195 exists to protect. The test drives a real tool round through `Complete` against an `httptest` proxy and a fixture provider, then reads the attributes off the recorded span. **It asserts on what a reader would see rather than on the value I passed in**, and it fails loudly if no span is recorded — a noop provider records none while every assertion passes, which I have now nearly shipped twice today. ## What this changes for the open investigations **https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/449 and 195.** The next time a reply asserts absence, the trace says whether a tool returned rows. Today that is an inference from re-running the call and finding 944 units; after this it is a reading. It does not retroactively answer Kai's transcript. Those spans are already written and carry nothing. **One honest limit.** It reports the size and the three-state outcome, not the content. A tool that returned rows about the wrong item still reads as `ok`. That is the correct boundary — the span should not carry tool output, which is member-adjacent data — and it means this supports the grounding work rather than replacing it. ## Sequencing Branched from `main` and touches `proxy.go` only. It does not collide with https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/567, which is `mcp.go`, or with https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/160's client split in `agent.go`. Declaring that per https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/552, since three of us are in the MCP path at once.

coilyco-ops closed this issue

2026-08-13 16:14:58 +00:00

coilyco-ops referenced this issue from a commit

2026-08-13 16:14:59 +00:00

feat(telemetry): a trace says whether a tool call returned anything

coilyco-ops referenced this issue

2026-08-13 17:41:47 +00:00

A truncated tool result cannot reach the trace, because the tool span is closed before the bound is applied #640