coilyco-gaming/sirens-echo

Watch

Tool surface drops from 86 tools to 0 mid-turn, and a dead tool is re-called six times in one turn with no breaker #943

New issue

Closed

opened 2026-08-18 01:54:36 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-18 01:54:36 +00:00

Member

Filed by Angie (ENG, claude seat) from a read-only sweep of the same 2026-08-18 owl.glass window as #932. Spans and agent-proxy capture logs only, no live system touched.

Operational improvement. Released by the #929 amendment: both changes below alter how reliably the lane does what it already does, not what it can do.

Deliberately not a round cap and not a roster reduction. #940 records that Kai declined both. Nothing here needs either.

Why this is a separate issue

The truncation and over-fetch findings I reached are already covered by #940 and #936 with the same byte counts, and the round-budget waste by #935. Those are not re-filed here. Two findings from the same window appear in no open issue I could find, and one of them is a mechanism nobody has specified yet.

1. The tool surface collapses to zero mid-turn

Request 1539078028959944794, trace 6d1efdd560dd868db1fdbea1394cd3cd, one Discord message at 01:06:36Z. Tool count offered to the model, per captured request:

round 1 - 86 tools
rounds 3, 5, 7 - 86 tools
rounds 9 and 11 - 0 tools
round 13 - back to 86 tools

The model was asked to continue a turn whose own message history is full of tool_calls, with an empty tool block. It is not a prompt the model can answer correctly by construction.

The mcp.tools.list spans over the 3h window show the same shape from the harness side:

configured: 12, listed: 0, reached: 0, cached: true - 13 calls
configured: 11, listed: 0, reached: 0, cached: true - 8 calls
configured: 11, listed: 11, reached: 11, cached: false - 3 calls
configured: 12, listed: 12, reached: 12, cached: false - 1 call

Every cold call reaches every configured server. Every cache hit reaches none. 21 of 25. That is either a cache that stores an empty roster and serves it, or a counter that is not written on the cached path. The two have very different severity and the distinction is cheap to settle.

Cannot be established from telemetry alone: which of those it is. Reading the tools-list cache path in internal/community settles it in one sitting. I did not open it, so the collapse is measured and the cause is not.

2. A dead tool is re-called six times in one turn

Same turn. playwright__browser_navigate against the same artifact URL:

round 1 - once
round 5 - once
round 7 - twice
round 9 - twice
round 11 - twice

Every one returned the identical error, verbatim:

mcp-beaver: upstream MCP session is closed (reconnect also failed:
reconnect upstream MCP "http://127.0.0.1:8931/mcp":
sending "notifications/initialized": Bad Request)

Rounds 9 and 11 were byte-identical requests at 27,867 bytes each. Not a retry with backoff and not a varied second attempt. The same request, twice.

Across the window browser_navigate was 9 calls, 0 successes, 100% failure, spread evenly through 00:20, 00:30, 00:40 and 01:00. Playwright has been dead since #897 and coilyco-flight-deck/mcp-beaver#80.

#932 records this as one bullet with two retries observed. Six in one turn is worse than recorded, and neither #932 nor #935 proposes the mechanism, so it is specified here.

What it cost, precisely

Same turn, from the spans:

discord.receive opens 01:06:36.875
a complete, correct reply is generated and validated at 01:07:16.323 - 39.4s
discord.reply runs 01:07:23.097 to 01:09:19.8 - 116.7s
total 163.7s

The answer was ready at 39 seconds. The remaining 124 seconds bought nothing. The passes after the first re-attempted the dead tool, twice per round, and re-derived substantially the same answer.

The breaker, specified

#873 closed having established that mcp.tool.outcome is the reliable failure signal, because span error status is not set on failed tool calls. That attribute is already written on every mcp.tool.call. The breaker keys on it and needs no new instrumentation.

Within a turn, once a tool returns outcome: failed, refuse further calls to that same tool and return the recorded error to the model as the tool result. One failure per tool per turn, not six.
Across turns, when a server has returned only failures for a configurable window, mark it unavailable and say so in the tool result rather than attempting the call. This is what stops Deep rediscovering the playwright outage on every single turn.
Never silently drop the tool. The model must be told the tool is out, which is what lets it do the thing it already did correctly here: say it cannot see the artifact and answer the part that does not need it.

This is per-tool and per-outcome. It does not cap rounds and it does not shrink the roster, so it is orthogonal to both decisions in #940.

Worth noting, because it changes what "fix" means

The reply this turn produced was good. It said it could not see the artifact, named the outage, linked #897 unprompted, declined to fake a verdict, and answered the answerable half well. The defect is not the text. It is that the harness spent two extra minutes and six dead tool calls after that text already existed. A breaker makes this turn a 40-second good answer instead of a 164-second good answer.

That also separates this from #935, which is about turns that confabulate. This turn did not confabulate. It was correct and slow.

Done when

No turn issues a second call to a tool that already returned outcome: failed in that turn.
A server with a standing failure record is reported as unavailable in the tool result rather than called.
mcp.tools.list never reports reached: 0 while configured is non-zero, or the counter is corrected and the cached path is confirmed to serve a populated roster.

All three are measurable from existing spans.

#932 - the incident review this window comes from, records the blind retry as one bullet
#935 - round budget spent on unrelated tools, the confabulation case this one is not
#940 and #936 - truncation and over-fetch from the same window, not re-filed here
#897 and coilyco-flight-deck/mcp-beaver#80 - the dead playwright session itself
#873 - closed, established mcp.tool.outcome as the signal this breaker keys on

**Filed by Angie (ENG, `claude` seat)** from a read-only sweep of the same 2026-08-18 owl.glass window as #932. Spans and agent-proxy capture logs only, no live system touched. **Operational improvement.** Released by the [#929 amendment](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/929#issuecomment-68413): both changes below alter how reliably the lane does what it already does, not what it can do. **Deliberately not a round cap and not a roster reduction.** #940 records that Kai declined both. Nothing here needs either. ## Why this is a separate issue The truncation and over-fetch findings I reached are already covered by #940 and #936 with the same byte counts, and the round-budget waste by #935. Those are not re-filed here. Two findings from the same window appear in no open issue I could find, and one of them is a mechanism nobody has specified yet. ## 1. The tool surface collapses to zero mid-turn Request `1539078028959944794`, trace `6d1efdd560dd868db1fdbea1394cd3cd`, one Discord message at 01:06:36Z. Tool count offered to the model, per captured request: * round 1 - **86 tools** * rounds 3, 5, 7 - 86 tools * rounds 9 and 11 - **0 tools** * round 13 - back to 86 tools The model was asked to continue a turn whose own message history is full of `tool_calls`, with an empty tool block. It is not a prompt the model can answer correctly by construction. The `mcp.tools.list` spans over the 3h window show the same shape from the harness side: * `configured: 12, listed: 0, reached: 0, cached: true` - **13 calls** * `configured: 11, listed: 0, reached: 0, cached: true` - **8 calls** * `configured: 11, listed: 11, reached: 11, cached: false` - 3 calls * `configured: 12, listed: 12, reached: 12, cached: false` - 1 call **Every cold call reaches every configured server. Every cache hit reaches none.** 21 of 25. That is either a cache that stores an empty roster and serves it, or a counter that is not written on the cached path. The two have very different severity and the distinction is cheap to settle. **Cannot be established from telemetry alone:** which of those it is. Reading the tools-list cache path in `internal/community` settles it in one sitting. I did not open it, so the collapse is measured and the cause is not. ## 2. A dead tool is re-called six times in one turn Same turn. `playwright__browser_navigate` against the same artifact URL: * round 1 - once * round 5 - once * round 7 - twice * round 9 - twice * round 11 - twice Every one returned the identical error, verbatim: ``` mcp-beaver: upstream MCP session is closed (reconnect also failed: reconnect upstream MCP "http://127.0.0.1:8931/mcp": sending "notifications/initialized": Bad Request) ``` Rounds 9 and 11 were **byte-identical requests at 27,867 bytes each**. Not a retry with backoff and not a varied second attempt. The same request, twice. Across the window `browser_navigate` was **9 calls, 0 successes, 100% failure**, spread evenly through 00:20, 00:30, 00:40 and 01:00. Playwright has been dead since #897 and coilyco-flight-deck/mcp-beaver#80. #932 records this as one bullet with two retries observed. Six in one turn is worse than recorded, and neither #932 nor #935 proposes the mechanism, so it is specified here. ## What it cost, precisely Same turn, from the spans: * `discord.receive` opens **01:06:36.875** * a complete, correct reply is generated and validated at **01:07:16.323** - **39.4s** * `discord.reply` runs **01:07:23.097 to 01:09:19.8** - **116.7s** * total **163.7s** **The answer was ready at 39 seconds. The remaining 124 seconds bought nothing.** The passes after the first re-attempted the dead tool, twice per round, and re-derived substantially the same answer. ## The breaker, specified #873 closed having established that `mcp.tool.outcome` is the reliable failure signal, because span error status is not set on failed tool calls. That attribute is already written on every `mcp.tool.call`. The breaker keys on it and needs no new instrumentation. * **Within a turn**, once a tool returns `outcome: failed`, refuse further calls to that same tool and return the recorded error to the model as the tool result. One failure per tool per turn, not six. * **Across turns**, when a server has returned only failures for a configurable window, mark it unavailable and say so in the tool result rather than attempting the call. This is what stops Deep rediscovering the playwright outage on every single turn. * **Never silently drop the tool.** The model must be told the tool is out, which is what lets it do the thing it already did correctly here: say it cannot see the artifact and answer the part that does not need it. This is per-tool and per-outcome. It does not cap rounds and it does not shrink the roster, so it is orthogonal to both decisions in #940. ## Worth noting, because it changes what "fix" means The reply this turn produced was good. It said it could not see the artifact, named the outage, linked #897 unprompted, declined to fake a verdict, and answered the answerable half well. The defect is not the text. **It is that the harness spent two extra minutes and six dead tool calls after that text already existed.** A breaker makes this turn a 40-second good answer instead of a 164-second good answer. That also separates this from #935, which is about turns that confabulate. This turn did not confabulate. It was correct and slow. ## Done when * No turn issues a second call to a tool that already returned `outcome: failed` in that turn. * A server with a standing failure record is reported as unavailable in the tool result rather than called. * `mcp.tools.list` never reports `reached: 0` while `configured` is non-zero, or the counter is corrected and the cached path is confirmed to serve a populated roster. All three are measurable from existing spans. ## Related * #932 - the incident review this window comes from, records the blind retry as one bullet * #935 - round budget spent on unrelated tools, the confabulation case this one is not * #940 and #936 - truncation and over-fetch from the same window, not re-filed here * #897 and coilyco-flight-deck/mcp-beaver#80 - the dead playwright session itself * #873 - closed, established `mcp.tool.outcome` as the signal this breaker keys on

coilyco-ops added the

labels

2026-08-18 01:54:36 +00:00

coilyco-ops referenced this issue

2026-08-18 02:43:14 +00:00

Scope freeze through August 19: six items, everything else behind August 20 #929

coilyco-ops referenced this issue

2026-08-18 20:39:11 +00:00

Epic: August 19 Temporal Vibe Check demo readiness, the work that is on camera #981

coilyco-ops referenced this issue

2026-08-18 20:46:19 +00:00

Epic: August 19 Temporal Vibe Check demo readiness, the work that is on camera #981

coilyco-ops referenced this issue

2026-08-18 21:58:24 +00:00

owl.glass dropped 3 of 14 turns with no reply, no error, and no model call, at a ~30s wall #939

coilyco-ops referenced this issue

2026-08-18 22:07:09 +00:00

fix(turns): a dropped turn always tells the member #987

coilyco-ops referenced this issue

2026-08-18 22:36:53 +00:00

Evaluate if skills based progressive disclosure is actually working #968

coilyco-ops referenced this issue from a commit

2026-08-19 01:37:20 +00:00

feat(tools): break a failed tool for the rest of its turn

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-19 01:37:56 +00:00

feat(tools): break a failed tool for the rest of its turn #1007

coilyco-ops referenced this issue

2026-08-19 01:38:39 +00:00

The docs band is saturated: four pages at their cap, and no room for a 41st page either #1009

coilyco-ops referenced this issue from a commit

2026-08-19 01:45:47 +00:00

fix(tools): key the breaker on the arguments, not just the tool

coilyco-ops commented

2026-08-19 01:46:04 +00:00

Author

Member

Fixed in #1007, with one deliberate divergence from the acceptance here. Engineer seat.

Finding 1 is an instrument defect, not a collapse

The question this issue could not answer from telemetry: configured: 12, listed: 0, reached: 0, cached: true on 21 of 25 listings is neither "a cache serving an empty roster" nor "a counter not written on the cached path".

reached counts servers that took a round trip, and it is legitimately zero in two unrelated states: every server warm, and every server backing off. mcp.tools.cached is derived from reached == 0, so it reads true for both. On the warm path Open registers from entry.tools and the model receives the full roster. So those 21 spans are healthy turns.

mcp.tools.unavailable and mcp.tools.registered now sit beside reached, and a test builds both states against a real in-process MCP server to prove they read differently. cached is kept so existing queries do not break.

Rounds 9 and 11 of 6d1efdd560dd868db1fdbea1394cd3cd remain unexplained. Both accounted-for states are now distinguishable and neither is that one, so the finding survives with a better instrument aimed at it.

Finding 2 is fixed, keyed on the call rather than the tool

This is the divergence and it is worth arguing with. The acceptance here says "No turn issues a second call to a tool that already returned outcome: failed in that turn." #1007 blocks a second identical call instead.

I changed it after reading #940. A tool that fails on bad arguments is correctable: forgejo list_issue rejecting a query is not playwright holding a closed session, and refusing the model's corrected retry would cost a recovery it is entitled to make. Blocking per tool buys the playwright case and pays for it on every tool that fails for a fixable reason.

The evidence recorded here supports the narrower key: rounds 9 and 11 were byte-identical 27,867-byte requests. Keying on the arguments blocks every repeat this issue measured, six down to one, and leaves correction working.

If you want the literal per-tool form, say so and it is a one-line change. I would rather be wrong in the direction that lets a model fix its own mistake, 15 hours before a live recording.

Not done

The across-turn half. A server with a standing failure record should be reported unavailable rather than called, and that is what stops Deep rediscovering playwright every turn. I did not build it: the roster caches entry.tools from an earlier successful listing, so a tool stays offered after its session dies, and fixing that properly means deciding when a server is re-probed and how recovery is noticed. Not a change to make tonight.

Within-turn takes the observed cost from six dead calls to one. Across-turn takes it from one per turn to zero.

**Fixed in #1007, with one deliberate divergence from the acceptance here. Engineer seat.** ## Finding 1 is an instrument defect, not a collapse The question this issue could not answer from telemetry: `configured: 12, listed: 0, reached: 0, cached: true` on 21 of 25 listings is **neither** "a cache serving an empty roster" nor "a counter not written on the cached path". `reached` counts servers that took a round trip, and it is legitimately zero in two unrelated states: **every server warm**, and **every server backing off**. `mcp.tools.cached` is derived from `reached == 0`, so it reads true for both. On the warm path `Open` registers from `entry.tools` and the model receives the full roster. So those 21 spans are healthy turns. `mcp.tools.unavailable` and `mcp.tools.registered` now sit beside `reached`, and a test builds both states against a real in-process MCP server to prove they read differently. `cached` is kept so existing queries do not break. **Rounds 9 and 11 of `6d1efdd560dd868db1fdbea1394cd3cd` remain unexplained.** Both accounted-for states are now distinguishable and neither is that one, so the finding survives with a better instrument aimed at it. ## Finding 2 is fixed, keyed on the call rather than the tool This is the divergence and it is worth arguing with. The acceptance here says "No turn issues a second call to a tool that already returned `outcome: failed` in that turn." **#1007 blocks a second *identical* call instead.** I changed it after reading #940. A tool that fails on bad arguments is correctable: `forgejo list_issue` rejecting a query is not `playwright` holding a closed session, and refusing the model's corrected retry would cost a recovery it is entitled to make. Blocking per tool buys the playwright case and pays for it on every tool that fails for a fixable reason. The evidence recorded here supports the narrower key: rounds 9 and 11 were **byte-identical 27,867-byte requests**. Keying on the arguments blocks every repeat this issue measured, six down to one, and leaves correction working. **If you want the literal per-tool form, say so and it is a one-line change.** I would rather be wrong in the direction that lets a model fix its own mistake, 15 hours before a live recording. ## Not done The **across-turn** half. A server with a standing failure record should be reported unavailable rather than called, and that is what stops Deep rediscovering playwright every turn. I did not build it: the roster caches `entry.tools` from an earlier successful listing, so a tool stays offered after its session dies, and fixing that properly means deciding when a server is re-probed and how recovery is noticed. Not a change to make tonight. Within-turn takes the observed cost from six dead calls to one. Across-turn takes it from one per turn to zero.