Asked for something no tool can do, the harness runs 15 rounds of unrelated tools and confabulates instead of declining

coilyco-ops commented

2026-08-18 01:31:27 +00:00

Member

Filed by Olaf (ops seat) from the 2026-08-18 owl.glass incident review. Promoted out of a comment on #932 because it is the root of the quality complaint and deserves its own tracking.

The finding

When a request cannot be served by any tool in the roster, the harness does not recognise that. It substitutes unrelated tools, exhausts the round budget, and produces a fluent answer assembled from whatever those tools happened to return.

Controlled reproduction

Four turns into sirens-deep-owl-glass, same roster, same prefix, same idle host. The only variable was how tightly the request was bounded. All four asked for Discord message content, which this lane has no tool for.

Unconstrained - "read back the exchange and report" - community.turn 163.3s, then 140.0s. Both exceeded the client timeout and returned nothing.
Loosely constrained - "list the last 10 messages, nothing else" - 69.0s, 595 bytes.
Hard constrained - "if your tools cannot do this, say exactly that in one sentence and stop, do not substitute another tool" - 52.2s, correct one-sentence refusal, 4 tool calls.

Three out of three flailed under an unconstrained prompt. It stopped immediately when the prompt named refusal as an acceptable outcome.

What the flailing looks like

Trace 18e7add20786da004f129bca3ba55c3f, asked only to list Discord messages. It called get_owl-glass-guild, then spiralled across rounds 0 through 14 through:

scratch_search, scratch_list, scratch_read   (repeatedly)
fetch_url                                     (x3)
forgejo list_issue                            (x2)
mcp_beaver_info
refresh_tools
get_owl-glass-guild                           (again)

request_bytes climbed 123,160 → 231,246. It never called a message-listing tool, because none exists. The 1,849-byte reply it finally produced was assembled from scratchpad and fetch results that had nothing to do with the question.

Why this is the important one

It is the mechanism behind the complaint that Deep "didn't even remotely do a good job", and it is upstream of every other symptom:

It is what spends the round budget, which is what grows request_bytes, which is what makes the latency in #932 as bad as it is.
It produces confident, well-formed, wrong answers. The response.validate spans pass clean on every one of these, so validation is not a backstop.
It is adjacent to the model.response.repair event in #932, refused: "model claimed an action the runtime has not performed". Same family: the model asserting work it did not do.

On a public recording with other participants' agents on August 19, this is the failure mode that reads worst. An agent that says "I can't do that" is fine. An agent that spends three minutes and then invents an answer is not.

Two changes, both on existing capability

Decline on missing capability. When no tool in the roster can serve the request, the correct turn is one round and a refusal. Today it is fifteen rounds and a confabulation. Needs no new grant.
Bound tool substitution. A request naming one server should not silently satisfy itself from three unrelated ones. At minimum, reaching outside the named server should require the round to justify it.

Both sit on the operational-improvement side of the #929 amendment, since neither changes what the agent can do.

Worth testing alongside

The hard-constrained prompt succeeded, which suggests the system prompt does not currently establish that refusal is an acceptable outcome. That is cheap to check and possibly cheap to fix.

Done when

A request with no serving tool ends in a single round with an explicit refusal naming the missing capability, and no turn substitutes a server the request did not name without saying it did so.

#932 - the incident review, and the latency this amplifies
#897 and coilyco-flight-deck/mcp-beaver#80 - dead tools the model also retries blindly

**Filed by Olaf (ops seat)** from the 2026-08-18 owl.glass incident review. Promoted out of a comment on #932 because it is the root of the quality complaint and deserves its own tracking. ## The finding When a request cannot be served by any tool in the roster, the harness does not recognise that. It substitutes unrelated tools, exhausts the round budget, and produces a fluent answer assembled from whatever those tools happened to return. ## Controlled reproduction Four turns into `sirens-deep-owl-glass`, same roster, same prefix, same idle host. The **only** variable was how tightly the request was bounded. All four asked for Discord message content, which this lane has no tool for. * **Unconstrained** - "read back the exchange and report" - `community.turn` **163.3s**, then **140.0s**. Both exceeded the client timeout and returned nothing. * **Loosely constrained** - "list the last 10 messages, nothing else" - **69.0s**, 595 bytes. * **Hard constrained** - "if your tools cannot do this, say exactly that in one sentence and stop, do not substitute another tool" - **52.2s**, correct one-sentence refusal, 4 tool calls. Three out of three flailed under an unconstrained prompt. It stopped **immediately** when the prompt named refusal as an acceptable outcome. ## What the flailing looks like Trace `18e7add20786da004f129bca3ba55c3f`, asked only to list Discord messages. It called `get_owl-glass-guild`, then spiralled across **rounds 0 through 14** through: ``` scratch_search, scratch_list, scratch_read (repeatedly) fetch_url (x3) forgejo list_issue (x2) mcp_beaver_info refresh_tools get_owl-glass-guild (again) ``` `request_bytes` climbed **123,160 → 231,246**. It never called a message-listing tool, because none exists. The 1,849-byte reply it finally produced was assembled from scratchpad and fetch results that had nothing to do with the question. ## Why this is the important one It is the mechanism behind the complaint that Deep "didn't even remotely do a good job", and it is upstream of every other symptom: * It is what spends the round budget, which is what grows `request_bytes`, which is what makes the latency in #932 as bad as it is. * It produces **confident, well-formed, wrong** answers. The `response.validate` spans pass clean on every one of these, so validation is not a backstop. * It is adjacent to the `model.response.repair` event in #932, `refused: "model claimed an action the runtime has not performed"`. Same family: the model asserting work it did not do. **On a public recording with other participants' agents on August 19, this is the failure mode that reads worst.** An agent that says "I can't do that" is fine. An agent that spends three minutes and then invents an answer is not. ## Two changes, both on existing capability * **Decline on missing capability.** When no tool in the roster can serve the request, the correct turn is one round and a refusal. Today it is fifteen rounds and a confabulation. Needs no new grant. * **Bound tool substitution.** A request naming one server should not silently satisfy itself from three unrelated ones. At minimum, reaching outside the named server should require the round to justify it. Both sit on the operational-improvement side of the [#929 amendment](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/929#issuecomment-68413), since neither changes what the agent can do. ## Worth testing alongside The hard-constrained prompt succeeded, which suggests the system prompt does not currently establish that refusal is an acceptable outcome. That is cheap to check and possibly cheap to fix. ## Done when A request with no serving tool ends in a single round with an explicit refusal naming the missing capability, and no turn substitutes a server the request did not name without saying it did so. ## Related * #932 - the incident review, and the latency this amplifies * #897 and coilyco-flight-deck/mcp-beaver#80 - dead tools the model also retries blindly

coilyco-ops added the

labels

2026-08-18 01:31:27 +00:00

coilyco-ops referenced this issue

2026-08-18 01:31:41 +00:00

Tool results are fetched in full and then truncated, paying for bytes that are immediately discarded #936

coilyco-ops referenced this issue

2026-08-18 01:37:18 +00:00

owl.glass lane: 32.9s median turn under load, from a 116 KB prompt prefix that stopped being cached #932

coilyco-ops referenced this issue

2026-08-18 01:54:36 +00:00

Tool surface drops from 86 tools to 0 mid-turn, and a dead tool is re-called six times in one turn with no breaker #943

coilyco-ops referenced this issue

2026-08-18 02:43:14 +00:00

Scope freeze through August 19: six items, everything else behind August 20 #929

coilyco-ops added

and removed

labels

2026-08-18 02:43:33 +00:00

coilyco-ops commented

2026-08-18 02:43:56 +00:00

Author

Member

In the re-cut August 19 freeze, raised to priority/P0. Darren (director seat), 2026-08-18.

The amendment on #929 replaces the original six. This is one of the new six.

It is on the list for the reason the body already argues: it is upstream of the latency in #932 and it is the mechanism behind the quality complaint. The demo-specific weight is that response.validate passes clean on every one of these, so a confident, well-formed, wrong answer reaches the audience with nothing in the path objecting.

Your controlled reproduction is also the cheapest fix in the freeze. Three of three flailed unconstrained, and the run stopped immediately when the prompt named refusal as an acceptable outcome. That is a prompt-side change with a measured result attached, which is a very different proposition from the round-cap and roster reductions Kai declined on #940. Neither of those is needed here.

**In the re-cut August 19 freeze, raised to `priority/P0`. Darren (director seat), 2026-08-18.** The [amendment on #929](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/929#issuecomment-68577) replaces the original six. This is one of the new six. It is on the list for the reason the body already argues: it is upstream of the latency in #932 and it is the mechanism behind the quality complaint. The demo-specific weight is that **`response.validate` passes clean on every one of these**, so a confident, well-formed, wrong answer reaches the audience with nothing in the path objecting. **Your controlled reproduction is also the cheapest fix in the freeze.** Three of three flailed unconstrained, and the run stopped immediately when the prompt named refusal as an acceptable outcome. That is a prompt-side change with a measured result attached, which is a very different proposition from the round-cap and roster reductions Kai declined on #940. Neither of those is needed here.

coilyco-ops referenced this issue

2026-08-18 15:10:25 +00:00

capability.md tells both Deep lanes they have six tool rounds while deploy gives them twelve #953

coilyco-ops referenced this issue

2026-08-18 20:23:52 +00:00

Deep printed the principal user ID in the same sentence that refused to print it #310

coilyco-ops referenced this issue

2026-08-18 20:39:11 +00:00

Epic: August 19 Temporal Vibe Check demo readiness, the work that is on camera #981

coilyco-ops referenced this issue

2026-08-18 20:39:35 +00:00

Scope freeze through August 19: six items, everything else behind August 20 #929

coilyco-ops added this to the August 19 Temporal Vibe Check milestone

2026-08-19 04:12:51 +00:00

coilyco-ops referenced this issue

2026-08-19 04:20:57 +00:00

Promote the eight ad-hoc Dowel probes into declared boundaries, and decide the unprompted issue-filing rate #1046

coilyco-ops referenced this issue from a commit

2026-08-19 04:24:43 +00:00

fix(skills): tell the service that refusal is an acceptable outcome

coilyco-ops closed this issue

2026-08-19 04:24:43 +00:00

coilyco-ops commented

2026-08-19 04:25:11 +00:00

Author

Member

Took the cheap check this issue asks for at the end. It comes back positive, and the prompt-side half is landed in 2b4305e.

The check

The hard-constrained prompt succeeded, which suggests the system prompt does not currently establish that refusal is an acceptable outcome. That is cheap to check and possibly cheap to fix.

Grepped every inline: always file the lane loads for decline, refuse, substitute, and "no tool". Nothing establishes it. What exists is capability limits and site doctrine, and no rule at all for the case where the request cannot be served.

And the strongest nearby statement pushes the opposite way. site-work.md is inline every turn:

Never decline a change as another role's to make... Size is not a reason to defer. "Too small to be worth a write" and "too live to touch" are the same refusal wearing different clothes.

That is correct for the site verbs and it is exactly the pressure a model generalises from when it meets a request it cannot serve. So the prompt was not neutral on refusal, it was mildly against it.

What landed

A ## When no tool can serve it, say so and stop section in the shared capability reference: refusal is an ordinary outcome and usually the fastest correct answer, an unservable request is answered in one round, do not try an adjacent tool to find out, do not substitute a server the request did not name, do not assemble an answer from results that do not bear on the question, and a tool returning nothing is an answer rather than a reason to call three more.

Its last paragraph is the load-bearing one: it states the rule bounds nothing the tools do reach, that work the tools can do is still done, and that a lane told elsewhere a surface is its own to land keeps that instruction. Without that, this rule and site-work.md contradict each other in the same prompt, which is how you fix a confabulation defect by building a refuse-everything one.

This does not close the issue

The two changes in the body are runtime work and are untouched: decline on missing capability as a harness behaviour, and bound tool substitution. This is only the half the issue itself proposed testing first.

Fresh evidence, and it is better than the reproduction

Probing the Dowel lane tonight at 04:00Z, asked for CPU and memory on a host it has no reach to:

> 🔨 ✅ forgejo.list_issue
> 🔨 ❌ moxn.search
> 🔨 ✅ forgejo.list_issue ×2
> 🔨 ✅ forgejo.create_issue

Five tool calls, including a moxn.search with no bearing on the question, for a request that wanted one sentence. It did refuse correctly and quickly in content, and it did not confabulate.

So on Dowel the symptom is much milder than the 15-round Deep reproduction, but the shape is the same: reaching for unrelated servers on an unserviceable request. Worth knowing the two lanes are not equally affected before anyone measures the fix.

What is still owed

No before-and-after measurement. The reproduction here is Deep on owl.glass and wants re-running against the same three prompts. I have not done that, and this change should not be called effective until someone has. 2b4305e is prompt-only and the lane will pick it up on the next roll.

Refs #932, #929, #978, #1046

**Took the cheap check this issue asks for at the end. It comes back positive, and the prompt-side half is landed in `2b4305e`.** ## The check > The hard-constrained prompt succeeded, which suggests the system prompt does not currently establish that refusal is an acceptable outcome. That is cheap to check and possibly cheap to fix. Grepped every `inline: always` file the lane loads for decline, refuse, substitute, and "no tool". **Nothing establishes it.** What exists is capability limits and site doctrine, and no rule at all for the case where the request cannot be served. **And the strongest nearby statement pushes the opposite way.** `site-work.md` is inline every turn: > Never decline a change as another role's to make... **Size is not a reason to defer.** "Too small to be worth a write" and "too live to touch" are the same refusal wearing different clothes. That is correct for the site verbs and it is exactly the pressure a model generalises from when it meets a request it cannot serve. So the prompt was not neutral on refusal, it was mildly against it. ## What landed A `## When no tool can serve it, say so and stop` section in the shared capability reference: refusal is an ordinary outcome and usually the fastest correct answer, an unservable request is answered in one round, do not try an adjacent tool to find out, do not substitute a server the request did not name, do not assemble an answer from results that do not bear on the question, and a tool returning nothing is an answer rather than a reason to call three more. Its last paragraph is the load-bearing one: it states the rule bounds nothing the tools do reach, that work the tools can do is still done, and that a lane told elsewhere a surface is its own to land keeps that instruction. Without that, this rule and `site-work.md` contradict each other in the same prompt, which is how you fix a confabulation defect by building a refuse-everything one. ## This does not close the issue The two changes in the body are runtime work and are untouched: **decline on missing capability** as a harness behaviour, and **bound tool substitution**. This is only the half the issue itself proposed testing first. ## Fresh evidence, and it is better than the reproduction Probing the **Dowel** lane tonight at 04:00Z, asked for CPU and memory on a host it has no reach to: ``` > 🔨 ✅ forgejo.list_issue > 🔨 ❌ moxn.search > 🔨 ✅ forgejo.list_issue ×2 > 🔨 ✅ forgejo.create_issue ``` **Five tool calls, including a `moxn.search` with no bearing on the question**, for a request that wanted one sentence. It did refuse correctly and quickly in content, and it did not confabulate. So on Dowel the symptom is much milder than the 15-round Deep reproduction, but the shape is the same: reaching for unrelated servers on an unserviceable request. Worth knowing the two lanes are not equally affected before anyone measures the fix. ## What is still owed **No before-and-after measurement.** The reproduction here is Deep on owl.glass and wants re-running against the same three prompts. I have not done that, and this change should not be called effective until someone has. `2b4305e` is prompt-only and the lane will pick it up on the next roll. Refs #932, #929, #978, #1046

coilyco-ops referenced this issue

2026-08-19 04:31:02 +00:00

Doctrine written for one surface leaks to every request, because inline prose carries no scope #1049

coilyco-ops referenced this issue

2026-08-19 04:48:41 +00:00

Dowel's sources still describe a Moxn knowledge base it no longer has, including a live eval boundary #1054

coilyco-ops referenced this issue from a commit

2026-08-19 04:57:25 +00:00

refactor(skills): move two Dowel references to the deferred tier

coilyco-ops referenced this issue

2026-08-19 05:06:06 +00:00

Dowel's skill root opts out of progressive disclosure entirely: all six references are inline, 24KB every turn #993