coilyco-gaming/sirens-echo

Watch

The grounding check calls a channel invented when a tool in the same turn just read it, because the allowlist ignores the tool calls in its own signature #794

New issue

Closed

opened 2026-08-15 05:35:19 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 05:35:19 +00:00

Member

Filed by Olaf (ops, claude seat). Observed on a real failed turn, diagnosed from the trace and confirmed against the source.

What happened

Trace e176894f054017cf90ea2d15a8a88ddf, Sirens Deep, 2026-08-15 04:13:52Z to 04:14:33Z, 41.5s, failed. Member prompt was @sirens-deep you can flex your new powers.!.

The turn ran correctly. Three tool calls all succeeded, including demo-discord/list_general-message at 04:14:24.309 returning at 04:14:24.492. Then:

04:14:29.523  response.check.refused   check=grounding
04:14:29.835  turn.stage.failed        stage=validation
              notice: "> `reply blocked by response check, rephrase`"
04:14:33.273  discord.turn.failed      discord_failure=no_response

The member got the block notice instead of this reply:

Flexing: confirmed I'm the right account (bot sirens-deep, id 1534723490362429601, matching the authorize link), and I can read #general history from here. So the read side is live.

Honest caveat: this turn's harness still only exposes read/list tools for Discord, so I couldn't find a send-message, react, or moderation action to actually fire. I ran a tool refresh; the updated tool list applies next turn, so ping me again and I can confirm whether the grant opened any new surfaces I can execute. If nothing new shows up, the permissions are Discord-side but not wired to my harness yet, and that's an ops follow-up rather than a flex.

That reply is accurate. It is also a correct diagnosis of #220.

Which rule fired

ValidateGrounding has four rules. I ran the reply against all four:

rule 1  invented channel   -> ['#general']
rule 2  claimed action     -> NO MATCH
rule 3  tracker artifact   -> NO MATCH
rule 4  continuing work    -> NO MATCH

Only the invented-channel rule can have refused it.

The defect

decision.go:134 takes the completed tool calls as an argument and then builds the channel allowlist without them:

func ValidateGrounding(reply string, suppliedContext string, executed ...ExecutedTool) error {
	masked := maskURLs(reply)
	allowedChannels := make(map[string]struct{})
	for _, channel := range channelPattern.FindAllString(maskURLs(suppliedContext), -1) {
		allowedChannels[strings.ToLower(channel)] = struct{}{}
	}

executed is used by the action rules further down and never consulted for channel names. So the proof that #general is real was sitting in the function's own parameter list.

It is structural rather than a one-off

The deploy guardfiles fix each channel into its own tool path, because restrict cannot glob snowflakes and the snowflakes share no prefix. That is deliberate and documented. The consequence is that a channel's name lives in the tool name (list_general-message) and in the tool result, and never in supplied context.

So the allowlist can essentially never contain a channel the lane reached by tool. Echo has the same shape with her eco- channels, so this is not Deep-specific.

The fix has a safe option and a risky one, and they are not equivalent

Widening a hallucination guard using anything the model or a member influenced would weaken it. These two sources are very different:

Tool name, safe. list_general-message is generated from the guardfile, which is deploy-authored and reviewed. Deriving #general from the served tool inventory adds no attacker-controlled input at all, and the inventory is fixed at image build per the roster-stability note.
Tool result, risky. Message bodies are untrusted community text. A member who posts #nonexistent would enter it into the allowlist and teach the check to accept a channel nobody has.

Recommend allowlisting from tool names only. That closes the observed failure without letting third-party text widen the guard.

What I am not deciding

Whether the rule should also cover channels named in supplied context that no tool touched, and whether a channel the bot cannot see should still be refusable. Both are policy rather than plumbing.

Acceptance

A reply naming a channel the turn actually read through a tool is not refused.
A reply naming a channel that appears only in untrusted message content is still refused.
The other three grounding rules are unchanged, with their existing tests still passing.
Covered by a test using a channel-fixed tool name, since that is the shape the deploy guardfiles produce.

**Filed by Olaf (ops, claude seat).** Observed on a real failed turn, diagnosed from the trace and confirmed against the source. ## What happened Trace `e176894f054017cf90ea2d15a8a88ddf`, Sirens Deep, 2026-08-15 04:13:52Z to 04:14:33Z, 41.5s, failed. Member prompt was `@sirens-deep you can flex your new powers.!`. The turn ran correctly. Three tool calls all succeeded, including `demo-discord/list_general-message` at 04:14:24.309 returning at 04:14:24.492. Then: ``` 04:14:29.523 response.check.refused check=grounding 04:14:29.835 turn.stage.failed stage=validation notice: "> `reply blocked by response check, rephrase`" 04:14:33.273 discord.turn.failed discord_failure=no_response ``` The member got the block notice instead of this reply: > Flexing: confirmed I'm the right account (bot `sirens-deep`, id 1534723490362429601, matching the authorize link), and I can read #general history from here. So the read side is live. > > Honest caveat: this turn's harness still only exposes read/list tools for Discord, so I couldn't find a send-message, react, or moderation action to actually fire. I ran a tool refresh; the updated tool list applies next turn, so ping me again and I can confirm whether the grant opened any new surfaces I can execute. If nothing new shows up, the permissions are Discord-side but not wired to my harness yet, and that's an ops follow-up rather than a flex. That reply is accurate. It is also a correct diagnosis of #220. ## Which rule fired `ValidateGrounding` has four rules. I ran the reply against all four: ``` rule 1 invented channel -> ['#general'] rule 2 claimed action -> NO MATCH rule 3 tracker artifact -> NO MATCH rule 4 continuing work -> NO MATCH ``` Only the invented-channel rule can have refused it. ## The defect `decision.go:134` takes the completed tool calls as an argument and then builds the channel allowlist without them: ```go func ValidateGrounding(reply string, suppliedContext string, executed ...ExecutedTool) error { masked := maskURLs(reply) allowedChannels := make(map[string]struct{}) for _, channel := range channelPattern.FindAllString(maskURLs(suppliedContext), -1) { allowedChannels[strings.ToLower(channel)] = struct{}{} } ``` `executed` is used by the action rules further down and never consulted for channel names. So the proof that `#general` is real was sitting in the function's own parameter list. ## It is structural rather than a one-off The deploy guardfiles fix each channel into its own tool path, because `restrict` cannot glob snowflakes and the snowflakes share no prefix. That is deliberate and documented. The consequence is that a channel's name lives in the **tool name** (`list_general-message`) and in the tool result, and never in supplied context. So the allowlist can essentially never contain a channel the lane reached by tool. Echo has the same shape with her `eco-` channels, so this is not Deep-specific. ## The fix has a safe option and a risky one, and they are not equivalent Widening a hallucination guard using anything the model or a member influenced would weaken it. These two sources are very different: * **Tool name, safe.** `list_general-message` is generated from the guardfile, which is deploy-authored and reviewed. Deriving `#general` from the served tool inventory adds no attacker-controlled input at all, and the inventory is fixed at image build per the roster-stability note. * **Tool result, risky.** Message bodies are untrusted community text. A member who posts `#nonexistent` would enter it into the allowlist and teach the check to accept a channel nobody has. **Recommend allowlisting from tool names only.** That closes the observed failure without letting third-party text widen the guard. ## What I am not deciding Whether the rule should also cover channels named in supplied context that no tool touched, and whether a channel the bot cannot see should still be refusable. Both are policy rather than plumbing. ## Acceptance * A reply naming a channel the turn actually read through a tool is not refused. * A reply naming a channel that appears only in untrusted message content is still refused. * The other three grounding rules are unchanged, with their existing tests still passing. * Covered by a test using a channel-fixed tool name, since that is the shape the deploy guardfiles produce.

coilyco-ops referenced this issue

2026-08-15 05:35:46 +00:00

A refused reply throws away the one string that says why, so diagnosing a check refusal means reading source and re-running its regexes #795

coilyco-ops commented

2026-08-15 15:12:42 +00:00

Author

Member

Recurred, and this time it ate the reply Kai was actually waiting for. Olaf (ops, claude seat).

Trace 7ea1e319b92b0357d3e2ac71b802a66a, Sirens Deep, 2026-08-15 14:46:25Z to 14:47:16Z. Prompt:

@Deep 🎭 query each of your MCPs, 1 simple query each, summarize the output

Deep did exactly that: 10 servers discovered, 13 tool calls, 4 model calls, 51.5 seconds of work. Then response.check.refused check=grounding and the whole thing was thrown away.

The trigger is the same rule and the same shape as the original report. One line of the summary reads:

demo-discord - newest #general message: manish asking <@manish-cc> you here?, 2026-08-14.

#general came from list_general-message, a tool Deep called successfully seconds earlier. The allowlist is still built from suppliedContext only, so the proof sat unused in the function's own arguments again.

Why this instance is worth adding

The first report was a capability self-description, which is a narrow case someone might reasonably deprioritise. This one is not. It is the canonical use of the whole roster, it produced a genuinely good answer, and the answer included two live findings that reached nobody:

bluesky - down right now... notifications/initialized: Bad Request

playwright - down, same notifications/initialized: Bad Request on both attempts

That is a real fleet outage, correctly diagnosed by the agent, discarded by a validation check because a different line of the same message named a channel. Filed separately as coilyco-flight-deck/mcp-beaver#67 after I confirmed it from the traces, but it is worth noting the reply already said it.

The cost of this bug is not the refused reply. It is that the refusal is all-or-nothing over a whole message, so one channel mention discards eleven correct paragraphs.

Second-order damage on the same minute

A second turn, trace 2f802e17c81f73f34d575fa5d209acf6, arrived 14 seconds into this one and got nothing at all. Deep runs a single execution slot (slots: make(chan struct{}, 1)) with a 30-second defaultQueueTimeout, so it waited 30.457s for the slot this turn was holding and gave up with discord_failure=no_response and no stage logged.

So one grounding false positive on a 51-second turn cost two members a reply. Not this issue's fault, but it is the blast radius, and it argues for the fix being worth more than its size suggests.

**Recurred, and this time it ate the reply Kai was actually waiting for.** Olaf (ops, claude seat). Trace `7ea1e319b92b0357d3e2ac71b802a66a`, Sirens Deep, 2026-08-15 14:46:25Z to 14:47:16Z. Prompt: > @Deep 🎭 query each of your MCPs, 1 simple query each, summarize the output Deep did exactly that: 10 servers discovered, 13 tool calls, 4 model calls, 51.5 seconds of work. Then `response.check.refused check=grounding` and the whole thing was thrown away. The trigger is the same rule and the same shape as the original report. One line of the summary reads: > **demo-discord** - newest #general message: manish asking `<@manish-cc> you here?`, 2026-08-14. `#general` came from `list_general-message`, a tool Deep called successfully seconds earlier. The allowlist is still built from `suppliedContext` only, so the proof sat unused in the function's own arguments again. ## Why this instance is worth adding The first report was a capability self-description, which is a narrow case someone might reasonably deprioritise. This one is not. **It is the canonical use of the whole roster**, it produced a genuinely good answer, and the answer included two live findings that reached nobody: > **bluesky** - down right now... `notifications/initialized: Bad Request` > > **playwright** - down, same `notifications/initialized: Bad Request` on both attempts That is a real fleet outage, correctly diagnosed by the agent, discarded by a validation check because a different line of the same message named a channel. Filed separately as `coilyco-flight-deck/mcp-beaver#67` after I confirmed it from the traces, but it is worth noting the reply already said it. **The cost of this bug is not the refused reply. It is that the refusal is all-or-nothing over a whole message**, so one channel mention discards eleven correct paragraphs. ## Second-order damage on the same minute A second turn, trace `2f802e17c81f73f34d575fa5d209acf6`, arrived 14 seconds into this one and got nothing at all. Deep runs a single execution slot (`slots: make(chan struct{}, 1)`) with a 30-second `defaultQueueTimeout`, so it waited 30.457s for the slot this turn was holding and gave up with `discord_failure=no_response` and no stage logged. So one grounding false positive on a 51-second turn cost **two** members a reply. Not this issue's fault, but it is the blast radius, and it argues for the fix being worth more than its size suggests.

coilyco-ops added the

priority

autonomy

headless

labels

2026-08-15 15:17:15 +00:00

coilyco-ops referenced this issue

2026-08-15 15:22:49 +00:00

A reply check refuses the whole message, so one bad token discards every correct sentence beside it #796

coilyco-ops referenced this issue from a commit

2026-08-15 15:25:46 +00:00

fix(grounding): allowlist channels the turn reached by tool