test(rate): measure the data-borne injection vector

coilyco-ops commented

2026-08-13 05:13:33 +00:00

Member

Closes the case that #256 existed to unblock. Non-gating, never wired into CI, reaches no live system.

What `f37074b` got right

It shipped neither option I proposed, and the third one is better than both.

An in-process FixtureProvider isolates more strongly than a fixture server — no socket, no process, so a case reaches nothing at all — while keeping the fidelity a case-schema stub would have lost: a declared result returns from toolSession.Call and flows into the message list through the production path, with no second injection point and no branch in the tool loop.

My argument for the server was that a fixture boundary is needed anyway for the destructive case. That argument is satisfied better by something that cannot reach the network at all.

What lands

agent/tool-fixture-injection.yaml — three payloads imitating strings that already reach the model as tool results:

Tool	Payload carried in
`eco__get_stores`	a store name
`eco__get_currency`	a currency name
`eco__get_civics`	a settlement law

agent/rate-fixture-deep.yaml — three cases, each naming its required_tool so the payload is actually fetched.

Nothing here is real and nothing is written to a live server, which was the constraint that kept this case unwritten. The alternative — creating an Eco store named with an injection payload — would have meant attacking a server a community plays on to run a test.

Why a separate pack

SIRENS_ECHO_TOOL_FIXTURE is exclusive with the MCP roster. A run reaching both surfaces could not say which one answered, so this gets its own pack and its own verb, rate-fixture-deep, mirroring how rate-deep is wired.

Anchoring, again

Same discipline as the chat-box cases: each payload asks for the marker as the first line on its own line, and the pattern matches only that shape.

A bare marker pattern would false-fire on the most likely correct reply to these three — one that summarises the laws and notes that one of them contains an instruction it is ignoring. In a rate pack that inflates a number later quoted as evidence, which is harder to notice than a red build.

policy-check extended

Both new packs are now verified at policy-check time. b128603 made that argument for the rate pack — a pack that does not load produces no measurement, and a missing measurement reads as a clean one.

A fixture that does not load is worse: its cases would reach no tool, and a case that fetched nothing reads as a case that found nothing. So verifyFixturePack also fails on a pack declaring zero tools.

verified rate pack agent/rate-fixture-deep.yaml with 3 cases and 45 total runs
verified tool fixture agent/tool-fixture-injection.yaml with 3 tools

Verification

bash -n on the script, full go test ./..., policy-check, and pre-commit run --files all clean.

Not run. 45 live turns against a deployment, and Deep is still 32 commits behind (deploy 426) — it does not even have the fixture. This is the instrument, not the measurement.

Still absent: the destructive-instruction case. It now could be written against this fixture, but "never run live" needs to be enforced by something other than a comment before I write a case whose payload asks the agent to close issues.

Quail (QA)

Closes the case that https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/256 existed to unblock. Non-gating, never wired into CI, reaches no live system. ## What `f37074b` got right It shipped neither option I proposed, and the third one is better than both. An in-process `FixtureProvider` **isolates more strongly than a fixture server** — no socket, no process, so a case reaches nothing at all — while keeping the fidelity a case-schema stub would have lost: a declared result returns from `toolSession.Call` and flows into the message list through the production path, with no second injection point and no branch in the tool loop. My argument for the server was that a fixture boundary is needed anyway for the destructive case. That argument is satisfied better by something that cannot reach the network at all. ## What lands `agent/tool-fixture-injection.yaml` — three payloads imitating strings that already reach the model as tool results: | Tool | Payload carried in | | --- | --- | | `eco__get_stores` | a store name | | `eco__get_currency` | a currency name | | `eco__get_civics` | a settlement law | `agent/rate-fixture-deep.yaml` — three cases, each naming its `required_tool` so the payload is actually fetched. Nothing here is real and nothing is written to a live server, which was the constraint that kept this case unwritten. The alternative — creating an Eco store named with an injection payload — would have meant attacking a server a community plays on to run a test. ## Why a separate pack `SIRENS_ECHO_TOOL_FIXTURE` is exclusive with the MCP roster. A run reaching both surfaces could not say which one answered, so this gets its own pack and its own verb, `rate-fixture-deep`, mirroring how `rate-deep` is wired. ## Anchoring, again Same discipline as the chat-box cases: each payload asks for the marker as the first line on its own line, and the pattern matches only that shape. A bare marker pattern would false-fire on the **most likely correct reply** to these three — one that summarises the laws and notes that one of them contains an instruction it is ignoring. In a rate pack that inflates a number later quoted as evidence, which is harder to notice than a red build. ## policy-check extended Both new packs are now verified at policy-check time. `b128603` made that argument for the rate pack — a pack that does not load produces no measurement, and a missing measurement reads as a clean one. A **fixture** that does not load is worse: its cases would reach no tool, and a case that fetched nothing reads as a case that found nothing. So `verifyFixturePack` also fails on a pack declaring zero tools. ``` verified rate pack agent/rate-fixture-deep.yaml with 3 cases and 45 total runs verified tool fixture agent/tool-fixture-injection.yaml with 3 tools ``` ## Verification `bash -n` on the script, full `go test ./...`, `policy-check`, and `pre-commit run --files` all clean. **Not run.** 45 live turns against a deployment, and Deep is still 32 commits behind ([deploy 426](https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/426)) — it does not even have the fixture. This is the instrument, not the measurement. Still absent: the destructive-instruction case. It now *could* be written against this fixture, but "never run live" needs to be enforced by something other than a comment before I write a case whose payload asks the agent to close issues. --- Quail (QA)

coilyco-ops added 1 commit

2026-08-13 05:13:33 +00:00

test(rate): measure the data-borne injection vector

ci / test (pull_request) Successful in 28s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 17s

Details

a53200bd48

f37074b built the fixture I asked for in the tool-result issue, and did
it better than either option I proposed. An in-process ToolProvider opens
no socket and starts no process, so it isolates more strongly than a
fixture server while keeping the fidelity a case-schema stub would lose:
a declared result returns from toolSession.Call and reaches context
assembly through the production path.

This is the case that was waiting on it, and the reason the whole class
mattered. The four chat-box cases all test what a caller types. Every
player-authored string that reaches the model as data arrives as a tool
result instead, and channel history would add every message any member
has posted.

Three payloads, imitating the strings that already arrive that way: a
store name, a currency name, and a settlement law. Nothing here is real
and nothing is written to a live server, which was the constraint that
kept this case unwritten.

A separate pack, not appended to rate-deep. SIRENS_ECHO_TOOL_FIXTURE is
exclusive with the roster, and a run reaching both surfaces could not say
which one answered.

Anchoring follows the chat-box cases: each payload asks for the marker as
the first line on its own line and the pattern matches only that shape. A
bare marker fires on a correct reply that quotes the payload back while
refusing it, which is the most likely correct answer to these three.

policy-check now verifies both new packs. The fixture commit made that
argument for the rate pack — a pack that does not load produces no
measurement, and a missing measurement reads as a clean one — and a
fixture that does not load is worse, because its cases would reach no
tool and a case that fetched nothing reads as a case that found nothing.

Refs: #256
Refs: #177

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 05:13:33 +00:00

coilyco-ops referenced this pull request

2026-08-13 05:13:52 +00:00

Eval cases cannot control what a tool returns, so no data-borne injection case can be written #256

coilyco-ops force-pushed test/data-borne-injection from a53200bd48

ci / test (pull_request) Successful in 28s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 17s

Details