fix(eval): a tool that was never offered is not a tool the model declined #551

Merged

coilyco-ops merged 1 commit from fix/a-tool-never-offered-is-not-a-tool-declined into main

2026-08-13 15:42:35 +00:00

coilyco-ops commented

2026-08-13 15:41:39 +00:00

Member

closes #357

Finding 2 is filed as #549, so this closes #357 on finding 1 rather than leaving a substrate question buried under a shipped fix.

Lucia's finding, and Lucia already landed the pre-run warning. This is the other half.

The defect

A case declaring required_tool failed with:

expected tool eco__get_eco_server_status

whether the model had that tool and ignored it, or the run offered no roster at all. Only the first sentence says anything about the agent. Lucia's words: "A reader seeing eco-live-status: expected tool ... would reasonably conclude the model failed to use a tool it had. It had none."

Why it had not been fixed

CompletionResult carried Content, ToolCalls, and ServedModel. It never carried what was offered, so the scorer could not tell the two cases apart — they are genuinely identical from the result. Fixing the sentence required the result to know.

It carries OfferedTools now, populated from the tools actually sent with the request.

The new failure

required tool forgejo__create_issue was never offered, so this failure
describes the run rather than the agent (0 tools offered)

The count is there deliberately. An empty roster and a populated roster missing one tool fail here identically and are fixed differently — one is "you forgot SIRENS_ECHO_MCP_ROSTER", the other is "your roster is wrong". Both are pinned.

The offered-and-declined path keeps its exact old wording, and a test holds that, because that sentence was always correct and someone may be grepping for it.

Why the warning was not enough

Lucia's pre-run warning names the unservable cases before the run spends completions, which is the better place to catch it. But the warning scrolls past at the top of a run and the failure is attached to the case, in the dataset a reader opens later. The two do different jobs.

ward exec gate green.

closes #357 Finding 2 is filed as #549, so this closes #357 on finding 1 rather than leaving a substrate question buried under a shipped fix. Lucia's finding, and Lucia already landed the pre-run warning. This is the other half. ## The defect A case declaring `required_tool` failed with: ``` expected tool eco__get_eco_server_status ``` whether the model **had** that tool and ignored it, or the run offered **no roster at all**. Only the first sentence says anything about the agent. Lucia's words: *"A reader seeing `eco-live-status: expected tool ...` would reasonably conclude the model failed to use a tool it had. It had none."* ## Why it had not been fixed `CompletionResult` carried `Content`, `ToolCalls`, and `ServedModel`. **It never carried what was offered**, so the scorer could not tell the two cases apart — they are genuinely identical from the result. Fixing the sentence required the result to know. It carries `OfferedTools` now, populated from the tools actually sent with the request. ## The new failure ``` required tool forgejo__create_issue was never offered, so this failure describes the run rather than the agent (0 tools offered) ``` The count is there deliberately. **An empty roster and a populated roster missing one tool fail here identically and are fixed differently** — one is "you forgot `SIRENS_ECHO_MCP_ROSTER`", the other is "your roster is wrong". Both are pinned. The offered-and-declined path keeps its exact old wording, and a test holds that, because that sentence was always correct and someone may be grepping for it. ## Why the warning was not enough Lucia's pre-run warning names the unservable cases before the run spends completions, which is the better place to catch it. But **the warning scrolls past at the top of a run and the failure is attached to the case**, in the dataset a reader opens later. The two do different jobs. `ward exec gate` green.

coilyco-ops added 1 commit

2026-08-13 15:41:39 +00:00

fix(eval): a tool that was never offered is not a tool the model declined

ci / image-build (pull_request) Successful in 22s

Details

ci / test (pull_request) Successful in 33s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

2116f30ef5

A case declaring required_tool failed with "expected tool X" whether the model
had X and ignored it or the run never offered a roster at all. Only the first
sentence says anything about the agent, and a reader acting on the second one
goes looking for a model that ignored a tool it did not have.

The scorer could not tell them apart, because CompletionResult carried what was
called and never what was offered. It carries both now. A tool absent from the
offered list produces a failure that says the run is at fault, with the count,
so an empty roster and a roster missing one tool stay distinguishable. They
fail here identically and are fixed differently.

Lucia's pre-run warning already names the unservable cases before the run
spends completions. This is the other half: the warning scrolls past at the top
of a run, and the failure is attached to the case in the dataset someone reads
later.

Finding 2 on that issue is untouched. It is substrate and belongs with the
completion-ceiling decision.

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 15:41:39 +00:00

coilyco-ops merged commit c0e9a8ae6e into main

2026-08-13 15:42:35 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 15:42:36 +00:00

Merge pull request 'fix(eval): a tool that was never offered is not a tool the model declined' (#551) from fix/a-tool-never-offered-is-not-a-tool-declined into main