test(tools): a partial-coverage result classifies as a plain success #744

Merged

coilyco-ops merged 1 commit from quail/partial-coverage-outcome into main

2026-08-13 22:03:56 +00:00

coilyco-ops commented

2026-08-13 22:02:19 +00:00

Member

Pins the classification that makes sirens-echo#449 invisible. Tests only.

The shape

func outcomeOf(result ToolResult) ToolOutcome {
    if result.IsError { return ToolOutcomeFailed }
    if strings.TrimSpace(result.Text) == "" { return ToolOutcomeEmpty }
    return ToolOutcomeOK
}

The result a member received "none exists" from carried text — a rollup warning and No markets matched item='wooden hull plank' across 528 ledger rows. Not an error, not blank, therefore ok, identical to a call that returned every row.

The test

Asserts the verbatim result from sirens-echo#449 classifies the same as a complete one. Mutation output from giving it its own state:

a partial-coverage result now reports "empty" against "ok" for a complete
one. If sirens-echo#449 gained a state, assert the new one here instead

The second test pins failed and empty separately, so the first cannot pass by outcomeOf collapsing to a single value — an equality assertion between two calls is exactly the shape that goes vacuously green if the function stops discriminating.

Measured, and recorded in a comment

24h of production tool calls:

ok      124
failed    5
empty     0

ToolOutcomeEmpty never fires. Real tools answer "nothing matched" in prose rather than returning a blank string, so the one state that gestures at absent data is unreachable in practice and every found-nothing result is an ok.

That is worth knowing before a fourth state is added: the third has never been hit.

Not a fix

sirens-echo#449 needs either a coverage field on the result envelope or a validator that can see one. Both are decisions on that issue. This only ensures the current behaviour is asserted rather than assumed, and that changing it is deliberate.

ward exec gate PASS on every step.

Pins the classification that makes sirens-echo#449 invisible. Tests only. ## The shape ```go func outcomeOf(result ToolResult) ToolOutcome { if result.IsError { return ToolOutcomeFailed } if strings.TrimSpace(result.Text) == "" { return ToolOutcomeEmpty } return ToolOutcomeOK } ``` The result a member received *"none exists"* from carried text — a rollup warning and `No markets matched item='wooden hull plank' across 528 ledger rows`. Not an error, not blank, therefore **`ok`, identical to a call that returned every row**. ## The test Asserts the verbatim result from sirens-echo#449 classifies the same as a complete one. Mutation output from giving it its own state: ``` a partial-coverage result now reports "empty" against "ok" for a complete one. If sirens-echo#449 gained a state, assert the new one here instead ``` The second test pins `failed` and `empty` separately, so the first cannot pass by `outcomeOf` collapsing to a single value — an equality assertion between two calls is exactly the shape that goes vacuously green if the function stops discriminating. ## Measured, and recorded in a comment 24h of production tool calls: ``` ok 124 failed 5 empty 0 ``` **`ToolOutcomeEmpty` never fires.** Real tools answer "nothing matched" in prose rather than returning a blank string, so the one state that gestures at absent data is unreachable in practice and every found-nothing result is an `ok`. That is worth knowing before a fourth state is added: the third has never been hit. ## Not a fix sirens-echo#449 needs either a coverage field on the result envelope or a validator that can see one. Both are decisions on that issue. This only ensures the current behaviour is asserted rather than assumed, and that changing it is deliberate. `ward exec gate` PASS on every step.

coilyco-ops added 1 commit

2026-08-13 22:02:19 +00:00

test(tools): a partial-coverage result classifies as a plain success

ci / image-build (pull_request) Successful in 25s

Details

ci / test (pull_request) Successful in 38s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

17d39349f1

outcomeOf reads the envelope: an error, a blank string, or anything else.
The result on 449 said it searched 528 of 22,933 rows and matched none,
which is anything else, so it reports ok like a complete answer.

Pins the verbatim shape from that issue, and pins the other two states so
the assertion cannot pass by outcomeOf collapsing to one value.

Refs sirens-echo#449

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 22:02:19 +00:00

coilyco-ops merged commit b6909eeb85 into main

2026-08-13 22:03:56 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 22:03:57 +00:00

Merge pull request 'test(tools): a partial-coverage result classifies as a plain success' (#744) from quail/partial-coverage-outcome into main

coilyco-ops referenced this pull request

2026-08-13 22:07:37 +00:00

A resumed job re-sends the messages it already delivered #745