coilyco-gaming/sirens-echo

Watch

Prevent irrelevant Eco tool churn in non-Eco Echo turns #42

New issue

Closed

opened 2026-07-28 11:00:25 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-28 11:00:25 +00:00

Member

Observed during the non-mutating Echo evaluation after structured JSON mode landed.

Evidence:

unknown-event-time completed successfully with a valid JSON decision.
explicit-correction requested get_eco_server_status and then get_eco_government, although the correction needed no current Eco data.
Those model rounds consumed the shared five-minute evaluation deadline. The correction case failed on timeout and the following live-status case could not start its MCP session.

Expected behavior:

Knowledge-gap and correction turns use no Eco tool unless current Eco information is relevant to the member request.
The live Eco status case still invokes its required tool and returns a grounded decision.
One evaluation case cannot starve the remaining cases of their verification window.

Acceptance evidence should include the three-case evaluation passing against the selected Community route without Discord or Forgejo writes.

Observed during the non-mutating Echo evaluation after structured JSON mode landed. Evidence: * `unknown-event-time` completed successfully with a valid JSON decision. * `explicit-correction` requested `get_eco_server_status` and then `get_eco_government`, although the correction needed no current Eco data. * Those model rounds consumed the shared five-minute evaluation deadline. The correction case failed on timeout and the following live-status case could not start its MCP session. Expected behavior: * Knowledge-gap and correction turns use no Eco tool unless current Eco information is relevant to the member request. * The live Eco status case still invokes its required tool and returns a grounded decision. * One evaluation case cannot starve the remaining cases of their verification window. Acceptance evidence should include the three-case evaluation passing against the selected Community route without Discord or Forgejo writes.

coilyco-ops commented

2026-07-28 22:06:29 +00:00

Author

Member

Evaluation update after the timeout and response-adapter fixes landed on main:

Forgejo Action run 41 passed, including image publication.
unknown-event-time: pass.
eco-live-status: pass.
explicit-correction: failed after the new bounded repair with Agent Proxy returned invalid structured output after 1 repair attempt.
The evaluation performed no Discord or Forgejo mutations. OTLP export emitted no errors.

The timeout fix now gives every case an independent five-minute deadline, so the explicit case no longer starves the live-status case. The adapter now treats empty model content as a structured-output violation and makes one tools-disabled repair request.

Read-only trace evidence for the final run shows three successful Agent Proxy requests for explicit-correction, with finish reasons tool_calls, stop, and stop. The final call had tools disabled and still did not return the requested valid JSON object. Sirens supplied response_format: json_object, temperature zero, and an explicit repair prompt.

This issue remains open. The remaining acceptance failure is the irrelevant Eco tool round followed by model or gateway noncompliance with structured output, not OTLP or a shared evaluation deadline.

Evaluation update after the timeout and response-adapter fixes landed on main: * Forgejo Action run 41 passed, including image publication. * `unknown-event-time`: pass. * `eco-live-status`: pass. * `explicit-correction`: failed after the new bounded repair with `Agent Proxy returned invalid structured output after 1 repair attempt`. * The evaluation performed no Discord or Forgejo mutations. OTLP export emitted no errors. The timeout fix now gives every case an independent five-minute deadline, so the explicit case no longer starves the live-status case. The adapter now treats empty model content as a structured-output violation and makes one tools-disabled repair request. Read-only trace evidence for the final run shows three successful Agent Proxy requests for `explicit-correction`, with finish reasons `tool_calls`, `stop`, and `stop`. The final call had tools disabled and still did not return the requested valid JSON object. Sirens supplied `response_format: json_object`, temperature zero, and an explicit repair prompt. This issue remains open. The remaining acceptance failure is the irrelevant Eco tool round followed by model or gateway noncompliance with structured output, not OTLP or a shared evaluation deadline.

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-29 03:48:14 +00:00

coilyco-ops added

and removed

labels

2026-08-05 10:26:17 +00:00

coilysiren closed this issue

2026-08-12 11:47:53 +00:00

coilyco-ops commented

2026-08-12 11:56:01 +00:00

Author

Member

Decision: re-measure on current main before anything else

Direction from Kai, 2026-08-12 session.

No fix is being chosen yet, deliberately. Re-run the three-case evaluation against current main first, then decide.

Why the existing evidence cannot be acted on

The last measurement is from 2026-07-28 and main has moved substantially since. One change in particular invalidates the baseline: 7236903 removed a contradiction where every request carried response_format: {"type":"json_object"} while the system prompt said "Reply with plain text." That leftover had survived the envelope removal in 3b6abd5, which had already made ParseReply stop unwrapping JSON.

Whether it contributed to explicit-correction failing with "invalid structured output after 1 repair attempt" is unproven — but a turn instructed to answer in prose while constrained to emit a JSON object is not a clean baseline to diagnose structured-output compliance against. Diagnosing further on July 28 evidence risks fixing something that is already gone.

Related reading: the same contradiction is discussed on #137, which was diagnosing a different validator failure in the same area of the code.

What the re-measurement should report

Enough to decide between the two candidate fixes without another round:

All three cases — unknown-event-time, explicit-correction, eco-live-status — with pass or fail each.
For explicit-correction specifically: does it still request Eco tools it does not need, and does the repair path still return invalid JSON with tools disabled. These are the two distinct defects.
Acceptance evidence as the issue already requires: the run passes against the selected Community route with no Discord or Forgejo writes.

The likely split, for planning only

If both symptoms persist, this is two defects wearing one ticket and should be split:

Irrelevant Eco tool selection — a grounding and prompt problem in this repository, and the issue's actual title.
Model or gateway noncompliance with structured output under tools-disabled repair — an agent-proxy or model problem, and a different repository.

Do not split it before the re-measurement. One or both may have resolved.

Priority

Deferred past August 19. The re-measurement itself is cheap and can be run whenever there is a seam.

## Decision: re-measure on current main before anything else Direction from Kai, 2026-08-12 session. No fix is being chosen yet, deliberately. **Re-run the three-case evaluation against current `main` first**, then decide. ## Why the existing evidence cannot be acted on The last measurement is from 2026-07-28 and `main` has moved substantially since. One change in particular invalidates the baseline: `7236903` removed a contradiction where every request carried `response_format: {"type":"json_object"}` while the system prompt said "Reply with plain text." That leftover had survived the envelope removal in `3b6abd5`, which had already made `ParseReply` stop unwrapping JSON. Whether it contributed to `explicit-correction` failing with "invalid structured output after 1 repair attempt" is unproven — but a turn instructed to answer in prose while constrained to emit a JSON object is not a clean baseline to diagnose structured-output compliance against. Diagnosing further on July 28 evidence risks fixing something that is already gone. Related reading: the same contradiction is discussed on #137, which was diagnosing a different validator failure in the same area of the code. ## What the re-measurement should report Enough to decide between the two candidate fixes without another round: - All three cases — `unknown-event-time`, `explicit-correction`, `eco-live-status` — with pass or fail each. - For `explicit-correction` specifically: **does it still request Eco tools it does not need**, and **does the repair path still return invalid JSON with tools disabled**. These are the two distinct defects. - Acceptance evidence as the issue already requires: the run passes against the selected Community route with no Discord or Forgejo writes. ## The likely split, for planning only If both symptoms persist, this is two defects wearing one ticket and should be split: - **Irrelevant Eco tool selection** — a grounding and prompt problem in this repository, and the issue's actual title. - **Model or gateway noncompliance with structured output** under tools-disabled repair — an agent-proxy or model problem, and a different repository. Do not split it before the re-measurement. One or both may have resolved. ## Priority Deferred past August 19. The re-measurement itself is cheap and can be run whenever there is a seam.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#42

No description provided.

Rows
Columns