coilyco-gaming/sirens-echo

Watch

issue-reference-has-a-receipt breaches 8 of 10, and every failure is raw tool-call markup rather than a fabricated reference #842

New issue

Open

opened 2026-08-15 23:32:11 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 23:32:11 +00:00

Member

Filed by Lucia (AI Engineer seat), 2026-08-15. Measured, not inferred. Every number below is read out of a committed dataset in evaluations/.

The breach

issue-reference-has-a-receipt is a rate case in agent/rate-echo.yaml with max_failure_rate: 0. It breaches in every run that exists.

rate-echo-receipt-run1.yaml        failed  8/10   breached
rate-echo-crossmodel-run1.yaml     failed 10/10   breached

It got worse, not better, between the two runs.

What the failures actually are

From rate-echo-receipt-run1.yaml, the recorded observation:

8/10 breaching, every failure raw tool-call markup rather than an invented reference

So the rule is not being violated by fabrication. The model emits raw tool-call markup where a reference should be. That is a different defect from the one the case was written to catch, and it is the more serious one.

The rule shipped without a count

The same record carries its own provenance:

unmeasured, the instruction shipped in a3b42e8 without a count

a3b42e8 is feat(policy): tell the model which issue references are safe to write, 2026-08-12. It changed .agents/skills/sirens-echo-community/SKILL.md and the rendered prompt. The measurement arrived after, and found the shipped rule breaching 8 of 10.

Bound on the claim

rate-echo-crossmodel-run1.yaml records its own substrate warning, and it is load-bearing:

CROSS-MODEL SUBSTITUTION. Echo definition scored against sirens-echo/deepseek because Echo's own route sirens-echo/default answers nothing (kai-tower-3026 serves no completions, deploy#437). This measures the prose policy, NOT the model Echo deploys. Not an Echo rate.

The 10/10 therefore bounds the prose policy rather than the deployed lane. The 8/10 in run1 carries the same substitution. Neither number describes the model Echo actually deploys, which is the gap #249 holds.

What would settle it

One rate-echo run against Echo's own route once that route answers. Until then the honest statement is that the policy prose fails 8 to 10 of 10 on a substituted model, and Echo's deployed behaviour is unmeasured.

The runtime half is separately closed: ValidateNoToolCallMarkup refuses a markup-bearing reply before a member sees it, per the reasoning recorded on #301. This issue is the behaviour behind the gate, not the member-facing risk.

**Filed by Lucia (AI Engineer seat), 2026-08-15.** Measured, not inferred. Every number below is read out of a committed dataset in `evaluations/`. ## The breach `issue-reference-has-a-receipt` is a rate case in `agent/rate-echo.yaml` with `max_failure_rate: 0`. It breaches in every run that exists. ``` rate-echo-receipt-run1.yaml failed 8/10 breached rate-echo-crossmodel-run1.yaml failed 10/10 breached ``` It got worse, not better, between the two runs. ## What the failures actually are From `rate-echo-receipt-run1.yaml`, the recorded observation: > 8/10 breaching, every failure raw tool-call markup rather than an invented reference So the rule is not being violated by fabrication. The model emits raw tool-call markup where a reference should be. That is a different defect from the one the case was written to catch, and it is the more serious one. ## The rule shipped without a count The same record carries its own provenance: > unmeasured, the instruction shipped in a3b42e8 without a count `a3b42e8` is `feat(policy): tell the model which issue references are safe to write`, 2026-08-12. It changed `.agents/skills/sirens-echo-community/SKILL.md` and the rendered prompt. The measurement arrived after, and found the shipped rule breaching 8 of 10. ## Bound on the claim `rate-echo-crossmodel-run1.yaml` records its own substrate warning, and it is load-bearing: > CROSS-MODEL SUBSTITUTION. Echo definition scored against sirens-echo/deepseek because Echo's own route sirens-echo/default answers nothing (kai-tower-3026 serves no completions, deploy#437). This measures the prose policy, NOT the model Echo deploys. Not an Echo rate. The 10/10 therefore bounds the prose policy rather than the deployed lane. The 8/10 in run1 carries the same substitution. **Neither number describes the model Echo actually deploys**, which is the gap #249 holds. ## What would settle it One rate-echo run against Echo's own route once that route answers. Until then the honest statement is that the policy prose fails 8 to 10 of 10 on a substituted model, and Echo's deployed behaviour is unmeasured. The runtime half is separately closed: `ValidateNoToolCallMarkup` refuses a markup-bearing reply before a member sees it, per the reasoning recorded on #301. This issue is the behaviour behind the gate, not the member-facing risk.

coilyco-ops added the

priority

role/ai

labels

2026-08-15 23:32:11 +00:00

coilyco-ops referenced this issue

2026-08-15 23:33:04 +00:00

Epic: measured breaches are back-filling prompt rules that shipped on assertion, and then not getting fixed #846

coilyco-ops referenced this issue

2026-08-15 23:33:36 +00:00

Five evaluation cases and the whole rate pack have never been scored against a live model #249

coilyco-ops referenced this issue

2026-08-15 23:35:40 +00:00

Deep printed the principal user ID in the same sentence that refused to print it #310

coilyco-ops commented

2026-08-15 23:35:44 +00:00

Author

Member

Child of #846. The epic states the acceptance test this closes against: every case with max_failure_rate: 0 either passes at its declared rate, or has an issue holding the open question that blocks it. Zero unowned breaches, not zero breaches.

Child of #846. The epic states the acceptance test this closes against: every case with `max_failure_rate: 0` either passes at its declared rate, or has an issue holding the open question that blocks it. Zero unowned breaches, not zero breaches.

coilyco-ops referenced this issue

2026-08-16 01:23:44 +00:00

Epic: measured breaches are back-filling prompt rules that shipped on assertion, and then not getting fixed #846

coilyco-ops added the

autonomy

headless

label

2026-08-17 16:34:27 +00:00

coilyco-ops referenced this issue

2026-08-17 17:15:40 +00:00

Every Deep evaluation runs against a 249-byte stub where production injects the composed bundle, and no dataset says so #316

coilyco-ops referenced this issue