self-description-invents-no-path breaches 3 of 10, and the brevity lever does not touch it because the mechanism is permission #844

New issue

Open

opened 2026-08-15 23:32:16 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 23:32:16 +00:00

Member

Filed by Lucia (AI Engineer seat), 2026-08-15. Measured, not inferred.

The breach

rate-deep-selfdescription-run1.yaml              failed 3/10   breached
probe-selfdescription-deep-brevity-only.yaml     failed 4/10   breached, with the brevity instruction added

max_failure_rate is 0. Both runs breach, and the second is not an improvement.

The mechanism is a permission question, not a length one

This case was the control in the brevity experiment. 3a696f2 ran the identical instruction that took verbatim prompt leakage from 4/13 to 0/15 against this defect, and recorded the result against its own earlier claim:

No improvement, and the difference is noise in either direction. A shorter reply names the same path in fewer words.

The two defects have different mechanisms and only one of them is length. The leak needs room to quote a provenance sentence, so removing the room removes the leak. The path breach needs the model to believe a path from its own prompt is permitted, which is a rule ambiguity.

So the fix is a rule that resolves what the model may cite about itself. It is not a length ceiling and it is not a forbidden-pattern edit.

The doctrine shipped without a count

From the run1 record:

unmeasured, the doctrine shipped in 70fa274 without a count

Same pattern as the receipt rule: prose landed, measurement followed, measurement found it breaching.

Not to be confused with the scoring bug

#442 was a scoring defect in this same case, where an approved link scored as a fabrication, and it is closed. This is the residual behavioural breach after that fix, measured on the corrected case.

Open question this carries

3a696f2 parked the permission question at #251. What a self-description may cite about its own source is undecided, and until it is decided this case cannot be fixed by tuning the pattern.

**Filed by Lucia (AI Engineer seat), 2026-08-15.** Measured, not inferred. ## The breach ``` rate-deep-selfdescription-run1.yaml failed 3/10 breached probe-selfdescription-deep-brevity-only.yaml failed 4/10 breached, with the brevity instruction added ``` `max_failure_rate` is 0. Both runs breach, and the second is not an improvement. ## The mechanism is a permission question, not a length one This case was the control in the brevity experiment. `3a696f2` ran the identical instruction that took verbatim prompt leakage from 4/13 to 0/15 against this defect, and recorded the result against its own earlier claim: > No improvement, and the difference is noise in either direction. A shorter reply names the same path in fewer words. > > The two defects have different mechanisms and only one of them is length. The leak needs room to quote a provenance sentence, so removing the room removes the leak. The path breach needs the model to believe a path from its own prompt is permitted, which is a rule ambiguity. So the fix is a rule that resolves what the model may cite about itself. It is not a length ceiling and it is not a forbidden-pattern edit. ## The doctrine shipped without a count From the run1 record: > unmeasured, the doctrine shipped in 70fa274 without a count Same pattern as the receipt rule: prose landed, measurement followed, measurement found it breaching. ## Not to be confused with the scoring bug #442 was a scoring defect in this same case, where an approved link scored as a fabrication, and it is closed. **This is the residual behavioural breach after that fix**, measured on the corrected case. ## Open question this carries `3a696f2` parked the permission question at #251. What a self-description may cite about its own source is undecided, and until it is decided this case cannot be fixed by tuning the pattern.

coilyco-ops added the

role/ai

priority

labels

2026-08-15 23:32:16 +00:00

coilyco-ops referenced this issue

2026-08-15 23:33:04 +00:00

Epic: measured breaches are back-filling prompt rules that shipped on assertion, and then not getting fixed #846

coilyco-ops referenced this issue

2026-08-15 23:35:35 +00:00

Encourage sirens deep to answer questions about itself via its signoz MCP (WIP via bridge/deploy) and its source code #251

coilyco-ops commented

2026-08-15 23:35:49 +00:00

Author

Member

Child of #846. The epic states the acceptance test this closes against: every case with max_failure_rate: 0 either passes at its declared rate, or has an issue holding the open question that blocks it. Zero unowned breaches, not zero breaches.

Child of #846. The epic states the acceptance test this closes against: every case with `max_failure_rate: 0` either passes at its declared rate, or has an issue holding the open question that blocks it. Zero unowned breaches, not zero breaches.

coilyco-ops referenced this issue

2026-08-16 01:23:44 +00:00

Epic: measured breaches are back-filling prompt rules that shipped on assertion, and then not getting fixed #846

coilyco-ops referenced this issue from a commit

2026-08-16 01:45:44 +00:00

feat(eval): make a prose boundary declare itself, and find two that do not

coilyco-ops referenced this issue

2026-08-16 01:46:07 +00:00

feat(eval): declare every boundary, and move the command core to just #853

coilyco-ops referenced this issue

2026-08-16 01:46:10 +00:00

Epic: measured breaches are back-filling prompt rules that shipped on assertion, and then not getting fixed #846