File on any unanswerable in-scope question, once per turn #821

Merged

coilysiren merged 1 commit from aos/claude/xh55-filing-trigger into main

2026-08-15 17:07:48 +00:00

coilyco-ops commented

2026-08-15 17:03:13 +00:00

Member

Refs #235. Implements Kai's filing trigger, decided 2026-08-15 and recorded by @Delphi, plus Kai's follow-up that Echo and Deep get the same settings. ward exec gate PASS on all six steps.

The trigger

The rule filed in exactly two cases: a user corrects a factual claim, or a needed capability turns out to be missing. It now also files when a question inside the remit could not be answered — any such question, not only a repeated one, which is the option Kai picked over explicit-ask-only and repeated-gap-only.

The cost Kai accepted, and its two limits

Kai chose the most responsive option knowing it files more, which is what makes dedupe "a requirement of the build rather than a nice-to-have". Both limits from the decision are now stated in the prose:

At most one issue per turn — a turn that fails three lookups files the one gap behind them, not three issues.
A question outside the remit is never filed — the trigger is "in remit and unanswerable", not "unanswerable".

Search-before-file was already there and matters more under a broader trigger, since it is what keeps the accepted duplicate cost bounded. The issue naming the gap rather than the member is now stated positively beside the existing prohibition on copying names, quotes, and identifiers.

"Echo and Deep should have the same settings here"

Already structurally true — issuePolicy is one function keyed only on whether a definition names a tracker, and both name forgejo. It was not pinned, so nothing stopped it drifting. TestBothProfilesShareOneFilingPolicy renders the rule from each definition and fails on any difference.

The 315-byte budget raise is identical on both snapshots, which is that shared block showing up as arithmetic.

Budget

Echo 21976 → 22291, Deep 12260 → 12575. Part of the widening was paid by cutting redundant wording; the rest is the new rule. Recorded in docs/sirens-echo-prompt-budget.md with its cause, as that doc requires.

Two things found on the way

The tracker rate packs were unrunnable through the supported surface. Both say "Run it with ward exec rate-fixture-tracker", and that verb existed only in scripts/ward-command.sh, never in .ward/ward.yaml. So the measurement @Lucia built to settle this issue could not be re-run by anyone following the file's own instructions. Both verbs are declared now.

A test matched the old sentence rather than the obligation. Rewording the policy failed TestBuildSystemPromptBoundsToolActionsAndAutomaticFollowUp on "Search for an open issue with the same title first". It now matches the obligation. That is the same defect recorded on this thread when a hard-coded line wrap failed a reworded paragraph — asserting a layout when the intent is a phrase.

Measurement

Two new cases in agent/rate-fixture-tracker.yaml for the new branches, marked unmeasured rather than carrying a number from the narrower rule they replace. The four existing branches are untouched, so they stay usable as the before-half of the comparison.

One thing I did not change

out-of-scope-question-files-nothing is classified partial in the language-scope registry: its tracker-URL pattern catches a translated filing, its English prose patterns do not. Its structural twin mild-confusion-files-nothing carries identical checks and is classified english-only. By the reasoning recorded on issue-reference-has-a-receipt, one of those two is wrong. I classified mine by that reasoning rather than copying the neighbour, and left the neighbour to whoever owns it rather than silently reclassifying someone else's decision.

Refs #235. Implements Kai's filing trigger, decided 2026-08-15 and recorded by @Delphi, plus Kai's follow-up that Echo and Deep get the same settings. `ward exec gate` PASS on all six steps. ## The trigger The rule filed in exactly two cases: a user corrects a factual claim, or a needed capability turns out to be missing. It now also files when **a question inside the remit could not be answered** — any such question, not only a repeated one, which is the option Kai picked over explicit-ask-only and repeated-gap-only. ## The cost Kai accepted, and its two limits Kai chose the most responsive option knowing it files more, which is what makes dedupe "a requirement of the build rather than a nice-to-have". Both limits from the decision are now stated in the prose: * **At most one issue per turn** — a turn that fails three lookups files the one gap behind them, not three issues. * **A question outside the remit is never filed** — the trigger is "in remit and unanswerable", not "unanswerable". Search-before-file was already there and matters more under a broader trigger, since it is what keeps the accepted duplicate cost bounded. The issue naming the gap rather than the member is now stated positively beside the existing prohibition on copying names, quotes, and identifiers. ## "Echo and Deep should have the same settings here" Already structurally true — `issuePolicy` is one function keyed only on whether a definition names a tracker, and both name `forgejo`. It was not **pinned**, so nothing stopped it drifting. `TestBothProfilesShareOneFilingPolicy` renders the rule from each definition and fails on any difference. The 315-byte budget raise is identical on both snapshots, which is that shared block showing up as arithmetic. ## Budget Echo 21976 → 22291, Deep 12260 → 12575. Part of the widening was paid by cutting redundant wording; the rest is the new rule. Recorded in `docs/sirens-echo-prompt-budget.md` with its cause, as that doc requires. ## Two things found on the way **The tracker rate packs were unrunnable through the supported surface.** Both say "Run it with `ward exec rate-fixture-tracker`", and that verb existed only in `scripts/ward-command.sh`, never in `.ward/ward.yaml`. So the measurement @Lucia built to settle this issue could not be re-run by anyone following the file's own instructions. Both verbs are declared now. **A test matched the old sentence rather than the obligation.** Rewording the policy failed `TestBuildSystemPromptBoundsToolActionsAndAutomaticFollowUp` on `"Search for an open issue with the same title first"`. It now matches the obligation. That is the same defect recorded on this thread when a hard-coded line wrap failed a reworded paragraph — asserting a layout when the intent is a phrase. ## Measurement Two new cases in `agent/rate-fixture-tracker.yaml` for the new branches, marked **unmeasured** rather than carrying a number from the narrower rule they replace. The four existing branches are untouched, so they stay usable as the before-half of the comparison. ## One thing I did not change `out-of-scope-question-files-nothing` is classified `partial` in the language-scope registry: its tracker-URL pattern catches a translated filing, its English prose patterns do not. Its structural twin `mild-confusion-files-nothing` carries identical checks and is classified `english-only`. By the reasoning recorded on `issue-reference-has-a-receipt`, one of those two is wrong. I classified mine by that reasoning rather than copying the neighbour, and left the neighbour to whoever owns it rather than silently reclassifying someone else's decision.

coilyco-ops added 1 commit

2026-08-15 17:03:13 +00:00

feat(prompt): file on any unanswerable in-scope question, once per turn

ci / image-build (pull_request) Successful in 26s

Details

ci / test (pull_request) Successful in 42s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

04943d573c

Kai decided the filing trigger on #235 and Delphi recorded it on the thread:
Echo files when it could not answer a question that is clearly inside its
remit. Any such question, not only a repeated one. Kai then added that Echo and
Deep get the same settings here.

The rule filed in exactly two cases, a user correcting a factual claim or a
needed capability turning out to be missing. Those stay, and the trigger now
also covers an in-remit question the turn simply could not answer, which is
what the pressure tests were complaining about.

THE COST KAI ACCEPTED, AND ITS TWO LIMITS

The chosen option files more, and duplicates were accepted knowingly. That
makes the limits load-bearing rather than decoration, so both are now stated:

  at most one issue per turn, so a turn failing three lookups files one gap
  a question outside the remit is never filed

Search-before-file was already there and matters more now than it did, because
it is what keeps the accepted duplicate cost bounded rather than open-ended.
The issue naming the gap rather than the member is stated positively beside the
existing prohibition on copying names, quotes, and identifiers.

ONE POLICY, TWO PROFILES

issuePolicy is one function keyed on whether a definition names a tracker, and
both sirens-echo.yaml and sirens-deep.yaml name forgejo, so Kai's "same
settings" was already structurally true. It was not pinned, so nothing stopped
it drifting. TestBothProfilesShareOneFilingPolicy renders the rule for each
definition and fails on a difference.

The 315-byte raise is identical on both snapshots, which is that shared block
showing up as arithmetic.

BUDGET

Echo 21976 to 22291, Deep 12260 to 12575. Part of the widening was paid by
cutting redundant wording; the rest is the new rule. Recorded in
docs/sirens-echo-prompt-budget.md with its cause, as that doc requires.

MEASUREMENT

The two new branches are added to agent/rate-fixture-tracker.yaml and marked
unmeasured rather than carrying a number from the narrower rule they replace.
The four existing branches are unchanged, so they remain the before-half of the
comparison.

Both tracker packs told the reader to run them with `ward exec
rate-fixture-tracker`, and that verb existed only in scripts/ward-command.sh
and not in .ward/ward.yaml, so the instruction was false and the measurement
was unrunnable through the supported surface. Both verbs are declared now.

One existing expectation matched the old sentence rather than the obligation,
and is rewritten to match the obligation. That is the same defect recorded on
this thread when a hard-coded line wrap failed a reworded paragraph.

Refs #235

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Angie <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-15 17:03:13 +00:00

coilyco-ops referenced this pull request

2026-08-15 17:05:14 +00:00

The agent is not proactive enough about filing fj issues around its knowledge gaps #235