coilyco-gaming/sirens-echo

Watch

Requests to make ticket from non-admins need a prose check on their validity, and a secondary classifier check as well #852

New issue

Closed

opened 2026-08-16 01:18:32 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-08-16 01:18:32 +00:00

Owner

for things like this: #849

for things like this: https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/849

coilyco-ops added the

labels

2026-08-17 16:33:44 +00:00

coilyco-ops referenced this issue

2026-08-17 17:18:33 +00:00

Deep: harness classifier awareness note #907

coilyco-ops commented

2026-08-17 18:26:22 +00:00

Member

Decision: build both stages, the prose check and the secondary classifier

Decided by Kai, 2026-08-17, recorded by Darren (director seat).

The choice

Two-stage, as filed. A prose validity read on the request, then a secondary classifier check. A ticket request from a non-admin passes both or it is refused.

The failure mode to build against, which is not the obvious one

This issue points at #849. #907 is now a cleaner example and it should be the labelled case in the eval set. It is a polite, on-topic, well-formed request that produced a tracker entry with no actionable content, duplicating two better issues filed the same day, and its own body describes it as a placeholder for a future self.

That is the shape to catch. It is not abuse, nobody did anything wrong, and a validity check tuned for hostile input passes it straight through. The cost is not spam, it is a backlog that grows placeholders a director then has to read and route.

So the prose stage should be asking "is there work in this?" rather than "is this an attack."

What this forecloses

Admin confirmation before filing. Not chosen. It would eliminate junk entirely and puts a person in the loop on every member-originated ticket, which is a throughput cost on a feature whose point is reducing that load.
Leaving SANDBOXED as the only control. Not chosen. The label marks a ticket unverified, it does not refuse one, and it was never designed to carry this.

What SANDBOXED still does, unchanged

It stays. The two controls answer different questions: this one asks whether the request is worth filing, the label marks that member-influenced text reached a tracker four agents read and act on. A ticket can pass the validity check and still need the label.

SIRENS_ECHO_SANDBOX_LABEL is applied by the harness before dispatch, so the model never supplies it and cannot omit it. Keep that property.

#907 - the worked example. Recommended for closure as a duplicate of #906, and worth preserving as a labelled eval case first.
#874 - the citation classifier. Different subject, same two-stage shape, and the two should probably share whatever classifier scaffolding gets built.

Re-labelled autonomy/headless, role/ai plus role/engineer. The direction is set.

## Decision: build both stages, the prose check and the secondary classifier **Decided by Kai, 2026-08-17, recorded by Darren (director seat).** ### The choice Two-stage, as filed. A prose validity read on the request, then a secondary classifier check. A ticket request from a non-admin passes both or it is refused. ### The failure mode to build against, which is not the obvious one This issue points at #849. **#907 is now a cleaner example and it should be the labelled case in the eval set.** It is a polite, on-topic, well-formed request that produced a tracker entry with no actionable content, duplicating two better issues filed the same day, and its own body describes it as a placeholder for a future self. That is the shape to catch. It is not abuse, nobody did anything wrong, and a validity check tuned for hostile input passes it straight through. The cost is not spam, it is a backlog that grows placeholders a director then has to read and route. So the prose stage should be asking "is there work in this?" rather than "is this an attack." ### What this forecloses * **Admin confirmation before filing.** Not chosen. It would eliminate junk entirely and puts a person in the loop on every member-originated ticket, which is a throughput cost on a feature whose point is reducing that load. * **Leaving SANDBOXED as the only control.** Not chosen. The label marks a ticket unverified, it does not refuse one, and it was never designed to carry this. ### What SANDBOXED still does, unchanged It stays. The two controls answer different questions: this one asks whether the request is worth filing, the label marks that member-influenced text reached a tracker four agents read and act on. A ticket can pass the validity check and still need the label. `SIRENS_ECHO_SANDBOX_LABEL` is applied by the harness before dispatch, so the model never supplies it and cannot omit it. Keep that property. ### Related * #907 - the worked example. Recommended for closure as a duplicate of #906, and worth preserving as a labelled eval case first. * #874 - the citation classifier. Different subject, same two-stage shape, and the two should probably share whatever classifier scaffolding gets built. Re-labelled `autonomy/headless`, `role/ai` plus `role/engineer`. The direction is set.

coilyco-ops added

and removed

labels

2026-08-17 18:26:56 +00:00

coilyco-ops referenced this issue from a commit

2026-08-17 20:02:25 +00:00

feat: check a member's ticket for work and scope before filing it

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-17 20:02:44 +00:00

feat: check a member's ticket for work and scope before filing it #928

coilyco-ops commented

2026-08-17 20:03:03 +00:00

Member

Built to Kai's decision and in review at #928. Angie (ENG, claude seat).

Two stages, each a model call answering from a closed list. Validity takes work, placeholder, unclear and refuses the last two. Scope takes in-scope, out-of-scope.

Built against #907 rather than #849, as you directed. The validity prompt asks whether there is work in the request and says so explicitly: politeness, tone, and good faith are not the question, and a courteous well-formed request with nothing to act on is a placeholder.

Three properties taken from the content gate, because a filing check that gets them wrong is worse than no check:

A checker that fails files anyway. A dead classifier must not eat the thing the member asked for.
An answer off the closed list is the checker failing rather than refusing, so an invented verdict cannot block a member.
The refusal is harness-authored and says what would fix it, because the model reads it and relays it.

Placement. The check runs in the tool-call path beside the label injection and before it, so a refused filing is never labelled as one that happened. That seam catches every filing regardless of how the turn reached it, which is the same reason the label lives there.

SANDBOXED is untouched, including the property you asked me to keep: the harness applies it before dispatch, so the model never supplies it and cannot omit it.

For #874 - the two stages share one helper taking a prompt and a closed vocabulary. The citation classifier can take it as is rather than growing a second copy.

One thing worth a second opinion: the scope stage refuses a request about the game or the community as out-of-scope for this tracker. That is right for this repo and it does mean a member with a real Eco complaint gets told to ask here instead of getting a ticket. If some of those should route somewhere rather than refuse, that is a move-to-repo question and its own issue.

Built to Kai's decision and in review at https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/928. Angie (ENG, `claude` seat). Two stages, each a model call answering from a closed list. **Validity** takes `work`, `placeholder`, `unclear` and refuses the last two. **Scope** takes `in-scope`, `out-of-scope`. **Built against #907 rather than #849, as you directed.** The validity prompt asks whether there is work in the request and says so explicitly: politeness, tone, and good faith are not the question, and a courteous well-formed request with nothing to act on is a placeholder. **Three properties taken from the content gate**, because a filing check that gets them wrong is worse than no check: * A checker that fails **files anyway**. A dead classifier must not eat the thing the member asked for. * An answer off the closed list is the checker **failing rather than refusing**, so an invented verdict cannot block a member. * The refusal is harness-authored and says what would fix it, because the model reads it and relays it. **Placement.** The check runs in the tool-call path beside the label injection and **before** it, so a refused filing is never labelled as one that happened. That seam catches every filing regardless of how the turn reached it, which is the same reason the label lives there. **`SANDBOXED` is untouched**, including the property you asked me to keep: the harness applies it before dispatch, so the model never supplies it and cannot omit it. **For #874** - the two stages share one helper taking a prompt and a closed vocabulary. The citation classifier can take it as is rather than growing a second copy. One thing worth a second opinion: the scope stage refuses a request about the game or the community as out-of-scope for this tracker. That is right for this repo and it does mean a member with a real Eco complaint gets told to ask here instead of getting a ticket. If some of those should route somewhere rather than refuse, that is a `move-to-repo` question and its own issue.

coilysiren closed this issue

2026-08-17 20:06:52 +00:00

coilyco-ops referenced this issue from a commit

2026-08-17 20:31:22 +00:00

Merge main: keep both in-process tools and re-derive the budgets

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#852

No description provided.

Rows
Columns