Read the admission split once denied_backlog and denied_slot_wait reach the cluster #1131

New issue

Open

opened 2026-08-23 01:18:59 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-23 01:18:59 +00:00

Member

Split out of #1083, whose investigation is complete and whose last step needs a deploy.

#1106 replaced the single denied_queue label with two, because it was two mechanisms under one name and no query could separate them:

denied_backlog - refused at admission because SIRENS_ECHO_MAX_PENDING is reached. The turn never waits.
denied_slot_wait - admitted, waited for an execution slot, gave up at the queue timeout. A turn that ran long rather than a queue that filled.

Those have opposite fixes, which is why the distinction matters.

What the pre-split data already says

Measured over seven days, and this is the hypothesis the read should confirm or kill:

Denial share ranks with turn length: Echo 18.8% at p50 300s, owl-glass 11.3% at 63s, deep 2.3% at 33s.
Dowel is the exception, 6.3% at p50 31s, and is the one lane whose load was a burst rather than sustained.
The rate tiers contribute nothing. Across every lane, the only outcomes recorded are accepted and denied_queue. Zero denied_user, denied_context, or denied_global.

The read to take

Once the split reaches the cluster, per lane:

Expected: Echo's denials are almost all denied_slot_wait, dowel's were almost all denied_backlog.
If that holds, the fix for the slot-wait lanes is the round bound in #1130 and not admission sizing at all.
If it does not hold, that is the more interesting result and worth a comment here rather than a shrug.

Note that dowel has produced no turns since 2026-08-19, so its share cannot be re-measured. sirens-deep is the lane to read.

Refs #1083, #1105, #1106, #1130

Split out of #1083, whose investigation is complete and whose last step needs a deploy. #1106 replaced the single `denied_queue` label with two, because it was two mechanisms under one name and no query could separate them: * **`denied_backlog`** - refused at admission because `SIRENS_ECHO_MAX_PENDING` is reached. The turn never waits. * **`denied_slot_wait`** - admitted, waited for an execution slot, gave up at the queue timeout. A turn that ran long rather than a queue that filled. Those have opposite fixes, which is why the distinction matters. ## What the pre-split data already says Measured over seven days, and this is the hypothesis the read should confirm or kill: * Denial share **ranks with turn length**: Echo 18.8% at p50 300s, owl-glass 11.3% at 63s, deep 2.3% at 33s. * Dowel is the exception, 6.3% at p50 31s, and is the one lane whose load was a burst rather than sustained. * **The rate tiers contribute nothing.** Across every lane, the only outcomes recorded are `accepted` and `denied_queue`. Zero `denied_user`, `denied_context`, or `denied_global`. ## The read to take Once the split reaches the cluster, per lane: * **Expected**: Echo's denials are almost all `denied_slot_wait`, dowel's were almost all `denied_backlog`. * **If that holds**, the fix for the slot-wait lanes is the round bound in #1130 and not admission sizing at all. * **If it does not hold**, that is the more interesting result and worth a comment here rather than a shrug. Note that dowel has produced no turns since 2026-08-19, so its share cannot be re-measured. `sirens-deep` is the lane to read. Refs #1083, #1105, #1106, #1130

coilyco-ops added this to the Headless engineer lane: the queue wait first milestone

2026-08-23 01:18:59 +00:00

coilyco-ops referenced this issue

2026-08-23 01:19:20 +00:00

denied_queue survives the coalescing flip, so something else is shedding admissions #1083

coilyco-ops removed this from the Headless engineer lane: the queue wait first milestone

2026-08-23 01:20:54 +00:00

coilyco-ops referenced this issue

2026-08-23 01:20:56 +00:00

denied_queue survives the coalescing flip, so something else is shedding admissions #1083

coilyco-ops referenced this issue

2026-08-23 01:42:07 +00:00