fix(admission): split denied_queue, which was two mechanisms in one label

coilyco-ops commented

2026-08-22 22:14:34 +00:00

Member

Milestone 17 phase 2, working #1083. This does not close #1083, so the slice is filed as #1105 and closed here rather than weakening the reference.

The finding

#1083's first question is which bound is actually rejecting. Read against the source, the recorded data cannot say: two different refusals increment denied_queue, and on the Discord path both go out with transport=discord.

ratelimit.go refuses at admission when MAX_PENDING is reached, so the turn never waits at all.
runSerialized refuses a turn that was already admitted, waited for a slot, and gave up at the queue timeout. It gets noticeQueueTimeout rather than a cooldown notice, which is the giveaway that it is a different thing.

Those are opposite findings. A backlog refusal is an admission-sizing problem. A slot-wait refusal is a turn that ran long, which #995's pool and #1076's iteration cap move and admission sizing does not. #1083 measured 7.3% on dowel and 11.7% on echo, and either mechanism produces those numbers. Its own text reaches for exactly this distinction and could not make it.

What changed

denied_backlog and denied_slot_wait replace denied_queue. That is the whole change: the cheapest thing that lets the next measurement answer its own question rather than repeat it.

TestABacklogRefusalAndASlotWaitAreSeparateOutcomes reads what sirens_echo.admissions actually recorded through a manual reader, rather than asserting what the code appears to record, and it fails against the old shared label with a slot wait was recorded as a backlog refusal: map[denied_queue:1].

What stays open on #1083

The post-deploy re-measurement, which needs the slot pool to reach the cluster first. Worth noting that the pool changes the prior: with one slot, sustained occupancy by long turns was the obvious mechanism, and with eight it is much less so, which is why the milestone put #1083 after #995.

closes #1105
Refs #1083, #976, #995, #1076

Milestone 17 phase 2, working #1083. **This does not close #1083**, so the slice is filed as #1105 and closed here rather than weakening the reference. ## The finding #1083's first question is which bound is actually rejecting. Read against the source, **the recorded data cannot say**: two different refusals increment `denied_queue`, and on the Discord path both go out with `transport=discord`. * `ratelimit.go` refuses at admission when `MAX_PENDING` is reached, so **the turn never waits at all**. * `runSerialized` refuses a turn that **was already admitted**, waited for a slot, and gave up at the queue timeout. It gets `noticeQueueTimeout` rather than a cooldown notice, which is the giveaway that it is a different thing. Those are opposite findings. A backlog refusal is an admission-sizing problem. A slot-wait refusal is a turn that ran long, which #995's pool and #1076's iteration cap move and admission sizing does not. #1083 measured 7.3% on dowel and 11.7% on echo, and **either mechanism produces those numbers**. Its own text reaches for exactly this distinction and could not make it. ## What changed `denied_backlog` and `denied_slot_wait` replace `denied_queue`. That is the whole change: the cheapest thing that lets the next measurement answer its own question rather than repeat it. `TestABacklogRefusalAndASlotWaitAreSeparateOutcomes` **reads what `sirens_echo.admissions` actually recorded** through a manual reader, rather than asserting what the code appears to record, and it fails against the old shared label with `a slot wait was recorded as a backlog refusal: map[denied_queue:1]`. ## What stays open on #1083 The post-deploy re-measurement, which needs the slot pool to reach the cluster first. Worth noting that the pool changes the prior: with one slot, sustained occupancy by long turns was the obvious mechanism, and with eight it is much less so, which is why the milestone put #1083 after #995. closes #1105 Refs #1083, #976, #995, #1076

coilyco-ops added this to the Headless engineer lane: the queue wait first milestone

2026-08-22 22:14:34 +00:00

coilyco-ops added 1 commit

2026-08-22 22:14:34 +00:00

fix(admission): split denied_queue, which was two mechanisms in one label

ci / test (pull_request) Successful in 58s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

ci / image-build (pull_request) Successful in 32s

Details

2d483436b4

#1083 asks which bound is actually rejecting. Read against the source,
the answer is that the measurement cannot say, because two different
refusals increment denied_queue on the same transport.

* The rate limiter refuses at admission when MAX_PENDING is reached, so
  the turn never waits at all.
* runSerialized refuses a turn that was already admitted, waited for an
  execution slot, and gave up at the queue timeout.

Those are opposite findings. The first is an admission-sizing problem.
The second is a turn that ran long, which #1076's iteration cap and the
slot pool in #995 both move and admission sizing does not. The 7.3%
denial share on dowel and 11.7% on echo could be either, and no query
against the recorded data separates them.

So they get their own outcomes, denied_backlog and denied_slot_wait,
which is the cheapest thing that makes the next measurement able to
answer its own question rather than repeating it.

The re-measurement #1083 also wants has to run after the slot pool
reaches the cluster, so it stays open carrying that half.

Refs #1083, #976, #995, #1076

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: engineer

Rows
Columns

fix(admission): split denied_queue, which was two mechanisms in one label #1106

The finding

What changed

What stays open on #1083

Right change, and the test is the right shape. It also breaks the verification query on #1083, in the direction that reads as success.

The trap

Historical continuity

What I checked so you do not have to

What is good and worth keeping in the next one

Merge it after the #1083 query