denied_queue survives the coalescing flip, so something else is shedding admissions

coilyco-ops commented

2026-08-19 16:10:14 +00:00

Member

Split out of #976, which is closed. That issue predicted coalescing would remove the denied_queue bottleneck. Coalescing landed, works, and did not remove it. That prediction failing is its own finding and does not belong in the issue that made it.

Evidence

From the QA pass on #976, SigNoz, read-only, window 2026-08-12 to 2026-08-19T15:09Z.

dowel, restricted to after its coalescing start - 115 accepted, 9 denied_queue. A 7.3% denial share with the lane on, and every one of those 9 falls after the flip.
echo, 7 days - 283 accepted, 38 denied_queue (11.7%), 5 denied_user.

So the bottleneck is real and it is not what the coalescing lane removes. #976 asserted "the denied_queue share is the bottleneck this removes" and the measurement contradicts it.

Why coalescing was never going to fix this

Worth stating so nobody re-runs the same reasoning. Coalescing merges comments that arrive inside the same window into one turn. Dowel runs roughly five admissions an hour against a 25s window, so comments essentially never coincide by chance. The lane batches correctly when they do (verified live, see #1082), but on this traffic shape it almost never fires, and a denial arriving minutes after the previous turn is untouched by a 25s window.

The two mechanisms barely overlap. Coalescing addresses burst; denied_queue here looks like sustained occupancy.

What is worth checking

Which bound is actually rejecting. #976 noted SIRENS_ECHO_MAX_PENDING (8) stops being the backlog bound while coalescing is on, with SIRENS_ECHO_COALESCE_CAPACITY (200) shedding instead. But coalesce.asks shows zero shed on every lane, so the denials are not coming from the coalescing path at all.
Whether the denials correlate with turn duration. #1076 measured turn p50 17.2s and p95 181s on dowel, with individual turns running past 235s. If a slow turn holds the only execution slot, a denial is a queue that is full because turns are long, and the fix is #1076's iteration cap rather than anything admission-side.
Whether the per-user, per-context, and global tiers are each contributing, since only denied_queue and a little denied_user show up.

That middle possibility is the interesting one: it would make this a symptom of turn latency rather than an admission-sizing problem, and would mean the turn budget landed in internal/community/turnbudget.go already moves it. Measuring after that rolls is cheap and should come first.

Read these as delta

The SigNoz catalog metadata declares several sirens_echo.* names cumulative and they are delta on the wire. Queried per the catalog they return empty, which reads as a broken pipeline and is not one. Pass temporality: delta explicitly.

#976 - where the prediction was made, closed
#1082 - the coalescing measurement successor
#1076 - turn-latency measurements and the iteration cap

Split out of #976, which is closed. That issue predicted coalescing would remove the `denied_queue` bottleneck. Coalescing landed, works, and **did not remove it**. That prediction failing is its own finding and does not belong in the issue that made it. ## Evidence From the QA pass on #976, SigNoz, read-only, window 2026-08-12 to 2026-08-19T15:09Z. * **dowel, restricted to after its coalescing start** - 115 accepted, 9 `denied_queue`. A 7.3% denial share with the lane on, and every one of those 9 falls after the flip. * **echo, 7 days** - 283 accepted, 38 `denied_queue` (11.7%), 5 `denied_user`. So the bottleneck is real and it is not what the coalescing lane removes. #976 asserted "the `denied_queue` share is the bottleneck this removes" and the measurement contradicts it. ## Why coalescing was never going to fix this Worth stating so nobody re-runs the same reasoning. Coalescing merges comments that arrive **inside the same window** into one turn. Dowel runs roughly five admissions an hour against a 25s window, so comments essentially never coincide by chance. The lane batches correctly when they do (verified live, see #1082), but on this traffic shape it almost never fires, and a denial arriving minutes after the previous turn is untouched by a 25s window. The two mechanisms barely overlap. Coalescing addresses burst; `denied_queue` here looks like sustained occupancy. ## What is worth checking * Which bound is actually rejecting. #976 noted `SIRENS_ECHO_MAX_PENDING` (8) stops being the backlog bound while coalescing is on, with `SIRENS_ECHO_COALESCE_CAPACITY` (200) shedding instead. But `coalesce.asks` shows **zero shed** on every lane, so the denials are not coming from the coalescing path at all. * Whether the denials correlate with turn duration. #1076 measured turn p50 17.2s and p95 181s on dowel, with individual turns running past 235s. If a slow turn holds the only execution slot, a denial is a queue that is full because turns are long, and the fix is #1076's iteration cap rather than anything admission-side. * Whether the per-user, per-context, and global tiers are each contributing, since only `denied_queue` and a little `denied_user` show up. That middle possibility is the interesting one: it would make this a symptom of turn latency rather than an admission-sizing problem, and would mean the turn budget landed in `internal/community/turnbudget.go` already moves it. Measuring after that rolls is cheap and should come first. ## Read these as delta The SigNoz catalog metadata declares several `sirens_echo.*` names cumulative and they are delta on the wire. Queried per the catalog they return empty, which reads as a broken pipeline and is not one. Pass `temporality: delta` explicitly. ## Related * #976 - where the prediction was made, closed * #1082 - the coalescing measurement successor * #1076 - turn-latency measurements and the iteration cap

coilyco-ops added the

labels

lane	denied_queue share	turn p50
`sirens-echo`	18.8%	300.8s
`sirens-deep-owl-glass`	11.3%	63.8s
`sirens-dowel`	6.3%	31.2s
`sirens-deep`	2.3%	32.9s

Rows
Columns

denied_queue survives the coalescing flip, so something else is shedding admissions #1083

Evidence

Why coalescing was never going to fix this

What is worth checking

Read these as delta

Related

"Which bound is actually rejecting" has a code answer, and it rules Discord out entirely

Facts, at the lines

What that leaves, marked as inference

The one query that settles it

Consequence for the ordering

One thing this does not touch

Measured, and the code read holds. The 9 are http and mcp, not Discord.

denied_queue by transport and service, daily

Discord stops producing this outcome entirely once the lane is on

So the answer to "which bound is actually rejecting"

What that does to this issue

Two things I am not claiming

Correcting my own verification query, because #1106 renamed the label out from under it

The corrected query

And the expected shape, which the split makes falsifiable

The before side

Reopened, because #1106's own body says this stays open and the merge closed it anyway

What remains to do here, in one place

denied_queue is still happening

The correlation is the interesting part

What the next window can now say

Where that leaves the three checks

Measured, and the code read holds. The 9 are `http` and `mcp`, not Discord.

`denied_queue` by transport and service, daily