A locked tenant parks a coalescing worker instead of deferring the batch, so one chatty member head-of-line blocks the shared channel #1097

New issue

Open

opened 2026-08-22 21:21:24 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-22 21:21:24 +00:00

Member

Filed by Darren (director seat), 2026-08-22. Rescued from a comment on #995, where it was the first of two findings that had no issue of their own. #995 is being worked now as the slot pool and will be retitled or closed around that work, so both findings needed a home before they went with it.

The finding, in the reporting seat's words

Olaf (ops seat), reading internal/coalesce on 2026-08-19:

The tenant lock blocks rather than skipping. Pool.serve takes locks.Lock(batch.Tenant.Key()) before doing anything, so a worker that picks up a batch for a member already being served parks until that member's turn finishes, and the shared channel head-of-line blocks behind it. A worker that instead deferred a locked batch and took the next one would waste nothing. At 3 workers a single chatty member can park a third of the pool. Sizing around it works, as coilyco-bridge/deploy#702 does with its extra two, but it is a workaround.

Why it is worth its own issue

The per-tenant lock is doing the right thing and should stay. It is what bounds one loud member to one worker's throughput, so six messages in a minute become two turns while a quiet member in the same channel is unaffected. That property came from the design and is not in question here.

What is in question is what the other workers do while one is parked. A blocked take holds a worker and the head of the shared channel, so the cost is not one worker but everything queued behind it. SIRENS_ECHO_COALESCE_WORKERS defaults to 3, and coilyco-bridge/deploy#702 raises it to 10 for the stream, which buys headroom without changing the shape.

Shape to consider

A worker that finds a batch's tenant locked defers that batch and takes the next one, rather than parking on the lock. Requeue discipline is the part that needs care rather than the skip itself: a deferred batch must not starve, and it must not be reordered ahead of a batch for a tenant that has been waiting longer.

Done when

A batch whose tenant is already being served does not block a worker that could serve a different tenant, demonstrated under a burst with one chatty member and several quiet ones, and no deferred batch is starved or answered out of order.

#995 - the slot pool, where this was found. Deliberately not carried there.
coilyco-bridge/deploy#702 - the worker count raise that sizes around this.

**Filed by Darren (director seat), 2026-08-22.** Rescued from a comment on #995, where it was the first of two findings that had no issue of their own. #995 is being worked now as the slot pool and will be retitled or closed around that work, so both findings needed a home before they went with it. ## The finding, in the reporting seat's words Olaf (ops seat), reading `internal/coalesce` on 2026-08-19: > **The tenant lock blocks rather than skipping.** `Pool.serve` takes `locks.Lock(batch.Tenant.Key())` before doing anything, so a worker that picks up a batch for a member already being served parks until that member's turn finishes, and the shared channel head-of-line blocks behind it. A worker that instead deferred a locked batch and took the next one would waste nothing. At 3 workers a single chatty member can park a third of the pool. Sizing around it works, as `coilyco-bridge/deploy#702` does with its extra two, but it is a workaround. ## Why it is worth its own issue The per-tenant lock is doing the right thing and should stay. It is what bounds one loud member to one worker's throughput, so six messages in a minute become two turns while a quiet member in the same channel is unaffected. That property came from the design and is not in question here. **What is in question is what the other workers do while one is parked.** A blocked take holds a worker and the head of the shared channel, so the cost is not one worker but everything queued behind it. `SIRENS_ECHO_COALESCE_WORKERS` defaults to 3, and `coilyco-bridge/deploy#702` raises it to 10 for the stream, which buys headroom without changing the shape. ## Shape to consider A worker that finds a batch's tenant locked defers that batch and takes the next one, rather than parking on the lock. Requeue discipline is the part that needs care rather than the skip itself: a deferred batch must not starve, and it must not be reordered ahead of a batch for a tenant that has been waiting longer. ## Done when A batch whose tenant is already being served does not block a worker that could serve a different tenant, demonstrated under a burst with one chatty member and several quiet ones, and no deferred batch is starved or answered out of order. ## Related * #995 - the slot pool, where this was found. Deliberately not carried there. * `coilyco-bridge/deploy#702` - the worker count raise that sizes around this.

coilyco-ops added the

labels

2026-08-22 21:21:24 +00:00

coilyco-ops referenced this issue

2026-08-22 21:21:45 +00:00

A lone ask waits the full coalescing window with nothing to coalesce, so the quiet-room case pays 25s of dead time #1098

coilyco-ops referenced this issue

2026-08-22 21:21:58 +00:00