coilyco-gaming/sirens-echo

Watch

Load test: Dowel refuses 3 of 8 summons at the stream's own arrival shape, each after a 50s wait for an execution slot #1010

New issue

Closed

opened 2026-08-19 01:44:54 +00:00 by coilyco-ops · 5 comments

coilyco-ops commented

2026-08-19 01:44:54 +00:00

Member

Load test run by Olaf (ops seat) 2026-08-19 01:40Z, roughly nine hours before stream prep. Driven over POST /v1/turn on NodePort 30122, so nothing was posted to Discord.

Under the arrival shape this lane was explicitly sized for, three of eight summons were refused.

Method

services/sirens-echo/docs/lane-values-rationale.md states the target shape directly:

Four humans and four agents share one room with nobody driving, and an @everyone draws agent replies within about two seconds of each other and human replies about ten seconds later, so eight summons land inside twelve seconds.

So: eight distinct X-Sirens-Caller values, eight independent admission tenants, arriving at 0, 1.5, 3, 4.5, 6, 7.5, 9, and 10.5 seconds. Ordinary short questions. A single unloaded warm-up turn beforehand returned in 11.7s.

Result

caller	code	wall
lt-human-a	200	16.3s
lt-agent-w	200	20.3s
lt-human-b	200	25.5s
lt-agent-x	200	34.4s
lt-human-c	200	84.9s
lt-agent-y	502	50.3s
lt-agent-z	502	50.3s
lt-human-d	502	50.3s

All three failures returned the same body:

{"reply":"> `busy, retry shortly`\n> `trace id ...`","error":"turn failed"}

37.5% refused. The three failures are within 0.1s of each other, which is a timeout rather than contention noise.

Cause, established rather than inferred

Spans for the burst window, service sirens-dowel:

POST /v1/turn - 8
community.turn - 5

Three requests never became a turn at all. Pulling one failed trace by the id in its own body (d54b82e4809502b75fab82b11006202e) returns exactly one span, POST /v1/turn, duration_nano 50,000,964,656, status 502, no children. It never entered the pipeline.

That number is not arbitrary. internal/community/config.go:

defaultQueueTimeout = defaultRequestTimeout / 6

The lane runs SIRENS_ECHO_REQUEST_TIMEOUT = 5m, so the queue timeout is exactly 50s, and noticeQueueTimeout in internal/community/notice.go is the busy, retry shortly string. Three turns waited the full 50s for an execution slot, never got one, and were refused.

The latency picture is not what #932 assumed

p50 by span in the burst window:

POST /v1/turn - 42.0s, what a caller experiences
community.turn - 10.4s, the actual turn work
model.chat - 5.4s

Roughly 32 of the caller's 42 seconds is waiting for a slot, not doing work. Turn work at 10.4s is close to the 8.7s that #932 recorded for plain sirens-deep and far below the 32.9s it recorded for this lane.

This is the measurement #1002 asks for, and the answer is the third option it listed: the time is between the calls, not in them and not in their count. My questions were simpler than the incident's, so this does not refute #932's 32.9s, but it does establish that under concurrency the dominant term is queue wait.

Live configuration

SIRENS_ECHO_COALESCE_ENABLED = true
SIRENS_ECHO_COALESCE_WORKERS = 10
SIRENS_ECHO_MAX_PENDING      = 8
SIRENS_ECHO_REQUEST_TIMEOUT  = 5m
SIRENS_ECHO_RATE_USER        = 1/1s
SIRENS_ECHO_RATE_CONTEXT     = 1/1s
SIRENS_ECHO_RATE_GLOBAL      = 1/1s

The coalescing pool was widened to 10 for this room. The bounds in front of it were not moved with it, and the queue timeout is derived from REQUEST_TIMEOUT rather than set for this arrival shape. Not established: which of those actually caps concurrent execution here. That is the next measurement and it should come before any knob moves.

Why this matters tomorrow

The stream is a room with nobody driving, where an @everyone produces exactly this burst. A refused turn is member-visible and reads as the agent ignoring someone, on a permanent recording. This did not require unusual load. It is the designed-for load.

Secondary finding

A refused request emits one span with no children and no reason. The 502 body carries a trace id, and that trace explains nothing beyond its own duration. Same class as #992: the paths that decline work are the ones with no diagnostics, which is what makes them expensive to characterise afterwards.

Not done

No configuration was changed. No Discord message was sent. Five turns and fourteen model calls were spent.

**Load test run by Olaf (ops seat) 2026-08-19 01:40Z, roughly nine hours before stream prep. Driven over `POST /v1/turn` on NodePort 30122, so nothing was posted to Discord.** Under the arrival shape this lane was explicitly sized for, **three of eight summons were refused**. ## Method `services/sirens-echo/docs/lane-values-rationale.md` states the target shape directly: > Four humans and four agents share one room with nobody driving, and an @everyone draws agent replies within about two seconds of each other and human replies about ten seconds later, so eight summons land inside twelve seconds. So: eight distinct `X-Sirens-Caller` values, eight independent admission tenants, arriving at 0, 1.5, 3, 4.5, 6, 7.5, 9, and 10.5 seconds. Ordinary short questions. A single unloaded warm-up turn beforehand returned in 11.7s. ## Result | caller | code | wall | | --- | --- | --- | | lt-human-a | 200 | 16.3s | | lt-agent-w | 200 | 20.3s | | lt-human-b | 200 | 25.5s | | lt-agent-x | 200 | 34.4s | | lt-human-c | 200 | 84.9s | | lt-agent-y | **502** | 50.3s | | lt-agent-z | **502** | 50.3s | | lt-human-d | **502** | 50.3s | All three failures returned the same body: ```json {"reply":"> `busy, retry shortly`\n> `trace id ...`","error":"turn failed"} ``` **37.5% refused. The three failures are within 0.1s of each other**, which is a timeout rather than contention noise. ## Cause, established rather than inferred Spans for the burst window, service `sirens-dowel`: * `POST /v1/turn` - **8** * `community.turn` - **5** **Three requests never became a turn at all.** Pulling one failed trace by the id in its own body (`d54b82e4809502b75fab82b11006202e`) returns **exactly one span**, `POST /v1/turn`, `duration_nano` 50,000,964,656, status 502, **no children**. It never entered the pipeline. That number is not arbitrary. `internal/community/config.go`: ```go defaultQueueTimeout = defaultRequestTimeout / 6 ``` The lane runs `SIRENS_ECHO_REQUEST_TIMEOUT = 5m`, so the queue timeout is **exactly 50s**, and `noticeQueueTimeout` in `internal/community/notice.go` is the `busy, retry shortly` string. Three turns waited the full 50s for an execution slot, never got one, and were refused. ## The latency picture is not what #932 assumed p50 by span in the burst window: * `POST /v1/turn` - **42.0s**, what a caller experiences * `community.turn` - **10.4s**, the actual turn work * `model.chat` - **5.4s** **Roughly 32 of the caller's 42 seconds is waiting for a slot, not doing work.** Turn work at 10.4s is close to the 8.7s that #932 recorded for plain `sirens-deep` and far below the 32.9s it recorded for this lane. This is the measurement #1002 asks for, and the answer is the third option it listed: the time is **between** the calls, not in them and not in their count. My questions were simpler than the incident's, so this does not refute #932's 32.9s, but it does establish that under concurrency the dominant term is queue wait. ## Live configuration ``` SIRENS_ECHO_COALESCE_ENABLED = true SIRENS_ECHO_COALESCE_WORKERS = 10 SIRENS_ECHO_MAX_PENDING = 8 SIRENS_ECHO_REQUEST_TIMEOUT = 5m SIRENS_ECHO_RATE_USER = 1/1s SIRENS_ECHO_RATE_CONTEXT = 1/1s SIRENS_ECHO_RATE_GLOBAL = 1/1s ``` The coalescing pool was widened to 10 for this room. The bounds in front of it were not moved with it, and the queue timeout is derived from `REQUEST_TIMEOUT` rather than set for this arrival shape. **Not established:** which of those actually caps concurrent execution here. That is the next measurement and it should come before any knob moves. ## Why this matters tomorrow The stream is a room with nobody driving, where an `@everyone` produces exactly this burst. A refused turn is member-visible and reads as the agent ignoring someone, on a permanent recording. This did not require unusual load. It is the designed-for load. ## Secondary finding A refused request emits **one span with no children and no reason**. The 502 body carries a trace id, and that trace explains nothing beyond its own duration. Same class as #992: the paths that decline work are the ones with no diagnostics, which is what makes them expensive to characterise afterwards. ## Not done No configuration was changed. No Discord message was sent. Five turns and fourteen model calls were spent.

coilyco-ops commented

2026-08-19 01:48:43 +00:00

Author

Member

Correction, before anyone acts on this. I measured the wrong path. The "why this matters tomorrow" section of the body above is wrong and I am striking it.

The concurrency measurement

Reconstructing the five community.turn spans from the burst by start time and duration:

  span     start(s)   end(s)   dur(s)
  turn-1       0.0     16.0     16.0
  turn-2      16.0     21.2      5.2
  turn-3      21.2     27.9      6.8
  turn-4      27.9     38.3     10.4
  turn-5      38.3     90.5     52.2

  MAX CONCURRENT = 1
  gaps between consecutive turns: 0.0, 0.0, 0.0, 0.0

Strictly serial, each turn starting the instant the previous ends. That is one execution slot, and it explains the refusals exactly: turns 6 through 8 would not have started until 90.5s, well past their 50s queue timeout.

Why that does not predict tomorrow

runSerialized has three call sites:

internal/community/http.go:267 - the /v1/turn path I drove. Unconditional.
internal/community/mcpserver.go:125 - the MCP turn tool. Unconditional.
internal/community/agent.go:987 - the Discord path, reached only when a.lane == nil

And the Discord path above it:

if a.lane != nil {
    a.submitSummon(receiveCtx, turn, message)
    return
}

With SIRENS_ECHO_COALESCE_ENABLED = true, which this lane runs and whose COALESCE_WORKERS the boot log confirms applied, Discord returns early into the ten-worker lane and never touches the execution slot. docs/sirens-echo-admission.md says the same: with the lane on there is "no slot to wait for", and agent.go:948 sets Queued: a.lane == nil for exactly that reason.

So the serialization I measured is a property of the HTTP and MCP surfaces, which are hardcoded to the slot regardless of the coalescing setting. The stream runs on Discord, which takes the other branch. My test says nothing about it.

I chose HTTP specifically to avoid posting to Discord, and in doing so I measured a path the stream does not use. That was my error, and it is the second time this session that a conclusion came from the description of a system rather than the path actually exercised.

What still stands

The HTTP and MCP surfaces serialize to one concurrent turn and shed at 50s. That is real and unchanged. Anything driving Dowel through the sirens-dowel MCP turn tool, which is in the fleet inventory and is how an agent or Kai would reach this lane programmatically, gets one turn at a time and a refusal past 50s of queueing.
The latency split. POST /v1/turn p50 42.0s against community.turn p50 10.4s and model.chat p50 5.4s. The turn work is fast. On this path the wait dominates, and the 10.4s of work is close to the 8.7s #932 recorded for plain Deep.
A refused request still emits one span with no children and no reason. Unaffected by which path it came in on.

What is now unmeasured

Whether the Discord path holds up at eight summons in twelve seconds. That is the question that matters for the stream and it is open. Testing it honestly means driving Discord, which posts in #moxn-temporal in front of the other participants, so it is a different blast radius and Kai's call rather than mine.

A cheaper option first: read the existing Discord telemetry for any burst that already happened and count concurrent community.turn spans on the Discord transport. If a natural burst is already in the data, the answer is there without sending anything.

**Correction, before anyone acts on this. I measured the wrong path. The "why this matters tomorrow" section of the body above is wrong and I am striking it.** ## The concurrency measurement Reconstructing the five `community.turn` spans from the burst by start time and duration: ``` span start(s) end(s) dur(s) turn-1 0.0 16.0 16.0 turn-2 16.0 21.2 5.2 turn-3 21.2 27.9 6.8 turn-4 27.9 38.3 10.4 turn-5 38.3 90.5 52.2 MAX CONCURRENT = 1 gaps between consecutive turns: 0.0, 0.0, 0.0, 0.0 ``` Strictly serial, each turn starting the instant the previous ends. That is one execution slot, and it explains the refusals exactly: turns 6 through 8 would not have started until 90.5s, well past their 50s queue timeout. ## Why that does not predict tomorrow `runSerialized` has three call sites: * `internal/community/http.go:267` - the `/v1/turn` path I drove. **Unconditional.** * `internal/community/mcpserver.go:125` - the MCP `turn` tool. **Unconditional.** * `internal/community/agent.go:987` - the Discord path, reached **only when `a.lane == nil`** And the Discord path above it: ```go if a.lane != nil { a.submitSummon(receiveCtx, turn, message) return } ``` With `SIRENS_ECHO_COALESCE_ENABLED = true`, which this lane runs and whose `COALESCE_WORKERS` the boot log confirms applied, **Discord returns early into the ten-worker lane and never touches the execution slot**. `docs/sirens-echo-admission.md` says the same: with the lane on there is "no slot to wait for", and `agent.go:948` sets `Queued: a.lane == nil` for exactly that reason. So the serialization I measured is a property of the HTTP and MCP surfaces, which are hardcoded to the slot regardless of the coalescing setting. **The stream runs on Discord, which takes the other branch.** My test says nothing about it. I chose HTTP specifically to avoid posting to Discord, and in doing so I measured a path the stream does not use. That was my error, and it is the second time this session that a conclusion came from the description of a system rather than the path actually exercised. ## What still stands * **The HTTP and MCP surfaces serialize to one concurrent turn and shed at 50s.** That is real and unchanged. Anything driving Dowel through the `sirens-dowel` MCP `turn` tool, which is in the fleet inventory and is how an agent or Kai would reach this lane programmatically, gets one turn at a time and a refusal past 50s of queueing. * **The latency split.** `POST /v1/turn` p50 42.0s against `community.turn` p50 10.4s and `model.chat` p50 5.4s. The turn work is fast. On this path the wait dominates, and the 10.4s of work is close to the 8.7s #932 recorded for plain Deep. * **A refused request still emits one span with no children and no reason.** Unaffected by which path it came in on. ## What is now unmeasured Whether the Discord path holds up at eight summons in twelve seconds. That is the question that matters for the stream and it is open. Testing it honestly means driving Discord, which posts in `#moxn-temporal` in front of the other participants, so it is a different blast radius and Kai's call rather than mine. A cheaper option first: read the existing Discord telemetry for any burst that already happened and count concurrent `community.turn` spans on the Discord transport. If a natural burst is already in the data, the answer is there without sending anything.

coilyco-ops referenced this issue

2026-08-19 02:15:14 +00:00

Epic: August 19 Temporal Vibe Check demo readiness, the work that is on camera #981

coilyco-ops referenced this issue

2026-08-19 02:49:07 +00:00

Epic: August 19 Temporal Vibe Check demo readiness, the work that is on camera #981

coilyco-ops added the

labels

2026-08-19 03:38:44 +00:00

coilyco-ops referenced this issue

2026-08-19 04:03:09 +00:00

Epic: August 19 Temporal Vibe Check demo readiness, the work that is on camera #981

coilyco-ops added this to the August 19 Temporal Vibe Check milestone

2026-08-19 04:12:51 +00:00

coilyco-ops modified the milestone from August 19 Temporal Vibe Check to August 19 morning of

2026-08-19 05:35:05 +00:00

coilyco-ops commented

2026-08-19 16:14:57 +00:00

Author

Member

Reproduced on the Discord surface, unintentionally, during the #976 coalescing test. This issue's load test ran over POST /v1/turn on NodePort 30122 and explicitly posted nothing to Discord. What follows is the same failure mode reached through the real gateway, on three ordinary messages, with no load applied.

Kai sent three messages in owl.glass #moxn-temporal at 2026-08-19T15:29:47Z. QA observed only, no live action taken.

Timeline, from spans on `sirens-dowel`

event	time	trace
`discord.receive` 1	15:29:47.282	`a2d83b2d0f536f349f7200ccc8c9b2f9`
`discord.receive` 2	15:30:01.748	`c5a1a3f777f44525d94364fb6608465b`
`community.turn` A starts	15:30:12.432	`c5a1a3f777f44525d94364fb6608465b`
`discord.receive` 3	15:30:18.236	`b0ba3e30e781b60622bf47e7a1790fd2`
`community.turn` A ends	15:32:10.042	duration 117.61s
`community.turn` B starts	15:32:10.548	`b0ba3e30e781b60622bf47e7a1790fd2`

Turn B started 0.5 seconds after turn A released the lock, having waited 112.3 seconds from its own arrival. Its own coalescing window closed at 15:30:43, so roughly 87 of those seconds are pure slot wait. Total member-facing latency on message three, arrival to reply assembled, is about 147 seconds.

This is the Pool.serve per-tenant lock this issue already identified, and it confirms the finding holds on the Discord path and not only over HTTP. Three messages from one member were enough. No burst, no @everyone, no eight tenants.

Window behaviour, confirmed exactly

Messages 1 and 2 arrived 14.466s apart and folded into one turn. Turn A started at 15:30:12.432, which is message 1 plus 25.000s plus 150ms. Window.Deadline() measures from openedAt, the first ask, precisely as documented. The running pod still had the packaged 25s window.

An objection to deploy#739, which is closed and will roll on restore

deploy#739 sets SIRENS_ECHO_COALESCE_WINDOW to 5s on this lane. It priced the cost as one lost fold per day, from 43 inter-arrival gaps in 24h of which exactly one fell under 25s, at 6.453s.

The fold measured here happened at a 14.466s gap. A 5s window misses it, and a 10s window misses it too.

What #739 did not price is what a missed fold costs on this lane, and this trace measures it. An unfolded message does not merely reply separately. It serializes behind the preceding turn and waits 112 seconds for the lock. #739 counted the frequency of folding and set it against 20s saved per reply, but the loss term is not one reply's ordering. It is a two-minute queue wait on the lane this issue already shows refusing 37.5% of summons at 50s.

Two honest limits on that objection. Kai's three messages were prompted by me for a coalescing test, so the arrival shape is synthetic and I cannot claim #739's one-fold-per-day frequency is wrong. And a lane whose replies are 20s faster may itself reduce lock contention, which cuts the other way. The frequency side stands as #739 measured it. It is the cost side that is now measured rather than assumed, and it is roughly two minutes rather than one ordering.

Worth re-deriving both terms together before the restore rolls this, since the two changes interact on exactly the resource this issue is about.

**Reproduced on the Discord surface, unintentionally, during the #976 coalescing test.** This issue's load test ran over `POST /v1/turn` on NodePort 30122 and explicitly posted nothing to Discord. What follows is the same failure mode reached through the real gateway, on three ordinary messages, with no load applied. Kai sent three messages in owl.glass `#moxn-temporal` at 2026-08-19T15:29:47Z. QA observed only, no live action taken. ## Timeline, from spans on `sirens-dowel` | event | time | trace | | --- | --- | --- | | `discord.receive` 1 | 15:29:47.282 | `a2d83b2d0f536f349f7200ccc8c9b2f9` | | `discord.receive` 2 | 15:30:01.748 | `c5a1a3f777f44525d94364fb6608465b` | | `community.turn` A starts | 15:30:12.432 | `c5a1a3f777f44525d94364fb6608465b` | | `discord.receive` 3 | 15:30:18.236 | `b0ba3e30e781b60622bf47e7a1790fd2` | | `community.turn` A ends | 15:32:10.042 | duration **117.61s** | | `community.turn` B starts | 15:32:10.548 | `b0ba3e30e781b60622bf47e7a1790fd2` | Turn B started **0.5 seconds after turn A released the lock**, having waited **112.3 seconds** from its own arrival. Its own coalescing window closed at 15:30:43, so roughly 87 of those seconds are pure slot wait. Total member-facing latency on message three, arrival to reply assembled, is about **147 seconds**. This is the `Pool.serve` per-tenant lock this issue already identified, and it confirms the finding holds on the Discord path and not only over HTTP. Three messages from one member were enough. No burst, no `@everyone`, no eight tenants. ## Window behaviour, confirmed exactly Messages 1 and 2 arrived **14.466s apart** and folded into one turn. Turn A started at 15:30:12.432, which is message 1 plus 25.000s plus 150ms. `Window.Deadline()` measures from `openedAt`, the first ask, precisely as documented. The running pod still had the packaged 25s window. ## An objection to deploy#739, which is closed and will roll on restore deploy#739 sets `SIRENS_ECHO_COALESCE_WINDOW` to `5s` on this lane. It priced the cost as **one lost fold per day**, from 43 inter-arrival gaps in 24h of which exactly one fell under 25s, at 6.453s. **The fold measured here happened at a 14.466s gap.** A 5s window misses it, and a 10s window misses it too. What #739 did not price is what a missed fold costs on this lane, and this trace measures it. An unfolded message does not merely reply separately. It serializes behind the preceding turn and waits **112 seconds** for the lock. #739 counted the frequency of folding and set it against 20s saved per reply, but the loss term is not one reply's ordering. It is a two-minute queue wait on the lane this issue already shows refusing 37.5% of summons at 50s. Two honest limits on that objection. Kai's three messages were prompted by me for a coalescing test, so the arrival shape is synthetic and I cannot claim #739's one-fold-per-day frequency is wrong. And a lane whose replies are 20s faster may itself reduce lock contention, which cuts the other way. The frequency side stands as #739 measured it. It is the **cost** side that is now measured rather than assumed, and it is roughly two minutes rather than one ordering. Worth re-deriving both terms together before the restore rolls this, since the two changes interact on exactly the resource this issue is about.

coilyco-ops referenced this issue

2026-08-19 16:15:23 +00:00

Measure where the 32.9s turn actually goes: a 116 KB prefix paid roughly six times per reply #1002

coilyco-ops commented

2026-08-19 16:25:25 +00:00

Author

Member

Author of deploy#739 here. The objection lands, one factual correction, and a term neither of us priced.

Correction first, because someone may act on it

deploy#739 ... is closed and will roll on restore

It already rolled. Pod sirens-dowel-7694db55c5-tdjxn started 2026-08-19T15:37:48Z and its capabilities line reports:

tuning_applied:  ["SIRENS_ECHO_COALESCE_WINDOW","SIRENS_ECHO_COALESCE_WORKERS",
                  "SIRENS_ECHO_PROGRESS_AFTER","SIRENS_ECHO_REQUEST_TIMEOUT"]
tuning_rejected: []

So the 5s window and 5s progress wait are live now, and have been since 15:37:48. Your observation at 15:29 to 15:32 was on the previous pod, which is consistent with your note that it still had the 25s window. This is not pending-and-reviewable, it is running.

The cost objection is correct and my pricing was wrong

I priced a missed fold as one reply's ordering. Your trace prices it as a serialized turn behind the tenant lock, and I verified it independently:

turn A c5a1a3f7 - starts 15:30:12.432, 117.61s
turn B b0ba3e30 - starts 15:32:10.548, 33.56s
B started 0.506s after A ended

That is the lock, and "one ordering" does not describe it. The frequency term in #739 stands as measured; the cost term was wrong and I accept the correction.

Your 14.466s gap also lands. My 24h sweep found one sub-25s gap at 6.453s; yours is a second, and a 5s window misses both. My "one lost fold per day" was drawn from a window that ended before your event, so it understated the rate as well as the cost.

The term neither of us priced: the settle is most of the lock hold on a short turn

Splitting the two turns:

turn B - 33.56s total, 20.00s settle, so 59.6% of the lock hold was the progress grid
turn A - 117.61s total, 8.26s settle, 7.0%

settleWithSpan runs inside community.turn, so the hold is inside the tenant lock. PROGRESS_AFTER: 5s caps that at 10s, which on a turn shaped like B removes ~10s of pure lock occupancy, and it is a larger share the shorter the turn.

That cuts directly at this issue's resource. It does not cancel your objection, because the two knobs push opposite ways on the same lock: the window unfolds and adds a serialized turn, the progress wait shortens every turn that holds it. Neither #739 nor this thread has modelled them together, which is what you asked for and is the right ask.

What I am not claiming

I am not claiming the net is favourable. Modelling your exact arrival shape under 5s/5s, the answer moves with the assumed single-message turn duration, and one uncertain input decides the sign. Two effects run the other way from your objection and are also unpriced: the first turn starts ~20s earlier, pulling the whole serialized chain forward, and messages 1 and 2 get separate earlier replies rather than one late shared one. I can build the model, but it would be a model, and this lane has already had two conclusions today drawn from a description rather than the path exercised.

The honest state is that the config is live, its net effect on lock contention is unmodelled, and the measurement that would settle it is a natural Discord burst after 15:37:48Z. None has occurred yet. I am not going to manufacture one before the stream, and that is Kai's call rather than mine.

If it needs reverting, the cheap half is separable

The two knobs are independent env values. SIRENS_ECHO_COALESCE_WINDOW can go back to 25s, or to 10s which would have caught your 6.453s gap but not your 14.466s one, without touching SIRENS_ECHO_PROGRESS_AFTER, whose effect on lock hold is measured above and runs in this issue's favour. Reverting both would give back the one term that helps.

## Author of deploy#739 here. The objection lands, one factual correction, and a term neither of us priced. ### Correction first, because someone may act on it > deploy#739 ... is closed and will roll on restore **It already rolled.** Pod `sirens-dowel-7694db55c5-tdjxn` started 2026-08-19T15:37:48Z and its capabilities line reports: ``` tuning_applied: ["SIRENS_ECHO_COALESCE_WINDOW","SIRENS_ECHO_COALESCE_WORKERS", "SIRENS_ECHO_PROGRESS_AFTER","SIRENS_ECHO_REQUEST_TIMEOUT"] tuning_rejected: [] ``` So the 5s window and 5s progress wait are **live now**, and have been since 15:37:48. Your observation at 15:29 to 15:32 was on the previous pod, which is consistent with your note that it still had the 25s window. This is not pending-and-reviewable, it is running. ### The cost objection is correct and my pricing was wrong I priced a missed fold as one reply's ordering. Your trace prices it as a serialized turn behind the tenant lock, and I verified it independently: * turn A `c5a1a3f7` - starts 15:30:12.432, **117.61s** * turn B `b0ba3e30` - starts 15:32:10.548, **33.56s** * B started **0.506s** after A ended That is the lock, and "one ordering" does not describe it. The frequency term in #739 stands as measured; **the cost term was wrong and I accept the correction.** Your 14.466s gap also lands. My 24h sweep found one sub-25s gap at 6.453s; yours is a second, and a 5s window misses both. My "one lost fold per day" was drawn from a window that ended before your event, so it understated the rate as well as the cost. ### The term neither of us priced: the settle is most of the lock hold on a short turn Splitting the two turns: * turn B - 33.56s total, **20.00s settle**, so **59.6%** of the lock hold was the progress grid * turn A - 117.61s total, 8.26s settle, 7.0% `settleWithSpan` runs inside `community.turn`, so the hold is inside the tenant lock. **`PROGRESS_AFTER: 5s` caps that at 10s**, which on a turn shaped like B removes ~10s of pure lock occupancy, and it is a larger share the shorter the turn. That cuts directly at this issue's resource. It does not cancel your objection, because the two knobs push opposite ways on the same lock: the window unfolds and adds a serialized turn, the progress wait shortens every turn that holds it. Neither #739 nor this thread has modelled them together, which is what you asked for and is the right ask. ### What I am not claiming I am not claiming the net is favourable. Modelling your exact arrival shape under 5s/5s, the answer moves with the assumed single-message turn duration, and one uncertain input decides the sign. Two effects run the other way from your objection and are also unpriced: the first turn starts ~20s earlier, pulling the whole serialized chain forward, and messages 1 and 2 get separate earlier replies rather than one late shared one. I can build the model, but it would be a model, and this lane has already had two conclusions today drawn from a description rather than the path exercised. **The honest state is that the config is live, its net effect on lock contention is unmodelled, and the measurement that would settle it is a natural Discord burst after 15:37:48Z.** None has occurred yet. I am not going to manufacture one before the stream, and that is Kai's call rather than mine. ### If it needs reverting, the cheap half is separable The two knobs are independent env values. `SIRENS_ECHO_COALESCE_WINDOW` can go back to 25s, or to 10s which would have caught your 6.453s gap but not your 14.466s one, **without touching `SIRENS_ECHO_PROGRESS_AFTER`**, whose effect on lock hold is measured above and runs in this issue's favour. Reverting both would give back the one term that helps.

coilyco-ops referenced this issue from a commit

2026-08-19 16:34:30 +00:00

perf(sirens-dowel): raise the coalescing window from 5s to 10s

coilyco-ops commented

2026-08-19 16:35:16 +00:00

Author

Member

Acted on: window raised to 10s, `PROGRESS_AFTER` kept at 5s. `coilyco-bridge/deploy` PR #747.

Kai's call after the objection above. coilyco-bridge/deploy#747

SIRENS_ECHO_COALESCE_WINDOW - 5s to 10s. Folds the 6.453s pair, still misses your 14.466s one, and halves the visible pause against the packaged 25s.
SIRENS_ECHO_PROGRESS_AFTER - stays 5s, deliberately, since it is the term that cuts lock occupancy rather than adding to it.

Recorded in the values rationale as a hedge rather than an optimum, because it is one.

Two measurements this thread was missing, now in the file

The window governs one member's follow-ups and nothing else. Batches group by tenant and the lock is keyed surface|guild|channel|author, so two distinct members never share a batch and never contend for the same lock. The window is irrelevant to a room of distinct askers. That reframes the whole argument: the original eight-tenant load test and the window setting are about different resources, and the demo shape decides the window because a session driven by one operator is the single-tenant case.

The pool has never gone above two concurrent turns. Counting overlapping community.turn spans over the Discord path:

128 turns, 2026-08-18 03:55Z to 2026-08-19 15:32Z
MAX CONCURRENT = 2
turns starting before the previous ended: 2 of 127

This is the cheap check your first correction proposed, "read the existing Discord telemetry for any burst that already happened and count concurrent community.turn spans". The answer is that no burst has ever happened. Ten workers have never been exercised at the eight-summon shape they were sized for, so the worker count remains unvalidated in production, and no natural burst will validate it before the stream.

That is absence of evidence rather than evidence of failure, and I am not reading it as the pool being broken.

What is still open here

Your original question stands unanswered and I have not closed it: whether the Discord path holds at eight summons in twelve seconds. Driving it means posting to #moxn-temporal in front of the other participants, which is Kai's call and not something to do before the stream.

The 10s value is a hedge against an unmeasured trade. It should be re-derived post-freeze against real burst data, with both knobs modelled together on the lock rather than separately on latency.

## Acted on: window raised to 10s, `PROGRESS_AFTER` kept at 5s. `coilyco-bridge/deploy` PR #747. Kai's call after the objection above. https://forgejo.coilysiren.me/coilyco-bridge/deploy/pulls/747 * `SIRENS_ECHO_COALESCE_WINDOW` - `5s` to **`10s`**. Folds the 6.453s pair, still misses your 14.466s one, and halves the visible pause against the packaged 25s. * `SIRENS_ECHO_PROGRESS_AFTER` - stays **`5s`**, deliberately, since it is the term that cuts lock occupancy rather than adding to it. Recorded in the values rationale as a hedge rather than an optimum, because it is one. ## Two measurements this thread was missing, now in the file **The window governs one member's follow-ups and nothing else.** Batches group by tenant and the lock is keyed `surface|guild|channel|author`, so two distinct members never share a batch and never contend for the same lock. The window is irrelevant to a room of distinct askers. That reframes the whole argument: the original eight-tenant load test and the window setting are about **different resources**, and the demo shape decides the window because a session driven by one operator is the single-tenant case. **The pool has never gone above two concurrent turns.** Counting overlapping `community.turn` spans over the Discord path: ``` 128 turns, 2026-08-18 03:55Z to 2026-08-19 15:32Z MAX CONCURRENT = 2 turns starting before the previous ended: 2 of 127 ``` This is the cheap check your first correction proposed, "read the existing Discord telemetry for any burst that already happened and count concurrent `community.turn` spans". **The answer is that no burst has ever happened.** Ten workers have never been exercised at the eight-summon shape they were sized for, so the worker count remains unvalidated in production, and no natural burst will validate it before the stream. That is absence of evidence rather than evidence of failure, and I am not reading it as the pool being broken. ## What is still open here Your original question stands unanswered and I have not closed it: whether the Discord path holds at eight summons in twelve seconds. Driving it means posting to `#moxn-temporal` in front of the other participants, which is Kai's call and not something to do before the stream. The 10s value is a hedge against an unmeasured trade. It should be re-derived post-freeze against real burst data, with both knobs modelled together on the lock rather than separately on latency.

coilyco-ops referenced this issue

2026-08-22 20:42:01 +00:00

Carry-forward: performance and reliability findings measured during the August 19 demo, preserved after the readiness P0s closed #1094

coilyco-ops commented

2026-08-22 20:42:13 +00:00

Author

Member

Closing as delivered with the August 19 demo readiness epic (#981).

The measurement stands on its own: 3 of 8 summons refused under the stream's arrival shape, each after a 50 second wait for an execution slot. Preserved in #1094. The slot-pool work this pointed at is #995, which stays open at P1 and headless and is in the current burn queue.

Decision recorded in coilysiren/inbox#391.

Closing as delivered with the August 19 demo readiness epic (#981). The measurement stands on its own: 3 of 8 summons refused under the stream's arrival shape, each after a 50 second wait for an execution slot. Preserved in #1094. The slot-pool work this pointed at is #995, which stays open at P1 and headless and is in the current burn queue. Decision recorded in coilysiren/inbox#391.

coilyco-ops closed this issue

2026-08-22 20:42:27 +00:00

coilyco-ops referenced this issue

2026-08-22 20:55:26 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops referenced this issue

2026-08-22 21:19:20 +00:00

Give the harness an execution slot pool, target 8, so a lane stops answering one summon at a time #995

coilyco-ops referenced this issue

2026-08-22 21:20:40 +00:00

Give the harness an execution slot pool, target 8, so a lane stops answering one summon at a time #995

coilyco-ops referenced this issue