Measure the coalescing lane once #1080 rolls, and check how a merged reply reads #1082

New issue

Open

opened 2026-08-19 16:09:51 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-19 16:09:51 +00:00

Member

Already settled, not to be re-litigated

The flip happened. SIRENS_ECHO_COALESCE_ENABLED is "true" on all three lanes. Echo flipped in deploy b8d12c8, cleared the pre-demo lockdown boundary.
Coalescing works against a live gateway. 2026-08-19T15:24Z on dowel: two messages ~5s apart merged into one turn. One batch of 2 and one batch of 1 from 2 turns. That closes #976's "never run against a live gateway" gap.
The flat-at-1 batch reading was wrong twice over. First as a traffic artifact (dowel runs ~5 admissions/hour against a 25s window, so two messages rarely coincide), then as an instrument defect (coalesce.batch.size used default boundaries whose first bucket is [0, 5), so a batch of 1 and a batch of 4 were indistinguishable). The lane was correctly idle, and the histogram could not have shown otherwise.

Blocked: the fix is merged but not running

#1080 landed the corrected histogram boundaries on main at c8139e6. The lane is running image 893c469e. Nothing measured before that rolls is worth recording, because the old boundaries top out at 10,000 ms while turn p50 is ~17s, so every percentile returns the boundary rather than a duration.

First action here is confirming the rollout, not querying.

Read these as delta or you will conclude they are broken

The SigNoz catalog metadata declares five sirens_echo.* names cumulative. The exporter has set DeltaTemporalitySelector since first release, so the declaration is simply wrong. Same metric, same 24h window:

queried as delta - dowel 121, deep 10, echo 4
queried as cumulative, per the catalog - empty

#976's QA pass read the empty result as a broken pipeline and filed it as a blocker. It was not. Pass temporality: delta explicitly. This trap is not specific to this issue and will catch anyone reading sirens-echo metrics until the catalog rows are corrected.

What to record, after the rollout

p50 and p99 sirens_echo.turn.duration, now that the buckets can express a real turn
sirens_echo.coalesce.batch.size distribution, now that small integers are separable
sirens_echo.coalesce.asks split by accepted and shed, and coalesce.queue.depth

Give it enough traffic to mean something. Echo's post-flip window at the time of #976 was 3 coalesce turns and 4 admissions, which is not a sample.

One caution: coalesce.turns and coalesce.turn.duration record on turn completion and dowel turns run long, so they lag batch.size by minutes. A snapshot taken too early shows batches with zero turns and reads as a stuck worker pool. Re-measure before concluding anything from that shape.

The before half cannot be produced, and that is final

Histogram boundaries change what an instrument can express, never what it already recorded. Pre-rollout turn latency is not recoverable from sirens_echo.turn.duration at any point in the future.

community.turn span durations in traces are the only before-window fallback, which is what #1076 used to establish p50 17.2s and p95 181s. Anyone expecting a clean histogram before-and-after should stop looking for one.

The human-only half

A member who posts several comments in quick succession now reads one reply covering them. #976 flagged this as worth watching, whether it reads as attentive or as dropped, and it is still unverified.

QA could not check it: the lane is scoped to owl.glass #moxn-temporal by sirens-dowel-access-policy.yml, and the agent's Discord identity is the Echo bot, which has no access to that guild. During the live test three admissions were observed where two messages were expected, and it was not resolvable from outside whether a third message was sent or a real member posted inside the window.

This one needs Kai reading the channel.

#976 - predecessor, closed
#1080 - the histogram fix, merged and awaiting rollout
#1076 - the trace-derived before-window latency figures

Successor to #976, which is closed. The flip it existed to decide is decided, shipped, and verified working. This carries only the measurement that is still owed, plus the one gap a human has to close. ## Already settled, not to be re-litigated * **The flip happened.** `SIRENS_ECHO_COALESCE_ENABLED` is `"true"` on all three lanes. Echo flipped in deploy `b8d12c8`, cleared the pre-demo lockdown boundary. * **Coalescing works against a live gateway.** 2026-08-19T15:24Z on dowel: two messages ~5s apart merged into one turn. One batch of 2 and one batch of 1 from 2 turns. That closes #976's "never run against a live gateway" gap. * **The flat-at-1 batch reading was wrong twice over.** First as a traffic artifact (dowel runs ~5 admissions/hour against a 25s window, so two messages rarely coincide), then as an instrument defect (`coalesce.batch.size` used default boundaries whose first bucket is `[0, 5)`, so a batch of 1 and a batch of 4 were indistinguishable). The lane was correctly idle, and the histogram could not have shown otherwise. ## Blocked: the fix is merged but not running #1080 landed the corrected histogram boundaries on `main` at `c8139e6`. The lane is running image `893c469e`. **Nothing measured before that rolls is worth recording**, because the old boundaries top out at 10,000 ms while turn p50 is ~17s, so every percentile returns the boundary rather than a duration. First action here is confirming the rollout, not querying. ## Read these as delta or you will conclude they are broken The SigNoz catalog metadata declares five `sirens_echo.*` names cumulative. The exporter has set `DeltaTemporalitySelector` since first release, so the declaration is simply wrong. Same metric, same 24h window: * queried as **delta** - dowel 121, deep 10, echo 4 * queried as **cumulative**, per the catalog - empty #976's QA pass read the empty result as a broken pipeline and filed it as a blocker. It was not. Pass `temporality: delta` explicitly. This trap is not specific to this issue and will catch anyone reading sirens-echo metrics until the catalog rows are corrected. ## What to record, after the rollout * p50 and p99 `sirens_echo.turn.duration`, now that the buckets can express a real turn * `sirens_echo.coalesce.batch.size` distribution, now that small integers are separable * `sirens_echo.coalesce.asks` split by `accepted` and `shed`, and `coalesce.queue.depth` Give it enough traffic to mean something. Echo's post-flip window at the time of #976 was 3 coalesce turns and 4 admissions, which is not a sample. **One caution:** `coalesce.turns` and `coalesce.turn.duration` record on turn completion and dowel turns run long, so they lag `batch.size` by minutes. A snapshot taken too early shows batches with zero turns and reads as a stuck worker pool. Re-measure before concluding anything from that shape. ## The before half cannot be produced, and that is final Histogram boundaries change what an instrument can express, never what it already recorded. Pre-rollout turn latency is not recoverable from `sirens_echo.turn.duration` at any point in the future. `community.turn` span durations in traces are the only before-window fallback, which is what #1076 used to establish p50 17.2s and p95 181s. Anyone expecting a clean histogram before-and-after should stop looking for one. ## The human-only half A member who posts several comments in quick succession now reads **one** reply covering them. #976 flagged this as worth watching, whether it reads as attentive or as dropped, and it is still unverified. QA could not check it: the lane is scoped to owl.glass `#moxn-temporal` by `sirens-dowel-access-policy.yml`, and the agent's Discord identity is the Echo bot, which has no access to that guild. During the live test three admissions were observed where two messages were expected, and it was not resolvable from outside whether a third message was sent or a real member posted inside the window. This one needs Kai reading the channel. ## Related * #976 - predecessor, closed * #1080 - the histogram fix, merged and awaiting rollout * #1076 - the trace-derived before-window latency figures

coilyco-ops added the

labels

2026-08-19 16:09:51 +00:00

coilyco-ops referenced this issue

2026-08-19 16:10:14 +00:00

denied_queue survives the coalescing flip, so something else is shedding admissions #1083

coilyco-ops referenced this issue

2026-08-19 16:10:34 +00:00