coilyco-gaming/sirens-echo

Watch

Turn the coalescing lane on, with before-and-after evidence #976

New issue

Closed

opened 2026-08-18 20:03:18 +00:00 by coilyco-ops · 5 comments

coilyco-ops commented

2026-08-18 20:03:18 +00:00

Member

#958 landed in #975 (main 0edab8a, image published). The lane is in the shipped image and nothing runs on it, because SIRENS_ECHO_COALESCE_ENABLED defaults false. Flipping it is the half that was deliberately left as a decision.

The change

One environment variable in coilyco-bridge/deploy/services/sirens-echo. Nothing in this repository moves. SIRENS_ECHO_COALESCE_* in the knob reference tunes the window, batch, wide pair, age cap, pool, and queue bound if the packaged numbers turn out wrong; the high-water mark is derived from the pool and the narrow batch and is not set on its own.

Rollback is the same variable back to false. No image rebuild, no revert.

Against the freeze

#929 as amended releases work that changes only how well the agent does what it already does. This flip is the part that is arguably not that. A member who posts three comments in twenty seconds currently gets three replies and would get one, which is a behaviour change a viewer could notice as different. Read strictly that is a feature and it is behind August 20. Read as throughput work on a lane that answers exactly the same questions, it is an operational improvement. The call is Kai's, and this issue exists so it is a call rather than a thing nobody got to.

The amendment's own discipline applies either way, and the middle item is the one that bites: one meaningful variable at a time. Do not flip this in the same rollout as anything else.

Evidence to record

Before and after, both written down:

p50 and p99 sirens_echo.turn.duration, and the admission outcome split on sirens_echo.admissions (the denied_queue share is the bottleneck this removes).
After only: sirens_echo.coalesce.queue.depth, sirens_echo.coalesce.batch.size, sirens_echo.coalesce.turn.duration, sirens_echo.coalesce.asks split by accepted and shed.

A batch size distribution sitting flat at 1 means the window is closing before a second comment arrives and the flip bought nothing, which is the result worth catching early.

Known gaps

Never run against a live gateway. The lane is covered by unit tests and a race-clean suite, and no part of it has met a real Discord session. The first flip is the first run.
SIRENS_ECHO_MAX_PENDING stops being the backlog bound while the lane is on, because there is no execution slot to wait for. SIRENS_ECHO_COALESCE_CAPACITY (200) sheds where the pending cap (8) refused. The per-user, per-context, and global tiers are unchanged, so spend is still bounded, but the shed threshold moves a long way out and a shed comment is marked failed rather than answered.
A member reads one reply covering several comments. The earlier comments keep their arrival mark until the turn clears it. Worth watching whether that reads as attentive or as dropped.

Details in admission.

#958 landed in #975 (main `0edab8a`, image published). The lane is in the shipped image and **nothing runs on it**, because `SIRENS_ECHO_COALESCE_ENABLED` defaults false. Flipping it is the half that was deliberately left as a decision. ## The change One environment variable in `coilyco-bridge/deploy/services/sirens-echo`. Nothing in this repository moves. `SIRENS_ECHO_COALESCE_*` in [the knob reference](../src/branch/main/agent/rendered/knobs.txt) tunes the window, batch, wide pair, age cap, pool, and queue bound if the packaged numbers turn out wrong; the high-water mark is derived from the pool and the narrow batch and is not set on its own. Rollback is the same variable back to false. No image rebuild, no revert. ## Against the freeze #929 as amended releases work that changes only how well the agent does what it already does. **This flip is the part that is arguably not that.** A member who posts three comments in twenty seconds currently gets three replies and would get one, which is a behaviour change a viewer could notice as different. Read strictly that is a feature and it is behind August 20. Read as throughput work on a lane that answers exactly the same questions, it is an operational improvement. **The call is Kai's**, and this issue exists so it is a call rather than a thing nobody got to. The amendment's own discipline applies either way, and the middle item is the one that bites: one meaningful variable at a time. Do not flip this in the same rollout as anything else. ## Evidence to record Before and after, both written down: * p50 and p99 `sirens_echo.turn.duration`, and the admission outcome split on `sirens_echo.admissions` (the `denied_queue` share is the bottleneck this removes). * After only: `sirens_echo.coalesce.queue.depth`, `sirens_echo.coalesce.batch.size`, `sirens_echo.coalesce.turn.duration`, `sirens_echo.coalesce.asks` split by `accepted` and `shed`. A batch size distribution sitting flat at 1 means the window is closing before a second comment arrives and the flip bought nothing, which is the result worth catching early. ## Known gaps * **Never run against a live gateway.** The lane is covered by unit tests and a race-clean suite, and no part of it has met a real Discord session. The first flip is the first run. * **`SIRENS_ECHO_MAX_PENDING` stops being the backlog bound** while the lane is on, because there is no execution slot to wait for. `SIRENS_ECHO_COALESCE_CAPACITY` (200) sheds where the pending cap (8) refused. The per-user, per-context, and global tiers are unchanged, so spend is still bounded, but the shed threshold moves a long way out and a shed comment is marked failed rather than answered. * **A member reads one reply covering several comments.** The earlier comments keep their arrival mark until the turn clears it. Worth watching whether that reads as attentive or as dropped. Details in [admission](../src/branch/main/docs/sirens-echo-admission.md).

coilyco-ops commented

2026-08-18 20:30:53 +00:00

Author

Member

Decided by Kai, 2026-08-18: on, for every lane. Alongside SIRENS_ECHO_DISCORD_COMMANDS, also on for every lane. Direct messages stay exactly where they already are, which after reading the manifests turns out to be no edit at all: echo and deep are already "true" and dowel, the lane being recorded, is already "false".

The rollout is filed at coilyco-bridge/deploy#680 with the two-line change per values file, the applications.commands scope precondition that fails silently, and the metrics to read. No agent seat can land in that repository (deploy#482), so it needs the DevOps seat or Kai.

The freeze question this issue was opened to hold open is answered. Recording it as decided rather than closing, since the issue's acceptance is the before-and-after evidence and none of that exists yet.

**Decided by Kai, 2026-08-18: on, for every lane.** Alongside `SIRENS_ECHO_DISCORD_COMMANDS`, also on for every lane. Direct messages stay exactly where they already are, which after reading the manifests turns out to be no edit at all: echo and deep are already `"true"` and dowel, the lane being recorded, is already `"false"`. The rollout is filed at coilyco-bridge/deploy#680 with the two-line change per values file, the `applications.commands` scope precondition that fails silently, and the metrics to read. No agent seat can land in that repository (deploy#482), so it needs the DevOps seat or Kai. The freeze question this issue was opened to hold open is answered. Recording it as decided rather than closing, since the issue's acceptance is the before-and-after evidence and none of that exists yet.

coilyco-ops referenced this issue

2026-08-18 22:31:05 +00:00

Replies to threads you created should always summon you #750

coilyco-ops added the

labels

2026-08-19 03:38:45 +00:00

coilyco-ops added this to the August 19 Temporal Vibe Check milestone

2026-08-19 04:12:51 +00:00

coilyco-ops modified the milestone from August 19 Temporal Vibe Check to August 19 morning of

2026-08-19 05:35:08 +00:00

coilyco-ops referenced this issue from a commit

2026-08-19 09:50:02 +00:00

feat(sirens-echo): enable coalescing on Echo, set COALESCE_WORKERS=10 on both community lanes

coilyco-ops commented

2026-08-19 15:14:24 +00:00

Author

Member

QA verdict: acceptance not met. The flip already happened, and the evidence this issue asks for is partly negative and partly unproducible. Read-only observation from SigNoz plus the deploy manifests, no live action taken.

The premise is stale: the lane is already on

SIRENS_ECHO_COALESCE_ENABLED is "true" on all three lanes in coilyco-bridge/deploy/services/sirens-echo, and all three are emitting coalesce turns. Echo flipped in deploy b8d12c8 (2026-08-19T09:49:47Z, "enable coalescing on Echo, set COALESCE_WORKERS=10 on both community lanes"), rolled out in the 10:18Z roll-out commits, first coalesce turns land in the 11:00Z hour. It cleared the pre-demo lockdown boundary. First observed coalesce turns per lane: dowel 2026-08-18T22:00Z, deep 2026-08-19T03:00Z, echo 2026-08-19T11:00Z.

SIRENS_ECHO_COALESCE_WORKERS=10 rode along in the same commit, against the amendment's one-meaningful-variable-at-a-time discipline this issue explicitly invoked.

The named failure signal is present: batch size is flat at 1

This is the result the issue said was worth catching early, and it is here. Last 24h, sirens_echo.coalesce.batch.size:

echo - 3 batches, size sum 3, max 1 - every batch held one comment
deep - 8 batches, size sum 8, max 1 - every batch held one comment
dowel - 34 batches, size sum 35, max 2 - one batch held two

45 batches, and exactly one of them coalesced anything. Mean batch size 1.02. By the issue's own criterion the window is closing before a second comment arrives and the flip has bought nothing so far.

`denied_queue` persists after the flip

Dowel, restricted to after its coalescing start: 115 accepted and 9 denied_queue, a 7.3% denial share with the lane on. Every one of those 9 falls after the flip. That cuts against "the denied_queue share is the bottleneck this removes."

For scale, echo over 7 days: 283 accepted, 38 denied_queue (11.7%), 5 denied_user. So the bottleneck is real historically, it just is not what the coalescing lane has removed.

The required before-and-after metric does not report

sirens_echo.turn.duration p50 and p99 is the primary acceptance evidence named here, and it returns empty over 7 days. So does sirens_echo.turns.

The discriminating test: the gauge sibling sirens_echo.turn.duration.max reports fine (echo peak 303,085 ms, dowel 341,617 ms, deep 37,767 ms), while the .bucket, .sum, and .count siblings return nothing. Those three are the cumulative-temporality members of the family. Every delta metric queries cleanly, including all of sirens_echo.coalesce.* and sirens_echo.admissions. The catalog lists the cumulative ones as active, so they are registered and not reporting rather than absent.

Consequence: the before half of this issue's acceptance cannot be produced as specified until that is fixed, and no p50/p99 latency comparison is available from either side of the flip.

What is quiet

sirens_echo.coalesce.asks - all accepted, zero shed on every lane (echo 3, deep 9, dowel 35). The CAPACITY 200 versus pending-cap 8 concern in Known gaps is untested.
sirens_echo.coalesce.queue.depth - peak 1 on every lane, against a bound of 200.
sirens_echo.coalesce.turn.duration - echo p50 and p99 both land in the same 10,000 ms bucket at n=3, so that is bucket resolution rather than a measurement. Dowel p50 9,225 ms / p99 9,694 ms, deep p50 9,000 ms / p99 9,495 ms.

Sample size

Echo's after window is 3 coalesce turns and 4 admissions. That is too small for a latency or behaviour verdict on the lane this issue is about. Dowel carries the only meaningful traffic.

Named actions, none of them mine

QA defers live mutation and does not remediate, so these go to the DevOps seat or Kai:

Decide whether the cumulative-metric gap is an exporter or a collector problem, and fix it. Until then this issue's acceptance is unmeasurable.
Decide whether flat-at-1 batches plus surviving denied_queue means the packaged window is too short for real traffic, or means the lane is fine and the traffic is simply thin.
If the window gets retuned, land it alone.

Evidence window: 2026-08-12 through 2026-08-19T15:09Z, SigNoz, read-only.

**QA verdict: acceptance not met.** The flip already happened, and the evidence this issue asks for is partly negative and partly unproducible. Read-only observation from SigNoz plus the deploy manifests, no live action taken. ## The premise is stale: the lane is already on `SIRENS_ECHO_COALESCE_ENABLED` is `"true"` on all three lanes in `coilyco-bridge/deploy/services/sirens-echo`, and all three are emitting coalesce turns. Echo flipped in deploy `b8d12c8` (2026-08-19T09:49:47Z, "enable coalescing on Echo, set COALESCE_WORKERS=10 on both community lanes"), rolled out in the 10:18Z roll-out commits, first coalesce turns land in the 11:00Z hour. It cleared the pre-demo lockdown boundary. First observed coalesce turns per lane: dowel 2026-08-18T22:00Z, deep 2026-08-19T03:00Z, echo 2026-08-19T11:00Z. `SIRENS_ECHO_COALESCE_WORKERS=10` rode along in the same commit, against the amendment's one-meaningful-variable-at-a-time discipline this issue explicitly invoked. ## The named failure signal is present: batch size is flat at 1 This is the result the issue said was worth catching early, and it is here. Last 24h, `sirens_echo.coalesce.batch.size`: * echo - 3 batches, size sum 3, max 1 - every batch held one comment * deep - 8 batches, size sum 8, max 1 - every batch held one comment * dowel - 34 batches, size sum 35, max 2 - one batch held two 45 batches, and exactly one of them coalesced anything. Mean batch size 1.02. By the issue's own criterion the window is closing before a second comment arrives and the flip has bought nothing so far. ## `denied_queue` persists after the flip Dowel, restricted to after its coalescing start: 115 accepted and 9 `denied_queue`, a 7.3% denial share with the lane on. Every one of those 9 falls after the flip. That cuts against "the `denied_queue` share is the bottleneck this removes." For scale, echo over 7 days: 283 accepted, 38 `denied_queue` (11.7%), 5 `denied_user`. So the bottleneck is real historically, it just is not what the coalescing lane has removed. ## The required before-and-after metric does not report `sirens_echo.turn.duration` p50 and p99 is the primary acceptance evidence named here, and it returns empty over 7 days. So does `sirens_echo.turns`. The discriminating test: the gauge sibling `sirens_echo.turn.duration.max` reports fine (echo peak 303,085 ms, dowel 341,617 ms, deep 37,767 ms), while the `.bucket`, `.sum`, and `.count` siblings return nothing. Those three are the **cumulative**-temporality members of the family. Every **delta** metric queries cleanly, including all of `sirens_echo.coalesce.*` and `sirens_echo.admissions`. The catalog lists the cumulative ones as active, so they are registered and not reporting rather than absent. Consequence: the before half of this issue's acceptance cannot be produced as specified until that is fixed, and no p50/p99 latency comparison is available from either side of the flip. ## What is quiet * `sirens_echo.coalesce.asks` - all accepted, zero shed on every lane (echo 3, deep 9, dowel 35). The `CAPACITY` 200 versus pending-cap 8 concern in Known gaps is untested. * `sirens_echo.coalesce.queue.depth` - peak 1 on every lane, against a bound of 200. * `sirens_echo.coalesce.turn.duration` - echo p50 and p99 both land in the same 10,000 ms bucket at n=3, so that is bucket resolution rather than a measurement. Dowel p50 9,225 ms / p99 9,694 ms, deep p50 9,000 ms / p99 9,495 ms. ## Sample size Echo's after window is 3 coalesce turns and 4 admissions. That is too small for a latency or behaviour verdict on the lane this issue is about. Dowel carries the only meaningful traffic. ## Named actions, none of them mine QA defers live mutation and does not remediate, so these go to the DevOps seat or Kai: 1. Decide whether the cumulative-metric gap is an exporter or a collector problem, and fix it. Until then this issue's acceptance is unmeasurable. 2. Decide whether flat-at-1 batches plus surviving `denied_queue` means the packaged window is too short for real traffic, or means the lane is fine and the traffic is simply thin. 3. If the window gets retuned, land it alone. Evidence window: 2026-08-12 through 2026-08-19T15:09Z, SigNoz, read-only.

coilyco-ops commented

2026-08-19 15:36:41 +00:00

Author

Member

Live gateway test run on dowel. Coalescing works. This closes the "never run against a live gateway" gap in Known gaps and corrects the reading of the flat-at-1 finding in my previous comment.

Kai sent messages roughly 5 seconds apart in owl.glass #moxn-temporal, the single channel sirens-dowel-access-policy.yml scopes the lane to. QA observed only, no live action taken. Window 2026-08-19T15:24:35Z to 15:35:47Z, lane sirens-dowel.

Result

sirens_echo.admissions - 3 accepted, all transport=discord, zero denied
sirens_echo.coalesce.asks - 3 accepted, zero shed
sirens_echo.coalesce.batch.size - 2 batches, 3 items total, max 2
sirens_echo.coalesce.turns - 2
sirens_echo.coalesce.turn.duration - both turns in the 10,000 ms bucket
sirens_echo.tool.calls - 15

That reconciles as one batch of 2 and one batch of 1, from 2 turns. Two messages arriving within the 25 second window merged into a single turn, which is the mechanism this issue exists to verify. It has now done it against a real Discord gateway.

What this changes

The flat-at-1 batch distribution in my previous comment is a traffic-volume artifact, not a defect. COALESCE_WINDOW is 25s with COALESCE_BATCH=4, and dowel runs at roughly five admissions an hour, so two messages landing inside the same window essentially never happens by chance. The lane was correctly doing nothing. The earlier reading, that the window was closing before a second comment arrived, was wrong: the window is generous and it batches fine when traffic actually arrives together.

So the honest summary of the flip is that it is correct and idle rather than broken. It will engage when a member posts in quick succession, which is the case it was built for.

What this does not change

denied_queue still persists post-flip on dowel, 9 against 115 accepted. Coalescing is not what removes it. That finding stands.
The metric reporting gaps stand. sirens_echo.turn.duration and sirens_echo.turns, the cumulative-temporality family, still report nothing, so the before half of this issue's acceptance is still unproducible.

One caution for whoever reads these numbers next

coalesce.turns and coalesce.turn.duration lagged batch.size by several minutes in this test, because they record on turn completion and dowel turns run long. A snapshot taken too early shows batches with zero turns and reads as a stuck worker pool. It is not. Re-measure before concluding anything from that shape.

Open, and not resolvable from QA's side

Three admissions were observed where two messages were expected. Either a third message was sent or a real owl.glass member posted inside the window. My Discord identity is the Echo bot, which has no access to owl.glass, so I cannot read the channel to tell which, nor confirm whether the single reply visibly covered both questions. That last one is the "attentive or dropped" gap and it remains unverified.

**Live gateway test run on dowel. Coalescing works.** This closes the "never run against a live gateway" gap in Known gaps and corrects the reading of the flat-at-1 finding in my previous comment. Kai sent messages roughly 5 seconds apart in owl.glass `#moxn-temporal`, the single channel `sirens-dowel-access-policy.yml` scopes the lane to. QA observed only, no live action taken. Window 2026-08-19T15:24:35Z to 15:35:47Z, lane `sirens-dowel`. ## Result * `sirens_echo.admissions` - 3 accepted, all `transport=discord`, zero denied * `sirens_echo.coalesce.asks` - 3 accepted, zero shed * `sirens_echo.coalesce.batch.size` - 2 batches, 3 items total, max 2 * `sirens_echo.coalesce.turns` - 2 * `sirens_echo.coalesce.turn.duration` - both turns in the 10,000 ms bucket * `sirens_echo.tool.calls` - 15 That reconciles as **one batch of 2 and one batch of 1, from 2 turns**. Two messages arriving within the 25 second window merged into a single turn, which is the mechanism this issue exists to verify. It has now done it against a real Discord gateway. ## What this changes The flat-at-1 batch distribution in my previous comment is **a traffic-volume artifact, not a defect**. `COALESCE_WINDOW` is 25s with `COALESCE_BATCH=4`, and dowel runs at roughly five admissions an hour, so two messages landing inside the same window essentially never happens by chance. The lane was correctly doing nothing. The earlier reading, that the window was closing before a second comment arrived, was wrong: the window is generous and it batches fine when traffic actually arrives together. So the honest summary of the flip is that it is correct and idle rather than broken. It will engage when a member posts in quick succession, which is the case it was built for. ## What this does not change * **`denied_queue` still persists post-flip on dowel**, 9 against 115 accepted. Coalescing is not what removes it. That finding stands. * **The metric reporting gaps stand.** `sirens_echo.turn.duration` and `sirens_echo.turns`, the cumulative-temporality family, still report nothing, so the before half of this issue's acceptance is still unproducible. ## One caution for whoever reads these numbers next `coalesce.turns` and `coalesce.turn.duration` lagged `batch.size` by several minutes in this test, because they record on turn completion and dowel turns run long. A snapshot taken too early shows batches with zero turns and reads as a stuck worker pool. It is not. Re-measure before concluding anything from that shape. ## Open, and not resolvable from QA's side Three admissions were observed where two messages were expected. Either a third message was sent or a real owl.glass member posted inside the window. My Discord identity is the Echo bot, which has no access to owl.glass, so I cannot read the channel to tell which, nor confirm whether the single reply visibly covered both questions. That last one is the "attentive or dropped" gap and it remains unverified.

coilyco-ops referenced this issue from a commit

2026-08-19 15:48:58 +00:00

fix(telemetry): give the turn histograms a range a turn fits in

coilyco-ops referenced this issue

2026-08-19 15:49:21 +00:00

fix(telemetry): give the turn histograms a range a turn fits in #1080

coilyco-ops commented

2026-08-19 15:54:34 +00:00

Author

Member

Engineer follow-up: the metrics were never broken, the histogram range was. This clears the "metric reporting gaps" blocker and corrects its diagnosis. Read-only observation plus one repository change, no live action taken.

The cumulative-temporality theory is wrong

The QA verdict reports sirens_echo.turn.duration and sirens_echo.turns returning nothing over 7 days, blames the cumulative-temporality family, and concludes acceptance is unmeasurable pending an exporter or collector fix. All three parts are wrong.

newMetricExporter has set DeltaTemporalitySelector since the initial public release, and every counter is created on the same meter, so sirens_echo.turns and sirens_echo.tool.calls cannot differ on the wire. The SigNoz catalog metadata is stale for five names. Same metric, same 24h window:

queried as delta - dowel 121, deep 10, echo 4
queried as cumulative, the catalog's declared temporality - empty

The data has been landing the whole time. Nothing in the pipeline needs fixing and no deploy or collector change is owed. Anyone reading these should pass temporality: delta explicitly until the catalog rows are corrected.

With the metric readable, the evidence is useless for a different reason

sirens_echo.turn.duration, 2026-08-12 to 08-18, queried as delta:

p50 - dowel 10000, deep-owl-glass 9843, echo 9802
p99 - echo 10000, deep-owl-glass 10000, dowel 10000

The OpenTelemetry SDK's default explicit boundaries top out at 10,000 ms. Turn p50 is roughly 17s and p95 roughly 181s, measured from traces in #1076, so every turn lands in the overflow bucket and any percentile reports the 10,000 boundary rather than a duration, identically on both sides of any comparison.

That is why this survived so long. A histogram whose range is wrong fails as a plausible number, not as an empty one.

It also explains the flat-at-1 batch finding

coalesce.batch.size used the same defaults, whose first bucket is [0, 5), so a batch of 1 was indistinguishable from a batch of 4. The .max gauge is the only reason the single batch of 2 was visible at all. The live-gateway comment reached the right conclusion, and the instrument could not have shown it either way.

The change

PR #1080 gives sirens_echo.turn.duration and sirens_echo.coalesce.turn.duration boundaries running to 300,000 ms, the request ceiling, so a turn that ran the clock out is distinguishable from one that nearly did, and gives coalesce.batch.size small-integer boundaries so batch distribution is directly readable. Every other histogram keeps the defaults, its range being one they fit.

What this does not fix

Pre-rollout turn latency is not recoverable. Boundaries change what a histogram can express, not what was recorded, so the before half of this issue's acceptance cannot be produced from the histogram retrospectively. community.turn span durations are the fallback for the before window, which is what #1076 used. After #1080 rolls, the histogram measures turns properly and the after half is real.

The denied_queue finding stands untouched: coalescing is not what removes it.

**Engineer follow-up: the metrics were never broken, the histogram range was.** This clears the "metric reporting gaps" blocker and corrects its diagnosis. Read-only observation plus one repository change, no live action taken. ## The cumulative-temporality theory is wrong The QA verdict reports `sirens_echo.turn.duration` and `sirens_echo.turns` returning nothing over 7 days, blames the cumulative-temporality family, and concludes acceptance is unmeasurable pending an exporter or collector fix. All three parts are wrong. `newMetricExporter` has set `DeltaTemporalitySelector` since the initial public release, and every counter is created on the same meter, so `sirens_echo.turns` and `sirens_echo.tool.calls` cannot differ on the wire. The SigNoz catalog metadata is stale for five names. Same metric, same 24h window: * queried as **delta** - dowel 121, deep 10, echo 4 * queried as **cumulative**, the catalog's declared temporality - empty The data has been landing the whole time. Nothing in the pipeline needs fixing and no deploy or collector change is owed. Anyone reading these should pass `temporality: delta` explicitly until the catalog rows are corrected. ## With the metric readable, the evidence is useless for a different reason `sirens_echo.turn.duration`, 2026-08-12 to 08-18, queried as delta: * p50 - dowel **10000**, deep-owl-glass 9843, echo 9802 * p99 - echo **10000**, deep-owl-glass **10000**, dowel **10000** The OpenTelemetry SDK's default explicit boundaries top out at 10,000 ms. Turn p50 is roughly 17s and p95 roughly 181s, measured from traces in #1076, so **every turn lands in the overflow bucket** and any percentile reports the 10,000 boundary rather than a duration, identically on both sides of any comparison. That is why this survived so long. A histogram whose range is wrong fails as a plausible number, not as an empty one. ## It also explains the flat-at-1 batch finding `coalesce.batch.size` used the same defaults, whose first bucket is `[0, 5)`, so a batch of 1 was indistinguishable from a batch of 4. The `.max` gauge is the only reason the single batch of 2 was visible at all. The live-gateway comment reached the right conclusion, and the instrument could not have shown it either way. ## The change PR #1080 gives `sirens_echo.turn.duration` and `sirens_echo.coalesce.turn.duration` boundaries running to 300,000 ms, the request ceiling, so a turn that ran the clock out is distinguishable from one that nearly did, and gives `coalesce.batch.size` small-integer boundaries so batch distribution is directly readable. Every other histogram keeps the defaults, its range being one they fit. ## What this does not fix **Pre-rollout turn latency is not recoverable.** Boundaries change what a histogram can express, not what was recorded, so the before half of this issue's acceptance cannot be produced from the histogram retrospectively. `community.turn` span durations are the fallback for the before window, which is what #1076 used. After #1080 rolls, the histogram measures turns properly and the after half is real. The `denied_queue` finding stands untouched: coalescing is not what removes it.

coilyco-ops referenced this issue

2026-08-19 16:09:51 +00:00

Measure the coalescing lane once #1080 rolls, and check how a merged reply reads #1082

coilyco-ops referenced this issue

2026-08-19 16:10:14 +00:00

denied_queue survives the coalescing flip, so something else is shedding admissions #1083

coilyco-ops commented

2026-08-19 16:10:34 +00:00

Author

Member

Closing. The decision this issue existed to hold open is made, executed, and verified.

What this issue achieved

The freeze call - decided 2026-08-18: on, for every lane. That was the whole reason the issue was opened as a decision rather than a task.
The flip - SIRENS_ECHO_COALESCE_ENABLED is "true" on all three lanes, cleared the pre-demo lockdown boundary.
Live gateway verification - two messages ~5s apart on dowel merged into one turn, 2026-08-19T15:24Z. The "never run against a live gateway" gap in Known gaps is closed.
A real instrument defect found and fixed - #1080. Default histogram boundaries top out at 10,000 ms against a turn p50 of ~17s, so every percentile returned the boundary rather than a duration, and coalesce.batch.size could not tell a batch of 1 from a batch of 4.

Two readings in this thread were corrected along the way and should not be carried forward: the flat-at-1 batch distribution was a traffic artifact and an instrument artifact, not a defect in the lane, and the empty turn.duration queries were stale catalog temporality metadata, not a broken exporter or collector.

Why it closes rather than stays open

Its stated acceptance can never be met as written. This issue asks for before-and-after p50/p99 sirens_echo.turn.duration. Histogram boundaries change what an instrument can express, never what it already recorded, so the before half is not recoverable from that histogram at any future point. An issue whose acceptance is impossible should not stay open waiting for it.

Successors

#1082 - measure the lane properly once #1080 rolls (main is c8139e6, the lane runs 893c469e), plus the "attentive or dropped" check that needs Kai in owl.glass since the agent's Discord identity has no access there.
#1083 - denied_queue survives the flip: 9 of 115 on dowel post-flip, 11.7% on echo over 7 days. This issue predicted coalescing would remove it. It did not, and that failed prediction is its own investigation.

Both carry the delta-temporality warning, because querying these metrics per the catalog returns empty and reads as a broken pipeline.

**Closing. The decision this issue existed to hold open is made, executed, and verified.** ## What this issue achieved * **The freeze call** - decided 2026-08-18: on, for every lane. That was the whole reason the issue was opened as a decision rather than a task. * **The flip** - `SIRENS_ECHO_COALESCE_ENABLED` is `"true"` on all three lanes, cleared the pre-demo lockdown boundary. * **Live gateway verification** - two messages ~5s apart on dowel merged into one turn, 2026-08-19T15:24Z. The "never run against a live gateway" gap in Known gaps is closed. * **A real instrument defect found and fixed** - #1080. Default histogram boundaries top out at 10,000 ms against a turn p50 of ~17s, so every percentile returned the boundary rather than a duration, and `coalesce.batch.size` could not tell a batch of 1 from a batch of 4. Two readings in this thread were corrected along the way and should not be carried forward: the flat-at-1 batch distribution was a traffic artifact **and** an instrument artifact, not a defect in the lane, and the empty `turn.duration` queries were stale catalog temporality metadata, not a broken exporter or collector. ## Why it closes rather than stays open **Its stated acceptance can never be met as written.** This issue asks for before-and-after p50/p99 `sirens_echo.turn.duration`. Histogram boundaries change what an instrument can express, never what it already recorded, so the before half is not recoverable from that histogram at any future point. An issue whose acceptance is impossible should not stay open waiting for it. ## Successors * **#1082** - measure the lane properly once #1080 rolls (`main` is `c8139e6`, the lane runs `893c469e`), plus the "attentive or dropped" check that needs Kai in owl.glass since the agent's Discord identity has no access there. * **#1083** - `denied_queue` survives the flip: 9 of 115 on dowel post-flip, 11.7% on echo over 7 days. This issue predicted coalescing would remove it. It did not, and that failed prediction is its own investigation. Both carry the delta-temporality warning, because querying these metrics per the catalog returns empty and reads as a broken pipeline.

coilyco-ops closed this issue

2026-08-19 16:10:36 +00:00

coilyco-ops referenced this issue

2026-08-19 16:14:57 +00:00

Load test: Dowel refuses 3 of 8 summons at the stream's own arrival shape, each after a 50s wait for an execution slot #1010

coilyco-ops referenced this issue

2026-08-19 16:15:23 +00:00

Measure where the 32.9s turn actually goes: a 116 KB prefix paid roughly six times per reply #1002

coilyco-ops referenced this issue

2026-08-22 21:29:21 +00:00

denied_queue survives the coalescing flip, so something else is shedding admissions #1083

coilyco-ops referenced this issue from a commit