fix(telemetry): give the turn histograms a range a turn fits in

coilyco-ops commented

2026-08-19 15:49:21 +00:00

Member

Refs #976. Answers the "metric reporting gaps" blocker in the QA verdict, and
corrects its diagnosis.

The metrics were never broken

The QA comment reports sirens_echo.turn.duration and sirens_echo.turns
returning nothing over 7 days, blames the cumulative-temporality family, and
concludes the issue's acceptance is unmeasurable pending an exporter or
collector fix. All three are wrong.

newMetricExporter has set DeltaTemporalitySelector since the initial public
release, and every counter is created on the same meter, so sirens_echo.turns
and sirens_echo.tool.calls cannot differ on the wire. The SigNoz catalog
metadata is simply stale for five names. Same metric, same 24h window:

queried as delta - dowel 121, deep 10, echo 4
queried as cumulative, the catalog's declared temporality - empty

The data was landing the whole time. Nothing in the pipeline needs fixing, and
no deploy or collector change is owed. Anyone reading these should pass
temporality: delta explicitly until the catalog rows are corrected.

What is actually broken

With the metric readable, the acceptance evidence comes back useless:

p50 turn duration, 2026-08-12 to 08-18 - dowel 10000, deep-owl-glass 9843, echo 9802
p99, same window - echo 10000, deep-owl-glass 10000, dowel 10000

The OpenTelemetry SDK's default explicit boundaries top out at 10,000 ms. Turn
p50 is ~17s and p95 ~181s (measured in #1076), so every turn lands in the
overflow bucket and any percentile reports the 10,000 boundary rather than a
duration, identically on both sides of any comparison.

This is why it survived: a histogram whose range is wrong fails as a plausible
number, not an empty one.

The same defect hit the other finding in that comment. coalesce.batch.size
used the same defaults, whose first bucket is [0, 5), so a batch of 1 was
indistinguishable from a batch of 4. The .max gauge is the only reason the
one batch of 2 was visible at all.

The change

sirens_echo.turn.duration and sirens_echo.coalesce.turn.duration take
boundaries running to 300,000 ms, the request ceiling, so a turn that ran the
clock out is distinguishable from one that nearly did.
sirens_echo.coalesce.batch.size takes small-integer boundaries so batch
distribution is directly readable rather than inferred from a max gauge.

Every other histogram keeps the defaults, its recorded range being one they
fit. Boundaries are a view on the meter provider rather than a knob: a
deployment that changed them would make its own history unreadable against
everyone else's.

Caveat on the evidence

Boundaries change what the histogram can express, not what was recorded.
Pre-rollout turn latency is not recoverable and the before half of #976's
acceptance cannot be produced from the histogram retrospectively. Trace
durations on community.turn are the fallback for the before window, which is
what #1076 used. After this rolls, the histogram measures turns properly.

The tests fail against the defaults, reporting the boundary set
[0 5 10 25 50 75 100 250 500 750 1000 2500 5000 7500 10000], so they are a
regression guard rather than a restatement.

go build, go vet, go test ./..., and pre-commit run --all-files are green.

Refs #976. Answers the "metric reporting gaps" blocker in the QA verdict, and corrects its diagnosis. ## The metrics were never broken The QA comment reports `sirens_echo.turn.duration` and `sirens_echo.turns` returning nothing over 7 days, blames the cumulative-temporality family, and concludes the issue's acceptance is unmeasurable pending an exporter or collector fix. All three are wrong. `newMetricExporter` has set `DeltaTemporalitySelector` since the initial public release, and every counter is created on the same meter, so `sirens_echo.turns` and `sirens_echo.tool.calls` cannot differ on the wire. The SigNoz catalog metadata is simply stale for five names. Same metric, same 24h window: * queried as **delta** - dowel 121, deep 10, echo 4 * queried as **cumulative**, the catalog's declared temporality - empty The data was landing the whole time. Nothing in the pipeline needs fixing, and no deploy or collector change is owed. Anyone reading these should pass `temporality: delta` explicitly until the catalog rows are corrected. ## What is actually broken With the metric readable, the acceptance evidence comes back useless: * p50 turn duration, 2026-08-12 to 08-18 - dowel **10000**, deep-owl-glass 9843, echo 9802 * p99, same window - echo **10000**, deep-owl-glass **10000**, dowel **10000** The OpenTelemetry SDK's default explicit boundaries top out at 10,000 ms. Turn p50 is ~17s and p95 ~181s (measured in #1076), so **every turn lands in the overflow bucket** and any percentile reports the 10,000 boundary rather than a duration, identically on both sides of any comparison. This is why it survived: a histogram whose range is wrong fails as a plausible number, not an empty one. The same defect hit the other finding in that comment. `coalesce.batch.size` used the same defaults, whose first bucket is `[0, 5)`, so a batch of 1 was indistinguishable from a batch of 4. The `.max` gauge is the only reason the one batch of 2 was visible at all. ## The change `sirens_echo.turn.duration` and `sirens_echo.coalesce.turn.duration` take boundaries running to 300,000 ms, the request ceiling, so a turn that ran the clock out is distinguishable from one that nearly did. `sirens_echo.coalesce.batch.size` takes small-integer boundaries so batch distribution is directly readable rather than inferred from a max gauge. Every other histogram keeps the defaults, its recorded range being one they fit. Boundaries are a view on the meter provider rather than a knob: a deployment that changed them would make its own history unreadable against everyone else's. ## Caveat on the evidence Boundaries change what the histogram can express, not what was recorded. **Pre-rollout turn latency is not recoverable** and the before half of #976's acceptance cannot be produced from the histogram retrospectively. Trace durations on `community.turn` are the fallback for the before window, which is what #1076 used. After this rolls, the histogram measures turns properly. The tests fail against the defaults, reporting the boundary set `[0 5 10 25 50 75 100 250 500 750 1000 2500 5000 7500 10000]`, so they are a regression guard rather than a restatement. `go build`, `go vet`, `go test ./...`, and `pre-commit run --all-files` are green.

coilyco-ops added 1 commit

2026-08-19 15:49:21 +00:00

fix(telemetry): give the turn histograms a range a turn fits in

ci / image-build (pull_request) Successful in 36s

Details

ci / test (pull_request) Successful in 1m1s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

40eba8c7e3

The OpenTelemetry SDK's default explicit boundaries top out at 10,000 ms and
a turn's p50 sits above that, so every turn landed in the overflow bucket and
a p50 or p99 over sirens_echo.turn.duration returned the 10,000 boundary
rather than a duration. Measured over 2026-08-12 to 08-18: p50 and p99 both
read 10000 on dowel, echo, and deep-owl-glass alike, on both sides of any
comparison.

That is why it survived: a histogram whose range is wrong fails as a
plausible number rather than as an empty one.

sirens_echo.turn.duration and sirens_echo.coalesce.turn.duration now take
boundaries running to 300,000 ms, the request ceiling, so a turn that ran the
clock out is distinguishable from one that nearly did.
sirens_echo.coalesce.batch.size takes small-integer boundaries, because a
batch is bounded by the wide batch size and the default first bucket held
every batch alike, so a lane batching nothing read like a lane batching four.

Every other histogram keeps the defaults, its recorded range being one they
fit. Boundaries are a view rather than a knob: a deployment that changed them
would make its own history unreadable against everyone else's.

Refs #976

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-19 15:49:21 +00:00

coilyco-ops added 1 commit

2026-08-19 15:50:38 +00:00