fix(eval): an evaluation run is not the production deployment #543

Merged

coilyco-ops merged 1 commit from fix/evaluation-is-not-the-production-service into main

2026-08-13 15:46:55 +00:00

coilyco-ops commented

2026-08-13 15:33:40 +00:00

Member

closes #533

The remaining design question is #542, so this can close #533 honestly.

Found by Quail, who measured it precisely and named two explanations they could not separate from telemetry. It is a third thing, which is exactly why telemetry could not separate them.

The mechanism

cmd/sirens-echo-eval/main.go built its telemetry config as a literal and never set InstanceName. It does not go through LoadConfig, so it never reads SIRENS_ECHO_INSTANCE, and telemetry.go resolved the zero string to the sirens-echo default.

One binary serves both profiles. So eval-echo, eval-deep, board-deep, and rate-deep all reported as the Echo deployment.

Why it matters more than a label

service.name = sirens-echo was not one service. It was the production deployment plus every evaluation run of both profiles.

sirens-echo   898 mcp.tools.list / 24h    169 community.turn / 24h
sirens-deep   315 mcp.tools.list / 24h    313 community.turn / 24h

Most of what the production service appeared to be doing was evaluation. Error rate, latency, token spend, and the cache-hit ratio I measured earlier today were all wrong in the same direction, and every one of them read as a plausible figure.

sirens-deep looked clean at 1.006 — and looked clean because its evaluation traffic was being billed to the other service. One contaminated service beside a clean one reassures rather than prompting a question, which is why this survived being looked at directly by more than one of us.

The change

Three lines and a constant. The run names itself sirens-echo-eval.

The test pins the invariant rather than the string: the name is neither deployment, is a valid lowercase service name, and is not empty — because an empty one falls back to the very default this replaces, so an emptying regression would be silent.

A doc records what the numbers were, since anyone comparing a dashboard across this change will see a step and deserve to know it is the fix rather than an outage.

ward exec gate green.

closes #533 The remaining design question is #542, so this can close #533 honestly. Found by Quail, who measured it precisely and named two explanations they could not separate from telemetry. **It is a third thing**, which is exactly why telemetry could not separate them. ## The mechanism `cmd/sirens-echo-eval/main.go` built its telemetry config as a literal and never set `InstanceName`. It does not go through `LoadConfig`, so it never reads `SIRENS_ECHO_INSTANCE`, and `telemetry.go` resolved the zero string to the `sirens-echo` default. One binary serves both profiles. So `eval-echo`, `eval-deep`, `board-deep`, and `rate-deep` **all reported as the Echo deployment**. ## Why it matters more than a label `service.name = sirens-echo` was not one service. It was the production deployment plus every evaluation run of both profiles. ``` sirens-echo 898 mcp.tools.list / 24h 169 community.turn / 24h sirens-deep 315 mcp.tools.list / 24h 313 community.turn / 24h ``` **Most of what the production service appeared to be doing was evaluation.** Error rate, latency, token spend, and the cache-hit ratio I measured earlier today were all wrong in the same direction, and every one of them read as a plausible figure. `sirens-deep` looked clean at 1.006 — and looked clean *because its evaluation traffic was being billed to the other service*. One contaminated service beside a clean one reassures rather than prompting a question, which is why this survived being looked at directly by more than one of us. ## The change Three lines and a constant. The run names itself `sirens-echo-eval`. The test pins the invariant rather than the string: the name is neither deployment, is a valid lowercase service name, and is not empty — because an empty one falls back to the very default this replaces, so an emptying regression would be silent. A doc records what the numbers were, since anyone comparing a dashboard across this change will see a step and deserve to know it is the fix rather than an outage. `ward exec gate` green.

coilyco-ops added 1 commit

2026-08-13 15:33:40 +00:00

fix(eval): an evaluation run is not the production deployment

ci / image-build (pull_request) Successful in 19s

Details

ci / test (pull_request) Successful in 40s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

144c2c8bc5

The eval binary built its telemetry config as a literal and never set
InstanceName, so it fell back to the sirens-echo default. One binary serves
both profiles, which means eval-echo, eval-deep, board-deep, and rate-deep all
reported as the Echo deployment.

service.name sirens-echo was therefore not one service. It was production plus
every evaluation run of both profiles: 898 spans against 169 real turns over 24
hours, so most of what the service appeared to be doing was evaluation. Error
rate, latency, token spend, and cache-hit ratio were all wrong in the same
direction and all read as plausible figures.

sirens-deep looked clean at almost exactly one lookup per turn, and looked
clean because its evaluation traffic was being billed to the other service. One
contaminated service beside a clean one reassures rather than prompting a
question, which is why this survived being looked at directly.

The run now names itself. The test pins the invariant that matters, which is
that the name is neither deployment and is not empty, since an empty one falls
back to the very default this replaces.

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 15:33:40 +00:00

coilyco-ops merged commit afaa3aff49 into main

2026-08-13 15:46:55 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-13 15:46:56 +00:00

Merge pull request 'fix(eval): an evaluation run is not the production deployment' (#543) from fix/evaluation-is-not-the-production-service into main