Export logs over OTLP beside traces and metrics #839

Merged

coilysiren merged 1 commit from aos/claude/xh55-otlp-logs into main

2026-08-15 21:30:20 +00:00

coilyco-ops commented

2026-08-15 21:14:32 +00:00

Member

Closes #810. ward exec gate PASS on all six steps.

What landed

An otlploghttp exporter and a LoggerProvider sharing the resource the tracer already built, so service.name comes along for free and matches the trace side exactly.

Stdout stays

@Olaf's weak preference, taken. Every line is now stored twice, once scraped and once over OTLP, with different attribute sets on the two rows.

Dropping stdout would make kubectl logs useless for these pods, and that is the path that still works when SigNoz is itself the thing that is unreachable — which is precisely the incident where the logs matter most. A duplicated line is cheaper than a blind incident.

The acceptance asked for this to be written down because a future reader cannot infer it from the code. docs/sirens-echo-log-export.md carries it.

The fan-out

slog has no fan-out, so multiHandler writes one record to every handler. Three properties matter and each has a test:

A failing exporter does not stop the line reaching stdout. Its error is reported rather than swallowed. This is the case the whole decision rests on: a collector refusing writes is the incident, and it must not take the remaining path down with it.
A quiet destination does not silence a talkative one — raising stdout's threshold cannot mute the exported copy.
Deriving with WithAttrs or WithGroup does not reshape the handler it came from, so one logger's attributes cannot leak into another's.

Records are cloned per destination, because a handler may retain or modify what it is given.

Verification against the acceptance

"A log row carries service.name matching its trace-side value" — TestALoggedLineIsExportedWithItsServiceName runs a fake collector and asserts the body reaching /v1/logs carries both service.name and the logged line.
"The same field name works across traces and logs" — TestLogsAndTracesShareOneServiceName asserts the same value on both signal paths.
"Whatever is decided about stdout is written down" — the doc above.

Checked by mutation, with a mutation that compiles. Pointing the bridge at a provider with no processor fails both tests. Removing the handler outright only breaks the build, which reports zero --- FAIL lines — the trap #653 recorded, so it is worth saying which mutation was used.

Dependencies

otlploghttp, sdk/log, and the otelslog bridge. Resolving them moved otel core 1.44.0 → 1.45.0 and left the two existing exporters behind at 1.44.0, so those are pinned forward and the family is uniform again. All are official go.opentelemetry.io modules already in this tree's dependency family.

Two notes for the deploy side

The eval lane gets this too. sirens-echo-eval builds telemetry the same way, so its logs now export against evaluationOTLPEndpoint() alongside its traces. Its dataset-on-stdout behaviour is unchanged and still covered by TestTheEvaluationRunnerKeepsLogsOffStdout.
Nothing needs configuring. The log endpoint is derived from the existing OTEL_EXPORTER_OTLP_ENDPOINT the same way the trace and metric paths are, so there is no new variable and no deploy change required for this to take effect.

Closes #810. `ward exec gate` PASS on all six steps. ## What landed An `otlploghttp` exporter and a `LoggerProvider` sharing the resource the tracer already built, so `service.name` comes along for free and matches the trace side exactly. ## Stdout stays @Olaf's weak preference, taken. Every line is now stored twice, once scraped and once over OTLP, with different attribute sets on the two rows. Dropping stdout would make `kubectl logs` useless for these pods, and that is the path that still works when SigNoz is itself the thing that is unreachable — which is precisely the incident where the logs matter most. A duplicated line is cheaper than a blind incident. The acceptance asked for this to be written down because a future reader cannot infer it from the code. `docs/sirens-echo-log-export.md` carries it. ## The fan-out `slog` has no fan-out, so `multiHandler` writes one record to every handler. Three properties matter and each has a test: * **A failing exporter does not stop the line reaching stdout.** Its error is reported rather than swallowed. This is the case the whole decision rests on: a collector refusing writes is the incident, and it must not take the remaining path down with it. * **A quiet destination does not silence a talkative one** — raising stdout's threshold cannot mute the exported copy. * **Deriving with `WithAttrs` or `WithGroup` does not reshape the handler it came from**, so one logger's attributes cannot leak into another's. Records are cloned per destination, because a handler may retain or modify what it is given. ## Verification against the acceptance * **"A log row carries `service.name` matching its trace-side value"** — `TestALoggedLineIsExportedWithItsServiceName` runs a fake collector and asserts the body reaching `/v1/logs` carries both `service.name` and the logged line. * **"The same field name works across traces and logs"** — `TestLogsAndTracesShareOneServiceName` asserts the same value on both signal paths. * **"Whatever is decided about stdout is written down"** — the doc above. **Checked by mutation, with a mutation that compiles.** Pointing the bridge at a provider with no processor fails both tests. Removing the handler outright only breaks the build, which reports zero `--- FAIL` lines — the trap #653 recorded, so it is worth saying which mutation was used. ## Dependencies `otlploghttp`, `sdk/log`, and the `otelslog` bridge. Resolving them moved otel core 1.44.0 → 1.45.0 and left the two existing exporters behind at 1.44.0, so those are pinned forward and the family is uniform again. All are official `go.opentelemetry.io` modules already in this tree's dependency family. ## Two notes for the deploy side * **The eval lane gets this too.** `sirens-echo-eval` builds telemetry the same way, so its logs now export against `evaluationOTLPEndpoint()` alongside its traces. Its dataset-on-stdout behaviour is unchanged and still covered by `TestTheEvaluationRunnerKeepsLogsOffStdout`. * **Nothing needs configuring.** The log endpoint is derived from the existing `OTEL_EXPORTER_OTLP_ENDPOINT` the same way the trace and metric paths are, so there is no new variable and no deploy change required for this to take effect.

coilyco-ops added 1 commit

2026-08-15 21:14:33 +00:00

feat(telemetry): export logs over OTLP beside traces and metrics

ci / image-build (pull_request) Successful in 32s

Details

ci / test (pull_request) Successful in 44s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

40eb6a28bc

The harness built a trace exporter and a metric exporter and no log exporter,
so no log line had ever left either lane over OTLP. Lines still reached SigNoZ,
because the cluster agent scrapes pod stdout, but a scraped row cannot carry
service.name: the scraper has no way to know it. Those rows keyed on
k8s.deployment.name instead.

That mostly worked, which is why this was not urgent, and three things stayed
awkward. service.name is the key every SigNoZ doc and the MCP's own shortcut
reaches for, and on those rows it matched nothing and returned an empty result
rather than an error. Traces and logs keyed differently, so no dashboard or
alert could use one field across both. And both lanes share an image, so
k8s.container.name is sirens-echo in both namespaces.

An otlploghttp exporter and a LoggerProvider now share the resource the tracer
already built, so service.name comes along for free and matches the trace side
exactly.

STDOUT STAYS, AND WHY

Olaf's weak preference, taken: every line is now stored twice, once scraped and
once over OTLP, with different attribute sets on the two rows. Dropping stdout
would make kubectl logs useless for these pods, and that is the path that works
when SigNoz is itself unreachable, which is the incident where logs matter
most. A duplicated line is cheaper than a blind incident.

The acceptance asked for the decision to be written down because a future
reader cannot infer it from the code. docs/sirens-echo-log-export.md carries it.

THE FAN-OUT

slog has no fan-out, so multiHandler writes one record to every handler. Three
properties have tests: a failing exporter does not stop the line reaching
stdout and its error is reported rather than swallowed, a destination that
wants a level gets it when another does not, and deriving with WithAttrs or
WithGroup does not reshape the handler it came from. Records are cloned per
destination because a handler may retain what it is given.

DEPENDENCIES

otlploghttp, sdk/log, and the otelslog bridge. Resolving them moved the otel
core from 1.44.0 to 1.45.0 and left the two existing exporters behind at
1.44.0, so those are pinned forward and the family is uniform again. All are
official go.opentelemetry.io modules already in the tree's dependency family.

VERIFICATION

TestALoggedLineIsExportedWithItsServiceName runs a fake collector and asserts
the body reaching /v1/logs carries both service.name and the line, which is the
issue's first acceptance criterion. TestLogsAndTracesShareOneServiceName
asserts the same value on both signal paths, which is the second.

Checked by mutation, with a mutation that compiles: pointing the bridge at a
provider with no processor fails both tests. Removing the handler outright only
breaks the build, which reports zero failures and is the trap #653 recorded.

Closes #810

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Angie <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-15 21:14:33 +00:00

coilyco-ops referenced this pull request

2026-08-15 21:16:17 +00:00

The harness exports traces and metrics over OTLP but not logs, so its log rows carry no service.name #810