Add a host-local SigNoz role for full high-detail localhost inspection (loopback-only OTLP backend) #435

New issue

Closed

opened 2026-07-02 09:31:16 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-02 09:31:16 +00:00

Owner

Want

A host-local SigNoz rollout for full high-detail inspection of agent-proxy (and other local services) on localhost, in the same shape as the existing dockerized host roles (agent-proxy, dozzle, logdy, mac-proxy). This is the local OTLP trace/metrics/logs backend the proxy points at during high-detail localhost debugging, kept separate from the shared fleet SigNoz on ser8 (which is redacted/aggregate; the local one is unsampled and may hold full request/response bodies, so it must stay loopback-only and never leave the host).

Shape (mirror the agent-proxy role)

Role under ansible/roles/signoz/ with defaults/main.yml (signoz_enabled: false, signoz_dir, UI port, retention knobs), files/compose.yaml, tasks/main.yml (gate on docker CLI, converge compose dir/file, bring up, wait on the UI port), and group_vars/mac.yml opt-in.
Use the upstream SigNoz docker-compose as the source of the stack (clickhouse + query-service + frontend + otel-collector). Pin a version rather than latest.
Bind to loopback only: the SigNoz UI (default :3301) and the OTLP collector ingest (:4317 gRPC, :4318 HTTP) published on 127.0.0.1 only - do not expose on the LAN. The proxy reaches it at http://localhost:4318.
Follow the auto-rebuild/converge lessons from the agent-proxy role: do not gate the bring-up such that it never re-converges (see infrastructure#434), and do not no_log the bring-up in a way that hides failures.

Notes / decisions for the engineer

SigNoz is a heavy multi-container stack. If it proves too heavy for the mac dev loop, note it and consider a lighter local OTLP trace backend (Jaeger all-in-one or Grafana Tempo) as an alternative in the docs - but default to SigNoz to match the fleet tool. Flag the resource cost.
Docs: a docs/signoz-local.md (or a section in an existing runbook) covering where it binds, the OTLP endpoint the proxy uses (PROXY_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318), how to validate (UI on 127.0.0.1:3301, a test span shows up), and that it is local-only / unredacted. Add a docs/FEATURES.md entry.

Acceptance

ward exec ansible-sync on a mac with signoz_enabled: true brings up SigNoz, UI reachable at 127.0.0.1:3301, OTLP ingest at 127.0.0.1:4318, nothing bound on the LAN.
With the proxy pointed at http://localhost:4318, a request's spans appear in the local SigNoz UI (once agent-proxy instrumentation lands).
Disabled by default; opt-in via host vars.

Refs

Pairs with the agent-proxy request-path instrumentation ticket (the producer of the traces).
Sibling dockerized roles: agent-proxy, dozzle, logdy. Heed infrastructure#434 (converge/rebuild + no_log lessons).

## Want A host-local **SigNoz** rollout for full high-detail inspection of agent-proxy (and other local services) on localhost, in the same shape as the existing dockerized host roles (`agent-proxy`, `dozzle`, `logdy`, `mac-proxy`). This is the local OTLP trace/metrics/logs backend the proxy points at during high-detail localhost debugging, kept separate from the shared fleet SigNoz on `ser8` (which is redacted/aggregate; the local one is unsampled and may hold full request/response bodies, so it must stay loopback-only and never leave the host). ## Shape (mirror the agent-proxy role) - Role under `ansible/roles/signoz/` with `defaults/main.yml` (`signoz_enabled: false`, `signoz_dir`, UI port, retention knobs), `files/compose.yaml`, `tasks/main.yml` (gate on docker CLI, converge compose dir/file, bring up, wait on the UI port), and `group_vars/mac.yml` opt-in. - Use the upstream SigNoz docker-compose as the source of the stack (clickhouse + query-service + frontend + otel-collector). Pin a version rather than `latest`. - **Bind to loopback only**: the SigNoz UI (default `:3301`) and the OTLP collector ingest (`:4317` gRPC, `:4318` HTTP) published on `127.0.0.1` only - do not expose on the LAN. The proxy reaches it at `http://localhost:4318`. - Follow the auto-rebuild/converge lessons from the agent-proxy role: do not gate the bring-up such that it never re-converges (see infrastructure#434), and do not `no_log` the bring-up in a way that hides failures. ## Notes / decisions for the engineer - SigNoz is a heavy multi-container stack. If it proves too heavy for the mac dev loop, note it and consider a lighter local OTLP trace backend (Jaeger all-in-one or Grafana Tempo) as an alternative in the docs - but default to SigNoz to match the fleet tool. Flag the resource cost. - Docs: a `docs/signoz-local.md` (or a section in an existing runbook) covering where it binds, the OTLP endpoint the proxy uses (`PROXY_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318`), how to validate (UI on `127.0.0.1:3301`, a test span shows up), and that it is local-only / unredacted. Add a `docs/FEATURES.md` entry. ## Acceptance - `ward exec ansible-sync` on a mac with `signoz_enabled: true` brings up SigNoz, UI reachable at `127.0.0.1:3301`, OTLP ingest at `127.0.0.1:4318`, nothing bound on the LAN. - With the proxy pointed at `http://localhost:4318`, a request's spans appear in the local SigNoz UI (once agent-proxy instrumentation lands). - Disabled by default; opt-in via host vars. ## Refs - Pairs with the agent-proxy request-path instrumentation ticket (the producer of the traces). - Sibling dockerized roles: `agent-proxy`, `dozzle`, `logdy`. Heed infrastructure#434 (converge/rebuild + no_log lessons).

coilyco-ops commented

2026-07-02 09:31:33 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-infrastructure-435 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T09:31:31Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-infrastructure-435` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T09:31:31Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-02 09:42:33 +00:00

feat(ansible): add host-local SigNoz OTLP backend role (closes #435)

coilysiren closed this issue

2026-07-02 09:42:33 +00:00

coilysiren referenced this issue from a commit

2026-07-02 09:42:33 +00:00

Merge issue-435: host-local SigNoz OTLP backend role (closes #435)

coilysiren commented

2026-07-02 09:43:49 +00:00

Author

Owner

WARD-OUTCOME: done - added the signoz ansible role (loopback-only host-local OTLP backend), vendored from SigNoz v0.90.1, merged to main and pushed.

Mostly straightforward since it mirrors the agent-proxy/dozzle/logdy shape, but SigNoz fought back in two ways worth flagging. First, upstream deprecated the docker-compose path entirely for their new "Foundry" installer, so main no longer ships a compose file - I had to vendor from v0.90.1, the last release that does, and it drags in a whole common/ tree of clickhouse/otel config the compose mounts by relative path, so this is ~11 vendored files, not a one-file compose. Second, the repo's code-comments hook forbids any YAML comment below the first content line, which collided with both the vendored files' upstream inline comments (excluded them under [tool.agentic-os.code-comments], since restyling third-party config is wrong) and my own defaults/tasks (folded all commentary into top header blocks).

Confidence: high on the ansible plumbing, port bindings (validated loopback-only), and pre-commit-green. Lower on the live stack: no docker-compose plugin in this Linux container and it's a heavy 5-container mac-targeted stack, so I could not actually bring it up - the real ward exec ansible-sync bring-up on a mac with signoz_enabled: true is unverified by me.

Rough edges / follow-ups worth filing: (1) retention has no compose/env knob in this SigNoz version, so the *_retention_hours defaults are documented-intent-only (set in the UI) - a bit of a smell; (2) it's opt-in per-host and OFF in group_vars (heavy stack), a deliberate call that diverges from the cheap siblings; (3) the paired agent-proxy request-path instrumentation is the actual trace producer and still needs to land before the end-to-end "span shows up in the UI" acceptance can be exercised.

WARD-OUTCOME: done - added the `signoz` ansible role (loopback-only host-local OTLP backend), vendored from SigNoz v0.90.1, merged to main and pushed. Mostly straightforward since it mirrors the agent-proxy/dozzle/logdy shape, but SigNoz fought back in two ways worth flagging. First, upstream **deprecated the docker-compose path entirely** for their new "Foundry" installer, so `main` no longer ships a compose file - I had to vendor from v0.90.1, the last release that does, and it drags in a whole `common/` tree of clickhouse/otel config the compose mounts by relative path, so this is ~11 vendored files, not a one-file compose. Second, the repo's `code-comments` hook forbids any YAML comment below the first content line, which collided with both the vendored files' upstream inline comments (excluded them under `[tool.agentic-os.code-comments]`, since restyling third-party config is wrong) and my own defaults/tasks (folded all commentary into top header blocks). Confidence: high on the ansible plumbing, port bindings (validated loopback-only), and pre-commit-green. Lower on the live stack: no docker-compose plugin in this Linux container and it's a heavy 5-container mac-targeted stack, so I could not actually bring it up - the real `ward exec ansible-sync` bring-up on a mac with `signoz_enabled: true` is unverified by me. Rough edges / follow-ups worth filing: (1) retention has no compose/env knob in this SigNoz version, so the `*_retention_hours` defaults are documented-intent-only (set in the UI) - a bit of a smell; (2) it's opt-in per-host and OFF in group_vars (heavy stack), a deliberate call that diverges from the cheap siblings; (3) the paired agent-proxy request-path instrumentation is the actual trace producer and still needs to land before the end-to-end "span shows up in the UI" acceptance can be exercised.

coilysiren referenced this issue

2026-07-02 10:04:01 +00:00

Wire agent-proxy to export OTLP to the host-local SigNoz backend #436

coilysiren referenced this issue

2026-07-02 10:04:01 +00:00

Enable host-local SigNoz on the local mac (host-scoped opt-in) #437

coilysiren referenced this issue from coilyco-flight-deck/agent-proxy

2026-07-02 10:09:55 +00:00

Build out agent-proxy observability (thin OTLP today -> full traces/metrics/logs) #29

coilysiren referenced this issue from a commit

2026-07-02 10:10:41 +00:00

feat(agent-proxy): export OTLP to host-local SigNoz (closes #436)