coilyco-flight-deck/infrastructure

Fork 0

Converge the always-on observability stack (SigNoz) on the Windows tower #459

New issue

Open

opened 2026-07-03 19:34:11 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-07-03 19:34:11 +00:00

Owner

Converge the always-on observability stack (SigNoz) on the Windows tower

The always-on observability stack (SigNoz + OTLP collector) is defined in ansible
but has not converged on the Windows tower (the WSL2 + Docker Desktop host).
Confirmed down from a container on that host - a port scan shows every endpoint
refusing:

SigNoz UI 3301, OTLP 4317/4318, Grafana 3000, Prometheus 9090,
VictoriaMetrics 8428, Jaeger 16686, ClickHouse 8123  -> all connection-refused

Why it matters

agent-proxy exports Prometheus /metrics (working) and OTel traces over OTLP,
but OTEL_EXPORTER_OTLP_ENDPOINT is unset because there is no collector to point
at, so every trace is a graceful no-op on this host. This blocks end-to-end
validation of the observability buildout in coilyco-flight-deck/agent-proxy#29.

Task

Make the observability always-on ansible target the Windows / WSL Docker
Desktop host and converge it: SigNoz stack up, OTLP collector accepting on
4317 (gRPC) / 4318 (HTTP), UI on 3301.
Document the host-run path. A sealed ward container cannot
docker compose up on the host (the container's docker is not the host's, and
the host does not expose docker over TCP), so this needs host-level execution -
an operator docker compose up on the tower, or an ansible path that runs with
host docker access. Not container-dispatchable as-is.

Acceptance

SigNoz UI reachable on the tower host, OTLP 4317/4318 accepting.
agent-proxy started with PROXY_OTEL_EXPORTER_OTLP_ENDPOINT set shows spans
for /v1/chat/completions in SigNoz.

Related: coilyco-flight-deck/agent-proxy#29 (proxy observability buildout),
#18 (OTel GenAI gen_ai.* semconv target). Filed from the agent-proxy
read-only director surface on the Windows tower.

## Converge the always-on observability stack (SigNoz) on the Windows tower The always-on observability stack (SigNoz + OTLP collector) is defined in ansible but **has not converged on the Windows tower** (the WSL2 + Docker Desktop host). Confirmed down from a container on that host - a port scan shows every endpoint refusing: ``` SigNoz UI 3301, OTLP 4317/4318, Grafana 3000, Prometheus 9090, VictoriaMetrics 8428, Jaeger 16686, ClickHouse 8123 -> all connection-refused ``` **Why it matters** `agent-proxy` exports Prometheus `/metrics` (working) and OTel traces over OTLP, but `OTEL_EXPORTER_OTLP_ENDPOINT` is unset because there is no collector to point at, so every trace is a graceful no-op on this host. This blocks end-to-end validation of the observability buildout in `coilyco-flight-deck/agent-proxy#29`. **Task** - Make the observability always-on ansible target the Windows / WSL Docker Desktop host and converge it: SigNoz stack up, OTLP collector accepting on 4317 (gRPC) / 4318 (HTTP), UI on 3301. - Document the host-run path. **A sealed ward container cannot `docker compose up` on the host** (the container's docker is not the host's, and the host does not expose docker over TCP), so this needs host-level execution - an operator `docker compose up` on the tower, or an ansible path that runs with host docker access. Not container-dispatchable as-is. **Acceptance** - SigNoz UI reachable on the tower host, OTLP 4317/4318 accepting. - `agent-proxy` started with `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT` set shows spans for `/v1/chat/completions` in SigNoz. **Related:** `coilyco-flight-deck/agent-proxy#29` (proxy observability buildout), `#18` (OTel GenAI `gen_ai.*` semconv target). Filed from the agent-proxy read-only director surface on the Windows tower.

coilysiren referenced this issue

2026-07-04 10:38:26 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #470

coilysiren referenced this issue

2026-07-04 10:38:33 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #471

coilyco-ops added the

label

2026-07-04 17:38:19 +00:00

coilyco-ops added the

consult

label

2026-07-07 04:08:39 +00:00

coilyco-ops added

interactive

and removed

consult

labels

2026-07-09 21:46:18 +00:00

coilyco-ops added

consult

and removed

interactive

labels

2026-07-10 08:59:57 +00:00

coilyco-ops commented

2026-07-10 13:16:54 +00:00

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Recommendation

Treat this as an infrastructure-only converge issue. The repo already has the right building block: ansible/roles/signoz vendors SigNoz v0.90.1, publishes UI on 127.0.0.1:3301, OTLP gRPC on 127.0.0.1:4317, and OTLP HTTP on 127.0.0.1:4318. The stack is currently off by default and only hostname-scoped for one Mac via group_vars/mac.yml, so kai-tower-3026-wsl never opts in.

The clean path is to opt in the WSL tower host only, then converge it from an attended host with Docker Desktop WSL integration visible to the WSL distro. Do not run this from a sealed ward container. The host needs the real Docker daemon, so the operational path should be something like ward exec ansible-sync hosts=kai-tower-3026-wsl tags=signoz after the inventory or host vars enable signoz_enabled for that host.

Preferred implementation shape

Add a host-scoped opt-in for kai-tower-3026-wsl, not a Linux group-wide default. This stack is heavy and may contain unredacted request data.
Keep the compose loopback-only. If UI access is needed off-box, use SSH or a deliberate authenticated tailnet front door later. Do not change the SigNoz ports to 0.0.0.0 just to make validation easier.
Document the host-run path in docs/signoz-local.md or a tower-specific runbook section. The key caveat is that a ward feature container cannot bring up host Docker.
After first bring-up, complete SigNoz's required admin/org setup at http://127.0.0.1:3301. The repo's troubleshooting doc notes that OTLP may not actually bind correctly until setup is complete.
Validate from both sides: host loopback (curl http://127.0.0.1:3301, curl http://127.0.0.1:4318/ expecting a non-refused OTLP response) and the producer container path used by agent-proxy.

Decision points

Target host: this should target the WSL host kai-tower-3026-wsl, not native Windows kai-tower-3026, unless #356 has landed and WinRM service management is ready.
Exposure model: keep local SigNoz private and loopback-only. The current role is designed for unsampled, high-detail local debugging.
Producer endpoint: if agent-proxy runs in Docker on the same WSL Docker Desktop daemon, try the existing documented endpoint first: PROXY_OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318. If Docker Desktop on WSL cannot reach a loopback-bound published port through host.docker.internal, the safer fix is to put the producer on the SigNoz Docker network and target http://signoz-otel-collector:4318, not to publish OTLP on the LAN.
Converge mechanism: prefer Ansible over a one-off docker compose up, because the role already owns vendored files, pinned image tags, .env, and validation waits.

Open questions for the engineer

Is the active tower host definitely kai-tower-3026-wsl rather than kai-desktop-tower-wsl?
Does docker ps work for the Ansible SSH user (coilysiren) inside that WSL distro?
Is the tower's agent-proxy managed by this infra role, or is it launched from the sibling repo by hand? If it is role-managed, the host probably also needs a tower-scoped agent_proxy_enabled decision. If it is not role-managed, the runbook should say where to set PROXY_OTEL_EXPORTER_OTLP_ENDPOINT.
Does host.docker.internal:4318 work from the actual agent-proxy container once SigNoz is up? This is the main networking trap to test before declaring the issue done.

Risks

Accidentally enabling signoz_enabled for the whole linux group would land a heavy ClickHouse-backed stack on kai-server, ser8, and both WSL towers.
Changing the compose to LAN-bind ports would expose unredacted local trace data.
First-run SigNoz setup can make OTLP look broken even when containers are up.
Docker Desktop WSL networking can differ from the Mac path already documented, so the producer-to-collector endpoint needs a real container-side test.

Acceptance evidence to ask for on the closing comment

docker ps --filter name=signoz shows the SigNoz stack running on the WSL tower.
curl -fsS http://127.0.0.1:3301 succeeds on the tower host.
OTLP 4317 and 4318 accept connections on the tower host after SigNoz setup is complete.
A request through agent-proxy with PROXY_OTEL_EXPORTER_OTLP_ENDPOINT set produces spans for /v1/chat/completions in the local SigNoz UI.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- **Recommendation** Treat this as an infrastructure-only converge issue. The repo already has the right building block: `ansible/roles/signoz` vendors SigNoz `v0.90.1`, publishes UI on `127.0.0.1:3301`, OTLP gRPC on `127.0.0.1:4317`, and OTLP HTTP on `127.0.0.1:4318`. The stack is currently off by default and only hostname-scoped for one Mac via `group_vars/mac.yml`, so `kai-tower-3026-wsl` never opts in. The clean path is to opt in the WSL tower host only, then converge it from an attended host with Docker Desktop WSL integration visible to the WSL distro. Do not run this from a sealed ward container. The host needs the real Docker daemon, so the operational path should be something like `ward exec ansible-sync hosts=kai-tower-3026-wsl tags=signoz` after the inventory or host vars enable `signoz_enabled` for that host. **Preferred implementation shape** * Add a host-scoped opt-in for `kai-tower-3026-wsl`, not a Linux group-wide default. This stack is heavy and may contain unredacted request data. * Keep the compose loopback-only. If UI access is needed off-box, use SSH or a deliberate authenticated tailnet front door later. Do not change the SigNoz ports to `0.0.0.0` just to make validation easier. * Document the host-run path in `docs/signoz-local.md` or a tower-specific runbook section. The key caveat is that a ward feature container cannot bring up host Docker. * After first bring-up, complete SigNoz's required admin/org setup at `http://127.0.0.1:3301`. The repo's troubleshooting doc notes that OTLP may not actually bind correctly until setup is complete. * Validate from both sides: host loopback (`curl http://127.0.0.1:3301`, `curl http://127.0.0.1:4318/` expecting a non-refused OTLP response) and the producer container path used by `agent-proxy`. **Decision points** * Target host: this should target the WSL host `kai-tower-3026-wsl`, not native Windows `kai-tower-3026`, unless #356 has landed and WinRM service management is ready. * Exposure model: keep local SigNoz private and loopback-only. The current role is designed for unsampled, high-detail local debugging. * Producer endpoint: if `agent-proxy` runs in Docker on the same WSL Docker Desktop daemon, try the existing documented endpoint first: `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318`. If Docker Desktop on WSL cannot reach a loopback-bound published port through `host.docker.internal`, the safer fix is to put the producer on the SigNoz Docker network and target `http://signoz-otel-collector:4318`, not to publish OTLP on the LAN. * Converge mechanism: prefer Ansible over a one-off `docker compose up`, because the role already owns vendored files, pinned image tags, `.env`, and validation waits. **Open questions for the engineer** * Is the active tower host definitely `kai-tower-3026-wsl` rather than `kai-desktop-tower-wsl`? * Does `docker ps` work for the Ansible SSH user (`coilysiren`) inside that WSL distro? * Is the tower's `agent-proxy` managed by this infra role, or is it launched from the sibling repo by hand? If it is role-managed, the host probably also needs a tower-scoped `agent_proxy_enabled` decision. If it is not role-managed, the runbook should say where to set `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT`. * Does `host.docker.internal:4318` work from the actual `agent-proxy` container once SigNoz is up? This is the main networking trap to test before declaring the issue done. **Risks** * Accidentally enabling `signoz_enabled` for the whole `linux` group would land a heavy ClickHouse-backed stack on kai-server, ser8, and both WSL towers. * Changing the compose to LAN-bind ports would expose unredacted local trace data. * First-run SigNoz setup can make OTLP look broken even when containers are up. * Docker Desktop WSL networking can differ from the Mac path already documented, so the producer-to-collector endpoint needs a real container-side test. **Acceptance evidence to ask for on the closing comment** * `docker ps --filter name=signoz` shows the SigNoz stack running on the WSL tower. * `curl -fsS http://127.0.0.1:3301` succeeds on the tower host. * OTLP 4317 and 4318 accept connections on the tower host after SigNoz setup is complete. * A request through `agent-proxy` with `PROXY_OTEL_EXPORTER_OTLP_ENDPOINT` set produces spans for `/v1/chat/completions` in the local SigNoz UI. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilysiren added

interactive

and removed

consult

labels

2026-07-10 23:31:49 +00:00