coilyco-flight-deck/infrastructure

Watch

Local SigNoz drops all telemetry on restart while login/dashboards survive (ClickHouse loses data, sqlite doesn't) #442

New issue

Closed

opened 2026-07-02 17:12:39 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-02 17:12:39 +00:00

Owner

Symptom

The host-local SigNoz stack (ansible/roles/signoz, docker-compose, on the operator's mac) drops all telemetry (traces / metrics / logs) across a restart, while the login / org / dashboards survive. Reported by Kai 2026-07-02.

This is distinct from #439 (OTLP-not-flowing / first-run org setup). Here ingest works and data accumulates, then a restart empties it. Not covered by docs/signoz-troubleshooting.md today.

Why the asymmetry is the key clue

SigNoz keeps two independent stores:

Metadata (users, org, dashboards, alerts, saved views) - sqlite at /var/lib/signoz/signoz.db, volume signoz-sqlite. This survives - hence login persists.
Telemetry (traces/metrics/logs) - ClickHouse at /var/lib/clickhouse, volume signoz-clickhouse. This is what vanishes.

Different fates across the same restart rule out a blanket volume wipe (a down -v / Docker Desktop reset would take sqlite too). So the loss is specific to the ClickHouse side.

Two candidate mechanisms (both fit the asymmetry, different fixes)

A - ReplicatedMergeTree loses ZooKeeper part-metadata (most likely). The vendored compose wires ClickHouse to ZooKeeper (common/clickhouse/cluster.xml, cluster cluster), and SigNoz v0.90.1 creates Replicated*MergeTree tables. Data parts live on ClickHouse local disk (persistent signoz-clickhouse) but every part is registered in ZK under /clickhouse/tables/.... If ZK returns without those znodes on restart, ClickHouse marks replicas read-only and/or detaches all local parts - tables read empty while bytes remain under …/detached/. sqlite is independent, so login stays. Recoverable (re-attach / SYSTEM RESTORE REPLICA). Open question: why ZK state does not survive the restart the way the ClickHouse/sqlite volumes do - the ZK volume signoz-zookeeper-1 is named/persistent in the compose, and ZOO_AUTOPURGE_INTERVAL=1 purges snapshots aggressively - is that (or the bitnamilegacy image data-dir layout) re-initializing ZK?

B - the signoz-clickhouse volume itself is not persisting. If only that named volume is pruned/recreated while signoz-sqlite is not, same visible symptom, but data genuinely gone.

Host triage to disambiguate (operator runs on the mac)

docker volume inspect signoz-clickhouse signoz-sqlite signoz-zookeeper-1 --format '{{.Name}}  created={{.CreatedAt}}'
docker exec signoz-clickhouse du -sh /var/lib/clickhouse/store 2>/dev/null
docker exec signoz-clickhouse bash -lc 'find /var/lib/clickhouse -type d -name detached -not -empty' 2>/dev/null
docker exec signoz-clickhouse clickhouse-client -q "SELECT database, table, is_readonly, is_session_expired FROM system.replicas WHERE is_readonly OR is_session_expired FORMAT PrettyCompact"
docker logs signoz-clickhouse 2>&1 | grep -iE 'readonly|read-only|No node|/clickhouse/tables|Detaching|Cannot attach|is lost' | tail -30

volumes all old + data in store + read-only/detached tables -> mechanism A (recoverable).
signoz-clickhouse newer than signoz-sqlite + empty store -> mechanism B.

Advisor design pass (do this first, post the plan on this thread)

Read ansible/roles/signoz/ (compose, cluster.xml, config.xml, users.xml, defaults, tasks) and produce a concrete fix plan ranked by robustness, without needing host access:

Confirm whether SigNoz v0.90.1's schema-migrator creates Replicated*MergeTree (ZK-backed) tables on this single-node stack, or plain MergeTree. This decides whether mechanism A is even possible.
If Replicated on a single node: evaluate dropping to a non-replicated single-node ClickHouse schema (SigNoz supports a single-node deployment that does not need ZooKeeper) so telemetry survival no longer depends on ZK state at all. Weigh against staying close to the vendored upstream compose (re-vendor discipline).
If staying replicated: what makes ZK state non-durable across restarts here (autopurge, bitnamilegacy data-dir, startup ordering / healthcheck race between ClickHouse and ZooKeeper), and the minimal role change to make it durable.
A recovery runbook for the current already-lost data if mechanism A confirms (re-attach detached parts / SYSTEM RESTORE REPLICA <db>.<table>), added to docs/signoz-troubleshooting.md.
A restart-survival assertion to add to the role's validation section and/or the compose (so this regression is caught, per the infrastructure#434 "never gate the converge so it goes stale" discipline).

Constraints

Local stack only, loopback-only, single mac (signoz_local_mac_hostname). Do not touch the shared fleet SigNoz on ser8.
Stay re-vendorable from SigNoz v0.90.1 where practical (the compose is a vendored artifact - see the header in docker-compose.yaml).
No behavior change that widens the LAN binding.

Done condition

Advisor plan posted here. Then a follow-up engineer run (gated on Kai's host-triage result naming mechanism A vs B) implements the chosen fix + the troubleshooting-doc recovery runbook + the restart-survival check.

Context

Local SigNoz: docs/signoz-local.md, role ansible/roles/signoz/. Related: #435 (local SigNoz role), #436 (proxy->SigNoz wiring), #439 (OTLP-not-flowing triage), and agent-proxy#29 (proxy-side observability build-out that consumes this backend).

## Symptom The **host-local** SigNoz stack (`ansible/roles/signoz`, docker-compose, on the operator's mac) drops **all telemetry** (traces / metrics / logs) across a restart, while the **login / org / dashboards survive**. Reported by Kai 2026-07-02. This is distinct from #439 (OTLP-not-flowing / first-run org setup). Here ingest works and data accumulates, then a restart empties it. Not covered by `docs/signoz-troubleshooting.md` today. ## Why the asymmetry is the key clue SigNoz keeps two independent stores: - **Metadata** (users, org, dashboards, alerts, saved views) - sqlite at `/var/lib/signoz/signoz.db`, volume `signoz-sqlite`. **This survives** - hence login persists. - **Telemetry** (traces/metrics/logs) - ClickHouse at `/var/lib/clickhouse`, volume `signoz-clickhouse`. **This is what vanishes.** Different fates across the same restart rule out a blanket volume wipe (a `down -v` / Docker Desktop reset would take sqlite too). So the loss is specific to the ClickHouse side. ## Two candidate mechanisms (both fit the asymmetry, different fixes) **A - ReplicatedMergeTree loses ZooKeeper part-metadata (most likely).** The vendored compose wires ClickHouse to ZooKeeper (`common/clickhouse/cluster.xml`, cluster `cluster`), and SigNoz v0.90.1 creates `Replicated*MergeTree` tables. Data parts live on ClickHouse local disk (persistent `signoz-clickhouse`) but every part is registered in ZK under `/clickhouse/tables/...`. If ZK returns without those znodes on restart, ClickHouse marks replicas read-only and/or **detaches all local parts** - tables read empty while bytes remain under `…/detached/`. sqlite is independent, so login stays. **Recoverable** (re-attach / `SYSTEM RESTORE REPLICA`). Open question: why ZK state does not survive the restart the way the ClickHouse/sqlite volumes do - the ZK volume `signoz-zookeeper-1` is named/persistent in the compose, and `ZOO_AUTOPURGE_INTERVAL=1` purges snapshots aggressively - is that (or the bitnamilegacy image data-dir layout) re-initializing ZK? **B - the `signoz-clickhouse` volume itself is not persisting.** If only that named volume is pruned/recreated while `signoz-sqlite` is not, same visible symptom, but data genuinely gone. ## Host triage to disambiguate (operator runs on the mac) ```bash docker volume inspect signoz-clickhouse signoz-sqlite signoz-zookeeper-1 --format '{{.Name}} created={{.CreatedAt}}' docker exec signoz-clickhouse du -sh /var/lib/clickhouse/store 2>/dev/null docker exec signoz-clickhouse bash -lc 'find /var/lib/clickhouse -type d -name detached -not -empty' 2>/dev/null docker exec signoz-clickhouse clickhouse-client -q "SELECT database, table, is_readonly, is_session_expired FROM system.replicas WHERE is_readonly OR is_session_expired FORMAT PrettyCompact" docker logs signoz-clickhouse 2>&1 | grep -iE 'readonly|read-only|No node|/clickhouse/tables|Detaching|Cannot attach|is lost' | tail -30 ``` - volumes all old + data in `store` + read-only/detached tables -> mechanism **A** (recoverable). - `signoz-clickhouse` newer than `signoz-sqlite` + empty `store` -> mechanism **B**. ## Advisor design pass (do this first, post the plan on this thread) Read `ansible/roles/signoz/` (compose, `cluster.xml`, `config.xml`, `users.xml`, defaults, tasks) and produce a concrete fix plan ranked by robustness, without needing host access: 1. Confirm whether SigNoz v0.90.1's schema-migrator creates `Replicated*MergeTree` (ZK-backed) tables on this single-node stack, or plain `MergeTree`. This decides whether mechanism A is even possible. 2. If Replicated on a single node: evaluate dropping to a **non-replicated single-node** ClickHouse schema (SigNoz supports a single-node deployment that does not need ZooKeeper) so telemetry survival no longer depends on ZK state at all. Weigh against staying close to the vendored upstream compose (re-vendor discipline). 3. If staying replicated: what makes ZK state non-durable across restarts here (autopurge, bitnamilegacy data-dir, startup ordering / healthcheck race between ClickHouse and ZooKeeper), and the minimal role change to make it durable. 4. A **recovery runbook** for the current already-lost data if mechanism A confirms (re-attach detached parts / `SYSTEM RESTORE REPLICA <db>.<table>`), added to `docs/signoz-troubleshooting.md`. 5. A restart-survival **assertion** to add to the role's validation section and/or the compose (so this regression is caught, per the infrastructure#434 "never gate the converge so it goes stale" discipline). ## Constraints - Local stack only, loopback-only, single mac (`signoz_local_mac_hostname`). Do not touch the shared fleet SigNoz on ser8. - Stay re-vendorable from SigNoz v0.90.1 where practical (the compose is a vendored artifact - see the header in `docker-compose.yaml`). - No behavior change that widens the LAN binding. ## Done condition Advisor plan posted here. Then a follow-up engineer run (gated on Kai's host-triage result naming mechanism A vs B) implements the chosen fix + the troubleshooting-doc recovery runbook + the restart-survival check. ## Context Local SigNoz: `docs/signoz-local.md`, role `ansible/roles/signoz/`. Related: #435 (local SigNoz role), #436 (proxy->SigNoz wiring), #439 (OTLP-not-flowing triage), and agent-proxy#29 (proxy-side observability build-out that consumes this backend).

coilyco-ops commented

2026-07-02 17:20:46 +00:00

Owner

🔎 ward agent advisor

ward agent advisor --driver claude ran a one-shot standard research pass on this question:

Read ansible/roles/signoz (compose, cluster.xml, config.xml, users.xml, defaults, tasks) and produce the fix-plan the issue asks for: confirm whether SigNoz v0.90.1's schema-migrator creates ReplicatedMergeTree (ZK-backed) or plain MergeTree tables on this single-node stack; if replicated, weigh dropping to a non-replicated single-node schema (no ZooKeeper dependency) vs hardening ZK durability, and identify what makes ZK state non-durable across restarts here (autopurge, bitnamilegacy data-dir, startup ordering); draft a recovery runbook for already-lost data (re-attach detached parts / SYSTEM RESTORE REPLICA) for docs/signoz-troubleshooting.md; and propose a restart-survival assertion for the role. Post the ranked plan on the issue thread. Repo-only, no host access needed.

I now have everything needed for a decisive, repo-grounded advisor plan. The key finding overturns the issue's ranked hypotheses. This is single-repo (infrastructure) work, so the full answer goes in summary with an empty issues array.

{
  "summary": "## Advisor plan: the telemetry loss is **not** a ZooKeeper problem \u2014 mechanism A is ruled out by the schema\n\nI read the vendored role (`ansible/roles/signoz/`) and the SigNoz schema-migrator source at the exact pinned tag (`signoz-otel-collector` **v0.128.2**, the `OTELCOL_TAG` the compose runs). The result reorders the whole issue: **hypothesis A (ReplicatedMergeTree loses ZK part-metadata) is architecturally impossible on this stack**, so the ZK-hardening and `SYSTEM RESTORE REPLICA` work the issue proposes would be a dead end. The cause lives on the ClickHouse-data side (mechanism B family). Below is Q1\u2013Q5 answered, re-ranked, plus a corrected recovery runbook, a corrected host-triage, and a restart-survival assertion.\n\n---\n\n## Q1 \u2014 Replicated or plain MergeTree? **Plain MergeTree. Confirmed from source.**\n\nThe compose runs the migrator with **neither** `--replication` **nor** `--cluster-name`:\n\n```yaml\nschema-migrator-sync:  command: [sync,  --dsn=tcp://clickhouse:9000, --up=]\nschema-migrator-async: command: [async, --dsn=tcp://clickhouse:9000, --up=]\n```\n\nIn `signoz-otel-collector` v0.128.2:\n\n- `cmd/signozschemamigrator/main.go:63` \u2014 `BoolVar(&replicationEnabled, \"replication\", false, ...)` \u2192 **default false**.\n- `schema_migrator/table.go:93` \u2014 `EngineType()` returns `\"ReplicatedMergeTree\"` **only** `if m.Replicated`, else `\"MergeTree\"` (same pattern for Replacing/Aggregating/Summing).\n- `m.Replicated` is set **only** by `WithReplication()`, and `manager.go:895` calls it **only** `if m.replicationEnabled`.\n\nSo every SigNoz table on this box is a **plain `MergeTree` / `ReplacingMergeTree` / `AggregatingMergeTree` / `SummingMergeTree`**. The `<macros>` in `config.xml` (`shard=01`, `replica=example01-01-1`) and the `cluster.xml` `<zookeeper>` block are wired, but **no table engine references them**.\n\n**What ZooKeeper is actually used for here:** only the DDL path. Tables are still created `ON CLUSTER cluster` (`manager.go:892`, applied because `clusterName` defaults to `\"cluster\"`), which routes CREATE/ALTER through ClickHouse's `distributed_ddl` queue (`config.xml` `/clickhouse/task_queue/ddl`). That needs ZK **at migration time only**. Once a plain `MergeTree` table exists, **all its data parts live on ClickHouse local disk (`/var/lib/clickhouse`, the persistent `signoz-clickhouse` volume) with zero ZK registration.** Migration *status* is read from the `schema_migrations_v2` table in ClickHouse (`manager.go:211-216`, via `clusterAllReplicas`), **not** from ZK znodes.\n\n**Consequence:** wiping ZooKeeper cannot empty a `MergeTree` table, cannot mark it read-only, and cannot detach its parts (those are `ReplicatedMergeTree` behaviors). **Mechanism A is off the table.** The asymmetry (login/dashboards survive, telemetry vanishes) is real, but it is *not* explained by ZK \u2014 it is a ClickHouse **data-persistence** asymmetry.\n\n---\n\n## Q2 \u2014 Drop to non-replicated single-node? **Already there \u2014 it's a no-op for this bug.**\n\nThe stack is *already* non-replicated single-node for data purposes. There is no replicated schema to drop. Two follow-on points:\n\n- **Do not chase a ZK-free deployment as the fix.** You cannot cleanly remove ZooKeeper by config in v0.128.2: `manager.go:149` hardcodes `CREATE DATABASE IF NOT EXISTS %s ON CLUSTER %s` and the status reads use `clusterAllReplicas($clusterName, ...)`, both of which assume a **non-empty** cluster. Setting `--cluster-name=\"\"` breaks the migrator (`ON CLUSTER ` is invalid SQL). Truly dropping `ON CLUSTER`/ZK would mean patching the migrator, which breaks re-vendor discipline for **zero** durability benefit \u2014 because the data path already never touches ZK.\n- **You could stop ZooKeeper after first migration** and telemetry would still read/write fine until the next DDL. That's a proof of the point, not a recommendation.\n\n**Verdict: no schema change. The fix is not on the ZK/replication axis at all.**\n\n---\n\n## Q3 \u2014 What makes ZK state non-durable here? **Wrong question for this symptom \u2014 and ZK is actually fine.**\n\nSince data survival doesn't depend on ZK, ZK durability is not the fix. For completeness, the issue's ZK suspicions don't hold up either:\n\n- **`ZOO_AUTOPURGE_INTERVAL=1` is a red herring.** Bitnami's autopurge runs the ZK `PurgeTxnLog` task every 1 hour and keeps `ZOO_AUTOPURGE_RETAIN_COUNT` (default **3**) most-recent snapshots plus their trailing txn logs. It **only trims superseded snapshots/logs**, never the current committed state. It cannot empty the live tree.\n- **The ZK data-dir is persistent.** `zookeeper-1:/bitnami/zookeeper` is a named volume (`signoz-zookeeper-1`); bitnamilegacy stores state under `/bitnami/zookeeper/data`, which is inside that volume. It survives restarts like any other named volume.\n- Even in the counterfactual where ZK *were* wiped, the migrator is idempotent and reads its status from ClickHouse, so a fresh ZK just means an empty `distributed_ddl` queue (re-populated on next DDL). No data loss path.\n\n**Verdict: leave ZK alone. Low priority. Not the regression.**\n\n---\n\n## Re-ranked cause: **mechanism B** \u2014 ClickHouse data not persisting (or being orphaned)\n\nSince the bytes-vs-empty asymmetry can't be ZK, it splits into two sub-cases, and the operator's triage should target *these*:\n\n- **B1 \u2014 genuine volume loss.** The `signoz-clickhouse` named volume is being recreated (empty `store/`). Data is gone, unrecoverable.\n- **B2 \u2014 orphaned Atomic store.** ClickHouse's default DB engine is **Atomic**: each table's parts live under `store/<uuid>/`, addressed by `metadata/<db>/<table>.sql`. If the tables get dropped-and-recreated (new UUIDs) while `store/` bytes remain, `du -sh store` shows data but the live tables read empty. Recoverable by re-attaching.\n\nI grepped the repo for a destructive restart path (`down -v`, `volume rm`, `volume prune`) in docs, the Makefile, `.ward/ward.yaml`, and the role \u2014 **none present**. So the trigger is host-side (an operator `docker compose down -v`, a `docker volume prune`, a Docker Desktop \u201cReset / purge data,\u201d or a first-run where the clickhouse volume was created before the config settled). The fix is to make the ClickHouse data **immune** to those, independent of which one it was.\n\n---\n\n## The actual fix, ranked by robustness\n\n**1. (Recommended) Move telemetry + metadata onto host bind-mounts under `signoz_dir`, not opaque named volumes.** Change the three data mounts so the data is a *visible directory on the mac* that `docker compose down -v` and `docker volume prune` cannot touch:\n\n```yaml\n# clickhouse service\n  - ${SIGNOZ_DATA:-./data}/clickhouse:/var/lib/clickhouse/\n# zookeeper-1 service\n  - ${SIGNOZ_DATA:-./data}/zookeeper:/bitnami/zookeeper\n# signoz service\n  - ${SIGNOZ_DATA:-./data}/sqlite:/var/lib/signoz/\n```\n\nRole change: `ansible.builtin.file` to pre-create `{{ signoz_dir }}/data/{clickhouse,sqlite,zookeeper}` (clickhouse needs uid 101 or the container's `user`, zookeeper runs `user: root`). This is a **localized, documented edit** in the same spirit as the existing loopback-port and bitnamilegacy edits already called out in the compose header, so re-vendor stays a clean diff. It kills B1 outright and makes B2 diagnosable with a plain `ls`/`du` on the mac (no `docker exec`). **This is the primary recommendation.**\n\n**2. (Lighter, if bind-mounts are unwanted) Mark the data volumes `external: true`.** `docker compose down -v` refuses to remove external volumes, so an accidental teardown can no longer wipe them. Operator creates them once (`docker volume create signoz-clickhouse ...`). Stays closer to vendor but leaves data hidden in the Docker VM and doesn't defend against `docker volume prune`/factory-reset.\n\n**3. (Cheap, do regardless) Document the safe restart contract.** Add to `docs/signoz-local.md`: restart with `docker compose restart` or `docker compose down && docker compose up -d` \u2014 **never** `down -v`, and never `docker volume prune` while this stack matters. The role already re-runs `up -d --remove-orphans` every converge (safe: removes orphan *containers*, not volumes).\n\n---\n\n## Q4 \u2014 Recovery runbook for already-lost data (for `docs/signoz-troubleshooting.md`)\n\n**Correction to the issue:** `SYSTEM RESTORE REPLICA` **does not apply here** \u2014 it operates only on `ReplicatedMergeTree`, and errors (`Table ... is not replicated`) on these plain `MergeTree` tables. Do not put it in the runbook. The correct recovery is classify-then-attach:\n\n```bash\n# 0. Confirm the schema is plain MergeTree (proves mechanism A is impossible on THIS box)\ndocker exec signoz-clickhouse clickhouse-client -q \\\n  \"SELECT database, name, engine FROM system.tables\n   WHERE database LIKE 'signoz_%' AND engine LIKE '%MergeTree%'\n   ORDER BY database, name FORMAT PrettyCompact\"\n#   -> engines read 'MergeTree'/'ReplacingMergeTree'/... (NO 'Replicated' prefix)\n\n# 1. Classify B1 vs B2: is there data on disk at all?\ndocker exec signoz-clickhouse du -sh /var/lib/clickhouse/store\ndocker exec signoz-clickhouse clickhouse-client -q \\\n  \"SELECT database, table, sum(rows) rows, formatReadableSize(sum(bytes_on_disk)) size\n   FROM system.parts WHERE active AND database LIKE 'signoz_%'\n   GROUP BY database, table ORDER BY rows DESC FORMAT PrettyCompact\"\n\n# 2a. B1 (store ~empty, parts = 0 rows): data is GONE. No recovery. Apply the fix above so it can't recur.\n\n# 2b. B2 (store has bytes but live tables empty): look for detached / orphaned parts\ndocker exec signoz-clickhouse clickhouse-client -q \\\n  \"SELECT database, table, count() FROM system.detached_parts\n   WHERE database LIKE 'signoz_%' GROUP BY database, table FORMAT PrettyCompact\"\ndocker exec signoz-clickhouse bash -lc \\\n  'find /var/lib/clickhouse -type d -name detached -not -empty'\n\n# 3. Re-attach detached parts, per table (plain-MergeTree recovery, NOT SYSTEM RESTORE REPLICA)\ndocker exec signoz-clickhouse clickhouse-client -q \\\n  \"ALTER TABLE signoz_traces.signoz_index_v3 ATTACH PARTITION ALL\"   # or per-part: ATTACH PART '<name>'\n#   Repeat for each affected signoz_* table shown in step 2b.\n```\n\nIf B2 is orphaned *Atomic store dirs* (parts present under `store/<uuid>` but no matching table), the recovery is to recreate the table and `ATTACH` the orphaned parts into its data dir, or use `SYSTEM RESTORE DATABASE`/manual `ATTACH TABLE` \u2014 note in the doc that this is fiddly and B1 prevention is the real answer.\n\n---\n\n## Q5 \u2014 Restart-survival assertion (per infra#434: never gate so it goes stale)\n\nTwo layers, both non-destructive to the operator's live data:\n\n**A. Always-on converge guard (cheap, encodes this exact bug's signature).** After bring-up, compare volume ages \u2014 the telemetry store being *younger* than the metadata store is precisely the fingerprint of \u201ctelemetry wiped, login survived\u201d:\n\n```yaml\n- name: Assert the telemetry volume was not recreated alone\n  ansible.builtin.shell: |\n    ch=$(docker volume inspect signoz-clickhouse --format '{{ '{{' }}.CreatedAt{{ '}}' }}')\n    sq=$(docker volume inspect signoz-sqlite     --format '{{ '{{' }}.CreatedAt{{ '}}' }}')\n    test \"$(date -d \"$ch\" +%s)\" -le \"$(date -d \"$sq\" +%s)\"\n  register: sz_vol_age\n  changed_when: false\n  failed_when: sz_vol_age.rc != 0\n  # Fails when signoz-clickhouse is newer than signoz-sqlite -> telemetry store\n  # was recreated without the metadata store. Runs every converge, never gated.\n```\n\n(If bind-mounts are adopted per fix #1, swap this for a check that the `data/clickhouse` dir exists and is non-empty when `data/sqlite` is.)\n\n**B. CI restart-survival smoke (molecule, against throwaway data \u2014 the real regression test).** In the role's molecule `verify`: bring the stack up, POST a canary span to `127.0.0.1:4318/v1/traces`, run `docker compose restart` **and** a `down && up -d` (no `-v`), then assert the canary is still queryable:\n\n```bash\ndocker exec signoz-clickhouse clickhouse-client -q \\\n  \"SELECT count() FROM signoz_traces.signoz_index_v3 WHERE serviceName='restart-canary'\"\n# must be > 0 after the restart cycle, else fail the build\n```\n\nThis catches any future re-vendor or volume-name change that reintroduces the regression, without ever touching the operator's personal telemetry.\n\n---\n\n## Refined host-triage to post back (disambiguate B1 vs B2, and nail the schema on the live box)\n\n```bash\n# (1) Prove plain MergeTree on the actual host (kills mechanism A definitively)\ndocker exec signoz-clickhouse clickhouse-client -q \\\n  \"SELECT database,name,engine FROM system.tables WHERE database LIKE 'signoz_%' FORMAT PrettyCompact\"\n# (2) volume ages: is signoz-clickhouse newer than signoz-sqlite?\ndocker volume inspect signoz-clickhouse signoz-sqlite signoz-zookeeper-1 --format '{{ '{{' }}.Name{{ '}}' }} {{ '{{' }}.CreatedAt{{ '}}' }}'\n# (3) is there data on disk?\ndocker exec signoz-clickhouse du -sh /var/lib/clickhouse/store\ndocker exec signoz-clickhouse clickhouse-client -q \\\n  \"SELECT database,table,sum(rows) FROM system.parts WHERE active AND database LIKE 'signoz_%' GROUP BY database,table FORMAT PrettyCompact\"\n```\n\n- engines show **no** `Replicated` prefix (they will) **+** volumes same-age **+** empty `store` \u2192 **B1** (volume recreated, data gone, apply fix #1).\n- engines plain **+** `store` has bytes but live parts = 0 \u2192 **B2** (orphaned/detached, recover via Q4 step 3).\n\nThe issue's original triage line looking for `readonly`/`is_session_expired`/`/clickhouse/tables` will come back **empty** \u2014 that's expected and confirms the schema is non-replicated, not a bug in the triage.\n\n---\n\n## Suggested follow-up (single repo: `coilyco-flight-deck/infrastructure`)\n\nOne engineer run, gated on the operator's B1-vs-B2 result: (1) implement fix #1 (bind-mount the three data dirs, role pre-creates them, `docs/signoz-local.md` safe-restart note), (2) add the Q4 recovery runbook to `docs/signoz-troubleshooting.md` with the `SYSTEM RESTORE REPLICA`-does-not-apply correction, (3) add both Q5 assertions. All confined to `ansible/roles/signoz/` + `docs/`, so this stays a single-repo change \u2014 no cross-repo fan-out.\n\n**Sources:** [SigNoz upgrade-0.113 (migrator env: DSN/CLUSTER/REPLICATION)](https://signoz.io/docs/operate/migration/upgrade-0-113/), plus the pinned source read directly at `signoz-otel-collector` v0.128.2 (`cmd/signozschemamigrator/main.go`, `schema_migrator/table.go`, `schema_migrator/manager.go`).",
  "issues": []
}

Researched and posted automatically by ward agent advisor --driver claude (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Claude (she/her), via ward agent

### 🔎 ward agent advisor `ward agent advisor --driver claude` ran a one-shot **standard** research pass on this question: > Read ansible/roles/signoz (compose, cluster.xml, config.xml, users.xml, defaults, tasks) and produce the fix-plan the issue asks for: confirm whether SigNoz v0.90.1's schema-migrator creates ReplicatedMergeTree (ZK-backed) or plain MergeTree tables on this single-node stack; if replicated, weigh dropping to a non-replicated single-node schema (no ZooKeeper dependency) vs hardening ZK durability, and identify what makes ZK state non-durable across restarts here (autopurge, bitnamilegacy data-dir, startup ordering); draft a recovery runbook for already-lost data (re-attach detached parts / SYSTEM RESTORE REPLICA) for docs/signoz-troubleshooting.md; and propose a restart-survival assertion for the role. Post the ranked plan on the issue thread. Repo-only, no host access needed. --- I now have everything needed for a decisive, repo-grounded advisor plan. The key finding overturns the issue's ranked hypotheses. This is single-repo (infrastructure) work, so the full answer goes in `summary` with an empty `issues` array. ```json { "summary": "## Advisor plan: the telemetry loss is **not** a ZooKeeper problem \u2014 mechanism A is ruled out by the schema\n\nI read the vendored role (`ansible/roles/signoz/`) and the SigNoz schema-migrator source at the exact pinned tag (`signoz-otel-collector` **v0.128.2**, the `OTELCOL_TAG` the compose runs). The result reorders the whole issue: **hypothesis A (ReplicatedMergeTree loses ZK part-metadata) is architecturally impossible on this stack**, so the ZK-hardening and `SYSTEM RESTORE REPLICA` work the issue proposes would be a dead end. The cause lives on the ClickHouse-data side (mechanism B family). Below is Q1\u2013Q5 answered, re-ranked, plus a corrected recovery runbook, a corrected host-triage, and a restart-survival assertion.\n\n---\n\n## Q1 \u2014 Replicated or plain MergeTree? **Plain MergeTree. Confirmed from source.**\n\nThe compose runs the migrator with **neither** `--replication` **nor** `--cluster-name`:\n\n```yaml\nschema-migrator-sync: command: [sync, --dsn=tcp://clickhouse:9000, --up=]\nschema-migrator-async: command: [async, --dsn=tcp://clickhouse:9000, --up=]\n```\n\nIn `signoz-otel-collector` v0.128.2:\n\n- `cmd/signozschemamigrator/main.go:63` \u2014 `BoolVar(&replicationEnabled, \"replication\", false, ...)` \u2192 **default false**.\n- `schema_migrator/table.go:93` \u2014 `EngineType()` returns `\"ReplicatedMergeTree\"` **only** `if m.Replicated`, else `\"MergeTree\"` (same pattern for Replacing/Aggregating/Summing).\n- `m.Replicated` is set **only** by `WithReplication()`, and `manager.go:895` calls it **only** `if m.replicationEnabled`.\n\nSo every SigNoz table on this box is a **plain `MergeTree` / `ReplacingMergeTree` / `AggregatingMergeTree` / `SummingMergeTree`**. The `<macros>` in `config.xml` (`shard=01`, `replica=example01-01-1`) and the `cluster.xml` `<zookeeper>` block are wired, but **no table engine references them**.\n\n**What ZooKeeper is actually used for here:** only the DDL path. Tables are still created `ON CLUSTER cluster` (`manager.go:892`, applied because `clusterName` defaults to `\"cluster\"`), which routes CREATE/ALTER through ClickHouse's `distributed_ddl` queue (`config.xml` `/clickhouse/task_queue/ddl`). That needs ZK **at migration time only**. Once a plain `MergeTree` table exists, **all its data parts live on ClickHouse local disk (`/var/lib/clickhouse`, the persistent `signoz-clickhouse` volume) with zero ZK registration.** Migration *status* is read from the `schema_migrations_v2` table in ClickHouse (`manager.go:211-216`, via `clusterAllReplicas`), **not** from ZK znodes.\n\n**Consequence:** wiping ZooKeeper cannot empty a `MergeTree` table, cannot mark it read-only, and cannot detach its parts (those are `ReplicatedMergeTree` behaviors). **Mechanism A is off the table.** The asymmetry (login/dashboards survive, telemetry vanishes) is real, but it is *not* explained by ZK \u2014 it is a ClickHouse **data-persistence** asymmetry.\n\n---\n\n## Q2 \u2014 Drop to non-replicated single-node? **Already there \u2014 it's a no-op for this bug.**\n\nThe stack is *already* non-replicated single-node for data purposes. There is no replicated schema to drop. Two follow-on points:\n\n- **Do not chase a ZK-free deployment as the fix.** You cannot cleanly remove ZooKeeper by config in v0.128.2: `manager.go:149` hardcodes `CREATE DATABASE IF NOT EXISTS %s ON CLUSTER %s` and the status reads use `clusterAllReplicas($clusterName, ...)`, both of which assume a **non-empty** cluster. Setting `--cluster-name=\"\"` breaks the migrator (`ON CLUSTER ` is invalid SQL). Truly dropping `ON CLUSTER`/ZK would mean patching the migrator, which breaks re-vendor discipline for **zero** durability benefit \u2014 because the data path already never touches ZK.\n- **You could stop ZooKeeper after first migration** and telemetry would still read/write fine until the next DDL. That's a proof of the point, not a recommendation.\n\n**Verdict: no schema change. The fix is not on the ZK/replication axis at all.**\n\n---\n\n## Q3 \u2014 What makes ZK state non-durable here? **Wrong question for this symptom \u2014 and ZK is actually fine.**\n\nSince data survival doesn't depend on ZK, ZK durability is not the fix. For completeness, the issue's ZK suspicions don't hold up either:\n\n- **`ZOO_AUTOPURGE_INTERVAL=1` is a red herring.** Bitnami's autopurge runs the ZK `PurgeTxnLog` task every 1 hour and keeps `ZOO_AUTOPURGE_RETAIN_COUNT` (default **3**) most-recent snapshots plus their trailing txn logs. It **only trims superseded snapshots/logs**, never the current committed state. It cannot empty the live tree.\n- **The ZK data-dir is persistent.** `zookeeper-1:/bitnami/zookeeper` is a named volume (`signoz-zookeeper-1`); bitnamilegacy stores state under `/bitnami/zookeeper/data`, which is inside that volume. It survives restarts like any other named volume.\n- Even in the counterfactual where ZK *were* wiped, the migrator is idempotent and reads its status from ClickHouse, so a fresh ZK just means an empty `distributed_ddl` queue (re-populated on next DDL). No data loss path.\n\n**Verdict: leave ZK alone. Low priority. Not the regression.**\n\n---\n\n## Re-ranked cause: **mechanism B** \u2014 ClickHouse data not persisting (or being orphaned)\n\nSince the bytes-vs-empty asymmetry can't be ZK, it splits into two sub-cases, and the operator's triage should target *these*:\n\n- **B1 \u2014 genuine volume loss.** The `signoz-clickhouse` named volume is being recreated (empty `store/`). Data is gone, unrecoverable.\n- **B2 \u2014 orphaned Atomic store.** ClickHouse's default DB engine is **Atomic**: each table's parts live under `store/<uuid>/`, addressed by `metadata/<db>/<table>.sql`. If the tables get dropped-and-recreated (new UUIDs) while `store/` bytes remain, `du -sh store` shows data but the live tables read empty. Recoverable by re-attaching.\n\nI grepped the repo for a destructive restart path (`down -v`, `volume rm`, `volume prune`) in docs, the Makefile, `.ward/ward.yaml`, and the role \u2014 **none present**. So the trigger is host-side (an operator `docker compose down -v`, a `docker volume prune`, a Docker Desktop \u201cReset / purge data,\u201d or a first-run where the clickhouse volume was created before the config settled). The fix is to make the ClickHouse data **immune** to those, independent of which one it was.\n\n---\n\n## The actual fix, ranked by robustness\n\n**1. (Recommended) Move telemetry + metadata onto host bind-mounts under `signoz_dir`, not opaque named volumes.** Change the three data mounts so the data is a *visible directory on the mac* that `docker compose down -v` and `docker volume prune` cannot touch:\n\n```yaml\n# clickhouse service\n - ${SIGNOZ_DATA:-./data}/clickhouse:/var/lib/clickhouse/\n# zookeeper-1 service\n - ${SIGNOZ_DATA:-./data}/zookeeper:/bitnami/zookeeper\n# signoz service\n - ${SIGNOZ_DATA:-./data}/sqlite:/var/lib/signoz/\n```\n\nRole change: `ansible.builtin.file` to pre-create `{{ signoz_dir }}/data/{clickhouse,sqlite,zookeeper}` (clickhouse needs uid 101 or the container's `user`, zookeeper runs `user: root`). This is a **localized, documented edit** in the same spirit as the existing loopback-port and bitnamilegacy edits already called out in the compose header, so re-vendor stays a clean diff. It kills B1 outright and makes B2 diagnosable with a plain `ls`/`du` on the mac (no `docker exec`). **This is the primary recommendation.**\n\n**2. (Lighter, if bind-mounts are unwanted) Mark the data volumes `external: true`.** `docker compose down -v` refuses to remove external volumes, so an accidental teardown can no longer wipe them. Operator creates them once (`docker volume create signoz-clickhouse ...`). Stays closer to vendor but leaves data hidden in the Docker VM and doesn't defend against `docker volume prune`/factory-reset.\n\n**3. (Cheap, do regardless) Document the safe restart contract.** Add to `docs/signoz-local.md`: restart with `docker compose restart` or `docker compose down && docker compose up -d` \u2014 **never** `down -v`, and never `docker volume prune` while this stack matters. The role already re-runs `up -d --remove-orphans` every converge (safe: removes orphan *containers*, not volumes).\n\n---\n\n## Q4 \u2014 Recovery runbook for already-lost data (for `docs/signoz-troubleshooting.md`)\n\n**Correction to the issue:** `SYSTEM RESTORE REPLICA` **does not apply here** \u2014 it operates only on `ReplicatedMergeTree`, and errors (`Table ... is not replicated`) on these plain `MergeTree` tables. Do not put it in the runbook. The correct recovery is classify-then-attach:\n\n```bash\n# 0. Confirm the schema is plain MergeTree (proves mechanism A is impossible on THIS box)\ndocker exec signoz-clickhouse clickhouse-client -q \\\n \"SELECT database, name, engine FROM system.tables\n WHERE database LIKE 'signoz_%' AND engine LIKE '%MergeTree%'\n ORDER BY database, name FORMAT PrettyCompact\"\n# -> engines read 'MergeTree'/'ReplacingMergeTree'/... (NO 'Replicated' prefix)\n\n# 1. Classify B1 vs B2: is there data on disk at all?\ndocker exec signoz-clickhouse du -sh /var/lib/clickhouse/store\ndocker exec signoz-clickhouse clickhouse-client -q \\\n \"SELECT database, table, sum(rows) rows, formatReadableSize(sum(bytes_on_disk)) size\n FROM system.parts WHERE active AND database LIKE 'signoz_%'\n GROUP BY database, table ORDER BY rows DESC FORMAT PrettyCompact\"\n\n# 2a. B1 (store ~empty, parts = 0 rows): data is GONE. No recovery. Apply the fix above so it can't recur.\n\n# 2b. B2 (store has bytes but live tables empty): look for detached / orphaned parts\ndocker exec signoz-clickhouse clickhouse-client -q \\\n \"SELECT database, table, count() FROM system.detached_parts\n WHERE database LIKE 'signoz_%' GROUP BY database, table FORMAT PrettyCompact\"\ndocker exec signoz-clickhouse bash -lc \\\n 'find /var/lib/clickhouse -type d -name detached -not -empty'\n\n# 3. Re-attach detached parts, per table (plain-MergeTree recovery, NOT SYSTEM RESTORE REPLICA)\ndocker exec signoz-clickhouse clickhouse-client -q \\\n \"ALTER TABLE signoz_traces.signoz_index_v3 ATTACH PARTITION ALL\" # or per-part: ATTACH PART '<name>'\n# Repeat for each affected signoz_* table shown in step 2b.\n```\n\nIf B2 is orphaned *Atomic store dirs* (parts present under `store/<uuid>` but no matching table), the recovery is to recreate the table and `ATTACH` the orphaned parts into its data dir, or use `SYSTEM RESTORE DATABASE`/manual `ATTACH TABLE` \u2014 note in the doc that this is fiddly and B1 prevention is the real answer.\n\n---\n\n## Q5 \u2014 Restart-survival assertion (per infra#434: never gate so it goes stale)\n\nTwo layers, both non-destructive to the operator's live data:\n\n**A. Always-on converge guard (cheap, encodes this exact bug's signature).** After bring-up, compare volume ages \u2014 the telemetry store being *younger* than the metadata store is precisely the fingerprint of \u201ctelemetry wiped, login survived\u201d:\n\n```yaml\n- name: Assert the telemetry volume was not recreated alone\n ansible.builtin.shell: |\n ch=$(docker volume inspect signoz-clickhouse --format '{{ '{{' }}.CreatedAt{{ '}}' }}')\n sq=$(docker volume inspect signoz-sqlite --format '{{ '{{' }}.CreatedAt{{ '}}' }}')\n test \"$(date -d \"$ch\" +%s)\" -le \"$(date -d \"$sq\" +%s)\"\n register: sz_vol_age\n changed_when: false\n failed_when: sz_vol_age.rc != 0\n # Fails when signoz-clickhouse is newer than signoz-sqlite -> telemetry store\n # was recreated without the metadata store. Runs every converge, never gated.\n```\n\n(If bind-mounts are adopted per fix #1, swap this for a check that the `data/clickhouse` dir exists and is non-empty when `data/sqlite` is.)\n\n**B. CI restart-survival smoke (molecule, against throwaway data \u2014 the real regression test).** In the role's molecule `verify`: bring the stack up, POST a canary span to `127.0.0.1:4318/v1/traces`, run `docker compose restart` **and** a `down && up -d` (no `-v`), then assert the canary is still queryable:\n\n```bash\ndocker exec signoz-clickhouse clickhouse-client -q \\\n \"SELECT count() FROM signoz_traces.signoz_index_v3 WHERE serviceName='restart-canary'\"\n# must be > 0 after the restart cycle, else fail the build\n```\n\nThis catches any future re-vendor or volume-name change that reintroduces the regression, without ever touching the operator's personal telemetry.\n\n---\n\n## Refined host-triage to post back (disambiguate B1 vs B2, and nail the schema on the live box)\n\n```bash\n# (1) Prove plain MergeTree on the actual host (kills mechanism A definitively)\ndocker exec signoz-clickhouse clickhouse-client -q \\\n \"SELECT database,name,engine FROM system.tables WHERE database LIKE 'signoz_%' FORMAT PrettyCompact\"\n# (2) volume ages: is signoz-clickhouse newer than signoz-sqlite?\ndocker volume inspect signoz-clickhouse signoz-sqlite signoz-zookeeper-1 --format '{{ '{{' }}.Name{{ '}}' }} {{ '{{' }}.CreatedAt{{ '}}' }}'\n# (3) is there data on disk?\ndocker exec signoz-clickhouse du -sh /var/lib/clickhouse/store\ndocker exec signoz-clickhouse clickhouse-client -q \\\n \"SELECT database,table,sum(rows) FROM system.parts WHERE active AND database LIKE 'signoz_%' GROUP BY database,table FORMAT PrettyCompact\"\n```\n\n- engines show **no** `Replicated` prefix (they will) **+** volumes same-age **+** empty `store` \u2192 **B1** (volume recreated, data gone, apply fix #1).\n- engines plain **+** `store` has bytes but live parts = 0 \u2192 **B2** (orphaned/detached, recover via Q4 step 3).\n\nThe issue's original triage line looking for `readonly`/`is_session_expired`/`/clickhouse/tables` will come back **empty** \u2014 that's expected and confirms the schema is non-replicated, not a bug in the triage.\n\n---\n\n## Suggested follow-up (single repo: `coilyco-flight-deck/infrastructure`)\n\nOne engineer run, gated on the operator's B1-vs-B2 result: (1) implement fix #1 (bind-mount the three data dirs, role pre-creates them, `docs/signoz-local.md` safe-restart note), (2) add the Q4 recovery runbook to `docs/signoz-troubleshooting.md` with the `SYSTEM RESTORE REPLICA`-does-not-apply correction, (3) add both Q5 assertions. All confined to `ansible/roles/signoz/` + `docs/`, so this stays a single-repo change \u2014 no cross-repo fan-out.\n\n**Sources:** [SigNoz upgrade-0.113 (migrator env: DSN/CLUSTER/REPLICATION)](https://signoz.io/docs/operate/migration/upgrade-0-113/), plus the pinned source read directly at `signoz-otel-collector` v0.128.2 (`cmd/signozschemamigrator/main.go`, `schema_migrator/table.go`, `schema_migrator/manager.go`).", "issues": [] } ``` --- Researched and posted automatically by `ward agent advisor --driver claude` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Claude (she/her), via `ward agent`

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-03 07:58:35 +00:00

coilyco-ops added

priority

and removed

priority

labels

2026-07-09 21:46:21 +00:00

coilysiren removed the

autonomy

async-consult

label

2026-07-10 23:32:55 +00:00

coilyco-ops commented

2026-07-16 00:49:55 +00:00

Owner

Backlog consolidation on 2026-07-15: the agent is closing this P4 item as a reversible icebox entry. The work remains intentionally parked rather than completed. Kai can reopen it if she promotes it back into active direction.

coilyco-ops closed this issue

2026-07-16 00:49:56 +00:00

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#442

No description provided.

Rows
Columns