coilyco-flight-deck/infrastructure

Fork 0

Retire kai-server observability stack after ser8 o11y cutover #505

New issue

Open

opened 2026-07-09 21:27:43 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-09 21:27:43 +00:00

Member

Kai noticed during a namespace memory review that observability is the largest remaining kai-server namespace:

2834Mi observability

That is about 8.9% of kai-server's 31.04GiB RAM, and it should not be the long-term shape. The deploy docs already say ser8 is the canonical o11y plane and kai-server's in-cluster observability stack is retiring:

docs/o11y.md: kai-server is the app plane; its old observability namespace is the retiring stack.
services/ser8-observability/: vmsingle/vmalert and SigNoz live on ser8.
Related deploy issue: coilyco-bridge/deploy#51 designs migrating glitchtip/signoz/vmsingle deploy ownership.
Related deploy issue: coilyco-bridge/deploy#114 tracks product-level ownership for the old shared observability bucket.

Please plan and execute the kai-server o11y retirement safely:

Inventory everything currently running in kai-server observability and classify each component: already replaced by ser8, still producing unique data, or legacy dead weight.
Confirm producers have moved or can move to ser8: metrics to ser8 VictoriaMetrics, traces/logs to ser8 SigNoz, errors to GlitchTip where applicable.
Preserve or explicitly drop historical data. Do not blindly delete PVC-backed stores.
Remove kai-server observability workloads that ser8 now replaces, including manifests/verbs/docs that would recreate them.
Keep any minimal bridge components only if still required during the cutover, and document their removal condition.
Coordinate with coilyco-bridge/deploy#51 rather than duplicating deploy-repo migration work.

Acceptance: kai-server no longer carries the old all-in-one observability stack once ser8 coverage is verified, and a namespace memory rollup no longer shows observability as a multi-GiB app-plane consumer.

Kai noticed during a namespace memory review that `observability` is the largest remaining kai-server namespace: ```text 2834Mi observability ``` That is about 8.9% of kai-server's 31.04GiB RAM, and it should not be the long-term shape. The deploy docs already say ser8 is the canonical o11y plane and kai-server's in-cluster observability stack is retiring: * `docs/o11y.md`: kai-server is the app plane; its old `observability` namespace is the retiring stack. * `services/ser8-observability/`: vmsingle/vmalert and SigNoz live on ser8. * Related deploy issue: `coilyco-bridge/deploy#51` designs migrating glitchtip/signoz/vmsingle deploy ownership. * Related deploy issue: `coilyco-bridge/deploy#114` tracks product-level ownership for the old shared `observability` bucket. Please plan and execute the kai-server o11y retirement safely: * Inventory everything currently running in kai-server `observability` and classify each component: already replaced by ser8, still producing unique data, or legacy dead weight. * Confirm producers have moved or can move to ser8: metrics to ser8 VictoriaMetrics, traces/logs to ser8 SigNoz, errors to GlitchTip where applicable. * Preserve or explicitly drop historical data. Do not blindly delete PVC-backed stores. * Remove kai-server `observability` workloads that ser8 now replaces, including manifests/verbs/docs that would recreate them. * Keep any minimal bridge components only if still required during the cutover, and document their removal condition. * Coordinate with `coilyco-bridge/deploy#51` rather than duplicating deploy-repo migration work. Acceptance: kai-server no longer carries the old all-in-one `observability` stack once ser8 coverage is verified, and a namespace memory rollup no longer shows `observability` as a multi-GiB app-plane consumer.

coilyco-ops commented

2026-07-09 21:30:58 +00:00

Author

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Advisory recommendation

The repo evidence supports treating this as a staged kai-server observability retirement, not a one-shot namespace delete.

The safest recommendation is:

Retire kai-server VictoriaMetrics and SigNoz first. Those are clearly described here as the old in-cluster observability stack, and they are also the parts most plausibly driving the 2834Mi observability footprint. The strongest likely RAM culprit is SigNoz ClickHouse alone, which is configured at 4Gi request / 8Gi limit in deploy/observability/signoz-values.yml.
Do not blindly delete PVC-backed data. This repo's backup path explicitly says metrics and traces are out of scope for restic in docs/restic-backups.md, and the backup script only includes forgejo registry coilysiren-backend ntfy namespaces plus /home/kai/saves. That means historical VictoriaMetrics and SigNoz data are not covered by the normal offsite backup flow. GlitchTip also is not named in the backup namespaces.
Leave GlitchTip as a separate decision gate unless coilyco-bridge/deploy#51 already settled its runtime move. In this repo, GlitchTip is newer than the old VM/SigNoz stack and is still documented as active on kai-server. If the real goal is "ser8 owns shared o11y, kai-server keeps only app-plane workloads", GlitchTip may remain temporarily without violating the memory goal, but that needs to be made explicit.
Keep bridge pieces only while they still have a named producer. The repo gives clear removal conditions for each bridge, so this can be done surgically.

What is still tracked on kai-server today

I cannot inspect the live cluster from this clone, so this is the tracked desired state plus documented host-side producers, not a live kubectl get all dump.

Tracked in deploy/observability/, scripts/k8s/, .ward/ward.yaml, Makefile, and docs/FEATURES.md:

VictoriaMetrics stack
- vmsingle with a 10Gi PVC on local-path
- vmagent
- prometheus-node-exporter DaemonSet
- ts-vmsingle Tailscale proxy
- vmsingle-nodeport on localhost:30428
SigNoz stack
- ClickHouse with 10Gi PVC
- ZooKeeper with 2Gi PVC
- SigNoz app with 1Gi PVC
- OTel collector
- schema migrator
- ts-signoz Tailscale proxy
- signoz-otlp-nodeport on localhost:30418
GlitchTip stack
- web pod with embedded worker
- Valkey
- CloudNativePG glitchtip-pg cluster with 10Gi storage
- ts-glitchtip Tailscale proxy
Host-side producers wired to kai-server observability endpoints
- scripts/process-memory-heartbeat.py posts OTLP metrics to http://localhost:30428/opentelemetry/v1/metrics and therefore depends on kai-server vmsingle-nodeport
- scripts/agent-health-heartbeat.py posts OTLP metrics to http://localhost:30418/v1/metrics and therefore depends on kai-server signoz-otlp-nodeport
- scripts/thermal-heartbeat.py writes node-exporter textfile metrics, so it depends on node-exporter plus vmagent/VictoriaMetrics if those metrics still matter

Component classification

Based on the repo contents and the issue brief:

Already replaced or intended to be replaced by ser8
- VictoriaMetrics on kai-server
- SigNoz on kai-server
- Their Tailscale access proxies
- Their localhost NodePort bridges
Still producing unique data from kai-server unless repointed or dropped
- process-memory-heartbeat -> unique host memory telemetry, currently only sent to kai-server VM
- agent-health-heartbeat -> unique Ollama/GPU telemetry, currently only sent to kai-server SigNoz
- thermal-heartbeat -> unique thermal telemetry, currently routed through node-exporter/vmagent and also still emits Sentry events
- GlitchTip data itself, if any apps already switched DSNs there
Legacy or stale control surface that would recreate the old stack
- ward exec observability
- ward exec signoz
- scripts/k8s/observability.py
- scripts/k8s/signoz.py
- deploy/observability/README.md sections for VM/SigNoz install/upgrade/uninstall
- docs/FEATURES.md entries describing kai-server VM/SigNoz as shipped features

The biggest decisions still needed

Does GlitchTip remain on kai-server for now, or is it part of the ser8 cutover?
- This repo does not settle that cleanly.
- The issue body points at ser8 as the canonical o11y plane.
- But this repo still documents active GlitchTip-on-kai-server rollout and a phased Sentry-to-GlitchTip DSN migration.
What historical data is worth preserving?
- VictoriaMetrics history
- SigNoz ClickHouse history
- GlitchTip Postgres history, if any real events already landed there
Do we preserve kai-server host telemetry by repointing it to ser8, or do we intentionally drop some of it?
- process-memory-heartbeat and agent-health-heartbeat are easy to repoint if ser8 endpoints exist.
- thermal-heartbeat is more opinionated because it currently mixes metrics with Sentry alerting.
Is coilyco-bridge/deploy#51 about runtime placement, repo ownership, or both?
- This matters because infra should not duplicate the deploy-repo migration plan.

Options

Option A. Memory-first retirement, recommended.
- Verify ser8 coverage for metrics and traces.
- Repoint or retire the three host-side producers.
- Remove kai-server VM/SigNoz workloads and the verbs/manifests/docs that recreate them.
- Leave GlitchTip alone until DSN cutover and deploy ownership are resolved.
- This is the best match for the issue acceptance if the real goal is to kill the multi-GiB observability footprint quickly and safely.
Option B. Full observability namespace retirement now.
- Move or remove GlitchTip too, then delete the namespace.
- Higher coordination cost and higher data-loss risk.
- I would not recommend this unless deploy#51 already defines the GlitchTip landing zone and the DSN cutover status is known.
Option C. Repo-only cleanup first.
- Remove the verbs/docs/manifests that recreate the old stack, but leave live workloads until a later manual cutover.
- Lowest immediate risk, but it does not satisfy the memory objective on its own.

Recommended execution shape

Stage 1. Inventory and prove replacements
- Confirm which live producers still hit kai-server VM/SigNoz/GlitchTip endpoints.
- Confirm ser8 equivalents for VM and SigNoz exist and are reachable from kai-server.
- Decide whether thermal/process-memory/agent-health are being preserved or intentionally dropped.
Stage 2. Preserve or waive history explicitly
- Export or snapshot VM and SigNoz data before teardown, or record a deliberate decision that historical data is being dropped.
- Do the same for GlitchTip only if it is in scope for this cutover.
Stage 3. Cut producers over
- process-memory-heartbeat: remove or repoint away from localhost:30428
- agent-health-heartbeat: remove or repoint away from localhost:30418
- thermal-heartbeat: either keep only the eventing side, or give it a new metrics sink
Stage 4. Remove kai-server VM/SigNoz surfaces
- workloads
- Tailscale proxies
- NodePorts
- ward verbs
- README / FEATURES references
Stage 5. Reassess namespace residue
- If only GlitchTip remains and memory is now small, that is probably acceptable as an interim state.
- If GlitchTip is also meant to move, that should be handled under the deploy#51 plan, not improvised here.

Bridge components and their removal conditions

deploy/observability/signoz-otlp-nodeport-service.yml
- Keep only while agent-health-heartbeat.py still posts to kai-server localhost:30418.
deploy/observability/vmsingle-nodeport-service.yml
- Keep only while process-memory-heartbeat.py still posts to kai-server localhost:30428.
node-exporter and vmagent
- Keep only while thermal-heartbeat metrics still need the VictoriaMetrics path.
ts-signoz and ts-vmsingle
- Remove with the kai-server SigNoz/VM services.
ts-glitchtip
- Keep only if GlitchTip itself remains on kai-server.

Risks and doc drift to call out

Backup gap: docs/restic-backups.md explicitly excludes metrics and traces, so deleting VM/SigNoz PVCs without an export is irreversible.
GlitchTip ambiguity: docs/glitchtip-deploy-plan-status.md says phase 1 is complete on kai-server, but the issue brief frames ser8 as the canonical o11y plane. That conflict needs one explicit decision.
Error-pane ambiguity: deploy/observability/README.md still has a ## Sentry stays put section saying hosted Sentry remains the app-error pane, while GlitchTip docs describe a phased DSN replacement. That means error producers are probably not fully cut over yet.
Control-surface risk: leaving ward exec observability and ward exec signoz in place after live teardown invites accidental re-creation.
Potential cluster dependency: if CloudNativePG was installed only for GlitchTip, GlitchTip retirement may also enable removing CNPG from kai-server, but that should be verified before assuming it is dead weight.

Bottom line

I recommend posting back that the issue should proceed as a two-step retirement:

Now: retire kai-server VictoriaMetrics and SigNoz after repointing or waiving the three host-side producers and preserving or waiving historical VM/SigNoz data.
Later or separately under deploy#51: decide GlitchTip's runtime home and complete the Sentry-to-GlitchTip DSN migration.

That gets the big RAM win, matches the repo evidence, avoids silent data loss, and avoids duplicating the deploy-repo migration design.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- ## Advisory recommendation The repo evidence supports treating this as a **staged kai-server observability retirement**, not a one-shot namespace delete. The safest recommendation is: - **Retire kai-server VictoriaMetrics and SigNoz first.** Those are clearly described here as the old in-cluster observability stack, and they are also the parts most plausibly driving the `2834Mi observability` footprint. The strongest likely RAM culprit is **SigNoz ClickHouse** alone, which is configured at `4Gi` request / `8Gi` limit in `deploy/observability/signoz-values.yml`. - **Do not blindly delete PVC-backed data.** This repo's backup path explicitly says **metrics and traces are out of scope** for restic in `docs/restic-backups.md`, and the backup script only includes `forgejo registry coilysiren-backend ntfy` namespaces plus `/home/kai/saves`. That means historical VictoriaMetrics and SigNoz data are **not covered by the normal offsite backup flow**. GlitchTip also is not named in the backup namespaces. - **Leave GlitchTip as a separate decision gate unless `coilyco-bridge/deploy#51` already settled its runtime move.** In this repo, GlitchTip is newer than the old VM/SigNoz stack and is still documented as active on kai-server. If the real goal is "ser8 owns shared o11y, kai-server keeps only app-plane workloads", GlitchTip may remain temporarily without violating the memory goal, but that needs to be made explicit. - **Keep bridge pieces only while they still have a named producer.** The repo gives clear removal conditions for each bridge, so this can be done surgically. ## What is still tracked on kai-server today I cannot inspect the live cluster from this clone, so this is the **tracked desired state plus documented host-side producers**, not a live `kubectl get all` dump. Tracked in `deploy/observability/`, `scripts/k8s/`, `.ward/ward.yaml`, `Makefile`, and `docs/FEATURES.md`: - **VictoriaMetrics stack** - `vmsingle` with a `10Gi` PVC on `local-path` - `vmagent` - `prometheus-node-exporter` DaemonSet - `ts-vmsingle` Tailscale proxy - `vmsingle-nodeport` on `localhost:30428` - **SigNoz stack** - ClickHouse with `10Gi` PVC - ZooKeeper with `2Gi` PVC - SigNoz app with `1Gi` PVC - OTel collector - schema migrator - `ts-signoz` Tailscale proxy - `signoz-otlp-nodeport` on `localhost:30418` - **GlitchTip stack** - web pod with embedded worker - Valkey - CloudNativePG `glitchtip-pg` cluster with `10Gi` storage - `ts-glitchtip` Tailscale proxy - **Host-side producers wired to kai-server observability endpoints** - `scripts/process-memory-heartbeat.py` posts OTLP metrics to `http://localhost:30428/opentelemetry/v1/metrics` and therefore depends on kai-server `vmsingle-nodeport` - `scripts/agent-health-heartbeat.py` posts OTLP metrics to `http://localhost:30418/v1/metrics` and therefore depends on kai-server `signoz-otlp-nodeport` - `scripts/thermal-heartbeat.py` writes node-exporter textfile metrics, so it depends on `node-exporter` plus `vmagent`/VictoriaMetrics if those metrics still matter ## Component classification Based on the repo contents and the issue brief: - **Already replaced or intended to be replaced by ser8** - VictoriaMetrics on kai-server - SigNoz on kai-server - Their Tailscale access proxies - Their localhost NodePort bridges - **Still producing unique data from kai-server unless repointed or dropped** - `process-memory-heartbeat` -> unique host memory telemetry, currently only sent to kai-server VM - `agent-health-heartbeat` -> unique Ollama/GPU telemetry, currently only sent to kai-server SigNoz - `thermal-heartbeat` -> unique thermal telemetry, currently routed through node-exporter/vmagent and also still emits Sentry events - GlitchTip data itself, if any apps already switched DSNs there - **Legacy or stale control surface that would recreate the old stack** - `ward exec observability` - `ward exec signoz` - `scripts/k8s/observability.py` - `scripts/k8s/signoz.py` - `deploy/observability/README.md` sections for VM/SigNoz install/upgrade/uninstall - `docs/FEATURES.md` entries describing kai-server VM/SigNoz as shipped features ## The biggest decisions still needed - **Does GlitchTip remain on kai-server for now, or is it part of the ser8 cutover?** - This repo does not settle that cleanly. - The issue body points at ser8 as the canonical o11y plane. - But this repo still documents active GlitchTip-on-kai-server rollout and a phased Sentry-to-GlitchTip DSN migration. - **What historical data is worth preserving?** - VictoriaMetrics history - SigNoz ClickHouse history - GlitchTip Postgres history, if any real events already landed there - **Do we preserve kai-server host telemetry by repointing it to ser8, or do we intentionally drop some of it?** - `process-memory-heartbeat` and `agent-health-heartbeat` are easy to repoint if ser8 endpoints exist. - `thermal-heartbeat` is more opinionated because it currently mixes metrics with Sentry alerting. - **Is `coilyco-bridge/deploy#51` about runtime placement, repo ownership, or both?** - This matters because infra should not duplicate the deploy-repo migration plan. ## Options - **Option A. Memory-first retirement, recommended.** - Verify ser8 coverage for metrics and traces. - Repoint or retire the three host-side producers. - Remove kai-server VM/SigNoz workloads and the verbs/manifests/docs that recreate them. - Leave GlitchTip alone until DSN cutover and deploy ownership are resolved. - This is the best match for the issue acceptance if the real goal is to kill the multi-GiB observability footprint quickly and safely. - **Option B. Full `observability` namespace retirement now.** - Move or remove GlitchTip too, then delete the namespace. - Higher coordination cost and higher data-loss risk. - I would not recommend this unless deploy#51 already defines the GlitchTip landing zone and the DSN cutover status is known. - **Option C. Repo-only cleanup first.** - Remove the verbs/docs/manifests that recreate the old stack, but leave live workloads until a later manual cutover. - Lowest immediate risk, but it does not satisfy the memory objective on its own. ## Recommended execution shape - **Stage 1. Inventory and prove replacements** - Confirm which live producers still hit kai-server VM/SigNoz/GlitchTip endpoints. - Confirm ser8 equivalents for VM and SigNoz exist and are reachable from kai-server. - Decide whether thermal/process-memory/agent-health are being preserved or intentionally dropped. - **Stage 2. Preserve or waive history explicitly** - Export or snapshot VM and SigNoz data before teardown, or record a deliberate decision that historical data is being dropped. - Do the same for GlitchTip only if it is in scope for this cutover. - **Stage 3. Cut producers over** - `process-memory-heartbeat`: remove or repoint away from `localhost:30428` - `agent-health-heartbeat`: remove or repoint away from `localhost:30418` - `thermal-heartbeat`: either keep only the eventing side, or give it a new metrics sink - **Stage 4. Remove kai-server VM/SigNoz surfaces** - workloads - Tailscale proxies - NodePorts - ward verbs - README / FEATURES references - **Stage 5. Reassess namespace residue** - If only GlitchTip remains and memory is now small, that is probably acceptable as an interim state. - If GlitchTip is also meant to move, that should be handled under the deploy#51 plan, not improvised here. ## Bridge components and their removal conditions - `deploy/observability/signoz-otlp-nodeport-service.yml` - Keep only while `agent-health-heartbeat.py` still posts to kai-server `localhost:30418`. - `deploy/observability/vmsingle-nodeport-service.yml` - Keep only while `process-memory-heartbeat.py` still posts to kai-server `localhost:30428`. - `node-exporter` and `vmagent` - Keep only while `thermal-heartbeat` metrics still need the VictoriaMetrics path. - `ts-signoz` and `ts-vmsingle` - Remove with the kai-server SigNoz/VM services. - `ts-glitchtip` - Keep only if GlitchTip itself remains on kai-server. ## Risks and doc drift to call out - **Backup gap:** `docs/restic-backups.md` explicitly excludes metrics and traces, so deleting VM/SigNoz PVCs without an export is irreversible. - **GlitchTip ambiguity:** `docs/glitchtip-deploy-plan-status.md` says phase 1 is complete on kai-server, but the issue brief frames ser8 as the canonical o11y plane. That conflict needs one explicit decision. - **Error-pane ambiguity:** `deploy/observability/README.md` still has a `## Sentry stays put` section saying hosted Sentry remains the app-error pane, while GlitchTip docs describe a phased DSN replacement. That means error producers are probably not fully cut over yet. - **Control-surface risk:** leaving `ward exec observability` and `ward exec signoz` in place after live teardown invites accidental re-creation. - **Potential cluster dependency:** if CloudNativePG was installed only for GlitchTip, GlitchTip retirement may also enable removing CNPG from kai-server, but that should be verified before assuming it is dead weight. ## Bottom line I recommend posting back that the issue should proceed as a **two-step retirement**: - **Now:** retire kai-server VictoriaMetrics and SigNoz after repointing or waiving the three host-side producers and preserving or waiving historical VM/SigNoz data. - **Later or separately under deploy#51:** decide GlitchTip's runtime home and complete the Sentry-to-GlitchTip DSN migration. That gets the big RAM win, matches the repo evidence, avoids silent data loss, and avoids duplicating the deploy-repo migration design. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilyco-ops added the

interactive

labels

2026-07-09 21:46:08 +00:00

coilyco-ops added

consult

and removed

interactive

labels

2026-07-10 08:59:51 +00:00

coilyco-ops commented

2026-07-10 12:22:47 +00:00

Author

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

pre-commit installed at .git/hooks/pre-commit
pre-commit installed at .git/hooks/commit-msg

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent