Tear down ntfy, consolidate fleet alerting into Telegram (coilyco_ops_bot, green/red chats) #369

New issue

Open

opened 2026-06-18 00:20:43 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-06-18 00:20:43 +00:00

Owner

Tear down self-hosted ntfy and consolidate all fleet alerting into Telegram via the existing coilyco_ops_bot. Replaces the ntfy server + the xenrox/ntfy-alertmanager bridge + the direct-curl producers.

Routing model: two chats

green (/coilysiren/telegram/green-chat-id) — good news: successful deploys, fleet-converge applied-a-commit, clean git-state sweeps.
red (/coilysiren/telegram/red-chat-id) — failures/alerts: every vmalert rule, converge FAILED, git drift/unreachable, deploy job failures.
Bot token: /coilysiren/telegram/bot-token (SecureString). Bot: coilyco_ops_bot (local: coilyco-ops).
Every producer already computes a high/warning-vs-min/ok priority — that existing branch becomes the green/red chat selector. The cluster Alertmanager only emits problems, so it routes wholesale to red.

Inventory (what posts to ntfy today)

Cluster path (deploy repo, services/ser8-observability/): vmalert → VictoriaMetrics bundled Alertmanager → xenrox/ntfy-alertmanager bridge → ntfy topic alerts. AM is upstream-Prometheus-compatible, so it speaks telegram_configs natively — drop the bridge, swap the ntfy receiver for a Telegram red receiver.

Direct-curl scripts (this repo):

ansible/roles/git-state-watch/files/fleet-git-state.sh — daily git drift sweep. clean→green, drift/unreachable→red.
ansible/roles/fleet-converge/files/fleet-converge.sh notify() — every 30 min. applied→green, FAILED→red.
scripts/coilysiren-mirror-reconcile.sh — already DEAD (its mirror-sync.yml never sets NTFY_URL); migrate for completeness.

Plumbing to retire: deploy/ntfy.yml (binwiederhier server, Flux + tailscale sidecar), ansible/roles/o11y-ntfy/ (SSM base-url cache role), SSM /coilysiren/ntfy/{base-url,ts-authkey}, the flux kustomization + tailscale service entries, ~/.config/fleet-o11y/ntfy-base-url.

Stages (dual-ship so no alert gap)

Bootstrap (operator/Kai): token + green/red chat ids into SSM via coilyco_ops_bot. Done out-of-band.
Add Telegram alongside ntfy: new o11y-telegram ansible role caching the 3 SSM params host-local (mirrors o11y-ntfy); swap the 3 scripts' ntfy curl for Telegram sendMessage with green/red selection; add the Telegram red receiver to deploy vmalert-values.yml. Both fire in parallel; confirm Telegram lands.
Tear down ntfy: delete the bridge + ntfy.yml + o11y-ntfy role + flux/tailscale/SSM entries; update all docs/FEATURES (infra + deploy).

Open detail

Alertmanager telegram_configs needs bot_token/chat_id — keep them out of tracked files. Either bot_token_file + out-of-band config Secret (same precedent as today's imperative ntfy-bridge Secret), or keep a thin Telegram webhook relay fed by an ExternalSecret. Decide at cluster-path authoring time.

Out of scope

Discord #coilyco-alerting — its only producer is the retired coily binary (coily repo, not here); deferred, tracked separately.

Decisions locked: two chats (green/red); Discord deferred. Bot coilyco_ops_bot pre-exists.

Tear down self-hosted ntfy and consolidate all fleet alerting into Telegram via the existing `coilyco_ops_bot`. Replaces the ntfy server + the `xenrox/ntfy-alertmanager` bridge + the direct-curl producers. ## Routing model: two chats - **green** (`/coilysiren/telegram/green-chat-id`) — good news: successful deploys, `fleet-converge` applied-a-commit, clean git-state sweeps. - **red** (`/coilysiren/telegram/red-chat-id`) — failures/alerts: every vmalert rule, converge FAILED, git drift/unreachable, deploy job failures. - Bot token: `/coilysiren/telegram/bot-token` (SecureString). Bot: `coilyco_ops_bot` (local: coilyco-ops). - Every producer already computes a high/warning-vs-min/ok priority — that existing branch becomes the green/red chat selector. The cluster Alertmanager only emits problems, so it routes wholesale to **red**. ## Inventory (what posts to ntfy today) **Cluster path (deploy repo, `services/ser8-observability/`):** vmalert → VictoriaMetrics bundled Alertmanager → `xenrox/ntfy-alertmanager` bridge → ntfy topic `alerts`. AM is upstream-Prometheus-compatible, so it speaks `telegram_configs` natively — drop the bridge, swap the `ntfy` receiver for a Telegram red receiver. **Direct-curl scripts (this repo):** - `ansible/roles/git-state-watch/files/fleet-git-state.sh` — daily git drift sweep. clean→green, drift/unreachable→red. - `ansible/roles/fleet-converge/files/fleet-converge.sh` `notify()` — every 30 min. applied→green, FAILED→red. - `scripts/coilysiren-mirror-reconcile.sh` — already DEAD (its `mirror-sync.yml` never sets `NTFY_URL`); migrate for completeness. **Plumbing to retire:** `deploy/ntfy.yml` (binwiederhier server, Flux + tailscale sidecar), `ansible/roles/o11y-ntfy/` (SSM base-url cache role), SSM `/coilysiren/ntfy/{base-url,ts-authkey}`, the flux kustomization + tailscale service entries, `~/.config/fleet-o11y/ntfy-base-url`. ## Stages (dual-ship so no alert gap) 1. **Bootstrap (operator/Kai):** token + green/red chat ids into SSM via `coilyco_ops_bot`. Done out-of-band. 2. **Add Telegram alongside ntfy:** new `o11y-telegram` ansible role caching the 3 SSM params host-local (mirrors o11y-ntfy); swap the 3 scripts' ntfy curl for Telegram `sendMessage` with green/red selection; add the Telegram red receiver to `deploy` `vmalert-values.yml`. Both fire in parallel; confirm Telegram lands. 3. **Tear down ntfy:** delete the bridge + `ntfy.yml` + `o11y-ntfy` role + flux/tailscale/SSM entries; update all docs/FEATURES (infra + deploy). ## Open detail Alertmanager `telegram_configs` needs `bot_token`/`chat_id` — keep them out of tracked files. Either `bot_token_file` + out-of-band config Secret (same precedent as today's imperative ntfy-bridge Secret), or keep a thin Telegram webhook relay fed by an ExternalSecret. Decide at cluster-path authoring time. ## Out of scope Discord `#coilyco-alerting` — its only producer is the retired `coily` binary (coily repo, not here); deferred, tracked separately. Decisions locked: two chats (green/red); Discord deferred. Bot `coilyco_ops_bot` pre-exists.

coilysiren referenced this issue from a commit

2026-06-18 00:46:33 +00:00

feat(o11y): dual-ship alertmanager to telegram red channel alongside ntfy

coilysiren commented

2026-06-18 01:06:20 +00:00

Author

Owner

Additional teardown item folded in (found during the cluster red-receiver apply):

Dead grafana-admin ExternalSecret on ser8 — infrastructure/deploy/observability/admin-password-externalsecret.yml syncs /grafana/admin-password into the ser8 observability namespace, but:

ser8 runs no Grafana (per ser8-observability docs/ser8-k3s.md: "No Grafana: vmui for ad-hoc queries").
ser8 has no external-secrets operator (the ExternalSecret CRD is not installed — confirmed when the telegram-token ExternalSecret apply failed with no matches for kind "ExternalSecret" in version "external-secrets.io/v1"), so it could never have applied anyway.

Action during the sweep: delete admin-password-externalsecret.yml, and (if unused) retire the /grafana/admin-password SSM param. Same root cause as the ntfy plumbing — ser8 secrets are imperative, not SSM-synced.

Additional teardown item folded in (found during the cluster red-receiver apply): **Dead `grafana-admin` ExternalSecret on ser8** — `infrastructure/deploy/observability/admin-password-externalsecret.yml` syncs `/grafana/admin-password` into the ser8 `observability` namespace, but: - ser8 runs **no Grafana** (per `ser8-observability` `docs/ser8-k3s.md`: "No Grafana: vmui for ad-hoc queries"). - ser8 has **no external-secrets operator** (the ExternalSecret CRD is not installed — confirmed when the telegram-token ExternalSecret apply failed with `no matches for kind "ExternalSecret" in version "external-secrets.io/v1"`), so it could never have applied anyway. Action during the sweep: delete `admin-password-externalsecret.yml`, and (if unused) retire the `/grafana/admin-password` SSM param. Same root cause as the ntfy plumbing — ser8 secrets are imperative, not SSM-synced.

coilysiren referenced this issue from a commit

2026-06-18 01:15:33 +00:00

feat(o11y): dual-ship fleet notifications to Telegram green/red channels

coilysiren referenced this issue

2026-06-18 01:33:14 +00:00

ward exec sandbox fails on ser8 (apparmor_restrict_unprivileged_userns) — fleet-converge silently failing #370

coilysiren referenced this issue from a commit

2026-06-18 03:43:10 +00:00

feat(o11y): de-provision ntfy, Telegram is the sole notification channel

coilysiren added the

consult

labels