Tear down ntfy, consolidate fleet alerting into Telegram (coilyco_ops_bot, green/red chats)

coilysiren commented

2026-06-18 00:20:43 +00:00

Owner

Tear down self-hosted ntfy and consolidate all fleet alerting into Telegram via the existing coilyco_ops_bot. Replaces the ntfy server + the xenrox/ntfy-alertmanager bridge + the direct-curl producers.

Routing model: two chats

green (/coilysiren/telegram/green-chat-id) — good news: successful deploys, fleet-converge applied-a-commit, clean git-state sweeps.
red (/coilysiren/telegram/red-chat-id) — failures/alerts: every vmalert rule, converge FAILED, git drift/unreachable, deploy job failures.
Bot token: /coilysiren/telegram/bot-token (SecureString). Bot: coilyco_ops_bot (local: coilyco-ops).
Every producer already computes a high/warning-vs-min/ok priority — that existing branch becomes the green/red chat selector. The cluster Alertmanager only emits problems, so it routes wholesale to red.

Inventory (what posts to ntfy today)

Cluster path (deploy repo, services/ser8-observability/): vmalert → VictoriaMetrics bundled Alertmanager → xenrox/ntfy-alertmanager bridge → ntfy topic alerts. AM is upstream-Prometheus-compatible, so it speaks telegram_configs natively — drop the bridge, swap the ntfy receiver for a Telegram red receiver.

Direct-curl scripts (this repo):

ansible/roles/git-state-watch/files/fleet-git-state.sh — daily git drift sweep. clean→green, drift/unreachable→red.
ansible/roles/fleet-converge/files/fleet-converge.sh notify() — every 30 min. applied→green, FAILED→red.
scripts/coilysiren-mirror-reconcile.sh — already DEAD (its mirror-sync.yml never sets NTFY_URL); migrate for completeness.

Plumbing to retire: deploy/ntfy.yml (binwiederhier server, Flux + tailscale sidecar), ansible/roles/o11y-ntfy/ (SSM base-url cache role), SSM /coilysiren/ntfy/{base-url,ts-authkey}, the flux kustomization + tailscale service entries, ~/.config/fleet-o11y/ntfy-base-url.

Stages (dual-ship so no alert gap)

Bootstrap (operator/Kai): token + green/red chat ids into SSM via coilyco_ops_bot. Done out-of-band.
Add Telegram alongside ntfy: new o11y-telegram ansible role caching the 3 SSM params host-local (mirrors o11y-ntfy); swap the 3 scripts' ntfy curl for Telegram sendMessage with green/red selection; add the Telegram red receiver to deploy vmalert-values.yml. Both fire in parallel; confirm Telegram lands.
Tear down ntfy: delete the bridge + ntfy.yml + o11y-ntfy role + flux/tailscale/SSM entries; update all docs/FEATURES (infra + deploy).

Open detail

Alertmanager telegram_configs needs bot_token/chat_id — keep them out of tracked files. Either bot_token_file + out-of-band config Secret (same precedent as today's imperative ntfy-bridge Secret), or keep a thin Telegram webhook relay fed by an ExternalSecret. Decide at cluster-path authoring time.

Out of scope

Discord #coilyco-alerting — its only producer is the retired coily binary (coily repo, not here); deferred, tracked separately.

Decisions locked: two chats (green/red); Discord deferred. Bot coilyco_ops_bot pre-exists.

Tear down self-hosted ntfy and consolidate all fleet alerting into Telegram via the existing `coilyco_ops_bot`. Replaces the ntfy server + the `xenrox/ntfy-alertmanager` bridge + the direct-curl producers. ## Routing model: two chats - **green** (`/coilysiren/telegram/green-chat-id`) — good news: successful deploys, `fleet-converge` applied-a-commit, clean git-state sweeps. - **red** (`/coilysiren/telegram/red-chat-id`) — failures/alerts: every vmalert rule, converge FAILED, git drift/unreachable, deploy job failures. - Bot token: `/coilysiren/telegram/bot-token` (SecureString). Bot: `coilyco_ops_bot` (local: coilyco-ops). - Every producer already computes a high/warning-vs-min/ok priority — that existing branch becomes the green/red chat selector. The cluster Alertmanager only emits problems, so it routes wholesale to **red**. ## Inventory (what posts to ntfy today) **Cluster path (deploy repo, `services/ser8-observability/`):** vmalert → VictoriaMetrics bundled Alertmanager → `xenrox/ntfy-alertmanager` bridge → ntfy topic `alerts`. AM is upstream-Prometheus-compatible, so it speaks `telegram_configs` natively — drop the bridge, swap the `ntfy` receiver for a Telegram red receiver. **Direct-curl scripts (this repo):** - `ansible/roles/git-state-watch/files/fleet-git-state.sh` — daily git drift sweep. clean→green, drift/unreachable→red. - `ansible/roles/fleet-converge/files/fleet-converge.sh` `notify()` — every 30 min. applied→green, FAILED→red. - `scripts/coilysiren-mirror-reconcile.sh` — already DEAD (its `mirror-sync.yml` never sets `NTFY_URL`); migrate for completeness. **Plumbing to retire:** `deploy/ntfy.yml` (binwiederhier server, Flux + tailscale sidecar), `ansible/roles/o11y-ntfy/` (SSM base-url cache role), SSM `/coilysiren/ntfy/{base-url,ts-authkey}`, the flux kustomization + tailscale service entries, `~/.config/fleet-o11y/ntfy-base-url`. ## Stages (dual-ship so no alert gap) 1. **Bootstrap (operator/Kai):** token + green/red chat ids into SSM via `coilyco_ops_bot`. Done out-of-band. 2. **Add Telegram alongside ntfy:** new `o11y-telegram` ansible role caching the 3 SSM params host-local (mirrors o11y-ntfy); swap the 3 scripts' ntfy curl for Telegram `sendMessage` with green/red selection; add the Telegram red receiver to `deploy` `vmalert-values.yml`. Both fire in parallel; confirm Telegram lands. 3. **Tear down ntfy:** delete the bridge + `ntfy.yml` + `o11y-ntfy` role + flux/tailscale/SSM entries; update all docs/FEATURES (infra + deploy). ## Open detail Alertmanager `telegram_configs` needs `bot_token`/`chat_id` — keep them out of tracked files. Either `bot_token_file` + out-of-band config Secret (same precedent as today's imperative ntfy-bridge Secret), or keep a thin Telegram webhook relay fed by an ExternalSecret. Decide at cluster-path authoring time. ## Out of scope Discord `#coilyco-alerting` — its only producer is the retired `coily` binary (coily repo, not here); deferred, tracked separately. Decisions locked: two chats (green/red); Discord deferred. Bot `coilyco_ops_bot` pre-exists.

coilysiren referenced this issue from a commit

2026-06-18 00:46:33 +00:00

feat(o11y): dual-ship alertmanager to telegram red channel alongside ntfy

coilysiren commented

2026-06-18 01:06:20 +00:00

Author

Owner

Additional teardown item folded in (found during the cluster red-receiver apply):

Dead grafana-admin ExternalSecret on ser8 — infrastructure/deploy/observability/admin-password-externalsecret.yml syncs /grafana/admin-password into the ser8 observability namespace, but:

ser8 runs no Grafana (per ser8-observability docs/ser8-k3s.md: "No Grafana: vmui for ad-hoc queries").
ser8 has no external-secrets operator (the ExternalSecret CRD is not installed — confirmed when the telegram-token ExternalSecret apply failed with no matches for kind "ExternalSecret" in version "external-secrets.io/v1"), so it could never have applied anyway.

Action during the sweep: delete admin-password-externalsecret.yml, and (if unused) retire the /grafana/admin-password SSM param. Same root cause as the ntfy plumbing — ser8 secrets are imperative, not SSM-synced.

Additional teardown item folded in (found during the cluster red-receiver apply): **Dead `grafana-admin` ExternalSecret on ser8** — `infrastructure/deploy/observability/admin-password-externalsecret.yml` syncs `/grafana/admin-password` into the ser8 `observability` namespace, but: - ser8 runs **no Grafana** (per `ser8-observability` `docs/ser8-k3s.md`: "No Grafana: vmui for ad-hoc queries"). - ser8 has **no external-secrets operator** (the ExternalSecret CRD is not installed — confirmed when the telegram-token ExternalSecret apply failed with `no matches for kind "ExternalSecret" in version "external-secrets.io/v1"`), so it could never have applied anyway. Action during the sweep: delete `admin-password-externalsecret.yml`, and (if unused) retire the `/grafana/admin-password` SSM param. Same root cause as the ntfy plumbing — ser8 secrets are imperative, not SSM-synced.

coilysiren referenced this issue from a commit

2026-06-18 01:15:33 +00:00

feat(o11y): dual-ship fleet notifications to Telegram green/red channels

coilysiren referenced this issue

2026-06-18 01:33:14 +00:00

ward exec sandbox fails on ser8 (apparmor_restrict_unprivileged_userns) — fleet-converge silently failing #370

coilysiren referenced this issue from a commit

2026-06-18 03:43:10 +00:00

feat(o11y): de-provision ntfy, Telegram is the sole notification channel

coilysiren added the

labels

2026-06-18 06:19:38 +00:00

coilyco-ops referenced this issue

2026-06-24 04:46:02 +00:00

Full-mesh reachability healthcheck (N×N connectivity matrix) across the fleet #255

coilyco-ops referenced this issue

2026-06-24 04:51:36 +00:00

Full-mesh reachability healthcheck (N×N connectivity matrix) across the fleet #255

coilysiren referenced this issue from a commit

2026-06-24 06:28:29 +00:00

fix(fleet-reachability): default ser8 Telegram chat-id SSM path to red-chat-id

coilyco-ops removed the

autonomy

async-consult

label

2026-07-16 01:08:18 +00:00

coilyco-ops commented

2026-07-23 04:36:22 +00:00

Owner

Live cluster follow-up from 2026-07-22: ntfy/ntfy-ts-authkey remains Ready=False because /coilysiren/ntfy/ts-authkey is absent. The teardown should remove this ExternalSecret and its dependent Tailscale wiring instead of recreating a credential for the retiring service. Infrastructure #500 also tracks deletion of the live ntfy namespace.

Live cluster follow-up from 2026-07-22: `ntfy/ntfy-ts-authkey` remains `Ready=False` because `/coilysiren/ntfy/ts-authkey` is absent. The teardown should remove this ExternalSecret and its dependent Tailscale wiring instead of recreating a credential for the retiring service. Infrastructure #500 also tracks deletion of the live ntfy namespace.

coilyco-ops added

and removed

labels

2026-07-29 03:45:49 +00:00

coilyco-ops added the

role/sysadmin

label

2026-08-25 08:02:35 +00:00

coilyco-ops commented

2026-08-27 03:09:28 +00:00

Owner

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901).

Verified as already landed: Closing: the teardown landed. Commit 2675b2de3b de-provisioned ntfy, deploy/ntfy.yml and ansible/roles/o11y-ntfy are gone along with the dead grafana admin-password ExternalSecret, and ansible/roles/o11y-telegram now records Telegram green/red as the sole o11y notification channel.

This issue was open only because nothing closed it when the work shipped. If the verification is wrong, reopen it. The whole set is recoverable with state:closed label:burndown-2026-08.

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901). Verified as already landed: Closing: the teardown landed. Commit 2675b2de3b de-provisioned ntfy, deploy/ntfy.yml and ansible/roles/o11y-ntfy are gone along with the dead grafana admin-password ExternalSecret, and ansible/roles/o11y-telegram now records Telegram green/red as the sole o11y notification channel. This issue was open only because nothing closed it when the work shipped. If the verification is wrong, reopen it. The whole set is recoverable with `state:closed label:burndown-2026-08`.

coilyco-ops

2026-08-27 03:09:30 +00:00

closed this issue
added the
burndown-2026-08
label

Rows
Columns

Tear down ntfy, consolidate fleet alerting into Telegram (coilyco_ops_bot, green/red chats) #369

Routing model: two chats

Inventory (what posts to ntfy today)

Stages (dual-ship so no alert gap)

Open detail

Out of scope