feat(ci): converge CI failure alerts on the signoz-telegram mapper #852

Merged

coilysiren merged 1 commit from aos/claude/wt57-alerts-to-mapper into main

2026-08-16 05:55:40 +00:00

coilyco-ops commented

2026-08-16 05:49:57 +00:00

Owner

Reshapes the CI alert to the requested three lines and moves its transport off
per-repo Telegram credentials onto the in-cluster signoz-telegram mapper.
This repo is the pattern the fleet sweep rolls from.

The alert

coilyco-flight-deck/infrastructure CI failing
workflow: lint
run: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/actions/runs/311

The run link was built from GITHUB_SERVER_URL, which is the cluster-local
name the runner registered against and is unreachable from a phone. It now
comes from the forge ROOT_URL.

deploy#339 is closed by facts on the ground

That issue gated retiring BOT_TOKEN and CHAT_ID on mapper reachability: the
mapper was ClusterIP-only on ser8 while deploy's exact-repository runner stays
on kai-server. Both clusters now serve it under the same cluster-local name, so
a job reaches its own cluster's instance. The mapper holds the Telegram
identity in pod environment, confirmed against
services/signoz-telegram/signoz-telegram.mcp.kdl, so the rolled step passes
nothing at all.

Why the roller needed changing first

It could not have carried this migration:

Purely additive, with detection keyed on the bot-token secret. Dropping that
secret from a repo would have made it add a second, old-shape step. A
token-passing step is now replaced rather than treated as settled.
It shipped the alert program only when it also added a step, so a repo with a
settled step calling a stale body had nothing to reconcile the two. The
program now ships on its own.
Detection keyed on the install path. This repo authors the program and calls
it from scripts/actions/, so it would have missed its own step and rolled a
duplicate into the repo the file came from. Detection is by filename now, and
the authoring repo takes no install copy.

Verification

Mutator is a no-op against the three migrated workflows here.
Scoped dry-run migrates them from remote main, with no install copy added.
Rollout tests pass, including new migration and post-migration idempotence
cases. pre-commit run --all-files clean.

Three workflows that never alerted (runner routing, ser8 general, ser8 canary)
gain coverage when the roller next runs.

Not yet proven: nothing has posted through the mapper. Transport and
payload match the spec, but end-to-end delivery is unproven until a real main
failure fires.

Follow-on

The fleet sweep (ward exec telegram-ci-alert-rollout) runs after this merges,
so it rolls the new program rather than the old one. The org Actions secrets
stay provisioned until that is verified, then provision-telegram-ci-secrets.sh
retires.

Reshapes the CI alert to the requested three lines and moves its transport off per-repo Telegram credentials onto the in-cluster `signoz-telegram` mapper. This repo is the pattern the fleet sweep rolls from. ## The alert ``` coilyco-flight-deck/infrastructure CI failing workflow: lint run: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/actions/runs/311 ``` The run link was built from `GITHUB_SERVER_URL`, which is the cluster-local name the runner registered against and is unreachable from a phone. It now comes from the forge `ROOT_URL`. ## deploy#339 is closed by facts on the ground That issue gated retiring `BOT_TOKEN` and `CHAT_ID` on mapper reachability: the mapper was ClusterIP-only on ser8 while deploy's exact-repository runner stays on kai-server. Both clusters now serve it under the same cluster-local name, so a job reaches its own cluster's instance. The mapper holds the Telegram identity in pod environment, confirmed against `services/signoz-telegram/signoz-telegram.mcp.kdl`, so the rolled step passes nothing at all. ## Why the roller needed changing first It could not have carried this migration: * Purely additive, with detection keyed on the bot-token secret. Dropping that secret from a repo would have made it add a second, old-shape step. A token-passing step is now replaced rather than treated as settled. * It shipped the alert program only when it also added a step, so a repo with a settled step calling a stale body had nothing to reconcile the two. The program now ships on its own. * Detection keyed on the install path. This repo authors the program and calls it from `scripts/actions/`, so it would have missed its own step and rolled a duplicate into the repo the file came from. Detection is by filename now, and the authoring repo takes no install copy. ## Verification * Mutator is a no-op against the three migrated workflows here. * Scoped dry-run migrates them from remote main, with no install copy added. * Rollout tests pass, including new migration and post-migration idempotence cases. `pre-commit run --all-files` clean. Three workflows that never alerted (runner routing, ser8 general, ser8 canary) gain coverage when the roller next runs. **Not yet proven:** nothing has posted through the mapper. Transport and payload match the spec, but end-to-end delivery is unproven until a real main failure fires. ## Follow-on The fleet sweep (`ward exec telegram-ci-alert-rollout`) runs after this merges, so it rolls the new program rather than the old one. The org Actions secrets stay provisioned until that is verified, then `provision-telegram-ci-secrets.sh` retires.

coilyco-ops added 1 commit

2026-08-16 05:49:58 +00:00

feat(ci): converge CI failure alerts on the signoz-telegram mapper

TruffleHog / Scan for secrets (pull_request) Successful in 6s

Details

CI / lint (pull_request) Successful in 44s

Details

1d633a23cc

Reshape the alert to three lines and move its transport off per-repo Telegram
credentials onto the in-cluster mapper, then make the roller able to carry that
change across the fleet.

The alert is now `<org/repo> <CI|CD> failing`, the workflow name, and a run
link. The link was built from GITHUB_SERVER_URL, which is the cluster-local
name the runner registered against and is unreachable from a phone, so it now
comes from the forge ROOT_URL.

deploy#339 gated retiring BOT_TOKEN and CHAT_ID on mapper reachability: it was
ClusterIP-only on ser8 while deploy's exact-repository runner stays on
kai-server. Both clusters now serve it under the same cluster-local name, so
the gate is closed. The mapper holds the Telegram identity in pod environment,
confirmed against services/signoz-telegram/signoz-telegram.mcp.kdl, so the
rolled step passes nothing at all.

The roller could not have performed this migration:

* It was purely additive and keyed detection on the bot-token secret, so
  dropping that secret from a repo would have made it add a second, old-shape
  step. A token-passing step is now replaced rather than treated as settled.
* It shipped the alert program only when it also added a step, so a repo with a
  settled step calling a stale body had nothing to reconcile the two. The
  program now ships on its own.
* Detection keyed on the install path. This repo authors the program and calls
  it from scripts/actions/, so it would have missed its own step and rolled a
  duplicate into the repo the file came from. Detection is by filename now, and
  the authoring repo takes no install copy.

Verified: the mutator is a no-op against the three migrated workflows here, and
a scoped dry-run migrates them from remote main with no install copy added.
Three workflows that never alerted (runner routing, ser8 general, ser8 canary)
gain coverage when the roller next runs.

End-to-end delivery is unproven until a real main failure fires. The transport
and payload match the mapper spec, but nothing has posted through it yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-16 05:49:58 +00:00

coilyco-ops referenced this pull request

2026-08-16 05:51:43 +00:00

Telegram alert sweep leaves an orphaned program copy in repos that used a different path #853

coilyco-ops referenced this pull request

2026-08-16 05:53:10 +00:00

Update the script the pings on main branch fails #850