Backfill the Telegram CI-alert secrets across the fleet: sirens-echo's red-main alert has never fired #857

New issue

Open

opened 2026-08-17 03:57:08 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-17 03:57:08 +00:00

Member

Filed by Angie (ENG, claude seat) from a live finding on coilyco-gaming/sirens-echo#196. Everything below is read-only evidence; the fix is an ops action and I have not taken any of it.

The finding

sirens-echo's red-main Telegram alert has never fired, because TELEGRAM_BOT_TOKEN and TELEGRAM_RED_CHAT_ID are unset.

Proof, from run 19008 - a failed push to main on 2026-08-13, which is the one condition the step fires on (failure() && github.ref == 'refs/heads/main'):

test-31024              telegram alert missing required secret
publish-observed-31027  telegram alert missing required secret
                        Failed to execute step (but continue-on-error is true): exitcode '2': failure

Exit code 2 is alert-telegram.py's if not bot_token or not chat_id branch. Two jobs, one run, same string.

It fails invisibly by construction. The step is continue-on-error: true, correctly, so the only trace is a line in the step log of a run that was already red - the one place nobody opens.

This repo already owns both fixes, and neither has been run against `coilyco-gaming`

scripts/provision-telegram-ci-secrets.sh already syncs /coilysiren/telegram/bot-token and /coilysiren/telegram/red-chat-id into org Actions secrets, and its ORG_DEFAULTS already includes coilyco-gaming:

ORG_DEFAULTS=(coilyco-bridge coilyco-flight-deck coilyco-gaming)

scripts/telegram-ci-alert-rollout.py has the same three in DEFAULT_OWNERS and discovers repos through /repos/search, so sirens-echo is in scope for both. Neither appears to have reached it.

So this is not a build. It is a run.

The ask

Run scripts/provision-telegram-ci-secrets.sh across all three orgs - the default. --dry-run first. This is the backfill and it unblocks every repo still on the token-passing shape in one action, because the secrets are org-level and inherited.
Verify. The next red main in any repo prints something other than telegram alert missing required secret in that step. No code changes anywhere.

Then the migration, which retires the thing being backfilled

Worth doing in this order rather than instead, because docs/telegram-ci-alerts.md already specifies it:

scripts/provision-telegram-ci-secrets.sh syncs [...] The migration retires the need for both. Leave them provisioned until the sweep is verified across the fleet, then retire the script.

sirens-echo is un-migrated. Its scripts/alert-telegram.py is the pre-migration copy and says so in its own docstring:

A workflow passes only what the runner cannot supply: BOT_TOKEN and CHAT_ID. [...] Retiring the two secrets needs the Ward /api/create_alert path, which is gated on deploy#339.

Against this repo's tracked scripts/actions/alert-telegram.py, which is the target:

The mapper holds the Telegram identity, so no workflow and no repository needs BOT_TOKEN or CHAT_ID.

sirens-echo's step still passes both, and docs/telegram-ci-alerts.md names that exact shape as one the roller replaces rather than leaves alone: "any step still passing BOT_TOKEN. The second is the migration itself, so a token-passing step cannot count as settled."

One thing I could not reconcile, flagging rather than guessing. This repo's doc says the mapper gate is closed - "Both clusters now serve it under the same cluster-local name [...] and the gate is closed" - but coilyco-bridge/deploy#339 is still open. Either the issue is stale or the doc is ahead of it. Worth settling before the rollout runs, since the rollout depends on which is true.

Scope I could and could not check

Repos I have on disk and grepped for TELEGRAM_BOT_TOKEN / alert-telegram:

Carry it - coilyco-gaming/sirens-echo, coilyco-bridge/agentic-os-hardware, coilyco-bridge/agentic-os-kai, coilyco-bridge/agentic-os-xxx, coilyco-bridge/deploy, coilyco-flight-deck/agentic-os, coilyco-flight-deck/infrastructure
Do not - coilysiren/coilysiren, coilysiren/lore, coilysiren/voice-corpus

Only sirens-echo is confirmed broken, because that is the only one where I found a red main to read. The others may already be fine - an org secret set for coilyco-bridge and coilyco-flight-deck would cover them, and I cannot read org secrets to check. The backfill is safe either way, since it is idempotent.

coilysiren/* is a user, not an org, so org-level secrets do not reach it and it is outside both scripts' ORG_DEFAULTS. None of the three I could inspect use the alert, so this is a note rather than a gap.

Why it matters beyond one repo

sirens-echo#838 is scoping CI-speed work around an explicit constraint:

The Telegram alert step must survive. It is the current backstop for a red main (#196).

That backstop has never caught anything. A safety net nobody has tested is worse than a known gap, because work gets planned around it.

Related - coilyco-gaming/sirens-echo#196 (where this was found), coilyco-gaming/sirens-echo#838 (the constraint that leans on it), coilyco-bridge/deploy#339 (the migration gate).

Filed by Angie (ENG, `claude` seat) from a live finding on `coilyco-gaming/sirens-echo#196`. Everything below is read-only evidence; the fix is an ops action and I have not taken any of it. ## The finding **`sirens-echo`'s red-`main` Telegram alert has never fired, because `TELEGRAM_BOT_TOKEN` and `TELEGRAM_RED_CHAT_ID` are unset.** Proof, from run `19008` - a failed push to `main` on 2026-08-13, which is the one condition the step fires on (`failure() && github.ref == 'refs/heads/main'`): ``` test-31024 telegram alert missing required secret publish-observed-31027 telegram alert missing required secret Failed to execute step (but continue-on-error is true): exitcode '2': failure ``` Exit code 2 is `alert-telegram.py`'s `if not bot_token or not chat_id` branch. Two jobs, one run, same string. **It fails invisibly by construction.** The step is `continue-on-error: true`, correctly, so the only trace is a line in the step log of a run that was already red - the one place nobody opens. ## This repo already owns both fixes, and neither has been run against `coilyco-gaming` `scripts/provision-telegram-ci-secrets.sh` already syncs `/coilysiren/telegram/bot-token` and `/coilysiren/telegram/red-chat-id` into org Actions secrets, and its `ORG_DEFAULTS` **already includes `coilyco-gaming`**: ```bash ORG_DEFAULTS=(coilyco-bridge coilyco-flight-deck coilyco-gaming) ``` `scripts/telegram-ci-alert-rollout.py` has the same three in `DEFAULT_OWNERS` and discovers repos through `/repos/search`, so `sirens-echo` is in scope for both. Neither appears to have reached it. **So this is not a build. It is a run.** ## The ask 1. **Run `scripts/provision-telegram-ci-secrets.sh`** across all three orgs - the default. `--dry-run` first. This is the backfill and it unblocks every repo still on the token-passing shape in one action, because the secrets are org-level and inherited. 2. **Verify.** The next red `main` in any repo prints something other than `telegram alert missing required secret` in that step. No code changes anywhere. ## Then the migration, which retires the thing being backfilled Worth doing in this order rather than instead, because `docs/telegram-ci-alerts.md` already specifies it: > `scripts/provision-telegram-ci-secrets.sh` syncs [...] **The migration retires the need for both. Leave them provisioned until the sweep is verified across the fleet, then retire the script.** **`sirens-echo` is un-migrated.** Its `scripts/alert-telegram.py` is the pre-migration copy and says so in its own docstring: > A workflow passes only what the runner cannot supply: BOT_TOKEN and CHAT_ID. [...] **Retiring the two secrets needs the Ward `/api/create_alert` path, which is gated on deploy#339.** Against this repo's tracked `scripts/actions/alert-telegram.py`, which is the target: > The mapper holds the Telegram identity, so **no workflow and no repository needs BOT_TOKEN or CHAT_ID.** `sirens-echo`'s step still passes both, and `docs/telegram-ci-alerts.md` names that exact shape as one the roller replaces rather than leaves alone: *"any step still passing `BOT_TOKEN`. The second is the migration itself, so a token-passing step cannot count as settled."* **One thing I could not reconcile, flagging rather than guessing.** This repo's doc says the mapper gate is closed - *"Both clusters now serve it under the same cluster-local name [...] and the gate is closed"* - but **`coilyco-bridge/deploy#339` is still open**. Either the issue is stale or the doc is ahead of it. Worth settling before the rollout runs, since the rollout depends on which is true. ## Scope I could and could not check Repos I have on disk and grepped for `TELEGRAM_BOT_TOKEN` / `alert-telegram`: * **Carry it** - `coilyco-gaming/sirens-echo`, `coilyco-bridge/agentic-os-hardware`, `coilyco-bridge/agentic-os-kai`, `coilyco-bridge/agentic-os-xxx`, `coilyco-bridge/deploy`, `coilyco-flight-deck/agentic-os`, `coilyco-flight-deck/infrastructure` * **Do not** - `coilysiren/coilysiren`, `coilysiren/lore`, `coilysiren/voice-corpus` **Only `sirens-echo` is confirmed broken**, because that is the only one where I found a red `main` to read. The others may already be fine - an org secret set for `coilyco-bridge` and `coilyco-flight-deck` would cover them, and I cannot read org secrets to check. **The backfill is safe either way, since it is idempotent.** **`coilysiren/*` is a user, not an org**, so org-level secrets do not reach it and it is outside both scripts' `ORG_DEFAULTS`. None of the three I could inspect use the alert, so this is a note rather than a gap. ## Why it matters beyond one repo `sirens-echo#838` is scoping CI-speed work around an explicit constraint: > **The Telegram alert step must survive.** It is the current backstop for a red `main` (#196). That backstop has never caught anything. A safety net nobody has tested is worse than a known gap, because work gets planned around it. **Related** - `coilyco-gaming/sirens-echo#196` (where this was found), `coilyco-gaming/sirens-echo#838` (the constraint that leans on it), `coilyco-bridge/deploy#339` (the migration gate).

coilyco-ops added the

labels

2026-08-17 03:57:17 +00:00

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-17 03:57:41 +00:00

add telegram alerts for various things #196