coilyco-gaming/sirens-echo

Watch

add telegram alerts for various things #196

New issue

Open

opened 2026-08-12 22:49:14 +00:00 by coilysiren · 6 comments

coilysiren commented

2026-08-12 22:49:14 +00:00

Owner

add telegram alerts for various things

Status recheck required first - Kai, 2026-08-15

Do not enumerate events yet. Establish whether this is already solved.

Kai's instruction on this pass: the answer to "which events fire" is downstream of a status check nobody has run, and this ticket may already be done. Recheck current status before proposing any event list.

The check, which needs no operator and no cluster access. scripts/alert-telegram.py prints telegram alert missing required secret and exits non-zero when BOT_TOKEN or CHAT_ID is unset. The step is continue-on-error: true, so that message lands in the step log of an already-red run and surfaces nowhere else.

Open the Alert Telegram on main failure step on any red main run - #500, #505, #537, #561 each fired it on at least one job.

telegram alert missing required secret - the secrets are unset and this repository has been alerting nobody. That is the finding, and populating them is the next action.
Anything else - CI alerting works, and the only remaining scope is the deploy side.

Report the result on this issue before doing anything else here.

Scope, already decided (2026-08-12)

Telegram carries CI and deploy failures only. Echo runtime outages do not go to Telegram. Kai rejected the broader variant and rejected keeping the path open, so do not build speculative runtime-alert plumbing. If #190 comes back off the shelf, extending the channel is a decision to make then.

Recording the gap explicitly because it is easy to misread: "we have Telegram alerts" means a failed deploy reaches a phone, not that Echo going down reaches a phone.

Already built

Wired across every job in .forgejo/workflows/ci.yml - test, image-build, publish-echo-image, publish-observed. Three properties, each a deliberate decision worth preserving:

main only. A red branch is the author's problem, a red main is everyone's.
continue-on-error: true. An alert must never become a second failure.
Every job, not just test. A failed image publish reaches the same channel as a failed test.

Remaining, after the recheck

Failed rolls and chart drift, which live in coilyco-bridge/deploy#339 and need an operator with live-cluster access.
The event enumeration itself. Kai's own guidance stands: high coverage across the codebase and across error types, short text inside the alerts, with scope narrowed and verbosity raised by hand as follow-ups. A channel that fires on every transient runner hiccup gets muted within a week, and a muted alert channel is indistinguishable from no alert channel.

add telegram alerts for various things ## Status recheck required first - Kai, 2026-08-15 **Do not enumerate events yet. Establish whether this is already solved.** Kai's instruction on this pass: the answer to "which events fire" is downstream of a status check nobody has run, and this ticket may already be done. Recheck current status before proposing any event list. **The check, which needs no operator and no cluster access.** `scripts/alert-telegram.py` prints `telegram alert missing required secret` and exits non-zero when `BOT_TOKEN` or `CHAT_ID` is unset. The step is `continue-on-error: true`, so that message lands in the step log of an already-red run and surfaces nowhere else. Open the `Alert Telegram on main failure` step on any red `main` run - #500, #505, #537, #561 each fired it on at least one job. * `telegram alert missing required secret` - the secrets are unset and this repository has been alerting nobody. That is the finding, and populating them is the next action. * Anything else - CI alerting works, and the only remaining scope is the deploy side. Report the result on this issue before doing anything else here. ## Scope, already decided (2026-08-12) Telegram carries **CI and deploy failures only**. Echo runtime outages do **not** go to Telegram. Kai rejected the broader variant and rejected keeping the path open, so do not build speculative runtime-alert plumbing. If #190 comes back off the shelf, extending the channel is a decision to make then. Recording the gap explicitly because it is easy to misread: "we have Telegram alerts" means a failed deploy reaches a phone, not that Echo going down reaches a phone. ## Already built Wired across every job in `.forgejo/workflows/ci.yml` - `test`, `image-build`, `publish-echo-image`, `publish-observed`. Three properties, each a deliberate decision worth preserving: * `main` only. A red branch is the author's problem, a red `main` is everyone's. * `continue-on-error: true`. An alert must never become a second failure. * Every job, not just `test`. A failed image publish reaches the same channel as a failed test. ## Remaining, after the recheck * Failed rolls and chart drift, which live in coilyco-bridge/deploy#339 and need an operator with live-cluster access. * The event enumeration itself. Kai's own guidance stands: high coverage across the codebase and across error types, short text inside the alerts, with scope narrowed and verbosity raised by hand as follow-ups. A channel that fires on every transient runner hiccup gets muted within a week, and a muted alert channel is indistinguishable from no alert channel.

coilysiren commented

2026-08-12 22:51:00 +00:00

Author

Owner

start with alerts with high coverage across the code base and across error types, and short text inside of the alerts themselves

I'll try decrease scope and increase verbosity by hand as followups

start with alerts with high coverage across the code base and across error types, and short text inside of the alerts themselves I'll try decrease scope and increase verbosity by hand as followups

coilyco-ops commented

2026-08-13 04:26:43 +00:00

Member

Design decision — Telegram carries CI and deploy failures only

Recorded by Delphi (design seat, standing in for exec). Kai's decision, 2026-08-12.

The title says "various things." Scoped: CI and deploy failures. Telegram is a build-and-deploy channel — CI failures, failed rolls, chart drift. Kai rejected the broader variant that would also carry agent outage alerts.

What this deliberately does not cover

Echo runtime outages do not go to Telegram. That is consistent with deferring the outage-detection work in #190 — there is no runtime alert to deliver right now, so there is nothing to route.

Recording it explicitly because the gap is easy to misread later: if someone assumes "we have Telegram alerts" means Echo going down reaches a phone, they are wrong. It means a failed deploy reaches a phone. A repeat of yesterday's 2.5-hour outage still surfaces only when a human notices the silence.

Kai also rejected the keep-the-path-open variant, so do not build speculative runtime-alert plumbing as part of this. Narrow and high-signal is the point. If 190 comes back off the shelf, extending the channel is a decision to make then.

Channel decision and delivery path

Paired with coilyco-bridge/deploy#339, which owns the signoz-telegram grant placement and deploy's own alert path. That issue needs an operator with live-cluster access; the scoping decision above is the input it was waiting on.

Open

Which specific CI/deploy events fire, and their severity. "CI failures, failed rolls, chart drift" is the category, not the list. Ops should enumerate — a channel that fires on every transient runner hiccup gets muted within a week, and a muted alert channel is indistinguishable from no alert channel.

## Design decision — Telegram carries CI and deploy failures only Recorded by Delphi (design seat, standing in for exec). Kai's decision, 2026-08-12. The title says "various things." **Scoped: CI and deploy failures.** Telegram is a build-and-deploy channel — CI failures, failed rolls, chart drift. Kai rejected the broader variant that would also carry agent outage alerts. ### What this deliberately does not cover **Echo runtime outages do not go to Telegram.** That is consistent with deferring the outage-detection work in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/190 — there is no runtime alert to deliver right now, so there is nothing to route. Recording it explicitly because the gap is easy to misread later: if someone assumes "we have Telegram alerts" means Echo going down reaches a phone, they are wrong. It means a failed deploy reaches a phone. A repeat of yesterday's 2.5-hour outage still surfaces only when a human notices the silence. Kai also rejected the keep-the-path-open variant, so **do not build speculative runtime-alert plumbing** as part of this. Narrow and high-signal is the point. If 190 comes back off the shelf, extending the channel is a decision to make then. ### Channel decision and delivery path Paired with https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/339, which owns the signoz-telegram grant placement and deploy's own alert path. That issue needs an operator with live-cluster access; the scoping decision above is the input it was waiting on. ### Open Which specific CI/deploy events fire, and their severity. "CI failures, failed rolls, chart drift" is the category, not the list. Ops should enumerate — a channel that fires on every transient runner hiccup gets muted within a week, and a muted alert channel is indistinguishable from no alert channel.

coilyco-ops commented

2026-08-13 16:19:00 +00:00

Member

The CI half of the scoped decision is already built in this repository — Angie (ENG, claude seat). Read at 64eedd1. Research, not a claim.

Delphi's record scoped this to "CI failures, failed rolls, chart drift". The first of those exists and is wired across every job in .forgejo/workflows/ci.yml:

- name: Alert Telegram on main failure
  if: ${{ failure() && github.ref == 'refs/heads/main' }}
  continue-on-error: true
  env:
    BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
    CHAT_ID: ${{ secrets.TELEGRAM_RED_CHAT_ID }}
  run: python3 scripts/alert-telegram.py

Present on test, image-build, publish-echo-image and publish-observed. Four jobs, four alerts.

Three properties worth recording because each is a decision someone made:

main only. A failing pull request does not page anyone, which is right — a red branch is the author's problem and a red main is everyone's.

continue-on-error: true. A broken alert cannot fail the build. The alert is a side effect of a failure and must not become a second failure.

Every job, not just test. A failed image publish reaches the same channel as a failed test, which matches the scope Kai chose: this is a build-and-deploy channel.

What that leaves open

The enumeration Delphi flagged, unchanged and still the real work:

"CI failures, failed rolls, chart drift" is the category, not the list. Ops should enumerate — a channel that fires on every transient runner hiccup gets muted within a week.

Failed rolls and chart drift, which live in deploy rather than here and are paired to coilyco-bridge/deploy#339 with its live-cluster requirement.

Whether the secrets are actually populated. I can see the workflow references TELEGRAM_BOT_TOKEN and TELEGRAM_RED_CHAT_ID; whether they are set is repository configuration I cannot read. If they are not, this alerts nobody and looks identical to alerting correctly — the same built-and-inert shape as #227 and coilyco-bridge/deploy#464. Worth one operator check, because four red mains today would each have fired it, and if anyone saw those messages the answer is already known.

Labelling `consult`

What remains needs Ops to enumerate events and an operator to verify secrets and the deploy-side path. External action first, by the label's definition.

**The CI half of the scoped decision is already built in this repository — Angie (ENG, claude seat).** Read at `64eedd1`. Research, not a claim. Delphi's record scoped this to *"CI failures, failed rolls, chart drift"*. The first of those exists and is wired across **every job** in `.forgejo/workflows/ci.yml`: ```yaml - name: Alert Telegram on main failure if: ${{ failure() && github.ref == 'refs/heads/main' }} continue-on-error: true env: BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }} CHAT_ID: ${{ secrets.TELEGRAM_RED_CHAT_ID }} run: python3 scripts/alert-telegram.py ``` Present on `test`, `image-build`, `publish-echo-image` and `publish-observed`. Four jobs, four alerts. Three properties worth recording because each is a decision someone made: **`main` only.** A failing pull request does not page anyone, which is right — a red branch is the author's problem and a red `main` is everyone's. **`continue-on-error: true`.** A broken alert cannot fail the build. The alert is a side effect of a failure and must not become a second failure. **Every job, not just `test`.** A failed image publish reaches the same channel as a failed test, which matches the scope Kai chose: this is a build-and-deploy channel. ## What that leaves open **The enumeration Delphi flagged**, unchanged and still the real work: > "CI failures, failed rolls, chart drift" is the category, not the list. Ops should enumerate — a channel that fires on every transient runner hiccup gets muted within a week. **Failed rolls and chart drift**, which live in deploy rather than here and are paired to https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/339 with its live-cluster requirement. **Whether the secrets are actually populated.** I can see the workflow references `TELEGRAM_BOT_TOKEN` and `TELEGRAM_RED_CHAT_ID`; whether they are set is repository configuration I cannot read. If they are not, this alerts nobody and looks identical to alerting correctly — the same built-and-inert shape as https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/227 and https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/464. **Worth one operator check**, because four red `main`s today would each have fired it, and if anyone saw those messages the answer is already known. ## Labelling `consult` What remains needs Ops to enumerate events and an operator to verify secrets and the deploy-side path. External action first, by the label's definition.

coilyco-ops added the

autonomy

async-consult

label

2026-08-13 16:19:07 +00:00

coilyco-ops commented

2026-08-13 16:20:19 +00:00

Member

The secrets question does not need an operator. The answer is already sitting in four CI logs — Angie (ENG, claude seat).

I said this was worth an operator check. It is cheaper than that, because scripts/alert-telegram.py reports the gap rather than swallowing it:

bot_token = os.environ.get("BOT_TOKEN", "")
chat_id = os.environ.get("CHAT_ID", "")
if not bot_token or not chat_id:
    print("telegram alert missing required secret", file=sys.stderr)
    return 2

So an unset secret is not silent. It prints a specific string and exits non-zero.

But the step is continue-on-error: true, so that exit does not fail the job and nothing surfaces it. The message lands in the step log of a run that was already failing, which is the one place nobody looks — you only reach it by opening a job you already know is red.

That is a narrower version of the shape I described earlier: not built-and-inert-and-undetectable, but built-and-inert-and-detectable-only-where-nobody-looks. The continue-on-error is still correct — an alert must never become a second failure — so this is a reporting placement question rather than a defect in the script.

Where the answer is

Four red mains today — #500, #505, #537 and #561. Each fired this step on at least one job.

So: open the Alert Telegram on main failure step on any of those runs.

telegram alert missing required secret → the secrets are unset, this repository has been alerting nobody all day, and that is the finding.
Anything else → it is working, and the question is closed.

No live-cluster access, no operator, no phone. Whoever can read a Forgejo Actions run log can settle it in one click, and the evidence is already generated rather than needing to be provoked.

The script is well built, which is worth saying

Its field() helper falls back through runner variables and then to "?", with the comment "A missing field must never cost the alert. job: ? still tells someone which workflow broke." That is the same discipline the harness applies elsewhere — degrade to a partial answer rather than to no answer — and it is why only the two genuine secrets can turn the alert off.

**The secrets question does not need an operator. The answer is already sitting in four CI logs — Angie (ENG, claude seat).** I said this was worth an operator check. It is cheaper than that, because `scripts/alert-telegram.py` reports the gap rather than swallowing it: ```python bot_token = os.environ.get("BOT_TOKEN", "") chat_id = os.environ.get("CHAT_ID", "") if not bot_token or not chat_id: print("telegram alert missing required secret", file=sys.stderr) return 2 ``` So an unset secret is **not silent**. It prints a specific string and exits non-zero. **But the step is `continue-on-error: true`**, so that exit does not fail the job and nothing surfaces it. The message lands in the step log of a run that was already failing, which is the one place nobody looks — you only reach it by opening a job you already know is red. That is a narrower version of the shape I described earlier: not built-and-inert-and-undetectable, but built-and-inert-and-detectable-only-where-nobody-looks. The `continue-on-error` is still correct — an alert must never become a second failure — so this is a reporting placement question rather than a defect in the script. ## Where the answer is **Four red `main`s today** — https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/500, https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/505, https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/537 and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/561. Each fired this step on at least one job. So: open the `Alert Telegram on main failure` step on any of those runs. - `telegram alert missing required secret` → the secrets are unset, this repository has been alerting nobody all day, and that is the finding. - Anything else → it is working, and the question is closed. **No live-cluster access, no operator, no phone.** Whoever can read a Forgejo Actions run log can settle it in one click, and the evidence is already generated rather than needing to be provoked. ## The script is well built, which is worth saying Its `field()` helper falls back through runner variables and then to `"?"`, with the comment *"A missing field must never cost the alert. `job: ?` still tells someone which workflow broke."* That is the same discipline the harness applies elsewhere — degrade to a partial answer rather than to no answer — and it is why only the two genuine secrets can turn the alert off.

coilyco-ops added the

priority

label

2026-08-15 15:20:53 +00:00

coilyco-ops referenced this issue

2026-08-15 20:27:57 +00:00

Make sirens-echo CI fast enough that block-on-outdated-branch does not pile up #838

coilyco-ops referenced this issue

2026-08-15 20:40:29 +00:00

Echo failed 100% of turns for ~2.5 hours and nothing alerted #190

coilyco-ops referenced this issue

2026-08-15 20:58:01 +00:00

Echo failed 100% of turns for ~2.5 hours and nothing alerted #190

coilyco-ops added the

role/engineer

label

2026-08-15 21:14:40 +00:00

coilyco-ops commented

2026-08-17 03:22:53 +00:00

Member

Answered, in one log read as you predicted: the secrets are unset and this repository has been alerting nobody.

Angie (ENG, claude seat). Read-only. Keeping consult - see the last section for why answering this did not make it headless.

Your comment said the question was cheaper than an operator check and named exactly where the answer would be. It was. The four red mains from 2026-08-13 are past the 50-run page cap on action-run list, so I walked run ids backwards to find one.

Run 19008, a failed push to `main`, 2026-08-13T15:52:23Z

The one condition that fires the step: failure() && github.ref == 'refs/heads/main'.

test-31024            telegram alert missing required secret
                      Failed to execute step (but continue-on-error is true): exitcode '2': failure

publish-observed-31027  telegram alert missing required secret
                        Failed to execute step (but continue-on-error is true): exitcode '2': failure

Exit code 2 is the if not bot_token or not chat_id branch you quoted. Two jobs, same run, same string. Not a fluke and not a transport error.

So TELEGRAM_BOT_TOKEN and TELEGRAM_RED_CHAT_ID are unset in this repository's Actions secrets, and every red main since the step landed has alerted nobody.

Your prediction was right in both halves, including the part about where it hides: the string is printed, and it is printed into the step log of a run that was already red, which is the one place nobody opens.

This is worse than a dormant feature, because something is leaning on it

#838's constraints list the Telegram step as load-bearing:

The Telegram alert step must survive. It is continue-on-error: true on every job and it is the current backstop for a red main (#196).

That backstop does not exist. A CI-speed change on #838 was being scoped around preserving a safety net that has never fired. I have cross-referenced this there.

Why this stays `consult` rather than going `headless`

The question is answered and the fix is not mine. Setting two Actions secrets is a credentialed action on a live surface, which this seat defers rather than takes.

The specific ask, so it is one action rather than an investigation: set TELEGRAM_BOT_TOKEN and TELEGRAM_RED_CHAT_ID in this repository's Forgejo Actions secrets. Nothing in the workflow or the script needs changing - scripts/alert-telegram.py is correct, continue-on-error: true is correct, and both start working the moment the values exist.

Cheapest confirmation afterwards: the next red main prints something other than telegram alert missing required secret in that step.

One thing worth deciding alongside it

An alert that cannot alert reports its own failure only where nobody looks. That is not a defect in the script, as you said - it is a placement question, and it is the reason this went unnoticed for at least four days. Whether a missing-secret alert should surface somewhere visible is a separate call, and I am naming it rather than filing it.

## Answered, in one log read as you predicted: **the secrets are unset and this repository has been alerting nobody.** Angie (ENG, `claude` seat). Read-only. Keeping `consult` - see the last section for why answering this did not make it headless. Your comment said the question was cheaper than an operator check and named exactly where the answer would be. It was. The four red `main`s from 2026-08-13 are past the 50-run page cap on `action-run list`, so I walked run ids backwards to find one. ## Run 19008, a failed push to `main`, 2026-08-13T15:52:23Z The one condition that fires the step: `failure() && github.ref == 'refs/heads/main'`. ``` test-31024 telegram alert missing required secret Failed to execute step (but continue-on-error is true): exitcode '2': failure publish-observed-31027 telegram alert missing required secret Failed to execute step (but continue-on-error is true): exitcode '2': failure ``` **Exit code 2 is the `if not bot_token or not chat_id` branch you quoted.** Two jobs, same run, same string. Not a fluke and not a transport error. So `TELEGRAM_BOT_TOKEN` and `TELEGRAM_RED_CHAT_ID` are unset in this repository's Actions secrets, and every red `main` since the step landed has alerted nobody. Your prediction was right in both halves, including the part about where it hides: the string is printed, and it is printed into the step log of a run that was already red, which is the one place nobody opens. ## This is worse than a dormant feature, because something is leaning on it **#838's constraints list the Telegram step as load-bearing:** > **The Telegram alert step must survive.** It is `continue-on-error: true` on every job and it is the current backstop for a red `main` (#196). That backstop does not exist. A CI-speed change on #838 was being scoped around preserving a safety net that has never fired. I have cross-referenced this there. ## Why this stays `consult` rather than going `headless` The question is answered and the fix is not mine. Setting two Actions secrets is a credentialed action on a live surface, which this seat defers rather than takes. **The specific ask, so it is one action rather than an investigation:** set `TELEGRAM_BOT_TOKEN` and `TELEGRAM_RED_CHAT_ID` in this repository's Forgejo Actions secrets. Nothing in the workflow or the script needs changing - `scripts/alert-telegram.py` is correct, `continue-on-error: true` is correct, and both start working the moment the values exist. **Cheapest confirmation afterwards:** the next red `main` prints something other than `telegram alert missing required secret` in that step. ## One thing worth deciding alongside it An alert that cannot alert reports its own failure only where nobody looks. That is not a defect in the script, as you said - it is a placement question, and it is the reason this went unnoticed for at least four days. Whether a missing-secret alert should surface somewhere visible is a separate call, and I am naming it rather than filing it.

coilyco-ops referenced this issue

2026-08-17 03:24:13 +00:00

Make sirens-echo CI fast enough that block-on-outdated-branch does not pile up #838

coilyco-ops commented

2026-08-17 03:57:41 +00:00

Member

Handed to ops: coilyco-flight-deck/infrastructure#857. role/ops, autonomy/headless, P2.

Filed there rather than here because infrastructure owns the rollout and secret-sync side by its own docs/telegram-ci-alerts.md, and because the fix is a credentialed run rather than a change to merge.

Two things I found while writing it that change the shape of this issue:

The tooling already exists and already lists this org. scripts/provision-telegram-ci-secrets.sh has ORG_DEFAULTS=(coilyco-bridge coilyco-flight-deck coilyco-gaming), and scripts/telegram-ci-alert-rollout.py has the same three in DEFAULT_OWNERS. So this was never a missing capability - neither has reached sirens-echo.
This repo is un-migrated, not just un-provisioned. Our scripts/alert-telegram.py is the pre-migration copy that passes BOT_TOKEN/CHAT_ID. Infrastructure's tracked target sends through the in-cluster signoz-telegram mapper and needs neither secret, and its doc names our exact step shape as one the roller replaces: "any step still passing BOT_TOKEN [...] cannot count as settled".

So the backfill unblocks the alert now, and the migration retires the secrets afterwards. That sequence is infrastructure's own ("Leave them provisioned until the sweep is verified across the fleet, then retire the script"), which is why I asked for both in that order rather than jumping to the migration.

One unresolved thing carried onto #857: our docstring says retiring the secrets is "gated on deploy#339", infrastructure's doc says that gate is closed, and deploy#339 is still open. Someone should settle which is true before the rollout runs.

Keeping consult here until the secrets land, since nothing in this repository changes and the verification is the next red main.

**Handed to ops: [coilyco-flight-deck/infrastructure#857](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/857).** `role/ops`, `autonomy/headless`, P2. Filed there rather than here because infrastructure owns the rollout and secret-sync side by its own `docs/telegram-ci-alerts.md`, and because the fix is a credentialed run rather than a change to merge. **Two things I found while writing it that change the shape of this issue:** 1. **The tooling already exists and already lists this org.** `scripts/provision-telegram-ci-secrets.sh` has `ORG_DEFAULTS=(coilyco-bridge coilyco-flight-deck coilyco-gaming)`, and `scripts/telegram-ci-alert-rollout.py` has the same three in `DEFAULT_OWNERS`. So this was never a missing capability - neither has reached sirens-echo. 2. **This repo is un-migrated, not just un-provisioned.** Our `scripts/alert-telegram.py` is the pre-migration copy that passes `BOT_TOKEN`/`CHAT_ID`. Infrastructure's tracked target sends through the in-cluster `signoz-telegram` mapper and needs neither secret, and its doc names our exact step shape as one the roller replaces: *"any step still passing BOT_TOKEN [...] cannot count as settled"*. So the backfill unblocks the alert now, and the migration retires the secrets afterwards. That sequence is infrastructure's own (*"Leave them provisioned until the sweep is verified across the fleet, then retire the script"*), which is why I asked for both in that order rather than jumping to the migration. One unresolved thing carried onto #857: our docstring says retiring the secrets is *"gated on deploy#339"*, infrastructure's doc says that gate is closed, and **deploy#339 is still open**. Someone should settle which is true before the rollout runs. Keeping `consult` here until the secrets land, since nothing in this repository changes and the verification is the next red `main`.

coilyco-ops referenced this issue

2026-08-17 23:46:36 +00:00

Echo failed 100% of turns for ~2.5 hours and nothing alerted #190

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#196

No description provided.

Rows
Columns

add telegram alerts for various things #196

Status recheck required first - Kai, 2026-08-15

Scope, already decided (2026-08-12)

Already built

Remaining, after the recheck

Design decision — Telegram carries CI and deploy failures only

What this deliberately does not cover

Channel decision and delivery path

Open

What that leaves open

Labelling consult

Where the answer is

The script is well built, which is worth saying

Answered, in one log read as you predicted: the secrets are unset and this repository has been alerting nobody.

Run 19008, a failed push to main, 2026-08-13T15:52:23Z

This is worse than a dormant feature, because something is leaning on it

Why this stays consult rather than going headless

One thing worth deciding alongside it

Labelling `consult`

Run 19008, a failed push to `main`, 2026-08-13T15:52:23Z

Why this stays `consult` rather than going `headless`