[interactive] Roll the Telegram alert action back out to consumer repos #768

New issue

Open

opened 2026-08-08 17:33:30 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-08 17:33:30 +00:00

Owner

Goal

Move consumer repos off the inline Telegram payload and onto the agentic-os action, then re-roll the fleet.

Analysis and evidence: coilyco-flight-deck/agentic-os#975. This issue is the rollout leg. AOS authors the action and the contract, this repo owns the fan-out, per the standing authoring-vs-rollout split.

Background

#522 inlined the alert on 2026-07-10 after a fleet-wide CI outage. The root cause was the bare ref form:

uses: coilysiren/agentic-os/actions/telegram-alert@main

The Forgejo runner resolved that against github.com, where the mirror was stale at 4dbb587 and its actions/ tree had no telegram-alert. Inlining was the correct emergency fix.

Both reasons that made it fail are now gone:

The GitHub mirror has caught up. coilysiren/agentic-os/actions/telegram-alert exists today, verified against the GitHub API.
AOS documents a fully-qualified form that never resolves through the mirror at all:

uses: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/telegram-alert@main

That fully-qualified form has never been exercised anywhere in the fleet, so it is an untested contract. Treat it as unproven until a real run says otherwise.

Current cost

41 workflow files carry 47 copies of a ~1.9KB single-line python3 -c payload with escaped newlines. It cannot be meaningfully reviewed, and any change to the alert requires a fleet re-roll.

scripts/telegram-ci-alert-rollout.py also restates the payload, both secret names, and the ref policy, none of which it sources from the published
actions/telegram-alert/defaults.json contract that exists for exactly that purpose.

Work

Canary first, by hand. Pick one low-traffic consumer (coilyco-gaming/galaxy-gen or coilyco-flight-deck/ward-mcp). Replace one alert step with the fully-qualified uses: form. Push and watch a real run.
Prove resolution. A fresh Forgejo Actions run must reach the real job steps rather than failing during action resolution.
Prove delivery. Force a failure on main in the canary and confirm a Telegram message actually arrives. Resolution succeeding is not delivery succeeding, and the runner egress path (FORGEJO_EGRESS_PROXY, nested DinD) is the part most likely to differ between the inline payload and the action.
Then change the rollout. telegram-ci-alert-rollout.py emits the uses: form and sources the URL, ref policy, and secret names from defaults.json rather than restating them. Update test_telegram_ci_alert_rollout.py, including replacement of the now-stale inline block.
Re-roll the fleet with dry_run=1 reviewed before apply.
Reconcile the docs. This repo currently states "Consumer repos no longer depend on an external action for this alert" while AOS states consumers use the fully qualified URL. One policy survives.

Open question to settle before step 4

AOS inlines its own alerts so a failure before checkout can still alert. That rationale clearly holds for its local ./actions/ refs, which need the checkout to exist. It is unclear whether it holds for a remote URL ref, which the runner fetches independently. Settle this before deciding whether any inline call site should deliberately survive, and record the answer next to the code.

Rollback

The current inline rollout is the rollback path. If the canary fails resolution or delivery, revert the canary workflow and leave the rollout script untouched. No fleet-wide change happens until the canary passes both checks, so a bad outcome is scoped to one repo.

Acceptance

Canary repo shows a green Forgejo Actions run reaching real job steps with the uses: form.
Canary repo shows a delivered Telegram alert from a deliberately failed main run.
telegram-ci-alert-rollout.py emits the uses: form and reads defaults.json.
Tests cover replacing the inline block with the action block.
No workflow in the fleet still carries the inline python3 -c payload, except call sites that carry a written reason.
docs/telegram-ci-alerts.md here and in AOS state the same policy.

Why this is operator work

Action resolution and runner egress are live behaviors that cannot be proven from a checkout, and this exact change has taken the fleet red once. Engineer and QA runs stop at the evidence boundary.

Related: #522, #521, coilyco-flight-deck/agentic-os#975, coilyco-flight-deck/agentic-os#632.

## Goal Move consumer repos off the inline Telegram payload and onto the `agentic-os` action, then re-roll the fleet. Analysis and evidence: `coilyco-flight-deck/agentic-os#975`. This issue is the rollout leg. AOS authors the action and the contract, this repo owns the fan-out, per the standing authoring-vs-rollout split. ## Background `#522` inlined the alert on 2026-07-10 after a fleet-wide CI outage. The root cause was the **bare** ref form: ```yaml uses: coilysiren/agentic-os/actions/telegram-alert@main ``` The Forgejo runner resolved that against `github.com`, where the mirror was stale at `4dbb587` and its `actions/` tree had no `telegram-alert`. Inlining was the correct emergency fix. Both reasons that made it fail are now gone: 1. The GitHub mirror has caught up. `coilysiren/agentic-os/actions/telegram-alert` exists today, verified against the GitHub API. 2. AOS documents a fully-qualified form that never resolves through the mirror at all: ```yaml uses: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/telegram-alert@main ``` That fully-qualified form has never been exercised anywhere in the fleet, so it is an untested contract. Treat it as unproven until a real run says otherwise. ## Current cost 41 workflow files carry 47 copies of a ~1.9KB single-line `python3 -c` payload with escaped newlines. It cannot be meaningfully reviewed, and any change to the alert requires a fleet re-roll. `scripts/telegram-ci-alert-rollout.py` also restates the payload, both secret names, and the ref policy, none of which it sources from the published [`actions/telegram-alert/defaults.json`](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/src/branch/main/actions/telegram-alert/defaults.json) contract that exists for exactly that purpose. ## Work 1. **Canary first, by hand.** Pick one low-traffic consumer (`coilyco-gaming/galaxy-gen` or `coilyco-flight-deck/ward-mcp`). Replace one alert step with the fully-qualified `uses:` form. Push and watch a real run. 2. **Prove resolution.** A fresh Forgejo Actions run must reach the real job steps rather than failing during action resolution. 3. **Prove delivery.** Force a failure on `main` in the canary and confirm a Telegram message actually arrives. Resolution succeeding is not delivery succeeding, and the runner egress path (`FORGEJO_EGRESS_PROXY`, nested DinD) is the part most likely to differ between the inline payload and the action. 4. **Then change the rollout.** `telegram-ci-alert-rollout.py` emits the `uses:` form and sources the URL, ref policy, and secret names from `defaults.json` rather than restating them. Update `test_telegram_ci_alert_rollout.py`, including replacement of the now-stale inline block. 5. **Re-roll the fleet** with `dry_run=1` reviewed before apply. 6. **Reconcile the docs.** This repo currently states "Consumer repos no longer depend on an external action for this alert" while AOS states consumers use the fully qualified URL. One policy survives. ## Open question to settle before step 4 AOS inlines its own alerts so a failure before checkout can still alert. That rationale clearly holds for its local `./actions/` refs, which need the checkout to exist. It is unclear whether it holds for a remote URL ref, which the runner fetches independently. Settle this before deciding whether any inline call site should deliberately survive, and record the answer next to the code. ## Rollback The current inline rollout is the rollback path. If the canary fails resolution or delivery, revert the canary workflow and leave the rollout script untouched. No fleet-wide change happens until the canary passes both checks, so a bad outcome is scoped to one repo. ## Acceptance - Canary repo shows a green Forgejo Actions run reaching real job steps with the `uses:` form. - Canary repo shows a delivered Telegram alert from a deliberately failed `main` run. - `telegram-ci-alert-rollout.py` emits the `uses:` form and reads `defaults.json`. - Tests cover replacing the inline block with the action block. - No workflow in the fleet still carries the inline `python3 -c` payload, except call sites that carry a written reason. - `docs/telegram-ci-alerts.md` here and in AOS state the same policy. ## Why this is operator work Action resolution and runner egress are live behaviors that cannot be proven from a checkout, and this exact change has taken the fleet red once. Engineer and QA runs stop at the evidence boundary. Related: #522, #521, `coilyco-flight-deck/agentic-os#975`, `coilyco-flight-deck/agentic-os#632`.

coilyco-ops added the

autonomy

async-consult

label

2026-08-08 17:33:35 +00:00

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:15 +00:00

coilyco-ops added this to the Observability and alerting milestone

2026-08-29 06:09:22 +00:00