Add a prep-safe Telegram CI alert renderer and reliability contract #632

New issue

Open

opened 2026-07-23 08:25:53 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-23 08:25:53 +00:00

Member

Outcome

AOS ships one canonical helper for Telegram CI failure alerts. The helper renders and verifies a self-contained workflow step, and the same source owns the send and retry behavior. Consumer repositories stop hand-copying or hand-maintaining the Python payload.

Working name: render-telegram-ci-alerts.

Evidence and constraints

Ward is the reference implementation. Its inline run step reaches Telegram without resolving another action.
AOS #411 and #416 plus infrastructure #521 and #522 proved that local and remote uses: forms are unsafe here. Forgejo resolves the action before checkout and may read the stale GitHub mirror.
Four independent failures on 2026-07-23 reached the alert step and then lost delivery to one TLS handshake timeout: Ward run 2387 attempt 2, AOS run 1581, Agent Proxy run 19 attempt 2, and Lunch Money run 53.
Infrastructure #519 already owns a fleet mutator and org-secret provisioner. Its mutator currently duplicates the entire transport implementation, so behavior can drift away from AOS.
Infrastructure #546 remains the live egress incident. Retries improve transient delivery but do not claim to repair a sustained runner-network outage.

Ownership boundary

AOS owns the public-safe renderer, checker, transport contract, tests, and documentation.
Infrastructure owns managed-repository discovery, cloning, commit and push orchestration, explicit opt-outs, and Forgejo secret convergence.
Consumer repositories own only the generated inline block and any explicit repo-local opt-out.
The AOS helper must not know Coilyco owner names, SSM paths, chat ids, Forgejo credentials, or fleet branch policy.
The AOS helper must not clone, commit, push, or mutate any file that the caller did not name.

CLI contract

The package exposes one entrypoint with three modes:

render-telegram-ci-alerts check <workflow...> - exits 0 when every eligible job carries the canonical block, 1 for drift, and 2 for an unsupported or ambiguous workflow shape.
render-telegram-ci-alerts write <workflow...> - rewrites only named workflow files, preserves unrelated bytes and comments, replaces legacy inline and local or remote action blocks, and is idempotent.
render-telegram-ci-alerts send --dry-run - exercises message construction without network access. The real send path reads only the documented environment contract.

check is the safe default when the mode is omitted. write never performs git operations. Infrastructure calls write, then retains its existing repo-specific validation and landing workflow.

Rendered workflow contract

The alert is an inline run step. It contains no uses: reference and needs no checkout.
The step runs only after an earlier step failed and only for refs/heads/main.
The step sets continue-on-error: true, so delivery failure never replaces or obscures the primary CI result.
Inputs remain TELEGRAM_BOT_TOKEN and TELEGRAM_RED_CHAT_ID. Context includes repository, workflow, job, ref, commit SHA, and run URL.
The helper appends one canonical block to every ordinary job with steps: and replaces any existing block by its stable step name.
Jobs implemented as reusable workflows have no step surface. The helper reports them as unsupported unless the workflow carries a documented explicit opt-out or a supported terminal notification job. The helper never silently calls such a workflow covered.
The renderer preserves matrix definitions, job conditions, dependencies, comments, and unrelated step order.

Delivery and retry contract

One alert gets at most three attempts.
Each attempt has a 15-second network timeout. Backoff is 2 seconds then 5 seconds, keeping the total wall-clock budget below 60 seconds.
The sender retries DNS failures, connection failures, resets, timeouts, TLS handshake failures, HTTP 429, and HTTP 5xx responses.
The sender does not retry missing inputs, malformed configuration, Telegram rejection, or permanent HTTP 4xx responses other than 429.
A successful HTTP exchange counts only when the Telegram response reports success.
Diagnostics name the attempt and failure class but never print the bot token, chat id, request URL containing the token, or encoded payload.
Final exhaustion emits one concise warning and returns nonzero to the step. continue-on-error keeps that secondary result non-blocking.

Rollout contract

The AOS implementation adds the renderer, checker, sender, console entrypoint, focused tests, and updated feature documentation.
AOS removes the unusable remote-action example from docs/telegram-ci-alerts.md. The implementation removes actions/telegram-alert after every workflow reference is gone, or marks it deprecated until that cleanup lands.
Infrastructure replaces its local alert template and mutation parser with the released AOS helper. Infrastructure keeps repository discovery, temporary clones, per-repo commit and push, and verification.
Infrastructure converges the two secrets at the correct owner layer for every managed org and for managed personal-owner repositories. Secret values never enter argv, logs, tracked files, or tests.
Infrastructure reruns the rollout and produces a complete classification: updated, current, explicitly opted out, or unsupported. No repository disappears from the report.
An ops-capable run performs one controlled live failure probe after rollout and records delivery evidence. Sealed engineer or QA runs stop at repository and deterministic transport tests.

Acceptance

Deterministic transport tests cover first-attempt success, transient failure then success, retry exhaustion, HTTP 429 or 5xx retry, permanent 4xx no-retry, missing secrets, malformed Telegram response, and redaction.
Golden renderer tests cover multi-job workflows, matrices, comments, an existing canonical block, both legacy remote refs, local action refs, reusable-workflow jobs, jobs without steps, and idempotent second render.
Tests never call live Telegram.
AOS workflows and all managed consumer workflows contain no Telegram alert uses: reference after rollout.
Infrastructure contains no second copy of the message builder, retry loop, or alert-step template.
Ward retains its current main-only alert semantics and gains the bounded retry behavior.
The fleet report accounts for every active managed repository across the configured owners.
Live delivery verification is recorded separately from repository acceptance, and infrastructure #546 remains open until sustained runner egress is proven healthy.

AOS #411 and #416
Infrastructure #519, #521, #522, and #546
Forgejo action guidance: https://forgejo.org/docs/latest/user/actions/actions/

## Outcome AOS ships one canonical helper for Telegram CI failure alerts. The helper renders and verifies a self-contained workflow step, and the same source owns the send and retry behavior. Consumer repositories stop hand-copying or hand-maintaining the Python payload. Working name: `render-telegram-ci-alerts`. ## Evidence and constraints * Ward is the reference implementation. Its inline `run` step reaches Telegram without resolving another action. * AOS #411 and #416 plus infrastructure #521 and #522 proved that local and remote `uses:` forms are unsafe here. Forgejo resolves the action before checkout and may read the stale GitHub mirror. * Four independent failures on 2026-07-23 reached the alert step and then lost delivery to one TLS handshake timeout: Ward run 2387 attempt 2, AOS run 1581, Agent Proxy run 19 attempt 2, and Lunch Money run 53. * Infrastructure #519 already owns a fleet mutator and org-secret provisioner. Its mutator currently duplicates the entire transport implementation, so behavior can drift away from AOS. * Infrastructure #546 remains the live egress incident. Retries improve transient delivery but do not claim to repair a sustained runner-network outage. ## Ownership boundary * AOS owns the public-safe renderer, checker, transport contract, tests, and documentation. * Infrastructure owns managed-repository discovery, cloning, commit and push orchestration, explicit opt-outs, and Forgejo secret convergence. * Consumer repositories own only the generated inline block and any explicit repo-local opt-out. * The AOS helper must not know Coilyco owner names, SSM paths, chat ids, Forgejo credentials, or fleet branch policy. * The AOS helper must not clone, commit, push, or mutate any file that the caller did not name. ## CLI contract The package exposes one entrypoint with three modes: * `render-telegram-ci-alerts check <workflow...>` - exits 0 when every eligible job carries the canonical block, 1 for drift, and 2 for an unsupported or ambiguous workflow shape. * `render-telegram-ci-alerts write <workflow...>` - rewrites only named workflow files, preserves unrelated bytes and comments, replaces legacy inline and local or remote action blocks, and is idempotent. * `render-telegram-ci-alerts send --dry-run` - exercises message construction without network access. The real send path reads only the documented environment contract. `check` is the safe default when the mode is omitted. `write` never performs git operations. Infrastructure calls `write`, then retains its existing repo-specific validation and landing workflow. ## Rendered workflow contract * The alert is an inline `run` step. It contains no `uses:` reference and needs no checkout. * The step runs only after an earlier step failed and only for `refs/heads/main`. * The step sets `continue-on-error: true`, so delivery failure never replaces or obscures the primary CI result. * Inputs remain `TELEGRAM_BOT_TOKEN` and `TELEGRAM_RED_CHAT_ID`. Context includes repository, workflow, job, ref, commit SHA, and run URL. * The helper appends one canonical block to every ordinary job with `steps:` and replaces any existing block by its stable step name. * Jobs implemented as reusable workflows have no step surface. The helper reports them as unsupported unless the workflow carries a documented explicit opt-out or a supported terminal notification job. The helper never silently calls such a workflow covered. * The renderer preserves matrix definitions, job conditions, dependencies, comments, and unrelated step order. ## Delivery and retry contract * One alert gets at most three attempts. * Each attempt has a 15-second network timeout. Backoff is 2 seconds then 5 seconds, keeping the total wall-clock budget below 60 seconds. * The sender retries DNS failures, connection failures, resets, timeouts, TLS handshake failures, HTTP 429, and HTTP 5xx responses. * The sender does not retry missing inputs, malformed configuration, Telegram rejection, or permanent HTTP 4xx responses other than 429. * A successful HTTP exchange counts only when the Telegram response reports success. * Diagnostics name the attempt and failure class but never print the bot token, chat id, request URL containing the token, or encoded payload. * Final exhaustion emits one concise warning and returns nonzero to the step. `continue-on-error` keeps that secondary result non-blocking. ## Rollout contract 1. The AOS implementation adds the renderer, checker, sender, console entrypoint, focused tests, and updated feature documentation. 2. AOS removes the unusable remote-action example from `docs/telegram-ci-alerts.md`. The implementation removes `actions/telegram-alert` after every workflow reference is gone, or marks it deprecated until that cleanup lands. 3. Infrastructure replaces its local alert template and mutation parser with the released AOS helper. Infrastructure keeps repository discovery, temporary clones, per-repo commit and push, and verification. 4. Infrastructure converges the two secrets at the correct owner layer for every managed org and for managed personal-owner repositories. Secret values never enter argv, logs, tracked files, or tests. 5. Infrastructure reruns the rollout and produces a complete classification: updated, current, explicitly opted out, or unsupported. No repository disappears from the report. 6. An ops-capable run performs one controlled live failure probe after rollout and records delivery evidence. Sealed engineer or QA runs stop at repository and deterministic transport tests. ## Acceptance * Deterministic transport tests cover first-attempt success, transient failure then success, retry exhaustion, HTTP 429 or 5xx retry, permanent 4xx no-retry, missing secrets, malformed Telegram response, and redaction. * Golden renderer tests cover multi-job workflows, matrices, comments, an existing canonical block, both legacy remote refs, local action refs, reusable-workflow jobs, jobs without steps, and idempotent second render. * Tests never call live Telegram. * AOS workflows and all managed consumer workflows contain no Telegram alert `uses:` reference after rollout. * Infrastructure contains no second copy of the message builder, retry loop, or alert-step template. * Ward retains its current main-only alert semantics and gains the bounded retry behavior. * The fleet report accounts for every active managed repository across the configured owners. * Live delivery verification is recorded separately from repository acceptance, and infrastructure #546 remains open until sustained runner egress is proven healthy. ## Related * AOS #411 and #416 * Infrastructure #519, #521, #522, and #546 * Forgejo action guidance: https://forgejo.org/docs/latest/user/actions/actions/

coilyco-ops commented

2026-07-23 08:26:19 +00:00

Author

Member

WARDED_WORKFLOW: needs-redispatch

The director filed this cross-repo alert-reliability work and attempted to dispatch it with Ward, Agent Proxy, Lunch Money, and Factory Game as explicit extra repositories. The host dispatch broker then refused the launch with broker-unreachable and instructed this surface to exit so the director heartbeat can recycle it. No engineer was launched for this issue in this attempt.

WARDED_WORKFLOW: needs-redispatch The director filed this cross-repo alert-reliability work and attempted to dispatch it with Ward, Agent Proxy, Lunch Money, and Factory Game as explicit extra repositories. The host dispatch broker then refused the launch with `broker-unreachable` and instructed this surface to exit so the director heartbeat can recycle it. No engineer was launched for this issue in this attempt.

coilyco-ops changed title from ~~Make CI failure alerts tolerate transient TLS outages~~ to Add a prep-safe Telegram CI alert renderer and reliability contract

2026-07-24 00:12:06 +00:00

coilyco-ops referenced this issue

2026-07-27 03:17:43 +00:00

Interactive: restore AOS CLI release after Telegram alert rollout #762

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-07-29 03:44:49 +00:00

coilyco-ops added

priority