Actions runners cannot reach api.telegram.org: SSL handshake timeout kills every CI failure alert #546

New issue

Open

opened 2026-07-12 07:08:25 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-07-12 07:08:25 +00:00

Owner

The "Alert Telegram on main failure" steps across aos/ward workflows fail in ~16s with:

telegram alert failed: <urlopen error _ssl.c:983: The handshake operation timed out>

Observed identically in aos runs 1077 and 1078 (2026-07-12) and consistent with the 16s alert-step failures across every red run of 7/10-7/11. Two independent problems stack here:

The repos never had TELEGRAM_BOT_TOKEN / TELEGRAM_RED_CHAT_ID Actions secrets set - fixed by ward exec sync-actions-secrets (aos#479) once Kai runs it.
Even with secrets, the k3s runner pods appear unable to complete a TLS handshake to api.telegram.org (egress policy, MTU, or DNS-to-blackhole). The 15s urllib timeout + ~1s startup matches the observed 16s exactly.

Check egress from the forgejo-runner pods (e.g. kubectl exec a runner and curl api.telegram.org), then either open the egress path or route alerts through something already reachable from the cluster. Until then main-failure alerting is silently dead, which is how the 7/10-7/11 release-red streak ran unnoticed.

The "Alert Telegram on main failure" steps across aos/ward workflows fail in ~16s with: telegram alert failed: <urlopen error _ssl.c:983: The handshake operation timed out> Observed identically in aos runs 1077 and 1078 (2026-07-12) and consistent with the 16s alert-step failures across every red run of 7/10-7/11. Two independent problems stack here: 1. The repos never had TELEGRAM_BOT_TOKEN / TELEGRAM_RED_CHAT_ID Actions secrets set - fixed by `ward exec sync-actions-secrets` (aos#479) once Kai runs it. 2. Even with secrets, the k3s runner pods appear unable to complete a TLS handshake to api.telegram.org (egress policy, MTU, or DNS-to-blackhole). The 15s urllib timeout + ~1s startup matches the observed 16s exactly. Check egress from the forgejo-runner pods (e.g. `kubectl exec` a runner and curl api.telegram.org), then either open the egress path or route alerts through something already reachable from the cluster. Until then main-failure alerting is silently dead, which is how the 7/10-7/11 release-red streak ran unnoticed.