CI runner cannot reach api.telegram.org - every failure alert dies with a TLS handshake timeout #553

New issue

Open

opened 2026-07-13 03:06:08 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-13 03:06:08 +00:00

Member

The Telegram failure alerts wired into aos/ward/cli-guard workflows (the Alert Telegram on main failure steps, secrets synced by ward exec sync-actions-secrets) never deliver. Every sampled failed run on the aos docker runner shows the same step output:

telegram alert failed: <urlopen error _ssl.c:983: The handshake operation timed out>

Samples: aos actions runs 1120 (promote.yml), 1121 (ci.yml), 1123 (promote.yml), all 2026-07-13. The step reaches the HTTPS call, so the secrets are configured; the TLS handshake to api.telegram.org:443 times out from inside the runner container. The same endpoint answers from Kai's desktop on the same network in ~0.5s, so this is runner/container egress, not an ISP block - likely MTU black-holing on the runner's docker/tailscale network path, or an IPv6 preference the runner cannot route.

Diagnosis steps for the runner host:

docker run --rm curlimages/curl -sv --max-time 15 https://api.telegram.org/ on the runner's docker network - reproduce the hang.
Compare curl -4 vs curl -6, and try --tls-max 1.2.
Check the docker network MTU against the host's (tailscale MTU 1280 vs docker default 1500 is the classic).

The alert scripts already parameterize API_BASE, so a fallback is pointing them at a reachable relay if fixing egress is not worth it.

Found while answering Kai's "I haven't gotten a Telegram all day" - the alert path has plausibly never delivered from CI. Runner networking is infrastructure's rollout surface, so the fix lands here.

The Telegram failure alerts wired into aos/ward/cli-guard workflows (the `Alert Telegram on main failure` steps, secrets synced by `ward exec sync-actions-secrets`) never deliver. Every sampled failed run on the aos docker runner shows the same step output: ``` telegram alert failed: <urlopen error _ssl.c:983: The handshake operation timed out> ``` Samples: aos actions runs 1120 (promote.yml), 1121 (ci.yml), 1123 (promote.yml), all 2026-07-13. The step reaches the HTTPS call, so the secrets are configured; the TLS handshake to `api.telegram.org:443` times out from inside the runner container. The same endpoint answers from Kai's desktop on the same network in ~0.5s, so this is runner/container egress, not an ISP block - likely MTU black-holing on the runner's docker/tailscale network path, or an IPv6 preference the runner cannot route. Diagnosis steps for the runner host: * `docker run --rm curlimages/curl -sv --max-time 15 https://api.telegram.org/` on the runner's docker network - reproduce the hang. * Compare `curl -4` vs `curl -6`, and try `--tls-max 1.2`. * Check the docker network MTU against the host's (tailscale MTU 1280 vs docker default 1500 is the classic). The alert scripts already parameterize `API_BASE`, so a fallback is pointing them at a reachable relay if fixing egress is not worth it. Found while answering Kai's "I haven't gotten a Telegram all day" - the alert path has plausibly never delivered from CI. Runner networking is infrastructure's rollout surface, so the fix lands here.