Add an allowlisted egress proxy for Forgejo Actions runner jobs #563

New issue

Closed

opened 2026-07-16 00:09:05 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-16 00:09:05 +00:00

Member

Forgejo Actions jobs need a stable outbound path for services that fail from the runner DinD job network. Start with Telegram CI alerts, but design the surface for more destinations later.

Context:

Related open issues: infrastructure#546 and infrastructure#553.
The Telegram alert wiring and secrets exist, but CI failure alerts do not deliver because runner job containers cannot complete the HTTPS path to api.telegram.org.
A plain Kubernetes pod in namespace forgejo can reach https://api.telegram.org/ over IPv4, and Python urllib from agentic-os:release as a normal pod succeeds in about 1.1s.
The same test through the Forgejo runner DinD path (kubectl exec forgejo-runner-0 -c dind -- docker run curlimages/curl ...) fails for Telegram. IPv6 is Network unreachable; forced IPv4 sends TLS ClientHello and times out. Non-Telegram HTTPS from that same DinD path still works for example.com and Forgejo.

Likely direction:

Add an infrastructure-owned forward proxy for runner egress, not an app-owned deploy service.
Put the manifest beside the runner substrate, for example deploy/egress-proxy.yml or an equivalent runner support manifest.
Use an allowlist-oriented proxy such as Squid, Envoy, or tinyproxy.
Start the allowlist with api.telegram.org:443.
Give the service a stable in-cluster address, for example an egress-proxy Service in forgejo or a dedicated namespace.
Configure CI alert steps to use HTTPS_PROXY for the Telegram call, while keeping NO_PROXY for cluster-local services, Forgejo, the in-cluster registry, and localhost.
Document how to add future destinations without turning the runner into an unrestricted outbound tunnel.

Acceptance:

A disposable job container on the Forgejo build runner can reach https://api.telegram.org/ through the proxy.
Existing Telegram CI alert steps can use the proxy without embedding new per-repo service-specific hacks.
The proxy config is allowlisted and documented for future destinations.
The runner networking docs mention why this exists and link back to infrastructure#546/#553.

This belongs in infrastructure because it is runner substrate, next to deploy/forgejo-runner.yml and the k3s runner network trap docs, not in the deploy repo.

Forgejo Actions jobs need a stable outbound path for services that fail from the runner DinD job network. Start with Telegram CI alerts, but design the surface for more destinations later. Context: - Related open issues: infrastructure#546 and infrastructure#553. - The Telegram alert wiring and secrets exist, but CI failure alerts do not deliver because runner job containers cannot complete the HTTPS path to `api.telegram.org`. - A plain Kubernetes pod in namespace `forgejo` can reach `https://api.telegram.org/` over IPv4, and Python `urllib` from `agentic-os:release` as a normal pod succeeds in about 1.1s. - The same test through the Forgejo runner DinD path (`kubectl exec forgejo-runner-0 -c dind -- docker run curlimages/curl ...`) fails for Telegram. IPv6 is `Network unreachable`; forced IPv4 sends TLS ClientHello and times out. Non-Telegram HTTPS from that same DinD path still works for `example.com` and Forgejo. Likely direction: - Add an infrastructure-owned forward proxy for runner egress, not an app-owned deploy service. - Put the manifest beside the runner substrate, for example `deploy/egress-proxy.yml` or an equivalent runner support manifest. - Use an allowlist-oriented proxy such as Squid, Envoy, or tinyproxy. - Start the allowlist with `api.telegram.org:443`. - Give the service a stable in-cluster address, for example an `egress-proxy` Service in `forgejo` or a dedicated namespace. - Configure CI alert steps to use `HTTPS_PROXY` for the Telegram call, while keeping `NO_PROXY` for cluster-local services, Forgejo, the in-cluster registry, and localhost. - Document how to add future destinations without turning the runner into an unrestricted outbound tunnel. Acceptance: - A disposable job container on the Forgejo build runner can reach `https://api.telegram.org/` through the proxy. - Existing Telegram CI alert steps can use the proxy without embedding new per-repo service-specific hacks. - The proxy config is allowlisted and documented for future destinations. - The runner networking docs mention why this exists and link back to infrastructure#546/#553. This belongs in infrastructure because it is runner substrate, next to `deploy/forgejo-runner.yml` and the k3s runner network trap docs, not in the deploy repo.

coilyco-ops added the

priority

label

2026-07-16 00:48:22 +00:00

coilyco-ops referenced this issue

2026-07-16 00:50:53 +00:00

CI runner cannot reach api.telegram.org - every failure alert dies with a TLS handshake timeout #553

coilyco-ops commented

2026-07-22 06:51:19 +00:00

Author

Member

Second affected destination confirmed from the same class of Forgejo runner path. eco-app Actions run 105, build-image task 14518, reached a Docker build that restores mods/jobs/src, then timed out fetching https://api.nuget.org/v3/index.json with NU1301. Restore spent about 10 minutes retrying before the image build failed, so none of the four downstream per-mod publishers started.

This strengthens the case that #563 owns a general destination-sensitive DinD egress defect rather than a Telegram-only integration problem. Add api.nuget.org:443 to the diagnostic and eventual allowlist alongside api.telegram.org:443. Acceptance should verify both endpoints from the general runner DinD path and the dedicated docker-build BuildKit path.

Related application blockers: coilyco-gaming/eco-app#111 and #141. The narrower NuGet-proxy proposal in #571 is being closed as a duplicate because a package cache would mask the shared runner-network symptom and still fail on cold misses.

Second affected destination confirmed from the same class of Forgejo runner path. eco-app Actions run 105, build-image task 14518, reached a Docker build that restores `mods/jobs/src`, then timed out fetching `https://api.nuget.org/v3/index.json` with NU1301. Restore spent about 10 minutes retrying before the image build failed, so none of the four downstream per-mod publishers started. This strengthens the case that #563 owns a general destination-sensitive DinD egress defect rather than a Telegram-only integration problem. Add `api.nuget.org:443` to the diagnostic and eventual allowlist alongside `api.telegram.org:443`. Acceptance should verify both endpoints from the general runner DinD path and the dedicated `docker-build` BuildKit path. Related application blockers: coilyco-gaming/eco-app#111 and #141. The narrower NuGet-proxy proposal in #571 is being closed as a duplicate because a package cache would mask the shared runner-network symptom and still fail on cold misses.

coilyco-ops referenced this issue from coilyco-gaming/eco-app

2026-07-22 06:51:19 +00:00

interactive: inspect repeated CI image-build failure #141

coilyco-ops referenced this issue

2026-07-22 06:51:33 +00:00

deploy: add an in-cluster NuGet caching proxy for runner builds #571

coilyco-ops referenced this issue from coilyco-gaming/eco-app

2026-07-22 06:51:35 +00:00

Design a real required CI path for C# / NuGet checks #111

coilyco-ops referenced this issue from coilyco-gaming/eco-app

2026-07-22 07:14:02 +00:00

ci: route the Eco image build through the runner egress proxy #142