coilyco-flight-deck/infrastructure

Watch

Forgejo Actions runners hairpin through the public edge, 59% of all ingress traffic #912

New issue

Closed

opened 2026-08-25 00:57:16 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-25 00:57:16 +00:00

Owner

Finding

With Traefik access logs newly enabled (coilyco-bridge/deploy#786, see coilyco-bridge/deploy#786), the largest single source of traffic through the public ingress turns out to be the estate's own CI runners.

Across 519 sampled requests:

   308  59.3%  forgejo-runner/v12.13.2
   101  19.5%  meta-externalagent
    56  10.8%  Chrome/Edge 120 on Windows
    27   5.2%  Chrome 119 on macOS
    11   2.1%  Claude-User
    ...

The runners are roughly three times the AI crawler that caused the OOM incident in #909.

What they are doing

Continuous long-poll against the task queue, via the public hostname:

POST /api/actions/runner.v1.RunnerService/FetchTask
RequestHost: forgejo.coilysiren.me
RouterName:  websecure-forgejo-forgejo-forgejo-coilysiren-me@kubernetes
ServiceURL:  http://10.42.0.87:3000

Several arrive within the same second (RequestCount 297, 299, 300 in one observed second).

Why it matters

The runner pods live in the forgejo namespace on kai-server, alongside the Forgejo pod they are polling. They resolve forgejo.coilysiren.me to the public IP 99.110.50.213, so each poll leaves the cluster, hairpins through the home router's port-forward, re-enters via klipper-lb, and traverses Traefik to reach a Service two hops away.

Three consequences:

Rate-limit contention. forgejo-ratelimit is average: 50, burst: 100, period: 1s, and because klipper-lb SNATs every client to one pod IP, it is a single global bucket rather than per-source. The runners are the heaviest occupant of that bucket. This is a strong candidate for why Flux source-controller took collateral 429s during the incident, and it means the runners and the crawler are actively competing with each other and with every human visitor.
Unnecessary edge load. Roughly 60% of all ingress work is traffic that never needed to leave the cluster, including TLS termination and routing for every poll.
Fragility. Any interruption at the router, the public IP, or the ingress path breaks CI, even when Forgejo itself is perfectly healthy inside the cluster. During the crash loop in #909, the runners were competing for the same degraded surface.

Possible direction, not yet validated

Point the runners at the in-cluster Service (http://forgejo.forgejo.svc.cluster.local or the existing NodePort) instead of the public hostname, so polling stays inside the cluster.

This needs verification before anyone acts on it. Specifically:

Whether Forgejo Actions runner registration tolerates an instance URL that differs from ROOT_URL, and whether tokens, artifact upload, and log streaming all follow the same base URL.
Whether anything in the workflow path depends on the runner reaching the public hostname specifically.
Whether a split-horizon DNS entry or a pod hostAlias is a lighter change than re-registration. Note the Forgejo pod itself already carries a hostAlias mapping forgejo.coilysiren.me to 192.168.0.194, so this pattern is established here and may be the cheapest route.

Runner definitions live under deploy/forgejo-runners-scoped/ and deploy/forgejo-runners-ser8/ in this repo.

How it was found

Not by looking for it. Traefik access logs were enabled to identify the crawler from #909, and the User-Agent breakdown surfaced this instead. Before access logs existed there was no way to see it, because every client appeared as the same klipper-lb pod IP and no per-request record was kept at the edge.

No change has been made. Filing so the observation is not lost.

## Finding With Traefik access logs newly enabled (`coilyco-bridge/deploy#786`, see https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/786), the largest single source of traffic through the public ingress turns out to be the estate's own CI runners. Across 519 sampled requests: ``` 308 59.3% forgejo-runner/v12.13.2 101 19.5% meta-externalagent 56 10.8% Chrome/Edge 120 on Windows 27 5.2% Chrome 119 on macOS 11 2.1% Claude-User ... ``` The runners are roughly three times the AI crawler that caused the OOM incident in https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/909. ## What they are doing Continuous long-poll against the task queue, via the public hostname: ``` POST /api/actions/runner.v1.RunnerService/FetchTask RequestHost: forgejo.coilysiren.me RouterName: websecure-forgejo-forgejo-forgejo-coilysiren-me@kubernetes ServiceURL: http://10.42.0.87:3000 ``` Several arrive within the same second (`RequestCount` 297, 299, 300 in one observed second). ## Why it matters The runner pods live in the `forgejo` namespace on kai-server, alongside the Forgejo pod they are polling. They resolve `forgejo.coilysiren.me` to the public IP `99.110.50.213`, so each poll leaves the cluster, hairpins through the home router's port-forward, re-enters via klipper-lb, and traverses Traefik to reach a Service two hops away. Three consequences: 1. **Rate-limit contention.** `forgejo-ratelimit` is `average: 50, burst: 100, period: 1s`, and because klipper-lb SNATs every client to one pod IP, it is a single global bucket rather than per-source. The runners are the heaviest occupant of that bucket. This is a strong candidate for why Flux `source-controller` took collateral 429s during the incident, and it means the runners and the crawler are actively competing with each other and with every human visitor. 2. **Unnecessary edge load.** Roughly 60% of all ingress work is traffic that never needed to leave the cluster, including TLS termination and routing for every poll. 3. **Fragility.** Any interruption at the router, the public IP, or the ingress path breaks CI, even when Forgejo itself is perfectly healthy inside the cluster. During the crash loop in https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/909, the runners were competing for the same degraded surface. ## Possible direction, not yet validated Point the runners at the in-cluster Service (`http://forgejo.forgejo.svc.cluster.local` or the existing NodePort) instead of the public hostname, so polling stays inside the cluster. **This needs verification before anyone acts on it.** Specifically: * Whether Forgejo Actions runner registration tolerates an instance URL that differs from `ROOT_URL`, and whether tokens, artifact upload, and log streaming all follow the same base URL. * Whether anything in the workflow path depends on the runner reaching the public hostname specifically. * Whether a split-horizon DNS entry or a pod `hostAlias` is a lighter change than re-registration. Note the Forgejo pod itself already carries a `hostAlias` mapping `forgejo.coilysiren.me` to `192.168.0.194`, so this pattern is established here and may be the cheapest route. Runner definitions live under `deploy/forgejo-runners-scoped/` and `deploy/forgejo-runners-ser8/` in this repo. ## How it was found Not by looking for it. Traefik access logs were enabled to identify the crawler from https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/909, and the User-Agent breakdown surfaced this instead. Before access logs existed there was no way to see it, because every client appeared as the same klipper-lb pod IP and no per-request record was kept at the edge. No change has been made. Filing so the observation is not lost.

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:05 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:54 +00:00

coilyco-ops commented

2026-08-28 21:11:48 +00:00

Author

Owner

Fixed 2026-08-28. Closing.

Split-horizon DNS applied on both clusters, which is the third of the three directions this issue proposed and the one it guessed would be lightest. It was.

coredns-custom on kai-server and ser8 now carries:

forgejo.server: |
  forgejo.coilysiren.me:53 {
      errors
      hosts {
          100.69.164.66 forgejo.coilysiren.me
      }
  }

100.69.164.66 is Traefik's LoadBalancer external IP, which is kai-server's tailnet address. SNI is preserved so the existing certificate validates unchanged, and no re-registration was needed. Landed declaratively in coilyco-flight-deck/infrastructure#976, so it survives a cluster rebuild.

Answers to the three open questions

Does runner registration tolerate an instance URL differing from ROOT_URL? Yes, and both spellings are already in production. kai-server's forgejo-runner-deploy-scoped uses http://forgejo.forgejo.svc.cluster.local, while ser8's forgejo-runner-ser8-flight-deck uses https://forgejo.coilysiren.me/. Both register and claim jobs. So re-registration was an available route, just a heavier one.

Does anything in the workflow path need the public hostname specifically? Nothing found. Job containers run under docker-in-docker with dnsPolicy: ClusterFirst, dockerd carries no --dns, and dnsConfig is unset, so DinD inherits the pod resolver and the override reaches the job as well as the daemon.

Was split-horizon lighter than re-registration? Yes. One ConfigMap key per cluster, no workload restarts beyond CoreDNS, no token or registration churn.

Correction to this issue's framing

The body says the runner pods "live in the forgejo namespace on kai-server, alongside the Forgejo pod they are polling." That is true of the deploy runners and not of the ones that matter most. The flight-deck runners that run agentic-os CI live on ser8, a separate cluster, and they were hairpinning their control plane as well as their job traffic. The fix covers both because it is applied per cluster, but the fault surface was larger than described here.

What this issue got right and is worth keeping

The klipper-lb observation. externalTrafficPolicy is Local and every request still arrives as ClientHost: 10.42.0.22, the svclb pod, so an external crawler and a runner are indistinguishable in the access log. I rediscovered this independently today while trying to verify the fix from the ingress side, and it is the reason that verification had to be structural instead. Worth its own issue if per-source rate limiting or real client IPs are ever wanted, since it requires replacing klipper-lb or fronting Traefik with PROXY protocol.

Not claimed

This did not resolve the CI outage that ran 17:42Z to roughly 18:15Z on 2026-08-28. CI recovered on its own about twenty minutes before the first apply, and neither end of that window is explained. The fix removes the structural fragility this issue documents. It is not the reason that day's incident ended. Full record in coilyco-flight-deck/agentic-os#1344.

## Fixed 2026-08-28. Closing. Split-horizon DNS applied on both clusters, which is the third of the three directions this issue proposed and the one it guessed would be lightest. It was. `coredns-custom` on kai-server and ser8 now carries: ``` forgejo.server: | forgejo.coilysiren.me:53 { errors hosts { 100.69.164.66 forgejo.coilysiren.me } } ``` `100.69.164.66` is Traefik's LoadBalancer external IP, which is kai-server's tailnet address. SNI is preserved so the existing certificate validates unchanged, and no re-registration was needed. Landed declaratively in coilyco-flight-deck/infrastructure#976, so it survives a cluster rebuild. ### Answers to the three open questions **Does runner registration tolerate an instance URL differing from ROOT_URL?** Yes, and both spellings are already in production. kai-server's `forgejo-runner-deploy-scoped` uses `http://forgejo.forgejo.svc.cluster.local`, while ser8's `forgejo-runner-ser8-flight-deck` uses `https://forgejo.coilysiren.me/`. Both register and claim jobs. So re-registration was an available route, just a heavier one. **Does anything in the workflow path need the public hostname specifically?** Nothing found. Job containers run under docker-in-docker with `dnsPolicy: ClusterFirst`, `dockerd` carries no `--dns`, and `dnsConfig` is unset, so DinD inherits the pod resolver and the override reaches the job as well as the daemon. **Was split-horizon lighter than re-registration?** Yes. One ConfigMap key per cluster, no workload restarts beyond CoreDNS, no token or registration churn. ### Correction to this issue's framing The body says the runner pods "live in the `forgejo` namespace on kai-server, alongside the Forgejo pod they are polling." That is true of the deploy runners and not of the ones that matter most. The flight-deck runners that run agentic-os CI live on **ser8**, a separate cluster, and they were hairpinning their control plane as well as their job traffic. The fix covers both because it is applied per cluster, but the fault surface was larger than described here. ### What this issue got right and is worth keeping The klipper-lb observation. `externalTrafficPolicy` is `Local` and every request still arrives as `ClientHost: 10.42.0.22`, the svclb pod, so an external crawler and a runner are indistinguishable in the access log. I rediscovered this independently today while trying to verify the fix from the ingress side, and it is the reason that verification had to be structural instead. Worth its own issue if per-source rate limiting or real client IPs are ever wanted, since it requires replacing klipper-lb or fronting Traefik with PROXY protocol. ### Not claimed This did not resolve the CI outage that ran 17:42Z to roughly 18:15Z on 2026-08-28. CI recovered on its own about twenty minutes before the first apply, and neither end of that window is explained. The fix removes the structural fragility this issue documents. It is not the reason that day's incident ended. Full record in coilyco-flight-deck/agentic-os#1344.

coilyco-ops

2026-08-28 21:11:53 +00:00