Set an explicit MTU on the Forgejo runner DinD daemons #783

New issue

Closed

opened 2026-08-11 23:35:08 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-11 23:35:08 +00:00

Member

Measured on kai-server 2026-08-12:

enp1s0   mtu 1500     physical NIC
cni0     mtu 1230     pod network
veth*    mtu 1230     every pod interface
docker0  mtu 1500     host docker bridge

docker -H tcp://10.42.0.29:2375 network inspect bridge
  "com.docker.network.driver.mtu": "1500"

No runner manifest passes --mtu to dockerd, so every daemon takes Docker's
1500 default regardless of the network it lands on.

Fix

Add --mtu=1230 to the dockerd args in each runner manifest, matching cni0.

Why this one matters most

Every other mitigation routes around this defect. The egress proxy exists
because of it, and it is the only candidate that also explains image-build
failures, where a layer pull hits the same ceiling with no pre-commit involved.

docs/forgejo-runner-egress-proxy.md already anticipates this: the proxy is
"a reliability route, not the final diagnosis of the DinD MTU or network
defect."

Verification

After rollout, docker -H tcp://<runner-pod-ip>:2375 network inspect bridge
reports 1230, and a large HTTPS fetch through a DinD-launched container
completes without the proxy.

Every job container launched through a runner's DinD daemon gets a bridge at MTU 1500, while the pod network underneath runs at 1230. Packets above 1230 bytes are dropped, so a TLS handshake completes and the server's certificate response disappears. That is the "ClientHello then stall" symptom the egress proxy was built to route around. Measured on kai-server 2026-08-12: ``` enp1s0 mtu 1500 physical NIC cni0 mtu 1230 pod network veth* mtu 1230 every pod interface docker0 mtu 1500 host docker bridge ``` ``` docker -H tcp://10.42.0.29:2375 network inspect bridge "com.docker.network.driver.mtu": "1500" ``` No runner manifest passes `--mtu` to `dockerd`, so every daemon takes Docker's 1500 default regardless of the network it lands on. ## Fix Add `--mtu=1230` to the `dockerd` args in each runner manifest, matching `cni0`. ## Why this one matters most Every other mitigation routes around this defect. The egress proxy exists because of it, and it is the only candidate that also explains image-build failures, where a layer pull hits the same ceiling with no pre-commit involved. `docs/forgejo-runner-egress-proxy.md` already anticipates this: the proxy is "a reliability route, not the final diagnosis of the DinD MTU or network defect." ## Verification After rollout, `docker -H tcp://<runner-pod-ip>:2375 network inspect bridge` reports 1230, and a large HTTPS fetch through a DinD-launched container completes without the proxy.

coilyco-ops referenced this issue from a commit

2026-08-11 23:42:28 +00:00

fix(forgejo): match the DinD bridge MTU to the pod network

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-11 23:42:49 +00:00

fix(forgejo): match every DinD bridge MTU to its node's pod network #789