fix(forgejo): match every DinD bridge MTU to its node's pod network #789

Merged

coilysiren merged 2 commits from aos/claude/dind-mtu into main

2026-08-12 00:13:42 +00:00

coilyco-ops commented

2026-08-11 23:42:49 +00:00

Member

Closes #783. The root cause behind the pre-commit and dependency-fetch failures, measured on both nodes.

The defect

Every job container launched through a runners DinD daemon got a bridge at MTU 1500, while the pod network underneath is smaller. Any reply larger than the pod MTU was dropped. A TLS ClientHello fits, the servers certificate response does not, which is exactly the "connects then stalls" symptom docs/forgejo-runner-egress-proxy.md was written about.

No runner manifest passed --mtu, so every daemon took Docker`s 1500 default regardless of the network it landed on.

Evidence, per node

kai-server - Tailscale-derived, 270 bytes of overshoot:

enp1s0   mtu 1500      cni0   mtu 1230      docker0  mtu 1500
docker -H tcp://10.42.0.29:2375 network inspect bridge  ->  mtu 1500

ser8 - plain flannel VXLAN, 50 bytes of overshoot:

enp1s0   mtu 1500      flannel.1 mtu 1450      cni0  mtu 1450      docker0  mtu 1500
docker -H tcp://10.42.0.167:2375 network inspect bridge  ->  mtu 1500

Change

kai-servers four DinD daemons take --mtu=1230`: general, build, deploy, publisher.
ser8s four kustomizations take --mtu=1450`.
tap-writer has no DinD and is untouched.

Why the value is per node, not one constant

kai-server sits behind Tailscale and ser8 does not, so their pod networks differ by 220 bytes. Applying kai-server`s 1230 to ser8 would have worked while quietly costing throughput on every job there, and reading either number off the other node is not evidence. Both were measured directly.

The first revision of this PR deliberately left ser8 alone because ssh ser8 was refused. That was the wrong username, not a policy boundary - ser8s account is coilysiren`. Measured and fixed.

What this should make redundant

If this holds, the egress proxy stops being load-bearing for ordinary dependency fetches, and the per-repo proxy wrappers in #782 and coilyco-bridge/deploy#366 become optional rather than necessary. Both are cheap and harmless to keep, and they cover the window before these runners roll.

Verification after rollout

docker -H tcp://<runner-pod-ip>:2375 network inspect bridge reports the node`s value on each runner, and a large HTTPS fetch from a DinD-launched container completes without the proxy.

This is a pod-template change on both nodes, so it rolls every runner and kills any in-flight job. Worth a quiet moment.

🤖 Generated with Claude Code

Closes #783. The root cause behind the pre-commit and dependency-fetch failures, measured on both nodes. ## The defect Every job container launched through a runner`s DinD daemon got a bridge at MTU 1500, while the pod network underneath is smaller. Any reply larger than the pod MTU was dropped. A TLS ClientHello fits, the server`s certificate response does not, which is exactly the "connects then stalls" symptom `docs/forgejo-runner-egress-proxy.md` was written about. No runner manifest passed `--mtu`, so every daemon took Docker`s 1500 default regardless of the network it landed on. ## Evidence, per node **kai-server** - Tailscale-derived, 270 bytes of overshoot: ``` enp1s0 mtu 1500 cni0 mtu 1230 docker0 mtu 1500 docker -H tcp://10.42.0.29:2375 network inspect bridge -> mtu 1500 ``` **ser8** - plain flannel VXLAN, 50 bytes of overshoot: ``` enp1s0 mtu 1500 flannel.1 mtu 1450 cni0 mtu 1450 docker0 mtu 1500 docker -H tcp://10.42.0.167:2375 network inspect bridge -> mtu 1500 ``` ## Change * kai-server`s four DinD daemons take `--mtu=1230`: general, build, deploy, publisher. * ser8`s four kustomizations take `--mtu=1450`. * `tap-writer` has no DinD and is untouched. ## Why the value is per node, not one constant kai-server sits behind Tailscale and ser8 does not, so their pod networks differ by 220 bytes. Applying kai-server`s 1230 to ser8 would have worked while quietly costing throughput on every job there, and reading either number off the other node is not evidence. Both were measured directly. The first revision of this PR deliberately left ser8 alone because `ssh ser8` was refused. That was the wrong username, not a policy boundary - ser8`s account is `coilysiren`. Measured and fixed. ## What this should make redundant If this holds, the egress proxy stops being load-bearing for ordinary dependency fetches, and the per-repo proxy wrappers in `#782` and `coilyco-bridge/deploy#366` become optional rather than necessary. Both are cheap and harmless to keep, and they cover the window before these runners roll. ## Verification after rollout `docker -H tcp://<runner-pod-ip>:2375 network inspect bridge` reports the node`s value on each runner, and a large HTTPS fetch from a DinD-launched container completes without the proxy. This is a pod-template change on both nodes, so it rolls every runner and kills any in-flight job. Worth a quiet moment. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

coilyco-ops added 1 commit

2026-08-11 23:42:49 +00:00

fix(forgejo): match the DinD bridge MTU to the pod network

TruffleHog / Scan for secrets (pull_request) Successful in 5s

Details

CI / lint (pull_request) Failing after 5m10s

Details

2482dbfb9a

Every job container launched through a runner's DinD daemon got a bridge
at MTU 1500 while the pod network underneath runs at 1230, so any reply
larger than 1230 bytes was dropped. A TLS ClientHello fits and the
server's certificate response does not, which is the "connects then
stalls" failure the egress proxy was built to route around.

Measured on kai-server:

  enp1s0   mtu 1500     physical NIC
  cni0     mtu 1230     pod network
  veth*    mtu 1230     every pod interface
  docker0  mtu 1500     host docker bridge

  docker -H tcp://10.42.0.29:2375 network inspect bridge
    "com.docker.network.driver.mtu": "1500"

No runner manifest passed --mtu, so each daemon took Docker's default
regardless of the network it landed on.

The four kai-server runners now pass --mtu=1230. The ser8 runners are
left alone: that node denies SSH under the current tailnet policy, so its
pod-network MTU is unmeasured, and a value derived from kai-server is not
evidence about ser8.

Closes #783

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-11 23:42:49 +00:00

coilyco-ops added 1 commit

2026-08-11 23:48:27 +00:00

fix(forgejo): match the ser8 DinD bridge MTU to its pod network

TruffleHog / Scan for secrets (pull_request) Successful in 5s

Details

CI / lint (pull_request) Successful in 1m11s

Details

7da5f075ad

ser8 has the same defect with a different number. Its pod network is
plain flannel VXLAN at 1450, where kai-server sits at 1230 behind
Tailscale, and both DinD daemons built bridges at Docker's 1500 default.

Measured on ser8:

  enp1s0      mtu 1500
  flannel.1   mtu 1450
  cni0        mtu 1450
  docker0     mtu 1500

  docker -H tcp://10.42.0.167:2375 network inspect bridge
    com.docker.network.driver.mtu = 1500

The four ser8 kustomizations now pass --mtu=1450. The gap there is 50
bytes rather than kai-server's 270, so the same class of failure with a
narrower trigger.

This is why the value is per node rather than one constant: applying
kai-server's 1230 here would have worked while quietly costing throughput
on every job, and reading either number off the other node is not
evidence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops changed title from ~~fix(forgejo): match the DinD bridge MTU to the pod network~~ to fix(forgejo): match every DinD bridge MTU to its node's pod network

2026-08-11 23:48:46 +00:00