revert(k3s): drop the traefik hostNetwork bypass, keep the #910 parity fix #947

Merged

coilysiren merged 1 commit from revert/traefik-hostnetwork into main

2026-08-26 04:58:09 +00:00

coilyco-ops commented

2026-08-26 04:45:10 +00:00

Owner

Merge this promptly. Until it lands, main says hostNetwork while the cluster says LoadBalancer, and the node's /var/lib/rancher/k3s/server/manifests/traefik-config.yaml still holds the hostNetwork version. k3s re-applies that directory on restart, so a k3s or node restart would repeat the outage unattended.

What happened

The hostNetwork/ClusterIP change from #941 took every public hostname down for ~25 minutes (roughly 04:14 to 04:40 UTC, 2026-08-26). Rolled back live by applying the previous HelmChartConfig; all six public surfaces and the git-upload-pack path verified 200 afterwards, at or better than the pre-change baseline. Forgejo was untouched throughout, 0 restarts.

Why it failed

error opening listener: listen tcp :80: bind: permission denied

The chart runs Traefik as UID 65532 with allowPrivilegeEscalation: false. That sets NoNewPrivs, and NoNewPrivs makes the kernel ignore file capabilities, so the NET_BIND_SERVICE I added never became effective. Kubernetes has no ambient-capability support, so a pod spec alone cannot grant a working CAP_NET_BIND_SERVICE to a non-root process.

Neither helm template nor the server-side dry run could have caught this. Both validate the manifest. The failure is kernel behaviour visible only once the container starts. #941 presented the render as the verification, and that was wrong.

The svclb ordering trap I predicted in #941 also fired: FailedScheduling ... didn't have free ports while svclb still held 80/443, then no path at all for external traffic once svclb was torn down and the replacement was still crash-looping.

Why this is not `git revert e1db70e`

e1db70e squashed two changes. Reverting it wholesale would also undo the template parity fix and re-arm the #910 landmine, where an ordinary ansible-sync rewrites the config with the dead service.loadBalancerIP key and drops the public IP for every hostname.

So this restores the template to the parity state and leaves it there. Net effect against the live cluster after merge and converge: no change, because parity is what is running.

Docs

The section in docs/k3s-deploy-notes-manifests.md is kept and rewritten to record what actually happened rather than what I predicted: the NoNewPrivs mechanism, the svclb port-ordering trap, the rollback that worked, and the note that a render is not proof of runtime behaviour. If someone retries this, the shape to try is hostPort: 80/443 onto unprivileged container ports 8000/8443, or MetalLB, and to validate on a throwaway workload that actually binds the port first.

After merge

just ansible-sync check hosts=kai-server tags=k3s-config ask_pass
just ansible-sync apply hosts=kai-server tags=k3s-config ask_pass

The check diff should show the node's traefik-config.yaml returning to the parity content. That converge rewrites the file so a k3s restart is safe again. It should not restart Traefik in any user-visible way, since the resulting HelmChartConfig matches what is already live.

**Merge this promptly.** Until it lands, `main` says hostNetwork while the cluster says LoadBalancer, and the node's `/var/lib/rancher/k3s/server/manifests/traefik-config.yaml` still holds the hostNetwork version. k3s re-applies that directory on restart, so a k3s or node restart would repeat the outage unattended. ## What happened The hostNetwork/ClusterIP change from #941 took **every public hostname down for ~25 minutes** (roughly 04:14 to 04:40 UTC, 2026-08-26). Rolled back live by applying the previous HelmChartConfig; all six public surfaces and the `git-upload-pack` path verified 200 afterwards, at or better than the pre-change baseline. Forgejo was untouched throughout, 0 restarts. ## Why it failed ``` error opening listener: listen tcp :80: bind: permission denied ``` The chart runs Traefik as UID 65532 with `allowPrivilegeEscalation: false`. That sets `NoNewPrivs`, and **NoNewPrivs makes the kernel ignore file capabilities**, so the `NET_BIND_SERVICE` I added never became effective. Kubernetes has no ambient-capability support, so a pod spec alone cannot grant a working `CAP_NET_BIND_SERVICE` to a non-root process. **Neither `helm template` nor the server-side dry run could have caught this.** Both validate the manifest. The failure is kernel behaviour visible only once the container starts. #941 presented the render as the verification, and that was wrong. The svclb ordering trap I predicted in #941 also fired: `FailedScheduling ... didn't have free ports` while svclb still held 80/443, then no path at all for external traffic once svclb was torn down and the replacement was still crash-looping. ## Why this is not `git revert e1db70e` `e1db70e` squashed two changes. Reverting it wholesale would also undo the template parity fix and **re-arm the #910 landmine**, where an ordinary `ansible-sync` rewrites the config with the dead `service.loadBalancerIP` key and drops the public IP for every hostname. So this restores the template to the parity state and leaves it there. Net effect against the live cluster after merge and converge: **no change**, because parity is what is running. ## Docs The section in `docs/k3s-deploy-notes-manifests.md` is kept and rewritten to record what actually happened rather than what I predicted: the NoNewPrivs mechanism, the svclb port-ordering trap, the rollback that worked, and the note that a render is not proof of runtime behaviour. If someone retries this, the shape to try is `hostPort: 80/443` onto unprivileged container ports 8000/8443, or MetalLB, and to validate on a throwaway workload that actually binds the port first. ## After merge ``` just ansible-sync check hosts=kai-server tags=k3s-config ask_pass just ansible-sync apply hosts=kai-server tags=k3s-config ask_pass ``` The check diff should show the node's `traefik-config.yaml` returning to the parity content. That converge rewrites the file so a k3s restart is safe again. It should not restart Traefik in any user-visible way, since the resulting HelmChartConfig matches what is already live.

coilyco-ops added 1 commit

2026-08-26 04:45:10 +00:00

revert(k3s): drop the traefik hostNetwork bypass, keep the #910 parity fix

TruffleHog / Scan for secrets (pull_request) Successful in 5s

Details

CI / lint (pull_request) Successful in 42s

Details

8a1f91e2dc

The hostNetwork/ClusterIP change from #941 took every public hostname down
for ~25 minutes on 2026-08-26 and had to be rolled back live.

Traefik could not bind :80. The chart runs it as UID 65532 with
allowPrivilegeEscalation: false, which sets NoNewPrivs, and NoNewPrivs makes
the kernel ignore file capabilities. NET_BIND_SERVICE therefore never became
effective, and Kubernetes has no ambient-capability support to grant it
another way.

  error opening listener: listen tcp :80: bind: permission denied

Neither helm template nor a server-side dry run could have caught this. Both
validate the manifest; the failure is kernel behaviour visible only once the
container starts. The PR claimed the render was the verification, and it was
not.

This is a targeted revert, not `git revert e1db70e`. That commit squashed two
changes, and reverting it wholesale would also undo the template parity fix
and re-arm the #910 landmine, where an ordinary ansible run drops the public
IP for every hostname. The template goes back to the parity state and stays
there.

The docs section is kept and rewritten to record the actual failure, the
NoNewPrivs mechanism, the svclb port-ordering trap, and the rollback that
worked, so the next attempt starts from evidence rather than from the same
reasoning.

Refs #909 #910

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: sysadmin