fix(k3s): restore traefik template parity, then bypass ServiceLB for real client IPs #941

Merged

coilysiren merged 2 commits from fix/traefik-hostnetwork-source-ip into main

2026-08-26 03:01:18 +00:00

coilyco-ops commented

2026-08-26 03:00:02 +00:00

Owner

Refs #909. Sequencing agreed with Kai: ServiceLB bypass first, then the 499-triggered self-healing script, because every actuator for that script is currently bent around the masked source IP.

Two commits, deliberately separate.

1. `fix(k3s)`: restore template parity with live — an armed landmine, no live change

The ansible template that converges HelmChartConfig/traefik still carried the dead service.loadBalancerIP key that #910 was closed for. #910 was fixed on the live object only. The authoring layer was never updated, so the fix was one ansible run away from being undone.

Running the k3s-config role today would rewrite /var/lib/rancher/k3s/server/manifests/traefik-config.yaml with the old spelling. k3s auto-applies that directory, which triggers a helm upgrade, which re-renders the Service without loadBalancerIP and drops the public IP binding for every public hostname on the cluster. That is the exact #910 outage, reachable from an ordinary ansible run rather than a deliberate edit.

The template was also missing externalTrafficPolicy and the entire logs.access block. Those access logs carry User-Agent and Referer, are the only reason the 2026-08-26 crawler could be attributed at all, and are the source for the 499 signal in #909.

This commit only makes the template match what is already live, so applying it is a no-op against the running cluster.

Same class as the 1500Mi vs 3Gi drift found earlier tonight: fixed live, never landed in the source.

2. `feat(k3s)`: hostNetwork bypass — this one has an outage

Service becomes ClusterIP (no ServiceLB, no svclb DaemonSet) and Traefik binds the node's :80/:443 via hostNetwork. Three traps carried:

NET_BIND_SERVICE added, since the chart runs as UID 65532 with all caps dropped and otherwise cannot bind privileged ports.
metrics moved 9100 to 9101, because 9100 is node-exporter's port and they collide under hostNetwork. The chart's own values file warns about this.
loadBalancerIP and externalTrafficPolicy dropped, because both are rejected on a ClusterIP Service. An earlier draft leaked them and would have failed the apply.

Verification

Per #910's instruction, verified by rendering rather than applying. Chart pulled from the k3s static path, helm template run with the k3s HelmChart valuesContent as base and these values on top:

type: ClusterIP          (no loadBalancerIP, no externalTrafficPolicy)
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containerPort: 80 / 443 / 9101
capabilities.add: NET_BIND_SERVICE

A server-side dry-run of the rendered Service is accepted by the API server.

Worth recording: my first render layered the previous HelmChartConfig underneath and leaked loadBalancerIP and externalTrafficPolicy onto the ClusterIP Service. k3s replaces the HelmChartConfig rather than merging with the old one, so that render was wrong. The trap is now in docs/k3s-deploy-notes-manifests.md.

Blast radius and rollback

DNS cannot move: all published ingresses pin external-dns.alpha.kubernetes.io/target and external-dns runs --policy=upsert-only, so records never derive from Service status and are never deleted.

Traefik is the only LoadBalancer Service in the cluster, so nothing else loses svclb. cert-manager uses DNS-01, so certificate issuance does not depend on port 80.

Applying commit 2 restarts Traefik and briefly drops every public hostname while svclb tears down and the hostNetwork pod binds the ports. Expect tens of seconds, longer if the pod hits CrashLoopBackOff backoff while svclb is still holding the ports. The k3s API is on the tailnet independently of Traefik, so rollback stays available even with ingress down.

Not done

The k3s_traefik_load_balancer_ip_available gate and its SSM lookup are now vestigial, since the config no longer consumes the public IP. Left in place because it fails safe by preserving the existing file, and because a role refactor does not belong in a change this risky. Tracked separately.

Refs #909. Sequencing agreed with Kai: ServiceLB bypass first, then the 499-triggered self-healing script, because every actuator for that script is currently bent around the masked source IP. Two commits, deliberately separate. ## 1. `fix(k3s)`: restore template parity with live — **an armed landmine, no live change** The ansible template that converges `HelmChartConfig/traefik` still carried the dead `service.loadBalancerIP` key that #910 was closed for. **#910 was fixed on the live object only.** The authoring layer was never updated, so the fix was one ansible run away from being undone. Running the `k3s-config` role today would rewrite `/var/lib/rancher/k3s/server/manifests/traefik-config.yaml` with the old spelling. k3s auto-applies that directory, which triggers a helm upgrade, which re-renders the Service without `loadBalancerIP` and drops the public IP binding for **every public hostname on the cluster**. That is the exact #910 outage, reachable from an ordinary ansible run rather than a deliberate edit. The template was also missing `externalTrafficPolicy` and the entire `logs.access` block. Those access logs carry `User-Agent` and `Referer`, are the only reason the 2026-08-26 crawler could be attributed at all, and are the source for the 499 signal in #909. This commit only makes the template match what is already live, so **applying it is a no-op against the running cluster.** Same class as the `1500Mi` vs `3Gi` drift found earlier tonight: fixed live, never landed in the source. ## 2. `feat(k3s)`: hostNetwork bypass — **this one has an outage** Service becomes `ClusterIP` (no ServiceLB, no svclb DaemonSet) and Traefik binds the node's :80/:443 via `hostNetwork`. Three traps carried: * `NET_BIND_SERVICE` added, since the chart runs as UID 65532 with all caps dropped and otherwise cannot bind privileged ports. * metrics moved 9100 to 9101, because 9100 is node-exporter's port and they collide under hostNetwork. The chart's own values file warns about this. * `loadBalancerIP` and `externalTrafficPolicy` dropped, because both are rejected on a ClusterIP Service. An earlier draft leaked them and would have failed the apply. ## Verification Per #910's instruction, verified by rendering rather than applying. Chart pulled from the k3s static path, `helm template` run with the k3s `HelmChart` valuesContent as base and these values on top: ``` type: ClusterIP (no loadBalancerIP, no externalTrafficPolicy) hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containerPort: 80 / 443 / 9101 capabilities.add: NET_BIND_SERVICE ``` A server-side dry-run of the rendered Service is accepted by the API server. Worth recording: my first render layered the *previous* HelmChartConfig underneath and leaked `loadBalancerIP` and `externalTrafficPolicy` onto the ClusterIP Service. k3s replaces the HelmChartConfig rather than merging with the old one, so that render was wrong. The trap is now in `docs/k3s-deploy-notes-manifests.md`. ## Blast radius and rollback DNS cannot move: all published ingresses pin `external-dns.alpha.kubernetes.io/target` and external-dns runs `--policy=upsert-only`, so records never derive from Service status and are never deleted. Traefik is the only LoadBalancer Service in the cluster, so nothing else loses svclb. cert-manager uses DNS-01, so certificate issuance does not depend on port 80. **Applying commit 2 restarts Traefik and briefly drops every public hostname** while svclb tears down and the hostNetwork pod binds the ports. Expect tens of seconds, longer if the pod hits CrashLoopBackOff backoff while svclb is still holding the ports. The k3s API is on the tailnet independently of Traefik, so rollback stays available even with ingress down. ## Not done The `k3s_traefik_load_balancer_ip_available` gate and its SSM lookup are now vestigial, since the config no longer consumes the public IP. Left in place because it fails safe by preserving the existing file, and because a role refactor does not belong in a change this risky. Tracked separately.

coilyco-ops added 2 commits

2026-08-26 03:00:02 +00:00

fix(k3s): restore traefik HelmChartConfig parity with live c68cc9e26b

The ansible template still carried the dead service.loadBalancerIP key that
infrastructure#910 was closed for. #910 was fixed on the live object only, so
the authoring layer that converges the file was left armed.

Running the k3s-config role today would rewrite
/var/lib/rancher/k3s/server/manifests/traefik-config.yaml with the old
spelling, which k3s auto-applies, which triggers a helm upgrade that
re-renders the Service without loadBalancerIP and drops the public IP binding
for every public hostname on the cluster. That is the exact #910 outage,
reachable from an ordinary ansible run rather than a deliberate edit.

The template was also missing externalTrafficPolicy and the whole
logs.access block. Those access logs carry User-Agent and Referer and are
the only reason the 2026-08-26 crawler could be attributed at all, and they
are the source for the 499 signal in #909.

This commit only makes the template match what is already live, so applying
it is a no-op against the running cluster.

Refs #909 #910

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: sysadmin

feat(k3s): bypass ServiceLB so Traefik sees real client IPs

TruffleHog / Scan for secrets (pull_request) Successful in 6s

Details

CI / lint (pull_request) Successful in 47s

Details

99373ec9ea

klipper-lb forwards through a pod network namespace and SNATs every inbound
packet to the svclb-traefik pod IP, so Traefik and every service behind it
saw one client address for the whole internet. That is why the forgejo
rate limiter bucketed a crawler, CI, and Flux together, and why an abusive
client could not be identified or throttled (infrastructure#909).

externalTrafficPolicy=Local was tried on 2026-08-24 and does not fix it: it
governs node eligibility, not the klipper-lb hop. Removing the hop is the
only thing that works, so the Service becomes ClusterIP (no ServiceLB, no
svclb DaemonSet) and Traefik binds the node's :80/:443 via hostNetwork.

Carries the three traps this shape needs:
- NET_BIND_SERVICE, since the chart runs as UID 65532 with caps dropped and
  cannot otherwise bind privileged ports.
- metrics moved 9100 -> 9101, because 9100 is node-exporter's port and they
  collide under hostNetwork. The chart's own values file warns about this.
- loadBalancerIP and externalTrafficPolicy dropped entirely, because both are
  rejected on a ClusterIP Service.

Verified by rendering, per infrastructure#910, not by applying: helm template
against the chart from the k3s static path, with the k3s HelmChart
valuesContent as base and these values on top, produces hostNetwork: true,
dnsPolicy ClusterFirstWithHostNet, containerPorts 80/443/9101,
NET_BIND_SERVICE, and a clean ClusterIP Service with no leaked LoadBalancer
fields. A server-side dry-run of that Service is accepted.

DNS is unaffected: all published ingresses pin
external-dns.alpha.kubernetes.io/target and external-dns is upsert-only, so
records never derive from Service status.

Applying this restarts Traefik and briefly drops every public hostname while
svclb tears down and the new pod binds the ports.

Refs #909 #910

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: sysadmin

coilyco-ops referenced this pull request

2026-08-26 03:00:23 +00:00

k3s-config: SSM public-IP lookup and its gate are vestigial once Traefik is ClusterIP #942