coilyco-flight-deck/infrastructure

Watch

HelmChartConfig/traefik carries a dead service.loadBalancerIP key, and hand-applied Service drift #910

New issue

Closed

opened 2026-08-24 19:02:28 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-08-24 19:02:28 +00:00

Owner

The landmine

HelmChartConfig/traefik in kube-system on kai-server currently carries:

spec:
  valuesContent: |-
    service:
      loadBalancerIP: "99.110.50.213"
    additionalArguments:
      - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s"

service.loadBalancerIP is a dead key in traefik chart 34.x. It moved to service.spec.loadBalancerIP. The chart ignores the old spelling silently, with no warning and no error.

Proof, from decoding the live Helm release sh.helm.release.v1.traefik.v3 (chart traefik-34.2.1+up34.2.0). The release config contains service.loadBalancerIP: 99.110.50.213, and the manifest the chart actually rendered contains no loadBalancerIP field at all:

# chart-rendered Service spec, verbatim
type: LoadBalancer
selector:
  app.kubernetes.io/name: traefik
  app.kubernetes.io/instance: traefik-kube-system
ipFamilyPolicy: PreferDualStack
ports:
- port: 80
  name: web
  targetPort: web
  protocol: TCP
- port: 443
  name: websecure
  targetPort: websecure
  protocol: TCP

The live Service has loadBalancerIP: 99.110.50.213 only because someone hand-applied over the Helm-managed object. That history is recorded in its own annotation:

kubernetes.io/change-cause: Update loadBalancerIP to public IP

Why this bites

Any edit to HelmChartConfig/traefik causes k3s to run a Helm upgrade job. That upgrade re-renders the Service from the chart, which drops loadBalancerIP, which drops the public IP binding for every public hostname on the cluster. As of 2026-08-24 that is 20+ ingresses across authelia, aws-ssm-mcp, bluesky-mcp, coilysiren-eco-app, coilysiren-eco-gnome, discord-mcp, factory-game, forgejo, forgejo-mcp, galaxy-gen, glama-mcp, node-stats-mcp, registry, skillsmp-mcp, steam-mcp, trello-mcp, and website.

The failure mode is nasty because the HelmChartConfig looks like it pins the IP. Nothing about reading the file suggests the value is inert.

The same trap applies to a k3s upgrade or anything else that re-runs the helm-install-traefik job, independent of whether a human edits the config.

Fix

Move the key under service.spec so the chart honors it, and so the live Service stops depending on a hand-applied patch:

spec:
  valuesContent: |-
    service:
      spec:
        loadBalancerIP: "99.110.50.213"
    additionalArguments:
      - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s"

Confirm service.spec passthrough exists in chart 34.2.1 before relying on it, rather than trusting this issue. Verify by rendering, not by applying: check that the re-rendered Service still carries loadBalancerIP: 99.110.50.213 before letting the upgrade job touch the live object.

Until that lands, anyone editing HelmChartConfig/traefik for any reason must make this correction in the same edit.

How it was found

While preparing the externalTrafficPolicy: Local change for #909. The HelmChartConfig was the obvious route for that change and would have caused a cluster-wide ingress outage. That change was instead prepared as a direct Service apply, which does not trigger a Helm upgrade. See #909 for the incident this came out of.

Discovered by inspection, not by triggering it. No outage occurred.

## The landmine `HelmChartConfig/traefik` in `kube-system` on kai-server currently carries: ```yaml spec: valuesContent: |- service: loadBalancerIP: "99.110.50.213" additionalArguments: - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s" ``` `service.loadBalancerIP` is a **dead key** in traefik chart 34.x. It moved to `service.spec.loadBalancerIP`. The chart ignores the old spelling silently, with no warning and no error. Proof, from decoding the live Helm release `sh.helm.release.v1.traefik.v3` (chart `traefik-34.2.1+up34.2.0`). The release config contains `service.loadBalancerIP: 99.110.50.213`, and the manifest the chart actually rendered contains no `loadBalancerIP` field at all: ```yaml # chart-rendered Service spec, verbatim type: LoadBalancer selector: app.kubernetes.io/name: traefik app.kubernetes.io/instance: traefik-kube-system ipFamilyPolicy: PreferDualStack ports: - port: 80 name: web targetPort: web protocol: TCP - port: 443 name: websecure targetPort: websecure protocol: TCP ``` The live Service has `loadBalancerIP: 99.110.50.213` only because someone hand-applied over the Helm-managed object. That history is recorded in its own annotation: ``` kubernetes.io/change-cause: Update loadBalancerIP to public IP ``` ## Why this bites Any edit to `HelmChartConfig/traefik` causes k3s to run a Helm upgrade job. That upgrade re-renders the Service from the chart, which drops `loadBalancerIP`, which drops the public IP binding for every public hostname on the cluster. As of 2026-08-24 that is 20+ ingresses across authelia, aws-ssm-mcp, bluesky-mcp, coilysiren-eco-app, coilysiren-eco-gnome, discord-mcp, factory-game, forgejo, forgejo-mcp, galaxy-gen, glama-mcp, node-stats-mcp, registry, skillsmp-mcp, steam-mcp, trello-mcp, and website. The failure mode is nasty because the HelmChartConfig *looks* like it pins the IP. Nothing about reading the file suggests the value is inert. The same trap applies to a k3s upgrade or anything else that re-runs the `helm-install-traefik` job, independent of whether a human edits the config. ## Fix Move the key under `service.spec` so the chart honors it, and so the live Service stops depending on a hand-applied patch: ```yaml spec: valuesContent: |- service: spec: loadBalancerIP: "99.110.50.213" additionalArguments: - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s" ``` Confirm `service.spec` passthrough exists in chart 34.2.1 before relying on it, rather than trusting this issue. Verify by rendering, not by applying: check that the re-rendered Service still carries `loadBalancerIP: 99.110.50.213` before letting the upgrade job touch the live object. Until that lands, **anyone editing `HelmChartConfig/traefik` for any reason must make this correction in the same edit.** ## How it was found While preparing the `externalTrafficPolicy: Local` change for #909. The HelmChartConfig was the obvious route for that change and would have caused a cluster-wide ingress outage. That change was instead prepared as a direct Service apply, which does not trigger a Helm upgrade. See #909 for the incident this came out of. Discovered by inspection, not by triggering it. No outage occurred.

coilyco-ops referenced this issue

2026-08-24 20:31:08 +00:00

Forgejo OOM crash loop under faceted issue-query flood, 2026-08-24 #909

coilyco-ops changed title from ~~Editing HelmChartConfig/traefik would drop the public loadBalancerIP for every service~~ to HelmChartConfig/traefik carries a dead service.loadBalancerIP key, and hand-applied Service drift

2026-08-24 20:36:55 +00:00

coilyco-ops commented

2026-08-24 20:37:20 +00:00

Author

Owner

Severity correction: this is drift, not an outage risk

I filed this claiming a Helm upgrade would "drop the public IP binding for every public hostname on the cluster." That is wrong. I traced the dead key correctly and then asserted an impact I had not traced at all. Retitled accordingly. The original description's "Why this bites" section should be read as retracted.

The dead-key finding stands. The consequence does not.

What actually publishes the public IP

spec.loadBalancerIP on the traefik Service participates in nothing. Traced end to end:

1. k3s ServiceLB ignores the field. The Service carries loadBalancerIP: 99.110.50.213, but its status advertises the node's InternalIP:

status:
  loadBalancer:
    ingress:
    - ip: 100.69.164.66
      ipMode: VIP

klipper-lb advertises node IPs and does not implement spec.loadBalancerIP assignment. Every Ingress inherits that same address, 100.69.164.66.

2. external-dns never reads this Service. It runs --source=ingress, not --source=service:

--source=ingress  --provider=aws  --domain-filter=coilysiren.me
--label-filter=external-dns.coilysiren.me/managed=true
--registry=txt  --txt-prefix=extdns-  --policy=upsert-only

3. The public IP comes from a per-ingress annotation. Each managed Ingress pins its own target, which is why the published record is 99.110.50.213 and not the 100.69.164.66 in ingress status:

# ingress/website/website
external-dns.alpha.kubernetes.io/hostname: website.coilysiren.me
external-dns.alpha.kubernetes.io/target: 99.110.50.213

Confirmed in DNS:

website.coilysiren.me   A    99.110.50.213
extdns-website...       TXT  "heritage=external-dns,...,external-dns/resource=ingress/website/website"

4. Forgejo is not even external-dns managed. Its Ingress carries no labels, and extdns-forgejo.coilysiren.me has no TXT record. forgejo.coilysiren.me A 99.110.50.213 is a manual Route53 record. 15 of 26 ingresses are external-dns managed; forgejo is not among them (the forgejo match in the managed set is forgejo-mcp).

So the real chain is: DNS (per-ingress annotation, or manual) -> 99.110.50.213 is the router WAN IP -> port-forward 80/443 -> kai-server -> klipper-lb hostPort -> traefik. The traefik Service's loadBalancerIP field appears nowhere in it. Deleting it would be cosmetic.

The dead key itself, now properly evidenced

Confirmed from the chart embedded in the release secret rather than from the version number. Chart default service block:

additionalServices: {}
annotations: {}
enabled: true
externalIPs: []
labels: {}
loadBalancerSourceRanges: []
single: true
spec: {}          # <- the passthrough
type: LoadBalancer

No loadBalancerIP key exists. And templates/_service.tpl references .service.loadBalancerClass, .service.spec, .service.loadBalancerSourceRanges, .service.externalIPs, .service.ipFamilyPolicy, .service.ipFamilies -- and never .service.loadBalancerIP. service.spec is dumped verbatim:

{{- with .service.spec }}
{{- toYaml . | nindent 2 }}

That also resolves the open question from the original description: service.spec passthrough is real in 34.2.1, so it is the correct home for both loadBalancerIP and externalTrafficPolicy.

What is actually worth fixing

Two real but mild problems:

A comprehension hazard. The HelmChartConfig reads as though it pins the public IP. It does not, and it never has. The next person to touch it will reasonably believe otherwise, and may make a change premised on it. Delete the key or move it under service.spec so the file states something true.
Hand-applied Service drift. The live Service carries two hand-applied fields the chart does not know about: loadBalancerIP (inert) and now externalTrafficPolicy: Local (applied 2026-08-24, see #909). A Helm upgrade reverts both. The externalTrafficPolicy revert is the one with actual behavior attached, and per #909 that transition costs roughly 12 minutes of degraded in-cluster access to public hostnames while the ServiceLB pod is recreated. So a routine k3s upgrade carries a small unannounced disruption, which is worth folding into service.spec to make it survive.

Neither is urgent. Neither is the cluster-wide outage I originally described.

## Severity correction: this is drift, not an outage risk I filed this claiming a Helm upgrade would "drop the public IP binding for every public hostname on the cluster." **That is wrong.** I traced the dead key correctly and then asserted an impact I had not traced at all. Retitled accordingly. The original description's "Why this bites" section should be read as retracted. The dead-key finding stands. The consequence does not. ### What actually publishes the public IP `spec.loadBalancerIP` on the traefik Service participates in **nothing**. Traced end to end: **1. k3s ServiceLB ignores the field.** The Service carries `loadBalancerIP: 99.110.50.213`, but its status advertises the node's InternalIP: ``` status: loadBalancer: ingress: - ip: 100.69.164.66 ipMode: VIP ``` klipper-lb advertises node IPs and does not implement `spec.loadBalancerIP` assignment. Every Ingress inherits that same address, `100.69.164.66`. **2. external-dns never reads this Service.** It runs `--source=ingress`, not `--source=service`: ``` --source=ingress --provider=aws --domain-filter=coilysiren.me --label-filter=external-dns.coilysiren.me/managed=true --registry=txt --txt-prefix=extdns- --policy=upsert-only ``` **3. The public IP comes from a per-ingress annotation.** Each managed Ingress pins its own target, which is why the published record is `99.110.50.213` and not the `100.69.164.66` in ingress status: ```yaml # ingress/website/website external-dns.alpha.kubernetes.io/hostname: website.coilysiren.me external-dns.alpha.kubernetes.io/target: 99.110.50.213 ``` Confirmed in DNS: ``` website.coilysiren.me A 99.110.50.213 extdns-website... TXT "heritage=external-dns,...,external-dns/resource=ingress/website/website" ``` **4. Forgejo is not even external-dns managed.** Its Ingress carries no labels, and `extdns-forgejo.coilysiren.me` has no TXT record. `forgejo.coilysiren.me A 99.110.50.213` is a manual Route53 record. 15 of 26 ingresses are external-dns managed; `forgejo` is not among them (the `forgejo` match in the managed set is `forgejo-mcp`). So the real chain is: DNS (per-ingress annotation, or manual) -> `99.110.50.213` is the router WAN IP -> port-forward 80/443 -> kai-server -> klipper-lb hostPort -> traefik. The traefik Service's `loadBalancerIP` field appears nowhere in it. Deleting it would be cosmetic. ### The dead key itself, now properly evidenced Confirmed from the chart embedded in the release secret rather than from the version number. Chart default `service` block: ```yaml additionalServices: {} annotations: {} enabled: true externalIPs: [] labels: {} loadBalancerSourceRanges: [] single: true spec: {} # <- the passthrough type: LoadBalancer ``` No `loadBalancerIP` key exists. And `templates/_service.tpl` references `.service.loadBalancerClass`, `.service.spec`, `.service.loadBalancerSourceRanges`, `.service.externalIPs`, `.service.ipFamilyPolicy`, `.service.ipFamilies` -- and **never `.service.loadBalancerIP`**. `service.spec` is dumped verbatim: ``` {{- with .service.spec }} {{- toYaml . | nindent 2 }} ``` That also resolves the open question from the original description: `service.spec` passthrough is real in 34.2.1, so it is the correct home for both `loadBalancerIP` and `externalTrafficPolicy`. ### What is actually worth fixing Two real but mild problems: 1. **A comprehension hazard.** The HelmChartConfig reads as though it pins the public IP. It does not, and it never has. The next person to touch it will reasonably believe otherwise, and may make a change premised on it. Delete the key or move it under `service.spec` so the file states something true. 2. **Hand-applied Service drift.** The live Service carries two hand-applied fields the chart does not know about: `loadBalancerIP` (inert) and now `externalTrafficPolicy: Local` (applied 2026-08-24, see #909). A Helm upgrade reverts both. The `externalTrafficPolicy` revert is the one with actual behavior attached, and per #909 that transition costs roughly 12 minutes of degraded in-cluster access to public hostnames while the ServiceLB pod is recreated. So a routine k3s upgrade carries a small unannounced disruption, which is worth folding into `service.spec` to make it survive. Neither is urgent. Neither is the cluster-wide outage I originally described.

coilyco-ops commented

2026-08-25 00:08:58 +00:00

Author

Owner

Fix prepared and dry-run verified, blocked on permission

Kai sequenced this ahead of the Traefik analytics work, because enabling access logs and fixing the metrics scrape both require editing this HelmChartConfig, and doing that today would silently revert the externalTrafficPolicy: Local applied in #909.

Prepared, dry-run verified, and blocked by the operator permission classifier. Not applied.

The dry run, rendered rather than reasoned

Previous comments argued from the chart template. This renders it. The chart is embedded in the release secret sh.helm.release.v1.traefik.v3, so it can be reconstructed and rendered locally with the exact values k3s supplies (release.config, which is the k3s base merged with this HelmChartConfig).

Rendered Service spec under current values:

type: LoadBalancer
selector: {app.kubernetes.io/name: traefik, app.kubernetes.io/instance: traefik-kube-system}
ipFamilyPolicy: PreferDualStack
ports: [web 80 -> web, websecure 443 -> websecure]

No externalTrafficPolicy. No loadBalancerIP. This confirms the drift claim directly: a Helm upgrade today reverts externalTrafficPolicy to Cluster and drops the IP field.

Rendered Service spec under proposed values:

type: LoadBalancer
externalTrafficPolicy: Local
loadBalancerIP: 99.110.50.213
selector: {...}
ipFamilyPolicy: PreferDualStack
ports: [...]

service.spec passthrough works as expected, and service.ipFamilyPolicy from the k3s base values survives the deep merge alongside it.

Blast radius: none

Diffing the entire rendered chart output between the two value sets, the only delta is two lines:

126a127,128
>   externalTrafficPolicy: Local
>   loadBalancerIP: 99.110.50.213

No Deployment change, no RBAC change, nothing else.

Diffing the proposed render against the live cluster (kubectl diff on the rendered Service and Deployment), the only difference is the annotation added by hand in #909:

-    kubernetes.io/change-cause: Set externalTrafficPolicy=Local to preserve client
-      source IP through klipper-lb (infrastructure#909)

So the Helm upgrade this triggers is a genuine no-op against live. No Deployment roll, no Service mutation, and therefore none of the roughly 12 minute svclb recreation window that the direct Service apply cost in #909. That window happened because the Service spec actually changed; here it does not.

The change

kubectl diff on the CR itself:

-      loadBalancerIP: "99.110.50.213"
+      spec:
+        loadBalancerIP: "99.110.50.213"
+        externalTrafficPolicy: Local

Full manifest, recorded so it survives the session:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    service:
      spec:
        # Chart 34.x reads these from service.spec only. The old top-level
        # service.loadBalancerIP was silently ignored (infrastructure#910).
        # loadBalancerIP is inert under k3s ServiceLB, which advertises node
        # IPs regardless; kept so a helm upgrade is a no-op against live.
        loadBalancerIP: "99.110.50.213"
        externalTrafficPolicy: Local
    additionalArguments:
      - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s"

loadBalancerIP is deliberately kept rather than deleted, even though it is inert. Keeping it makes the rendered Service byte-match live, which is what makes the upgrade a no-op. Removing it is a separate, deliberate decision and should not ride along with this one.

After applying

The k3s helm-install-traefik job runs a Helm upgrade. Expected observations:

Service externalTrafficPolicy stays Local, loadBalancerIP stays 99.110.50.213.
nodePorts stay 31400 and 31428. The rendered manifest omits them and Helm's three-way merge leaves them untouched, since they are absent from both the old and new release manifests.
No new svclb-traefik pod, because the Service spec does not change.
Public edge unaffected.

Rollback is rollback-helmchartconfig-traefik-current.yaml, or restoring the two-line service.loadBalancerIP form.

What this unblocks

Once landed, HelmChartConfig/traefik is safe to edit, which is the prerequisite for the Traefik observability work: correcting the metrics scrape port and enabling access logs. Both are currently dark. See the analytics discussion for detail.

## Fix prepared and dry-run verified, blocked on permission Kai sequenced this ahead of the Traefik analytics work, because enabling access logs and fixing the metrics scrape both require editing this HelmChartConfig, and doing that today would silently revert the `externalTrafficPolicy: Local` applied in #909. Prepared, dry-run verified, and **blocked by the operator permission classifier**. Not applied. ## The dry run, rendered rather than reasoned Previous comments argued from the chart template. This renders it. The chart is embedded in the release secret `sh.helm.release.v1.traefik.v3`, so it can be reconstructed and rendered locally with the exact values k3s supplies (`release.config`, which is the k3s base merged with this HelmChartConfig). **Rendered Service spec under current values:** ```yaml type: LoadBalancer selector: {app.kubernetes.io/name: traefik, app.kubernetes.io/instance: traefik-kube-system} ipFamilyPolicy: PreferDualStack ports: [web 80 -> web, websecure 443 -> websecure] ``` No `externalTrafficPolicy`. No `loadBalancerIP`. **This confirms the drift claim directly**: a Helm upgrade today reverts `externalTrafficPolicy` to `Cluster` and drops the IP field. **Rendered Service spec under proposed values:** ```yaml type: LoadBalancer externalTrafficPolicy: Local loadBalancerIP: 99.110.50.213 selector: {...} ipFamilyPolicy: PreferDualStack ports: [...] ``` `service.spec` passthrough works as expected, and `service.ipFamilyPolicy` from the k3s base values survives the deep merge alongside it. ## Blast radius: none Diffing the **entire** rendered chart output between the two value sets, the only delta is two lines: ``` 126a127,128 > externalTrafficPolicy: Local > loadBalancerIP: 99.110.50.213 ``` No Deployment change, no RBAC change, nothing else. Diffing the proposed render against the **live cluster** (`kubectl diff` on the rendered Service and Deployment), the only difference is the annotation added by hand in #909: ``` - kubernetes.io/change-cause: Set externalTrafficPolicy=Local to preserve client - source IP through klipper-lb (infrastructure#909) ``` So the Helm upgrade this triggers is a genuine no-op against live. **No Deployment roll, no Service mutation, and therefore none of the roughly 12 minute svclb recreation window** that the direct Service apply cost in #909. That window happened because the Service spec actually changed; here it does not. ## The change `kubectl diff` on the CR itself: ``` - loadBalancerIP: "99.110.50.213" + spec: + loadBalancerIP: "99.110.50.213" + externalTrafficPolicy: Local ``` Full manifest, recorded so it survives the session: ```yaml apiVersion: helm.cattle.io/v1 kind: HelmChartConfig metadata: name: traefik namespace: kube-system spec: valuesContent: |- service: spec: # Chart 34.x reads these from service.spec only. The old top-level # service.loadBalancerIP was silently ignored (infrastructure#910). # loadBalancerIP is inert under k3s ServiceLB, which advertises node # IPs regardless; kept so a helm upgrade is a no-op against live. loadBalancerIP: "99.110.50.213" externalTrafficPolicy: Local additionalArguments: - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s" ``` `loadBalancerIP` is deliberately **kept rather than deleted**, even though it is inert. Keeping it makes the rendered Service byte-match live, which is what makes the upgrade a no-op. Removing it is a separate, deliberate decision and should not ride along with this one. ## After applying The k3s `helm-install-traefik` job runs a Helm upgrade. Expected observations: * Service `externalTrafficPolicy` stays `Local`, `loadBalancerIP` stays `99.110.50.213`. * nodePorts stay 31400 and 31428. The rendered manifest omits them and Helm's three-way merge leaves them untouched, since they are absent from both the old and new release manifests. * No new `svclb-traefik` pod, because the Service spec does not change. * Public edge unaffected. Rollback is `rollback-helmchartconfig-traefik-current.yaml`, or restoring the two-line `service.loadBalancerIP` form. ## What this unblocks Once landed, `HelmChartConfig/traefik` is safe to edit, which is the prerequisite for the Traefik observability work: correcting the metrics scrape port and enabling access logs. Both are currently dark. See the analytics discussion for detail.

coilyco-ops commented

2026-08-25 00:18:29 +00:00

Author

Owner

Applied and verified. Resolved.

Kai applied it at 00:14 UTC on 2026-08-25. The k3s helm-install-traefik job ran a Helm upgrade and completed. Every prediction from the dry run held.

After-state

1. The hand-applied field survived a real Helm upgrade. This is the whole point of the change, and it is now demonstrated rather than modelled:

externalTrafficPolicy: Local
loadBalancerIP: 99.110.50.213

Before this change, the same upgrade would have reverted the policy to Cluster and dropped the IP, as the rendered dry run showed.

2. Allocations preserved, as predicted for Helm's three-way merge with nodePort absent from both old and new release manifests:

nodePort: 31400
nodePort: 31428
healthCheckNodePort: 30975
clusterIP: 10.43.103.250

3. No ServiceLB recreation. Same pod as before the upgrade:

svclb-traefik-bcc27bfa-2m44w   Running   age=4h16m   ip=10.42.0.93

This is the key operational difference from #909. That change mutated the Service spec and cost roughly 12 minutes of degraded in-cluster access while klipper-lb rebuilt. This one did not touch the spec, so it cost nothing.

4. No Deployment roll. traefik-59f5fc745f-8rphd, 27d old, unchanged.

5. Public edge healthy. All six sampled hostnames returned 200. First probes ran 0.21-0.43s, elevated against the 0.026-0.13s baseline, but a re-probe 20 seconds later settled to normal:

forgejo.coilysiren.me   0.116  0.060  0.078
website.coilysiren.me   0.073  0.064  0.055
auth.coilysiren.me      0.041  0.041  0.040

Cold connections immediately after the Helm job, not a regression.

6. Nothing else disturbed. Forgejo forgejo-6fb99c98fc-8hsfp at 0 restarts and 5h22m. Both Flux GitRepositories READY=True.

Disposition

The dead key is gone, the values now render a Service that matches live, and HelmChartConfig/traefik is safe to edit. Closing.

Two things deliberately left undone, neither blocking:

loadBalancerIP is retained under service.spec and remains functionally inert, since k3s ServiceLB advertises node IPs regardless. It is now at least in the location the chart reads, and the config carries a comment saying so. Deleting it is a separate decision.
The Forgejo Deployment is still hand-applied and outside GitOps. That belongs to #909, not here.

Unblocks the Traefik observability work: the metrics scrape port correction and enabling access logs, both of which require editing this HelmChartConfig and are currently dark.

## Applied and verified. Resolved. Kai applied it at 00:14 UTC on 2026-08-25. The k3s `helm-install-traefik` job ran a Helm upgrade and completed. Every prediction from the dry run held. ### After-state **1. The hand-applied field survived a real Helm upgrade.** This is the whole point of the change, and it is now demonstrated rather than modelled: ``` externalTrafficPolicy: Local loadBalancerIP: 99.110.50.213 ``` Before this change, the same upgrade would have reverted the policy to `Cluster` and dropped the IP, as the rendered dry run showed. **2. Allocations preserved**, as predicted for Helm's three-way merge with nodePort absent from both old and new release manifests: ``` nodePort: 31400 nodePort: 31428 healthCheckNodePort: 30975 clusterIP: 10.43.103.250 ``` **3. No ServiceLB recreation.** Same pod as before the upgrade: ``` svclb-traefik-bcc27bfa-2m44w Running age=4h16m ip=10.42.0.93 ``` This is the key operational difference from #909. That change mutated the Service spec and cost roughly 12 minutes of degraded in-cluster access while klipper-lb rebuilt. This one did not touch the spec, so it cost nothing. **4. No Deployment roll.** `traefik-59f5fc745f-8rphd`, 27d old, unchanged. **5. Public edge healthy.** All six sampled hostnames returned 200. First probes ran 0.21-0.43s, elevated against the 0.026-0.13s baseline, but a re-probe 20 seconds later settled to normal: ``` forgejo.coilysiren.me 0.116 0.060 0.078 website.coilysiren.me 0.073 0.064 0.055 auth.coilysiren.me 0.041 0.041 0.040 ``` Cold connections immediately after the Helm job, not a regression. **6. Nothing else disturbed.** Forgejo `forgejo-6fb99c98fc-8hsfp` at 0 restarts and 5h22m. Both Flux GitRepositories `READY=True`. ### Disposition The dead key is gone, the values now render a Service that matches live, and `HelmChartConfig/traefik` is safe to edit. Closing. Two things deliberately left undone, neither blocking: * `loadBalancerIP` is retained under `service.spec` and remains functionally inert, since k3s ServiceLB advertises node IPs regardless. It is now at least in the location the chart reads, and the config carries a comment saying so. Deleting it is a separate decision. * The Forgejo Deployment is still hand-applied and outside GitOps. That belongs to #909, not here. Unblocks the Traefik observability work: the metrics scrape port correction and enabling access logs, both of which require editing this HelmChartConfig and are currently dark.

coilyco-ops closed this issue

2026-08-25 00:18:36 +00:00

coilyco-ops referenced this issue

2026-08-26 02:29:33 +00:00

Forgejo OOM crash loop under faceted issue-query flood, 2026-08-24 #909

coilyco-ops referenced this issue from a commit

2026-08-26 02:59:38 +00:00

fix(k3s): restore traefik HelmChartConfig parity with live

coilyco-ops referenced this issue from a commit

2026-08-26 02:59:38 +00:00

feat(k3s): bypass ServiceLB so Traefik sees real client IPs

coilyco-ops referenced this issue