feat(coredns): declare the hairpin fix both clusters are running undeclared #976

Merged

coilyco-ops merged 1 commit from aos/claude/mt75-coredns-hairpin into main

2026-08-28 18:59:35 +00:00

coilyco-ops commented

2026-08-28 18:58:12 +00:00

Owner

Closes the durability gap Vera recorded in agentic-os#1344. She applied the fix by hand to both clusters; this declares it so a rebuild cannot lose it.

Why it was drift

Both clusters run Flux. The applies were imperative, so neither ConfigMap is declared anywhere. Nothing prunes them, so nothing was going to break today - but a cluster rebuild loses the fix silently and brings the hairpin back.

What the key does

forgejo.coilysiren.me resolves to its tailnet address (100.69.164.66), so in-cluster traffic stops leaving the estate and transiting the router to reach a service on the same estate. Masqueraded once at flannel and again at the gateway, that hairpin degrades under concurrency rather than failing cleanly - which is why it presented as three 133-second connect timeouts in actions/checkout beside a clean HTTP 404 from the same host two minutes later.

coredns-custom rather than the Corefile ConfigMap, because the latter is owned by a k3s Addon and reverts hand-edits. kai-server has already lost api.coilysiren.me and eco.coilysiren.me hosts blocks exactly that way; they survive only in a stale last-applied-configuration annotation.

Transcribed from live state, not from the issue

This mattered. kai-server's ConfigMap also carries ser8-observability.server, a tailnet forwarder unrelated to this incident. A manifest written from the issue text alone would have declared only forgejo.server, and Flux would then have deleted a working forwarder. Both keys are read out of the live object.

ser8 had no coredns-custom at all before #1344, so its file is the whole content.

Adoption is a no-op, proven

kubectl diff -k deploy/_flux/coredns-kai-server --context kai-server   -> empty, exit 0
kubectl diff -k deploy/_flux/coredns-ser8       --context ser8         -> empty, exit 0

Declared content already equals live on both clusters, so Flux takes ownership without changing a byte. The cluster-level Kustomization CRs were validated the same way: a server-side dry-run against ser8 renders coredns as a clean addition with nothing else in the cluster dir differing, so the CR is schema-valid per the API server rather than per my reading.

prune: false matches every other Kustomization in this repo. This change adopts objects that already exist, and arming a prune on a CoreDNS input in the same commit that declares it is not a trade worth making. Promote it the way infrastructure.yaml documents: observe, then flip.

What this does not claim

It does not explain the incident. CI recovered on its own at 18:15Z, roughly twenty minutes before Vera's first apply, and she caught that herself rather than reporting a confirmed fix. Both ends of the window - 17:42Z breaking, ~18:15Z recovering - remain unexplained.

The fix prevents recurrence. It did not resolve this episode.

Scope

forgejo.coilysiren.me only, per Kai. No other name is touched on either cluster.

Verification: pre-commit run --all-files green, including yamllint and the k3s guards.

Closes the durability gap Vera recorded in agentic-os#1344. She applied the fix by hand to both clusters; this declares it so a rebuild cannot lose it. ## Why it was drift Both clusters run Flux. The applies were imperative, so neither ConfigMap is declared anywhere. Nothing prunes them, so nothing was going to break today - but a cluster rebuild loses the fix silently and brings the hairpin back. ## What the key does `forgejo.coilysiren.me` resolves to its tailnet address (`100.69.164.66`), so in-cluster traffic stops leaving the estate and transiting the router to reach a service on the same estate. Masqueraded once at flannel and again at the gateway, that hairpin degrades under concurrency rather than failing cleanly - which is why it presented as three 133-second connect timeouts in `actions/checkout` beside a clean HTTP 404 from the same host two minutes later. `coredns-custom` rather than the `Corefile` ConfigMap, because the latter is owned by a k3s Addon and reverts hand-edits. kai-server has already lost `api.coilysiren.me` and `eco.coilysiren.me` hosts blocks exactly that way; they survive only in a stale `last-applied-configuration` annotation. ## Transcribed from live state, not from the issue This mattered. kai-server's ConfigMap also carries `ser8-observability.server`, a tailnet forwarder unrelated to this incident. A manifest written from the issue text alone would have declared only `forgejo.server`, and Flux would then have **deleted a working forwarder**. Both keys are read out of the live object. ser8 had no `coredns-custom` at all before #1344, so its file is the whole content. ## Adoption is a no-op, proven ``` kubectl diff -k deploy/_flux/coredns-kai-server --context kai-server -> empty, exit 0 kubectl diff -k deploy/_flux/coredns-ser8 --context ser8 -> empty, exit 0 ``` Declared content already equals live on both clusters, so Flux takes ownership without changing a byte. The cluster-level `Kustomization` CRs were validated the same way: a server-side dry-run against ser8 renders `coredns` as a clean addition with nothing else in the cluster dir differing, so the CR is schema-valid per the API server rather than per my reading. `prune: false` matches every other Kustomization in this repo. This change adopts objects that already exist, and arming a prune on a CoreDNS input in the same commit that declares it is not a trade worth making. Promote it the way `infrastructure.yaml` documents: observe, then flip. ## What this does not claim **It does not explain the incident.** CI recovered on its own at 18:15Z, roughly twenty minutes before Vera's first apply, and she caught that herself rather than reporting a confirmed fix. Both ends of the window - 17:42Z breaking, ~18:15Z recovering - remain unexplained. The fix prevents recurrence. It did not resolve this episode. ## Scope `forgejo.coilysiren.me` only, per Kai. No other name is touched on either cluster. Verification: `pre-commit run --all-files` green, including yamllint and the k3s guards.

coilyco-ops added 1 commit

2026-08-28 18:58:12 +00:00

feat(coredns): declare the hairpin fix both clusters are running undeclared

TruffleHog / Scan for secrets (pull_request) Successful in 4s

Details

CI / lint (pull_request) Successful in 43s

Details

081eaff474

Vera applied a `coredns-custom` key to ser8 and kai-server by hand while
closing agentic-os#1344. Both clusters run Flux, so both were drift: nothing
prunes the ConfigMaps, but nothing declares them either, and a rebuild loses
the fix silently and brings the hairpin back.

What the key does: forgejo.coilysiren.me resolves to its tailnet address, so
in-cluster traffic stops leaving the estate and transiting the router to
reach a service on the same estate. The double-NAT hairpin that produced
degraded under concurrency rather than failing cleanly, which is why it read
as 133s connect timeouts in checkout beside a clean 404 two minutes later.

`coredns-custom` rather than the Corefile ConfigMap: the latter is owned by a
k3s Addon and reverts hand-edits. kai-server has already lost `api` and `eco`
hosts blocks that way, surviving only in a stale last-applied annotation.

Transcribed from live state rather than from the issue text, which matters
for kai-server: its ConfigMap also carries `ser8-observability.server`, and a
manifest omitting it would have had Flux delete a working tailnet forwarder.

Adoption is a no-op, proven rather than assumed. `kubectl diff -k` against
both clusters returns empty, so the declared content already equals live.
`prune: false` matches every other Kustomization here: this change takes
ownership of existing objects, and arming a prune on a CoreDNS input in the
same commit that declares it is not a trade worth making.

This does not explain the incident. CI recovered on its own at 18:15Z, about
twenty minutes before the first apply, and both ends of the 17:42Z window are
still unexplained. The fix prevents recurrence; it did not resolve this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: platform

coilyco-ops merged commit b4e0ab1a60 into main

2026-08-28 18:59:35 +00:00

coilyco-ops referenced this pull request from a commit

2026-08-28 18:59:36 +00:00

feat(coredns): declare the hairpin fix both clusters are running undeclared (#976)

coilyco-ops referenced this pull request from coilyco-flight-deck/agentic-os

2026-08-28 19:00:37 +00:00

CI is blocked repo-wide: ward-doctor 404s because the install ref falls back to a release-tag path that cannot exist #1344

coilyco-ops referenced this pull request

2026-08-28 21:11:48 +00:00

Forgejo Actions runners hairpin through the public edge, 59% of all ingress traffic #912

coilyco-ops referenced this pull request