Forgejo OOM crash loop under faceted issue-query flood, 2026-08-24 #909
Labels
No labels
burndown-2026-06
burndown-2026-08
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/advocate
role/director
role/exec
role/frontend
role/gamedev
role/human
role/platform
role/qa
role/science
role/sysadmin
state
ambient
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
coilyco-flight-deck/infrastructure#909
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
forgejo/forgejo-6df4ddf4d5-f6zjhon kai-server is in an OOMKill crash loop. 46 container restarts on a pod whose current instance dates to 2026-08-12, with 5 of those restarts inside a 90-minute window on 2026-08-24. Each kill produced a user-visible outage of up to ~14 minutes because kubelet CrashLoopBackOff had grown to its 5-minute ceiling.Reported by Kai as "forgejo was just down for like 10 minutes" at 2026-08-24 18:28 UTC.
Observed before-state
Container status at 18:30 UTC, from
kubectl describe pod:Process start banners (
cmd/web.go:250:runWeb() [I] Starting Forgejo on PID: 7), instance number fromk8s.container.restart_count:The gap Kai noticed is the 18:02:30 to 18:16:37 window, ~14 minutes with no serving process.
Node was healthy throughout.
MemoryPressure=False,DiskPressure=False,PIDPressure=False,Ready=True, 32 GiB capacity. This is a container-limit OOM, not node exhaustion.Trigger
Immediately before instance 45 died, Forgejo was serving a flood of faceted issue and pull list queries across
ward,umbra, andsirens-echo, permutingassignee,labels,milestone,poster,project,state, andtype. Latency climbed from ~3s to 298s inside 9 minutes:The permutation pattern is consistent with a crawler walking every filter-combination URL. Forgejo 16 serves those combinations without a crawl guard.
Contributing conditions
The rate limiter cannot see individual clients.
forgejo/forgejo-ratelimit(94d old) isaverage: 50, burst: 100, period: 1s. Traefik's defaultsourceCriterionkeys on the direct remote address, and klipper-lb (svclb-traefik,externalTrafficPolicy: Cluster) SNATs every inbound connection to10.42.0.167. Every client on the internet therefore shares one 50 rps bucket. The limiter does not bound the crawler per-source, and it does 429 legitimate traffic that shares the bucket.Client attribution is lost. Every request in Forgejo's router log shows
for 10.42.0.167:0, the klipper-lb pod IP, despiteFORGEJO__security__REVERSE_PROXY_TRUSTED_PROXIES=10.42.0.0/16. The real source address is destroyed at the klipper-lb hop before Traefik setsX-Forwarded-For, so there is currently no way to identify or block the offending client.Blast radius
Flux GitOps reconciliation is stalled on the same failures.
bridge-deployandinfrastructureKustomizations both report empty READY status.503 during the crash windows, 429 from the shared rate-limit bucket once Forgejo is back up.
After-state at time of filing
Forgejo is up and fast:
/api/healthzreturns 200 withcache:pinganddatabase:pingbothpass, in 27-102ms. Instance 46 has held for ~7 minutes with no further restart. Container RSS 313 MB against the 1500Mi limit.No change has been applied. The conditions that produced the crash loop are all still in place, so a recurrence is expected rather than possible.
Containment still outstanding
Nothing is fixed. Candidate work, unsequenced:
externalTrafficPolicy: Localon the traefik service, or another path that preserves the source address through klipper-lb. Everything below is guesswork without this.robots.txtat minimum, and consider whether the combinations should be reachable unauthenticated at all.repo.Issuesactually costs under concurrency. Raising it alone buys time and does not stop the flood.kustomize.toolkit,helm.toolkit, ormeta.helmlabels and appears in neitherinfrastructurenordeploy. It is hand-applied, so none of the above is reviewable or reproducible today.Evidence sources
kai-server k3s API (
kubectl describe pod,get middleware,get kustomization, node conditions and events), SigNoz logs fork8s.namespace.name = 'forgejo', metrics-server container memory, and direct HTTPS probes offorgejo.coilysiren.me.Change 1 of 2 applied: Forgejo memory limit 1500Mi -> 3Gi
Applied 2026-08-24 18:55:36 UTC. Kai approved the memory bump plus source attribution, and declined the crawl guard for now.
kubectl diffbefore apply showed exactly one spec change, memory limit only. cpu, ephemeral-storage, and all requests untouched.Deployment strategy is
Recreate, so this replaced the pod.forgejo-6df4ddf4d5-f6zjh(46 restarts) is gone, replaced byforgejo-6fb99c98fc-8hsfp.Node headroom at time of change: 19.98 GiB available of 33.3 GiB, 40% used. The extra 1.5Gi ceiling is comfortably affordable.
After-state, verified end-to-end rather than by probe alone
The
git-upload-packpath is the one Flux was failing on with 503 and 429. It now returns 200.As of 19:01 UTC the new pod has held 5m48s with 0 restarts.
Rollback:
kubectl apply -fthe capturedrollback-deployment-forgejo.yaml, or set the limit back to 1500Mi.Change 2 of 2 NOT applied: traefik externalTrafficPolicy
Prepared, diffed, and blocked by the operator permission classifier. Not applied. This is unresolved and the source-attribution half of the fix is still outstanding.
A trap found while preparing it, worth recording
The obvious route, editing
HelmChartConfig/traefik, would have taken down public ingress for every service on the cluster.Decoding the live Helm release (
sh.helm.release.v1.traefik.v3, charttraefik-34.2.1+up34.2.0) shows the user-supplied values containservice.loadBalancerIP: 99.110.50.213, but the chart-rendered Service spec contains noloadBalancerIPat all:Traefik chart 34.x moved that key to
service.spec.loadBalancerIP. The old top-level key is silently ignored. The public IP is present on the live Service only because someone hand-applied over the Helm-managed object, which is what thekubernetes.io/change-cause: Update loadBalancerIP to public IPannotation records.Editing the HelmChartConfig triggers a Helm upgrade, which would re-render the Service without
loadBalancerIPand drop the public IP binding for all 20+ public hostnames.Anyone touching the traefik HelmChartConfig must move
loadBalancerIPunderservice.specin the same edit. This is a live landmine independent of this incident and deserves its own issue.The prepared change, direct Service apply
Chosen because the Service is already hand-managed, so this matches the existing pattern and does not trigger a Helm upgrade. It is not durable across a future Helm upgrade, but neither is the current
loadBalancerIP, so it makes nothing worse.kubectl diffconfirmed the change is scoped to the policy flip, its auto-allocatedhealthCheckNodePort, and the change-cause annotation.loadBalancerIP, ports, nodePorts (31400/31428), and selector all unchanged.Full manifest, recorded here so it survives the session:
Safety argument: kai-server is the only node in this cluster and it runs the traefik pod, so
Localcannot blackhole traffic to a node without a backend. Expected impact is a brief blip while kube-proxy and klipper-lb reprogram.Pre-change baseline for comparison after it lands, all via the public IP 99.110.50.213:
Residual risk
The memory bump raises the OOM ceiling. It does not stop the crawler. If the flood returns at the same intensity, Forgejo may still reach 3Gi.
Once
externalTrafficPolicy: Locallands, the existingforgejo-ratelimitmiddleware becomes per-source automatically, because Traefik's defaultsourceCriterionkeys on the remote address, which will finally be the real client. That stops Flux and other legitimate traffic taking collateral 429s. But 50 rps per source is still enough for a single crawler to hurt Forgejo, so the rate limit likely wants lowering, and item 3 in the original description, the crawl guard, remains the actual fix for the trigger.Correction: change 2 landed but did NOT achieve source attribution
Kai applied the traefik Service change at ~20:00 UTC. It applied cleanly and broke nothing. It also did not do what I said it would. Correcting the record.
What I claimed, and why it was wrong
In the previous comment I wrote:
That is false. Client source IP is still not preserved.
Evidence
The policy change caused k3s to recreate the ServiceLB pod. It came back with a new name and a new IP:
Forgejo's router log post-change, timestamps after the apply:
Every external request is still masked. The masking IP simply moved from the old ServiceLB pod
10.42.0.167to the new one10.42.0.93.Why
k3s ServiceLB (klipper-lb) forwards traffic through a pod's network namespace, so the packet is SNATed to that pod's IP before it ever reaches Traefik.
externalTrafficPolicy: Localgoverns which nodes are eligible to receive traffic. It does not remove the klipper-lb hop, so it cannot preserve the source address here. On a single-node cluster it changes effectively nothing.The rate limiter therefore still keys the entire internet into one shared 50 rps bucket, exactly as described in the original report. Nothing about that has improved.
What would actually work
Source preservation requires removing the klipper-lb hop, not reconfiguring around it:
hostPorton 80/443, orhostNetwork: true, and disable ServiceLB for that Service. Traefik then binds the node's ports directly and sees the router's DNAT source address. Simplest option for a single-node cluster.externalTrafficPolicy: Localfor source preservation. Heavier, and more moving parts than this cluster needs.Both need their own change window. Neither is done.
Current state of the applied change
Verified against the pre-change baseline, all via the public IP:
loadBalancerIP: 99.110.50.213, nodePorts 31400/31428, and the selector all survived as intended.healthCheckNodePort: 30975was allocated.The change is harmless and is a reasonable prerequisite for option 1 above, so leaving it in place is defensible. Reverting it is equally defensible since it currently buys nothing. Rollback source is
rollback-svc-traefik.yaml, or setexternalTrafficPolicyback toCluster.Forgejo itself is holding
forgejo-6fb99c98fc-8hsfp, 0 restarts at 66 minutes, against 46 restarts on the previous pod. The 3Gi limit is doing its job so far.Confirmed the git path is genuinely working rather than returning an HTML page:
Flux, partial recovery
bridge-deployGitRepository recovered:stored artifact for revision 'main@sha1:be74966d852b73d186415e3bdf43377154959cda'.flux-systemGitRepository (the infrastructure repo) is still failing, and the error is current as of 20:02:09 UTC, not stale:The failure mode changed from 503 and 429 to timeout. Its spec sets
timeout: 60sand it clones withsecretRef: flux-system, so this is an authenticated full clone exceeding 60s, a different code path from the unauthenticated advertisement that returns in 25ms for me. Whether that is repo size, residual Forgejo slowness, or something else is not yet established.Worth noting for anyone reading this as a GitOps outage:
infrastructureandbridge-deployKustomizations both havesuspend: trueandobservedGeneration: -1. They are deliberately paused and were before this incident. Theflux-systemandappsKustomizations areREADY=Trueonmain@sha1:b5fbbf45. So nothing is auto-applying regardless, and the stale GitRepository is not currently blocking a deploy.Resolved: the Flux timeouts were transient, no revert needed
Correcting my own call
Mid-investigation I concluded the traefik change had caused "textbook
externalTrafficPolicy: Localhairpin breakage" and started preparing a revert. That was premature. I asserted a persistent break from a correlation without first checking whether the failure was still occurring. It was not.What the timeouts actually were
Failure-mode timeline from source-controller, bucketed:
The traefik apply caused k3s to recreate the ServiceLB pod (
10.42.0.167->10.42.0.93). In-cluster traffic to the public hostname broke for roughly 12 minutes while NAT and conntrack state rebuilt against the new pod, then recovered on its own. A disruption window during the change, not a standing break.The two 429s at 20:20 are the original shared-bucket problem, unrelated to this.
Current state, 20:30 UTC
Both GitRepositories recovered with fresh artifacts:
Kustomizations
flux-systemandappsareREADY=True.infrastructureandbridge-deployremain blank because they carrysuspend: true, which predates this incident and is deliberate.Forgejo:
forgejo-6fb99c98fc-8hsfp, 0 restarts at 95 minutes, against 46 restarts on the pod it replaced.Public edge unchanged:
Disposition
Kai chose to leave
externalTrafficPolicy: Localin place. That decision stands and needs no revisiting. A revert manifest was prepared and diffed but not applied, and is no longer warranted.One correction to my earlier assessment stands, though: I described the change as harmless. It is harmless in steady state, but applying it cost ~12 minutes of degraded in-cluster access to public hostnames, because it recreates the ServiceLB pod. Anyone changing
externalTrafficPolicyon this Service in future should expect that window rather than treating it as a no-op edit.Incident closed, containment not
Forgejo is stable and Flux is reconciling. Nothing in this incident is fixed at the cause level:
externalTrafficPolicy: Localdoes not fix that, as established above. It needs traefik onhostPort/hostNetworkwith ServiceLB bypassed, or MetalLB.HelmChartConfig/traefikedit or k3s upgrade drops the public IP for every hostname.The crawler ignores robots.txt, so item 3 needs rewriting
Kai asked whether the crawler is actually giving useful visibility. Checked it, and the answer changes the recommended fix.
Forgejo already ships a restrictive robots.txt
Served live at
forgejo.coilysiren.me/robots.txt, the Forgejo default, including:It is being ignored
Classifying 1010 recent request paths from the router log against those rules:
The client fetched
/robots.txtat 20:43:10 UTC and is still generating 27.7% disallowed traffic. It reads the file and does not honor it.It is not a search engine
The request-shape distribution rules that out. 60% raw/blame/commit plus 22%
/src/at pinned tags and commits, against 1.3% repo landing pages. Search indexing wants landing pages, descriptions, and READMEs. This wants file contents at every revision, which is the signature of source harvesting rather than indexing.Sample of the live pattern:
Deep file content at a pinned tag, under a path robots.txt disallows.
Consequence for the fix
Item 3 in the description said "robots.txt at minimum". That is now known-insufficient and should be treated as retracted. The policy it would express is already published and already ignored, so adding rules to it changes nothing for this client.
The guard has to be enforced rather than advisory:
/*/*/issues?*and/*/*/pulls?*permutations, which are the queries that actually drove the OOM viarepo.Issues./*/*/src/and/api/paths, since publishing that policy has demonstrably not worked.Worth stating plainly for anyone weighing this against SEO: blocking the faceted query permutations costs zero search visibility. Those URLs are an infinite duplicate-content space that search engines penalize rather than reward, and repo landing pages, READMEs, and ordinary code browsing stay reachable for well-behaved crawlers. Discovery for these projects happens on the GitHub mirrors regardless.
Caveat
Client attribution is still masked by klipper-lb (now
10.42.0.93), so all of the above describes aggregate traffic and cannot be attributed to a single client with certainty. A well-behaved crawler and an abusive one are indistinguishable by source address today. That limitation is unchanged and is why the source-IP work still matters.Posture decision: steer, do not block
Kai decided 2026-08-24: source harvesting is acceptable. The only thing to stop is harvesters reaching endpoints that are expensive to serve.
This narrows the guard and retracts the broader option from my previous comment, which offered to "extend enforcement to the already-disallowed
/*/*/src/and/api/paths." That is declined. Code browsing, raw file access, and/src/stay open.What remains in scope for this issue
A narrow enforced guard on the faceted issue and pull query permutation space only:
These are what drove
repo.Issuesto 298336ms and OOM-killed the process. They are an unbounded-cost endpoint rather than content anyone would want indexed, and blocking the permutations costs nothing discoverable because the unfiltered issue and pull pages stay reachable.Everything else in the original item 3 is dropped.
The complementary half
The "steer" side is now designed in
coilyco-bridge/deploy#785: serverobots.txtandllms.txtat the infra level across public surfaces, giving automated readers a cheap curated path instead of brute-forcing expensive ones. That issue is design-only and explicitly does not mitigate this incident.Worth restating so the two are not confused:
llms.txtis an unenforced convention and the current crawler already ignoresrobots.txt. deploy#785 improves how the estate presents itself. It provides no protection. This issue remains the only thing standing between the crawler and another OOM.Still unmitigated
Nothing has changed on the cause since the 3Gi bump. Forgejo has been stable, but the trigger is untouched and the rate limiter is still a single global bucket.
Recurrence 2026-08-26, and the guard is now authored in #939
Kai reported "getting some 503s from forgejo" at ~01:44 UTC. Same incident, higher intensity. The 3Gi ceiling did not hold.
Before-state
forgejo-6fb99c98fc-8hsfp, the pod created by the 3Gi change in this issue, is no longer at 0 restarts.That instance lived 21 minutes. Container memory sampled live: 892 MB at 01:44:21 UTC, 975 MB at ~01:47:30 UTC, roughly 26 MB/min of climb after startup. Node stayed healthy throughout (
MemoryPressure=False,DiskPressure=False), so this is a container-limit OOM again, not node exhaustion.Kubelet log volume for the dead instance was 9.68 MB across 21 minutes, about 460 KB/min.
Trigger, same shape, worse
347s, against 298s at the peak of the 2026-08-24 event.
repo.Issueshits per 10 minutes over 6h: baseline ~2,000-3,600, ramping from ~00:40 UTC to a peak of 28,883 (~48/s).New evidence: attribution by user-agent
Traefik access logs do carry
request_User-Agent, which the earlier comments did not use. Over a 2h window onforgejo.coilysiren.me:forgejo-runner/v12.13.2- 44,712meta-externalagent/1.1- 4,251 + 2,200git/2.52.0- 952Eight distinct "browser" agents landing within 3 percent of each other is not organic traffic. That is one fleet rotating UA strings, roughly 145k requests over 2h, plus ~6.4k that openly declare Meta's crawler.
This does not overcome the attribution limit recorded above, since
ClientHostis still10.42.0.93for every one of them. It does mean UA is a usable signal even though source IP is not, if a future guard wants one.Confirmations of prior findings
externalTrafficPolicy: Localis still live onsvc/traefikwith the #909 change-cause annotation,svclb-traefikDaemonSet is still 1/1, and Forgejo still logs every client as10.42.0.93. The correction in this comment holds.forgejo-ratelimitmiddleware is unchanged ataverage: 50, burst: 100, period: 1s, still a single global bucket, still above the ~20/s the crawler sustains.Also found: memory limit drift
Live was 3Gi, tracked
deploy/forgejo.ymlsaid 1500Mi. The 2026-08-24 bump was applied live but never landed in the manifest, so the next Ansible converge would have reset a still-OOMing service to the limit it was already dying at. Worth noting for item 5 in the description, which is the same root cause: the Deployment is hand-applied.What is authored
#939 implements the narrow guard decided in the posture comment, plus the 4Gi limit Kai approved, plus a
docs/forgejo-deploy-plan.mdsection recording whyexternalTrafficPolicy: Localandrobots.txtare both retracted, so they stop being re-proposed.kubectl diffpasses server-side dry-run andpre-commitis green.Nothing has been applied live.
kubectl applywas blocked by the operator permission classifier in my session. The rate numbers and thepriority: 100route are unverified at runtime, and Forgejo is still exposed to the crawler right now.Still unmitigated
Everything in the previous closing list, unchanged, plus the observation that the crawl intensity has grown between the two events. #938 was filed before I found this issue and has been closed as a duplicate.
Applied: #939 merged and live. Guard works mechanically, but is NOT yet proven against the crawler.
Kai merged #939 and applied it at ~02:26 UTC.
Gate check before apply
mainstill carried the stale1500Miuntil the merge landed, so the firstkubectl diffaftergit pullproposed 3Gi -> 1500Mi, a reduction on a service that was OOMing at 3Gi. Caught before apply. Post-merge the diff read correctly:Anyone repeating this should read the diff rather than trusting the command sequence. The check is now in
docs/forgejo-deploy-plan.md.Apply result
forgejo-6fb99c98fc-8hsfp(3 restarts) replaced byforgejo-86b7b6f66f-qk7th, 0 restarts, live limit 4Gi, container memory 171 MB.After-state, verified end-to-end
Guard behaviour:
The control path is unthrottled, so ordinary filtering and browsing are unaffected. The faceted path does 429 under repetition. The route priority is correct and the middleware is attached.
The honest caveat: this is not attributable to the guard
The crawl stopped on its own at 01:40 UTC, 46 minutes before the guard was applied.
repo.Issueshits per 5 minutes:Traefik has issued exactly 3
429s onforgejo.coilysiren.mein the last 30 minutes, and all 3 are mine. The crawler has not been subject to this guard even once.So Forgejo is stable right now because the crawler left, not because of anything applied here. The guard is verified as a mechanism and unverified as a mitigation. The real test is the next wave, and the numbers (
average: 2, burst: 10) may need revisiting once there is production evidence rather than a synthetic curl loop.Worth noting the crawl ended right after the 01:36:21 OOMKill, which is also roughly what happened on 2026-08-24. Whether the crawler backs off when the origin dies, or simply finished its pass, is not established.
Containment status
Closed by this change:
externalTrafficPolicy: Localandrobots.txtdead ends are recorded indocs/forgejo-deploy-plan.mdso they stop being re-proposed.Still open, unchanged:
loadBalancerIPlandmine on anyHelmChartConfig/traefikedit or k3s upgrade, is untouched.Leaving this issue open: the mitigation has not met an adversarial test yet.
Outage: the hostNetwork bypass failed. ~25 minutes of total public ingress loss, rolled back.
I caused this one. Recording it fully.
Timeline, 2026-08-26 UTC
Roughly 25 minutes with no external path at all to any public service.
Cause
The chart runs Traefik as UID 65532 with
allowPrivilegeEscalation: false. That setsNoNewPrivs, and NoNewPrivs makes the kernel ignore file capabilities, so theNET_BIND_SERVICEcapability I added never became effective. Kubernetes has no ambient-capability support, so there is no way to grant a workingCAP_NET_BIND_SERVICEto a non-root process through the pod spec alone.The svclb ordering trap fired as predicted in #941, and made it a full outage rather than a failed rollout:
FailedScheduling ... didn't have free portswhile svclb still held 80/443, then once the Service flipped to ClusterIP and svclb was torn down, external traffic had no path while the replacement pod was still crash-looping.Where my verification was insufficient
I verified by
helm templateplus a server-side dry run of the rendered Service, and stated in #941 that this was the verification. It was not. Both validate the manifest. Whether a non-root process can actually acquire a capability is kernel behaviour, observable only once the container starts. I treated manifest correctness as runtime correctness.The correction for next time is concrete: validate a privileged-port bind on a throwaway workload before touching a shared ingress. That check costs a minute and would have caught this before any outage.
Rollback
kubectl applyof the previous HelmChartConfig. Helm upgraded back to a LoadBalancer Service, svclb returned within seconds, and the old Traefik pod was still Running 1/1 the whole time, so traffic resumed as soon as the path existed. That the old pod survived is why recovery took under a minute once the right lever was pulled.After-state against the 03:53 baseline:
Forgejo was never affected:
forgejo-86b7b6f66f-qk7th, 0 restarts, 134 minutes at the time of check. The 4Gi limit and the faceted-query guard are untouched and still in place.Flux both GitRepositories went
Falsewithconnect: connection refusedduring the window, as in the original incident. Expected to self-recover as they did on 2026-08-24, but that is an assumption rather than something I have observed yet, so it wants a check.Still open
#947 reverts the template to the parity state. It is deliberately not
git revert e1db70e, because that commit squashed two changes and reverting it wholesale would re-arm the #910 landmine. Until #947 merges and converges,mainand the node's manifest file both still hold the hostNetwork version, and a k3s or node restart would repeat this outage unattended.Source IP is still masked by klipper-lb, so the original problem in this issue is exactly where it was. The retry shape recorded in
docs/k3s-deploy-notes-manifests.mdishostPort: 80/443onto unprivileged container ports 8000/8443, which keeps the container non-root and needs no capabilities, or MetalLB.