fix(forgejo): guard faceted issue/pull queries, raise limit to 4Gi #939

Merged

coilysiren merged 1 commit from fix/forgejo-crawler-oom into main

2026-08-26 02:23:28 +00:00

coilyco-ops commented

2026-08-26 02:16:22 +00:00

Owner

Refs #909. Forgejo has OOMKilled 3 times in the last 30h and is doing it again right now, at a higher crawl intensity than the 2026-08-24 incident.

What is happening

A crawler is walking the permuted issue/pull filter space. repo.Issues responses reached 347548ms on the container that died at 01:36:21 UTC. repo.Issues hits peaked at 28,883 per 10 minutes (~48/s), up from a ~2,000-3,600 baseline, ramping from ~00:40 UTC.

Traefik attribution over 2h on forgejo.coilysiren.me:

forgejo-runner/v12.13.2 - 44,712 (legitimate)
eight rotating desktop browser UA strings - 18,400 / 18,390 / 18,205 / 18,204 / 18,073 / 18,053 / 18,004 / 17,819
meta-externalagent/1.1 - 4,251 + 2,200
git/2.52.0 - 952

Eight "browsers" within 3 percent of each other is one fleet rotating UA strings, not organic traffic.

What this changes

1. Enforced guard on the faceted query space only. New Middleware/forgejo-faceted-query-limit (average: 2, burst: 10, period: 1s) and IngressRoute/forgejo-faceted-query at priority: 100, matching /<org>/<repo>/(issues|pulls) only when one of labels, milestone, assignee, poster, project is present.

This implements the "steer, do not block" posture from #909: plain ?state= and ?q= pages stay on the fast path, and code browsing, raw, /src/, and /api/ stay open.

2. Memory limit 1500Mi -> 4Gi. The tracked value had drifted below the live 3Gi, so an Ansible converge would have cut headroom on a service that was already OOMing. Kai approved 4Gi on both.

What this deliberately does not do

Not externalTrafficPolicy: Local. Already applied 2026-08-24 and proven ineffective: klipper-lb forwards through a pod netns, so the SNAT happens before Traefik. Documented in docs/forgejo-deploy-plan.md so it stops being re-proposed.
Not robots.txt. Already shipped, already ignored (27.7 percent of crawler requests hit disallowed paths after it fetched the file).

The rate bucket is global because source IP is still masked. Per-source needs ServiceLB bypassed (Traefik on hostPort/hostNetwork) or MetalLB, which is separate work.

Verification status

kubectl diff -f deploy/forgejo.yml passes server-side dry-run: both new CRs are schema-valid, and the Deployment diff is scoped to the memory line plus dropping a stale restartedAt annotation. pre-commit run --all-files is green.

The guard is not runtime-verified. kubectl apply was blocked by the operator permission classifier in my session, so nothing has been applied live. The rate numbers and the priority: 100 route need a real apply plus the check in docs/forgejo-deploy-plan.md before anyone should trust them.

Refs #909. Forgejo has OOMKilled 3 times in the last 30h and is doing it again right now, at a higher crawl intensity than the 2026-08-24 incident. ## What is happening A crawler is walking the permuted issue/pull filter space. `repo.Issues` responses reached **347548ms** on the container that died at 01:36:21 UTC. `repo.Issues` hits peaked at **28,883 per 10 minutes (~48/s)**, up from a ~2,000-3,600 baseline, ramping from ~00:40 UTC. Traefik attribution over 2h on `forgejo.coilysiren.me`: * `forgejo-runner/v12.13.2` - 44,712 (legitimate) * eight rotating desktop browser UA strings - 18,400 / 18,390 / 18,205 / 18,204 / 18,073 / 18,053 / 18,004 / 17,819 * `meta-externalagent/1.1` - 4,251 + 2,200 * `git/2.52.0` - 952 Eight "browsers" within 3 percent of each other is one fleet rotating UA strings, not organic traffic. ## What this changes **1. Enforced guard on the faceted query space only.** New `Middleware/forgejo-faceted-query-limit` (`average: 2, burst: 10, period: 1s`) and `IngressRoute/forgejo-faceted-query` at `priority: 100`, matching `/<org>/<repo>/(issues|pulls)` only when one of `labels`, `milestone`, `assignee`, `poster`, `project` is present. This implements the "steer, do not block" posture from #909: plain `?state=` and `?q=` pages stay on the fast path, and code browsing, raw, `/src/`, and `/api/` stay open. **2. Memory limit 1500Mi -> 4Gi.** The tracked value had drifted below the live 3Gi, so an Ansible converge would have cut headroom on a service that was already OOMing. Kai approved 4Gi on both. ## What this deliberately does not do * **Not `externalTrafficPolicy: Local`.** Already applied 2026-08-24 and proven ineffective: klipper-lb forwards through a pod netns, so the SNAT happens before Traefik. Documented in `docs/forgejo-deploy-plan.md` so it stops being re-proposed. * **Not robots.txt.** Already shipped, already ignored (27.7 percent of crawler requests hit disallowed paths after it fetched the file). The rate bucket is global because source IP is still masked. Per-source needs ServiceLB bypassed (Traefik on hostPort/hostNetwork) or MetalLB, which is separate work. ## Verification status `kubectl diff -f deploy/forgejo.yml` passes server-side dry-run: both new CRs are schema-valid, and the Deployment diff is scoped to the memory line plus dropping a stale `restartedAt` annotation. `pre-commit run --all-files` is green. **The guard is not runtime-verified.** `kubectl apply` was blocked by the operator permission classifier in my session, so nothing has been applied live. The rate numbers and the `priority: 100` route need a real apply plus the check in `docs/forgejo-deploy-plan.md` before anyone should trust them.

coilyco-ops added 1 commit

2026-08-26 02:16:22 +00:00

fix(forgejo): guard faceted issue/pull queries, raise limit to 4Gi

TruffleHog / Scan for secrets (pull_request) Successful in 4s

Details

CI / lint (pull_request) Successful in 41s

Details

4d5890925f

Forgejo OOMKilled 3 times in 30h, each kill a user-visible 503 window on a
single-replica Recreate deployment. Cause is a crawler walking the permuted
issue/pull filter space, driving repo.Issues to 347s per request and peaking
at 28,883 hits per 10 minutes.

Adds an enforced rate limit on the faceted query space only, matching the
"steer, do not block" posture: plain ?state= and ?q= pages stay on the fast
path, code browsing and /src/ stay open. The bucket is global because
klipper-lb masks source IPs, which externalTrafficPolicy=Local was already
proven not to fix (infrastructure#909).

Also raises the memory limit 1500Mi -> 4Gi. The tracked value had drifted
below the live 3Gi, so an Ansible converge would have cut headroom on a
service that was already OOMing.

Refs #909

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: sysadmin

coilyco-ops referenced this pull request

2026-08-26 02:17:01 +00:00

Forgejo OOM crash loop under faceted issue-query flood, 2026-08-24 #909

coilyco-ops referenced this pull request

2026-08-26 02:17:08 +00:00

Forgejo OOMKilled repeatedly: AI crawler enumerating issue-filter URL space, ingress rate limit structurally blind #938