coilyco-flight-deck/infrastructure

Watch

Forgejo OOMKilled repeatedly: AI crawler enumerating issue-filter URL space, ingress rate limit structurally blind #938

New issue

Closed

opened 2026-08-26 01:51:12 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-26 01:51:12 +00:00

Owner

Symptom

Intermittent 503s from forgejo.coilysiren.me, reported 2026-08-26 ~01:44 UTC.

Root cause

forgejo-6fb99c98fc-8hsfp (single replica, strategy: Recreate) is being OOMKilled. Every kill produces a 503 window until the readiness probe passes again.

Evidence from kubectl describe pod:

Last State: Terminated - Reason: OOMKilled - Exit Code: 137
Previous container lifetime 18:15:38 -> 18:36:21 -0700, so 21 minutes
Restart Count: 3 over the pod's 30h life, and accelerating

Container memory sampled live while writing this: 892 MB at 01:44:21 UTC, 975 MB at ~01:47:30 UTC, so roughly 26 MB/min of sustained climb after startup against a 3Gi limit.

What is driving it

An AI/scraper fleet is enumerating the combinatorial issue-filter URL space. Sample from a single 0.3 second window in the container that died:

GET /coilyco-flight-deck/ward/pulls?labels=201%2C211%2C325%2C345%2C347&milestone=6&poster=0&project=-1&state=open&type=all ... 200 OK in 347548.2ms @ repo/issue.go:515(repo.Issues)
GET /coilyco-gaming/eco-app/issues?assignee=7&labels=216%2C361%2C364&milestone=0&poster=0&project=0&state=closed&type=all ... 200 OK in 345539.1ms @ repo/issue.go:515(repo.Issues)

Note the 347 second response times. Requests pile up, each holding memory, until the cgroup limit is hit.

Rate of repo.Issues hits per 10 minutes over the last 6h: baseline ~2,000-3,600, ramping from ~00:40 UTC to a peak of 28,883 per 10 minutes (~48/s).

Traefik access logs give attribution over a 2h window on forgejo.coilysiren.me:

forgejo-runner/v12.13.2 - 44,712 (legitimate)
eight rotating desktop browser UA strings - 18,400 / 18,390 / 18,205 / 18,204 / 18,073 / 18,053 / 18,004 / 17,819
meta-externalagent/1.1 - 4,251 + 2,200
git/2.52.0 - 952

Eight distinct "browser" agents landing within 3 percent of each other is not organic traffic. That is one scraper fleet rotating UA strings, roughly 145k requests / 2h (~20/s), plus ~6.4k that openly declare Meta's crawler.

Why the existing guard did not fire

Middleware/forgejo-ratelimit is attached to the router and reads average: 50, burst: 100, period: 1s.

Threshold is above the attack. 50 req/s allowed, crawler sustains ~20 req/s. It never trips.
The limiter is structurally blind. k3s ServiceLB SNATs every external connection to the svclb-traefik pod IP 10.42.0.93. Forgejo logs every single client as 10.42.0.93:0, and Traefik's ClientHost is 10.42.0.93 for all of them. So the per-source rate limiter buckets the crawler, the runners, and real users into one shared bucket. It cannot distinguish clients at all, which also means tightening the number alone would throttle the runners first.
robots.txt does not cover the crawled space. The stock Forgejo robots.txt disallows /*/*/src/, /*q=*, /*sort=*, /*tab=*, but not /*/*/issues?... or /*/*/pulls?... filtered by labels=, assignee=, milestone=, state=, poster=, project=, type=. Those are exactly the URLs being walked. A UA-rotating scraper likely ignores robots.txt regardless.

Separate landmine: memory limit drift

The live deployment and the tracked manifest disagree.

live Limits.memory - 3Gi
deploy/forgejo.yml - 1500Mi

CPU and ephemeral-storage match, memory does not. Forgejo is being OOMKilled at 3Gi today. The next Ansible converge would reset it to 1500Mi and cut time-to-OOM by more than half. This should be reconciled deliberately, in whichever direction, rather than discovered during the next incident.

Not yet done

No live change has been made. Everything above is read-only observation. The ingress-filtering decision (block by UA, fix source-IP visibility then tighten the limit, extend robots.txt, or some combination) is a public-surface change and is with Kai.

Candidate remediations, roughly in order of leverage:

Set externalTrafficPolicy: Local on the Traefik service (or otherwise preserve client IP) so the rate limiter and any block rule can actually see distinct sources. This is the prerequisite for 2 being meaningful.
Re-scope forgejo-ratelimit once source IPs are real, and consider a separate tighter limit on the issues/pulls filter paths.
Extend robots.txt to disallow the filter query space.
Reconcile the 3Gi vs 1500Mi drift.

Filed by the sysadmin seat during live diagnosis.

## Symptom Intermittent 503s from forgejo.coilysiren.me, reported 2026-08-26 ~01:44 UTC. ## Root cause `forgejo-6fb99c98fc-8hsfp` (single replica, `strategy: Recreate`) is being **OOMKilled**. Every kill produces a 503 window until the readiness probe passes again. Evidence from `kubectl describe pod`: * `Last State: Terminated` - `Reason: OOMKilled` - `Exit Code: 137` * Previous container lifetime `18:15:38 -> 18:36:21 -0700`, so 21 minutes * `Restart Count: 3` over the pod's 30h life, and accelerating Container memory sampled live while writing this: 892 MB at 01:44:21 UTC, 975 MB at ~01:47:30 UTC, so roughly 26 MB/min of sustained climb after startup against a 3Gi limit. ## What is driving it An AI/scraper fleet is enumerating the combinatorial issue-filter URL space. Sample from a single 0.3 second window in the container that died: ``` GET /coilyco-flight-deck/ward/pulls?labels=201%2C211%2C325%2C345%2C347&milestone=6&poster=0&project=-1&state=open&type=all ... 200 OK in 347548.2ms @ repo/issue.go:515(repo.Issues) GET /coilyco-gaming/eco-app/issues?assignee=7&labels=216%2C361%2C364&milestone=0&poster=0&project=0&state=closed&type=all ... 200 OK in 345539.1ms @ repo/issue.go:515(repo.Issues) ``` Note the 347 second response times. Requests pile up, each holding memory, until the cgroup limit is hit. Rate of `repo.Issues` hits per 10 minutes over the last 6h: baseline ~2,000-3,600, ramping from ~00:40 UTC to a peak of **28,883 per 10 minutes (~48/s)**. Traefik access logs give attribution over a 2h window on `forgejo.coilysiren.me`: * `forgejo-runner/v12.13.2` - 44,712 (legitimate) * eight rotating desktop browser UA strings - 18,400 / 18,390 / 18,205 / 18,204 / 18,073 / 18,053 / 18,004 / 17,819 * `meta-externalagent/1.1` - 4,251 + 2,200 * `git/2.52.0` - 952 Eight distinct "browser" agents landing within 3 percent of each other is not organic traffic. That is one scraper fleet rotating UA strings, roughly 145k requests / 2h (~20/s), plus ~6.4k that openly declare Meta's crawler. ## Why the existing guard did not fire `Middleware/forgejo-ratelimit` is attached to the router and reads `average: 50, burst: 100, period: 1s`. 1. **Threshold is above the attack.** 50 req/s allowed, crawler sustains ~20 req/s. It never trips. 2. **The limiter is structurally blind.** k3s ServiceLB SNATs every external connection to the `svclb-traefik` pod IP `10.42.0.93`. Forgejo logs every single client as `10.42.0.93:0`, and Traefik's `ClientHost` is `10.42.0.93` for all of them. So the per-source rate limiter buckets the crawler, the runners, and real users into **one** shared bucket. It cannot distinguish clients at all, which also means tightening the number alone would throttle the runners first. 3. **robots.txt does not cover the crawled space.** The stock Forgejo robots.txt disallows `/*/*/src/`, `/*q=*`, `/*sort=*`, `/*tab=*`, but not `/*/*/issues?...` or `/*/*/pulls?...` filtered by `labels=`, `assignee=`, `milestone=`, `state=`, `poster=`, `project=`, `type=`. Those are exactly the URLs being walked. A UA-rotating scraper likely ignores robots.txt regardless. ## Separate landmine: memory limit drift The live deployment and the tracked manifest disagree. * live `Limits.memory` - **3Gi** * [`deploy/forgejo.yml`](deploy/forgejo.yml) - **1500Mi** CPU and ephemeral-storage match, memory does not. Forgejo is being OOMKilled at 3Gi today. The next Ansible converge would reset it to 1500Mi and cut time-to-OOM by more than half. This should be reconciled deliberately, in whichever direction, rather than discovered during the next incident. ## Not yet done No live change has been made. Everything above is read-only observation. The ingress-filtering decision (block by UA, fix source-IP visibility then tighten the limit, extend robots.txt, or some combination) is a public-surface change and is with Kai. Candidate remediations, roughly in order of leverage: 1. Set `externalTrafficPolicy: Local` on the Traefik service (or otherwise preserve client IP) so the rate limiter and any block rule can actually see distinct sources. This is the prerequisite for 2 being meaningful. 2. Re-scope `forgejo-ratelimit` once source IPs are real, and consider a separate tighter limit on the `issues`/`pulls` filter paths. 3. Extend robots.txt to disallow the filter query space. 4. Reconcile the 3Gi vs 1500Mi drift. Filed by the sysadmin seat during live diagnosis.

coilyco-ops referenced this issue

2026-08-26 02:17:01 +00:00

Forgejo OOM crash loop under faceted issue-query flood, 2026-08-24 #909

coilyco-ops commented

2026-08-26 02:17:08 +00:00

Author

Owner

Duplicate of #909, which was filed 2026-08-24 for the same crawler and the same OOM and is still open at P0. I filed this before finding it.

Fresh evidence from tonight's recurrence has been added to #909 as a comment, and the guard is authored in #939. Closing here so the record stays in one place.

Two things in this issue's description were wrong and are corrected in #909, noting them so nobody acts on this text:

It proposed externalTrafficPolicy: Local to restore client attribution. That was already applied on 2026-08-24 and does not work, because klipper-lb SNATs through a pod network namespace before Traefik sees the packet.
It proposed extending robots.txt. The crawler fetches robots.txt and ignores it, measured at 27.7 percent disallowed-path traffic, so that route was already retracted in #909.

Duplicate of #909, which was filed 2026-08-24 for the same crawler and the same OOM and is still open at P0. I filed this before finding it. Fresh evidence from tonight's recurrence has been added to #909 as a comment, and the guard is authored in #939. Closing here so the record stays in one place. Two things in this issue's description were wrong and are corrected in #909, noting them so nobody acts on this text: * It proposed `externalTrafficPolicy: Local` to restore client attribution. That was already applied on 2026-08-24 and does not work, because klipper-lb SNATs through a pod network namespace before Traefik sees the packet. * It proposed extending robots.txt. The crawler fetches robots.txt and ignores it, measured at 27.7 percent disallowed-path traffic, so that route was already retracted in #909.

coilyco-ops closed this issue

2026-08-26 02:17:12 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#938

No description provided.

Rows
Columns