Forgejo takes crawler-shaped external traffic on expensive blame and commit endpoints, peaking at 3x baseline #842

New issue

Open

opened 2026-08-15 16:53:21 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-15 16:53:21 +00:00

Owner

Summary

Public Forgejo traffic is dominated by automated traversal of the most expensive read endpoints. It is absorbed fine today, but the pattern has no ceiling and there is nothing in front of it.

Evidence

All external traffic arrives through svclb-traefik at 10.42.0.167, which source-NATs it, so per-client attribution is not currently possible from the router log.

Request volume through that path, 15 minute buckets on 2026-08-15:

13:00Z   5361
13:45Z  10135
14:00Z  12692     <- peak, about 14 req/s
14:30Z   9127
15:15Z   6518
16:00Z   3993
16:15Z   3871     <- back to baseline

Baseline sits around 4000 per 15 minutes, roughly 4.4 req/s. The peak is about 3x that.

Endpoint mix over one hour, out of 17,430 completed requests:

15,738 from the external path
4,642 requests with commit/ in the path
1,079 requests to /blame/

The access pattern walks /blame/commit/<sha>/<file>, /src/commit/<sha>/<path>, and /commits/commit/<sha>/<path> across many repositories in rapid succession, including 404s and 301s on paths that do not exist. That is crawler behaviour, not human browsing.

Forgejo is currently coping

Worth stating plainly so this is not treated as an outage. Zero of the 17,430 requests took a full second. Sampled durations were 1.7ms, 2.6ms, 28.5ms, 33.0ms, 42.8ms, 53.8ms, 129.2ms, and 622.3ms for a git push. The gitea process was using 13.7% of one core.

Why file it anyway

/blame/ is one of the most expensive endpoints a git forge exposes, and it is being walked systematically
The 3x spike had no mitigation in place and nothing would have stopped a 30x one
Everything shares kai-server's single NVMe, so a sustained crawl competes with CI and the Eco server for disk

Options

robots.txt disallowing /blame/, /commits/, /src/commit/, and /archive/ on the public host
Traefik rate limiting on those path prefixes
Preserve the real client IP through svclb so crawlers can be attributed and blocked individually, which is a prerequisite for anything targeted

Acceptance

Expensive traversal endpoints are either rate limited or excluded from crawling, and the real client IP is visible in Forgejo's router log.

## Summary Public Forgejo traffic is dominated by automated traversal of the most expensive read endpoints. It is absorbed fine today, but the pattern has no ceiling and there is nothing in front of it. ## Evidence All external traffic arrives through `svclb-traefik` at `10.42.0.167`, which source-NATs it, so per-client attribution is not currently possible from the router log. Request volume through that path, 15 minute buckets on 2026-08-15: ``` 13:00Z 5361 13:45Z 10135 14:00Z 12692 <- peak, about 14 req/s 14:30Z 9127 15:15Z 6518 16:00Z 3993 16:15Z 3871 <- back to baseline ``` Baseline sits around 4000 per 15 minutes, roughly 4.4 req/s. The peak is about 3x that. Endpoint mix over one hour, out of 17,430 completed requests: - 15,738 from the external path - 4,642 requests with `commit/` in the path - 1,079 requests to `/blame/` The access pattern walks `/blame/commit/<sha>/<file>`, `/src/commit/<sha>/<path>`, and `/commits/commit/<sha>/<path>` across many repositories in rapid succession, including 404s and 301s on paths that do not exist. That is crawler behaviour, not human browsing. ## Forgejo is currently coping Worth stating plainly so this is not treated as an outage. Zero of the 17,430 requests took a full second. Sampled durations were 1.7ms, 2.6ms, 28.5ms, 33.0ms, 42.8ms, 53.8ms, 129.2ms, and 622.3ms for a git push. The `gitea` process was using 13.7% of one core. ## Why file it anyway - `/blame/` is one of the most expensive endpoints a git forge exposes, and it is being walked systematically - The 3x spike had no mitigation in place and nothing would have stopped a 30x one - Everything shares kai-server's single NVMe, so a sustained crawl competes with CI and the Eco server for disk ## Options - `robots.txt` disallowing `/blame/`, `/commits/`, `/src/commit/`, and `/archive/` on the public host - Traefik rate limiting on those path prefixes - Preserve the real client IP through svclb so crawlers can be attributed and blocked individually, which is a prerequisite for anything targeted ## Acceptance Expensive traversal endpoints are either rate limited or excluded from crawling, and the real client IP is visible in Forgejo's router log.

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:09 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:31:00 +00:00