Plan: real high-availability path for Forgejo (replicas>1) #381

New issue

Open

opened 2026-06-19 07:58:56 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-06-19 07:58:56 +00:00

Member

Why this exists

A request to scale deploy/forgejo to 2 replicas surfaced that the current setup cannot safely run more than one pod. Naively scaling would land both pods on the single node sharing one ReadWriteOnce data dir and risk corrupting git repos / LFS / bleve indexers, with zero availability gain. This issue captures the proper HA path instead.

Upstream caveat to set expectations: Forgejo's own helm HA docs state Forgejo is not fully HA-ready yet due to race conditions in the queue, cron, and cache processes without leader-follower logic. So "HA" here means horizontally-scaled + zero-downtime deploys + node-failure tolerance, not a fully race-free guarantee.

Current single points of failure

Single node. kai-server is the only node; everything (Forgejo, postgres, runners) runs there. Replicas alone buy nothing against node failure.
RWO local-path PVC. forgejo-data (/var/lib/gitea: git repos, LFS, attachments, avatars, bleve indexers) is ReadWriteOnce node-local storage. Cannot be shared across nodes.
In-memory cache + queue + session. Cache and queue default to per-instance in-memory; session is db (the one piece already HA-friendly). Multiple instances would diverge / duplicate background work.
Bleve indexers. Issue + repo indexers are local-file bleve, single-connection only, not multi-instance safe.
Single postgres. forgejo-db StatefulSet is one replica — itself a SPOF.
Recreate strategy. Forces full downtime on every deploy; incompatible with zero-downtime rollout.

Required changes (per Forgejo helm HA docs)

Component	HA requirement	Current state
Repository storage	RWX filesystem owned by uid/gid 1000	RWO local-path
Object storage (LFS/attachments/avatars/packages)	S3-compatible (MinIO) or RWX	local disk
Cache	Valkey/Redis	in-memory
Queue	Valkey/Redis	in-memory (per-instance)
Session	Valkey/Redis provider	`db` (OK, can stay or move)
Issue indexer	Elasticsearch or Meilisearch	bleve
Repo/code indexer	must be disabled for HA (meili only does issue indexer)	bleve
Database	external HA postgres	single postgres pod
Cron/leader election	not implemented upstream	N/A

Proposed phased rollout

Phase 1 - zero-downtime deploys on the current single node (no node-failure HA yet).

Stand up Valkey (single pod ok for now) in the forgejo namespace.
Set [cache] ADAPTER=redis, [queue] TYPE=redis, optionally [session] PROVIDER=redis, pointing at Valkey.
Move issue indexer off bleve (ISSUE_INDEXER_TYPE=meilisearch + a Meilisearch pod, or db as a lighter interim) and disable the repo indexer (REPO_INDEXER_ENABLED=false).
Move LFS/attachments/avatars/packages to MinIO (S3) so only bare git repos still need the filesystem.
Provision RWX storage for /var/lib/gitea (Longhorn RWX or an NFS export), uid/gid 1000.
Switch deployment strategy Recreate -> RollingUpdate, set replicas: 2.
Result: rolling/zero-downtime deploys and load spreading, but both pods still on kai-server -> no protection from node loss.

Phase 2 - true node-failure HA.

Add a second cluster node.
podAntiAffinity to spread Forgejo replicas across nodes (requires the Phase 1 RWX storage to actually be cross-node, e.g. Longhorn/NFS, not local-path).
HA postgres (e.g. CloudNativePG operator) to remove the DB SPOF.
Scale Valkey/indexer for redundancy.

Open decisions for Kai

Is node-failure HA actually wanted, or just zero-downtime deploys? Phase 1 alone may be enough and is far cheaper.
RWX choice: Longhorn vs NFS on the homelab.
Whether the added operational surface (Valkey + Meilisearch + MinIO + HA postgres + 2nd node) is worth it for a personal Forgejo, vs accepting single-instance with good backups.

Sources

forgejo-helm HA setup doc: https://code.forgejo.org/forgejo-helm/forgejo-helm/src/branch/main/docs/ha-setup.md
Forgejo config cheat sheet: https://forgejo.org/docs/latest/admin/config-cheat-sheet/
Codeberg clustered-Forgejo journey: https://codeberg.org/forgejo/discussions/issues/259

Filed instead of scaling to 2 replicas, which would have risked data corruption with no HA benefit on the current single-node RWO setup.

## Why this exists A request to scale `deploy/forgejo` to 2 replicas surfaced that the current setup cannot safely run more than one pod. Naively scaling would land both pods on the single node sharing one `ReadWriteOnce` data dir and risk corrupting git repos / LFS / bleve indexers, with **zero** availability gain. This issue captures the proper HA path instead. Upstream caveat to set expectations: Forgejo's own helm HA docs state Forgejo is **not fully HA-ready yet** due to race conditions in the queue, cron, and cache processes without leader-follower logic. So "HA" here means horizontally-scaled + zero-downtime deploys + node-failure tolerance, not a fully race-free guarantee. ## Current single points of failure - **Single node.** `kai-server` is the only node; everything (Forgejo, postgres, runners) runs there. Replicas alone buy nothing against node failure. - **RWO `local-path` PVC.** `forgejo-data` (`/var/lib/gitea`: git repos, LFS, attachments, avatars, bleve indexers) is `ReadWriteOnce` node-local storage. Cannot be shared across nodes. - **In-memory cache + queue + session.** Cache and queue default to per-instance in-memory; session is `db` (the one piece already HA-friendly). Multiple instances would diverge / duplicate background work. - **Bleve indexers.** Issue + repo indexers are local-file bleve, single-connection only, not multi-instance safe. - **Single postgres.** `forgejo-db` StatefulSet is one replica — itself a SPOF. - **`Recreate` strategy.** Forces full downtime on every deploy; incompatible with zero-downtime rollout. ## Required changes (per Forgejo helm HA docs) | Component | HA requirement | Current state | |---|---|---| | Repository storage | RWX filesystem owned by uid/gid 1000 | RWO local-path | | Object storage (LFS/attachments/avatars/packages) | S3-compatible (MinIO) or RWX | local disk | | Cache | Valkey/Redis | in-memory | | Queue | Valkey/Redis | in-memory (per-instance) | | Session | Valkey/Redis provider | `db` (OK, can stay or move) | | Issue indexer | Elasticsearch or Meilisearch | bleve | | Repo/code indexer | must be disabled for HA (meili only does issue indexer) | bleve | | Database | external HA postgres | single postgres pod | | Cron/leader election | not implemented upstream | N/A | ## Proposed phased rollout **Phase 1 - zero-downtime deploys on the current single node (no node-failure HA yet).** - Stand up Valkey (single pod ok for now) in the `forgejo` namespace. - Set `[cache] ADAPTER=redis`, `[queue] TYPE=redis`, optionally `[session] PROVIDER=redis`, pointing at Valkey. - Move issue indexer off bleve (`ISSUE_INDEXER_TYPE=meilisearch` + a Meilisearch pod, or `db` as a lighter interim) and disable the repo indexer (`REPO_INDEXER_ENABLED=false`). - Move LFS/attachments/avatars/packages to MinIO (S3) so only bare git repos still need the filesystem. - Provision RWX storage for `/var/lib/gitea` (Longhorn RWX or an NFS export), uid/gid 1000. - Switch deployment strategy `Recreate` -> `RollingUpdate`, set `replicas: 2`. - Result: rolling/zero-downtime deploys and load spreading, but both pods still on `kai-server` -> no protection from node loss. **Phase 2 - true node-failure HA.** - Add a second cluster node. - `podAntiAffinity` to spread Forgejo replicas across nodes (requires the Phase 1 RWX storage to actually be cross-node, e.g. Longhorn/NFS, not local-path). - HA postgres (e.g. CloudNativePG operator) to remove the DB SPOF. - Scale Valkey/indexer for redundancy. ## Open decisions for Kai - Is node-failure HA actually wanted, or just zero-downtime deploys? Phase 1 alone may be enough and is far cheaper. - RWX choice: Longhorn vs NFS on the homelab. - Whether the added operational surface (Valkey + Meilisearch + MinIO + HA postgres + 2nd node) is worth it for a personal Forgejo, vs accepting single-instance with good backups. ## Sources - forgejo-helm HA setup doc: https://code.forgejo.org/forgejo-helm/forgejo-helm/src/branch/main/docs/ha-setup.md - Forgejo config cheat sheet: https://forgejo.org/docs/latest/admin/config-cheat-sheet/ - Codeberg clustered-Forgejo journey: https://codeberg.org/forgejo/discussions/issues/259 Filed instead of scaling to 2 replicas, which would have risked data corruption with no HA benefit on the current single-node RWO setup.

coilyco-ops added the

consult

labels

2026-06-24 05:38:27 +00:00

coilyco-ops commented

2026-06-24 05:38:27 +00:00

Author

Member

Triage: P3 / consult - Plan issue for Forgejo HA; design/planning, no code yet.

(goose-style pass; claude as the judgment engine, applied 2026-06-23)

**Triage:** `P3` / `consult` - Plan issue for Forgejo HA; design/planning, no code yet. (goose-style pass; claude as the judgment engine, applied 2026-06-23)

coilyco-ops added

and removed

labels

2026-07-10 09:00:06 +00:00