Audit Forgejo branch-heavy storage growth and define safe retention cleanup #490

New issue

Closed

opened 2026-07-09 18:01:03 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented 2026-07-09 18:01:03 +00:00

Member

Copy link

Context: follow-up from infrastructure#488. The disk guardrail sweep landed xrdp/journald/rootfs telemetry and documented a read-only Forgejo PVC hotspot drill, but Kai called out that Forgejo is the critical service and that branches are the biggest durable-content offender. This needs its own focused issue because #488 is closed and destructive cleanup was explicitly out of scope.

Goal:

• identify which Forgejo storage paths and database concepts are dominated by branch-heavy growth
• find Forgejo-managed mechanisms for pruning stale branches, branch artifacts, packages, actions artifacts/logs, repo archives, or related generated content
• distinguish safe managed cleanup from unsafe filesystem deletion
• propose retention defaults that protect Forgejo first while reducing avoidable local-path PVC growth

Constraints:

• do not delete live Forgejo data autonomously
• do not rm inside the PVC
• prefer Forgejo UI/API/CLI managed cleanup paths with rollback/risk notes
• include a read-only measurement checklist Kai can run or an ops role can later run
• if repo changes are safe, land docs/config only; live cleanup remains an explicit operator action

References:

• infrastructure#488
• deploy/forgejo.yml
• docs/forgejo-deploy-plan-followups.md
• docs/ansible-kai-disk-guardrails-role.md
• Forgejo storage/admin docs for the currently deployed version.

Context: follow-up from infrastructure#488. The disk guardrail sweep landed xrdp/journald/rootfs telemetry and documented a read-only Forgejo PVC hotspot drill, but Kai called out that Forgejo is the critical service and that branches are the biggest durable-content offender. This needs its own focused issue because #488 is closed and destructive cleanup was explicitly out of scope. Goal: * identify which Forgejo storage paths and database concepts are dominated by branch-heavy growth * find Forgejo-managed mechanisms for pruning stale branches, branch artifacts, packages, actions artifacts/logs, repo archives, or related generated content * distinguish safe managed cleanup from unsafe filesystem deletion * propose retention defaults that protect Forgejo first while reducing avoidable local-path PVC growth Constraints: * do not delete live Forgejo data autonomously * do not `rm` inside the PVC * prefer Forgejo UI/API/CLI managed cleanup paths with rollback/risk notes * include a read-only measurement checklist Kai can run or an ops role can later run * if repo changes are safe, land docs/config only; live cleanup remains an explicit operator action References: * infrastructure#488 * `deploy/forgejo.yml` * `docs/forgejo-deploy-plan-followups.md` * `docs/ansible-kai-disk-guardrails-role.md` * Forgejo storage/admin docs for the currently deployed version.

coilyco-ops commented 2026-07-09 18:06:45 +00:00

Author

Member

Copy link

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

---

Recommendation

Treat this as a single-repo docs/config follow-up, not a multi-repo fan-out. The safest advisory is:

1. Measure first on the live pod and DB, read-only.
2. Clean up only through Forgejo-managed paths: branch delete in UI/API, package delete in UI/API, Actions cleanup cron, archive cleanup cron, and optional Git GC.
3. Do not rm inside the PVC. On this deployment, raw filesystem deletion is the high-risk path because the whole Forgejo work tree is mounted on one local-path PVC.
4. Harden defaults after measurement: shorten Actions retention, keep archive/deleted-branch cleanup enabled, and consider quotas only after the first size breakdown.

What is likely growing

Local repo context:

• deploy/forgejo.yml mounts a single local-path PVC named forgejo-data at /var/lib/gitea, requested at 20Gi, while Postgres lives on a separate 5Gi PVC. Actions and LFS are enabled, and there are no explicit storage, retention, package-limit, or quota overrides in the manifest today. See deploy/forgejo.yml lines 39-49 and 174-280.
• The existing repo docs already point Kai at a read-only subtree drill on /var/lib/gitea/data and explicitly say to prefer Forgejo UI/API cleanup over truncating the PVC. See docs/forgejo-deploy-plan-followups.md lines 7-13 and docs/ansible-kai-disk-guardrails-role.md lines 50-62.

The main storage nuance is that branches are not large by themselves. A branch ref is tiny. Branch-heavy growth becomes durable when many long-lived branches keep unique Git objects reachable, which prevents packfiles from shrinking.

For this deployment, the likely hot paths are:

• Primary branch-heavy path: Git repositories under /var/lib/gitea/git/repositories.
  • This is where unique commits, trees, and blobs held alive by stale branches accumulate.
  • If many feature branches are merged but not deleted, their unique objects stay reachable and Git GC cannot reclaim them.
• Other Forgejo-managed asset paths under /var/lib/gitea/data according to Forgejo v15 storage docs:
  • lfs/
  • packages/
  • actions_artifacts/
  • actions_log/
  • repo-archive/
  • attachments/
  • Source: Forgejo storage settings and config cheat sheet.
• Not the main branch-heavy risk: Postgres metadata on the separate DB PVC.
  • Branch rows, PR metadata, package metadata, and Actions metadata do grow, but they usually do not dominate disk the way Git packfiles, LFS, packages, or Actions artifacts do.
  • The DB still deserves a read-only size check, but the issue’s “branch-heavy local-path PVC growth” is much more likely to be the Forgejo data PVC than the DB PVC.

One more important separation: Forgejo’s own docs say Actions cache is stored by the runner itself and never sent to Forgejo. In this repo, the runner also has its own PVCs and an emptyDir for Docker state, so runner cache pressure is not the same thing as Forgejo server PVC pressure. See deploy/forgejo-runner.yml and Forgejo Actions admin docs.

Safe managed cleanup vs unsafe cleanup

Safe managed cleanup

• Delete stale merged branches through the Forgejo UI or API.
  • This is the only cleanup that actually attacks branch-heavy Git growth at the source.
  • Risk: deleting branches can break open PR expectations, branch-based automation, or human workflows if done too aggressively.
• Let Forgejo’s deleted-branch cleanup cron remove deleted-branch remnants.
  • Forgejo v15 default: cron.deleted_branches_cleanup enabled, OLDER_THAN = 24h.
  • Important limitation: this only cleans up already deleted branches. It is not an instance-wide stale-branch retention policy.
  • Source: config cheat sheet.
• Enable cron.git_gc_repos after branch cleanup.
  • Forgejo marks repository GC as an extended cron task and it is disabled by default.
  • This is the safe managed mechanism for reclaiming unreachable Git objects after refs are deleted.
  • Risk: CPU and I/O spikes. Run off-hours and only with backup confidence.
  • Source: config cheat sheet.
• Keep cron.archive_cleanup enabled.
  • Generated repo archives are already safe to prune through Forgejo. Default is enabled with OLDER_THAN = 24h.
  • This cleans snapshot archives, not the underlying Git repository.
  • Source: config cheat sheet.
• Use Actions retention plus Forgejo’s cleanup cron.
  • Forgejo v15 defaults: LOG_RETENTION_DAYS = 365, ARTIFACT_RETENTION_DAYS = 90.
  • The admin cron UI can manually trigger cleanup of expired logs and artifacts.
  • This is safe and managed. It directly targets actions_log/ and actions_artifacts/.
  • Source: Forgejo Actions admin docs and config cheat sheet.
• Delete package versions through Forgejo UI/API when they are truly obsolete.
  • Forgejo’s cron.cleanup_packages only removes unreferenced package data older than 24h by default. It is not a policy for deleting package versions that are still intentionally present.
  • Source: config cheat sheet.
• Consider quotas as a guardrail, not a cleanup tool.
  • Forgejo v15 quota is disabled by default and explicitly documented as soft quota.
  • Useful subjects include size:repos:all, size:git:lfs, size:assets:artifacts, and size:assets:packages:all.
  • Risk: soft quotas do not perfectly stop already-started growth, and the feature is still described as early support.
  • Source: Soft-Quota docs and config cheat sheet.

Unsafe cleanup

• rm -rf inside /var/lib/gitea/git/repositories, /var/lib/gitea/data/lfs, /var/lib/gitea/data/packages, /var/lib/gitea/data/actions_*, or any other Forgejo-managed subtree.
• Deleting files directly from the PVC to “match” what the UI should show.
• Deleting Git refs or packfiles by hand inside repos instead of via Forgejo/Git-managed commands.

Those paths can desynchronize Forgejo’s database, break repository integrity, or remove data that Forgejo still believes exists.

What Forgejo does not give you out of the box

The docs point to a real gap that matters for this issue:

• Forgejo has built-in cleanup for deleted branch remnants, generated archives, expired Actions assets, and unreferenced package blobs.
• Forgejo does not appear to provide an instance-level policy that says “delete merged branches older than N days” across repos.

So the branch-retention decision is operational, not automatic. If Kai wants branch-heavy growth controlled, the actual levers are:

• repo-owner discipline on deleting merged branches,
• optional tooling/reporting to identify stale branches,
• and scheduled Git GC after those deletions.

Recommended retention defaults

These are the defaults I would recommend documenting first, because they protect Forgejo without surprising normal Git use:

• Keep cron.archive_cleanup at 24h.
  • Low risk. Generated archives are disposable.
• Keep cron.deleted_branches_cleanup at 24h.
  • Low risk. It only affects branches already deleted.
• Enable cron.git_gc_repos, scheduled off-hours.
  • Start with Forgejo’s documented @every 72h cadence or a similar off-hours interval.
  • This is the most important missing managed cleanup for branch-heavy repo growth.
• Reduce Actions artifact retention from 90 days to 14-30 days.
  • Recommendation: 14 days unless a repo has a known need for longer retention.
• Reduce Actions log retention from 365 days to 30-60 days.
  • Recommendation: 30 days for a small self-hosted instance unless longer audit history is clearly needed.
• Do not set package or quota limits blindly before measurement.
  • If packages are a real hotspot, then add owner/package size limits or quota groups in a second pass.
• Do not change repo-archive retention below 24h unless measurement proves it matters.
  • It is already short, and it is probably not the main branch-heavy offender.

If Kai wants a single “protect Forgejo first” bias, the best first config changes are:

• enable repo Git GC,
• shorten Actions log retention,
• shorten Actions artifact retention.

Those three reduce durable local-path growth without touching live Git history.

Read-only measurement checklist

The existing checklist should be widened beyond /var/lib/gitea/data, because the most branch-heavy path is probably in /var/lib/gitea/git.

1. Top-level Forgejo PVC split

ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "du -xhd1 /var/lib/gitea | sort -h"'

Goal: separate git/ from data/.

2. Existing documented data-subtree drill

ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "du -xhd1 /var/lib/gitea/data | sort -h"'

Goal: identify lfs, packages, actions_artifacts, actions_log, repo-archive, attachments hotspots.

3. Largest repositories on disk

ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "du -xhd2 /var/lib/gitea/git/repositories | sort -h | tail -n 50"'

Goal: find which repos actually account for Git growth.

4. Largest packfiles

ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "find /var/lib/gitea/git/repositories -name \*.pack -printf \"%s %p\\n\" | sort -n | tail -n 50"'

Goal: confirm packfile-heavy repos rather than loose-file noise.

5. Branch counts on the largest repos

For a suspect repo path:

ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "git -C /var/lib/gitea/git/repositories/<owner>/<repo>.git for-each-ref refs/heads --format=%(refname) | wc -l"'

Goal: distinguish “many refs” from “few refs but very large history”.

6. Postgres size sanity check

ssh kai-server 'k3s kubectl -n forgejo exec statefulset/forgejo-db -- sh -lc "psql -U forgejo -d forgejo -c \"select pg_size_pretty(pg_database_size('"'"'forgejo'"'"'));\""'

Goal: confirm whether DB growth is material or incidental.

7. Largest DB relations

ssh kai-server 'k3s kubectl -n forgejo exec statefulset/forgejo-db -- sh -lc "psql -U forgejo -d forgejo -c \"select relname, pg_size_pretty(pg_total_relation_size(oid)) as size from pg_class where relkind = 'r' order by pg_total_relation_size(oid) desc limit 20;\""'

Goal: identify whether Actions/package metadata tables are becoming abnormal.

8. Admin cron status review

In Forgejo admin UI, review admin/monitor/cron:

• confirm archive cleanup is running,
• confirm Actions cleanup is running,
• note whether a manual run is needed after any retention change,
• confirm whether Git GC is still disabled.

Open questions to resolve before any live cleanup

• Is the real hotspot git/repositories, lfs, packages, actions_artifacts, or something else?
• Are there repos that intentionally keep long-lived release/support branches?
• Do any workflows genuinely need artifacts older than 14-30 days?
• Are package registries actively used enough that owner/package limits would be disruptive?
• Does Kai want only docs/config landed now, with all live cleanup left as an operator runbook step? That is the safest shape.

Advisory conclusion

The best answer to post back on the issue is:

• Yes, branch-heavy growth deserves its own audit, but the risky assumption is thinking “branches” means a Forgejo data/ subtree. The dominant branch-heavy growth is more likely the Git repo store under /var/lib/gitea/git/repositories, while /var/lib/gitea/data holds other Forgejo-managed assets.
• Safe cleanup is available, but mostly for generated/expired content: repo archives, Actions logs/artifacts, deleted-branch remnants, and unreferenced package blobs.
• Forgejo does not appear to offer an instance-wide stale-live-branch retention policy. Live stale branches still need deliberate UI/API deletion, followed by managed Git GC if the goal is real space recovery.
• The first safe config recommendations are: enable repo Git GC, shorten Actions log retention, shorten Actions artifact retention, and keep existing archive/deleted-branch cleanup in place.
• Do not delete files directly from the PVC. Measure first, then use Forgejo-managed cleanup paths, and leave any actual live cleanup as an explicit operator action.

Sources

• Local repo: deploy/forgejo.yml, deploy/forgejo-runner.yml, docs/forgejo-deploy-plan-followups.md, docs/ansible-kai-disk-guardrails-role.md
• Forgejo v15 docs: storage settings, config cheat sheet, Actions admin guide, soft quota

---

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- ## Recommendation Treat this as a **single-repo docs/config follow-up**, not a multi-repo fan-out. The safest advisory is: 1. **Measure first** on the live pod and DB, read-only. 2. **Clean up only through Forgejo-managed paths**: branch delete in UI/API, package delete in UI/API, Actions cleanup cron, archive cleanup cron, and optional Git GC. 3. **Do not `rm` inside the PVC**. On this deployment, raw filesystem deletion is the high-risk path because the whole Forgejo work tree is mounted on one `local-path` PVC. 4. **Harden defaults after measurement**: shorten Actions retention, keep archive/deleted-branch cleanup enabled, and consider quotas only after the first size breakdown. ## What is likely growing Local repo context: - `deploy/forgejo.yml` mounts a single `local-path` PVC named `forgejo-data` at `/var/lib/gitea`, requested at **20Gi**, while Postgres lives on a separate **5Gi** PVC. Actions and LFS are enabled, and there are **no explicit storage, retention, package-limit, or quota overrides** in the manifest today. See [deploy/forgejo.yml](deploy/forgejo.yml) lines 39-49 and 174-280. - The existing repo docs already point Kai at a read-only subtree drill on `/var/lib/gitea/data` and explicitly say to prefer Forgejo UI/API cleanup over truncating the PVC. See [docs/forgejo-deploy-plan-followups.md](docs/forgejo-deploy-plan-followups.md) lines 7-13 and [docs/ansible-kai-disk-guardrails-role.md](docs/ansible-kai-disk-guardrails-role.md) lines 50-62. The main storage nuance is that **branches are not large by themselves**. A branch ref is tiny. Branch-heavy growth becomes durable when many long-lived branches keep **unique Git objects** reachable, which prevents packfiles from shrinking. For this deployment, the likely hot paths are: - **Primary branch-heavy path**: Git repositories under `/var/lib/gitea/git/repositories`. - This is where unique commits, trees, and blobs held alive by stale branches accumulate. - If many feature branches are merged but not deleted, their unique objects stay reachable and Git GC cannot reclaim them. - **Other Forgejo-managed asset paths** under `/var/lib/gitea/data` according to Forgejo v15 storage docs: - `lfs/` - `packages/` - `actions_artifacts/` - `actions_log/` - `repo-archive/` - `attachments/` - Source: [Forgejo storage settings](https://forgejo.org/docs/v15.0/admin/setup/storage/) and [config cheat sheet](https://forgejo.org/docs/v15.0/admin/config-cheat-sheet/). - **Not the main branch-heavy risk**: Postgres metadata on the separate DB PVC. - Branch rows, PR metadata, package metadata, and Actions metadata do grow, but they usually do **not** dominate disk the way Git packfiles, LFS, packages, or Actions artifacts do. - The DB still deserves a read-only size check, but the issue’s “branch-heavy local-path PVC growth” is much more likely to be the Forgejo data PVC than the DB PVC. One more important separation: Forgejo’s own docs say **Actions cache is stored by the runner itself and never sent to Forgejo**. In this repo, the runner also has its own PVCs and an `emptyDir` for Docker state, so runner cache pressure is **not** the same thing as Forgejo server PVC pressure. See [deploy/forgejo-runner.yml](deploy/forgejo-runner.yml) and [Forgejo Actions admin docs](https://forgejo.org/docs/v15.0/admin/actions/). ## Safe managed cleanup vs unsafe cleanup ### Safe managed cleanup - **Delete stale merged branches through the Forgejo UI or API**. - This is the only cleanup that actually attacks branch-heavy Git growth at the source. - Risk: deleting branches can break open PR expectations, branch-based automation, or human workflows if done too aggressively. - **Let Forgejo’s deleted-branch cleanup cron remove deleted-branch remnants**. - Forgejo v15 default: `cron.deleted_branches_cleanup` enabled, `OLDER_THAN = 24h`. - Important limitation: this only cleans up **already deleted** branches. It is **not** an instance-wide stale-branch retention policy. - Source: [config cheat sheet](https://forgejo.org/docs/v15.0/admin/config-cheat-sheet/). - **Enable `cron.git_gc_repos` after branch cleanup**. - Forgejo marks repository GC as an extended cron task and it is **disabled by default**. - This is the safe managed mechanism for reclaiming unreachable Git objects after refs are deleted. - Risk: CPU and I/O spikes. Run off-hours and only with backup confidence. - Source: [config cheat sheet](https://forgejo.org/docs/v15.0/admin/config-cheat-sheet/). - **Keep `cron.archive_cleanup` enabled**. - Generated repo archives are already safe to prune through Forgejo. Default is enabled with `OLDER_THAN = 24h`. - This cleans snapshot archives, not the underlying Git repository. - Source: [config cheat sheet](https://forgejo.org/docs/v15.0/admin/config-cheat-sheet/). - **Use Actions retention plus Forgejo’s cleanup cron**. - Forgejo v15 defaults: `LOG_RETENTION_DAYS = 365`, `ARTIFACT_RETENTION_DAYS = 90`. - The admin cron UI can manually trigger cleanup of expired logs and artifacts. - This is safe and managed. It directly targets `actions_log/` and `actions_artifacts/`. - Source: [Forgejo Actions admin docs](https://forgejo.org/docs/v15.0/admin/actions/) and [config cheat sheet](https://forgejo.org/docs/v15.0/admin/config-cheat-sheet/). - **Delete package versions through Forgejo UI/API when they are truly obsolete**. - Forgejo’s `cron.cleanup_packages` only removes **unreferenced** package data older than `24h` by default. It is not a policy for deleting package versions that are still intentionally present. - Source: [config cheat sheet](https://forgejo.org/docs/v15.0/admin/config-cheat-sheet/). - **Consider quotas as a guardrail, not a cleanup tool**. - Forgejo v15 quota is disabled by default and explicitly documented as **soft quota**. - Useful subjects include `size:repos:all`, `size:git:lfs`, `size:assets:artifacts`, and `size:assets:packages:all`. - Risk: soft quotas do not perfectly stop already-started growth, and the feature is still described as early support. - Source: [Soft-Quota docs](https://forgejo.org/docs/v15.0/admin/advanced/quota/) and [config cheat sheet](https://forgejo.org/docs/v15.0/admin/config-cheat-sheet/). ### Unsafe cleanup - `rm -rf` inside `/var/lib/gitea/git/repositories`, `/var/lib/gitea/data/lfs`, `/var/lib/gitea/data/packages`, `/var/lib/gitea/data/actions_*`, or any other Forgejo-managed subtree. - Deleting files directly from the PVC to “match” what the UI should show. - Deleting Git refs or packfiles by hand inside repos instead of via Forgejo/Git-managed commands. Those paths can desynchronize Forgejo’s database, break repository integrity, or remove data that Forgejo still believes exists. ## What Forgejo does **not** give you out of the box The docs point to a real gap that matters for this issue: - Forgejo has built-in cleanup for **deleted branch remnants**, **generated archives**, **expired Actions assets**, and **unreferenced package blobs**. - Forgejo does **not** appear to provide an instance-level policy that says “delete merged branches older than N days” across repos. So the branch-retention decision is operational, not automatic. If Kai wants branch-heavy growth controlled, the actual levers are: - repo-owner discipline on deleting merged branches, - optional tooling/reporting to identify stale branches, - and scheduled Git GC after those deletions. ## Recommended retention defaults These are the defaults I would recommend documenting first, because they protect Forgejo without surprising normal Git use: - **Keep `cron.archive_cleanup` at 24h**. - Low risk. Generated archives are disposable. - **Keep `cron.deleted_branches_cleanup` at 24h**. - Low risk. It only affects branches already deleted. - **Enable `cron.git_gc_repos`**, scheduled off-hours. - Start with Forgejo’s documented `@every 72h` cadence or a similar off-hours interval. - This is the most important missing managed cleanup for branch-heavy repo growth. - **Reduce Actions artifact retention from 90 days to 14-30 days**. - Recommendation: **14 days** unless a repo has a known need for longer retention. - **Reduce Actions log retention from 365 days to 30-60 days**. - Recommendation: **30 days** for a small self-hosted instance unless longer audit history is clearly needed. - **Do not set package or quota limits blindly before measurement**. - If packages are a real hotspot, then add owner/package size limits or quota groups in a second pass. - **Do not change repo-archive retention below 24h unless measurement proves it matters**. - It is already short, and it is probably not the main branch-heavy offender. If Kai wants a single “protect Forgejo first” bias, the best first config changes are: - enable repo Git GC, - shorten Actions log retention, - shorten Actions artifact retention. Those three reduce durable local-path growth without touching live Git history. ## Read-only measurement checklist The existing checklist should be widened beyond `/var/lib/gitea/data`, because the most branch-heavy path is probably in `/var/lib/gitea/git`. ### 1. Top-level Forgejo PVC split ```sh ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "du -xhd1 /var/lib/gitea | sort -h"' ``` Goal: separate `git/` from `data/`. ### 2. Existing documented data-subtree drill ```sh ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "du -xhd1 /var/lib/gitea/data | sort -h"' ``` Goal: identify `lfs`, `packages`, `actions_artifacts`, `actions_log`, `repo-archive`, `attachments` hotspots. ### 3. Largest repositories on disk ```sh ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "du -xhd2 /var/lib/gitea/git/repositories | sort -h | tail -n 50"' ``` Goal: find which repos actually account for Git growth. ### 4. Largest packfiles ```sh ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "find /var/lib/gitea/git/repositories -name \*.pack -printf \"%s %p\\n\" | sort -n | tail -n 50"' ``` Goal: confirm packfile-heavy repos rather than loose-file noise. ### 5. Branch counts on the largest repos For a suspect repo path: ```sh ssh kai-server 'k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc "git -C /var/lib/gitea/git/repositories/<owner>/<repo>.git for-each-ref refs/heads --format=%(refname) | wc -l"' ``` Goal: distinguish “many refs” from “few refs but very large history”. ### 6. Postgres size sanity check ```sh ssh kai-server 'k3s kubectl -n forgejo exec statefulset/forgejo-db -- sh -lc "psql -U forgejo -d forgejo -c \"select pg_size_pretty(pg_database_size('"'"'forgejo'"'"'));\""' ``` Goal: confirm whether DB growth is material or incidental. ### 7. Largest DB relations ```sh ssh kai-server 'k3s kubectl -n forgejo exec statefulset/forgejo-db -- sh -lc "psql -U forgejo -d forgejo -c \"select relname, pg_size_pretty(pg_total_relation_size(oid)) as size from pg_class where relkind = 'r' order by pg_total_relation_size(oid) desc limit 20;\""' ``` Goal: identify whether Actions/package metadata tables are becoming abnormal. ### 8. Admin cron status review In Forgejo admin UI, review `admin/monitor/cron`: - confirm archive cleanup is running, - confirm Actions cleanup is running, - note whether a manual run is needed after any retention change, - confirm whether Git GC is still disabled. ## Open questions to resolve before any live cleanup - Is the real hotspot `git/repositories`, `lfs`, `packages`, `actions_artifacts`, or something else? - Are there repos that intentionally keep long-lived release/support branches? - Do any workflows genuinely need artifacts older than 14-30 days? - Are package registries actively used enough that owner/package limits would be disruptive? - Does Kai want only docs/config landed now, with all live cleanup left as an operator runbook step? That is the safest shape. ## Advisory conclusion The best answer to post back on the issue is: - **Yes, branch-heavy growth deserves its own audit**, but the risky assumption is thinking “branches” means a Forgejo `data/` subtree. The dominant branch-heavy growth is more likely the Git repo store under `/var/lib/gitea/git/repositories`, while `/var/lib/gitea/data` holds other Forgejo-managed assets. - **Safe cleanup is available**, but mostly for generated/expired content: repo archives, Actions logs/artifacts, deleted-branch remnants, and unreferenced package blobs. - **Forgejo does not appear to offer an instance-wide stale-live-branch retention policy**. Live stale branches still need deliberate UI/API deletion, followed by managed Git GC if the goal is real space recovery. - **The first safe config recommendations are**: enable repo Git GC, shorten Actions log retention, shorten Actions artifact retention, and keep existing archive/deleted-branch cleanup in place. - **Do not delete files directly from the PVC**. Measure first, then use Forgejo-managed cleanup paths, and leave any actual live cleanup as an explicit operator action. ### Sources - Local repo: [deploy/forgejo.yml](deploy/forgejo.yml), [deploy/forgejo-runner.yml](deploy/forgejo-runner.yml), [docs/forgejo-deploy-plan-followups.md](docs/forgejo-deploy-plan-followups.md), [docs/ansible-kai-disk-guardrails-role.md](docs/ansible-kai-disk-guardrails-role.md) - Forgejo v15 docs: [storage settings](https://forgejo.org/docs/v15.0/admin/setup/storage/), [config cheat sheet](https://forgejo.org/docs/v15.0/admin/config-cheat-sheet/), [Actions admin guide](https://forgejo.org/docs/v15.0/admin/actions/), [soft quota](https://forgejo.org/docs/v15.0/admin/advanced/quota/) --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it. <!-- ward-agent-reply --> <!-- ward-agent-signature --> — Codex, via `ward agent`

coilyco-ops added the

autonomy

async-consult

priority

P3

labels 2026-07-09 21:46:11 +00:00

coilyco-ops commented 2026-07-10 12:13:04 +00:00

Author

Member

Copy link

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-490 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T12:13:03Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

• Resolved: coilyco-flight-deck/infrastructure#490 · branch issue-490 · harness codex · workflow direct-main
• Run: engineer-codex-infrastructure-490 · ward v0.580.0 · dispatched 2026-07-10T12:13:03Z
• Comment thread: 1 included in the pre-flight read, 0 stripped (ward's own automated comments).
  • included: @coilyco-ops (2026-07-09T18:06:45Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0).

— Codex, via ward agent

<!-- ward-agent-reservation --> WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-490` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T12:13:03Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#490` · branch `issue-490` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-490` · ward `v0.580.0` · dispatched `2026-07-10T12:13:03Z` - **Comment thread:** 1 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-09T18:06:45Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0). </details> </details> <!-- ward-agent-signature --> — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit 2026-07-10 12:19:19 +00:00

docs: add Forgejo retention audit

coilyco-ops closed this issue 2026-07-10 12:19:19 +00:00

coilyco-ops removed the

autonomy

async-consult

label 2026-07-16 01:07:05 +00:00

coilyco-ops referenced this issue 2026-07-23 09:41:38 +00:00

Define and automate Forgejo package and attachment retention after the disk-pressure purge #600

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

ops/393-aos-pin-rollout

ops/708-o11y-telegram-per-host-cache

chore/claude-permissions-allow-all

fix/comfyui-preflight-whatif

fix/comfyui-supervisor-job-object

fix/forgejo-creds-windows-ssm

fix/comfyui-orphan-on-refused-restart

fix/windows-projects-root

fix/comfyui-restart-race

windows-tasks-s4u

windows-attribution-bash-shim-python

fix/comfyui-whatif-verification

windows-terminal-kai-tower-3026

fix/setup-git-lfs-windows-path

chore/drop-ward-yaml

fix/terraform-docs-action-arg

fix/mint-key-orphan

feat/public-asset-host

aos/claude/ka54-failclosed

aos/claude/ka54-cleanup

aos/claude/ka54

aos/claude/uq98-unattended-forgejo

aos/claude/uq98-disk-domains

aos/claude/xk58

aos/claude/aw85-bands

aos/claude/ym96-docs-band

aos/claude/wt57-alerts-to-mapper

aos/claude/xu59-role-trailer

aos/claude/ym96-verb-convert

aos/claude/ym96-token-cleanup

aos/claude/ym96-label-sync

aos/claude/wt57-atlas-cleanup

fix/mirror-unskip-sirens-echo

aos/claude/uz86-session-name

retire-code-fleet-watch-830

codex/alacritty-link-rollout

chore/mcp-beaver-fleet-watch

aos/claude/attribution-interpreter

aos/claude/mg96-fm

aos/claude/js44

coilyco-ops/deprovision-tangled-knot

aos/claude/mg96

aos/claude/zv96

outdated-branch-815

aos/claude/agent-proxy-publisher-runner

aos/claude/git-attribution-pin-interpreter

aos/claude/restore-tag-quarantine

aos/claude/mirror-workflows-permission

aos/claude/forgejo-16

aos/claude/dind-mtu

aos/claude/precommit-egress-proxy

aos/claude/qa57-branch-protection

chore/egress-proxy-open

chore/egress-allow-code-forgejo

aos/claude/qa57

chore/aos-precommit-v0.18-bump

aos/claude/ck67

infra/claude/ser8-runner-force-pull

aos/claude/runner-scratch-10gi

aos/claude/runner-recycle-selector

aos/claude/xk86

aos/codex/20260806t002312z-7943-00371332

codex/sirens-echo-deepseek-tailnet-20260805

codex/daily-aos-sync

issue-756-remove-role-provider-symlink

aos/codex/sirens-echo-runner-rename

aos/codex/20260731t171534z-9817-e7959fba

issue-697-aos-check-config

issue-694-aos-convergence

ops/remove-mac-models-launchd-20260729

ops/remove-mac-models-20260729

tailscale-proxy-converge-self-heal

issue-587

backup/issue-785-first-person

issue-785-first-person

ops/galaxy-gen-registry-rotate

ops/website-deploy-runner

codex/history-derived-skill-20260728

ops/k3s-pod-capacity-config

ops/publisher-runners

codex/code-fleet-watch-v2

codex/code-fleet-watch

recovery/2026-07-28-triaged-branch-archive

recovery/2026-07-27-local-work

recovery/machine-exit-20260727

issue-653-registry-retirement

issue-593

fix/aws-iam-ssm-import

feat/warded-ssm-identity

issue-500

No results found.

Labels

Clear labels   

burndown-2026-06

Backlog burndown June 2026

  

autonomy

async-consult

A human needs to consult on the issue to upgrade it to headless

  

autonomy

epic

This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  

autonomy

headless

The agent can perform the work on its own

  

autonomy

live-collab

The agent and the human need to work together in realtime

  

coherence-core

Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  

priority

P0

priority tier

  

priority

P1

priority tier

  

priority

P2

priority tier

  

priority

P3

priority tier

  

priority

P4

priority tier

  

qa-fixture

Disposable issue admitted to the bounded Ward QA verification lane.

  

role/ai

requires work from the AI Engineer role

  

role/creator

requires work from Content Creator role

  

role/design

requires work from the design role

  

role/director

requires work from the director role

  

role/engineer

requires work from the engineer role

  

role/exec

requires work from the exec role

  

role/human

requires a person, and specifically not an agent seat

  

role/ops

requires work from the ops role

  

role/qa

requires work from the QA role

No labels

burndown-2026-06

autonomy

async-consult

autonomy

epic

autonomy

headless

autonomy

live-collab

coherence-core

priority

P0

priority

P1

priority

P2

priority

P3

priority

P4

qa-fixture

role/ai

role/creator

role/design

role/director

role/engineer

role/exec

role/human

role/ops

role/qa

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#490

No description provided.