coilyco-flight-deck/infrastructure

Watch

Forgejo package registry has no retention rules: apply the Stage 1 cleanup spec (disk pressure since relieved) #903

New issue

Open

opened 2026-08-23 00:24:39 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-08-23 00:24:39 +00:00

Owner

Measured 2026-08-23 ~00:20Z, ops seat, read-only throughout. Nothing was changed. Classification only, per the disk-pressure runbook's contract.

Headline

kai-server root is at 90.75%, status critical, 44 GiB available, 27.6 GiB past the critical threshold.

The dominant consumer is the Forgejo package registry at 93 GB, and it has no cleanup rules configured at all. That is not a tuning problem, it is an absent policy:

===== Forgejo package cleanup rules =====
 owner | type | enabled | keep_count | keep_pattern | remove_days | ...
-------+------+---------+------------+--------------+-------------+----
(0 rows)

Attribution, top down

Every figure below is a complete scan, not a lower bound, unless marked.

/                     479.5 GiB total, 90.75% used, 44.4 GiB available
├── /var              274.7
│   └── /var/lib      271.2
│       ├── rancher   231.6
│       │   ├── k3s/storage   159.9   (local-path PVCs)
│       │   ├── k3s/server/db   8.5   (kine sqlite)
│       │   └── k3s/agent/containerd  ~55 by subtraction, scan timed out
│       ├── kubelet    29.2
│       ├── snapd       5.5
│       └── containerd  4.2
├── /home              85.9   (not investigated)
├── /swapfile          32.0
├── /usr               10.5
└── /srv                6.6

Within the 159.9 GiB of PVCs, one claim is 81% of it:

   130.0 GiB  81.3%  forgejo/forgejo-data
    13.1       8.2   registry/registry-data
     6.5       4.0   forgejo/docker-lib-forgejo-runner-build-0
     1.8-1.9   ea.   forgejo/data-forgejo-runner-{0,1,2,3}
     0.7             forgejo/forgejo-db-data

And inside forgejo-data (139.7 GiB as Forgejo sees it):

    93.3 GiB  packages          <- the container registry
    39.4      data/attachments
     3.5      tmp
     1.9      repo-archive
     1.3      git
     0.33     actions_log
     0.08     indexers

So Forgejo application data is ~133 GiB, about 28% of the entire disk, and two directories account for all of it. Git objects are a rounding error at 1.3 GiB, and PostgreSQL is healthy at 604 MB.

Why the registry cannot self-heal

 total_blobs | total_blob_bytes | unreferenced_blobs | unreferenced_blob_bytes
        9633 | 93 GB            |                  0 | 0 bytes

Zero unreferenced blobs. There is nothing for garbage collection to reclaim. Every byte is held by a live package version, so the only lever is retention, and retention is unset.

Version counts by package:

 owner               | package     | external_versions | referenced_blob_bytes
 coilyco-flight-deck | agentic-os  |               695 | 59 GB
 coilyco-gaming      | sirens-echo |               469 | 29 GB
 coilyco-flight-deck | ward        |                33 | 17 GB
 coilyco-gaming      | eco-app     |                51 | 13 GB
 coilyco-bridge      | website     |               121 | 6592 MB

2,039 package versions retained, and older_than_30d is 0 for every owner. Oldest version is 2026-07-24. So this entire 93 GB accumulated in about 30 days, which is roughly 3 GB/day of permanent growth, and none of it has aged out because nothing ages it out.

agentic-os alone is 695 versions and 59 GB. That is the dev-base image, which publishes five language payloads across two architectures per release.

This reframes agentic-os#1060

coilyco-flight-deck/agentic-os#1060 reports the dev-base release blocked because publish-lang-rust builds but cannot push, with a generic unknown error, at 87% disk. It reasoned circumstantially toward "out of room" and correctly declined to claim it.

The measurement supports that reading and adds the mechanism: the registry is full of 695 prior versions of the very image whose push is failing. The publish pipeline fills the disk that the next publish needs. It is a self-reinforcing loop, not a registry fault, and no amount of retrying publish-lang-rust will clear it.

Tonight's step

The 7-day trend is a sawtooth oscillating 69% to 84%, reset daily by the runner recycle, with the floor creeping up (70%, then 71%, then 72%). Then from roughly 21:00Z it stepped 72% to 90.75% and plateaued.

That correlates with active agentic-os publishing: a package version created 23:33:49Z, a successful release run at 00:18Z, and further runs at 00:15 to 00:17Z. Correlation with timestamps, not a proven causal chain. Stated as the leading hypothesis.

Remediation plan

Stage 1: stop the growth (config only, no data deleted)

Configure Forgejo package cleanup rules per owner and type. This is a supported built-in feature that is currently entirely unconfigured, and it is by far the highest-leverage change. It needs one decision: how many versions to keep per container package, plus whether to express it as keep_count, remove_days, or both.

A keep_count in the 10 to 20 range on container packages would cut agentic-os from 695 versions to a small fraction while keeping every recent rollback target. Immutable-SHA deploy tags mean old versions are only needed as far back as you would realistically roll back.

Stage 2: reclaim (attended, needs a rollback artifact)

Once rules exist, run the managed cleanup then registry GC. Per references/forgejo-storage.md this requires a fresh restic recovery artifact first, since packages are application data under the backup contract. Expected reclaim is most of the 93 GB.

Stage 3: attachments, separately

data/attachments is 39.4 GiB and is not covered by package retention. Nothing here establishes what is writing it. That wants its own measurement before any policy.

Stage 4: structural

The PVC request is fiction. forgejo-data requests 20Gi and holds 139.7 GiB. local-path enforces no quota, so the claim size is documentation rather than a limit. Either move to a quota-enforcing class or monitor the claim directly.
Detection works, reclamation does not. kai-disk-pressure-watch polls from ser8 every five minutes and alerts at 85%, deliberately read-only. The only automated reclaim on the box is the daily runner recycle, which frees DinD scratch and nothing else. Stage 1 may make that acceptable, and that is worth deciding explicitly rather than by default.
k3s/server/db is 8.5 GiB, large for a single-node kine datastore. Cluster events live there and this estate has had sustained event storms. Inference, not established.

Not established

/home at 85.9 GiB is unexamined.
/var/lib/rancher/k3s/agent/containerd is ~55 GiB by subtraction. The direct scan timed out, so that figure is arithmetic rather than measurement.
get_deleted_open_files returned 360 permission errors and timed out, so deleted-but-open files are unmeasured rather than ruled out.
Tonight's step is correlated with publishing, not proven.

coilyco-flight-deck/agentic-os#1060 - the release blocked by this, reframed above
coilyco-flight-deck/infrastructure#901 - three stuck local-path PVs, ~3 GiB, separate mechanism
coilyco-bridge/deploy#677 - closed today, reclaimed 5 GiB

Measured 2026-08-23 ~00:20Z, ops seat, read-only throughout. **Nothing was changed.** Classification only, per the disk-pressure runbook's contract. ## Headline `kai-server` root is at **90.75%, status `critical`**, 44 GiB available, 27.6 GiB past the critical threshold. **The dominant consumer is the Forgejo package registry at 93 GB, and it has no cleanup rules configured at all.** That is not a tuning problem, it is an absent policy: ``` ===== Forgejo package cleanup rules ===== owner | type | enabled | keep_count | keep_pattern | remove_days | ... -------+------+---------+------------+--------------+-------------+---- (0 rows) ``` ## Attribution, top down Every figure below is a complete scan, not a lower bound, unless marked. ``` / 479.5 GiB total, 90.75% used, 44.4 GiB available ├── /var 274.7 │ └── /var/lib 271.2 │ ├── rancher 231.6 │ │ ├── k3s/storage 159.9 (local-path PVCs) │ │ ├── k3s/server/db 8.5 (kine sqlite) │ │ └── k3s/agent/containerd ~55 by subtraction, scan timed out │ ├── kubelet 29.2 │ ├── snapd 5.5 │ └── containerd 4.2 ├── /home 85.9 (not investigated) ├── /swapfile 32.0 ├── /usr 10.5 └── /srv 6.6 ``` Within the 159.9 GiB of PVCs, one claim is 81% of it: ``` 130.0 GiB 81.3% forgejo/forgejo-data 13.1 8.2 registry/registry-data 6.5 4.0 forgejo/docker-lib-forgejo-runner-build-0 1.8-1.9 ea. forgejo/data-forgejo-runner-{0,1,2,3} 0.7 forgejo/forgejo-db-data ``` And inside `forgejo-data` (139.7 GiB as Forgejo sees it): ``` 93.3 GiB packages <- the container registry 39.4 data/attachments 3.5 tmp 1.9 repo-archive 1.3 git 0.33 actions_log 0.08 indexers ``` So **Forgejo application data is ~133 GiB, about 28% of the entire disk**, and two directories account for all of it. Git objects are a rounding error at 1.3 GiB, and PostgreSQL is healthy at 604 MB. ## Why the registry cannot self-heal ``` total_blobs | total_blob_bytes | unreferenced_blobs | unreferenced_blob_bytes 9633 | 93 GB | 0 | 0 bytes ``` **Zero unreferenced blobs.** There is nothing for garbage collection to reclaim. Every byte is held by a live package version, so the only lever is retention, and retention is unset. Version counts by package: ``` owner | package | external_versions | referenced_blob_bytes coilyco-flight-deck | agentic-os | 695 | 59 GB coilyco-gaming | sirens-echo | 469 | 29 GB coilyco-flight-deck | ward | 33 | 17 GB coilyco-gaming | eco-app | 51 | 13 GB coilyco-bridge | website | 121 | 6592 MB ``` **2,039 package versions retained, and `older_than_30d` is 0 for every owner.** Oldest version is 2026-07-24. So this entire 93 GB accumulated in about 30 days, which is roughly **3 GB/day of permanent growth**, and none of it has aged out because nothing ages it out. `agentic-os` alone is 695 versions and 59 GB. That is the dev-base image, which publishes five language payloads across two architectures per release. ## This reframes agentic-os#1060 `coilyco-flight-deck/agentic-os#1060` reports the dev-base release blocked because `publish-lang-rust` builds but cannot push, with a generic `unknown` error, at 87% disk. It reasoned circumstantially toward "out of room" and correctly declined to claim it. The measurement supports that reading and adds the mechanism: **the registry is full of 695 prior versions of the very image whose push is failing.** The publish pipeline fills the disk that the next publish needs. It is a self-reinforcing loop, not a registry fault, and no amount of retrying `publish-lang-rust` will clear it. ## Tonight's step The 7-day trend is a sawtooth oscillating 69% to 84%, reset daily by the runner recycle, with the floor creeping up (70%, then 71%, then 72%). Then from roughly 21:00Z it stepped 72% to 90.75% and plateaued. That correlates with active `agentic-os` publishing: a package version created 23:33:49Z, a successful `release` run at 00:18Z, and further runs at 00:15 to 00:17Z. **Correlation with timestamps, not a proven causal chain.** Stated as the leading hypothesis. ## Remediation plan ### Stage 1: stop the growth (config only, no data deleted) Configure Forgejo package cleanup rules per owner and type. This is a supported built-in feature that is currently entirely unconfigured, and it is by far the highest-leverage change. It needs one decision: **how many versions to keep per container package**, plus whether to express it as `keep_count`, `remove_days`, or both. A `keep_count` in the 10 to 20 range on container packages would cut `agentic-os` from 695 versions to a small fraction while keeping every recent rollback target. Immutable-SHA deploy tags mean old versions are only needed as far back as you would realistically roll back. ### Stage 2: reclaim (attended, needs a rollback artifact) Once rules exist, run the managed cleanup then registry GC. Per `references/forgejo-storage.md` this requires a fresh restic recovery artifact first, since packages are application data under the backup contract. Expected reclaim is most of the 93 GB. ### Stage 3: attachments, separately `data/attachments` is 39.4 GiB and is **not** covered by package retention. Nothing here establishes what is writing it. That wants its own measurement before any policy. ### Stage 4: structural * **The PVC request is fiction.** `forgejo-data` requests `20Gi` and holds 139.7 GiB. local-path enforces no quota, so the claim size is documentation rather than a limit. Either move to a quota-enforcing class or monitor the claim directly. * **Detection works, reclamation does not.** `kai-disk-pressure-watch` polls from ser8 every five minutes and alerts at 85%, deliberately read-only. The only automated reclaim on the box is the daily runner recycle, which frees DinD scratch and nothing else. Stage 1 may make that acceptable, and that is worth deciding explicitly rather than by default. * `k3s/server/db` is **8.5 GiB**, large for a single-node kine datastore. Cluster events live there and this estate has had sustained event storms. Inference, not established. ## Not established * `/home` at 85.9 GiB is unexamined. * `/var/lib/rancher/k3s/agent/containerd` is ~55 GiB by subtraction. The direct scan timed out, so that figure is arithmetic rather than measurement. * `get_deleted_open_files` returned 360 permission errors and timed out, so deleted-but-open files are unmeasured rather than ruled out. * Tonight's step is correlated with publishing, not proven. ## Related * `coilyco-flight-deck/agentic-os#1060` - the release blocked by this, reframed above * `coilyco-flight-deck/infrastructure#901` - three stuck local-path PVs, ~3 GiB, separate mechanism * `coilyco-bridge/deploy#677` - closed today, reclaimed 5 GiB

coilyco-ops commented

2026-08-23 00:56:04 +00:00

Author

Owner

Stage 1 spec: Forgejo package cleanup rules

Grounded in the running version (data.forgejo.org/forgejo/forgejo:16.0.2-rootless), the upstream source, and this instance's own swagger. Config only. Deletes nothing until Stage 2.

What Forgejo actually offers

One rule is a PackageCleanupRule row with seven configurable fields:

Enabled - on/off
Type - the package type the rule governs (container, generic, ...)
KeepCount - how many versions to always keep, per package
KeepPattern - regex, matches are never deleted
RemoveDays - only versions older than N days are eligible
RemovePattern - regex, only matches are eligible
MatchFullName - patterns match package/version rather than bare version

Scope, and why it is workable here

The model carries UNIQUE(s) across OwnerID and Type, so there is exactly one rule per owner per package type. Rules cannot be written per package.

KeepCount applies per package inside the rule, which is what makes a single rule per owner acceptable. One rule on coilyco-flight-deck/container trims agentic-os from 695 versions while leaving bluesky-mcp at 11 and quire at 1 untouched.

Two semantics that determine the config

KeepCount alone is sufficient. services/packages/cleanup/cleanup.go runs protections in order and deletes when none fire:

skip digest-based container versions (sha256:), handled by mass-digest cleanup
keep if within KeepCount
keep if KeepPattern matches
keep if newer than RemoveDays
keep if RemovePattern is set and does not match

With RemovePattern empty and RemoveDays zero, checks 4 and 5 never protect anything. They are narrowing filters, not prerequisites.

RemoveDays is a no-op on this data. The measurement above shows older_than_30d = 0 for every owner, oldest version 2026-07-24. Any age-based rule deletes zero bytes. This has to be KeepCount.

The rules

Four rules, KeepCount = 15, every other field left empty.

owner                 type       enabled  keep_count  keep_pattern  remove_days  remove_pattern  match_full_name
coilyco-flight-deck   container  true     15          (empty)       0            (empty)         false
coilyco-gaming        container  true     15          (empty)       0            (empty)         false
coilyco-bridge        container  true     15          (empty)       0            (empty)         false
coilyco-gaming        generic    true     15          (empty)       0            (empty)         false

Projected effect on version counts, derived from the per-package table above:

coilyco-flight-deck - 896 to ~137. agentic-os 695→15, mcp-beaver 47→15, agent-proxy 38→15, ward 33→15, ward-mcp 18→15, code-fleet-watch 18→15. Untouched: node-stats-mcp 15, bluesky-mcp 11, reddit-mcp 10, lunch-money-mcp 10, quire 1
coilyco-gaming container - 621 to ~86. sirens-echo 469→15, eco-app 51→15, galaxy-gen 36→15, factory-game-v3 33→15, steam-mcp 21→15. Untouched: sirens-discord-ops 11
coilyco-bridge - 138 to ~32. website 121→15. Untouched: reference-media 12, atlas 4, playwright-mcp 1
coilyco-gaming generic - 384 versions, only 46 MB total. Included for hygiene, not for bytes

Roughly 1,400 of 2,039 versions removed. No byte estimate is given deliberately: container layers are shared across versions, so reclaim is not linear in version count. The preview produces the real number.

Why 15 and not lower: deploys pin immutable source SHAs, so old versions matter only as far back as a realistic rollback. Fifteen covers well over a week for every package except the two highest-churn ones, and the number is trivially adjustable before anything is applied.

Sequencing, preview first

Create all four rules disabled.
Run the built-in preview on each. Every rule can list exactly which versions it would affect, which is a genuine dry run.
Paste the preview output into this issue. That is the byte figure the plan currently lacks, and Stage 2 wants it recorded anyway.
Only then enable, and only after the Stage 2 restic recovery artifact exists.

Risk to weigh before enabling

Forgejo #9358 reports container cleanup pinning one core at 100% for about 50 minutes, with pages timing out and database queries taking 40 seconds, on an instance holding 3,178 artifacts / 42 GiB.

This registry is 9,633 blobs / 93 GB, roughly double that. Two things to check first:

The fix skips sha256: digest versions and defers them to mass-digest cleanup (#10297, backported to v13.0). This instance runs 16.0.2, so it should be included. That is inference from version ordering and has not been verified against the running binary.
The forgejo pod already shows 21 restarts, most recent 9h ago. A CPU-heavy cleanup pass is exactly the load that would worsen that, and preview runs the same query path as the rule.

Preferably run the first enabled pass at a quiet hour with the pod under observation.

These rules cannot be tracked in git, and that is a real gap

Checked this instance's own swagger.v1.json. There is no cleanup-rule endpoint. The complete package API surface is:

/packages/{owner}                                   get
/packages/{owner}/{type}/{name}/{version}           get, delete
/packages/{owner}/{type}/{name}/{version}/files     get
/packages/{owner}/{type}/{name}/-/link/{repo_name}  post
/packages/{owner}/{type}/{name}/-/unlink            post
/orgs/{org}/quota/packages                          get
/user/quota/packages                                get

So cleanup rules are web UI only, persisted in the package_cleanup_rule table with no tracked source. That conflicts with this estate's config-placement rule, and it means the retention policy becomes hand-set state that nothing converges or validates.

Two options, and I recommend the first:

Accept the UI rules. The disk-pressure runbook explicitly says to prefer Forgejo-managed retention, and the native path gets the mass-digest optimisation that a hand-rolled loop would not. Record the four rows in this issue so the intended state is at least written down, and file a follow-up for a doctor-style drift check that reads the table and compares against a tracked expectation.
Script it against DELETE /packages/{owner}/{type}/{name}/{version}. Tracked in git, but it reimplements a feature Forgejo already has, deletes version by version, and would likely hit exactly the performance profile #9358 describes.

Verification after enabling

aosguard ops forgejo-storage measure reruns cleanly and shows packages well below 93.3 GiB
unreferenced_blobs becomes non-zero between the rule pass and blob GC, then returns to 0
get_filesystem_pressure returns status below critical
forgejo pod restart count does not increase across the pass

Out of scope for Stage 1

data/attachments at 39.4 GiB is untouched by package retention and stays on Stage 3. Quotas (/orgs/{org}/quota/packages exists in the API) are a future guardrail per the runbook, not a cleanup mechanism, and belong in Stage 4.

## Stage 1 spec: Forgejo package cleanup rules Grounded in the running version (`data.forgejo.org/forgejo/forgejo:16.0.2-rootless`), the upstream source, and this instance's own swagger. Config only. **Deletes nothing until Stage 2.** ### What Forgejo actually offers One rule is a `PackageCleanupRule` row with seven configurable fields: * `Enabled` - on/off * `Type` - the package type the rule governs (`container`, `generic`, ...) * `KeepCount` - how many versions to always keep, **per package** * `KeepPattern` - regex, matches are never deleted * `RemoveDays` - only versions older than N days are eligible * `RemovePattern` - regex, only matches are eligible * `MatchFullName` - patterns match `package/version` rather than bare `version` ### Scope, and why it is workable here The model carries `UNIQUE(s)` across `OwnerID` and `Type`, so there is **exactly one rule per owner per package type**. Rules cannot be written per package. **`KeepCount` applies per package inside the rule**, which is what makes a single rule per owner acceptable. One rule on `coilyco-flight-deck/container` trims `agentic-os` from 695 versions while leaving `bluesky-mcp` at 11 and `quire` at 1 untouched. ### Two semantics that determine the config **`KeepCount` alone is sufficient.** `services/packages/cleanup/cleanup.go` runs protections in order and deletes when none fire: 1. skip digest-based container versions (`sha256:`), handled by mass-digest cleanup 2. keep if within `KeepCount` 3. keep if `KeepPattern` matches 4. keep if newer than `RemoveDays` 5. keep if `RemovePattern` is set and does **not** match With `RemovePattern` empty and `RemoveDays` zero, checks 4 and 5 never protect anything. They are narrowing filters, not prerequisites. **`RemoveDays` is a no-op on this data.** The measurement above shows `older_than_30d = 0` for every owner, oldest version 2026-07-24. Any age-based rule deletes zero bytes. This has to be `KeepCount`. ### The rules Four rules, `KeepCount = 15`, every other field left empty. ``` owner type enabled keep_count keep_pattern remove_days remove_pattern match_full_name coilyco-flight-deck container true 15 (empty) 0 (empty) false coilyco-gaming container true 15 (empty) 0 (empty) false coilyco-bridge container true 15 (empty) 0 (empty) false coilyco-gaming generic true 15 (empty) 0 (empty) false ``` Projected effect on version counts, derived from the per-package table above: * `coilyco-flight-deck` - 896 to ~137. agentic-os 695→15, mcp-beaver 47→15, agent-proxy 38→15, ward 33→15, ward-mcp 18→15, code-fleet-watch 18→15. Untouched: node-stats-mcp 15, bluesky-mcp 11, reddit-mcp 10, lunch-money-mcp 10, quire 1 * `coilyco-gaming` container - 621 to ~86. sirens-echo 469→15, eco-app 51→15, galaxy-gen 36→15, factory-game-v3 33→15, steam-mcp 21→15. Untouched: sirens-discord-ops 11 * `coilyco-bridge` - 138 to ~32. website 121→15. Untouched: reference-media 12, atlas 4, playwright-mcp 1 * `coilyco-gaming` generic - 384 versions, only 46 MB total. Included for hygiene, not for bytes Roughly **1,400 of 2,039 versions removed**. **No byte estimate is given deliberately**: container layers are shared across versions, so reclaim is not linear in version count. The preview produces the real number. Why 15 and not lower: deploys pin immutable source SHAs, so old versions matter only as far back as a realistic rollback. Fifteen covers well over a week for every package except the two highest-churn ones, and the number is trivially adjustable before anything is applied. ### Sequencing, preview first 1. Create all four rules **disabled**. 2. Run the built-in **preview** on each. Every rule can list exactly which versions it would affect, which is a genuine dry run. 3. Paste the preview output into this issue. That is the byte figure the plan currently lacks, and Stage 2 wants it recorded anyway. 4. Only then enable, and only after the Stage 2 restic recovery artifact exists. ### Risk to weigh before enabling Forgejo [#9358](https://codeberg.org/forgejo/forgejo/issues/9358) reports container cleanup pinning one core at 100% for about 50 minutes, with pages timing out and database queries taking 40 seconds, on an instance holding **3,178 artifacts / 42 GiB**. **This registry is 9,633 blobs / 93 GB, roughly double that.** Two things to check first: * The fix skips `sha256:` digest versions and defers them to mass-digest cleanup ([#10297](https://codeberg.org/forgejo/forgejo/pulls/10334), backported to v13.0). This instance runs 16.0.2, so it should be included. **That is inference from version ordering and has not been verified against the running binary.** * The `forgejo` pod already shows **21 restarts**, most recent 9h ago. A CPU-heavy cleanup pass is exactly the load that would worsen that, and preview runs the same query path as the rule. Preferably run the first enabled pass at a quiet hour with the pod under observation. ### These rules cannot be tracked in git, and that is a real gap Checked this instance's own `swagger.v1.json`. **There is no cleanup-rule endpoint.** The complete package API surface is: ``` /packages/{owner} get /packages/{owner}/{type}/{name}/{version} get, delete /packages/{owner}/{type}/{name}/{version}/files get /packages/{owner}/{type}/{name}/-/link/{repo_name} post /packages/{owner}/{type}/{name}/-/unlink post /orgs/{org}/quota/packages get /user/quota/packages get ``` So cleanup rules are **web UI only**, persisted in the `package_cleanup_rule` table with no tracked source. That conflicts with this estate's config-placement rule, and it means the retention policy becomes hand-set state that nothing converges or validates. Two options, and I recommend the first: * **Accept the UI rules.** The disk-pressure runbook explicitly says to prefer Forgejo-managed retention, and the native path gets the mass-digest optimisation that a hand-rolled loop would not. Record the four rows in this issue so the intended state is at least written down, and file a follow-up for a `doctor`-style drift check that reads the table and compares against a tracked expectation. * **Script it against `DELETE /packages/{owner}/{type}/{name}/{version}`.** Tracked in git, but it reimplements a feature Forgejo already has, deletes version by version, and would likely hit exactly the performance profile #9358 describes. ### Verification after enabling * `aosguard ops forgejo-storage measure` reruns cleanly and shows `packages` well below 93.3 GiB * `unreferenced_blobs` becomes non-zero between the rule pass and blob GC, then returns to 0 * `get_filesystem_pressure` returns `status` below `critical` * `forgejo` pod restart count does not increase across the pass ### Out of scope for Stage 1 `data/attachments` at 39.4 GiB is untouched by package retention and stays on Stage 3. Quotas (`/orgs/{org}/quota/packages` exists in the API) are a future guardrail per the runbook, not a cleanup mechanism, and belong in Stage 4.

coilyco-ops commented

2026-08-23 01:26:00 +00:00

Author

Owner

Stage 1 canary result: 29 GB reclaimed from one owner, out of `critical`

Kai enabled the coilyco-flight-deck / container rule and ran the cleanup_packages cron from the admin UI. Rule as applied was keep_count = 10, remove_days = 7, which is more conservative than the spec above proposed: a version must be both outside the ten most recent and older than seven days.

Preview reported 219 versions eligible.

Result

                        before        after       delta
root filesystem         90.75%        83.73%      -7.02 pts
                        critical      warning     out of critical
available               47.6 GiB      83.8 GiB    +36.2 GiB
inodes used              9.34%         8.65%

/var/lib/gitea         139.7 GiB     113.4 GiB    -26.3 GiB
  packages              93.3 GiB      66.5 GiB    -26.8 GiB
  data/attachments      39.4 GiB      39.9 GiB    unchanged

package blobs             9,633         7,882     -1,751
package blob bytes         93 GB         66 GB    -27 GB
unreferenced_blobs             0             0    prune completed in-run

Owner split, showing the change is entirely the treated owner:

                                    before              after
coilyco-flight-deck  container   3,283 / 64 GB      1,448 / 35 GB
coilyco-gaming       container   5,682 / 35 GB      5,722 / 35 GB   untreated
coilyco-bridge       container     521 / 8163 MB      521 / 8163 MB untreated
coilyco-gaming       generic       434 / 46 MB        434 / 46 MB   untreated

One owner gave up 29 GB and 1,835 blobs. Root filesystem is now 6.5 GiB below the critical line for the first time tonight.

The three questions this canary was built to answer

Does retention actually reclaim bytes, given the blob sharing? Yes, decisively. The open worry was that 695 versions sharing 2,465 blobs meant deleting tagged versions would unreference almost nothing. Deleting ~219 version rows removed 1,835 blobs, so sharing was nowhere near total. Retention is the right lever.

Does the blob prune lag behind the rule? No. unreferenced_blobs is 0 both before and after, and bytes dropped in the same pass, so OLDER_THAN did not gate anything. No second run is needed.

Does forgejo#9358 bite at this scale? No. Peak observed CPU was 466m with memory at 1005Mi, restart count held at 21 (unchanged, still 10h old), and the site answered HTTP 200 in 0.028s throughout. The digest-skip fix is evidently present in 16.0.2. The sha256: rows are exactly why: they are ~58% of version rows and the rule passes straight over them.

The 1.6 KiB red herring, recorded

The preview showed the 219 versions at roughly 1.6 KiB each, about 350 KiB in total, which reads as pointless. That figure is the manifest size, not the reclaim. A container version row holds a small JSON manifest and the payload lives in shared blobs, so the preview is structurally unable to show what a deletion frees. It understated the real result by roughly five orders of magnitude.

Worth knowing for Stage 2 planning: the preview's byte column should be ignored entirely. Version count is a weak proxy and manifest size is no proxy at all.

What this predicts for the remaining owners

coilyco-flight-deck went 64 GB to 35 GB, a 45% reduction, under a deliberately conservative rule. Applying the same shape:

coilyco-gaming container - 35 GB, 621 versions, sirens-echo dominant at 469. Note it holds more blobs than flight-deck did (5,722) for the same bytes, so its blobs are smaller on average and the ratio may differ
coilyco-bridge container - 8163 MB, 138 versions, website dominant at 121
coilyco-gaming generic - 46 MB, not worth a run on its own

A comparable outcome puts packages near 47 GiB and root near 79%, back under the warn line.

Recommended next steps

Apply the same rule to coilyco-gaming / container and coilyco-bridge / container, one at a time, running the cron between each so attribution stays clean.
Stage 3 is now the largest single item. With packages at 66.5 GiB, data/attachments at 39.9 GiB is the second-biggest consumer and no package rule touches it. It has not moved and nothing yet establishes what writes it.
Still open from the spec above: these rules have no API and no tracked source. Now that they are demonstrably load-bearing, the drift-check follow-up matters more, not less.
The four lang-*-buildcache moving tags remain unexamined. No keep_count rule touches a moving tag, and each rebuild re-points them, so they are a plausible home for retained blobs. Worth a look once the other owners are done.

Nothing else on the cluster was changed.

## Stage 1 canary result: 29 GB reclaimed from one owner, out of `critical` Kai enabled the `coilyco-flight-deck` / `container` rule and ran the `cleanup_packages` cron from the admin UI. Rule as applied was **`keep_count = 10`, `remove_days = 7`**, which is more conservative than the spec above proposed: a version must be both outside the ten most recent **and** older than seven days. Preview reported 219 versions eligible. ### Result ``` before after delta root filesystem 90.75% 83.73% -7.02 pts critical warning out of critical available 47.6 GiB 83.8 GiB +36.2 GiB inodes used 9.34% 8.65% /var/lib/gitea 139.7 GiB 113.4 GiB -26.3 GiB packages 93.3 GiB 66.5 GiB -26.8 GiB data/attachments 39.4 GiB 39.9 GiB unchanged package blobs 9,633 7,882 -1,751 package blob bytes 93 GB 66 GB -27 GB unreferenced_blobs 0 0 prune completed in-run ``` Owner split, showing the change is entirely the treated owner: ``` before after coilyco-flight-deck container 3,283 / 64 GB 1,448 / 35 GB coilyco-gaming container 5,682 / 35 GB 5,722 / 35 GB untreated coilyco-bridge container 521 / 8163 MB 521 / 8163 MB untreated coilyco-gaming generic 434 / 46 MB 434 / 46 MB untreated ``` **One owner gave up 29 GB and 1,835 blobs.** Root filesystem is now 6.5 GiB below the critical line for the first time tonight. ### The three questions this canary was built to answer **Does retention actually reclaim bytes, given the blob sharing?** Yes, decisively. The open worry was that 695 versions sharing 2,465 blobs meant deleting tagged versions would unreference almost nothing. Deleting ~219 version rows removed **1,835 blobs**, so sharing was nowhere near total. Retention is the right lever. **Does the blob prune lag behind the rule?** No. `unreferenced_blobs` is 0 both before and after, and bytes dropped in the same pass, so `OLDER_THAN` did not gate anything. No second run is needed. **Does forgejo#9358 bite at this scale?** No. Peak observed CPU was **466m** with memory at 1005Mi, restart count held at **21** (unchanged, still 10h old), and the site answered `HTTP 200` in 0.028s throughout. The digest-skip fix is evidently present in 16.0.2. The `sha256:` rows are exactly why: they are ~58% of version rows and the rule passes straight over them. ### The 1.6 KiB red herring, recorded The preview showed the 219 versions at roughly 1.6 KiB each, about 350 KiB in total, which reads as pointless. That figure is the **manifest size**, not the reclaim. A container version row holds a small JSON manifest and the payload lives in shared blobs, so the preview is structurally unable to show what a deletion frees. It understated the real result by roughly five orders of magnitude. **Worth knowing for Stage 2 planning: the preview's byte column should be ignored entirely.** Version count is a weak proxy and manifest size is no proxy at all. ### What this predicts for the remaining owners `coilyco-flight-deck` went 64 GB to 35 GB, a 45% reduction, under a deliberately conservative rule. Applying the same shape: * `coilyco-gaming` container - 35 GB, 621 versions, `sirens-echo` dominant at 469. Note it holds **more blobs than flight-deck did (5,722) for the same bytes**, so its blobs are smaller on average and the ratio may differ * `coilyco-bridge` container - 8163 MB, 138 versions, `website` dominant at 121 * `coilyco-gaming` generic - 46 MB, not worth a run on its own A comparable outcome puts packages near 47 GiB and root near 79%, back under the warn line. ### Recommended next steps 1. Apply the same rule to `coilyco-gaming` / `container` and `coilyco-bridge` / `container`, one at a time, running the cron between each so attribution stays clean. 2. **Stage 3 is now the largest single item.** With packages at 66.5 GiB, `data/attachments` at **39.9 GiB** is the second-biggest consumer and no package rule touches it. It has not moved and nothing yet establishes what writes it. 3. Still open from the spec above: these rules have **no API and no tracked source**. Now that they are demonstrably load-bearing, the drift-check follow-up matters more, not less. 4. The four `lang-*-buildcache` moving tags remain unexamined. No `keep_count` rule touches a moving tag, and each rebuild re-points them, so they are a plausible home for retained blobs. Worth a look once the other owners are done. Nothing else on the cluster was changed.

coilyco-ops commented

2026-08-23 01:32:37 +00:00

Author

Owner

Stage 1 complete for two owners: root filesystem is `ok` for the first time tonight

coilyco-gaming rules applied and the cron run. Cumulative state across tonight's three measurements:

                        start      after FD     after gaming
root filesystem         90.75%      83.73%        78.80%
status                  critical    warning       ok
available              47.6 GiB    83.8 GiB     109.1 GiB

/var/lib/gitea         139.7 GiB   113.4 GiB     89.9 GiB
  packages              93.3 GiB    66.5 GiB     43.0 GiB
  data/attachments      39.4 GiB    39.9 GiB     39.9 GiB   untouched

package blobs             9,633       7,882         3,245
package blob bytes         93 GB       66 GB         43 GB
unreferenced_blobs             0           0             0

57.3 GiB reclaimed in total. Root is 6.2 GiB below the warn line and 31.9 GiB below critical.

Owner split

                                start          after FD        after gaming
coilyco-flight-deck container  3,283 / 64 GB   1,448 / 35 GB   1,108 / 22 GB
coilyco-gaming      container  5,682 / 35 GB   5,722 / 35 GB   1,548 / 19 GB
coilyco-gaming      generic      434 / 46 MB     434 / 46 MB     171 / 29 MB
coilyco-bridge      container    521 / 8163 MB   521 / 8163 MB   521 / 8163 MB   UNTREATED

coilyco-bridge is unchanged across all three readings, which is a clean control: every byte reclaimed is attributable to a rule that was actually enabled.

Note that coilyco-flight-deck continued to shrink from 35 GB to 22 GB during the gaming pass. The post-flight-deck measurement was taken while that cleanup was still completing, so the 29 GB figure in my previous comment understated it. Flight-deck's true contribution is 42 GB, not 29.

Version counts

owner                type       packages  versions  (was)   older_than_7d
coilyco-flight-deck  container        11       267   896              59
coilyco-gaming       container         6       142   621              35
coilyco-gaming       generic          13       139   384              20
coilyco-bridge       container         4       138   138             118   untreated

2,039 versions down to 686. The older_than_7d column is now small everywhere treated, which is the steady state the remove_days = 7 guard produces: roughly a week of history per package plus the ten most recent.

Health throughout

No degradation at any point across two cleanup passes on a registry that started at 93 GB:

peak CPU 586m, memory 1127Mi
forgejo pod restart count held at 21 for the entire operation, unchanged and still 10h old
site answered HTTP 200 in 0.025s during the gaming pass

forgejo#9358 does not apply to 16.0.2 at this scale. Recording that plainly so the next operator does not treat it as a blocker.

What is left

coilyco-bridge / container - the last untreated owner. 521 blobs, 8163 MB, 138 versions, website dominant at 121. Small, and the only remaining Stage 1 work.
data/attachments at 39.9 GiB is now effectively tied with packages (43.0 GiB) as the largest consumer, and after bridge it becomes the single largest. It has not moved by a byte all night and no package rule touches it. Stage 3 is now the main event, and nothing yet establishes what writes it.
lang-*-buildcache moving tags, still unexamined.
The no-API / no-tracked-source gap from the Stage 1 spec. These rules just moved 57 GiB, so they are load-bearing config with nothing converging or validating them.

The structural change, which matters more than the bytes

Before tonight the registry grew about 3 GB/day with nothing ageing out and older_than_30d = 0 only because the instance had not been running long enough for anything to reach 30 days. That was unbounded.

Retention now caps it. Steady-state package storage becomes a function of publish rate times the retention window rather than a monotonic climb, which is what actually resolves agentic-os#1060 rather than deferring it.

## Stage 1 complete for two owners: root filesystem is `ok` for the first time tonight `coilyco-gaming` rules applied and the cron run. Cumulative state across tonight's three measurements: ``` start after FD after gaming root filesystem 90.75% 83.73% 78.80% status critical warning ok available 47.6 GiB 83.8 GiB 109.1 GiB /var/lib/gitea 139.7 GiB 113.4 GiB 89.9 GiB packages 93.3 GiB 66.5 GiB 43.0 GiB data/attachments 39.4 GiB 39.9 GiB 39.9 GiB untouched package blobs 9,633 7,882 3,245 package blob bytes 93 GB 66 GB 43 GB unreferenced_blobs 0 0 0 ``` **57.3 GiB reclaimed in total.** Root is 6.2 GiB below the warn line and 31.9 GiB below critical. ### Owner split ``` start after FD after gaming coilyco-flight-deck container 3,283 / 64 GB 1,448 / 35 GB 1,108 / 22 GB coilyco-gaming container 5,682 / 35 GB 5,722 / 35 GB 1,548 / 19 GB coilyco-gaming generic 434 / 46 MB 434 / 46 MB 171 / 29 MB coilyco-bridge container 521 / 8163 MB 521 / 8163 MB 521 / 8163 MB UNTREATED ``` `coilyco-bridge` is unchanged across all three readings, which is a clean control: every byte reclaimed is attributable to a rule that was actually enabled. **Note that `coilyco-flight-deck` continued to shrink** from 35 GB to 22 GB during the gaming pass. The post-flight-deck measurement was taken while that cleanup was still completing, so the 29 GB figure in my previous comment understated it. Flight-deck's true contribution is **42 GB**, not 29. ### Version counts ``` owner type packages versions (was) older_than_7d coilyco-flight-deck container 11 267 896 59 coilyco-gaming container 6 142 621 35 coilyco-gaming generic 13 139 384 20 coilyco-bridge container 4 138 138 118 untreated ``` 2,039 versions down to 686. The `older_than_7d` column is now small everywhere treated, which is the steady state the `remove_days = 7` guard produces: roughly a week of history per package plus the ten most recent. ### Health throughout No degradation at any point across two cleanup passes on a registry that started at 93 GB: * peak CPU **586m**, memory 1127Mi * `forgejo` pod restart count held at **21** for the entire operation, unchanged and still 10h old * site answered `HTTP 200` in 0.025s during the gaming pass forgejo#9358 does not apply to 16.0.2 at this scale. Recording that plainly so the next operator does not treat it as a blocker. ### What is left 1. **`coilyco-bridge` / `container`** - the last untreated owner. 521 blobs, 8163 MB, 138 versions, `website` dominant at 121. Small, and the only remaining Stage 1 work. 2. **`data/attachments` at 39.9 GiB is now effectively tied with packages** (43.0 GiB) as the largest consumer, and after bridge it becomes the single largest. It has not moved by a byte all night and no package rule touches it. **Stage 3 is now the main event**, and nothing yet establishes what writes it. 3. `lang-*-buildcache` moving tags, still unexamined. 4. The no-API / no-tracked-source gap from the Stage 1 spec. These rules just moved 57 GiB, so they are load-bearing config with nothing converging or validating them. ### The structural change, which matters more than the bytes Before tonight the registry grew about **3 GB/day with nothing ageing out** and `older_than_30d = 0` only because the instance had not been running long enough for anything to reach 30 days. That was unbounded. Retention now caps it. Steady-state package storage becomes a function of publish rate times the retention window rather than a monotonic climb, which is what actually resolves agentic-os#1060 rather than deferring it.

coilyco-ops commented

2026-08-25 05:46:21 +00:00

Author

Owner

The pressure is gone. The spec in the body is not implemented. Re-scoping rather than closing, because those are two different things and the title only tracks the first.

Live state

/dev/nvme0n1p7  480G  316G  140G  70%  /

70.86% used, status: ok, 72 GB until the 85% critical line. The node's DiskPressure condition last transitioned to False at 2026-08-23T00:35:35Z, eleven minutes after this issue was filed.

So the 90.75% in the title is two days stale and nothing is on fire.

What did not cause the relief

forgejo-package-upload-cleanup runs every 15 minutes and is healthy, but it is 31 days old, so it predates this issue and is not the fix. It also cleans up incomplete uploads rather than enforcing registry retention, which is what the body specifies.

I did not establish what actually reclaimed the ~20 points. Recording that as unknown rather than guessing.

What is still open

The Stage 1 spec in the body stands on its own merits and is unimplemented. It says plainly that it deletes nothing until Stage 2, so it was never going to relieve pressure by itself. The four PackageCleanupRule rows at KeepCount = 15 are still worth having, because without them the registry has no retention at all and the growth that produced this incident can simply recur.

The reasoning in the body is good and I am not relitigating it. KeepCount alone being sufficient, RemoveDays being a no-op against data with older_than_30d = 0, and one rule per owner per type being workable because KeepCount applies per package are all specific and checked against the running version.

Suggested re-scope

This stops being an incident and becomes planned work: apply the Stage 1 rules, then decide on Stage 2. No urgency at 70% with 140 GB free, but it should not be closed as though the underlying gap were addressed.

Retitling accordingly.

**The pressure is gone. The spec in the body is not implemented.** Re-scoping rather than closing, because those are two different things and the title only tracks the first. ## Live state ``` /dev/nvme0n1p7 480G 316G 140G 70% / ``` `70.86%` used, `status: ok`, 72 GB until the 85% critical line. The node's `DiskPressure` condition last transitioned to `False` at `2026-08-23T00:35:35Z`, eleven minutes after this issue was filed. So the 90.75% in the title is two days stale and nothing is on fire. ## What did not cause the relief `forgejo-package-upload-cleanup` runs every 15 minutes and is healthy, but it is **31 days old**, so it predates this issue and is not the fix. It also cleans up incomplete uploads rather than enforcing registry retention, which is what the body specifies. I did not establish what actually reclaimed the ~20 points. Recording that as unknown rather than guessing. ## What is still open The Stage 1 spec in the body stands on its own merits and is unimplemented. It says plainly that it **deletes nothing until Stage 2**, so it was never going to relieve pressure by itself. The four `PackageCleanupRule` rows at `KeepCount = 15` are still worth having, because without them the registry has no retention at all and the growth that produced this incident can simply recur. The reasoning in the body is good and I am not relitigating it. `KeepCount` alone being sufficient, `RemoveDays` being a no-op against data with `older_than_30d = 0`, and one rule per owner per type being workable because `KeepCount` applies per package are all specific and checked against the running version. ## Suggested re-scope This stops being an incident and becomes planned work: apply the Stage 1 rules, then decide on Stage 2. No urgency at 70% with 140 GB free, but it should not be closed as though the underlying gap were addressed. Retitling accordingly.

coilyco-ops changed title from ~~kai-server root at 90.75% critical: Forgejo package registry has no retention rules and is growing ~3 GB/day~~ to Forgejo package registry has no retention rules: apply the Stage 1 cleanup spec (disk pressure since relieved)

2026-08-25 05:46:22 +00:00

coilyco-ops referenced this issue

2026-08-25 06:10:26 +00:00

forgejo-data is 97.7 GB against a 21.5 GB request, and #870 counts the orphaned runner PVCs as legitimate data #846

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:06 +00:00

coilyco-ops added the

autonomy

async-consult