Forgejo attachments are 39.9 GiB and it is release binaries: 6,900+ assets across 820 releases, no retention #905

New issue

Open

opened 2026-08-23 04:46:59 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-23 04:46:59 +00:00

Owner

Stage 3 of coilyco-flight-deck/infrastructure#903. Measured 2026-08-23 ~03:40Z, read-only throughout. Nothing changed.

After Stage 1 cut the package registry from 93.3 to 37.5 GiB, data/attachments at 39.9 GiB became the single largest consumer on kai-server and had not moved a byte all night. This establishes what is in it.

It is release assets, and the arithmetic closes

/var/lib/gitea/data/attachments   39.9 GiB   (filesystem)
release assets, all orgs          39.6 GiB   (Forgejo API)

99.2% accounted for. Forgejo stores release attachments under data/attachments, so these are the same bytes counted two ways. avatars is 1.9 MiB and repo-avatars 64 KiB, so nothing else in data/ is material.

Totals: 820 releases carrying 6,933 assets, concentrated in four repositories.

     GiB   releases  repo
   20.16        504  coilyco-flight-deck/agentic-os
   11.85        957  coilyco-flight-deck/ward
    4.22        159  coilyco-flight-deck/umbra
    3.63        161  coilyco-flight-deck/agent-compose

Everything else across all four orgs is a rounding error.

Why it grows

These are cross-platform Go binaries, republished on every release, and nothing ever removes an old one.

agentic-os, 3,102 assets over 504 releases, ~41 MiB per release:

  196  aos-darwin-arm64          196  aos-linux-amd64
  196  aos-linux-arm64           196  aos-windows-amd64.exe
  196  aos.json                  196  aos.rb
  196  SHA256SUMS                111  aosguard-darwin-arm64

ward, 2,384 assets over 957 releases, ~13 MiB per release, the same shape across five platform triples plus per-file .sha256 and a SHA256SUMS.

Largest single assets are the Windows builds at 19.1 MiB (aosguard-windows-amd64.exe) and 18.4 MiB (ward-windows-amd64.exe), each retained once per release.

The release trains are automated. Per agentic-os/AGENTS.md the standalone AOS CLI bumps its minor version automatically whenever a push to main changes a shipped binary input. Oldest release here is 2026-06-15 for agentic-os and 2026-06-12 for ward, so 39.6 GiB accumulated in roughly ten weeks, and it is monotonic.

This is structurally the same failure Stage 1 fixed for packages: an automated publisher plus no retention policy. Different storage domain, same shape.

What this is not

Not issue or comment attachments. The asset names are entirely build artifacts. No human upload appears in the top shapes.
Not Actions artifacts. Those live in actions_artifacts, which is 296 KiB, and actions_log is a separate 329.8 MiB.
Not the database. Forgejo PostgreSQL is 608 MB and healthy.
Not fixable by package cleanup rules. Those govern the package registry only. Release assets are a different subsystem with a different retention story, which is why Stage 1 left this figure untouched.

What needs deciding

Unlike packages, Forgejo has no built-in retention for release assets. There is no equivalent of a cleanup rule, so this cannot be closed by turning something on. Three options, and the choice is a genuine one:

Delete old release assets, keep the releases. DELETE /repos/{owner}/{repo}/releases/{id}/assets/{attachment_id} exists in the API, so this is scriptable. Release notes, tags and history survive; only the binaries go. Recovers most of the 39.6 GiB.
Stop attaching binaries to routine releases. Attach only on tagged milestones and let Homebrew and Scoop consume those. Changes what downstream consumers can resolve, so it needs checking against homebrew-tap and scoop-bucket before anyone commits to it.
Reduce release frequency. 957 ward releases in about ten weeks is roughly 14 per day. That is the automation working as designed, but it is worth asking whether every binary-input change warrants a published artifact set.

The consumer question gates all three. homebrew-tap and scoop-bucket resolve binaries from these releases. Before deleting any asset, someone has to establish how far back those formulas and manifests can still point, and whether any pinned version outside the retained window is still referenced. I have not checked that and it is the first task here.

Suggested first step

Establish the retention floor from the consumers, then script a keep-N sweep over the four repositories using the release-asset delete endpoint. A keep_count in the same spirit as the package rules would put this at a couple of GiB.

Worth noting for whoever picks this up: the release-asset API is a real API, unlike package cleanup rules which are web-UI only (infrastructure#903). So this one can be scripted, tracked in git, and run on a schedule, which is the better end state anyway.

Current disk position

Not urgent. After Stage 1 the root filesystem is 78.2%, status ok, with 109 GiB available. This is the largest remaining consumer and an uncapped growth curve, not an active incident.

coilyco-flight-deck/infrastructure#903 - the parent disk analysis, Stage 3
coilyco-flight-deck/agentic-os#1060 - dev-base release blocked, should be retryable now that Stage 1 freed 56 GiB

Stage 3 of `coilyco-flight-deck/infrastructure#903`. Measured 2026-08-23 ~03:40Z, read-only throughout. **Nothing changed.** After Stage 1 cut the package registry from 93.3 to 37.5 GiB, `data/attachments` at **39.9 GiB** became the single largest consumer on `kai-server` and had not moved a byte all night. This establishes what is in it. ## It is release assets, and the arithmetic closes ``` /var/lib/gitea/data/attachments 39.9 GiB (filesystem) release assets, all orgs 39.6 GiB (Forgejo API) ``` **99.2% accounted for.** Forgejo stores release attachments under `data/attachments`, so these are the same bytes counted two ways. `avatars` is 1.9 MiB and `repo-avatars` 64 KiB, so nothing else in `data/` is material. Totals: **820 releases carrying 6,933 assets**, concentrated in four repositories. ``` GiB releases repo 20.16 504 coilyco-flight-deck/agentic-os 11.85 957 coilyco-flight-deck/ward 4.22 159 coilyco-flight-deck/umbra 3.63 161 coilyco-flight-deck/agent-compose ``` Everything else across all four orgs is a rounding error. ## Why it grows These are cross-platform Go binaries, republished on every release, and nothing ever removes an old one. `agentic-os`, 3,102 assets over 504 releases, ~41 MiB per release: ``` 196 aos-darwin-arm64 196 aos-linux-amd64 196 aos-linux-arm64 196 aos-windows-amd64.exe 196 aos.json 196 aos.rb 196 SHA256SUMS 111 aosguard-darwin-arm64 ``` `ward`, 2,384 assets over 957 releases, ~13 MiB per release, the same shape across five platform triples plus per-file `.sha256` and a `SHA256SUMS`. Largest single assets are the Windows builds at **19.1 MiB** (`aosguard-windows-amd64.exe`) and **18.4 MiB** (`ward-windows-amd64.exe`), each retained once per release. **The release trains are automated.** Per `agentic-os/AGENTS.md` the standalone AOS CLI bumps its minor version automatically whenever a push to main changes a shipped binary input. Oldest release here is 2026-06-15 for agentic-os and 2026-06-12 for ward, so **39.6 GiB accumulated in roughly ten weeks**, and it is monotonic. This is structurally the same failure Stage 1 fixed for packages: an automated publisher plus no retention policy. Different storage domain, same shape. ## What this is not * **Not issue or comment attachments.** The asset names are entirely build artifacts. No human upload appears in the top shapes. * **Not Actions artifacts.** Those live in `actions_artifacts`, which is 296 KiB, and `actions_log` is a separate 329.8 MiB. * **Not the database.** Forgejo PostgreSQL is 608 MB and healthy. * **Not fixable by package cleanup rules.** Those govern the package registry only. Release assets are a different subsystem with a different retention story, which is why Stage 1 left this figure untouched. ## What needs deciding Unlike packages, **Forgejo has no built-in retention for release assets.** There is no equivalent of a cleanup rule, so this cannot be closed by turning something on. Three options, and the choice is a genuine one: 1. **Delete old release assets, keep the releases.** `DELETE /repos/{owner}/{repo}/releases/{id}/assets/{attachment_id}` exists in the API, so this is scriptable. Release notes, tags and history survive; only the binaries go. Recovers most of the 39.6 GiB. 2. **Stop attaching binaries to routine releases.** Attach only on tagged milestones and let Homebrew and Scoop consume those. Changes what downstream consumers can resolve, so it needs checking against `homebrew-tap` and `scoop-bucket` before anyone commits to it. 3. **Reduce release frequency.** 957 ward releases in about ten weeks is roughly 14 per day. That is the automation working as designed, but it is worth asking whether every binary-input change warrants a published artifact set. **The consumer question gates all three.** `homebrew-tap` and `scoop-bucket` resolve binaries from these releases. Before deleting any asset, someone has to establish how far back those formulas and manifests can still point, and whether any pinned version outside the retained window is still referenced. I have not checked that and it is the first task here. ## Suggested first step Establish the retention floor from the consumers, then script a keep-N sweep over the four repositories using the release-asset delete endpoint. A `keep_count` in the same spirit as the package rules would put this at a couple of GiB. Worth noting for whoever picks this up: **the release-asset API is a real API**, unlike package cleanup rules which are web-UI only (`infrastructure#903`). So this one can be scripted, tracked in git, and run on a schedule, which is the better end state anyway. ## Current disk position Not urgent. After Stage 1 the root filesystem is **78.2%, status `ok`**, with 109 GiB available. This is the largest remaining consumer and an uncapped growth curve, not an active incident. ## Related * `coilyco-flight-deck/infrastructure#903` - the parent disk analysis, Stage 3 * `coilyco-flight-deck/agentic-os#1060` - dev-base release blocked, should be retryable now that Stage 1 freed 56 GiB

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:06 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:55 +00:00

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00