Finish kai-server backup rollout and gated Forgejo package retention #733

New issue

Open

opened 2026-08-01 18:54:36 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-01 18:54:36 +00:00

Member

Current state

kai-server root is 88% used with 56 GiB free as of 2026-08-01.
Forgejo uses about 141.6 GiB. Packages account for 120.3 GiB and attachments for 18.6 GiB.
coilyco-flight-deck/container/agentic-os accounts for about 116 GiB across 749 external versions.
The corrected backup role is on main at 02a35b7. Local validation and Forgejo Actions passed.
ser8's append-only rest-server repository and prune timer are converged and idempotent.
kai-server is already updated to 02a35b7. Direct read-only checks confirmed SSM visibility and decryption permission.
No package versions were deleted during this run.

Human boundary

Kai's sudo password is required for the kai-server Ansible apply. Do not ask Kai to repeat the completed diagnostics.

Remaining work

Run the tagged restic-backup Ansible apply on kai-server with attended sudo authentication.
Launch and monitor the first backup, then verify a fresh snapshot contains the Forgejo database dump and expected PVC data.
Start the already-enabled backup timer and verify service health and snapshot freshness.
Complete the isolated restore drill and record recovery timing under #599.
Re-run the agentic-os package retention preview. After a verified fresh backup and fresh explicit destructive approval, remove only the versions outside the newest 50 while retaining moving aliases.
Configure durable Forgejo cleanup policy so package growth does not recur.
Re-measure disk usage and verify Forgejo, PostgreSQL, runners, and package pulls remain healthy.

Safety gates

Do not delete Forgejo packages before the fresh backup is verified.
Do not treat the prior preview as destructive approval.
Preserve latest, release, buildcache, and lang-{node,go,dotnet,rust,python}-{release,buildcache} aliases.

Related: #599.

## Current state * kai-server root is 88% used with 56 GiB free as of 2026-08-01. * Forgejo uses about 141.6 GiB. Packages account for 120.3 GiB and attachments for 18.6 GiB. * coilyco-flight-deck/container/agentic-os accounts for about 116 GiB across 749 external versions. * The corrected backup role is on main at 02a35b7. Local validation and Forgejo Actions passed. * ser8's append-only rest-server repository and prune timer are converged and idempotent. * kai-server is already updated to 02a35b7. Direct read-only checks confirmed SSM visibility and decryption permission. * No package versions were deleted during this run. ## Human boundary Kai's sudo password is required for the kai-server Ansible apply. Do not ask Kai to repeat the completed diagnostics. ## Remaining work 1. Run the tagged restic-backup Ansible apply on kai-server with attended sudo authentication. 2. Launch and monitor the first backup, then verify a fresh snapshot contains the Forgejo database dump and expected PVC data. 3. Start the already-enabled backup timer and verify service health and snapshot freshness. 4. Complete the isolated restore drill and record recovery timing under #599. 5. Re-run the agentic-os package retention preview. After a verified fresh backup and fresh explicit destructive approval, remove only the versions outside the newest 50 while retaining moving aliases. 6. Configure durable Forgejo cleanup policy so package growth does not recur. 7. Re-measure disk usage and verify Forgejo, PostgreSQL, runners, and package pulls remain healthy. ## Safety gates * Do not delete Forgejo packages before the fresh backup is verified. * Do not treat the prior preview as destructive approval. * Preserve latest, release, buildcache, and lang-{node,go,dotnet,rust,python}-{release,buildcache} aliases. Related: #599.

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-08-01 18:54:36 +00:00

coilyco-ops added

priority

and removed

priority

labels

2026-08-05 10:25:59 +00:00