Restore and drill Forgejo backup coverage after kai-server restic drift

coilyco-ops commented

2026-07-23 09:41:38 +00:00

Member

Context

During the 2026-07-23 DiskPressure incident, the documented Forgejo backup gate could not run because restic was absent from kai-server. The live cluster also had no Forgejo backup CronJob or backup PVC. The emergency purge proceeded only after Kai explicitly waived the backup gate.

The documentation currently describes nightly host-level restic coverage to ser8, so the documented and live states have drifted. Related work: #596 and #320.

Work

Infrastructure identifies the owning Ansible or host-management rollout for the restic client, repository configuration, and timer.
The operator safely resolves any package-manager contention before convergence.
Infrastructure restores unattended backup coverage for Forgejo data and its database.
Monitoring alerts on stale or failed snapshots.
The operator takes and verifies the first successful post-repair snapshot.
The operator performs a restore drill into an isolated location and records recovery steps and timing.
docs/restic-backups.md is updated if the actual ownership or topology differs from the documented design.

Done when

The configured timer or job is enabled and succeeding on kai-server.
A current snapshot contains the expected Forgejo data and database material.
An isolated restore drill succeeds.
Backup freshness and failure monitoring are active.
The runbook matches the deployed system.

## Context During the 2026-07-23 DiskPressure incident, the documented Forgejo backup gate could not run because `restic` was absent from kai-server. The live cluster also had no Forgejo backup CronJob or backup PVC. The emergency purge proceeded only after Kai explicitly waived the backup gate. The documentation currently describes nightly host-level restic coverage to ser8, so the documented and live states have drifted. Related work: #596 and #320. ## Work * Infrastructure identifies the owning Ansible or host-management rollout for the restic client, repository configuration, and timer. * The operator safely resolves any package-manager contention before convergence. * Infrastructure restores unattended backup coverage for Forgejo data and its database. * Monitoring alerts on stale or failed snapshots. * The operator takes and verifies the first successful post-repair snapshot. * The operator performs a restore drill into an isolated location and records recovery steps and timing. * `docs/restic-backups.md` is updated if the actual ownership or topology differs from the documented design. ## Done when * The configured timer or job is enabled and succeeding on kai-server. * A current snapshot contains the expected Forgejo data and database material. * An isolated restore drill succeeds. * Backup freshness and failure monitoring are active. * The runbook matches the deployed system.

coilyco-ops added the

labels

2026-07-23 09:41:38 +00:00

coilyco-ops commented

2026-07-25 05:56:26 +00:00

Author

Member

2026-07-24 recurrence evidence

Infrastructure#642 records another operator-authorized Forgejo package purge without a new backup.

The package store had regrown to 110.9 GiB and kai-server root usage reached 89.2%.
Kai accepted the recovery cost and authorized deletion of package versions older than 24 hours.
The supported API and cleanup path reclaimed about 81.7 GiB and returned root usage to 71%.
The operator verified service health and old-object absence, but that verification is not a substitute for a recoverable snapshot.

This is the second emergency package purge documented while the backup and restore-drill gate remains unresolved. The acceptance criteria on this issue remain unchanged and materially urgent.

## 2026-07-24 recurrence evidence Infrastructure#642 records another operator-authorized Forgejo package purge without a new backup. * The package store had regrown to 110.9 GiB and kai-server root usage reached 89.2%. * Kai accepted the recovery cost and authorized deletion of package versions older than 24 hours. * The supported API and cleanup path reclaimed about 81.7 GiB and returned root usage to 71%. * The operator verified service health and old-object absence, but that verification is not a substitute for a recoverable snapshot. This is the second emergency package purge documented while the backup and restore-drill gate remains unresolved. The acceptance criteria on this issue remain unchanged and materially urgent.

coilyco-ops added

and removed

labels

2026-07-29 06:27:52 +00:00

coilyco-ops referenced this issue

2026-08-01 17:59:07 +00:00

Converge the restic backup plane on kai-server and ser8 #729

coilyco-ops referenced this issue

2026-08-01 18:09:30 +00:00

Converge the restic backup plane on kai-server and ser8 #729

coilyco-ops commented

2026-08-01 18:19:41 +00:00

Author

Member

2026-08-01 backup-plane repair checkpoint: commits 60967a2, c3b5221, 4d3aabc, and 8db55d7 landed the Ansible-owned restic topology, check-mode safety, readiness evidence, and the upstream executable-mode correction. The missing /restic/kai-server/repo-password SecureString was created through the guarded file-source path. ser8 now has restic plus the pinned checksummed rest-server 0.14.0, the append-only repository initialized successfully, the prune timer active, and a second apply reported changed=0. kai-server remains untouched because its sudo boundary requires an attended apply. The first large snapshot, freshness monitoring, and restore drill remain open here.

2026-08-01 backup-plane repair checkpoint: commits 60967a2, c3b5221, 4d3aabc, and 8db55d7 landed the Ansible-owned restic topology, check-mode safety, readiness evidence, and the upstream executable-mode correction. The missing /restic/kai-server/repo-password SecureString was created through the guarded file-source path. ser8 now has restic plus the pinned checksummed rest-server 0.14.0, the append-only repository initialized successfully, the prune timer active, and a second apply reported changed=0. kai-server remains untouched because its sudo boundary requires an attended apply. The first large snapshot, freshness monitoring, and restore drill remain open here.

coilyco-ops commented

2026-08-01 18:45:15 +00:00

Author

Member

2026-08-01 checkpoint: ser8 rest-server and prune timer are converged and idempotent. The kai-server secret-resolution failures were traced to an unavailable native aosguard binary on the server-class control host. Commit 02a35b7 now uses the installed AWS CLI bootstrap, pins us-east-1, adds a metadata-only preflight, and passed local plus Forgejo Actions validation. A direct read-only probe from kai-server confirmed both parameter visibility and decryption permission without emitting the value. The kai-server checkout is already at 02a35b7. Remaining live boundary: Ansible convergence and the first attended backup need Kai's sudo password. No Forgejo package versions have been deleted.

2026-08-01 checkpoint: ser8 rest-server and prune timer are converged and idempotent. The kai-server secret-resolution failures were traced to an unavailable native aosguard binary on the server-class control host. Commit 02a35b7 now uses the installed AWS CLI bootstrap, pins us-east-1, adds a metadata-only preflight, and passed local plus Forgejo Actions validation. A direct read-only probe from kai-server confirmed both parameter visibility and decryption permission without emitting the value. The kai-server checkout is already at 02a35b7. Remaining live boundary: Ansible convergence and the first attended backup need Kai's sudo password. No Forgejo package versions have been deleted.

coilyco-ops referenced this issue

2026-08-01 18:54:36 +00:00

Finish kai-server backup rollout and gated Forgejo package retention #733

coilyco-ops added

and removed

labels

2026-08-05 10:26:00 +00:00

coilyco-ops referenced this issue

2026-08-12 00:46:49 +00:00

feat(forgejo): upgrade to 16.0.2 and pin trusted proxies explicitly #790

coilyco-ops commented

2026-08-12 01:27:11 +00:00

Author

Member

Restore drill completed 2026-08-12, database scope

Run entirely on ser8 against the local repository path, so the rest-server and the append-only HTTP path were untouched. No live system was written to.

Method

snapshot   0db62842  (2026-08-11 11:33 UTC, 95.848 GiB, 6 paths)
restore    restic restore 0db62842 --target /var/tmp/restore-drill-20260812 \
             --include /var/lib/restic-backup/forgejo     (nice 10, ionice c2 n7)
verify     docker run --rm postgres:17 pg_restore --list
load       throwaway postgres:17 container, no published ports, createdb + pg_restore
compare    row counts against the live forgejo-db

Timing

Extract the 64 MiB dump from the repository: 1 second
createdb plus pg_restore of the full database: 8 seconds
Structure check: 128 TABLE DATA entries, and 128 tables present in information_schema after load

Fidelity

table	restored (2026-08-11 11:33 UTC)	live at drill time	delta
user	8	8	0
repository	36	36	0
action_run	16534	16685	+151
issue	6654	6692	+38
package_version	813	819	+6

Stable tables match exactly. Growing tables grew by amounts consistent with roughly fourteen hours of ordinary activity. That is the signature of a faithful snapshot rather than a truncated one.

Non-interference

ser8 was at 40% disk with 528 GiB free, load average 1.4, 49 GiB RAM available.
restic restore takes a non-exclusive lock, so it cannot block the nightly backup. The exclusive-lock operation is restic-prune, whose timer next fires 2026-08-16, four days out.
The kai-server nightly backup was roughly ten hours away.
Restore was niced and ioniced to protect runner and SigNoz I/O.
Container and scratch directory removed afterwards. Disk returned to 528 GiB free.

Scope and what is still open

This drill proves the database is recoverable, which is the material risk for the Forgejo 16 upgrade in #620, since that upgrade runs irreversible schema migrations.

It does not cover the PVC payload. The same snapshot carries forgejo-data, forgejo-db-data, registry-data, and code-fleet-watch-state, totalling about 95.8 GiB, and a full-payload restore drill remains unproven. That is the remaining work on this issue.

## Restore drill completed 2026-08-12, database scope Run entirely on ser8 against the local repository path, so the rest-server and the append-only HTTP path were untouched. No live system was written to. ### Method ``` snapshot 0db62842 (2026-08-11 11:33 UTC, 95.848 GiB, 6 paths) restore restic restore 0db62842 --target /var/tmp/restore-drill-20260812 \ --include /var/lib/restic-backup/forgejo (nice 10, ionice c2 n7) verify docker run --rm postgres:17 pg_restore --list load throwaway postgres:17 container, no published ports, createdb + pg_restore compare row counts against the live forgejo-db ``` ### Timing * Extract the 64 MiB dump from the repository: **1 second** * `createdb` plus `pg_restore` of the full database: **8 seconds** * Structure check: 128 `TABLE DATA` entries, and 128 tables present in `information_schema` after load ### Fidelity | table | restored (2026-08-11 11:33 UTC) | live at drill time | delta | | --- | --- | --- | --- | | user | 8 | 8 | 0 | | repository | 36 | 36 | 0 | | action_run | 16534 | 16685 | +151 | | issue | 6654 | 6692 | +38 | | package_version | 813 | 819 | +6 | Stable tables match exactly. Growing tables grew by amounts consistent with roughly fourteen hours of ordinary activity. That is the signature of a faithful snapshot rather than a truncated one. ### Non-interference * ser8 was at 40% disk with 528 GiB free, load average 1.4, 49 GiB RAM available. * `restic restore` takes a non-exclusive lock, so it cannot block the nightly backup. The exclusive-lock operation is `restic-prune`, whose timer next fires 2026-08-16, four days out. * The kai-server nightly backup was roughly ten hours away. * Restore was niced and ioniced to protect runner and SigNoz I/O. * Container and scratch directory removed afterwards. Disk returned to 528 GiB free. ### Scope and what is still open This drill proves the **database** is recoverable, which is the material risk for the Forgejo 16 upgrade in #620, since that upgrade runs irreversible schema migrations. It does **not** cover the PVC payload. The same snapshot carries `forgejo-data`, `forgejo-db-data`, `registry-data`, and `code-fleet-watch-state`, totalling about 95.8 GiB, and a full-payload restore drill remains unproven. That is the remaining work on this issue.

coilyco-ops referenced this issue

2026-08-12 01:27:14 +00:00

Upgrade Forgejo from 15 LTS to 16.0.1 #620

Rows
Columns

Restore and drill Forgejo backup coverage after kai-server restic drift #599

Context

Work

Done when

2026-07-24 recurrence evidence

Restore drill completed 2026-08-12, database scope

Method

Timing

Fidelity

Non-interference

Scope and what is still open