coilyco-flight-deck/infrastructure

Watch

Upgrade Forgejo from 15 LTS to 16.0.1 #620

New issue

Open

opened 2026-07-24 06:19:52 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-24 06:19:52 +00:00

Member

Goal

Infrastructure upgrades the Forgejo deployment from the pinned 15.0.2-rootless image to 16.0.1-rootless. The upgrade unlocks native Actions run cancellation, Authorized Integrations, workflow and job log APIs, and Actions artifact APIs.

Decision boundary

Forgejo 16 is a non-LTS release supported through 2026-10-29. Forgejo 15 LTS is supported through 2027-07-15. Kai accepts the shorter upgrade cadence, or the operator records a decision to remain on 15 and defers the linked adoption work. Infrastructure pins an exact patch release rather than a floating major tag.

Work

The operator reads the complete Forgejo 16 release notes and upgrade guide before applying the migration.
The operator takes and verifies a complete database and data backup before migration.
Infrastructure updates the rootless image pin through the established deployment path.
The operator validates database migrations, PostgreSQL connectivity, ingress, SSH and HTTP Git, OAuth login, package and OCI registries, Actions, runners, and API access.
The operator checks mirror redirect behavior affected by the new SSRF hardening.
The operator confirms that the reverse-proxy authentication and trusted-proxy breaking change does not apply, or configures explicit trusted proxies before rollout.
Infrastructure documents a rollback path using the verified backup and previous image.
The operator performs live post-deploy verification and records the result on this issue.

Done

Infrastructure pins the selected Forgejo 16 patch release.
The operator verifies migrations and core Forgejo functions.
Infrastructure records the rollback procedure.
The operator unblocks the linked Forgejo 16 adoption issues.

## Goal Infrastructure upgrades the Forgejo deployment from the pinned `15.0.2-rootless` image to `16.0.1-rootless`. The upgrade unlocks native Actions run cancellation, Authorized Integrations, workflow and job log APIs, and Actions artifact APIs. ## Decision boundary Forgejo 16 is a non-LTS release supported through 2026-10-29. Forgejo 15 LTS is supported through 2027-07-15. Kai accepts the shorter upgrade cadence, or the operator records a decision to remain on 15 and defers the linked adoption work. Infrastructure pins an exact patch release rather than a floating major tag. ## Work * The operator reads the complete Forgejo 16 release notes and upgrade guide before applying the migration. * The operator takes and verifies a complete database and data backup before migration. * Infrastructure updates the rootless image pin through the established deployment path. * The operator validates database migrations, PostgreSQL connectivity, ingress, SSH and HTTP Git, OAuth login, package and OCI registries, Actions, runners, and API access. * The operator checks mirror redirect behavior affected by the new SSRF hardening. * The operator confirms that the reverse-proxy authentication and trusted-proxy breaking change does not apply, or configures explicit trusted proxies before rollout. * Infrastructure documents a rollback path using the verified backup and previous image. * The operator performs live post-deploy verification and records the result on this issue. ## Done * Infrastructure pins the selected Forgejo 16 patch release. * The operator verifies migrations and core Forgejo functions. * Infrastructure records the rollback procedure. * The operator unblocks the linked Forgejo 16 adoption issues.

coilyco-ops referenced this issue

2026-07-24 06:20:47 +00:00

Adopt Forgejo 16 run cancellation for stale-job reaping #621

coilyco-ops referenced this issue

2026-07-24 06:20:49 +00:00

Adopt Forgejo 16 Authorized Integrations for automation identity #622

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-24 06:20:52 +00:00

Replace the Forgejo Actions web log bridge with the v16 logs API #696

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-24 06:20:54 +00:00

Add guarded Forgejo Actions artifact API operations #697

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-07-29 03:46:28 +00:00

coilyco-ops referenced this issue from a commit

2026-08-12 00:46:23 +00:00

feat(forgejo): upgrade to 16.0.2 and pin trusted proxies explicitly

coilyco-ops referenced this issue

2026-08-12 00:46:49 +00:00

feat(forgejo): upgrade to 16.0.2 and pin trusted proxies explicitly #790

coilyco-ops referenced this issue

2026-08-12 01:27:11 +00:00

Restore and drill Forgejo backup coverage after kai-server restic drift #599

coilyco-ops commented

2026-08-12 01:27:14 +00:00

Author

Member

The rollback path this upgrade depends on is now tested for the database.

A restore drill ran on ser8 today against snapshot 0db62842: the Forgejo pg_dump extracts in 1 second and loads into a clean PostgreSQL 17 in 8 seconds, producing 128 tables whose row counts match the live database exactly on stable tables and differ only by ordinary activity on growing ones. Full method, timing, and fidelity table are recorded on #599.

That closes the specific risk raised on #790: Forgejo 16 runs irreversible schema migrations, so rolling the image back does not roll the database back, and the only real rollback is restoring the snapshot. That restore is no longer untested.

Still true and worth stating plainly: the drill covered the database only. The PVC payload, about 95.8 GiB across forgejo-data, forgejo-db-data, registry-data, and code-fleet-watch-state, has still never been restore-tested.

Recommended sequence before applying #790 remains: take a fresh backup rather than relying on the nightly, then apply, then validate.

The rollback path this upgrade depends on is now tested for the database. A restore drill ran on ser8 today against snapshot `0db62842`: the Forgejo `pg_dump` extracts in 1 second and loads into a clean PostgreSQL 17 in 8 seconds, producing 128 tables whose row counts match the live database exactly on stable tables and differ only by ordinary activity on growing ones. Full method, timing, and fidelity table are recorded on #599. That closes the specific risk raised on #790: Forgejo 16 runs irreversible schema migrations, so rolling the image back does not roll the database back, and the only real rollback is restoring the snapshot. That restore is no longer untested. Still true and worth stating plainly: the drill covered the database only. The PVC payload, about 95.8 GiB across `forgejo-data`, `forgejo-db-data`, `registry-data`, and `code-fleet-watch-state`, has still never been restore-tested. Recommended sequence before applying #790 remains: take a fresh backup rather than relying on the nightly, then apply, then validate.

coilyco-ops commented

2026-08-12 04:50:09 +00:00

Author

Member

Applied 2026-08-12, verified

16.0.2-rootless is live. Applied 04:32:01Z, pod Ready 04:33:08Z, so 43
seconds of downtime on the Recreate strategy.

Migrations

Clean. v16b_add-login-source-id-to-forgejo-auth-token,
v16b_add_comment_line_count, v16b_authorized_integration_name_description,
v16c_action_run_priority, v16c_add_team_invite_invited_id,
v16c_authorized_integration_ui, v16c_cleanup_package_blob_indexes,
v16d_action_run_warnings, and v16e_add-granular-watch all applied, followed
by a dangling sha256:* image cleanup. No error, fatal, or panic in the
startup log.

Verified

Version endpoint reports 16.0.2+gitea-1.22.0.
PostgreSQL connected, authenticated issue reads return data.
HTTP git clone succeeds.
Ingress serving through Traefik.
Packages and OCI registry API return 200.
Runner pods all healthy, 21 running in the namespace, and CI runs have
completed successfully since the upgrade.
The GitHub mirror survived the upgrade and reports synced=31 with no
failures.

OAuth login was not verified. Kai confirms it is no longer used, so it is not a
gap.

The reason for the upgrade works

Both endpoints that 15 had no route for now answer:

/repos/{owner}/{repo}/actions/runs/{run_id}/jobs   -> jobs with id, name, status
/repos/{owner}/{repo}/actions/jobs/{job_id}/logs   -> HTTP 200, 23935 bytes

CI failures are diagnosable from the API. That resolves issue 786 and unblocks
issues 621 and 622.

Trusted proxy

FORGEJO__security__REVERSE_PROXY_TRUSTED_PROXIES was set to the pod CIDR in
the same change, so the removal of the image default
REVERSE_PROXY_TRUSTED_PROXIES = * in Forgejo 16 did not silently drop
Traefik's forwarded headers.

Backup deviation, stated plainly

The planned pre-upgrade restic run did not happen. The agent has no
passwordless sudo on kai-server, so systemctl start restic-backup.service was
not available.

Substituted: a fresh pg_dump -Fc taken through kubectl exec, 30 seconds,
65 MiB, copied off-host to ser8 at
/var/tmp/forgejo-preupgrade-20260812.pgdump and verified to parse to 128
TABLE DATA entries, matching the restore drill recorded on issue 599.

That is the material a schema-migration rollback needs, since migrations touch
the database and not the PVC payload, which last night's snapshot still covers.
It is a narrower safety net than planned and the deviation is recorded rather
than absorbed.

Residual, non-blocking

Five peripheral errors in the first thirty minutes, none migration or
data-related:

one GetDiff broken pipe on ward.git, a render aborted by a closed
connection
three Failed to create commit status for job 27537: head commit is missing in event payload
one Libravatar lookup failing on a bracketed bot email address

The commit-status one is the only one with functional effect, and only for that
job. Worth a glance if it recurs on ordinary push events.

Follow-up found during verification

aosguard ops forgejo action-job logs fails with invalid character '-' after top-level value. It JSON-parses a plaintext response. Forgejo serves the
endpoint correctly, so this is a client-side fix in the guardfile or its
response handling.

## Applied 2026-08-12, verified `16.0.2-rootless` is live. Applied 04:32:01Z, pod Ready 04:33:08Z, so **43 seconds** of downtime on the `Recreate` strategy. ### Migrations Clean. `v16b_add-login-source-id-to-forgejo-auth-token`, `v16b_add_comment_line_count`, `v16b_authorized_integration_name_description`, `v16c_action_run_priority`, `v16c_add_team_invite_invited_id`, `v16c_authorized_integration_ui`, `v16c_cleanup_package_blob_indexes`, `v16d_action_run_warnings`, and `v16e_add-granular-watch` all applied, followed by a dangling `sha256:*` image cleanup. No error, fatal, or panic in the startup log. ### Verified * Version endpoint reports `16.0.2+gitea-1.22.0`. * PostgreSQL connected, authenticated issue reads return data. * HTTP git clone succeeds. * Ingress serving through Traefik. * Packages and OCI registry API return 200. * Runner pods all healthy, 21 running in the namespace, and CI runs have completed successfully since the upgrade. * The GitHub mirror survived the upgrade and reports `synced=31` with no failures. OAuth login was not verified. Kai confirms it is no longer used, so it is not a gap. ### The reason for the upgrade works Both endpoints that 15 had no route for now answer: ``` /repos/{owner}/{repo}/actions/runs/{run_id}/jobs -> jobs with id, name, status /repos/{owner}/{repo}/actions/jobs/{job_id}/logs -> HTTP 200, 23935 bytes ``` CI failures are diagnosable from the API. That resolves issue 786 and unblocks issues 621 and 622. ### Trusted proxy `FORGEJO__security__REVERSE_PROXY_TRUSTED_PROXIES` was set to the pod CIDR in the same change, so the removal of the image default `REVERSE_PROXY_TRUSTED_PROXIES = *` in Forgejo 16 did not silently drop Traefik's forwarded headers. ### Backup deviation, stated plainly The planned pre-upgrade restic run **did not happen**. The agent has no passwordless sudo on kai-server, so `systemctl start restic-backup.service` was not available. Substituted: a fresh `pg_dump -Fc` taken through `kubectl exec`, 30 seconds, 65 MiB, copied off-host to ser8 at `/var/tmp/forgejo-preupgrade-20260812.pgdump` and verified to parse to 128 `TABLE DATA` entries, matching the restore drill recorded on issue 599. That is the material a schema-migration rollback needs, since migrations touch the database and not the PVC payload, which last night's snapshot still covers. It is a narrower safety net than planned and the deviation is recorded rather than absorbed. ### Residual, non-blocking Five peripheral errors in the first thirty minutes, none migration or data-related: * one `GetDiff` broken pipe on `ward.git`, a render aborted by a closed connection * three `Failed to create commit status for job 27537: head commit is missing in event payload` * one Libravatar lookup failing on a bracketed bot email address The commit-status one is the only one with functional effect, and only for that job. Worth a glance if it recurs on ordinary push events. ### Follow-up found during verification `aosguard ops forgejo action-job logs` fails with `invalid character '-' after top-level value`. It JSON-parses a plaintext response. Forgejo serves the endpoint correctly, so this is a client-side fix in the guardfile or its response handling.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#620

No description provided.

Rows
Columns