Mirror the flight-deck bootstrap image set to ghcr.io #794

New issue

Open

opened 2026-08-12 04:11:55 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-12 04:11:55 +00:00

Owner

Stand up a ghcr.io copy of the images a rebuilt cluster needs, so losing
kai-server does not also lose every image required to bring one back.

This issue is written to be picked up cold. Everything below marked verified
was measured on 2026-08-12 and does not need re-deriving. Everything under
settle first is an open decision.

Why

Every workload image lives in exactly one place, the Forgejo package registry on
kai-server. The standalone registry in the registry namespace on NodePort
30500 is a different failure domain from the Forgejo application, so it
survives a bad Forgejo upgrade, but it does not survive losing the machine,
because it is on the machine.

Restic snapshots to ser8 cover data. They do not make images pullable into a
cluster that does not exist yet.

Scope, decided

Mirror these, and only these.

Six runtime images at the exact SHAs the fleet runs, nine tags in total because
two images run at more than one SHA:

agent-proxy:20d53e4c5b41656a00b28d9319ebdf192a0cca55
bluesky-mcp:9a57ad43376eb65831be432f813d2c3b19cd0bb4
lunch-money-mcp:2c49705ce882adc767e7bc700a1bb0eeeacd3173
node-stats-mcp:9e486430f44f35edc39277e6b3c9d956c5abf415
node-stats-mcp:f15a3fc5c57887bf7884712ae8853dcbab6c2718
reddit-mcp:e5e30bc3da3dad2869214f72d778e58cddda681e
ward-mcp:82e685c87a38c530a3f183f056cb0152bdbf4315
ward-mcp:97a100274a72a281b11440af89a9f7638c2754cb
ward-mcp:fa601e9e11165349c2a199c39702236fd9fcf003

Plus agentic-os at its :release alias only. That is the CI and dev-base
image, so without it a recovered cluster can run workloads but cannot build new
ones.

That list drifts as pins move. Regenerate it rather than trusting the snapshot
above:

{ ssh kai-server 'k3s kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{.image}{\"\n\"}{end}{end}"';
  ssh coilysiren@ser8 'kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{.image}{\"\n\"}{end}{end}"'; } \
| grep "forgejo.coilysiren.me/coilyco-flight-deck/" | sort -u

Explicitly out of scope

Image history. There are 432 container versions in the org, 344 of them
agentic-os, which alone is about 116 GiB per issue 733. Mirroring history
costs nearly all the bytes and buys nearly nothing for recovery.
Repointing manifests at ghcr. This is a recovery copy, not a second
serving path. Deploy keeps pulling from Forgejo. Changing image references is
a separate decision with its own blast radius.
The imagePullPolicy: Always problem. Tracked in
coilyco-bridge/deploy issue 372. A second registry is not its fix, and
fixing it here would confuse two unrelated failures.
Other orgs. coilyco-gaming and coilyco-bridge images are not in this
pass.

Verified, do not re-derive

Namespace and credential. The coilyco-ops GitHub App holds
packages: write on all four installations, including the
coilyco-flight-deck organisation with repository_selection: all. So
ghcr.io/coilyco-flight-deck/<image> is reachable with the credential that
already exists. No new account, no new secret, no token-refresh timer.
App identity in SSM. /github/coilyco-ops-app-id and
/github/coilyco-ops-private-key-base64. The mirror controller in
scripts/forgejo_github_mirror_sync.py already mints short-lived installation
tokens from these and is the working reference for how to do it.
ghcr is already in use. kai-server runs
ghcr.io/coilysiren/lunch-money-k8s:latest, and the default/docker-registry
Secret's auth entry is ghcr.io.
Forgejo packages API pages at 50. Passing a larger limit is silently
capped, so enumerate with &page=N until a short page returns.
Source registry auth. /forgejo/coilyco-ops/registry-read-token exists
for pulling from the Forgejo side.

Settle first

Where it runs. ser8 is the natural home: it already hosts the
Forgejo-to-GitHub mirror as a user systemd timer, it has passwordless sudo,
and it is not the machine this protects against losing. kai-server is the
wrong host for the same reason the current registry is.
Trigger. A timer like the GitHub mirror, or a push-time hook when a
deploy pin changes. A timer is simpler and tolerates missed runs. A hook is
timelier but couples release to a second registry being up.
Copy mechanism. skopeo copy moves manifests and layers between
registries without a local daemon and preserves digests. crane copy is
equivalent. Pulling and re-pushing through Docker is the option to avoid,
because it rewrites manifests and breaks digest equality.
Retention. A cleanup policy so this does not grow the way the Forgejo
package store did, which reached 120.3 GiB and forced two emergency purges,
recorded in issues 733 and 599.

Constraints

Immutable tags only. Copy the full source SHA tag. Do not create :latest
or any moving alias except agentic-os:release, which is already a moving
alias upstream and is mirrored as-is. The deploy contract says deployment uses
the full upstream source SHA and does not substitute latest, and a recovery
mirror gains nothing from breaking that. During a rebuild, "which build is
:latest" is the worst time to be unsure.
Public on purpose. ghcr packages default to private, and private packages
on the free tier carry storage and bandwidth limits. These images are already
published to a public Forgejo registry, so publish them public deliberately
rather than discovering the limit later.
Never delete on the mirror. Follow the GitHub mirror's posture: create and
update, never prune, never force. A stale extra tag is harmless. A deleted one
during recovery is not.
Fail loudly. A failed copy should make the unit non-zero rather than
logging and continuing, so a silently empty mirror is not discovered during a
recovery.

Done when

The nine runtime tags plus agentic-os:release exist under
ghcr.io/coilyco-flight-deck/, with digests matching the Forgejo source.
The copy runs unattended and reports failure.
A documented recovery path exists: given a bare cluster, which references to
change and in what order to pull from the mirror instead.
docs/ carries the runbook, and the trifecta is updated if a feature entry is
warranted.

coilyco-bridge/deploy issue 372, the pull-policy problem this is not.
Issues 733 and 599 for the package-growth history that motivates retention.
scripts/forgejo_github_mirror_sync.py as the working reference for App-token
minting, per-target scoping, and never-force mirroring.

One loose end found while scoping

deploy/lunch-money/values.yaml in this repository pins
docker.io/coilysiren/lunch-money-k8s. That Docker Hub repository returns 404,
while the running pod uses a ghcr.io copy. The tracked reference points at a
registry path that does not exist, so a reconcile from that file would fail to
pull. Worth fixing independently of this work.

Stand up a ghcr.io copy of the images a rebuilt cluster needs, so losing kai-server does not also lose every image required to bring one back. This issue is written to be picked up cold. Everything below marked **verified** was measured on 2026-08-12 and does not need re-deriving. Everything under **settle first** is an open decision. ## Why Every workload image lives in exactly one place, the Forgejo package registry on kai-server. The standalone registry in the `registry` namespace on NodePort 30500 is a different failure domain from the Forgejo *application*, so it survives a bad Forgejo upgrade, but it does not survive losing the machine, because it is on the machine. Restic snapshots to ser8 cover data. They do not make images pullable into a cluster that does not exist yet. ## Scope, decided **Mirror these, and only these.** Six runtime images at the exact SHAs the fleet runs, nine tags in total because two images run at more than one SHA: ``` agent-proxy:20d53e4c5b41656a00b28d9319ebdf192a0cca55 bluesky-mcp:9a57ad43376eb65831be432f813d2c3b19cd0bb4 lunch-money-mcp:2c49705ce882adc767e7bc700a1bb0eeeacd3173 node-stats-mcp:9e486430f44f35edc39277e6b3c9d956c5abf415 node-stats-mcp:f15a3fc5c57887bf7884712ae8853dcbab6c2718 reddit-mcp:e5e30bc3da3dad2869214f72d778e58cddda681e ward-mcp:82e685c87a38c530a3f183f056cb0152bdbf4315 ward-mcp:97a100274a72a281b11440af89a9f7638c2754cb ward-mcp:fa601e9e11165349c2a199c39702236fd9fcf003 ``` Plus `agentic-os` at its `:release` alias only. That is the CI and dev-base image, so without it a recovered cluster can run workloads but cannot build new ones. That list drifts as pins move. Regenerate it rather than trusting the snapshot above: ```sh { ssh kai-server 'k3s kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{.image}{\"\n\"}{end}{end}"'; ssh coilysiren@ser8 'kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{.image}{\"\n\"}{end}{end}"'; } \ | grep "forgejo.coilysiren.me/coilyco-flight-deck/" | sort -u ``` ## Explicitly out of scope * **Image history.** There are 432 container versions in the org, 344 of them `agentic-os`, which alone is about 116 GiB per issue 733. Mirroring history costs nearly all the bytes and buys nearly nothing for recovery. * **Repointing manifests at ghcr.** This is a recovery copy, not a second serving path. Deploy keeps pulling from Forgejo. Changing image references is a separate decision with its own blast radius. * **The `imagePullPolicy: Always` problem.** Tracked in `coilyco-bridge/deploy` issue 372. A second registry is not its fix, and fixing it here would confuse two unrelated failures. * **Other orgs.** `coilyco-gaming` and `coilyco-bridge` images are not in this pass. ## Verified, do not re-derive * **Namespace and credential.** The `coilyco-ops` GitHub App holds `packages: write` on all four installations, including the `coilyco-flight-deck` **organisation** with `repository_selection: all`. So `ghcr.io/coilyco-flight-deck/<image>` is reachable with the credential that already exists. No new account, no new secret, no token-refresh timer. * **App identity in SSM.** `/github/coilyco-ops-app-id` and `/github/coilyco-ops-private-key-base64`. The mirror controller in `scripts/forgejo_github_mirror_sync.py` already mints short-lived installation tokens from these and is the working reference for how to do it. * **ghcr is already in use.** kai-server runs `ghcr.io/coilysiren/lunch-money-k8s:latest`, and the `default/docker-registry` Secret's auth entry is `ghcr.io`. * **Forgejo packages API pages at 50.** Passing a larger `limit` is silently capped, so enumerate with `&page=N` until a short page returns. * **Source registry auth.** `/forgejo/coilyco-ops/registry-read-token` exists for pulling from the Forgejo side. ## Settle first * **Where it runs.** ser8 is the natural home: it already hosts the Forgejo-to-GitHub mirror as a user systemd timer, it has passwordless sudo, and it is not the machine this protects against losing. kai-server is the wrong host for the same reason the current registry is. * **Trigger.** A timer like the GitHub mirror, or a push-time hook when a deploy pin changes. A timer is simpler and tolerates missed runs. A hook is timelier but couples release to a second registry being up. * **Copy mechanism.** `skopeo copy` moves manifests and layers between registries without a local daemon and preserves digests. `crane copy` is equivalent. Pulling and re-pushing through Docker is the option to avoid, because it rewrites manifests and breaks digest equality. * **Retention.** A cleanup policy so this does not grow the way the Forgejo package store did, which reached 120.3 GiB and forced two emergency purges, recorded in issues 733 and 599. ## Constraints * **Immutable tags only.** Copy the full source SHA tag. Do not create `:latest` or any moving alias except `agentic-os:release`, which is already a moving alias upstream and is mirrored as-is. The deploy contract says deployment uses the full upstream source SHA and does not substitute `latest`, and a recovery mirror gains nothing from breaking that. During a rebuild, "which build is `:latest`" is the worst time to be unsure. * **Public on purpose.** ghcr packages default to private, and private packages on the free tier carry storage and bandwidth limits. These images are already published to a public Forgejo registry, so publish them public deliberately rather than discovering the limit later. * **Never delete on the mirror.** Follow the GitHub mirror's posture: create and update, never prune, never force. A stale extra tag is harmless. A deleted one during recovery is not. * **Fail loudly.** A failed copy should make the unit non-zero rather than logging and continuing, so a silently empty mirror is not discovered during a recovery. ## Done when * The nine runtime tags plus `agentic-os:release` exist under `ghcr.io/coilyco-flight-deck/`, with digests matching the Forgejo source. * The copy runs unattended and reports failure. * A documented recovery path exists: given a bare cluster, which references to change and in what order to pull from the mirror instead. * `docs/` carries the runbook, and the trifecta is updated if a feature entry is warranted. ## Related * `coilyco-bridge/deploy` issue 372, the pull-policy problem this is not. * Issues 733 and 599 for the package-growth history that motivates retention. * `scripts/forgejo_github_mirror_sync.py` as the working reference for App-token minting, per-target scoping, and never-force mirroring. ## One loose end found while scoping `deploy/lunch-money/values.yaml` in this repository pins `docker.io/coilysiren/lunch-money-k8s`. That Docker Hub repository returns 404, while the running pod uses a `ghcr.io` copy. The tracked reference points at a registry path that does not exist, so a reconcile from that file would fail to pull. Worth fixing independently of this work.

coilyco-ops changed title from ~~Evaluate an ECR disaster-recovery mirror for fleet images~~ to Mirror fleet images to ghcr.io for cold-rebuild recovery

2026-08-12 05:13:50 +00:00

coilyco-ops changed title from ~~Mirror fleet images to ghcr.io for cold-rebuild recovery~~ to Mirror the flight-deck bootstrap image set to ghcr.io

2026-08-12 05:23:31 +00:00

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:13 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:31:02 +00:00

coilyco-ops added this to the CD and rollout milestone