Converge k3s-registry role on kai-server: registries.yaml missing the kai-registry.local mirror alias (silent drift) #540

New issue

Closed

opened 2026-07-10 17:44:58 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-10 17:44:58 +00:00

Member

Problem

kai-server's /etc/rancher/k3s/registries.yaml is out of sync with the k3s-registry ansible role, and nothing catches the drift. The role keys the containerd mirror on the logical alias kai-registry.local (ansible/roles/k3s-registry/defaults/main.yml: k3s_registry_authority: kai-registry.local -> http://192.168.0.194:30500), but the live host file only has the pre-migration IP-keyed mirror:

mirrors:
  "192.168.0.194:30500":
    endpoint:
      - "http://192.168.0.194:30500"
configs:
  "192.168.0.194:30500":
    tls:
      insecure_skip_verify: true

There is no kai-registry.local mirror entry. Deploy manifests were migrated to pull via the alias (deploy/forgejo-runner-deploy.yml uses kai-registry.local/forgejo-runner-deploy:latest, deploy#122), but the host was never reconverged, so containerd cannot resolve kai-registry.local and the deploy runner sat in ImagePullBackOff for ~2h after the 2026-07-10 reboot (infrastructure#538). The registry itself is healthy - the pod is Running, the PVC is Bound, and the registry serves forgejo-runner-deploy:latest fine at the IP. Only the alias route was missing.

The reboot did not cause the drift, it only exposed it: before the reboot the node had the image cached, so the missing mirror alias never mattered until a fresh pull was forced.

Do

Confirm the k3s-registry role is actually applied to kai-server: check the host playbook/inventory (ansible/) includes the role for the kai-server host. If it is not wired into the play that converges kai-server, add it, so ward exec ansible-sync local=1 renders the kai-registry.local mirror.
Make the rendered registries.yaml idempotently contain the kai-registry.local mirror alias (and keep the raw 192.168.0.194:30500 entry only if grep -rn "192.168.0.194:30500" deploy/ shows manifests still pull via the IP - otherwise the alias alone is correct).
Add a verification so this drift cannot go silent again: a post-reboot / converge-time smoke check that asserts /etc/rancher/k3s/registries.yaml carries the kai-registry.local mirror pointing at the NodePort endpoint. The Gatus fleet-reachability bundle and the post-reboot hardening discussed in infrastructure#538 are the natural home; a role-level assertion (or molecule test) that the template renders the alias key also works.
Do not restore the raw-IP push path assumptions: the raw IP:port is the Docker build/push endpoint per the role defaults, while containerd pull goes through the kai-registry.local alias. Keep that split intact.

Acceptance

ward exec ansible-sync local=1 on kai-server produces a registries.yaml whose mirrors block includes kai-registry.local -> http://192.168.0.194:30500, idempotently.
A committed check fails loudly if the kai-registry.local mirror is absent from the host registry config, so a future reboot or reconverge cannot silently drop it.
forgejo-runner-deploy-0 pulls kai-registry.local/forgejo-runner-deploy:latest and reaches Running (operator confirms on the live host - the one-time converge was done during the infrastructure#538 recovery).

Context

The one-time live fix (adding the alias to the host file + systemctl restart k3s --no-block + deleting the stuck pod) was applied by the operator during the infrastructure#538 recovery to unblock the coilyco-bridge deploy runner. This issue is the durable follow-up so the config lives in the role and the drift is caught by a check, not by the next fleet outage. Sibling hardening from the same incident: infrastructure#539 (runner-token bootstrap script). Refs: infrastructure#538, deploy#122, deploy/registry.yml, deploy/forgejo-runner-deploy.yml, ansible/roles/k3s-registry/.

## Problem kai-server's `/etc/rancher/k3s/registries.yaml` is out of sync with the `k3s-registry` ansible role, and nothing catches the drift. The role keys the containerd mirror on the logical alias `kai-registry.local` (`ansible/roles/k3s-registry/defaults/main.yml`: `k3s_registry_authority: kai-registry.local` -> `http://192.168.0.194:30500`), but the live host file only has the pre-migration IP-keyed mirror: ```yaml mirrors: "192.168.0.194:30500": endpoint: - "http://192.168.0.194:30500" configs: "192.168.0.194:30500": tls: insecure_skip_verify: true ``` There is **no `kai-registry.local` mirror entry**. Deploy manifests were migrated to pull via the alias (`deploy/forgejo-runner-deploy.yml` uses `kai-registry.local/forgejo-runner-deploy:latest`, deploy#122), but the host was never reconverged, so containerd cannot resolve `kai-registry.local` and the deploy runner sat in `ImagePullBackOff` for ~2h after the 2026-07-10 reboot (infrastructure#538). The registry itself is healthy - the pod is `Running`, the PVC is `Bound`, and the registry serves `forgejo-runner-deploy:latest` fine at the IP. Only the alias route was missing. The reboot did not cause the drift, it only exposed it: before the reboot the node had the image cached, so the missing mirror alias never mattered until a fresh pull was forced. ## Do - Confirm the `k3s-registry` role is actually applied to kai-server: check the host playbook/inventory (`ansible/`) includes the role for the kai-server host. If it is not wired into the play that converges kai-server, add it, so `ward exec ansible-sync local=1` renders the `kai-registry.local` mirror. - Make the rendered `registries.yaml` idempotently contain the `kai-registry.local` mirror alias (and keep the raw `192.168.0.194:30500` entry only if `grep -rn "192.168.0.194:30500" deploy/` shows manifests still pull via the IP - otherwise the alias alone is correct). - Add a **verification** so this drift cannot go silent again: a post-reboot / converge-time smoke check that asserts `/etc/rancher/k3s/registries.yaml` carries the `kai-registry.local` mirror pointing at the NodePort endpoint. The Gatus fleet-reachability bundle and the post-reboot hardening discussed in infrastructure#538 are the natural home; a role-level assertion (or molecule test) that the template renders the alias key also works. - Do not restore the raw-IP push path assumptions: the raw `IP:port` is the Docker build/push endpoint per the role defaults, while containerd pull goes through the `kai-registry.local` alias. Keep that split intact. ## Acceptance - `ward exec ansible-sync local=1` on kai-server produces a `registries.yaml` whose `mirrors` block includes `kai-registry.local -> http://192.168.0.194:30500`, idempotently. - A committed check fails loudly if the `kai-registry.local` mirror is absent from the host registry config, so a future reboot or reconverge cannot silently drop it. - `forgejo-runner-deploy-0` pulls `kai-registry.local/forgejo-runner-deploy:latest` and reaches `Running` (operator confirms on the live host - the one-time converge was done during the infrastructure#538 recovery). ## Context The one-time live fix (adding the alias to the host file + `systemctl restart k3s --no-block` + deleting the stuck pod) was applied by the operator during the infrastructure#538 recovery to unblock the coilyco-bridge deploy runner. This issue is the **durable** follow-up so the config lives in the role and the drift is caught by a check, not by the next fleet outage. Sibling hardening from the same incident: infrastructure#539 (runner-token bootstrap script). Refs: infrastructure#538, deploy#122, `deploy/registry.yml`, `deploy/forgejo-runner-deploy.yml`, `ansible/roles/k3s-registry/`.

coilyco-ops added the

headless

labels

2026-07-10 17:44:58 +00:00

coilyco-ops commented

2026-07-10 17:45:04 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-claude-infrastructure-540 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness claude (reserved 2026-07-10T17:45:03Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#540 · branch issue-540 · harness claude · workflow direct-main
Run: engineer-claude-infrastructure-540 · ward v0.584.0 · dispatched 2026-07-10T17:45:03Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0).

— Claude (she/her), via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-claude-infrastructure-540` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness claude` (reserved 2026-07-10T17:45:03Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#540` · branch `issue-540` · harness `claude` · workflow `direct-main` - **Run:** `engineer-claude-infrastructure-540` · ward `v0.584.0` · dispatched `2026-07-10T17:45:03Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0). </details> </details>  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-10 17:55:47 +00:00

fix(ansible): drift-guard the kai-registry.local mirror alias

coilyco-ops closed this issue

2026-07-10 17:55:47 +00:00

coilyco-ops referenced this issue

2026-07-10 18:12:52 +00:00

[epic] Eliminate the recurring post-bounce fleet full-stop (self-recovery + drift gate + blast-radius) #541