coilyco-flight-deck/infrastructure

Fork 0

ansible: manage the Caddy apt repo + signing key on kai-server so it self-heals #455

New issue

Closed

opened 2026-07-03 18:19:37 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-03 18:19:37 +00:00

Member

Why

The docker-ce converge (infrastructure#450) failed at apt-get update - not on the docker repo (that fetched fine) but on an unrelated stale source:

Err: https://dl.cloudsmith.io/public/caddy/stable/... EXPKEYSIG ABA1F9B8875A6661 Caddy Web Server
E: The repository '...caddy...' is not signed.

Caddy is a real running service on kai-server (v2.10.2, active+enabled). Its signing key ABA1F9B8875A6661 expired; upstream rotated to 155B6D79CA56EA34 (no expiry). Because the key was hand-installed and unmanaged, it expired silently and now any apt operation on the box fails - it just happened to surface via the docker converge.

Fix (object-level, done out of band)

Refresh /usr/share/keyrings/caddy-stable-archive-keyring.gpg from https://dl.cloudsmith.io/public/caddy/stable/gpg.key. Verified: refreshed key validates the repo (Get: InRelease + Packages, no errors).

Meta (this issue)

Add an ansible caddy role (or fold into an existing one) that manages the Caddy apt repo + keyring the same way the docker role manages docker's - get_url the current key to a keyring path, deb822_repository the source. Then a rotated/expired upstream key self-heals on the next converge instead of silently breaking the whole fleet's apt. Same authoring-vs-rollout pattern as the docker role.

Gate to kai-server (where Caddy runs) like the docker role. Origin: surfaced during the docker-ce converge, infrastructure#450.

## Why The docker-ce converge (infrastructure#450) failed at `apt-get update` - not on the docker repo (that fetched fine) but on an unrelated stale source: ``` Err: https://dl.cloudsmith.io/public/caddy/stable/... EXPKEYSIG ABA1F9B8875A6661 Caddy Web Server E: The repository '...caddy...' is not signed. ``` Caddy is a real running service on kai-server (v2.10.2, active+enabled). Its signing key `ABA1F9B8875A6661` **expired**; upstream rotated to `155B6D79CA56EA34` (no expiry). Because the key was hand-installed and unmanaged, it expired silently and now **any** apt operation on the box fails - it just happened to surface via the docker converge. ## Fix (object-level, done out of band) Refresh `/usr/share/keyrings/caddy-stable-archive-keyring.gpg` from `https://dl.cloudsmith.io/public/caddy/stable/gpg.key`. Verified: refreshed key validates the repo (`Get: InRelease` + `Packages`, no errors). ## Meta (this issue) Add an ansible `caddy` role (or fold into an existing one) that manages the Caddy apt repo + keyring the same way the `docker` role manages docker's - `get_url` the current key to a keyring path, `deb822_repository` the source. Then a rotated/expired upstream key self-heals on the next converge instead of silently breaking the whole fleet's apt. Same authoring-vs-rollout pattern as the docker role. Gate to kai-server (where Caddy runs) like the docker role. Origin: surfaced during the docker-ce converge, infrastructure#450.

coilysiren referenced this issue

2026-07-04 10:38:26 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #470

coilysiren referenced this issue

2026-07-04 10:38:33 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #471

coilyco-ops added the

label

2026-07-04 17:38:08 +00:00

coilyco-ops added the

consult

label

2026-07-07 04:08:46 +00:00

coilyco-ops added

headless

and removed

consult

labels

2026-07-09 21:46:19 +00:00

coilyco-ops added

consult

and removed

headless

labels

2026-07-10 08:59:57 +00:00

coilyco-ops commented

2026-07-10 11:32:38 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-455 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T11:32:38Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#455 · branch issue-455 · harness codex · workflow direct-main
Run: engineer-codex-infrastructure-455 · ward v0.580.0 · dispatched 2026-07-10T11:32:38Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-455` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T11:32:38Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#455` · branch `issue-455` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-455` · ward `v0.580.0` · dispatched `2026-07-10T11:32:38Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-10 11:40:21 +00:00

ansible: manage Caddy apt repo keyring

coilyco-ops closed this issue

2026-07-10 11:40:21 +00:00

coilyco-ops commented

2026-07-10 11:51:21 +00:00

Author

Member

WARD-OUTCOME: done

details

workflow: direct-to-main; review summary: skipped by ~/.ward/config.yaml default. review gate skipped by ~/.ward/config.yaml default. Felt like a straightforward role-add plus one merge/doc-size cleanup. Confidence: high. Surprises: Forgejo CI stayed queued after the push, and the repo-verb helper required a synced clean tree. Follow-ups: none.

WARD-OUTCOME: done <details><summary>details</summary>workflow: direct-to-main; review summary: skipped by ~/.ward/config.yaml default. review gate skipped by ~/.ward/config.yaml default. Felt like a straightforward role-add plus one merge/doc-size cleanup. Confidence: high. Surprises: Forgejo CI stayed queued after the push, and the repo-verb helper required a synced clean tree. Follow-ups: none.</details>