Apply ComfyUI Tailscale authkey and verify ExternalSecret #498

New issue

Open

opened 2026-07-09 19:24:56 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-09 19:24:56 +00:00

Member

Problem

deploy-comfyui.yml now gets past the previous deploy-owned RBAC blocker, but the latest rerun still fails waiting for the ComfyUI Tailscale authkey ExternalSecret.

Latest observed deploy run:

workflow: deploy-comfyui.yml
run number: 625
event: workflow_dispatch
head SHA: d482205
status: failure
URL: https://forgejo.coilysiren.me/coilyco-bridge/deploy/actions/runs/625

The deploy engineer on coilyco-bridge/deploy#108 reported the exact remaining blocker as:

services/comfyui/scripts/rollout.sh times out on kubectl -n comfyui wait --for=condition=Ready externalsecret/ts-authkey --timeout=2m.

Prior infra issue #479 added comfyui to Tailscale authkey minting, but it closed with the apply still outstanding because the run lacked operator-held Tailscale admin credentials. The live deploy failure now proves the secret is still not ready for the cluster.

Do

Check whether the current guarded infra surface can run the Tailscale Terraform apply.
Apply the existing comfyui Tailscale authkey config if credentials are available.
Verify the SSM/source secret exists and ExternalSecret comfyui/ts-authkey becomes Ready on the cluster.
If the apply still requires a human-held credential, comment with the exact one-line command and the exact missing env/secret, then leave this issue open.
After the secret is Ready, redispatch deploy-comfyui.yml and report the new run number.

Acceptance

kubectl -n comfyui wait --for=condition=Ready externalsecret/ts-authkey --timeout=2m succeeds, or this issue contains exact proof of the missing credential wall.
A fresh deploy-comfyui.yml run gets past the authkey wait.

## Problem `deploy-comfyui.yml` now gets past the previous deploy-owned RBAC blocker, but the latest rerun still fails waiting for the ComfyUI Tailscale authkey ExternalSecret. Latest observed deploy run: * workflow: `deploy-comfyui.yml` * run number: `625` * event: `workflow_dispatch` * head SHA: `d482205` * status: `failure` * URL: https://forgejo.coilysiren.me/coilyco-bridge/deploy/actions/runs/625 The deploy engineer on `coilyco-bridge/deploy#108` reported the exact remaining blocker as: `services/comfyui/scripts/rollout.sh` times out on `kubectl -n comfyui wait --for=condition=Ready externalsecret/ts-authkey --timeout=2m`. Prior infra issue `#479` added `comfyui` to Tailscale authkey minting, but it closed with the apply still outstanding because the run lacked operator-held Tailscale admin credentials. The live deploy failure now proves the secret is still not ready for the cluster. ## Do * Check whether the current guarded infra surface can run the Tailscale Terraform apply. * Apply the existing `comfyui` Tailscale authkey config if credentials are available. * Verify the SSM/source secret exists and ExternalSecret `comfyui/ts-authkey` becomes Ready on the cluster. * If the apply still requires a human-held credential, comment with the exact one-line command and the exact missing env/secret, then leave this issue open. * After the secret is Ready, redispatch `deploy-comfyui.yml` and report the new run number. ## Acceptance * `kubectl -n comfyui wait --for=condition=Ready externalsecret/ts-authkey --timeout=2m` succeeds, or this issue contains exact proof of the missing credential wall. * A fresh `deploy-comfyui.yml` run gets past the authkey wait.

coilyco-ops commented

2026-07-09 19:25:07 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-498 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T19:25:05Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#498 · branch issue-498 · harness codex · workflow direct-to-main
Run: engineer-codex-infrastructure-498 · ward v0.495.0 · dispatched 2026-07-09T19:25:05Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.495.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-498` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T19:25:05Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#498` · branch `issue-498` · harness `codex` · workflow `direct-to-main` - **Run:** `engineer-codex-infrastructure-498` · ward `v0.495.0` · dispatched `2026-07-09T19:25:05Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.495.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-09 19:27:32 +00:00

Author

Member

WARD-OUTCOME: blocked 🛑

details

workflow: direct-to-main; review summary: skipped by ~/.ward/config.yaml default retrospective: the infra surface is fine, but `ward exec terraform-tailscale action=apply` is blocked in this container by missing operator-held Tailscale admin creds. The missing env is `TAILSCALE_API_KEY` or the `TAILSCALE_OAUTH_CLIENT_ID` + `TAILSCALE_OAUTH_CLIENT_SECRET` pair. confidence: high surprises: `ward` needed writable `UV_CACHE_DIR` and `UV_PYTHON_INSTALL_DIR` overrides to get far enough to prove the credential wall. follow-ups: export the admin Tailscale credential in-shell, rerun `ward exec terraform-tailscale action=apply`, verify `kubectl -n comfyui wait --for=condition=Ready externalsecret/ts-authkey --timeout=2m`, then redispatch `deploy-comfyui.yml`.

WARD-OUTCOME: blocked 🛑 <details><summary>details</summary> workflow: direct-to-main; review summary: skipped by ~/.ward/config.yaml default retrospective: the infra surface is fine, but `ward exec terraform-tailscale action=apply` is blocked in this container by missing operator-held Tailscale admin creds. The missing env is `TAILSCALE_API_KEY` or the `TAILSCALE_OAUTH_CLIENT_ID` + `TAILSCALE_OAUTH_CLIENT_SECRET` pair. confidence: high surprises: `ward` needed writable `UV_CACHE_DIR` and `UV_PYTHON_INSTALL_DIR` overrides to get far enough to prove the credential wall. follow-ups: export the admin Tailscale credential in-shell, rerun `ward exec terraform-tailscale action=apply`, verify `kubectl -n comfyui wait --for=condition=Ready externalsecret/ts-authkey --timeout=2m`, then redispatch `deploy-comfyui.yml`. </details>

coilyco-ops added the

interactive

labels

2026-07-09 21:46:10 +00:00

coilyco-ops added

consult

and removed

interactive

labels

2026-07-10 08:59:52 +00:00