coilyco-flight-deck/infrastructure

Watch

Nine ExternalSecrets are permanently failing against deleted SSM keys #766

New issue

Closed

opened 2026-08-08 01:54:11 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-08 01:54:11 +00:00

Owner

Nine ExternalSecret objects on kai-server are stuck in SecretSyncedError,
each retrying continuously (observed counts 654 to 1610 and climbing). Every one
points at an SSM key that no longer exists.

Why this is not cosmetic

The Secret each one targets still exists, aged 56 to 478 days. Workloads
are running on that stale material right now, so nothing is visibly broken. What
is broken is the ability to ever refresh it.

The credentials cannot be rotated. The source of truth is gone.
If any of these Secret objects is deleted, or its namespace recreated, the
workload cannot come back. There is no path to resynthesise the value.
external-secrets emits a Warning event per retry per object, which is most of
the recent Warning volume on the cluster and buries real signal.

Most pointed example: StatefulSet default/null-db is Running 1/1 and takes
POSTGRES_PASSWORD from the stale null-db Secret (78d). A live Postgres whose
password is unrotatable and unrecoverable.

The nine, grouped by what to do

Declared in this repo, so fix the declaration

external-secrets/github-pat - key /github/pat (missing). Declared in
deploy/externalsecret.yml. Secret is 478d old.
forgejo/forgejo-tap-writer-secrets - keys /forgejo/api-token (missing) and
/forgejo/coilyco-ops/ci-release-token (exists). Declared in
deploy/forgejo-runner-tap-writer.yml.
/forgejo/coilyco-ops/api-token does exist, so this looks like a leftover
from the coilyco-ops token-family migration and the fix is likely a one-line
repath. Its StatefulSet forgejo-runner-tap-writer is scaled to 0, so the
breakage is latent rather than active, and would surface the first time
anyone scales it up.

Declared nowhere, so orphaned cluster state to delete

kubectl apply never prunes, so these survived the work that removed their
definitions. Not in this repo, not in deploy.

coilysiren-eco-app/ts-authkey - key /coilysiren/eco-mcp/ts-authkey.
coilysiren-eco-app/coilysiren-eco-app-sentry - key /sentry-dsn/eco-mcp-app.

Both use retired eco-mcp naming and are leftovers from the predecessor
namespaces that coilyco-bridge/deploy#101 replaced. The eco-app manifest that is
actually deployed declares neither. Related: infrastructure#627 owns deleting
the predecessor workloads.

default/ts-authkey - key /coilysiren/eco-mcp/ts-authkey. Same retired path.
Live tailscale consumers use per-service paths such as
/coilysiren/playwright-mcp/ts-authkey, which sync fine.
default/null-db - key /coilysiren/backend/db-password. Has a running
consumer, see above. Needs a decision, not a blind delete.
default/null-datastore-token - key /coilysiren/backend/datastore-token.
default/null-sentry - key /sentry-dsn/eco-mcp-app.
default/docker-registry - key /github/pat. Secret is 476d old, from the
GitHub-era pull-secret setup.

Distinct missing keys

Six, several shared by two consumers:

/github/pat
/sentry-dsn/eco-mcp-app
/coilysiren/eco-mcp/ts-authkey
/coilysiren/backend/db-password
/coilysiren/backend/datastore-token
/forgejo/api-token

Suggested order

forgejo-tap-writer-secrets, because the correct key demonstrably exists and
it is a real runner that will fail when next scaled up.
The coilysiren-eco-app and default orphans, which are almost certainly
deletions but each needs its consumer checked first.
null-db last and deliberately, since a running database depends on it and
the question is whether the null-* stack should still exist at all.
github-pat and docker-registry, both GitHub-era and likely retired
wholesale.

Each deletion is a live cluster mutation, so this wants an attended operator
rather than an unattended run.

Nine `ExternalSecret` objects on kai-server are stuck in `SecretSyncedError`, each retrying continuously (observed counts 654 to 1610 and climbing). Every one points at an SSM key that no longer exists. ## Why this is not cosmetic The `Secret` each one targets **still exists**, aged 56 to 478 days. Workloads are running on that stale material right now, so nothing is visibly broken. What is broken is the ability to ever refresh it. - The credentials cannot be rotated. The source of truth is gone. - If any of these `Secret` objects is deleted, or its namespace recreated, the workload cannot come back. There is no path to resynthesise the value. - external-secrets emits a Warning event per retry per object, which is most of the recent Warning volume on the cluster and buries real signal. Most pointed example: `StatefulSet default/null-db` is Running 1/1 and takes `POSTGRES_PASSWORD` from the stale `null-db` Secret (78d). A live Postgres whose password is unrotatable and unrecoverable. ## The nine, grouped by what to do **Declared in this repo, so fix the declaration** - `external-secrets/github-pat` - key `/github/pat` (missing). Declared in [deploy/externalsecret.yml](deploy/externalsecret.yml). Secret is 478d old. - `forgejo/forgejo-tap-writer-secrets` - keys `/forgejo/api-token` (missing) and `/forgejo/coilyco-ops/ci-release-token` (exists). Declared in [deploy/forgejo-runner-tap-writer.yml](deploy/forgejo-runner-tap-writer.yml). `/forgejo/coilyco-ops/api-token` **does** exist, so this looks like a leftover from the coilyco-ops token-family migration and the fix is likely a one-line repath. Its `StatefulSet forgejo-runner-tap-writer` is scaled to 0, so the breakage is latent rather than active, and would surface the first time anyone scales it up. **Declared nowhere, so orphaned cluster state to delete** `kubectl apply` never prunes, so these survived the work that removed their definitions. Not in this repo, not in deploy. - `coilysiren-eco-app/ts-authkey` - key `/coilysiren/eco-mcp/ts-authkey`. - `coilysiren-eco-app/coilysiren-eco-app-sentry` - key `/sentry-dsn/eco-mcp-app`. Both use retired `eco-mcp` naming and are leftovers from the predecessor namespaces that coilyco-bridge/deploy#101 replaced. The eco-app manifest that is actually deployed declares neither. Related: infrastructure#627 owns deleting the predecessor workloads. - `default/ts-authkey` - key `/coilysiren/eco-mcp/ts-authkey`. Same retired path. Live tailscale consumers use per-service paths such as `/coilysiren/playwright-mcp/ts-authkey`, which sync fine. - `default/null-db` - key `/coilysiren/backend/db-password`. **Has a running consumer**, see above. Needs a decision, not a blind delete. - `default/null-datastore-token` - key `/coilysiren/backend/datastore-token`. - `default/null-sentry` - key `/sentry-dsn/eco-mcp-app`. - `default/docker-registry` - key `/github/pat`. Secret is 476d old, from the GitHub-era pull-secret setup. ## Distinct missing keys Six, several shared by two consumers: - `/github/pat` - `/sentry-dsn/eco-mcp-app` - `/coilysiren/eco-mcp/ts-authkey` - `/coilysiren/backend/db-password` - `/coilysiren/backend/datastore-token` - `/forgejo/api-token` ## Suggested order 1. `forgejo-tap-writer-secrets`, because the correct key demonstrably exists and it is a real runner that will fail when next scaled up. 2. The `coilysiren-eco-app` and `default` orphans, which are almost certainly deletions but each needs its consumer checked first. 3. `null-db` last and deliberately, since a running database depends on it and the question is whether the `null-*` stack should still exist at all. 4. `github-pat` and `docker-registry`, both GitHub-era and likely retired wholesale. Each deletion is a live cluster mutation, so this wants an attended operator rather than an unattended run.

coilyco-ops commented

2026-08-08 02:02:46 +00:00

Author

Owner

Correction on `forgejo-tap-writer-secrets`: this is not a repath

I guessed "one-line repath" in the description. Wrong. The repo declaration is
already correct. Both keys it names exist in SSM:

/forgejo/runner-registration-token EXISTS
/forgejo/coilyco-ops/tap-bump-token EXISTS

The live object is the stale one. It still carries the pre-migration spec, so
nothing needs editing. The manifest simply was never applied.

What the drift actually is

e0e1de3 (2026-07-09) "fix: remove admin PAT from runner pods" rewrote this
runner to stop minting a registration token with a site-admin PAT inside the
pod, using a pre-minted bounded token instead. That commit has been on main
for a month and never reached the cluster. The live object dates from
2026-06-06.

So this is an unapplied security fix, not a broken key. Server-side dry-run
diff of deploy/forgejo-runner-tap-writer.yml against live:

ExternalSecret

/forgejo/api-token -> /forgejo/runner-registration-token
secretKey api-token -> runner-token
/forgejo/coilyco-ops/ci-release-token -> /forgejo/coilyco-ops/tap-bump-token
drops a leftover force-sync: repoint-a8d7ded annotation, evidence of an
earlier attempt to fix this by forcing a resync rather than applying

StatefulSet

removes the init container's admin-PAT block that wgets
/api/v1/admin/runners/registration-token using API_TOKEN
API_TOKEN / api-token -> RUNNER_TOKEN / runner-token
adds the 30-day stale-runner-registration TTL refresh
replicas: 0 -> 1

Live exposure today is low: the pod count is zero and /forgejo/api-token was
deleted from SSM, so the admin PAT is not merely unused but gone. The cluster
object is just describing a design that no longer exists.

The one consequence to decide

Applying converges to declared desired state, which includes starting the
tap-writer runner for the first time since it was scaled down. That is the repo's
intent (replicas: 1), and it should now succeed where it previously could not,
because the token it needs exists. Rollback is
kubectl scale --replicas=0 statefulset/forgejo-runner-tap-writer -n forgejo.

Applying the ExternalSecret alone would silence the error without starting
anything, but would leave the StatefulSet on the old admin-PAT spec, so the
security fix would stay unapplied. Not recommended as an end state.

## Correction on `forgejo-tap-writer-secrets`: this is not a repath I guessed "one-line repath" in the description. Wrong. **The repo declaration is already correct.** Both keys it names exist in SSM: - `/forgejo/runner-registration-token` EXISTS - `/forgejo/coilyco-ops/tap-bump-token` EXISTS The live object is the stale one. It still carries the pre-migration spec, so nothing needs editing. The manifest simply was never applied. ## What the drift actually is `e0e1de3` (2026-07-09) "fix: remove admin PAT from runner pods" rewrote this runner to stop minting a registration token with a site-admin PAT inside the pod, using a pre-minted bounded token instead. That commit has been on `main` for a month and never reached the cluster. The live object dates from 2026-06-06. So this is **an unapplied security fix**, not a broken key. Server-side dry-run diff of `deploy/forgejo-runner-tap-writer.yml` against live: **ExternalSecret** - `/forgejo/api-token` -> `/forgejo/runner-registration-token` - secretKey `api-token` -> `runner-token` - `/forgejo/coilyco-ops/ci-release-token` -> `/forgejo/coilyco-ops/tap-bump-token` - drops a leftover `force-sync: repoint-a8d7ded` annotation, evidence of an earlier attempt to fix this by forcing a resync rather than applying **StatefulSet** - removes the init container's admin-PAT block that wgets `/api/v1/admin/runners/registration-token` using `API_TOKEN` - `API_TOKEN` / `api-token` -> `RUNNER_TOKEN` / `runner-token` - adds the 30-day stale-runner-registration TTL refresh - **`replicas: 0` -> `1`** Live exposure today is low: the pod count is zero and `/forgejo/api-token` was deleted from SSM, so the admin PAT is not merely unused but gone. The cluster object is just describing a design that no longer exists. ## The one consequence to decide Applying converges to declared desired state, which includes starting the tap-writer runner for the first time since it was scaled down. That is the repo's intent (`replicas: 1`), and it should now succeed where it previously could not, because the token it needs exists. Rollback is `kubectl scale --replicas=0 statefulset/forgejo-runner-tap-writer -n forgejo`. Applying the ExternalSecret alone would silence the error without starting anything, but would leave the StatefulSet on the old admin-PAT spec, so the security fix would stay unapplied. Not recommended as an end state.

coilyco-ops commented

2026-08-25 06:15:51 +00:00

Author

Owner

Resolved. Zero ExternalSecrets are failing on kai-server.

$ kubectl get externalsecret -A -o custom-columns=STATUS:.status.conditions[0].reason,READY:.status.conditions[0].status
    100  SecretSynced True

All 100, every one synced. The default namespace, called out here specifically, now holds exactly one ExternalSecret (eco-admin-token) and it is healthy.

I did not establish how they were resolved, only that they are. The consumers appear to have been deleted rather than repointed: the three named in the sibling issue #589 all return NotFound, and /github/pat does not exist in SSM either. So the drift was cleaned up from the consumer side at some point and nothing recorded it here.

This and #766 or #809 (whichever you are not reading) shared 12 rare identifiers in a duplicate scan, which is how the pair surfaced. Closing both.

Resolved. Zero ExternalSecrets are failing on kai-server. ``` $ kubectl get externalsecret -A -o custom-columns=STATUS:.status.conditions[0].reason,READY:.status.conditions[0].status 100 SecretSynced True ``` All 100, every one synced. The `default` namespace, called out here specifically, now holds exactly one ExternalSecret (`eco-admin-token`) and it is healthy. I did not establish *how* they were resolved, only that they are. The consumers appear to have been deleted rather than repointed: the three named in the sibling issue #589 all return `NotFound`, and `/github/pat` does not exist in SSM either. So the drift was cleaned up from the consumer side at some point and nothing recorded it here. This and #766 or #809 (whichever you are not reading) shared 12 rare identifiers in a duplicate scan, which is how the pair surfaced. Closing both.

coilyco-ops closed this issue

2026-08-25 06:15:54 +00:00

coilyco-ops referenced this issue

2026-08-25 06:15:57 +00:00

Untracked ExternalSecrets in the default namespace retry a missing SSM key every hour, forever #809

coilyco-ops referenced this issue

2026-08-25 06:16:01 +00:00

reconcile: retire or replace stale /github/pat ExternalSecrets #589