coilyco-flight-deck/infrastructure

Watch

Untracked ExternalSecrets in the default namespace retry a missing SSM key every hour, forever #809

New issue

Closed

opened 2026-08-13 06:32:14 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-08-13 06:32:14 +00:00

Owner

What this is

Seven ExternalSecrets on kai-server are failing continuously with UpdateFailed ... Secret does not exist, and most of them are not defined in any repository I can find.

Namespace	ExternalSecret	SSM key	Occurrences
`external-secrets`	`github-pat`	`/github/pat`	2672
`default`	`docker-registry`	`/github/pat`	2674
`default`	`null-db`	`/coilysiren/backend/db-password`	2673
`default`	`null-datastore-token`	`/coilysiren/backend/datastore-token`	2672
`default`	`null-sentry`	`/sentry-dsn/eco-mcp-app`	2672
`default`	`ts-authkey`	`/coilysiren/eco-mcp/ts-authkey`	2530
`coilysiren-eco-app`	`ts-authkey`	`/coilysiren/eco-mcp/ts-authkey`	2530
`coilysiren-eco-app`	`coilysiren-eco-app-sentry`	`/sentry-dsn/eco-mcp-app`	2672

At refreshInterval: 1h, ~2670 retries is roughly 111 days of hourly failure.

Which are tracked, and which are not

Only one has a definition I could locate:

external-secrets/github-pat → deploy/externalsecret.yml in this repository, reading /github/pat via a SecretStore named aws-parameter-store.

The rest — default/docker-registry, default/null-db, default/null-datastore-token, default/null-sentry, default/ts-authkey — appear in no .yml, .yaml, or .tf file across infrastructure, deploy, agentic-os-kai, or the other checked-out repositories. Note that default/docker-registry is a different object from deploy's docker-registry-edge-basic-auth, which reads /docker-registry/basic-auth-users and is healthy.

The null- prefix suggests these were deliberately-nulled placeholders at some point. That reading is undercut by null-db-0 being a live running Postgres pod in default with a bound 5Gi PVC holding 47MB, so at least that one is not inert.

Why they survive

prune: false is deliberate on the Flux path during the soak, so an object whose manifest was deleted upstream stays in the cluster and keeps reconciling. That is the intended behavior of the soak, but its cost is exactly this: a growing set of objects nobody owns, failing forever, with no signal distinguishing "orphan we forgot" from "live thing that is broken".

The part that actually needs a decision

external-secrets/github-pat is tracked, and it is failing. Either /github/pat does not exist in SSM, or the SecretStore cannot read it. That is a real credential that this repository declares it needs.

I could not verify SSM directly — the aws-ssm MCP exposes exactly one policy-approved parameter, and /github/pat is not it. Someone with SSM read access should check whether /github/pat exists before anything is deleted. If it does exist, this is a store/permission problem, not a missing-key problem, and the error message is misleading.

Why it is worth clearing

Two of these are ts-authkey for eco-mcp. If eco-app ever needs a fresh Tailscale identity, that path is already broken and has been for months — it will fail at the moment it is needed rather than announcing itself now. The rest is event-stream noise that makes real ExternalSecret failures harder to notice.

Acceptance

/github/pat is confirmed present or absent in SSM, and external-secrets/github-pat either syncs or is removed.
Each default-namespace ExternalSecret above is identified as orphan or live. Orphans are deleted; live ones get a tracked manifest.
null-db specifically: it is running with real data on a PVC and is declared nowhere. It needs an owner or a decommission.

Notes

No changes made. Deleting cluster objects that are not in version control is not something to do from an investigation — the manifests may exist somewhere I cannot see, and null-db holds data.

Next owner

Ops, with a director call on the null-* set.

## What this is Seven ExternalSecrets on kai-server are failing continuously with `UpdateFailed ... Secret does not exist`, and most of them are not defined in any repository I can find. | Namespace | ExternalSecret | SSM key | Occurrences | | --- | --- | --- | --- | | `external-secrets` | `github-pat` | `/github/pat` | 2672 | | `default` | `docker-registry` | `/github/pat` | 2674 | | `default` | `null-db` | `/coilysiren/backend/db-password` | 2673 | | `default` | `null-datastore-token` | `/coilysiren/backend/datastore-token` | 2672 | | `default` | `null-sentry` | `/sentry-dsn/eco-mcp-app` | 2672 | | `default` | `ts-authkey` | `/coilysiren/eco-mcp/ts-authkey` | 2530 | | `coilysiren-eco-app` | `ts-authkey` | `/coilysiren/eco-mcp/ts-authkey` | 2530 | | `coilysiren-eco-app` | `coilysiren-eco-app-sentry` | `/sentry-dsn/eco-mcp-app` | 2672 | At `refreshInterval: 1h`, ~2670 retries is roughly **111 days** of hourly failure. ## Which are tracked, and which are not Only one has a definition I could locate: - `external-secrets/github-pat` → `deploy/externalsecret.yml` in this repository, reading `/github/pat` via a `SecretStore` named `aws-parameter-store`. The rest — `default/docker-registry`, `default/null-db`, `default/null-datastore-token`, `default/null-sentry`, `default/ts-authkey` — appear in **no** `.yml`, `.yaml`, or `.tf` file across `infrastructure`, `deploy`, `agentic-os-kai`, or the other checked-out repositories. Note that `default/docker-registry` is a different object from deploy's `docker-registry-edge-basic-auth`, which reads `/docker-registry/basic-auth-users` and is healthy. The `null-` prefix suggests these were deliberately-nulled placeholders at some point. That reading is undercut by `null-db-0` being a **live running Postgres pod** in `default` with a bound 5Gi PVC holding 47MB, so at least that one is not inert. ## Why they survive `prune: false` is deliberate on the Flux path during the soak, so an object whose manifest was deleted upstream stays in the cluster and keeps reconciling. That is the intended behavior of the soak, but its cost is exactly this: a growing set of objects nobody owns, failing forever, with no signal distinguishing "orphan we forgot" from "live thing that is broken". ## The part that actually needs a decision `external-secrets/github-pat` **is** tracked, and it is failing. Either `/github/pat` does not exist in SSM, or the `SecretStore` cannot read it. That is a real credential that this repository declares it needs. I could not verify SSM directly — the `aws-ssm` MCP exposes exactly one policy-approved parameter, and `/github/pat` is not it. **Someone with SSM read access should check whether `/github/pat` exists before anything is deleted.** If it does exist, this is a store/permission problem, not a missing-key problem, and the error message is misleading. ## Why it is worth clearing Two of these are `ts-authkey` for `eco-mcp`. If eco-app ever needs a fresh Tailscale identity, that path is already broken and has been for months — it will fail at the moment it is needed rather than announcing itself now. The rest is event-stream noise that makes real ExternalSecret failures harder to notice. ## Acceptance - `/github/pat` is confirmed present or absent in SSM, and `external-secrets/github-pat` either syncs or is removed. - Each `default`-namespace ExternalSecret above is identified as orphan or live. Orphans are deleted; live ones get a tracked manifest. - `null-db` specifically: it is running with real data on a PVC and is declared nowhere. It needs an owner or a decommission. ## Notes No changes made. Deleting cluster objects that are not in version control is not something to do from an investigation — the manifests may exist somewhere I cannot see, and `null-db` holds data. ## Next owner Ops, with a director call on the `null-*` set.

coilyco-ops referenced this issue

2026-08-13 06:32:42 +00:00

Three PVs retry deletion 1473 times each after their data is already gone #810

coilyco-ops referenced this issue

2026-08-13 06:50:07 +00:00

Three PVs retry deletion 1473 times each after their data is already gone #810

coilyco-ops commented

2026-08-13 06:50:27 +00:00

Author

Owner

Correction: the "~111 days" figure is wrong, and there is a real break date

kubectl describe externalsecret github-pat -n external-secrets:

Status:
  Conditions:
    Last Transition Time:   2026-05-25T06:11:13Z
    Message:                could not get secret data from provider
    Reason:                 SecretSyncedError
    Status:                 False
  Refresh Time:             2026-05-25T05:11:13Z
  Synced Resource Version:  3-58be32f11357c9f970866db775993e2d
Events:
  Warning  UpdateFailed  2m11s (x2690 over 12d)  external-secrets
    error processing spec.data[0] (key: /github/pat), err: Secret does not exist

Two things the body gets wrong

The retry arithmetic. I read ~2670 occurrences as hourly retries and derived "roughly 111 days". The event line reads x2690 over 12d — the counter covers 12 days, not 111, and works out to ~224 failures/day, about one per 6.4 minutes. refreshInterval: 1h governs successful refresh; failures retry far faster. The occurrence counts in the body's table are therefore a measure of retry aggressiveness, not of duration, and should not be read as age for any row.

It is not "always been broken". Synced Resource Version: 3-... and Refresh Time: 2026-05-25T05:11:13Z say this ExternalSecret synced successfully and then broke exactly one hour later, at its next refresh:

2026-05-25T05:11:13Z — last successful refresh
2026-05-25T06:11:13Z — transitioned to SecretSyncedError

That is roughly 80 days of failure, and more usefully, a precise timestamp for what changed. Something happened to /github/pat or to the aws-parameter-store SecretStore's access to it between 05:11 and 06:11 UTC on 2026-05-25. That is a far better lead than the body's framing.

Why it sharpens the SSM question

The body asks someone to check whether /github/pat exists. Still the right check, with a better prior: the parameter was readable on 2026-05-25 at 05:11Z. So the likely candidates are a deleted or renamed parameter, or a credential/permission change on the store — not a key that was never created.

Message: could not get secret data from provider is the provider-level error, distinct from the event's Secret does not exist. Worth confirming the IAM identity behind the SecretStore still has ssm:GetParameter on that path, since a permissions failure can surface as a not-found.

Unchanged

The rest of the issue stands: the default-namespace ExternalSecrets remain untracked in any repository, null-db is still a live Postgres with data and no manifest, and the prune: false explanation for why orphans persist is unaffected. I have only re-derived timings for external-secrets/github-pat; the other rows' occurrence counts are subject to the same 12-day-window caveat and none of them should be read as age.

## Correction: the "~111 days" figure is wrong, and there is a real break date `kubectl describe externalsecret github-pat -n external-secrets`: ``` Status: Conditions: Last Transition Time: 2026-05-25T06:11:13Z Message: could not get secret data from provider Reason: SecretSyncedError Status: False Refresh Time: 2026-05-25T05:11:13Z Synced Resource Version: 3-58be32f11357c9f970866db775993e2d Events: Warning UpdateFailed 2m11s (x2690 over 12d) external-secrets error processing spec.data[0] (key: /github/pat), err: Secret does not exist ``` ### Two things the body gets wrong **The retry arithmetic.** I read ~2670 occurrences as hourly retries and derived "roughly 111 days". The event line reads `x2690 over 12d` — the counter covers **12 days**, not 111, and works out to ~224 failures/day, about one per 6.4 minutes. `refreshInterval: 1h` governs successful refresh; failures retry far faster. The occurrence counts in the body's table are therefore a measure of retry aggressiveness, not of duration, and should not be read as age for any row. **It is not "always been broken".** `Synced Resource Version: 3-...` and `Refresh Time: 2026-05-25T05:11:13Z` say this ExternalSecret **synced successfully** and then broke exactly one hour later, at its next refresh: - `2026-05-25T05:11:13Z` — last successful refresh - `2026-05-25T06:11:13Z` — transitioned to `SecretSyncedError` That is roughly **80 days** of failure, and more usefully, a **precise timestamp for what changed**. Something happened to `/github/pat` or to the `aws-parameter-store` `SecretStore`'s access to it between 05:11 and 06:11 UTC on 2026-05-25. That is a far better lead than the body's framing. ### Why it sharpens the SSM question The body asks someone to check whether `/github/pat` exists. Still the right check, with a better prior: the parameter was readable on 2026-05-25 at 05:11Z. So the likely candidates are a deleted or renamed parameter, or a credential/permission change on the store — not a key that was never created. `Message: could not get secret data from provider` is the provider-level error, distinct from the event's `Secret does not exist`. Worth confirming the IAM identity behind the `SecretStore` still has `ssm:GetParameter` on that path, since a permissions failure can surface as a not-found. ### Unchanged The rest of the issue stands: the `default`-namespace ExternalSecrets remain untracked in any repository, `null-db` is still a live Postgres with data and no manifest, and the `prune: false` explanation for why orphans persist is unaffected. I have only re-derived timings for `external-secrets/github-pat`; the other rows' occurrence counts are subject to the same 12-day-window caveat and none of them should be read as age.

coilyco-ops commented

2026-08-13 06:59:55 +00:00

Author

Owner

Correction: `null-db` is empty. The "it holds data" caution is void.

Kai pointed out the obvious thing I argued past: the name says what it is. Checking the numbers, it holds up.

The body says the null- prefix reading is "undercut by null-db-0 being a live running Postgres pod ... with a bound 5Gi PVC holding 47MB, so at least that one is not inert." That inference was wrong. 47MB is the empty-Postgres floor, not evidence of content.

The comparison that settles it

Postgres	PVC on disk	Container RSS
`sirens-echo/sirens-echo-job-store-data`	48,291,840	26,451,968
`sirens-deep/sirens-deep-job-store-data`	48,291,840	25,079,808
`default/data-null-db-0`	47,767,552	18,464,768
`forgejo/forgejo-db-data`	584,519,680	169,431,040
`coilysiren-eco-gnome/eco-gnome-db-data`	81,584,128	34,570,240

The two job stores are byte-identical at 48,291,840 — and deploy's own README describes those manifests as provisioning "an empty database", with the harness applying its own migrations on boot. So that figure is a known empty-postgres:17 baseline.

null-db sits 524,288 bytes below that baseline and carries the lowest RSS of any Postgres in the cluster. It has less on disk than two databases documented as empty, and it is idle.

What it is

StatefulSet/null-db, 84 days old, 1/1
Headless ClusterIP Service null-db on 5432
Alongside null-service (ClusterIP, port 80, 90 days old)

A consistent null- family: null-db, null-service, null-datastore-token, null-sentry. That reads as a deliberate null-object set — placeholders that exist so something has a resolvable target — which is also the most natural explanation for why their ExternalSecrets point at SSM keys that do not exist. I do not know their original purpose and am not asserting one.

What changes

The Notes section's blocker is removed. I wrote that deletion was risky because null-db holds data. It does not. If these are confirmed as decommissionable placeholders, there is no data-loss risk in removing the StatefulSet, its PVC, and the null-* ExternalSecrets — only the ordinary care of confirming nothing resolves null-db:5432 or null-service.

The null-* ExternalSecret failures may be by design. If these are null objects, three of the eight rows in the body's table are expected-failing and not defects. That would leave the real signal as:

external-secrets/github-pat — tracked, previously synced, broke 2026-05-25 (see the correction above)
default/docker-registry — untracked, reads /github/pat
coilysiren-eco-app/ts-authkey and coilysiren-eco-app-sentry — a live service whose credentials do not resolve

That last pair is the one worth attention, and it was getting lost among the placeholders.

Limits of this check

I inferred emptiness from disk footprint and RSS, not from pg_database_size or a table listing — the guarded kubectl surface has no exec verb, so I could not query the database directly. The evidence is strong but circumstantial. Anyone with a shell should confirm before deleting, which is a one-command check.

## Correction: `null-db` is empty. The "it holds data" caution is void. Kai pointed out the obvious thing I argued past: the name says what it is. Checking the numbers, it holds up. The body says the `null-` prefix reading is *"undercut by `null-db-0` being a live running Postgres pod ... with a bound 5Gi PVC holding 47MB, so at least that one is not inert."* That inference was wrong. **47MB is the empty-Postgres floor, not evidence of content.** ### The comparison that settles it | Postgres | PVC on disk | Container RSS | | --- | --- | --- | | `sirens-echo/sirens-echo-job-store-data` | 48,291,840 | 26,451,968 | | `sirens-deep/sirens-deep-job-store-data` | 48,291,840 | 25,079,808 | | `default/data-null-db-0` | **47,767,552** | **18,464,768** | | `forgejo/forgejo-db-data` | 584,519,680 | 169,431,040 | | `coilysiren-eco-gnome/eco-gnome-db-data` | 81,584,128 | 34,570,240 | The two job stores are byte-identical at 48,291,840 — and deploy's own README describes those manifests as provisioning *"an empty database"*, with the harness applying its own migrations on boot. So that figure is a known empty-`postgres:17` baseline. `null-db` sits **524,288 bytes below** that baseline and carries the **lowest RSS of any Postgres in the cluster**. It has less on disk than two databases documented as empty, and it is idle. ### What it is - `StatefulSet/null-db`, 84 days old, `1/1` - Headless `ClusterIP` Service `null-db` on 5432 - Alongside `null-service` (ClusterIP, port 80, 90 days old) A consistent `null-` family: `null-db`, `null-service`, `null-datastore-token`, `null-sentry`. That reads as a deliberate null-object set — placeholders that exist so something has a resolvable target — which is also the most natural explanation for why their ExternalSecrets point at SSM keys that do not exist. I do not know their original purpose and am not asserting one. ### What changes **The Notes section's blocker is removed.** I wrote that deletion was risky because `null-db` holds data. It does not. If these are confirmed as decommissionable placeholders, there is no data-loss risk in removing the StatefulSet, its PVC, and the `null-*` ExternalSecrets — only the ordinary care of confirming nothing resolves `null-db:5432` or `null-service`. **The `null-*` ExternalSecret failures may be by design.** If these are null objects, three of the eight rows in the body's table are expected-failing and not defects. That would leave the real signal as: - `external-secrets/github-pat` — tracked, previously synced, broke 2026-05-25 (see the correction above) - `default/docker-registry` — untracked, reads `/github/pat` - `coilysiren-eco-app/ts-authkey` and `coilysiren-eco-app-sentry` — a **live service** whose credentials do not resolve That last pair is the one worth attention, and it was getting lost among the placeholders. ### Limits of this check I inferred emptiness from disk footprint and RSS, not from `pg_database_size` or a table listing — the guarded kubectl surface has no `exec` verb, so I could not query the database directly. The evidence is strong but circumstantial. Anyone with a shell should confirm before deleting, which is a one-command check.

coilyco-ops referenced this issue

2026-08-25 06:15:51 +00:00

Nine ExternalSecrets are permanently failing against deleted SSM keys #766

coilyco-ops commented

2026-08-25 06:15:57 +00:00

Author

Owner

Resolved. Zero ExternalSecrets are failing on kai-server.

$ kubectl get externalsecret -A -o custom-columns=STATUS:.status.conditions[0].reason,READY:.status.conditions[0].status
    100  SecretSynced True

All 100, every one synced. The default namespace, called out here specifically, now holds exactly one ExternalSecret (eco-admin-token) and it is healthy.

I did not establish how they were resolved, only that they are. The consumers appear to have been deleted rather than repointed: the three named in the sibling issue #589 all return NotFound, and /github/pat does not exist in SSM either. So the drift was cleaned up from the consumer side at some point and nothing recorded it here.

This and #766 or #809 (whichever you are not reading) shared 12 rare identifiers in a duplicate scan, which is how the pair surfaced. Closing both.

Resolved. Zero ExternalSecrets are failing on kai-server. ``` $ kubectl get externalsecret -A -o custom-columns=STATUS:.status.conditions[0].reason,READY:.status.conditions[0].status 100 SecretSynced True ``` All 100, every one synced. The `default` namespace, called out here specifically, now holds exactly one ExternalSecret (`eco-admin-token`) and it is healthy. I did not establish *how* they were resolved, only that they are. The consumers appear to have been deleted rather than repointed: the three named in the sibling issue #589 all return `NotFound`, and `/github/pat` does not exist in SSM either. So the drift was cleaned up from the consumer side at some point and nothing recorded it here. This and #766 or #809 (whichever you are not reading) shared 12 rare identifiers in a duplicate scan, which is how the pair surfaced. Closing both.

coilyco-ops closed this issue

2026-08-25 06:15:59 +00:00

coilyco-ops referenced this issue

2026-08-25 06:16:01 +00:00

reconcile: retire or replace stale /github/pat ExternalSecrets #589

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#809

No description provided.

Rows
Columns

Untracked ExternalSecrets in the default namespace retry a missing SSM key every hour, forever #809

What this is

Which are tracked, and which are not

Why they survive

The part that actually needs a decision

Why it is worth clearing

Acceptance

Notes

Next owner

Correction: the "~111 days" figure is wrong, and there is a real break date

Two things the body gets wrong

Why it sharpens the SSM question

Unchanged

Correction: null-db is empty. The "it holds data" caution is void.

The comparison that settles it

What it is

What changes

Limits of this check

Correction: `null-db` is empty. The "it holds data" caution is void.