securely cache every aws ssm parameter inside k3s #417

New issue

Open

opened 2026-06-26 15:58:28 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-26 15:58:28 +00:00

Owner

Kai's note: securely cache every aws ssm parameter inside k3s. Likely via a formal secrets management tool.

The real problem (why this matters today)

The forgejo API token (and every other secret) lives in AWS SSM Parameter Store. The only way to read it today is a live AWS call (ward-kdl ops aws ssm get-parameter --name /forgejo/api-token ..., see agentic-os/scripts/git-credential-forgejo-ssm.sh). That call needs AWS auth, which needs 2FA. So a lost/absent 2FA device locks Kai out of her own git token. The fix is to mirror SSM into the cluster so day-to-day retrieval is a kubectl get secret over the tailnet, never a 2FA-gated AWS round-trip.

Current state (verified in-repo - do NOT rebuild this, extend it)

The cluster already runs external-secrets with the spine you need:

deploy/secretstore.yml - a SecretStore and ClusterSecretStore named aws-parameter-store, provider AWS ParameterStore, region us-east-1, authenticating with a static aws-credentials secret in the external-secrets namespace (bootstrapped once via coily aws-secrets aws_access_key_id=<ID> aws_secret_access_key=<SECRET>). That static key is NOT 2FA-gated - it is exactly what removes the 2FA dependency once params are mirrored.
Per-param ExternalSecrets already exist, hand-written one at a time: deploy/externalsecret.yml (/github/pat), deploy/forgejo.yml (/forgejo/secret-key, /forgejo/internal-token, ...), deploy/forgejo-runner.yml (/forgejo/api-token -> secret forgejo-runner-secrets key api-token, ns forgejo).

The gap: every param is mirrored by hand. This issue is to mirror all of SSM automatically.

What to build

Add deploy/externalsecret-ssm-cache.yml using external-secrets bulk discovery (dataFrom -> find) instead of enumerating keys. Example shape (verify against the external-secrets version installed - kubectl get deploy -n external-secrets):

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: ssm-cache-coilysiren, namespace: external-secrets }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-parameter-store, kind: ClusterSecretStore }
  target: { name: ssm-cache-coilysiren, creationPolicy: Owner }
  dataFrom:
    - find:
        path: "/coilysiren"          # bulk-pull everything under this prefix
  # rewrite slashes in param names to valid k8s secret keys:
  # (use spec.dataFrom[].rewrite or a rewrite block per the installed CRD version)

Do one ExternalSecret per top-level SSM prefix (/coilysiren, /forgejo, /github, ...), each landing in its own namespaced Secret - do NOT dump all of SSM into a single secret (blast radius: any reader of that one secret reads everything). Scope by prefix and lock down RBAC so only intended consumers read each.
Register the new manifest in deploy/_flux/infrastructure/kustomization.yaml so Flux GitOps-reconciles it (this repo is Flux-managed under deploy/_flux/).
Write docs/ssm-cache.md: what is mirrored, the prefix->namespace->secret mapping, the refresh interval, the RBAC/blast-radius reasoning, and the new everyday-retrieval recipe (kubectl get secret ... -o jsonpath).

Acceptance

kubectl get secret ssm-cache-<prefix> -n <ns> -o json shows the SSM params under that prefix, base64-decodable, refreshing on the interval.
Retrieving the forgejo token (and any other param) needs only tailnet + kubectl, no AWS call, no 2FA.
docs/ssm-cache.md exists and the manifests are wired into Flux.

Files

deploy/secretstore.yml (read), deploy/externalsecret.yml (template), deploy/_flux/infrastructure/kustomization.yaml, new deploy/externalsecret-ssm-cache.yml, new docs/ssm-cache.md. Related: this unblocks the secrets gap in #418; see also #419.

> **Kai's note:** securely cache every aws ssm parameter inside k3s. Likely via a formal secrets management tool. ## The real problem (why this matters today) The forgejo API token (and every other secret) lives in **AWS SSM Parameter Store**. The only way to read it today is a live AWS call (`ward-kdl ops aws ssm get-parameter --name /forgejo/api-token ...`, see `agentic-os/scripts/git-credential-forgejo-ssm.sh`). That call needs AWS auth, which needs **2FA**. So a lost/absent 2FA device locks Kai out of her own git token. The fix is to **mirror SSM into the cluster** so day-to-day retrieval is a `kubectl get secret` over the tailnet, never a 2FA-gated AWS round-trip. ## Current state (verified in-repo - do NOT rebuild this, extend it) The cluster **already runs external-secrets** with the spine you need: - `deploy/secretstore.yml` - a `SecretStore` **and** `ClusterSecretStore` named `aws-parameter-store`, provider AWS `ParameterStore`, region `us-east-1`, authenticating with a **static** `aws-credentials` secret in the `external-secrets` namespace (bootstrapped once via `coily aws-secrets aws_access_key_id=<ID> aws_secret_access_key=<SECRET>`). **That static key is NOT 2FA-gated - it is exactly what removes the 2FA dependency once params are mirrored.** - Per-param `ExternalSecret`s already exist, hand-written one at a time: `deploy/externalsecret.yml` (`/github/pat`), `deploy/forgejo.yml` (`/forgejo/secret-key`, `/forgejo/internal-token`, ...), `deploy/forgejo-runner.yml` (`/forgejo/api-token` -> secret `forgejo-runner-secrets` key `api-token`, ns `forgejo`). The gap: every param is mirrored **by hand**. This issue is to mirror **all of SSM** automatically. ## What to build 1. Add `deploy/externalsecret-ssm-cache.yml` using external-secrets **bulk discovery** (`dataFrom` -> `find`) instead of enumerating keys. Example shape (verify against the external-secrets version installed - `kubectl get deploy -n external-secrets`): ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: { name: ssm-cache-coilysiren, namespace: external-secrets } spec: refreshInterval: 1h secretStoreRef: { name: aws-parameter-store, kind: ClusterSecretStore } target: { name: ssm-cache-coilysiren, creationPolicy: Owner } dataFrom: - find: path: "/coilysiren" # bulk-pull everything under this prefix # rewrite slashes in param names to valid k8s secret keys: # (use spec.dataFrom[].rewrite or a rewrite block per the installed CRD version) ``` 2. Do **one ExternalSecret per top-level SSM prefix** (`/coilysiren`, `/forgejo`, `/github`, ...), each landing in its own namespaced Secret - do NOT dump all of SSM into a single secret (blast radius: any reader of that one secret reads everything). Scope by prefix and lock down RBAC so only intended consumers read each. 3. Register the new manifest in `deploy/_flux/infrastructure/kustomization.yaml` so Flux GitOps-reconciles it (this repo is Flux-managed under `deploy/_flux/`). 4. Write `docs/ssm-cache.md`: what is mirrored, the prefix->namespace->secret mapping, the refresh interval, the RBAC/blast-radius reasoning, and the new everyday-retrieval recipe (`kubectl get secret ... -o jsonpath`). ## Acceptance - `kubectl get secret ssm-cache-<prefix> -n <ns> -o json` shows the SSM params under that prefix, base64-decodable, refreshing on the interval. - Retrieving the forgejo token (and any other param) needs **only** tailnet + kubectl, **no AWS call, no 2FA**. - `docs/ssm-cache.md` exists and the manifests are wired into Flux. ## Files `deploy/secretstore.yml` (read), `deploy/externalsecret.yml` (template), `deploy/_flux/infrastructure/kustomization.yaml`, new `deploy/externalsecret-ssm-cache.yml`, new `docs/ssm-cache.md`. Related: this unblocks the secrets gap in #418; see also #419.

coilysiren referenced this issue

2026-06-26 16:00:14 +00:00

airgapped developement audit #418

coilysiren referenced this issue

2026-06-26 16:07:12 +00:00

airgapped developement audit #418

coilysiren referenced this issue

2026-06-26 16:07:12 +00:00

find the push button tailnet airgap mode #419