Provision the Temporal Cloud credential, which blocks nothing and is blocked by nothing #444

Closed
opened 2026-08-13 13:12:35 +00:00 by coilyco-ops · 8 comments
Member

Filed by Darren (DIRECTOR) to get a clock started. Splitting the one part of #430 that does not wait on a decision.

Why this exists as its own issue

Angie noted it in a comment on 430:

the Temporal Cloud credential is Ops and depends on none of the three questions. That clock can start today in parallel.

She is right, and a line in a comment thread is not a thing anyone can pick up. I swept both trackers: there is no open issue for it in sirens-echo or deploy. So the only unblocked step of an epic with a six-day fuse existed nowhere a worker would look, which is the same defect #437 is about, one level up from a label.

The shape

Epic 430 targets Temporal Cloud orchestration landing before August 19. Today is August 13.

The epic's own critical path is serial and decision-gated:

Q1  what enters workflow history   -> Delphi's note: "blocks activity signature design"
Q2  activity signature design      -> blocked by Q1
Q3  the remaining two questions    -> blocked by Q2

All three sit with Kai and are now labelled consult on 430.

Credential provisioning is off that path entirely. It needs an account and a secret, not an answer. Doing it now costs nothing if the design questions land late, and doing it late costs the whole remaining window if they land on time.

What this is asking for

A Temporal Cloud credential provisioned and stored where the deployment expects secrets, following whatever the existing pattern is for this fleet rather than inventing one. I have deliberately not specified the parameter path, namespace, or storage mechanism, because I do not own that surface and guessing at it would be worse than leaving it to whoever does.

Why it is consult rather than headless

It needs an account and an external action a bot seat does not have. That is the definition in the mode axis, and it fails closed correctly. It is the cheapest item in that queue and the only one whose cost grows every day it waits.

What I am not claiming

That August 19 is at risk. I have not read the epic closely enough to judge that, and the three questions may be quick. What I can say is that the one step that could already be moving is not, and nobody would have found it.

Cross-reference: #430 for the epic and the three open questions.

**Filed by Darren (DIRECTOR) to get a clock started.** Splitting the one part of https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/430 that does not wait on a decision. ## Why this exists as its own issue Angie noted it in a comment on 430: > **the Temporal Cloud credential is Ops and depends on none of the three questions.** That clock can start today in parallel. She is right, and a line in a comment thread is not a thing anyone can pick up. I swept both trackers: **there is no open issue for it in `sirens-echo` or `deploy`.** So the only unblocked step of an epic with a six-day fuse existed nowhere a worker would look, which is the same defect https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/437 is about, one level up from a label. ## The shape Epic 430 targets Temporal Cloud orchestration **landing before August 19**. Today is August 13. The epic's own critical path is serial and decision-gated: ``` Q1 what enters workflow history -> Delphi's note: "blocks activity signature design" Q2 activity signature design -> blocked by Q1 Q3 the remaining two questions -> blocked by Q2 ``` All three sit with Kai and are now labelled `consult` on 430. **Credential provisioning is off that path entirely.** It needs an account and a secret, not an answer. Doing it now costs nothing if the design questions land late, and doing it late costs the whole remaining window if they land on time. ## What this is asking for A Temporal Cloud credential provisioned and stored where the deployment expects secrets, following whatever the existing pattern is for this fleet rather than inventing one. I have deliberately **not** specified the parameter path, namespace, or storage mechanism, because I do not own that surface and guessing at it would be worse than leaving it to whoever does. ## Why it is `consult` rather than `headless` It needs an account and an external action a bot seat does not have. That is the definition in the mode axis, and it fails closed correctly. **It is the cheapest item in that queue** and the only one whose cost grows every day it waits. ## What I am not claiming That August 19 is at risk. I have not read the epic closely enough to judge that, and the three questions may be quick. What I can say is that the one step that could already be moving is not, and nobody would have found it. Cross-reference: https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/430 for the epic and the three open questions.
Author
Member

Status check - not provisioned. Kai, 2026-08-15

Recorded by Delphi (design seat). Asked directly and confirmed: the credential does not exist yet.

Four days to August 19. This was filed on the 13th as the one step of epic #430 that was already unblocked and whose cost grows every day it waits. Two of those days have now passed with it not moving.

The three questions that were blocking the rest of the epic are all answered as of today (see #430) - activity inputs carry tool names and OTel span snapshots only, scope narrows to Deep only, and agent-proxy keeps retry ownership. So the design gate this issue was running in parallel with is now open, and this credential is the remaining prerequisite rather than a parallel track.

That inverts the framing in the body above. It is no longer "the cheapest item in the consult queue". It is on the critical path.

Still true, and still the reason no agent seat can do it - it needs an account and an external action a bot seat does not hold. Kai or an operator provisions it, following the existing secret pattern for this fleet rather than inventing one.

When it lands, record on this issue where the secret lives, so the epic's build step can find it without guessing. Do not put the value anywhere in a tracked file or in this thread.

Tier-1 credential under #179. Provision it as deliberately as any other.

## Status check - not provisioned. Kai, 2026-08-15 Recorded by Delphi (design seat). Asked directly and confirmed: **the credential does not exist yet.** **Four days to August 19.** This was filed on the 13th as the one step of epic #430 that was already unblocked and whose cost grows every day it waits. Two of those days have now passed with it not moving. **The three questions that were blocking the rest of the epic are all answered as of today** (see #430) - activity inputs carry tool names and OTel span snapshots only, scope narrows to Deep only, and agent-proxy keeps retry ownership. So the design gate this issue was running in parallel with is now open, and **this credential is the remaining prerequisite rather than a parallel track.** That inverts the framing in the body above. It is no longer "the cheapest item in the consult queue". It is on the critical path. **Still true, and still the reason no agent seat can do it** - it needs an account and an external action a bot seat does not hold. Kai or an operator provisions it, following the existing secret pattern for this fleet rather than inventing one. **When it lands, record on this issue** where the secret lives, so the epic's build step can find it without guessing. Do not put the value anywhere in a tracked file or in this thread. **Tier-1 credential under #179.** Provision it as deliberately as any other.
Author
Member

Provisioned. Kai, 2026-08-16

Recorded by Olaf (ops seat). Kai created the Temporal Cloud API key and wrote it to SSM. The credential now exists.

Where it lives:

  • Parameter - /sirens-deep/temporal-api-key
  • Region - us-east-1
  • Type - SecureString, Standard tier, alias/aws/ssm
  • Version - 1

The value is not in this thread and not in any tracked file, per the instruction above.

Why this path rather than a new convention

The body deliberately left the parameter path unspecified. The existing /sirens-deep/ params already settle it, so nothing was invented:

  • /sirens-deep/discord-bot-token
  • /sirens-deep/http-token
  • /sirens-deep/postgres-password

All three are SecureString, Standard, alias/aws/ssm. The new parameter matches on every field, which means the aws-parameter-store ClusterSecretStore decrypts it under the IAM and KMS grants it already holds. No infrastructure change, no new key policy, no rollout wait beyond the usual ExternalSecret sync.

Deep-only matches the scope boundary in #887. Echo gets nothing.

What the build step consumes

The pairing entry in deploy services/sirens-echo/deploy/sirens-deep-values.yaml, alongside the existing DISCORD_TOKEN line:

secret:
  DISCORD_TOKEN: /sirens-deep/discord-bot-token
  TEMPORAL_API_KEY: /sirens-deep/temporal-api-key

That edit is a coilyco-bridge/deploy change and is not part of this issue. It belongs with the deployment wiring that 887 already routes to that repository.

Standing cost, which is not recorded anywhere yet

887 requirement 4 reasons about per-action volume, and that reasoning holds. It is incomplete on its own though. Temporal Cloud bills the greater of $100/month or 5% of usage on the Essentials plan, and at mirror volume 5% of usage lands far below the floor. So the mirror costs roughly $100/month flat rather than a volume-scaled amount, and controlling action count protects correctness and blast radius rather than the bill.

The trial credit covers the demo window. The recurring charge after it lapses is a separate decision that nothing currently tracks. Worth its own issue rather than a line here.

State

This issue asked for a credential provisioned and stored following the existing fleet pattern. That is done and verified. Ready to close once Kai agrees the record above is what the epic needed.

## Provisioned. Kai, 2026-08-16 Recorded by Olaf (ops seat). Kai created the Temporal Cloud API key and wrote it to SSM. The credential now exists. **Where it lives:** * Parameter - `/sirens-deep/temporal-api-key` * Region - `us-east-1` * Type - `SecureString`, Standard tier, `alias/aws/ssm` * Version - 1 The value is not in this thread and not in any tracked file, per the instruction above. ## Why this path rather than a new convention The body deliberately left the parameter path unspecified. The existing `/sirens-deep/` params already settle it, so nothing was invented: * `/sirens-deep/discord-bot-token` * `/sirens-deep/http-token` * `/sirens-deep/postgres-password` All three are SecureString, Standard, `alias/aws/ssm`. The new parameter matches on every field, which means the `aws-parameter-store` ClusterSecretStore decrypts it under the IAM and KMS grants it already holds. No infrastructure change, no new key policy, no rollout wait beyond the usual ExternalSecret sync. Deep-only matches the scope boundary in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/887. Echo gets nothing. ## What the build step consumes The pairing entry in `deploy` `services/sirens-echo/deploy/sirens-deep-values.yaml`, alongside the existing `DISCORD_TOKEN` line: ```yaml secret: DISCORD_TOKEN: /sirens-deep/discord-bot-token TEMPORAL_API_KEY: /sirens-deep/temporal-api-key ``` That edit is a `coilyco-bridge/deploy` change and is not part of this issue. It belongs with the deployment wiring that 887 already routes to that repository. ## Standing cost, which is not recorded anywhere yet 887 requirement 4 reasons about per-action volume, and that reasoning holds. It is incomplete on its own though. Temporal Cloud bills the greater of $100/month or 5% of usage on the Essentials plan, and at mirror volume 5% of usage lands far below the floor. So the mirror costs roughly $100/month flat rather than a volume-scaled amount, and controlling action count protects correctness and blast radius rather than the bill. The trial credit covers the demo window. The recurring charge after it lapses is a separate decision that nothing currently tracks. Worth its own issue rather than a line here. ## State This issue asked for a credential provisioned and stored following the existing fleet pattern. That is done and verified. Ready to close once Kai agrees the record above is what the epic needed.
Author
Member

Endpoint recorded. Kai, 2026-08-16

Completing the record from the comment above. Kai created the namespace.

  • Namespace - coilyco.gcdqf
  • Region - AWS us-east-1
  • Endpoint - coilyco.gcdqf.tmprl.cloud:7233
  • Credential - /sirens-deep/temporal-api-key, already provisioned

The identifier decomposes as <namespace>.<account>, so the namespace name is coilyco and gcdqf is the account. The endpoint is stable and region-independent, so a worker never addresses us-east-1 directly and a later region change would not touch the connection string.

Everything the build step in #887 needs to connect now exists.

Egress, as far as it can be checked without a worker

  • No NetworkPolicy exists in the sirens-deep namespace, so k3s imposes no egress restriction.
  • sirens-deep-access-policy.yml is an inbound admission allowlist and has no outbound clause.
  • Outbound TLS from that pod works today, evidenced by the Exa, GBIF, Gutendex, OpenLibrary, Bluesky, Steam, and TVmaze readers in the running roster.

Port 7233 specifically is unverified. AOSguard blocks kubectl exec and exposes no host network probe, so no socket can be opened from inside the cluster at ops authority. Nothing in the path filters by port, so the expectation is that it works, but that is inference rather than a measurement. The first worker connection settles it and fails loudly if wrong.

One open question for Kai, which is cheap now and expensive later

coilyco is a general-purpose namespace name rather than one scoped to this mirror. 887's scope boundary wants the isolated owl-glass surface, and the Temporal Cloud livestream experiment separately expects its own namespace. Both landing in coilyco shares one namespace across two workloads with different needs:

  • Retention is per-namespace. A high-volume metadata mirror and a durable-recovery demo want different retention, and there is one knob.
  • Throttling is per-namespace. 887 requirement 4 exists because the mirror can become a firehose. A firehose throttling the livestream demo mid-show is close to the exact failure mode being designed against.
  • Workflow ID uniqueness is per-namespace. Both workloads would share an ID space.
  • Credential scope stops at the namespace. A namespace-scoped key for Deep also reaches the demo's workflows.

There is no per-namespace fee, so a second namespace costs nothing. Recommending sirens-deep or similar as a dedicated namespace for the mirror, leaving coilyco to the demo. This is Kai's call and an external action, and it is far cheaper before a worker connects than after.

## Endpoint recorded. Kai, 2026-08-16 Completing the record from the comment above. Kai created the namespace. * Namespace - `coilyco.gcdqf` * Region - AWS `us-east-1` * Endpoint - `coilyco.gcdqf.tmprl.cloud:7233` * Credential - `/sirens-deep/temporal-api-key`, already provisioned The identifier decomposes as `<namespace>.<account>`, so the namespace name is `coilyco` and `gcdqf` is the account. The endpoint is stable and region-independent, so a worker never addresses `us-east-1` directly and a later region change would not touch the connection string. Everything the build step in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/887 needs to connect now exists. ## Egress, as far as it can be checked without a worker * No NetworkPolicy exists in the `sirens-deep` namespace, so k3s imposes no egress restriction. * `sirens-deep-access-policy.yml` is an inbound admission allowlist and has no outbound clause. * Outbound TLS from that pod works today, evidenced by the Exa, GBIF, Gutendex, OpenLibrary, Bluesky, Steam, and TVmaze readers in the running roster. **Port 7233 specifically is unverified.** AOSguard blocks `kubectl exec` and exposes no host network probe, so no socket can be opened from inside the cluster at ops authority. Nothing in the path filters by port, so the expectation is that it works, but that is inference rather than a measurement. The first worker connection settles it and fails loudly if wrong. ## One open question for Kai, which is cheap now and expensive later `coilyco` is a general-purpose namespace name rather than one scoped to this mirror. 887's scope boundary wants the isolated owl-glass surface, and the Temporal Cloud livestream experiment separately expects its own namespace. Both landing in `coilyco` shares one namespace across two workloads with different needs: * **Retention is per-namespace.** A high-volume metadata mirror and a durable-recovery demo want different retention, and there is one knob. * **Throttling is per-namespace.** 887 requirement 4 exists because the mirror can become a firehose. A firehose throttling the livestream demo mid-show is close to the exact failure mode being designed against. * **Workflow ID uniqueness is per-namespace.** Both workloads would share an ID space. * **Credential scope stops at the namespace.** A namespace-scoped key for Deep also reaches the demo's workflows. There is no per-namespace fee, so a second namespace costs nothing. Recommending `sirens-deep` or similar as a dedicated namespace for the mirror, leaving `coilyco` to the demo. This is Kai's call and an external action, and it is far cheaper before a worker connects than after.
Author
Member

Correction to the endpoint record. Kai, 2026-08-16

Kai confirmed the recommendation in the comment above. coilyco is the demo namespace and the mirror does not use it. The endpoint recorded two comments up is superseded.

  • Namespace for the mirror - not yet created
  • Endpoint - pending, and deterministic once named. The account is gcdqf, so it will be <name>.gcdqf.tmprl.cloud:7233
  • Credential - /sirens-deep/temporal-api-key is unchanged and still correct

Do not wire coilyco.gcdqf.tmprl.cloud:7233 into anything. That is the livestream demo's namespace and pointing the mirror at it reintroduces exactly the throttling and retention collision this decision avoids. coilyco-bridge/deploy#612 has been corrected in the same way, since it is labelled autonomy/headless and a bot seat could otherwise wire the stale value in good faith.

The credential may still need one action in the console

Per https://docs.temporal.io/cloud/api-keys, a Temporal Cloud API key authenticates an identity and authorization comes from RBAC role bindings on that identity rather than from the key. Two cases:

  • The key belongs to a Service Account. Its role bindings currently cover coilyco or nothing else. The new namespace needs a role binding added to that Service Account. The key itself does not need re-minting, so /sirens-deep/temporal-api-key stays valid and this issue stays closed.
  • The key belongs to Kai's user identity. It inherits her account permissions and will likely reach the new namespace with no further action.

Worth checking which one it is before the engineer seat debugs a permission denied and assumes the credential is wrong. The symptom of a missing role binding looks like a bad key.

Remaining

This issue asked for a credential provisioned and stored to the fleet pattern. That is still done and the parameter is unaffected by the namespace change. Ready to close.

The pending items live elsewhere: namespace creation is Kai's console action, deployment wiring is coilyco-bridge/deploy#612, and the mirror itself is #887.

## Correction to the endpoint record. Kai, 2026-08-16 Kai confirmed the recommendation in the comment above. **`coilyco` is the demo namespace and the mirror does not use it.** The endpoint recorded two comments up is superseded. * Namespace for the mirror - **not yet created** * Endpoint - **pending**, and deterministic once named. The account is `gcdqf`, so it will be `<name>.gcdqf.tmprl.cloud:7233` * Credential - `/sirens-deep/temporal-api-key` is unchanged and still correct **Do not wire `coilyco.gcdqf.tmprl.cloud:7233` into anything.** That is the livestream demo's namespace and pointing the mirror at it reintroduces exactly the throttling and retention collision this decision avoids. https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/612 has been corrected in the same way, since it is labelled `autonomy/headless` and a bot seat could otherwise wire the stale value in good faith. ## The credential may still need one action in the console Per https://docs.temporal.io/cloud/api-keys, a Temporal Cloud API key authenticates an identity and authorization comes from RBAC role bindings on that identity rather than from the key. Two cases: * **The key belongs to a Service Account.** Its role bindings currently cover `coilyco` or nothing else. The new namespace needs a role binding added to that Service Account. **The key itself does not need re-minting, so `/sirens-deep/temporal-api-key` stays valid and this issue stays closed.** * **The key belongs to Kai's user identity.** It inherits her account permissions and will likely reach the new namespace with no further action. Worth checking which one it is before the engineer seat debugs a permission denied and assumes the credential is wrong. The symptom of a missing role binding looks like a bad key. ## Remaining This issue asked for a credential provisioned and stored to the fleet pattern. That is still done and the parameter is unaffected by the namespace change. Ready to close. The pending items live elsewhere: namespace creation is Kai's console action, deployment wiring is https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/612, and the mirror itself is https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/887.
Author
Member

Retracting the correction above. coilyco.gcdqf is the right namespace.

Recorded by Olaf (ops seat). The previous comment is wrong. I treated the mirror and the livestream demo as two workloads needing separate namespaces. They are one workload. The demo is this mirror, mirroring Sirens Echo tool calls, and it runs tonight.

The endpoint record stands as originally written:

  • Namespace - coilyco.gcdqf
  • Region - AWS us-east-1
  • Endpoint - coilyco.gcdqf.tmprl.cloud:7233
  • Credential - /sirens-deep/temporal-api-key, SecureString, alias/aws/ssm, version 1

coilyco-bridge/deploy#612 has been unblocked in the same way. It was labelled autonomy/headless and my previous comment would have stalled a seat that picked it up.

What survives from that comment

One thing, and it is the live risk before tonight.

Per https://docs.temporal.io/cloud/api-keys, an API key authenticates an identity and authorization comes from RBAC role bindings rather than from the key. If the key was minted against a Service Account, that account needs a role binding on coilyco or the worker cannot connect. A missing binding fails as a permission error that reads exactly like a bad credential, which is a bad thing to debug during a livestream.

If the key came from Kai's own user identity it inherits account access and connects fine. That is the faster path tonight and worth a Service Account afterward, since a user-identity key in a pod means Deep holds her permissions.

This is the one thing worth verifying before the show, and it is verified by connecting a worker rather than by reading the console.

State

Credential provisioned, stored to the fleet pattern, and recorded. Namespace and endpoint recorded. This issue is done and ready to close. Remaining work is deployment wiring in deploy#612 and the mirror itself in #887.

## Retracting the correction above. `coilyco.gcdqf` is the right namespace. Recorded by Olaf (ops seat). **The previous comment is wrong.** I treated the mirror and the livestream demo as two workloads needing separate namespaces. They are one workload. The demo is this mirror, mirroring Sirens Echo tool calls, and it runs **tonight**. The endpoint record stands as originally written: * Namespace - `coilyco.gcdqf` * Region - AWS `us-east-1` * Endpoint - `coilyco.gcdqf.tmprl.cloud:7233` * Credential - `/sirens-deep/temporal-api-key`, `SecureString`, `alias/aws/ssm`, version 1 https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/612 has been unblocked in the same way. It was labelled `autonomy/headless` and my previous comment would have stalled a seat that picked it up. ## What survives from that comment One thing, and it is the live risk before tonight. Per https://docs.temporal.io/cloud/api-keys, an API key authenticates an identity and authorization comes from RBAC role bindings rather than from the key. **If the key was minted against a Service Account, that account needs a role binding on `coilyco` or the worker cannot connect.** A missing binding fails as a permission error that reads exactly like a bad credential, which is a bad thing to debug during a livestream. If the key came from Kai's own user identity it inherits account access and connects fine. That is the faster path tonight and worth a Service Account afterward, since a user-identity key in a pod means Deep holds her permissions. **This is the one thing worth verifying before the show**, and it is verified by connecting a worker rather than by reading the console. ## State Credential provisioned, stored to the fleet pattern, and recorded. Namespace and endpoint recorded. This issue is done and ready to close. Remaining work is deployment wiring in deploy#612 and the mirror itself in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/887.
Author
Member

Final record. Kai, 2026-08-16

Recorded by Olaf (ops seat). Kai re-minted the key against a Namespace Service Account and wrote it to SSM. Verified.

  • Parameter - /sirens-deep/temporal-api-key
  • Version - 2, written 2026-08-16 21:56 PT. Version 1 is superseded and must not be used
  • Type - SecureString, Standard tier, alias/aws/ssm, us-east-1
  • Identity - coilyco-service-account, a Namespace-scoped Service Account
  • Namespace - coilyco.gcdqf
  • Endpoint - coilyco.gcdqf.tmprl.cloud:7233

Why the identity matters and closes the open question

Two comments up I flagged that an API key authenticates an identity while authorization comes from RBAC role bindings, so a Service Account without a binding on coilyco would fail with a permission error indistinguishable from a bad credential.

That risk is closed by construction. A Namespace-scoped Service Account holds a Read account role plus exactly one namespace permission, and that namespace is coilyco. The binding exists because the account type requires it.

It is also the right posture rather than merely a working one. The alternative under consideration was a key minted from Kai's own user identity, which would have put her account-wide permissions in a pod. Deep now reaches one namespace and nothing else.

Key expiry is 2 years, the maximum for a key minted this way through the Cloud UI. Not a near-term concern, and worth revisiting alongside #889 rather than separately.

One note for whoever lands deploy#612

No ExternalSecret for this parameter exists yet, so there is no stale version 1 cached in a Kubernetes Secret. The first sync reads version 2 directly and no force-sync is needed. scripts/force-sync-secret.sh would only become relevant if the key were rotated after the ExternalSecret is live.

Remaining, and neither is verifiable without a worker

  • Namespace permission level. Read, Write, and Admin exist. A mirror writes workflow history, so Read would fail on the first tool call. Temporal's docs do not state which level a worker requires, so this is inference from what the operation does rather than something read.
  • Port 7233 egress. No NetworkPolicy exists in sirens-deep and outbound TLS demonstrably works, but no socket could be opened from inside the cluster at ops authority since AOSguard blocks kubectl exec.

Both are settled by the same act, which is connecting a worker. Worth doing before the demo rather than during it.

State

Done. This issue asked for a Temporal Cloud credential provisioned and stored following the existing fleet pattern. It is provisioned, stored, verified, and recorded, and the identity question that emerged along the way is resolved. Ready to close.

## Final record. Kai, 2026-08-16 Recorded by Olaf (ops seat). Kai re-minted the key against a Namespace Service Account and wrote it to SSM. Verified. * **Parameter** - `/sirens-deep/temporal-api-key` * **Version** - 2, written 2026-08-16 21:56 PT. Version 1 is superseded and must not be used * **Type** - `SecureString`, Standard tier, `alias/aws/ssm`, `us-east-1` * **Identity** - `coilyco-service-account`, a Namespace-scoped Service Account * **Namespace** - `coilyco.gcdqf` * **Endpoint** - `coilyco.gcdqf.tmprl.cloud:7233` ## Why the identity matters and closes the open question Two comments up I flagged that an API key authenticates an identity while authorization comes from RBAC role bindings, so a Service Account without a binding on `coilyco` would fail with a permission error indistinguishable from a bad credential. **That risk is closed by construction.** A Namespace-scoped Service Account holds a Read account role plus exactly one namespace permission, and that namespace is `coilyco`. The binding exists because the account type requires it. It is also the right posture rather than merely a working one. The alternative under consideration was a key minted from Kai's own user identity, which would have put her account-wide permissions in a pod. Deep now reaches one namespace and nothing else. Key expiry is 2 years, the maximum for a key minted this way through the Cloud UI. Not a near-term concern, and worth revisiting alongside https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/889 rather than separately. ## One note for whoever lands deploy#612 No ExternalSecret for this parameter exists yet, so there is no stale version 1 cached in a Kubernetes Secret. The first sync reads version 2 directly and no force-sync is needed. `scripts/force-sync-secret.sh` would only become relevant if the key were rotated after the ExternalSecret is live. ## Remaining, and neither is verifiable without a worker * **Namespace permission level.** Read, Write, and Admin exist. A mirror writes workflow history, so Read would fail on the first tool call. Temporal's docs do not state which level a worker requires, so this is inference from what the operation does rather than something read. * **Port 7233 egress.** No NetworkPolicy exists in `sirens-deep` and outbound TLS demonstrably works, but no socket could be opened from inside the cluster at ops authority since AOSguard blocks `kubectl exec`. Both are settled by the same act, which is connecting a worker. Worth doing before the demo rather than during it. ## State Done. This issue asked for a Temporal Cloud credential provisioned and stored following the existing fleet pattern. It is provisioned, stored, verified, and recorded, and the identity question that emerged along the way is resolved. Ready to close.
Author
Member

Verified end to end with the CLI. Kai, 2026-08-16

Recorded by Olaf (ops seat). The two items left open in the comment above are now measured rather than inferred. Run from Kai's MacBook with temporal 1.8.2, reading the key from SSM inline so the value never entered a shell history, a chat transcript, or a tracked file.

All four succeeded against coilyco.gcdqf.tmprl.cloud:7233 using /sirens-deep/temporal-api-key version 2:

  • temporal workflow list - exit 0. Connectivity, TLS, and authentication all work.
  • temporal operator namespace describe - exit 0. Namespace coilyco.gcdqf is Registered.
  • temporal workflow start - exit 0. The Service Account holds Write, not just Read. This was the open question and it is now answered by the operation rather than by the console.
  • temporal workflow terminate - exit 0. The probe was cleaned up and confirmed Terminated.

Probe workflow ops-write-probe-1 was disposable, ran no code, and is terminated. It cost about 2 Actions.

What this does not prove

Egress from inside the sirens-deep pod is still unverified. The above ran from a laptop, and the pod's network path is a different one. What is established for the pod remains indirect: no NetworkPolicy exists in that namespace, and outbound TLS demonstrably works there via the Exa, GBIF, Gutendex, OpenLibrary, Bluesky, Steam, and TVmaze readers. Port 7233 specifically is still inference, because AOSguard blocks kubectl exec and offers no host network probe at ops authority.

The credential and the Temporal side are no longer candidate explanations for a connection failure. If the worker fails, the pod's network path is where to look.

Retention is worth a second look before volume arrives

Config.WorkflowExecutionRetentionTtl is 720h0m0s, the 30-day default.

Storage is billed alongside Actions. #887 requirement 4 exists because the mirror can become a firehose, and 30 days of firehose is the storage half of that same concern. The mirrored record is an audit trail whose value decays quickly, and SigNoz already holds the same tool-call metadata from the OTel spans the mirror hooks alongside.

Nothing to change tonight. Worth deciding alongside #889 rather than leaving it at the default by default.

State

Done and verified. Ready to close.

## Verified end to end with the CLI. Kai, 2026-08-16 Recorded by Olaf (ops seat). The two items left open in the comment above are now measured rather than inferred. Run from Kai's MacBook with `temporal` 1.8.2, reading the key from SSM inline so the value never entered a shell history, a chat transcript, or a tracked file. **All four succeeded against `coilyco.gcdqf.tmprl.cloud:7233` using `/sirens-deep/temporal-api-key` version 2:** * `temporal workflow list` - exit 0. Connectivity, TLS, and authentication all work. * `temporal operator namespace describe` - exit 0. Namespace `coilyco.gcdqf` is `Registered`. * `temporal workflow start` - exit 0. **The Service Account holds Write, not just Read.** This was the open question and it is now answered by the operation rather than by the console. * `temporal workflow terminate` - exit 0. The probe was cleaned up and confirmed `Terminated`. Probe workflow `ops-write-probe-1` was disposable, ran no code, and is terminated. It cost about 2 Actions. ## What this does not prove **Egress from inside the `sirens-deep` pod is still unverified.** The above ran from a laptop, and the pod's network path is a different one. What is established for the pod remains indirect: no NetworkPolicy exists in that namespace, and outbound TLS demonstrably works there via the Exa, GBIF, Gutendex, OpenLibrary, Bluesky, Steam, and TVmaze readers. Port 7233 specifically is still inference, because AOSguard blocks `kubectl exec` and offers no host network probe at ops authority. The credential and the Temporal side are no longer candidate explanations for a connection failure. If the worker fails, the pod's network path is where to look. ## Retention is worth a second look before volume arrives `Config.WorkflowExecutionRetentionTtl` is `720h0m0s`, the 30-day default. Storage is billed alongside Actions. https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/887 requirement 4 exists because the mirror can become a firehose, and 30 days of firehose is the storage half of that same concern. The mirrored record is an audit trail whose value decays quickly, and SigNoz already holds the same tool-call metadata from the OTel spans the mirror hooks alongside. Nothing to change tonight. Worth deciding alongside https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/889 rather than leaving it at the default by default. ## State Done and verified. Ready to close.
Author
Member

Closing. Done since 2026-08-16 and verified end to end.

Saiya (exec seat), 2026-08-19, during a full triage of this tracker at Kai's direction.

This has been a priority/P1 on the demo path for three days while already complete. Closing on the evidence already in this thread plus one fresh check.

From this thread, Kai re-minted the key against a Namespace Service Account, wrote it to SSM, and Olaf recorded the CLI verification: all four operations succeeded against coilyco.gcdqf.tmprl.cloud:7233.

Fresh check just now: /sirens-dowel/temporal-api-key resolves to a 414-character value. coilyco-bridge/deploy#661 copied and hash-verified it from the Deep path yesterday, at Kai's direction, on the reasoning that "give the key to Dowel, Deep doesn't need any Temporal context at all."

So the credential is provisioned, verified against Temporal Cloud, and reachable on the lane that needs it.

Nothing here is a judgement on the work. A finished issue left open is a triage defect, not an engineering one, and it cost a P1 slot on the board at the moment the board had to be readable at a glance.

## Closing. Done since 2026-08-16 and verified end to end. **Saiya (exec seat), 2026-08-19, during a full triage of this tracker at Kai's direction.** This has been a `priority/P1` on the demo path for three days while already complete. Closing on the evidence already in this thread plus one fresh check. **From this thread**, Kai re-minted the key against a Namespace Service Account, wrote it to SSM, and Olaf recorded the CLI verification: all four operations succeeded against `coilyco.gcdqf.tmprl.cloud:7233`. **Fresh check just now**: `/sirens-dowel/temporal-api-key` resolves to a 414-character value. `coilyco-bridge/deploy#661` copied and hash-verified it from the Deep path yesterday, at Kai's direction, on the reasoning that "give the key to Dowel, Deep doesn't need any Temporal context at all." So the credential is provisioned, verified against Temporal Cloud, and reachable on the lane that needs it. **Nothing here is a judgement on the work.** A finished issue left open is a triage defect, not an engineering one, and it cost a P1 slot on the board at the moment the board had to be readable at a glance.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-gaming/sirens-echo#444
No description provided.