tailscale-operator OAuth token 401, new ts-proxies don't provision #42

New issue

Closed

opened 2026-05-23 20:54:32 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-05-23 20:54:32 +00:00

Owner

Originally filed by @coilysiren on 2026-05-20T06:16:12Z - https://github.com/coilysiren/infrastructure/issues/198

The tailscale operator on kai-server is failing to provision new ts-proxies. Every reconcile against a Service with tailscale.com/expose: "true" returns 401 Unauthorized from api.tailscale.com/api/v2/tailnet/-/keys:

failed to provision: failed to create or get API key secret:
Post "https://api.tailscale.com/api/v2/tailnet/-/keys":
oauth2: cannot fetch token: 401 Unauthorized
Response: {"message":"API token invalid"}

Surfaced today while bootstrapping repo-recall: the Deployment is 2/2 Running, the Service has the right annotations, but no ts-proxy stands up in the tailscale namespace.

The operator is also failing for default/null-service, so the token is invalid org-wide, not specific to repo-recall. Pre-existing exposures (forgejo, coilysiren-backend, galaxy-gen, eco-mcp-app, eco-spec-tracker) keep working because their tailnet keys were provisioned earlier and persisted as Secrets.

Operator Secret is presumably the OAuth client credentials managed by terraform/tailscale-oidc/ (or similar). Rotate / re-create, then bounce the operator. Anything new that wants tailnet exposure is blocked until this lands.

_Originally filed by @coilysiren on 2026-05-20T06:16:12Z - [https://github.com/coilysiren/infrastructure/issues/198](https://github.com/coilysiren/infrastructure/issues/198)_ The tailscale operator on kai-server is failing to provision new ts-proxies. Every reconcile against a Service with `tailscale.com/expose: "true"` returns 401 Unauthorized from `api.tailscale.com/api/v2/tailnet/-/keys`: ``` failed to provision: failed to create or get API key secret: Post "https://api.tailscale.com/api/v2/tailnet/-/keys": oauth2: cannot fetch token: 401 Unauthorized Response: {"message":"API token invalid"} ``` Surfaced today while bootstrapping repo-recall: the Deployment is `2/2 Running`, the Service has the right annotations, but no ts-proxy stands up in the `tailscale` namespace. The operator is also failing for `default/null-service`, so the token is invalid org-wide, not specific to repo-recall. Pre-existing exposures (forgejo, coilysiren-backend, galaxy-gen, eco-mcp-app, eco-spec-tracker) keep working because their tailnet keys were provisioned earlier and persisted as Secrets. Operator Secret is presumably the OAuth client credentials managed by `terraform/tailscale-oidc/` (or similar). Rotate / re-create, then bounce the operator. Anything new that wants tailnet exposure is blocked until this lands.

coilysiren added the

label

2026-06-04 08:16:56 +00:00

coilysiren added

and removed

labels

2026-06-17 08:40:35 +00:00

coilysiren referenced this issue

2026-07-04 10:38:26 +00:00

Apply priority triage to 78 un-triaged flight-deck+bridge tickets (labels out of director tier) #470

coilysiren referenced this issue

2026-07-04 10:38:33 +00:00