coilyco-flight-deck/infrastructure

Watch

Tailscale operator OAuth credentials are rejected, blocking device cleanup and namespace termination #765

New issue

Open

opened 2026-08-06 22:54:41 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-08-06 22:54:41 +00:00

Owner

Summary

The Tailscale operator on kai-server cannot authenticate to the Tailscale API. Every device deletion fails with 401 Unauthorized, so the operator never releases tailscale.com/finalizer on the Services it manages. This blocks namespace termination and leaves orphaned tailnet devices behind.

This is not caused by any single workload. It affects the operator globally, including a live service.

Evidence

The operator's service-reconciler logs the same failure on a loop:

failed to cleanup: deleting device: tailscale.DeleteDevice:
Delete "https://api.tailscale.com/api/v2/device/<redacted>":
oauth2: cannot fetch token: 401 Unauthorized
Response: {"message":"API token invalid"}

1430 occurrences in the last 2000 log lines. Observed continuously from at least 16:19 UTC on 2026-08-06 through 20:12 UTC, retrying roughly every 17 minutes per affected Service.

Impact

Three namespaces retired by ward exec retire-zombie-namespaces -- apply are stuck in Terminating. Their workloads, PVCs, and ExternalSecrets are gone. Only the namespace objects remain, each held by a Service carrying the Tailscale finalizer:

coilysiren-backend
coilysiren-eco-mcp-app
coilysiren-eco-spec-tracker

The operator also fails to reconcile the live forgejo Service in the forgejo namespace. That failure predates and is independent of the retirement work, which means the operator cannot currently manage tailnet devices for anything, including services in active use.

Each failed cleanup also leaves its device registered in the tailnet.

Root cause candidates

The operator reads its credential from the operator-oauth Secret in the tailscale namespace, mounted at /oauth/client_id and /oauth/client_secret via CLIENT_ID_FILE and CLIENT_SECRET_FILE.

That Secret was created on 2025-04-24 and is not backed by an ExternalSecret. No ExternalSecret exists in the tailscale namespace at all, so the value is static, hand-created, over a year old, and has no rotation path. An expired or revoked OAuth client is the leading explanation.

Proposed work

Confirm in the Tailscale admin console whether the OAuth client behind operator-oauth is expired, revoked, or missing the scopes needed to delete devices.
Mint a replacement OAuth client with device-write scope and the existing tag:k8s ownership.
Store it in SSM and convert operator-oauth to an ExternalSecret so it converges like every other credential and stops being a hand-created one-off. This is the durable fix, since the static Secret is the reason there is no rotation path today.
Restart the operator and confirm the three namespaces finish terminating on their own.
Audit the tailnet for orphaned devices left by the failed cleanups and remove them.

Verification

kubectl -n tailscale logs deploy/operator reports no API token invalid errors.
The three namespaces above are gone without any finalizer being force-removed.
The forgejo Service reconciles cleanly.

Notes

Do not force-remove tailscale.com/finalizer as a shortcut. It unblocks the namespace but orphans the backing tailnet device and hides the credential failure that also affects the live forgejo Service.

Found while retiring zombie namespaces on kai-server. Runbook is scripts/k8s/retire-zombie-namespaces.sh, which now names this finalizer explicitly when a namespace survives.

## Summary The Tailscale operator on kai-server cannot authenticate to the Tailscale API. Every device deletion fails with `401 Unauthorized`, so the operator never releases `tailscale.com/finalizer` on the Services it manages. This blocks namespace termination and leaves orphaned tailnet devices behind. This is not caused by any single workload. It affects the operator globally, including a live service. ## Evidence The operator's service-reconciler logs the same failure on a loop: ``` failed to cleanup: deleting device: tailscale.DeleteDevice: Delete "https://api.tailscale.com/api/v2/device/<redacted>": oauth2: cannot fetch token: 401 Unauthorized Response: {"message":"API token invalid"} ``` 1430 occurrences in the last 2000 log lines. Observed continuously from at least 16:19 UTC on 2026-08-06 through 20:12 UTC, retrying roughly every 17 minutes per affected Service. ## Impact Three namespaces retired by `ward exec retire-zombie-namespaces -- apply` are stuck in `Terminating`. Their workloads, PVCs, and ExternalSecrets are gone. Only the namespace objects remain, each held by a Service carrying the Tailscale finalizer: * `coilysiren-backend` * `coilysiren-eco-mcp-app` * `coilysiren-eco-spec-tracker` The operator also fails to reconcile the **live** `forgejo` Service in the `forgejo` namespace. That failure predates and is independent of the retirement work, which means the operator cannot currently manage tailnet devices for anything, including services in active use. Each failed cleanup also leaves its device registered in the tailnet. ## Root cause candidates The operator reads its credential from the `operator-oauth` Secret in the `tailscale` namespace, mounted at `/oauth/client_id` and `/oauth/client_secret` via `CLIENT_ID_FILE` and `CLIENT_SECRET_FILE`. That Secret was created on 2025-04-24 and is **not** backed by an ExternalSecret. No `ExternalSecret` exists in the `tailscale` namespace at all, so the value is static, hand-created, over a year old, and has no rotation path. An expired or revoked OAuth client is the leading explanation. ## Proposed work 1. Confirm in the Tailscale admin console whether the OAuth client behind `operator-oauth` is expired, revoked, or missing the scopes needed to delete devices. 2. Mint a replacement OAuth client with device-write scope and the existing `tag:k8s` ownership. 3. Store it in SSM and convert `operator-oauth` to an ExternalSecret so it converges like every other credential and stops being a hand-created one-off. This is the durable fix, since the static Secret is the reason there is no rotation path today. 4. Restart the operator and confirm the three namespaces finish terminating on their own. 5. Audit the tailnet for orphaned devices left by the failed cleanups and remove them. ## Verification * `kubectl -n tailscale logs deploy/operator` reports no `API token invalid` errors. * The three namespaces above are gone without any finalizer being force-removed. * The `forgejo` Service reconciles cleanly. ## Notes Do not force-remove `tailscale.com/finalizer` as a shortcut. It unblocks the namespace but orphans the backing tailnet device and hides the credential failure that also affects the live `forgejo` Service. Found while retiring zombie namespaces on kai-server. Runbook is `scripts/k8s/retire-zombie-namespaces.sh`, which now names this finalizer explicitly when a namespace survives.

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:15 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:31:03 +00:00

coilyco-ops commented

2026-08-28 21:14:57 +00:00

Author

Owner

Still failing, 22 days later. Raising P3 to P2.

Re-verified against the live operator on kai-server 2026-08-28. The failure this issue describes has not stopped for a moment:

{"level":"error","ts":"2026-08-28T21:07:22Z","msg":"Reconciler error",
 "controller":"service-reconciler","namespace":"forgejo","name":"forgejo",
 "error":"failed to cleanup: deleting device: tailscale.DeleteDevice:
  Delete \"https://api.tailscale.com/api/v2/device/<redacted>\":
  oauth2: cannot fetch token: 401 Unauthorized
  Response: {\"message\":\"API token invalid\"}"}

Six occurrences in the last 25 log lines alone, at 20:00Z, 20:17Z, 20:20Z, 20:34Z, 20:50Z and 21:07Z. Same 401, same API token invalid, same forgejo Service. The operator pod itself is 1/1 Running with 0 restarts and 28 days uptime, which is exactly why this stays invisible: nothing crashes, nothing alerts, and a health check on the pod passes while the controller it runs is completely unable to do its job.

Why the priority is wrong

Filed as P3 on 2026-08-06 and untouched since. Three things make that the wrong tier:

It has run continuously for 22 days with no self-recovery path, because the credential is a hand-created static Secret with no ExternalSecret behind it
Every failed cleanup leaks a tailnet device, so the cost accrues rather than holding steady
It blocks namespace termination, so coilysiren-backend, coilysiren-eco-mcp-app and coilysiren-eco-spec-tracker remain stuck in Terminating with their workloads already gone

Raising to P2. Not P1: nothing user-facing is down and the live forgejo Service still works, it just cannot have its tailnet device managed.

The diagnosis in the body still holds

I checked nothing that contradicts it. The static operator-oauth Secret from 2025-04-24, with no rotation path and no ExternalSecret in the tailscale namespace, remains the leading explanation. Step 3 of the proposed work, converting it to an ExternalSecret, is the part that stops this recurring rather than just clearing it once.

Also still true

The body's warning not to force-remove tailscale.com/finalizer as a shortcut. That unblocks the namespaces and hides both the device leak and the credential failure affecting the live Service. Nothing here changes that.

## Still failing, 22 days later. Raising P3 to P2. Re-verified against the live operator on kai-server 2026-08-28. The failure this issue describes has not stopped for a moment: ``` {"level":"error","ts":"2026-08-28T21:07:22Z","msg":"Reconciler error", "controller":"service-reconciler","namespace":"forgejo","name":"forgejo", "error":"failed to cleanup: deleting device: tailscale.DeleteDevice: Delete \"https://api.tailscale.com/api/v2/device/<redacted>\": oauth2: cannot fetch token: 401 Unauthorized Response: {\"message\":\"API token invalid\"}"} ``` Six occurrences in the last 25 log lines alone, at 20:00Z, 20:17Z, 20:20Z, 20:34Z, 20:50Z and 21:07Z. Same `401`, same `API token invalid`, same `forgejo` Service. The operator pod itself is `1/1 Running` with `0` restarts and 28 days uptime, which is exactly why this stays invisible: nothing crashes, nothing alerts, and a health check on the pod passes while the controller it runs is completely unable to do its job. ### Why the priority is wrong Filed as P3 on 2026-08-06 and untouched since. Three things make that the wrong tier: * It has run continuously for 22 days with no self-recovery path, because the credential is a hand-created static Secret with no ExternalSecret behind it * Every failed cleanup leaks a tailnet device, so the cost accrues rather than holding steady * It blocks namespace termination, so `coilysiren-backend`, `coilysiren-eco-mcp-app` and `coilysiren-eco-spec-tracker` remain stuck in `Terminating` with their workloads already gone Raising to **P2**. Not P1: nothing user-facing is down and the live `forgejo` Service still works, it just cannot have its tailnet device managed. ### The diagnosis in the body still holds I checked nothing that contradicts it. The static `operator-oauth` Secret from 2025-04-24, with no rotation path and no ExternalSecret in the `tailscale` namespace, remains the leading explanation. Step 3 of the proposed work, converting it to an ExternalSecret, is the part that stops this recurring rather than just clearing it once. ### Also still true The body's warning not to force-remove `tailscale.com/finalizer` as a shortcut. That unblocks the namespaces and hides both the device leak and the credential failure affecting the live Service. Nothing here changes that.

coilyco-ops added

priority

and removed

priority

labels

2026-08-28 21:15:02 +00:00

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops commented

2026-08-28 21:45:19 +00:00

Author

Owner

Half the impact is gone, and the way it went is the shortcut this issue warned against

Verified 2026-08-28. All three namespaces this issue names as stuck in Terminating are now absent:

coilysiren-backend            Error from server (NotFound)
coilysiren-eco-mcp-app        Error from server (NotFound)
coilysiren-eco-spec-tracker   Error from server (NotFound)

36 namespaces on kai-server, none in a non-Active phase.

Why that is not good news

The credential is still broken. The operator was logging 401 Unauthorized / API token invalid as recently as 21:07Z today, so it still cannot call tailscale.DeleteDevice.

Those two facts together mean the namespaces did not finish terminating through the normal path, because the normal path requires the operator to release tailscale.com/finalizer and it demonstrably still cannot. The most likely explanation is that the finalizer was force-removed, which is precisely what the Notes section of this issue says not to do:

Do not force-remove tailscale.com/finalizer as a shortcut. It unblocks the namespace but orphans the backing tailnet device and hides the credential failure that also affects the live forgejo Service.

I did not do this and I do not know who or what did. Stated as the leading explanation given the evidence, not as established fact. An alternative is that something cleared them by another route, but any route that worked would have needed a working API token, and the token is not working.

What this changes

The namespace-termination half of the impact is resolved and should not be used as justification for this issue any more
The orphaned-device half is now worse, not better. Every one of those three namespaces had a Service holding the finalizer, so if it was force-removed there are now at least three orphaned tailnet devices on top of whatever the ongoing 401 loop has been leaking since 2026-08-06
The live forgejo Service failure is untouched and is the remaining live symptom

Revised remaining work

Steps 1 through 3 of the original plan are unchanged and still the fix: confirm the OAuth client state, mint a replacement with device-write scope, and convert operator-oauth to an ExternalSecret so it stops being a hand-created one-off with no rotation path.

Step 4 is now moot, since the namespaces are gone.

Step 5 grew. The tailnet device audit was already needed for the ongoing failures. It now also needs to cover the three devices behind those namespaces, which nothing will clean up automatically.

## Half the impact is gone, and the way it went is the shortcut this issue warned against Verified 2026-08-28. All three namespaces this issue names as stuck in `Terminating` are now **absent**: ``` coilysiren-backend Error from server (NotFound) coilysiren-eco-mcp-app Error from server (NotFound) coilysiren-eco-spec-tracker Error from server (NotFound) ``` 36 namespaces on kai-server, **none** in a non-`Active` phase. ### Why that is not good news The credential is still broken. The operator was logging `401 Unauthorized` / `API token invalid` as recently as 21:07Z today, so it still cannot call `tailscale.DeleteDevice`. Those two facts together mean the namespaces did not finish terminating through the normal path, because the normal path requires the operator to release `tailscale.com/finalizer` and it demonstrably still cannot. The most likely explanation is that the finalizer was **force-removed**, which is precisely what the Notes section of this issue says not to do: > Do not force-remove `tailscale.com/finalizer` as a shortcut. It unblocks the namespace but orphans the backing tailnet device and hides the credential failure that also affects the live `forgejo` Service. I did not do this and I do not know who or what did. Stated as the leading explanation given the evidence, not as established fact. An alternative is that something cleared them by another route, but any route that worked would have needed a working API token, and the token is not working. ### What this changes * The **namespace-termination** half of the impact is resolved and should not be used as justification for this issue any more * The **orphaned-device** half is now worse, not better. Every one of those three namespaces had a Service holding the finalizer, so if it was force-removed there are now at least three orphaned tailnet devices on top of whatever the ongoing 401 loop has been leaking since 2026-08-06 * The live `forgejo` Service failure is untouched and is the remaining live symptom ### Revised remaining work Steps 1 through 3 of the original plan are unchanged and still the fix: confirm the OAuth client state, mint a replacement with device-write scope, and convert `operator-oauth` to an ExternalSecret so it stops being a hand-created one-off with no rotation path. Step 4 is now moot, since the namespaces are gone. **Step 5 grew.** The tailnet device audit was already needed for the ongoing failures. It now also needs to cover the three devices behind those namespaces, which nothing will clean up automatically.

coilyco-ops commented

2026-08-29 05:14:51 +00:00

Author

Owner

Still failing, now measured rather than sampled. Most recent failure 2026-08-29T05:11:01Z, minutes before this comment.

#981 raised this P3 to P2 on the strength of one log line at 21:07Z. Here is the shape of it.

The error, verbatim and unchanging

Reconciler error   controller=service-reconciler   namespace=forgejo   name=forgejo
failed to cleanup: deleting device: tailscale.DeleteDevice:
  Delete "https://api.tailscale.com/api/v2/device/<device-id>":
  oauth2: cannot fetch token: 401 Unauthorized
  Response: {"message":"API token invalid"}

API token invalid. Not expired, not rate-limited, not a network failure. The OAuth client credential the operator holds is rejected outright by the Tailscale API.

The cadence is exact

Consecutive failures from the last seven hours:

22:30:45  22:47:26  23:04:06  23:20:47  23:37:28  23:54:08  00:10:49  00:27:30
00:44:10  01:00:51  01:17:32  01:34:12  01:50:53  02:07:34  02:24:14  02:40:55
02:57:35  03:14:16  03:30:57  03:47:38  04:04:18  04:20:59  04:37:39  04:54:20
05:11:01

Every 1000 seconds, to the second. That is the controller-runtime backoff cap, so it has been retrying at its slowest permitted rate rather than degrading. At 86.4 failures a day and the 22 days #981 established, that is roughly 1,900 consecutive failures.

Every one is the same reconcile: same controller, same forgejo/forgejo Service, same single device id. The operator is wedged trying to delete one orphaned tailnet device and cannot authenticate to do it.

The pod is still lying about it

tailscale/operator-748ccd7c75-bb4mx   1/1 Running   restarts 0   age 28d

Zero restarts in 28 days. #981 called this out as the reason nobody noticed and it is worth restating with the number attached: a component that has failed 1,900 times in a row presents as the healthiest pod on the cluster. Nothing in kubectl get pod distinguishes it from one doing its job.

Third instance today of the deploy#637 class, after deploy#818 at 851 restarts and deploy#436 at sixteen OOMKills. This one is the worst of the three for detection, because the other two at least move a restart counter.

What is actually blocked, stated as inference

The observed failure is a delete that cannot authenticate. Marked as inference rather than measurement, because I did not test it: an invalid OAuth credential should fail device creation the same way, which would mean no new tailnet-exposed Service can be provisioned on kai-server at all, not merely that one stale device cannot be cleaned up.

If that holds it reframes this issue from a cleanup failure into a provisioning outage, and it touches deploy#803 and deploy#456, both of which reason about ingress-tailscale behaviour. The cheap test is to apply a throwaway Service with the tailnet annotation and see whether a device appears. That wants an attended window, not a backlog pass.

The fix is not reachable from this seat

A new Tailscale OAuth client is minted in the Tailscale admin console. No agent can do that. Same shape as deploy#818, and it is the second credential today whose rotation only Kai can perform.

Sequence once the credential exists: update the SSM-backed projection the operator reads, restart the operator, and confirm the 1000-second error cadence stops. The cadence itself is the verification signal and it is unusually clean, since silence for over 1000 seconds is proof rather than encouragement.

No change made. The credential is out of reach and restarting the operator without a valid one would only reset a counter that is already at zero.

## Still failing, now measured rather than sampled. Most recent failure 2026-08-29T05:11:01Z, minutes before this comment. `#981` raised this P3 to P2 on the strength of one log line at 21:07Z. Here is the shape of it. ### The error, verbatim and unchanging ``` Reconciler error controller=service-reconciler namespace=forgejo name=forgejo failed to cleanup: deleting device: tailscale.DeleteDevice: Delete "https://api.tailscale.com/api/v2/device/<device-id>": oauth2: cannot fetch token: 401 Unauthorized Response: {"message":"API token invalid"} ``` **`API token invalid`.** Not expired, not rate-limited, not a network failure. The OAuth client credential the operator holds is rejected outright by the Tailscale API. ### The cadence is exact Consecutive failures from the last seven hours: ``` 22:30:45 22:47:26 23:04:06 23:20:47 23:37:28 23:54:08 00:10:49 00:27:30 00:44:10 01:00:51 01:17:32 01:34:12 01:50:53 02:07:34 02:24:14 02:40:55 02:57:35 03:14:16 03:30:57 03:47:38 04:04:18 04:20:59 04:37:39 04:54:20 05:11:01 ``` **Every 1000 seconds, to the second.** That is the controller-runtime backoff cap, so it has been retrying at its slowest permitted rate rather than degrading. At 86.4 failures a day and the 22 days `#981` established, that is roughly **1,900 consecutive failures**. Every one is the same reconcile: same controller, same `forgejo/forgejo` Service, same single device id. The operator is wedged trying to delete one orphaned tailnet device and cannot authenticate to do it. ### The pod is still lying about it ``` tailscale/operator-748ccd7c75-bb4mx 1/1 Running restarts 0 age 28d ``` **Zero restarts in 28 days.** `#981` called this out as the reason nobody noticed and it is worth restating with the number attached: a component that has failed 1,900 times in a row presents as the healthiest pod on the cluster. Nothing in `kubectl get pod` distinguishes it from one doing its job. Third instance today of the `deploy#637` class, after `deploy#818` at 851 restarts and `deploy#436` at sixteen OOMKills. **This one is the worst of the three for detection**, because the other two at least move a restart counter. ### What is actually blocked, stated as inference The observed failure is a **delete** that cannot authenticate. Marked as inference rather than measurement, because I did not test it: an invalid OAuth credential should fail device **creation** the same way, which would mean **no new tailnet-exposed Service can be provisioned on kai-server at all**, not merely that one stale device cannot be cleaned up. If that holds it reframes this issue from a cleanup failure into a provisioning outage, and it touches `deploy#803` and `deploy#456`, both of which reason about `ingress-tailscale` behaviour. **The cheap test is to apply a throwaway Service with the tailnet annotation and see whether a device appears.** That wants an attended window, not a backlog pass. ### The fix is not reachable from this seat A new Tailscale OAuth client is minted in the Tailscale admin console. **No agent can do that.** Same shape as `deploy#818`, and it is the second credential today whose rotation only Kai can perform. Sequence once the credential exists: update the SSM-backed projection the operator reads, restart the operator, and confirm the 1000-second error cadence stops. **The cadence itself is the verification signal** and it is unusually clean, since silence for over 1000 seconds is proof rather than encouragement. **No change made.** The credential is out of reach and restarting the operator without a valid one would only reset a counter that is already at zero.

coilyco-ops added this to the Networking and DNS milestone

2026-08-29 06:09:35 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#765

No description provided.

Rows
Columns