/eco-app/discord/token is rejected by Discord with 401, so the eco-app Discord worker has crash-looped 362 times #965

New issue

Open

opened 2026-08-27 09:51:17 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-27 09:51:17 +00:00

Owner

Found while checking cluster health after coilyco-bridge/deploy#811. Unrelated to
that incident, and it predates it.

Observed state

coilysiren-eco-app/coilysiren-eco-app-discord reports READY 0/1,
AVAILABLE 0, on a deployment that is 35 days old.

The discord-worker container is in CrashLoopBackOff with 362 restarts.
It runs for about 13 seconds and exits 1:

Started:   Thu, 27 Aug 2026 02:46:34 -0700
Finished:  Thu, 27 Aug 2026 02:46:47 -0700
Exit Code: 1

kubelet has logged 7849 BackOff events for it.

Cause

Discord rejects the bot token:

discord.errors.HTTPException: 401 Unauthorized (error code: 0): 401: Unauthorized
  ...
discord.errors.LoginFailure: Improper token has been passed.

The failure is on GET /users/@me during static_login, so it is the token
itself and not an intent, permission, or guild problem. The two WARNING lines
above it (Guilds intent seems to be disabled, PyNaCl ... not installed) are
noise and not the failure.

Delivery is not the problem

The ExternalSecret is healthy, so nothing is wrong with the sync path:

NAME                         STORE                 REFRESH   STATUS         READY
coilysiren-eco-app-discord   aws-parameter-store   1h        SecretSynced   True

refreshTime: 2026-08-27T09:37:09Z, condition Ready=True reason=SecretSynced.

It maps four parameters, and the failing one is the first:

ECO_DISCORD_TOKEN <- /eco-app/discord/token
ECO_DISCORD_APPLICATION_ID <- /eco-app/discord/application-id
ECO_DISCORD_TEST_GUILD_ID <- /eco-app/discord/test-guild-id
ECO_DISCORD_INFO_CHANNEL_ID <- /eco-app/discord/info-channel-id

So External Secrets is faithfully delivering a value that Discord will not
accept. The stored parameter is stale, revoked, regenerated in the Discord
developer portal, or otherwise wrong. I did not read the value.

What this is not

Not a deploy-repo defect. The manifest, image pin, and env wiring are intact.
Not an application defect. The worker fails at login, before any of its own
logic runs. Image is coilyco-gaming/eco-app:48590dc7d7873f39db2a1ddeac760974c2fff985.
Not caused by deploy#811 or its fixes. The current pod is 30h old and the
ReplicaSet dates to 2026-07-23.

Remediation

Operator work, because it needs the Discord developer portal and a SecureString
write:

Confirm the bot application still exists and read its current token from the
Discord developer portal, regenerating if needed.
Write it to /eco-app/discord/token as a SecureString, without echoing it.
Force the ExternalSecret to reconcile, or wait out the 1h refresh, then
restart the deployment.
Confirm coilysiren-eco-app-discord reaches 1/1 and that the restart count
stops climbing.

The shape matches #593, a non-empty but unusable value sitting in SSM behind a
healthy ExternalSecret. If that issue's validation-and-rotation path lands, this
parameter is a candidate for the same treatment: a token that logs in cleanly
once at write time is worth more than one that is merely non-empty.

Application context for the worker itself is
coilyco-gaming/eco-app#144.

Found while checking cluster health after coilyco-bridge/deploy#811. Unrelated to that incident, and it predates it. ## Observed state `coilysiren-eco-app/coilysiren-eco-app-discord` reports `READY 0/1`, `AVAILABLE 0`, on a deployment that is 35 days old. The `discord-worker` container is in `CrashLoopBackOff` with **362 restarts**. It runs for about 13 seconds and exits 1: ``` Started: Thu, 27 Aug 2026 02:46:34 -0700 Finished: Thu, 27 Aug 2026 02:46:47 -0700 Exit Code: 1 ``` kubelet has logged 7849 `BackOff` events for it. ## Cause Discord rejects the bot token: ``` discord.errors.HTTPException: 401 Unauthorized (error code: 0): 401: Unauthorized ... discord.errors.LoginFailure: Improper token has been passed. ``` The failure is on `GET /users/@me` during `static_login`, so it is the token itself and not an intent, permission, or guild problem. The two `WARNING` lines above it (`Guilds intent seems to be disabled`, `PyNaCl ... not installed`) are noise and not the failure. ## Delivery is not the problem The ExternalSecret is healthy, so nothing is wrong with the sync path: ``` NAME STORE REFRESH STATUS READY coilysiren-eco-app-discord aws-parameter-store 1h SecretSynced True ``` `refreshTime: 2026-08-27T09:37:09Z`, condition `Ready=True reason=SecretSynced`. It maps four parameters, and the failing one is the first: * `ECO_DISCORD_TOKEN` <- `/eco-app/discord/token` * `ECO_DISCORD_APPLICATION_ID` <- `/eco-app/discord/application-id` * `ECO_DISCORD_TEST_GUILD_ID` <- `/eco-app/discord/test-guild-id` * `ECO_DISCORD_INFO_CHANNEL_ID` <- `/eco-app/discord/info-channel-id` So External Secrets is faithfully delivering a value that Discord will not accept. The stored parameter is stale, revoked, regenerated in the Discord developer portal, or otherwise wrong. I did not read the value. ## What this is not * Not a deploy-repo defect. The manifest, image pin, and env wiring are intact. * Not an application defect. The worker fails at login, before any of its own logic runs. Image is `coilyco-gaming/eco-app:48590dc7d7873f39db2a1ddeac760974c2fff985`. * Not caused by deploy#811 or its fixes. The current pod is 30h old and the ReplicaSet dates to 2026-07-23. ## Remediation Operator work, because it needs the Discord developer portal and a SecureString write: 1. Confirm the bot application still exists and read its current token from the Discord developer portal, regenerating if needed. 2. Write it to `/eco-app/discord/token` as a SecureString, without echoing it. 3. Force the ExternalSecret to reconcile, or wait out the 1h refresh, then restart the deployment. 4. Confirm `coilysiren-eco-app-discord` reaches `1/1` and that the restart count stops climbing. ## Related The shape matches #593, a non-empty but unusable value sitting in SSM behind a healthy ExternalSecret. If that issue's validation-and-rotation path lands, this parameter is a candidate for the same treatment: a token that logs in cleanly once at write time is worth more than one that is merely non-empty. Application context for the worker itself is coilyco-gaming/eco-app#144.

coilyco-ops added this to the Credentials and access milestone

2026-08-29 06:09:51 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#965

No description provided.

Rows
Columns