eco-server: harden boot path + resolve -userToken leak and DiscordLink error (post-reboot incident) #537

New issue

Open

opened 2026-07-10 15:58:14 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-10 15:58:14 +00:00

Owner

Incident (2026-07-10): Eco game server did not come back after kai-server reboot

eco.coilysiren.me (EcoServer 0.13.0.4) crash-looped after the reboot ~08:28. Two independent regressions, both now fixed in bdc6bab:

1. Missing boot scripts (fixed)

eco-server.service runs scripts/eco-server-pre.sh (steamcmd validate) and scripts/eco-server-start.sh (token injection + launch) as its ExecStartPre/ExecStart. Both were deleted in d51837a (coilysiren/inbox#174) as "never called in a real deploy" - but they are the actual boot path. The running process predated the deletion, so it only broke on the first reboot since. Boot failed with exit 127 (No such file or directory). Restored from d51837a^.

bc5a48b dropped -userToken from the EcoServer launch on the theory that injecting the token into Network.eco as ServerAPIToken replaces it. It does not: EcoServer 0.13.0.4 only accepts Strange Cloud login via argv (-userToken / -username+-password / -steamTicket / -offline). Without the flag every boot failed with No authentication methods were available, even though the JWT (valid, non-expired, play.eco-issued) was present in the config. Re-added -userToken; the server now authenticates (Logged in as coilysiren) and players connect.

Open follow-ups

Token leak tradeoff (was issue #2). -userToken=<jwt> re-exposes the bearer token in /proc/<pid>/cmdline and any systemctl status/ps/journald scrape. The token is valid until 2052, so the leak is meaningful. Options: find a non-argv token path in EcoServer 0.13 (none in the accepted-methods list), rotate to a shorter-lived token, or accept + document. bc5a48b's fix regressed availability, so it can't simply be reinstated.
DiscordLink error on every message. System.InvalidOperationException: TextCommandProcessor has not been configured fires from the DiscordLink plugin on each Discord MessageCreated event. Non-fatal (server + players fine) but noisy in the journal. Likely a DiscordLink config/version mismatch after the steamcmd update.
Guard against silent boot-path removal. The removal sweep didn't cross-check systemd/*.service ExecStart references before deleting scripts/. Consider a pre-commit check that fails when a tracked ExecStart/ExecStartPre path points at a non-existent repo file.

Refs: coilysiren/inbox#174 (removal record), commits d51837a, bc5a48b, bdc6bab.

## Incident (2026-07-10): Eco game server did not come back after kai-server reboot `eco.coilysiren.me` (EcoServer 0.13.0.4) crash-looped after the reboot ~08:28. Two independent regressions, both now fixed in `bdc6bab`: ### 1. Missing boot scripts (fixed) `eco-server.service` runs `scripts/eco-server-pre.sh` (steamcmd validate) and `scripts/eco-server-start.sh` (token injection + launch) as its `ExecStartPre`/`ExecStart`. Both were deleted in `d51837a` (coilysiren/inbox#174) as "never called in a real deploy" - but they are the actual boot path. The running process predated the deletion, so it only broke on the first reboot since. Boot failed with exit 127 (`No such file or directory`). Restored from `d51837a^`. ### 2. Dropped Strange Cloud login (fixed) `bc5a48b` dropped `-userToken` from the `EcoServer` launch on the theory that injecting the token into `Network.eco` as `ServerAPIToken` replaces it. It does not: EcoServer 0.13.0.4 only accepts Strange Cloud login via argv (`-userToken` / `-username`+`-password` / `-steamTicket` / `-offline`). Without the flag every boot failed with `No authentication methods were available`, even though the JWT (valid, non-expired, play.eco-issued) was present in the config. Re-added `-userToken`; the server now authenticates (`Logged in as coilysiren`) and players connect. ## Open follow-ups - [ ] **Token leak tradeoff (was issue #2).** `-userToken=<jwt>` re-exposes the bearer token in `/proc/<pid>/cmdline` and any `systemctl status`/`ps`/journald scrape. The token is valid until 2052, so the leak is meaningful. Options: find a non-argv token path in EcoServer 0.13 (none in the accepted-methods list), rotate to a shorter-lived token, or accept + document. `bc5a48b`'s fix regressed availability, so it can't simply be reinstated. - [ ] **DiscordLink error on every message.** `System.InvalidOperationException: TextCommandProcessor has not been configured` fires from the DiscordLink plugin on each Discord `MessageCreated` event. Non-fatal (server + players fine) but noisy in the journal. Likely a DiscordLink config/version mismatch after the steamcmd update. - [ ] **Guard against silent boot-path removal.** The removal sweep didn't cross-check `systemd/*.service` ExecStart references before deleting `scripts/`. Consider a pre-commit check that fails when a tracked `ExecStart`/`ExecStartPre` path points at a non-existent repo file. Refs: coilysiren/inbox#174 (removal record), commits d51837a, bc5a48b, bdc6bab.

coilyco-ops referenced this issue

2026-07-10 16:58:42 +00:00

Post-reboot recovery incomplete: Forgejo Actions runners + DNS/tailnet degraded on kai-server, landing-blocking fleet burndown #538

coilyco-ops commented

2026-07-10 18:13:09 +00:00

Author

Owner

Folded under the resilience epic #541 - eco-server post-reboot boot path is another surface of the same self-recovery gap