coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

eco-server.service: stop leaking userToken JWT via ExecStart argv #2

New issue
Closed
opened 2026-05-23 20:54:25 +00:00 by coilysiren · 4 comments
coilysiren commented 2026-05-23 20:54:25 +00:00
Owner
Copy link

Originally filed by @coilysiren on 2026-05-23T19:19:13Z - https://github.com/coilysiren/infrastructure/issues/297

Problem - The eco-server systemd unit invokes EcoServer -userToken="$eco_server_api_token" (see scripts/eco-server-start.sh), which puts the JWT from SSM /eco/server-api-token into /proc/<pid>/cmdline. Any systemctl status eco-server, ps, or journald scrape that captures the cmdline (including agent transcripts) leaks the token in cleartext.

Surfaced - 2026-05-23, mid-incident chat transcript captured the JWT verbatim from systemctl status output during the LFS-pointer mod recovery (sibling issue: install-eco-mod-source.sh hardening).

Fix shape - Pass the token via env var or stdin instead of argv. Eco supports userToken via config file (Configs/Network.eco -> ServerAPIToken) as well; writing it there at ExecStartPre time and dropping the argv flag is the cleanest path. Rotate /eco/server-api-token once the unit no longer exposes it.

Out of scope - The DiscordLink token already gets injected via jq into Configs/DiscordLink.eco rather than argv. Mirror that pattern for the user token.

_Originally filed by @coilysiren on 2026-05-23T19:19:13Z - [https://github.com/coilysiren/infrastructure/issues/297](https://github.com/coilysiren/infrastructure/issues/297)_ **Problem** - The eco-server systemd unit invokes `EcoServer -userToken="$eco_server_api_token"` (see `scripts/eco-server-start.sh`), which puts the JWT from SSM `/eco/server-api-token` into `/proc/<pid>/cmdline`. Any `systemctl status eco-server`, `ps`, or journald scrape that captures the cmdline (including agent transcripts) leaks the token in cleartext. **Surfaced** - 2026-05-23, mid-incident chat transcript captured the JWT verbatim from `systemctl status` output during the LFS-pointer mod recovery (sibling issue: install-eco-mod-source.sh hardening). **Fix shape** - Pass the token via env var or stdin instead of argv. Eco supports `userToken` via config file (`Configs/Network.eco` -> `ServerAPIToken`) as well; writing it there at ExecStartPre time and dropping the argv flag is the cleanest path. Rotate `/eco/server-api-token` once the unit no longer exposes it. **Out of scope** - The DiscordLink token already gets injected via `jq` into `Configs/DiscordLink.eco` rather than argv. Mirror that pattern for the user token.
coilysiren added
P1
 and removed
P0
 labels 2026-06-17 08:40:35 +00:00
coilyco-ops commented 2026-07-04 17:25:58 +00:00
Member
Copy link

🔒 Reserved by ward agent --driver claude — container engineer-claude-infrastructure-2 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-04T17:25:53Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

The pre-flight judged this issue GO for an unattended run. Its justification:

pre-flight read (GO)

Context to front-load: In the fresh clone I will read, before any edit: (1) scripts/eco-server-start.sh — the current SSM fetch and -userToken argv wiring; (2) the eco-server systemd unit definition wherever the repo keeps it (service file / ansible / cloud-init) to see ExecStart/ExecStartPre structure; (3) the existing DiscordLink jq-into-Configs/DiscordLink.eco injection, since the issue says to mirror it exactly; (4) how/where Configs/Network.eco lives on the host relative to the script (path conventions, ownership/permissions on the written file); and (5) repo plumbing — .ward/ward.yaml verbs, pre-commit hooks, and the Forgejo-main push flow from AGENTS.md — so the land step doesn't stall.

Main risk: verification is indirect — the container can't run the real Eco server or systemd, so correctness rests on faithfully mirroring the proven DiscordLink jq pattern and confirming Network.eco is jq-editable JSON with the right ServerAPIToken key path; if the file turns out to be non-JSON or the key nesting differs from the issue's sketch, I'd have to infer it from repo fixtures or upstream docs. The token rotation step is operational (needs the live server/SSM), so I'd land the code fix and file a follow-up issue for rotation rather than block the merge on it. Everything else — the fix shape, the file to touch, the pattern to copy — is fully specified by the issue.

GO

— Claude (she/her), via ward agent

<!-- ward-agent-reservation --> 🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-infrastructure-2` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-04T17:25:53Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). The pre-flight judged this issue **GO** for an unattended run. Its justification: <details><summary>pre-flight read (GO)</summary> Context to front-load: In the fresh clone I will read, before any edit: (1) `scripts/eco-server-start.sh` — the current SSM fetch and `-userToken` argv wiring; (2) the eco-server systemd unit definition wherever the repo keeps it (service file / ansible / cloud-init) to see ExecStart/ExecStartPre structure; (3) the existing DiscordLink `jq`-into-`Configs/DiscordLink.eco` injection, since the issue says to mirror it exactly; (4) how/where `Configs/Network.eco` lives on the host relative to the script (path conventions, ownership/permissions on the written file); and (5) repo plumbing — `.ward/ward.yaml` verbs, pre-commit hooks, and the Forgejo-main push flow from AGENTS.md — so the land step doesn't stall. Main risk: verification is indirect — the container can't run the real Eco server or systemd, so correctness rests on faithfully mirroring the proven DiscordLink jq pattern and confirming `Network.eco` is jq-editable JSON with the right `ServerAPIToken` key path; if the file turns out to be non-JSON or the key nesting differs from the issue's sketch, I'd have to infer it from repo fixtures or upstream docs. The token rotation step is operational (needs the live server/SSM), so I'd land the code fix and file a follow-up issue for rotation rather than block the merge on it. Everything else — the fix shape, the file to touch, the pattern to copy — is fully specified by the issue. GO </details> <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`
coilyco-ops commented 2026-07-04 17:26:04 +00:00
Member
Copy link

🔓 Reservation released by ward container reap — container engineer-claude-infrastructure-2 (--driver claude) exited without launching the agent (smoke-test death, ward#222/#264), so the hold it took is retracted. A plain ward agent retry no longer needs --force.

— Claude (she/her), via ward agent

<!-- ward-agent-reservation-released --> 🔓 Reservation released by `ward container reap` — container `engineer-claude-infrastructure-2` (`--driver claude`) exited without launching the agent (smoke-test death, ward#222/#264), so the hold it took is retracted. A plain `ward agent` retry no longer needs `--force`. <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`
coilyco-ops commented 2026-07-04 17:27:13 +00:00
Member
Copy link

🔒 Reserved by ward agent --driver claude — container engineer-claude-infrastructure-2 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-04T17:27:08Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

The pre-flight judged this issue GO for an unattended run. Its justification:

pre-flight read (GO)

Context to front-load: (1) the eco-server systemd unit definition in the clone (ExecStart/ExecStartPre wiring and how the unit file is deployed — ansible, cloud-init, or checked-in unit), (2) scripts/eco-server-start.sh — how the SSM token is fetched and the argv is built today, (3) the existing DiscordLink jq-into-Configs/DiscordLink.eco injection, since the issue names it as the pattern to mirror, (4) the repo's SSM fetch conventions (instance role vs. coily ops aws) so /eco/server-api-token is read the house way, and (5) .ward/ward.yaml plus the pre-commit hook set so validation runs through ward. I will read each of these in the fresh clone before making my first edit.

The main unknown is Configs/Network.eco lifecycle: Eco may generate that file only on first server boot, so an ExecStartPre jq injection needs create-or-merge handling and the exact key path (ServerAPIToken) verified against the DiscordLink precedent rather than assumed — I can resolve that from the existing injection code and Eco config structure in the clone. The change itself is shell/unit-file editing with no live Eco server to smoke-test in the container, so verification will be shellcheck/inspection-level, which is acceptable for this shape of change. The token-rotation step is an operational action against SSM and the live server that shouldn't happen from an unattended container; I'd land the leak fix and file a follow-up issue for rotation, which matches the issue's "once the unit no longer exposes it" sequencing. The only comment is a reservation-release notice, so nothing supersedes the original description.

GO

— Claude (she/her), via ward agent

<!-- ward-agent-reservation --> 🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-infrastructure-2` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-04T17:27:08Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). The pre-flight judged this issue **GO** for an unattended run. Its justification: <details><summary>pre-flight read (GO)</summary> Context to front-load: (1) the eco-server systemd unit definition in the clone (ExecStart/ExecStartPre wiring and how the unit file is deployed — ansible, cloud-init, or checked-in unit), (2) `scripts/eco-server-start.sh` — how the SSM token is fetched and the argv is built today, (3) the existing DiscordLink `jq`-into-`Configs/DiscordLink.eco` injection, since the issue names it as the pattern to mirror, (4) the repo's SSM fetch conventions (instance role vs. `coily ops aws`) so `/eco/server-api-token` is read the house way, and (5) `.ward/ward.yaml` plus the pre-commit hook set so validation runs through ward. I will read each of these in the fresh clone before making my first edit. The main unknown is `Configs/Network.eco` lifecycle: Eco may generate that file only on first server boot, so an ExecStartPre `jq` injection needs create-or-merge handling and the exact key path (`ServerAPIToken`) verified against the DiscordLink precedent rather than assumed — I can resolve that from the existing injection code and Eco config structure in the clone. The change itself is shell/unit-file editing with no live Eco server to smoke-test in the container, so verification will be shellcheck/inspection-level, which is acceptable for this shape of change. The token-rotation step is an operational action against SSM and the live server that shouldn't happen from an unattended container; I'd land the leak fix and file a follow-up issue for rotation, which matches the issue's "once the unit no longer exposes it" sequencing. The only comment is a reservation-release notice, so nothing supersedes the original description. GO </details> <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`
coilysiren commented 2026-07-04 17:34:07 +00:00
Author
Owner
Copy link

WARD-OUTCOME: done - eco-server-start.sh now jq-injects the API token into Configs/Network.eco (ServerAPIToken) and launches EcoServer with no -userToken argv, so the JWT is off /proc//cmdline. Landed on main as bc5a48b.

The fix was smaller than the leak was scary: the DiscordLink BotToken injection sitting right above in the same script was an exact template, so this was really just "do that, for the other token, into the other config." I added a first-boot fallback that creates Network.eco with jq -n when the file isn't there yet, so a fresh Steam install doesn't dead-end on a missing config.

What actually fought back was the pre-commit suite, not the code. The doc-size cap (80 lines / 4000 chars) on docs/ops-eco.md was already near its ceiling, so my ops note kept blowing it. I moved the durable explanation into the eco-server-setup skill doc and left the script comment pointing there instead. Separately, pylint couldn't run at first because uv wanted to download a managed Python into a root-owned /opt/uv/python. Redirecting UV_PYTHON_INSTALL_DIR to a writable path let it pass 10.00/10 without --no-verify.

Confidence: high on the shell logic (it mirrors a pattern already running in prod, and .eco files are JSON that jq handles cleanly). One caveat I couldn't close from inside the container: no AWS creds here, so I couldn't verify the live CI run or confirm the exact ServerAPIToken key name against a running Network.eco - I trusted the issue's stated field name. Worth an eyeball on the next restart that the token lands and the server authenticates.

Follow-ups: (1) rotate /eco/server-api-token now that the old value rode argv and was captured in transcripts - I can't touch the play.eco account or SSM from here; (2) confirm the first restart picks up ServerAPIToken from the config and drops the argv path cleanly.

WARD-OUTCOME: done - eco-server-start.sh now jq-injects the API token into Configs/Network.eco (ServerAPIToken) and launches EcoServer with no -userToken argv, so the JWT is off /proc/<pid>/cmdline. Landed on main as bc5a48b. The fix was smaller than the leak was scary: the DiscordLink BotToken injection sitting right above in the same script was an exact template, so this was really just "do that, for the other token, into the other config." I added a first-boot fallback that creates Network.eco with `jq -n` when the file isn't there yet, so a fresh Steam install doesn't dead-end on a missing config. What actually fought back was the pre-commit suite, not the code. The doc-size cap (80 lines / 4000 chars) on docs/ops-eco.md was already near its ceiling, so my ops note kept blowing it. I moved the durable explanation into the eco-server-setup skill doc and left the script comment pointing there instead. Separately, pylint couldn't run at first because uv wanted to download a managed Python into a root-owned /opt/uv/python. Redirecting UV_PYTHON_INSTALL_DIR to a writable path let it pass 10.00/10 without --no-verify. Confidence: high on the shell logic (it mirrors a pattern already running in prod, and .eco files are JSON that jq handles cleanly). One caveat I couldn't close from inside the container: no AWS creds here, so I couldn't verify the live CI run or confirm the exact ServerAPIToken key name against a running Network.eco - I trusted the issue's stated field name. Worth an eyeball on the next restart that the token lands and the server authenticates. Follow-ups: (1) rotate /eco/server-api-token now that the old value rode argv and was captured in transcripts - I can't touch the play.eco account or SSM from here; (2) confirm the first restart picks up ServerAPIToken from the config and drops the argv path cleanly.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#2
No description provided.