coilyco-flight-deck/ward

Archived

Watch

warded --aws injects no working AWS/SSM creds on the dispatch host - SSM provision scripts cannot self-serve #579

Closed

opened 2026-07-03 18:44:13 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-07-03 18:44:13 +00:00

Owner

Symptom

infra#456 dispatched a warded engineer ... --aws to run provision-scoop-write-token.sh (an SSM-stashing rotation). The run found no working AWS creds in the container: ward ops aws sts get-caller-identity -> NoCredentials, no ~/.aws/config|credentials, no IMDS, no AWS_CONTAINER_CREDENTIALS_*, no access-key env. So --aws on a warded engineer did not deliver SSM access on the dispatch host that ran it.

Why it matters

The token-minting skill (and the "rotate these yourself" callout just landed in infra#454) tell agents to rotate FJ CI/tap/flux/scoop tokens themselves from "a box with AWS/SSM (--aws on a warded engineer, or a creds-having host)." This finding shows the --aws half of that promise does not hold on the current dispatch host - so no SSM-touching provision script can self-serve there. The self-serve rotation story has a hole.

Root cause (to determine)

Either (a) --aws mounts the host ~/.aws but the dispatch host has none configured, or (b) the broker-forwarded dispatch does not propagate the mount. The engineer's probes (no ~/.aws, no IMDS, no env) point at the host simply having no AWS identity.

Fix direction

Either provision working AWS/SSM creds on the dispatch host so --aws actually injects them, or correct the docs to say SSM rotation runs only on a creds-having host (Kai's Mac, per remote.py "assumes keys") and --aws is a no-op where the host has no AWS identity. Whichever is intended, the self-serve prose in the token-minting skill should match reality.

infra#456 (blocked scoop rotation), infra#454 (the self-serve callout), ward#576 (the secrets it blocks).

## Symptom infra#456 dispatched a `warded engineer ... --aws` to run `provision-scoop-write-token.sh` (an SSM-stashing rotation). The run found **no working AWS creds in the container**: `ward ops aws sts get-caller-identity` -> `NoCredentials`, no `~/.aws/config|credentials`, no IMDS, no `AWS_CONTAINER_CREDENTIALS_*`, no access-key env. So `--aws` on a warded engineer did **not** deliver SSM access on the dispatch host that ran it. ## Why it matters The `token-minting` skill (and the "rotate these yourself" callout just landed in infra#454) tell agents to rotate FJ CI/tap/flux/scoop tokens themselves from "a box with AWS/SSM (`--aws` on a warded engineer, or a creds-having host)." This finding shows the `--aws` half of that promise does not hold on the current dispatch host - so **no SSM-touching provision script can self-serve** there. The self-serve rotation story has a hole. ## Root cause (to determine) Either (a) `--aws` mounts the host `~/.aws` but the dispatch host has none configured, or (b) the broker-forwarded dispatch does not propagate the mount. The engineer's probes (no `~/.aws`, no IMDS, no env) point at the host simply having no AWS identity. ## Fix direction Either provision working AWS/SSM creds on the dispatch host so `--aws` actually injects them, **or** correct the docs to say SSM rotation runs only on a creds-having host (Kai's Mac, per `remote.py` "assumes keys") and `--aws` is a no-op where the host has no AWS identity. Whichever is intended, the self-serve prose in the token-minting skill should match reality. ## Related infra#456 (blocked scoop rotation), infra#454 (the self-serve callout), ward#576 (the secrets it blocks).

coilysiren commented

2026-07-03 19:42:53 +00:00

Author

Owner

Resolution direction: host creds, not a skill reword. Kai is provisioning real AWS creds on the dispatch hosts (KAI-DESKTOP-TOWER, KAI-SERVER) so --aws actually injects SSM into warded engineer containers. The creds need to sit in each host's ~/.aws (what --aws mounts read-only) with SSM read+write.

So the token-minting skill's --aws-on-a-warded-engineer line should not be walked back - it is becoming true. A separate narrow fix corrects only the orthogonal CI_RELEASE_TOKEN org-secret mis-grouping (a different, admin-PAT issue), leaving the --aws line intact.

Once the host creds land: re-fire infra#456 (scoop rotation) - it stalled on exactly this NoCredentials wall.

**Resolution direction: host creds, not a skill reword.** Kai is provisioning real AWS creds on the dispatch hosts (KAI-DESKTOP-TOWER, KAI-SERVER) so `--aws` actually injects SSM into warded engineer containers. The creds need to sit in each host's `~/.aws` (what `--aws` mounts read-only) with SSM read+write. So the token-minting skill's `--aws`-on-a-warded-engineer line should **not** be walked back - it is becoming true. A separate narrow fix corrects only the orthogonal CI_RELEASE_TOKEN org-secret mis-grouping (a different, admin-PAT issue), leaving the --aws line intact. Once the host creds land: re-fire infra#456 (scoop rotation) - it stalled on exactly this `NoCredentials` wall.

coilyco-ops commented

2026-07-03 20:13:47 +00:00

Owner

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-579 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T20:13:43Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-579` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T20:13:43Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-03 20:22:06 +00:00

fix(agent): warn loudly when --aws mount has no host creds (closes #579)

coilysiren closed this issue

2026-07-03 20:22:06 +00:00

coilysiren commented

2026-07-03 20:22:39 +00:00

Author

Owner

WARD-OUTCOME: done - ward now warns loudly at launch when the --aws capability is on but the dispatch host has no ~/.aws creds, instead of silently mounting an empty dir.

Honest retro: the fix itself was small, but the root cause was sneakier than the symptom read. The engineer probes said "no creds, no IMDS, no env" - which sounds like the mount is broken, but the mount was working exactly as coded. The real trap is docker: bind-mounting a missing source silently mints an empty dir, so a creds-less host produces a perfectly-mounted, perfectly-empty /root/.aws that reads as "delivered" right up until an ssm call face-plants on NoCredentials deep in a provision script. Classic silent-success hole.

Per Kai's call in the thread I did NOT walk back the token-minting promise - that stays "becoming true" via host-side creds. ward's job here was just to make the gap loud, which slotted cleanly next to the existing maybeWarnHostNet: same shape (capability requested, host can't deliver, warn don't fail). That precedent made me fairly confident the design is idiomatic rather than bolted-on.

What fought back: nothing in the logic, but the doc/comment size caps (4000-char doc, 2-line comment blocks, 90-char lines) took three trim passes to satisfy - the docs hook is strict. No test flakiness.

Confidence: high on the warning behavior and tests. The one thing I could NOT verify end-to-end is the actual host provisioning - that's Kai's side. Worth a follow-up once creds land on the dispatch hosts: re-run infra#456 (scoop rotation) to confirm --aws now self-serves, and consider whether the warning should also probe env/IMDS before firing (currently it only checks ~/.aws files, so a host carrying creds purely via AWS_* env would still get a false warning - deliberately conservative for now).

WARD-OUTCOME: done - ward now warns loudly at launch when the --aws capability is on but the dispatch host has no ~/.aws creds, instead of silently mounting an empty dir. Honest retro: the fix itself was small, but the root cause was sneakier than the symptom read. The engineer probes said "no creds, no IMDS, no env" - which sounds like the mount is broken, but the mount was working exactly as coded. The real trap is docker: bind-mounting a *missing* source silently mints an empty dir, so a creds-less host produces a perfectly-mounted, perfectly-empty /root/.aws that reads as "delivered" right up until an ssm call face-plants on NoCredentials deep in a provision script. Classic silent-success hole. Per Kai's call in the thread I did NOT walk back the token-minting promise - that stays "becoming true" via host-side creds. ward's job here was just to make the gap loud, which slotted cleanly next to the existing maybeWarnHostNet: same shape (capability requested, host can't deliver, warn don't fail). That precedent made me fairly confident the design is idiomatic rather than bolted-on. What fought back: nothing in the logic, but the doc/comment size caps (4000-char doc, 2-line comment blocks, 90-char lines) took three trim passes to satisfy - the docs hook is strict. No test flakiness. Confidence: high on the warning behavior and tests. The one thing I could NOT verify end-to-end is the actual host provisioning - that's Kai's side. Worth a follow-up once creds land on the dispatch hosts: re-run infra#456 (scoop rotation) to confirm --aws now self-serves, and consider whether the warning should also probe env/IMDS before firing (currently it only checks ~/.aws files, so a host carrying creds purely via AWS_* env would still get a false warning - deliberately conservative for now).

coilysiren referenced this issue

2026-07-03 20:34:34 +00:00

--aws: export-credentials on the host and inject AWS_* env, instead of only mounting ~/.aws #586

coilysiren referenced this issue from a commit

2026-07-03 20:51:48 +00:00

feat(agent): --aws exports the host credential chain and injects AWS_* env (closes #586)

coilysiren referenced this issue

2026-07-03 20:52:25 +00:00

--aws: export-credentials on the host and inject AWS_* env, instead of only mounting ~/.aws #586

coilyco-ops referenced this issue

2026-07-08 22:19:00 +00:00

Make agent workflow defaults configurable and default engineers to PRs #707