Restore the snapshot Ward workflow on current operator surfaces

coilyco-ops commented

2026-08-01 21:47:10 +00:00

Member

Problem

A macOS snapshot refresh on 2026-08-01 exposed three failures in the documented workflow:

ward exec snapshot-pull exits because BSD realpath does not support GNU -m.
snapshot-capture and snapshot-push still invoke the retired ward-kdl ops aws spelling. Capture therefore used Ward help text as the API token and produced zero responses until the token was injected through AOSGuard. Push cannot be trusted through the documented verb for the same reason.
A valid cycle 14 day 28 capture returned 547 responses with 138 recorded failures. Nearly all were catalog-advertised growth/seed raster paths returning 404. Replay pagination also returned an invalid events payload.

No invalid or zero-response archive was uploaded. The valid archive was replayed locally and uploaded through the guarded current AWS surface.

Acceptance

Snapshot AWS and SSM calls use the current guarded operator surface without exposing secret values.
Pull resolves and validates its destination portably on macOS and Linux.
Capture and push fail closed when the credential command returns non-secret help or error text.
A capture, timestamped push, latest update, pull, and local replay round trip succeeds on macOS.
Expected unavailable rasters and the current replay payload are either captured correctly or reported in an actionable bounded failure summary.

## Problem A macOS snapshot refresh on 2026-08-01 exposed three failures in the documented workflow: * `ward exec snapshot-pull` exits because BSD `realpath` does not support GNU `-m`. * `snapshot-capture` and `snapshot-push` still invoke the retired `ward-kdl ops aws` spelling. Capture therefore used Ward help text as the API token and produced zero responses until the token was injected through AOSGuard. Push cannot be trusted through the documented verb for the same reason. * A valid cycle 14 day 28 capture returned 547 responses with 138 recorded failures. Nearly all were catalog-advertised growth/seed raster paths returning 404. Replay pagination also returned an invalid events payload. No invalid or zero-response archive was uploaded. The valid archive was replayed locally and uploaded through the guarded current AWS surface. ## Acceptance * Snapshot AWS and SSM calls use the current guarded operator surface without exposing secret values. * Pull resolves and validates its destination portably on macOS and Linux. * Capture and push fail closed when the credential command returns non-secret help or error text. * A capture, timestamped push, latest update, pull, and local replay round trip succeeds on macOS. * Expected unavailable rasters and the current replay payload are either captured correctly or reported in an actionable bounded failure summary.

coilyco-ops added the

priority

P2

label

2026-08-01 22:42:06 +00:00

coilyco-ops commented

2026-08-04 22:32:20 +00:00

Author

Member

Live ops audit evidence from 2026-08-04 22:21-22:30 UTC:

The /replay SPA document returned 200 and rendered its shell.
GET /replay/api/v1/events returned 503.
GET /replay/api/v1/events/stats returned 503.
Both failures were captured as eco-app traces in SigNoz. No private replay payloads were inspected or copied.

This confirms the current replay payload problem is also visible through the deployed browser route, not only the snapshot workflow.

Live ops audit evidence from 2026-08-04 22:21-22:30 UTC: * The `/replay` SPA document returned 200 and rendered its shell. * `GET /replay/api/v1/events` returned 503. * `GET /replay/api/v1/events/stats` returned 503. * Both failures were captured as eco-app traces in SigNoz. No private replay payloads were inspected or copied. This confirms the current replay payload problem is also visible through the deployed browser route, not only the snapshot workflow.

coilyco-ops commented

2026-08-04 23:04:42 +00:00

Author

Member

Replay deployment status from the production verification:

The EcoReplay source contract and package build are healthy. CI published the immutable package from 0239532af8eaf15059a16e3be18dfbaee0d11f66.
EcoReplay is not installed on the native Eco server. Both expected replay paths currently return the Eco World HTML shell instead of JSON.
eco-app correctly rejects that malformed upstream response and returns a structured 503. The Replay page shell itself returns 200.
The current guarded ops surface exposes no Eco mod inventory, install, rollback, or service-restart action. Its Tailscale surface intentionally denies SSH, and the read-only node surface cannot inspect the mod installation.
I opened coilyco-flight-deck/infrastructure#749 to add the missing guarded install and restart path, then install and verify this published package.

I am leaving this issue open because its replay data acceptance criteria are not live, and it also retains the separate snapshot work.

Replay deployment status from the production verification: * The EcoReplay source contract and package build are healthy. CI published the immutable package from `0239532af8eaf15059a16e3be18dfbaee0d11f66`. * EcoReplay is not installed on the native Eco server. Both expected replay paths currently return the Eco World HTML shell instead of JSON. * eco-app correctly rejects that malformed upstream response and returns a structured 503. The Replay page shell itself returns 200. * The current guarded ops surface exposes no Eco mod inventory, install, rollback, or service-restart action. Its Tailscale surface intentionally denies SSH, and the read-only node surface cannot inspect the mod installation. * I opened coilyco-flight-deck/infrastructure#749 to add the missing guarded install and restart path, then install and verify this published package. I am leaving this issue open because its replay data acceptance criteria are not live, and it also retains the separate snapshot work.

coilyco-ops commented

2026-08-04 23:19:20 +00:00

Author

Member

Rendered-browser verification confirms the replay deployment diagnosis:

The React Replay page rendered successfully for three query variants.
Each render produced exactly two failed data requests: the events endpoint and the stats endpoint, both HTTP 503.
SigNoz contains all three page requests and all six 503 data requests as top-level server spans.
No health-check spans were emitted during the browser audit.
The same ten-minute trace window contains 64 distinct HTTP targets. Replay is the only current non-2xx application path.

The missing native EcoReplay installation remains tracked by coilyco-flight-deck/infrastructure#749.

Rendered-browser verification confirms the replay deployment diagnosis: * The React Replay page rendered successfully for three query variants. * Each render produced exactly two failed data requests: the events endpoint and the stats endpoint, both HTTP 503. * SigNoz contains all three page requests and all six 503 data requests as top-level server spans. * No health-check spans were emitted during the browser audit. * The same ten-minute trace window contains 64 distinct HTTP targets. Replay is the only current non-2xx application path. The missing native EcoReplay installation remains tracked by coilyco-flight-deck/infrastructure#749.

coilyco-ops added

and removed

labels

2026-08-05 10:26:09 +00:00

coilyco-ops added the

role/engineer