coilyco-flight-deck/ward

Fork 0

Build the Eco content pipeline (eco-ops#30): ward eco test [Windows] + auto-promote to kai-server w/ snapshot+health+rollback #582

New issue

Closed

opened 2026-07-03 19:20:16 +00:00 by coilysiren · 4 comments

coilysiren commented

2026-07-03 19:20:16 +00:00

Owner

Build the Eco content pipeline specified in coilyco-gaming/eco-ops#30 (read it for the full design - do not re-derive it).

Summary

ward eco test (native Windows, against Kai's local Steam EcoServer install at a configured path) boots a throwaway world with the working-copy mod/config change and runs a headless smoke gate. On green it auto-promotes to the live Linux kai-server, wrapped in snapshot -> apply -> restart -> health-check + canary -> healthy|auto-rollback.

Blocking requirements (auto-promote is only safe with all three)

Real smoke gate, not a liveness ping: server reaches ready, every mod loaded with NO ModKit load exception (eco-ops#7), UserCode *.cs compiled. Non-zero exit blocks promote.
Snapshot -> health-check -> auto-rollback on the live side: snapshot Mods/+Configs/ before touching kai-server; after restart, probe (service up, journal clean of ModKit exceptions, server ready); on failure auto-restore the snapshot + restart and fail loud. The live world only ever ends green or reverted.
Canary watch window: hold and watch post-restart, roll back on degradation - not a single instant probe.

Placement + reuse

ward: the eco test verb + the auto-promote orchestration (Windows-native). Ties into the existing ward eco verb surface (ward eco mod push, restart).
infrastructure (granted repo): the kai-server-side snapshot/restart/health-check/rollback scripts, next to scripts/eco-server-{pre,start}.sh + the eco-server-setup skill. Read remote.py for the kai-server access pattern.
Reuse the existing promote pieces (install-eco-mod.sh, ward eco mod push, the systemd restart) - wrap them, do not rebuild.

Do NOT

Run a real auto-promote against the live kai-server as part of building/testing this. Unit-test the gate + rollback logic and dry-run the promote path; the first real run is Kai's.
Reintroduce any container/baked-image/Steam path - local is Windows-native (eco-ops#30 explains why that scoping is dead).

eco-ops#30 (the spec), eco-ops#7 (the gate's target failure), eco-ops#26 (health-check signals).

Build the Eco content pipeline specified in **coilyco-gaming/eco-ops#30** (read it for the full design - do not re-derive it). ## Summary `ward eco test` (native Windows, against Kai's local Steam EcoServer install at a configured path) boots a throwaway world with the working-copy mod/config change and runs a headless smoke gate. On green it **auto-promotes** to the live Linux kai-server, wrapped in snapshot -> apply -> restart -> health-check + canary -> healthy|auto-rollback. ## Blocking requirements (auto-promote is only safe with all three) 1. **Real smoke gate**, not a liveness ping: server reaches ready, every mod loaded with NO ModKit load exception (eco-ops#7), UserCode *.cs compiled. Non-zero exit blocks promote. 2. **Snapshot -> health-check -> auto-rollback** on the live side: snapshot Mods/+Configs/ before touching kai-server; after restart, probe (service up, journal clean of ModKit exceptions, server ready); on failure auto-restore the snapshot + restart and fail loud. The live world only ever ends green or reverted. 3. **Canary watch window**: hold and watch post-restart, roll back on degradation - not a single instant probe. ## Placement + reuse - **ward**: the `eco test` verb + the auto-promote orchestration (Windows-native). Ties into the existing `ward eco` verb surface (`ward eco mod push`, restart). - **infrastructure** (granted repo): the kai-server-side snapshot/restart/health-check/rollback scripts, next to `scripts/eco-server-{pre,start}.sh` + the eco-server-setup skill. Read remote.py for the kai-server access pattern. - Reuse the existing promote pieces (`install-eco-mod.sh`, `ward eco mod push`, the systemd restart) - wrap them, do not rebuild. ## Do NOT - Run a real auto-promote against the live kai-server as part of building/testing this. Unit-test the gate + rollback logic and dry-run the promote path; the first real run is Kai's. - Reintroduce any container/baked-image/Steam path - local is Windows-native (eco-ops#30 explains why that scoping is dead). ## Related eco-ops#30 (the spec), eco-ops#7 (the gate's target failure), eco-ops#26 (health-check signals).

coilyco-ops commented

2026-07-03 19:20:35 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-ward-582 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T19:20:30Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-ward-582` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T19:20:30Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-03 19:47:45 +00:00

feat(eco): ward eco test - fail-closed smoke gate + guarded auto-promote (closes #582)

coilysiren closed this issue

2026-07-03 19:47:45 +00:00

coilysiren commented

2026-07-03 19:48:37 +00:00

Author

Owner

WARD-OUTCOME: done - ward eco test (fail-closed smoke gate + opt-in guarded auto-promote) landed on ward main, with the kai-server snapshot/health-check/rollback scripts landed on infrastructure main.

Honest retrospective: the shape of this was clearer than the substrate. The safety-critical core - the smoke gate verdict and the snapshot->apply->restart->health->canary->rollback state machine - fell out naturally as pure functions behind an EcoExecutor seam, so the ordering guarantees (snapshot before any mutation, roll back on the first canary degradation, fail loud only when a rollback itself doesn't recover) are all unit-tested with a fake executor. That part I'm confident in.

What fought back: I couldn't reach coilyco-gaming/eco-ops#30 (private/unreachable from here), so I built strictly from the frozen issue body rather than the full spec - the design maps cleanly, but a second pass against the real eco-ops#30 wording is worth a glance. Two honest rough edges to flag as follow-ups: (1) the EcoServer ready/ModKit-exception/compile-fail log markers are best-guess defaults (tunable, kept in step between the Go gate and the health-check script) - they need one real boot log to confirm; and (2) the local Windows boot + the ssh apply-step (currently reuses install-eco-mod.sh per mod, which pulls a released asset rather than the raw working copy) are the two seams no test here could exercise. Per the issue, no real promote ran - the path is dry-runnable and the first live run is yours.

The ward eco surface didn't previously exist here (it lived in coily), so this is the first eco verb in ward - scoped tight to test on purpose.

WARD-OUTCOME: done - `ward eco test` (fail-closed smoke gate + opt-in guarded auto-promote) landed on ward `main`, with the kai-server snapshot/health-check/rollback scripts landed on infrastructure `main`. Honest retrospective: the shape of this was clearer than the substrate. The safety-critical core - the smoke gate verdict and the snapshot->apply->restart->health->canary->rollback state machine - fell out naturally as pure functions behind an `EcoExecutor` seam, so the ordering guarantees (snapshot before any mutation, roll back on the *first* canary degradation, fail loud only when a rollback itself doesn't recover) are all unit-tested with a fake executor. That part I'm confident in. What fought back: I couldn't reach coilyco-gaming/eco-ops#30 (private/unreachable from here), so I built strictly from the frozen issue body rather than the full spec - the design maps cleanly, but a second pass against the real eco-ops#30 wording is worth a glance. Two honest rough edges to flag as follow-ups: (1) the EcoServer *ready*/ModKit-exception/compile-fail log markers are best-guess defaults (tunable, kept in step between the Go gate and the health-check script) - they need one real boot log to confirm; and (2) the local Windows boot + the ssh apply-step (currently reuses `install-eco-mod.sh` per mod, which pulls a *released* asset rather than the raw working copy) are the two seams no test here could exercise. Per the issue, no real promote ran - the path is dry-runnable and the first live run is yours. The `ward eco` surface didn't previously exist here (it lived in coily), so this is the first `eco` verb in ward - scoped tight to `test` on purpose.

coilysiren reopened this issue

2026-07-03 19:49:06 +00:00

coilysiren commented

2026-07-03 19:49:06 +00:00

Author

Owner

⚠️ Reopened: a granted `--repo` push did not land

This run held --repo grants and closed against coilyco-flight-deck/ward, but the reaper could not confirm every granted repo's work reached its main. A secondary push can be silently rejected (a non-fast-forward on a busy main, a dead/rotated PAT) while the primary push succeeds, so the issue is reopened rather than left reading "done" with the cross-repo half lost.

`coilyco-flight-deck/infrastructure`

Verdict: 1 local commit(s) never reached origin/main.
Preserved on: ward-salvage/infrastructure-26e284ba

git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure.git ward-salvage/infrastructure-26e284ba
git checkout -b ward-salvage/infrastructure-26e284ba FETCH_HEAD

Re-run the cross-repo half, or - per ward#291 - file it as a native issue in the granted repo so it becomes a single-repo run that sidesteps this failure mode.

— Claude (she/her), via ward agent

## ⚠️ Reopened: a granted `--repo` push did not land This run held `--repo` grants and closed against `coilyco-flight-deck/ward`, but the reaper could not confirm every granted repo's work reached its `main`. A secondary push can be silently rejected (a non-fast-forward on a busy `main`, a dead/rotated PAT) while the primary push succeeds, so the issue is **reopened** rather than left reading "done" with the cross-repo half lost. ### `coilyco-flight-deck/infrastructure` - **Verdict:** 1 local commit(s) never reached `origin/main`. - **Preserved on:** `ward-salvage/infrastructure-26e284ba` ```bash git fetch https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure.git ward-salvage/infrastructure-26e284ba git checkout -b ward-salvage/infrastructure-26e284ba FETCH_HEAD ``` Re-run the cross-repo half, or - per ward#291 - file it as a native issue in the granted repo so it becomes a single-repo run that sidesteps this failure mode.  — Claude (she/her), via `ward agent`

coilysiren commented

2026-07-03 19:51:21 +00:00

Author

Owner

Verified landed - same false positive as ward#578 (fixed by ward#583)

The infrastructure (granted) work landed on infra main:

infra main HEAD cc8973e97d88 ("Merge ... issue-582"), preceded by 6a6a97531bad "feat(eco): kai-server snapshot/health-check/rollback scripts".
New scripts present on infra main: eco-server-{snapshot,health-check,rollback}.sh.
Salvage branch ward-salvage/infrastructure-26e284ba HEAD == infra main HEAD (cc8973e97d88).

Same merge-commit SHA-mismatch false negative as ward#578 - root-caused and fixed in ward#583, which landed just after this run's reap, so this run still hit the old check. Re-closing.

Follow-ups from the run's own retro (real, tracked):

Smoke-gate log markers are best-guess - the ready / ModKit-exception / compile-fail patterns need one real EcoServer boot log to confirm. Auto-promote should not be trusted until these are validated against a real boot (a false-pass here auto-ships a broken mod - the exact failure the gate exists to stop).
Apply-step ships a released asset, not the tested working copy - breaks test->promote integrity. Filed as ward#584.
Built from the ward#582 body (cross-org eco-ops#30 was unreachable from the run) - a pass against the full eco-ops#30 wording is worth a glance.

## Verified landed - same false positive as ward#578 (fixed by ward#583) The infrastructure (granted) work landed on infra `main`: - infra main HEAD `cc8973e97d88` ("Merge ... issue-582"), preceded by `6a6a97531bad` "feat(eco): kai-server snapshot/health-check/rollback scripts". - New scripts present on infra main: `eco-server-{snapshot,health-check,rollback}.sh`. - Salvage branch `ward-salvage/infrastructure-26e284ba` HEAD == infra main HEAD (`cc8973e97d88`). Same merge-commit SHA-mismatch false negative as ward#578 - root-caused and fixed in **ward#583**, which landed just after this run's reap, so this run still hit the old check. Re-closing. Follow-ups from the run's own retro (real, tracked): 1. **Smoke-gate log markers are best-guess** - the *ready* / ModKit-exception / compile-fail patterns need one real EcoServer boot log to confirm. **Auto-promote should not be trusted until these are validated against a real boot** (a false-pass here auto-ships a broken mod - the exact failure the gate exists to stop). 2. **Apply-step ships a released asset, not the tested working copy** - breaks test->promote integrity. Filed as **ward#584**. 3. Built from the ward#582 body (cross-org eco-ops#30 was unreachable from the run) - a pass against the full eco-ops#30 wording is worth a glance.

coilysiren closed this issue

2026-07-03 19:51:21 +00:00