P1: prod deploy frozen since #60 — publish gates on the always-failing mods job (app image has no mods). Decouple + fix the mods gate #68

New issue

Closed

opened 2026-07-05 06:47:05 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-07-05 06:47:05 +00:00

Owner

Severity: production deploys frozen. Fix ASAP.

The live app (eco-app.coilysiren.me) is up and serving, but stale - frozen on the pre-#60 image. Confirmed live via /mcp/ tools/list: it registers get_eco_trades (#6), get_eco_stores (#50), get_eco_currency (#53) but is missing everything that landed after #60 - get_eco_civics (#61), get_eco_world (#62), get_eco_social (#63), get_eco_progression (#64), the logistics/watchers tools (#51/#52), plus the deepen-existing wave. The corresponding /preview/*.json for those routes 404.

Root cause

.forgejo/workflows/build-publish.yml: the publish job has needs: [test, frontend, mods]. The mods job (added by #60 to compile the four C# mods) has failed on every run since it was added - forgejo actions tasks show runs 48-57 all failure/cancelled, never success. So publish never runs, no image is pushed to the in-cluster registry, and the CD reconciler (deploy#43) has no new image to roll. Prod froze the moment #60 landed.

The coupling is wrong

Per the workflow's own comment, the mods are built by the build-mod-* verbs and shipped to the game server out of band - they are NOT in the eco-app image. The image is SPA + MCP + jobs (Python/Node). So gating the app image publish on C# mod compilation is incorrect: a mod that does not compile has zero bearing on whether the app image is publishable. #60's gate was meant to catch mod typos, but it was wired into the wrong path and took production down with it.

Fix

1. Unstick prod now (the P0 change): in build-publish.yml, change the publish job to needs: [test, frontend] - drop mods. The app image is still fully validated (pytest/ruff/mypy/smoke via test, plus frontend); it just no longer waits on mod compilation it does not contain. This lets the next push publish and the CD reconciler roll prod forward.

2. Fix the red mods job (same run): it has never passed - investigate and repair so the mod-compile signal is restored. Likely suspects: the self-installed .NET 10 SDK step, Eco.ReferenceAssemblies restore (the pin was bumped 998 -> 1024 by #66 - confirm 1024 actually restores from the feed and all four mods, including stores #55, compile against it), or ICU/invariant-globalization. Keep mods as a required status check / signal for the out-of-band mod-shipping step, but never back in publish's needs.

Verify (before closing)

After the change lands, a build-publish run reaches the publish job and pushes a new image.
The CD path actually rolls it - confirm deploy#43's reconciler (or a manual rollout.sh eco-app) brings the new image live; do not assume publish == deployed.
Live /mcp/ tools/list gains get_eco_civics / get_eco_world / get_eco_social / get_eco_progression and the logistics/watchers tools, and their /preview/*.json return 200.
The mods job is green again (part 2).

Also worth noting (do not block on)

eco-mcp.coilysiren.me returns 503 "no available server" - that is the deprecated standalone (#41), backend deprovisioned. Separate cleanup (decommission the hostname or redirect it to eco-app.coilysiren.me so stale connectors do not 503). Not part of this fix.

Refs

Introduced by #60 (mod-compile gate). Pin bump: #66. CI parity: #57. CD: deploy#43. Deprecated host: #41. Diagnosed live from the director session, 2026-07-05.

## Severity: production deploys frozen. Fix ASAP. The live app (`eco-app.coilysiren.me`) is up and serving, but **stale** - frozen on the pre-#60 image. Confirmed live via `/mcp/` `tools/list`: it registers `get_eco_trades` (#6), `get_eco_stores` (#50), `get_eco_currency` (#53) but is **missing** everything that landed after #60 - `get_eco_civics` (#61), `get_eco_world` (#62), `get_eco_social` (#63), `get_eco_progression` (#64), the logistics/watchers tools (#51/#52), plus the deepen-existing wave. The corresponding `/preview/*.json` for those routes 404. ## Root cause `.forgejo/workflows/build-publish.yml`: the `publish` job has **`needs: [test, frontend, mods]`**. The `mods` job (added by #60 to compile the four C# mods) has **failed on every run since it was added** - forgejo actions tasks show runs 48-57 all `failure`/`cancelled`, never `success`. So `publish` never runs, no image is pushed to the in-cluster registry, and the CD reconciler (deploy#43) has no new image to roll. Prod froze the moment #60 landed. ## The coupling is wrong Per the workflow's own comment, the mods are built by the `build-mod-*` verbs and **shipped to the game server out of band - they are NOT in the eco-app image.** The image is SPA + MCP + jobs (Python/Node). So gating the **app image publish** on **C# mod compilation** is incorrect: a mod that does not compile has zero bearing on whether the app image is publishable. #60's gate was meant to catch mod typos, but it was wired into the wrong path and took production down with it. ## Fix **1. Unstick prod now (the P0 change):** in `build-publish.yml`, change the `publish` job to **`needs: [test, frontend]`** - drop `mods`. The app image is still fully validated (pytest/ruff/mypy/smoke via `test`, plus `frontend`); it just no longer waits on mod compilation it does not contain. This lets the next push publish and the CD reconciler roll prod forward. **2. Fix the red `mods` job (same run):** it has never passed - investigate and repair so the mod-compile signal is restored. Likely suspects: the self-installed .NET 10 SDK step, `Eco.ReferenceAssemblies` restore (the pin was bumped 998 -> 1024 by #66 - confirm 1024 actually restores from the feed and all four mods, including `stores` #55, compile against it), or ICU/invariant-globalization. Keep `mods` as a **required status check / signal** for the out-of-band mod-shipping step, but **never** back in `publish`'s `needs`. ## Verify (before closing) * After the change lands, a `build-publish` run reaches the `publish` job and pushes a new image. * The CD path actually rolls it - confirm deploy#43's reconciler (or a manual `rollout.sh eco-app`) brings the new image live; do not assume publish == deployed. * Live `/mcp/` `tools/list` gains `get_eco_civics` / `get_eco_world` / `get_eco_social` / `get_eco_progression` and the logistics/watchers tools, and their `/preview/*.json` return 200. * The `mods` job is green again (part 2). ## Also worth noting (do not block on) `eco-mcp.coilysiren.me` returns 503 "no available server" - that is the **deprecated** standalone (#41), backend deprovisioned. Separate cleanup (decommission the hostname or redirect it to `eco-app.coilysiren.me` so stale connectors do not 503). Not part of this fix. ## Refs Introduced by #60 (mod-compile gate). Pin bump: #66. CI parity: #57. CD: deploy#43. Deprecated host: #41. Diagnosed live from the director session, 2026-07-05.

coilyco-ops commented

2026-07-05 06:47:16 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-eco-app-68 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-05T06:47:11Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-eco-app-68` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-05T06:47:11Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-05 06:57:54 +00:00

fix(ci): decouple app-image publish from the mods compile gate (#68)

coilysiren closed this issue

2026-07-05 06:57:54 +00:00

coilysiren referenced this issue

2026-07-05 07:13:36 +00:00

Fix the red mods CI compile gate (part 2 of #68: decoupled but never repaired) #69

coilysiren referenced this issue from a commit

2026-07-05 07:33:19 +00:00

fix(ci): harden the mods compile gate against the runner's flaky NuGet egress (#69)

coilyco-ops referenced this issue