Unstick deploy-factory-game Forgejo workflow queue

coilyco-ops commented

2026-07-10 09:46:03 +00:00

Member

Problem

The factory-game deploy repair in coilyco-bridge/deploy#130 landed deploy-side changes but reported WARD-OUTCOME: blocked because the Forgejo deploy workflow stayed queued and the public hostname still does not resolve.

Evidence from the director surface on 2026-07-10 UTC:

GET https://factory-game.coilysiren.me/
result: URLError [Errno -2] Name or service not known

Evidence from deploy#130's latest engineer:

Forgejo run 6276 stayed in waiting for several minutes after the push.
follow-ups: restore the deploy runner or rerun the workflow, then recheck factory-game DNS and HTTP 200

Do

Inspect the Forgejo Actions runner state for coilyco-bridge/deploy, especially the queued deploy-factory-game workflow run 6276.
Restore or unstick the runner path needed for deploy workflows.
Rerun or otherwise complete the factory-game deployment after the queued workflow can execute.
Preserve the deploy-runner placement constraints already tracked in infrastructure#507.
Post verification evidence back here and on coilyco-bridge/deploy#130.

Acceptance

The queued deploy-factory-game workflow runs or is superseded by a successful equivalent run.
DNS for factory-game.coilysiren.me resolves from a normal client.
https://factory-game.coilysiren.me/ returns HTTP 200 with the static app HTML.

Related: coilyco-bridge/deploy#130 and infrastructure#507.

Codex, via director surface

## Problem The factory-game deploy repair in coilyco-bridge/deploy#130 landed deploy-side changes but reported `WARD-OUTCOME: blocked` because the Forgejo deploy workflow stayed queued and the public hostname still does not resolve. Evidence from the director surface on 2026-07-10 UTC: ```text GET https://factory-game.coilysiren.me/ result: URLError [Errno -2] Name or service not known ``` Evidence from deploy#130's latest engineer: ```text Forgejo run 6276 stayed in waiting for several minutes after the push. follow-ups: restore the deploy runner or rerun the workflow, then recheck factory-game DNS and HTTP 200 ``` ## Do * Inspect the Forgejo Actions runner state for coilyco-bridge/deploy, especially the queued deploy-factory-game workflow run 6276. * Restore or unstick the runner path needed for deploy workflows. * Rerun or otherwise complete the factory-game deployment after the queued workflow can execute. * Preserve the deploy-runner placement constraints already tracked in infrastructure#507. * Post verification evidence back here and on coilyco-bridge/deploy#130. ## Acceptance * The queued deploy-factory-game workflow runs or is superseded by a successful equivalent run. * DNS for `factory-game.coilysiren.me` resolves from a normal client. * `https://factory-game.coilysiren.me/` returns HTTP 200 with the static app HTML. Related: coilyco-bridge/deploy#130 and infrastructure#507.  Codex, via director surface

coilyco-ops added the

headless

P2

labels

2026-07-10 09:46:03 +00:00

coilyco-ops commented

2026-07-10 09:46:06 +00:00

Author

Member

WARD-DISPATCH: deferred ⏸

deferred details

This forwarded dispatch was deferred after the issue was already reserved.

Attempted harness: codex
Attempted run: ward agent engineer coilyco-flight-deck/infrastructure#531 --harness codex --ward-version v0.573.0 --force
Container: engineer-codex-infrastructure-531
Container created: no running engineer was observed.
Host log: /Users/kai/.ward/agent-logs/dispatch/20260710T094603Z-director-codex-jq75-coilyco-flight-deck-infrastructure-531.log
Capacity: ward agent engineer --harness codex: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run ward agent reap for stale engineers

Retry: the issue stays queued and the director will try again when a slot opens.

— Codex, via ward agent

WARD-DISPATCH: deferred ⏸ <details><summary>deferred details</summary> This forwarded dispatch was deferred after the issue was already reserved. Attempted harness: `codex` Attempted run: `ward agent engineer coilyco-flight-deck/infrastructure#531 --harness codex --ward-version v0.573.0 --force` Container: `engineer-codex-infrastructure-531` Container created: no running engineer was observed. Host log: `/Users/kai/.ward/agent-logs/dispatch/20260710T094603Z-director-codex-jq75-coilyco-flight-deck-infrastructure-531.log` Capacity: `ward agent engineer --harness codex: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run `ward agent reap` for stale engineers` Retry: the issue stays queued and the director will try again when a slot opens. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 09:51:09 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-531 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T09:51:08Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#531 · branch issue-531 · harness codex · workflow direct-main
Run: engineer-codex-infrastructure-531 · ward v0.573.0 · dispatched 2026-07-10T09:51:08Z
Comment thread: 1 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T09:46:06Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-531` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T09:51:08Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#531` · branch `issue-531` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-531` · ward `v0.573.0` · dispatched `2026-07-10T09:51:08Z` - **Comment thread:** 1 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T09:46:06Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.573.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-10 10:04:08 +00:00

fix(cd): restart deploy runner on manifest change

coilyco-ops commented

2026-07-10 10:07:24 +00:00

Author

Member

Reopening after post-close live verification. The issue closed at 2026-07-10T10:04:08Z, but the dependent factory-game deployment is still not live:

GET https://factory-game.coilysiren.me/ -> URLError [Errno -2] Name or service not known.

The deploy-side issue coilyco-bridge/deploy#130 remains open and blocked until the workflow runner or rollout path is actually restored and the public DNS/HTTP check passes.

Reopening after post-close live verification. The issue closed at 2026-07-10T10:04:08Z, but the dependent factory-game deployment is still not live: - GET https://factory-game.coilysiren.me/ -> URLError [Errno -2] Name or service not known. The deploy-side issue coilyco-bridge/deploy#130 remains open and blocked until the workflow runner or rollout path is actually restored and the public DNS/HTTP check passes.

coilyco-ops commented

2026-07-10 10:07:29 +00:00

Author

Member

WARD-DISPATCH: deferred ⏸

deferred details

This forwarded dispatch was deferred after the issue was already reserved.

Attempted harness: codex
Attempted run: ward agent engineer coilyco-flight-deck/infrastructure#531 --harness codex --ward-version v0.573.0 --force
Container: engineer-codex-infrastructure-531
Container created: no running engineer was observed.
Host log: /Users/kai/.ward/agent-logs/dispatch/20260710T100725Z-director-codex-jq75-coilyco-flight-deck-infrastructure-531.log
Capacity: ward agent engineer --harness codex: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run ward agent reap for stale engineers

Retry: the issue stays queued and the director will try again when a slot opens.

— Codex, via ward agent

WARD-DISPATCH: deferred ⏸ <details><summary>deferred details</summary> This forwarded dispatch was deferred after the issue was already reserved. Attempted harness: `codex` Attempted run: `ward agent engineer coilyco-flight-deck/infrastructure#531 --harness codex --ward-version v0.573.0 --force` Container: `engineer-codex-infrastructure-531` Container created: no running engineer was observed. Host log: `/Users/kai/.ward/agent-logs/dispatch/20260710T100725Z-director-codex-jq75-coilyco-flight-deck-infrastructure-531.log` Capacity: `ward agent engineer --harness codex: global engineer limit is reached: 12 running (limit 12); wait for a run to finish or run `ward agent reap` for stale engineers` Retry: the issue stays queued and the director will try again when a slot opens. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 10:55:56 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-531 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T10:55:55Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#531 · branch issue-531 · harness codex · workflow direct-main
Run: engineer-codex-infrastructure-531 · ward v0.580.0 · dispatched 2026-07-10T10:55:55Z
Comment thread: 3 included in the pre-flight read, 1 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T09:46:06Z), @coilyco-ops (2026-07-10T10:07:24Z), @coilyco-ops (2026-07-10T10:07:29Z)
- stripped: @coilyco-ops (2026-07-10T09:51:09Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-531` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T10:55:55Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#531` · branch `issue-531` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-infrastructure-531` · ward `v0.580.0` · dispatched `2026-07-10T10:55:55Z` - **Comment thread:** 3 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T09:46:06Z), @coilyco-ops (2026-07-10T10:07:24Z), @coilyco-ops (2026-07-10T10:07:29Z) - stripped: @coilyco-ops (2026-07-10T09:51:09Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.580.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 11:24:38 +00:00

Author

Member

WARD-OUTCOME: blocked 🛑

details

workflow: direct-to-main; review summary: skipped (review gate skipped by ~/.ward/config.yaml default) retrospective: runner pin landed cleanly in repo, but live runner pickup and DNS never flipped from this container. confidence: low surprises: Forgejo dispatch worked, yet run 6438 stayed waiting and factory-game stayed NXDOMAIN. follow-ups: apply the manifest on the live cluster or restore runner availability, then recheck DNS and HTTP 200.

WARD-OUTCOME: blocked 🛑 <details><summary>details</summary> workflow: direct-to-main; review summary: skipped (review gate skipped by ~/.ward/config.yaml default) retrospective: runner pin landed cleanly in repo, but live runner pickup and DNS never flipped from this container. confidence: low surprises: Forgejo dispatch worked, yet run 6438 stayed waiting and factory-game stayed NXDOMAIN. follow-ups: apply the manifest on the live cluster or restore runner availability, then recheck DNS and HTTP 200. </details>

coilyco-ops referenced this issue

2026-07-10 16:58:42 +00:00

Post-reboot recovery incomplete: Forgejo Actions runners + DNS/tailnet degraded on kai-server, landing-blocking fleet burndown #538