Surface OOMKilled state when director surface Codex dies by SIGKILL #883

New issue

Closed

opened 2026-07-09 23:09:58 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-09 23:09:58 +00:00

Member

Observed while investigating director-codex-vg94 on 2026-07-09.

Facts:

Docker events showed container oom at 2026-07-09T15:59:24-07:00.
docker inspect director-codex-vg94 --format {{json .State}} showed OOMKilled=true and ExitCode=0.
The container had no explicit memory cap (HostConfig.Memory=0), so this points at Docker Desktop VM memory pressure rather than ward setting a per-run cap.
The operator-visible line only said agent exited non-zero (signal: killed); reaping anyway, then the read-only reaper returned cleanly with nothing to salvage.
No drained ~/.ward/agent-logs/director-codex-vg94/ directory existed for the interactive director surface, so the clearest OOM breadcrumb remained in Docker state and events.

Requested change:

When host-side surface or drain code has Docker state available, inspect .State.OOMKilled and include that fact in operator-visible output or meta.json.
For interactive director/read-only surfaces, make the final death diagnostic distinguish an OOM kill from a generic signal: killed when host state is available.
Add a troubleshooting entry for this symptom.

Acceptance criteria:

A director Codex child killed by OOM leaves a clear OOMKilled=true breadcrumb in host output or drained metadata.
The UX no longer reads like a normal read-only reaper success when the child was killed by the Docker runtime.

Observed while investigating `director-codex-vg94` on 2026-07-09. Facts: * Docker events showed `container oom` at `2026-07-09T15:59:24-07:00`. * `docker inspect director-codex-vg94 --format {{json .State}}` showed `OOMKilled=true` and `ExitCode=0`. * The container had no explicit memory cap (`HostConfig.Memory=0`), so this points at Docker Desktop VM memory pressure rather than ward setting a per-run cap. * The operator-visible line only said `agent exited non-zero (signal: killed); reaping anyway`, then the read-only reaper returned cleanly with `nothing to salvage`. * No drained `~/.ward/agent-logs/director-codex-vg94/` directory existed for the interactive director surface, so the clearest OOM breadcrumb remained in Docker state and events. Requested change: * When host-side surface or drain code has Docker state available, inspect `.State.OOMKilled` and include that fact in operator-visible output or `meta.json`. * For interactive director/read-only surfaces, make the final death diagnostic distinguish an OOM kill from a generic `signal: killed` when host state is available. * Add a troubleshooting entry for this symptom. Acceptance criteria: * A director Codex child killed by OOM leaves a clear `OOMKilled=true` breadcrumb in host output or drained metadata. * The UX no longer reads like a normal read-only reaper success when the child was killed by the Docker runtime.

coilyco-ops added the

headless

labels

2026-07-09 23:09:58 +00:00

coilyco-ops commented

2026-07-10 04:46:31 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-883 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T04:46:30Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#883 · branch issue-883 · harness codex · workflow pull-requests
Run: engineer-codex-ward-883 · ward v0.555.0 · dispatched 2026-07-10T04:46:30Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-883` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T04:46:30Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#883` · branch `issue-883` · harness `codex` · workflow `pull-requests` - **Run:** `engineer-codex-ward-883` · ward `v0.555.0` · dispatched `2026-07-10T04:46:30Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.555.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 04:59:18 +00:00

Surface OOMKilled state when director surface Codex dies by SIGKILL #946

coilyco-ops commented

2026-07-10 05:00:11 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-request; review summary: review gate skipped by ~/.ward/config.yaml default
summary: surfaced Docker OOMKilled state in the surface death path and drained it into meta.json.
felt: straightforward code, but the container disk exhaustion made validation noisy.
confidence: medium.
surprises: local hook and Go test temp space both hit overlay limits before the PR check went green.
follow-ups: none.

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-request; review summary: review gate skipped by ~/.ward/config.yaml default summary: surfaced Docker OOMKilled state in the surface death path and drained it into meta.json. felt: straightforward code, but the container disk exhaustion made validation noisy. confidence: medium. surprises: local hook and Go test temp space both hit overlay limits before the PR check went green. follow-ups: none. </details>

coilyco-ops closed this issue

2026-07-10 05:09:01 +00:00