ward-launched containers run with no memory limit; default container create to --memory=2g #1171

New issue

Closed

opened 2026-07-13 04:15:16 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-13 04:15:16 +00:00

Member

Problem

ward launches agent containers with no memory limit, so each container's ceiling is the entire Docker VM. Observed on Kai's Windows host on 2026-07-12: six running containers (3 engineer, 3 director) each showed MEM LIMIT = 15.57GiB, the full WSL2 VM. On a 32 GB host also running a game client, the uncapped WSL VM ballooned to 16 GB committed and pushed the machine into paging.

Kai capped the running containers by hand as a stopgap:

docker update --memory=1280m --memory-swap=2560m on the engineer containers
docker update --memory=768m --memory-swap=1536m on the director containers

That sticks for those containers only. Every newly spawned ward container comes up uncapped again, so the fix does not survive normal fleet churn.

Ask

ward passes a default memory limit when it creates agent containers.

Default: --memory=2g, per Kai. Observed steady-state usage is 300-900 MiB per container, so 2 GB leaves real headroom without letting one container eat the VM.
Pair it with a memory-swap ceiling, docker rejects a memory limit without one when updating. 2x memory (--memory-swap=4g) is a reasonable default: a brief spike swaps instead of OOM-killing an agent mid-task.
The default lands in the fleet-tuning layer (ward-kdl dialect 2), since every ward user melds their own values.
A per-host override belongs in ~/.ward/config.yaml as operator-local preference, same placement argument as ward#1160's staging-dir override.

Done condition

A freshly dispatched engineer or director container shows a 2 GiB MEM LIMIT in docker stats without hand-run docker update, and the default is overridable per host.

Filed from Kai's desktop session while diagnosing host memory pressure.

## Problem ward launches agent containers with no memory limit, so each container's ceiling is the entire Docker VM. Observed on Kai's Windows host on 2026-07-12: six running containers (3 engineer, 3 director) each showed `MEM LIMIT = 15.57GiB`, the full WSL2 VM. On a 32 GB host also running a game client, the uncapped WSL VM ballooned to 16 GB committed and pushed the machine into paging. Kai capped the running containers by hand as a stopgap: * `docker update --memory=1280m --memory-swap=2560m` on the engineer containers * `docker update --memory=768m --memory-swap=1536m` on the director containers That sticks for those containers only. Every newly spawned ward container comes up uncapped again, so the fix does not survive normal fleet churn. ## Ask ward passes a default memory limit when it creates agent containers. * Default: `--memory=2g`, per Kai. Observed steady-state usage is 300-900 MiB per container, so 2 GB leaves real headroom without letting one container eat the VM. * Pair it with a memory-swap ceiling, docker rejects a memory limit without one when updating. 2x memory (`--memory-swap=4g`) is a reasonable default: a brief spike swaps instead of OOM-killing an agent mid-task. * The default lands in the fleet-tuning layer (ward-kdl dialect 2), since every ward user melds their own values. * A per-host override belongs in `~/.ward/config.yaml` as operator-local preference, same placement argument as ward#1160's staging-dir override. ## Done condition A freshly dispatched engineer or director container shows a 2 GiB `MEM LIMIT` in `docker stats` without hand-run `docker update`, and the default is overridable per host. Filed from Kai's desktop session while diagnosing host memory pressure.

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-14 00:37:55 +00:00

fix: default ward containers to 2g memory #1223

coilyco-ops commented

2026-07-14 00:39:28 +00:00

Author

Member

WARDED_WORKFLOW: #1223

details

workflow: pull-request-and-merge; review summary: director review passed in #1223 (comment); Forgejo CI passed.

The implementation felt contained once the launch path was traced. Confidence is high. The surprise was that the existing smart-defaults seam already carried the right host-local override shape, so the fix stayed inside the fleet-tuning layer and the container argv builder.

Follow-ups: none.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1223 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: director review passed in https://forgejo.coilysiren.me/coilyco-flight-deck/ward/pulls/1223#issuecomment-32592; Forgejo CI passed. The implementation felt contained once the launch path was traced. Confidence is high. The surprise was that the existing smart-defaults seam already carried the right host-local override shape, so the fix stayed inside the fleet-tuning layer and the container argv builder. Follow-ups: none. </details>

coilyco-ops closed this issue

2026-07-14 00:41:08 +00:00