coilyco-flight-deck/ward

Archived

Watch

Config-driven ward: per-harness model-config validation - fall back vs fail (#650-6, design fork) #658

Closed

opened 2026-07-08 03:33:07 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-08 03:33:07 +00:00

Owner

Soft-depends on #653 (the resolve seam gives the fleet.kdl that carries these model strings a live source). This is a design decision to settle in parallel.

Breakdown of the config-driven ward epic (#650). Raised by Kai reading the landed .ward/ward-kdl.fleet.kdl (aos#332). This is a design fork the from-scratch epic design did not cover - it needs a decision before build. Labeled consult.

The new failure mode live-resolve introduces

The fleet config carries per-agent and per-role model strings: gpt-5.4 / gpt-5.4-mini, qwen3-coder:30b, and the role overlays claude-fable-5, claude-opus-4-8[1m], gpt-5.5 (ward-kdl.fleet.kdl:52,66,91,104,108). Under the old baked config these shipped lockstep with the binary that understood them - a ward release and its model strings versioned together. Under live-resolve from a git ref, the bundle on main and the locally-installed harness binaries (claude / codex / opencode / goose) version independently. So a fleet.kdl on main can name a model the local harness rejects, and models move fast - renames, deprecations, new tiers land constantly.

This is distinct from the epic's source fail-loud (the "does the bundle resolve / parse" layer). Here the bundle resolves and parses cleanly - the problem is a value inside it is stale for the specific harness it targets.

Why detection must be per-harness

ward does not own the model namespaces. Each harness validates differently:

claude - --model claude-opus-4-8[1m], its own id set. ward#616 already makes model override-only (empty model "" = bare launch = the harness's own default).
codex - --model gpt-5.5, OpenAI id set, separate cadence.
opencode - --model qwen3-coder:30b against a local ollama endpoint (:67) - an invalid model is a 404 from ollama, a totally different surface from a CLI-flag rejection.
goose - ollama auth, no model field today.

A central ward-side model allowlist would stale immediately (the exact "moves fast" problem). Detection has to sit per-harness.

The fork to decide (do not build until picked)

Two axes, both Kai's call:

On an invalid/stale model value: fall back or fail?
- Fall back to the harness default (the empty-string bare-launch path claude already has) - resilient, the fleet keeps running when one model string goes stale, but it silently downgrades (a director expecting claude-opus-4-8[1m] quietly gets whatever the harness defaults to).
- Fail loud - honest, but one stale model string in a live-tracked bundle could brick the whole fleet.
- Likely role-sensitive: a director losing its strong model silently is worse than an engineer fan-out losing claude-fable-5. The answer may differ by role.
Where does detection live?
- Probe before launch (dry-run / capability query the harness) - clean, but adds a per-launch call and not every harness exposes a probe.
- Catch-at-launch - launch, catch the harness's own "unknown model" error, then retry-with-default or surface. Reuses each harness's own validation, no ward-side model knowledge, but couples to each harness's error shape.

docs/agent-config-overrides.md (overlay precedence, ward#616 / ward#620 / cli-guard#192), the model/effort overlay on roles in ward-kdl.fleet.kdl. This issue is the validity layer under that overlay resolution.

Filed from the read-only director surface (she/her).

**Soft-depends on #653** (the resolve seam gives the fleet.kdl that carries these model strings a live source). This is a design decision to settle in parallel. Breakdown of the [config-driven ward epic (#650)](https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/650). Raised by Kai reading the landed `.ward/ward-kdl.fleet.kdl` (aos#332). **This is a design fork the from-scratch epic design did not cover** - it needs a decision before build. Labeled `consult`. ## The new failure mode live-resolve introduces The fleet config carries per-agent and per-role **model** strings: `gpt-5.4` / `gpt-5.4-mini`, `qwen3-coder:30b`, and the role overlays `claude-fable-5`, `claude-opus-4-8[1m]`, `gpt-5.5` (`ward-kdl.fleet.kdl:52,66,91,104,108`). Under the **old baked config** these shipped lockstep with the binary that understood them - a ward release and its model strings versioned together. Under **live-resolve from a git ref**, the bundle on `main` and the locally-installed harness binaries (claude / codex / opencode / goose) version **independently**. So a fleet.kdl on `main` can name a model the local harness rejects, and **models move fast** - renames, deprecations, new tiers land constantly. This is **distinct from the epic's source fail-loud** (the "does the bundle resolve / parse" layer). Here the bundle resolves and parses cleanly - the problem is a **value inside it** is stale for the **specific harness** it targets. ## Why detection must be per-harness ward does not own the model namespaces. Each harness validates differently: * **claude** - `--model claude-opus-4-8[1m]`, its own id set. ward#616 already makes model override-only (empty `model ""` = bare launch = the harness's own default). * **codex** - `--model gpt-5.5`, OpenAI id set, separate cadence. * **opencode** - `--model qwen3-coder:30b` against a **local ollama** `endpoint` (`:67`) - an invalid model is a 404 from ollama, a totally different surface from a CLI-flag rejection. * **goose** - ollama auth, no model field today. A central ward-side model allowlist would stale immediately (the exact "moves fast" problem). Detection has to sit per-harness. ## The fork to decide (do not build until picked) Two axes, both Kai's call: * **On an invalid/stale model value: fall back or fail?** * **Fall back to the harness default** (the empty-string bare-launch path claude already has) - resilient, the fleet keeps running when one model string goes stale, but it **silently downgrades** (a director expecting `claude-opus-4-8[1m]` quietly gets whatever the harness defaults to). * **Fail loud** - honest, but one stale model string in a live-tracked bundle could brick the whole fleet. * Likely **role-sensitive**: a director losing its strong model silently is worse than an engineer fan-out losing `claude-fable-5`. The answer may differ by role. * **Where does detection live?** * **Probe before launch** (dry-run / capability query the harness) - clean, but adds a per-launch call and not every harness exposes a probe. * **Catch-at-launch** - launch, catch the harness's own "unknown model" error, then retry-with-default or surface. Reuses each harness's own validation, no ward-side model knowledge, but couples to each harness's error shape. ## Related `docs/agent-config-overrides.md` (overlay precedence, ward#616 / ward#620 / cli-guard#192), the model/effort overlay on `roles` in `ward-kdl.fleet.kdl`. This issue is the **validity** layer under that **overlay** resolution. Filed from the read-only director surface (she/her).

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-08 03:33:07 +00:00

coilyco-ops referenced this issue

2026-07-08 03:33:46 +00:00

[epic] Config-driven ward: one precompiled binary, personal config resolved live from a git ref (supersedes the build-variant matrix) #650

coilyco-ops referenced this issue

2026-07-08 03:40:46 +00:00

Config-driven ward: built-in defaults for frontier harnesses - fleet blocks compress, frontier agents need no KDL (#650-7) #659

coilyco-ops referenced this issue

2026-07-08 03:40:57 +00:00

[epic] Config-driven ward: one precompiled binary, personal config resolved live from a git ref (supersedes the build-variant matrix) #650

coilyco-ops referenced this issue

2026-07-08 04:49:15 +00:00

[epic] Config-driven ward: one precompiled binary, personal config resolved live from a git ref (supersedes the build-variant matrix) #650

coilyco-ops commented

2026-07-08 09:33:33 +00:00

Author

Owner

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Recommend the concrete policy for stale per-harness model config under live WARD_CONFIG_REF: fail vs fallback, where detection should live per harness, and which follow-up engineering issue should be filed next. Post the answer as a concise issue comment.

Recommendation: treat a stale per-harness model under live WARD_CONFIG_REF as a fail-loud config error, not an automatic fallback.

Why:

A silent fallback is acceptable only when the fallback semantics are real and uniform. They are not. claude can omit --model, codex may have its own config default, opencode pins an Ollama model in generated config, and goose has no fleet model knob today.
The worst failure here is not "one run failed". It is "the run succeeded on the wrong model and nobody noticed". That is especially bad for the director role, but it is still the wrong default for the rest of the fleet because the config author asked for a specific model.
Ward already has the right operational shape for this: harness-owned pre-launch gates that fail before launch and surface a named recovery path through the reaper. A stale model fits that pattern better than a best-effort downgrade.

Where detection should live: in the harness-specific LaunchGate / PreLaunchCheck, after config composition and before launch.

That matches the current code shape:

claude already has a real pre-launch gate.
opencode and goose already have harness-owned Ollama reachability gates.
codex has no gate yet, but the seam already exists.

This keeps model knowledge out of ward's central resolver and avoids a fast-staling ward-side allowlist.

Per harness policy:

claude: extend the existing auth smoke test so that when WARD_CLAUDE_MODEL resolves non-empty, the probe runs with that --model and classifies "unknown/invalid model" as a distinct pre-launch failure. No retry without --model.
codex: add a new in-container PreLaunchCheck that exercises codex with the resolved config/model in a bounded one-shot and classifies codex's own unknown-model rejection. No retry with model removed.
opencode: extend the current Ollama gate from reachability only to reachability + configured-model existence at the resolved endpoint. A TCP-success / model-404 should fail pre-launch as stale config, not fall through to the harness.
goose: no new validation in this fork. The fleet config does not currently carry a goose model field, so there is no live-resolve stale-model problem to solve there yet.

Why not probe centrally before launch: ward does not own these namespaces, and the failure surfaces differ too much. The right abstraction is "each harness validates its own resolved model in its own gate", not "ward learns all model catalogs".

Why not catch-at-launch and retry: that couples ward to stderr/error-shape parsing after the real launch handoff, complicates stream/log handling, and weakens the existing "pre-launch death names its gate" diagnostics. A dedicated launch gate is cleaner and already matches ward's operator model.

Concrete policy text:

If a resolved per-harness model value is invalid for the selected harness, ward aborts the run at a harness-owned pre-launch gate and reports a stale model-config error. Ward does not silently fall back to the harness default.

Next engineering issue to file: a single ward issue, not cross-repo fan-out.

Suggested scope:

Implement per-harness stale-model validation in LaunchGate.
Add codex's first launch gate.
Extend opencode's Ollama probe from endpoint reachability to model existence.
Distinguish the failure in gate reporting/recovery text (for example a model-config gate or equivalent harness-specific classification) so the reaper tells the operator to update the fleet model string or move WARD_CONFIG_REF to a compatible ref.
Depend on #653, and close this design fork as the settled policy for #650.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Recommend the concrete policy for stale per-harness model config under live WARD_CONFIG_REF: fail vs fallback, where detection should live per harness, and which follow-up engineering issue should be filed next. Post the answer as a concise issue comment. --- **Recommendation:** treat a stale per-harness model under live `WARD_CONFIG_REF` as a **fail-loud config error**, not an automatic fallback. Why: - A silent fallback is acceptable only when the fallback semantics are real and uniform. They are not. `claude` can omit `--model`, `codex` may have its own config default, `opencode` pins an Ollama model in generated config, and `goose` has no fleet model knob today. - The worst failure here is not "one run failed". It is "the run succeeded on the wrong model and nobody noticed". That is especially bad for the `director` role, but it is still the wrong default for the rest of the fleet because the config author asked for a specific model. - Ward already has the right operational shape for this: harness-owned pre-launch gates that fail before launch and surface a named recovery path through the reaper. A stale model fits that pattern better than a best-effort downgrade. **Where detection should live:** in the harness-specific `LaunchGate` / `PreLaunchCheck`, after config composition and before launch. That matches the current code shape: - `claude` already has a real pre-launch gate. - `opencode` and `goose` already have harness-owned Ollama reachability gates. - `codex` has no gate yet, but the seam already exists. This keeps model knowledge out of ward's central resolver and avoids a fast-staling ward-side allowlist. **Per harness policy:** - `claude`: extend the existing auth smoke test so that when `WARD_CLAUDE_MODEL` resolves non-empty, the probe runs with that `--model` and classifies "unknown/invalid model" as a distinct pre-launch failure. No retry without `--model`. - `codex`: add a new in-container `PreLaunchCheck` that exercises codex with the resolved config/model in a bounded one-shot and classifies codex's own unknown-model rejection. No retry with model removed. - `opencode`: extend the current Ollama gate from **reachability only** to **reachability + configured-model existence** at the resolved endpoint. A TCP-success / model-404 should fail pre-launch as stale config, not fall through to the harness. - `goose`: no new validation in this fork. The fleet config does not currently carry a goose model field, so there is no live-resolve stale-model problem to solve there yet. **Why not probe centrally before launch:** ward does not own these namespaces, and the failure surfaces differ too much. The right abstraction is "each harness validates its own resolved model in its own gate", not "ward learns all model catalogs". **Why not catch-at-launch and retry:** that couples ward to stderr/error-shape parsing after the real launch handoff, complicates stream/log handling, and weakens the existing "pre-launch death names its gate" diagnostics. A dedicated launch gate is cleaner and already matches ward's operator model. **Concrete policy text:** > If a resolved per-harness model value is invalid for the selected harness, ward aborts the run at a harness-owned pre-launch gate and reports a stale model-config error. Ward does not silently fall back to the harness default. **Next engineering issue to file:** a single ward issue, not cross-repo fan-out. Suggested scope: - Implement per-harness stale-model validation in `LaunchGate`. - Add codex's first launch gate. - Extend opencode's Ollama probe from endpoint reachability to model existence. - Distinguish the failure in gate reporting/recovery text (for example a `model-config` gate or equivalent harness-specific classification) so the reaper tells the operator to update the fleet model string or move `WARD_CONFIG_REF` to a compatible ref. - Depend on #653, and close this design fork as the settled policy for #650. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-08 09:49:26 +00:00

Implement stale model-config launch gates for live WARD_CONFIG_REF #670

coilyco-ops commented

2026-07-08 09:51:57 +00:00

Author

Owner

Converted the advisor recommendation into implementation issue #670 and dispatched a Codex engineer.

Decision captured there: stale per-harness model config from live WARD_CONFIG_REF fails loud in a harness-owned launch gate. No silent fallback. Keep this consult open until #670 lands, then close it as resolved by #670.

Converted the advisor recommendation into implementation issue #670 and dispatched a Codex engineer. Decision captured there: stale per-harness model config from live `WARD_CONFIG_REF` fails loud in a harness-owned launch gate. No silent fallback. Keep this consult open until #670 lands, then close it as resolved by #670.

coilyco-ops referenced this issue

2026-07-08 09:51:57 +00:00

[epic] Config-driven ward: one precompiled binary, personal config resolved live from a git ref (supersedes the build-variant matrix) #650

coilyco-ops added

priority

and removed

priority

labels

2026-07-10 09:00:37 +00:00

coilyco-ops added

priority

and removed