coilyco-flight-deck/ward

Fork 0

Design broker-exercising dispatch smoke test #932

New issue

Open

opened 2026-07-10 03:48:04 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-10 03:48:04 +00:00

Member

Context:

ward#931 is the implementation ticket for CI dispatch smoke coverage, but it is currently reserved by an engineer run. This issue captures the concrete broker-exercising test design so it is not lost and can be read after the active run completes.

Recommended smoke design:

Add an in-process fake dispatch broker using the real broker protocol, not a stubbed function call. It should bind 127.0.0.1:0, require the same token shape as production, accept exactly one request, record the decoded operation/target/argv, and return a successful broker response without launching Docker.
Run the actual CLI entrypoint or command runner for ward agent engineer coilyco-flight-deck/ward#<fixture> from a simulated read-only surface environment: WARD_READONLY=1, WARD_DISPATCH_BROKER_ADDR=<fake listener>, WARD_DISPATCH_BROKER_TOKEN=<token>, no /var/run/docker.sock, and no usable local Docker env.
Use a fixture issue provider instead of live Forgejo. The fixture only needs enough issue metadata for dispatch resolution: owner, repo, issue number, title, open state, labels if the mode gate needs them, and repo-authority mapping that trusts coilyco-flight-deck.
Assert the fake broker received a launch request for role engineer, target coilyco-flight-deck/ward#<fixture>, harness codex or the configured default, and no forbidden local-Docker error occurred.
Add a negative companion test: same read-only environment but no broker address/token should fail with an explicit broker-unavailable error, not silently fall through to Docker.
Add a config regression test: with the fleet config fixture loaded, coilyco-flight-deck is trusted; with the neutral example config, it is rejected. This makes the WARD_CONFIG_REF/smart-default bug visible in CI.

The important assertion is behavioral: read-only + broker configured means broker launch request recorded. Docker must not be consulted on that path.

Related: ward#931, ward#900, ward#876.

Context: ward#931 is the implementation ticket for CI dispatch smoke coverage, but it is currently reserved by an engineer run. This issue captures the concrete broker-exercising test design so it is not lost and can be read after the active run completes. Recommended smoke design: * Add an in-process fake dispatch broker using the real broker protocol, not a stubbed function call. It should bind `127.0.0.1:0`, require the same token shape as production, accept exactly one request, record the decoded operation/target/argv, and return a successful broker response without launching Docker. * Run the actual CLI entrypoint or command runner for `ward agent engineer coilyco-flight-deck/ward#<fixture>` from a simulated read-only surface environment: `WARD_READONLY=1`, `WARD_DISPATCH_BROKER_ADDR=<fake listener>`, `WARD_DISPATCH_BROKER_TOKEN=<token>`, no `/var/run/docker.sock`, and no usable local Docker env. * Use a fixture issue provider instead of live Forgejo. The fixture only needs enough issue metadata for dispatch resolution: owner, repo, issue number, title, open state, labels if the mode gate needs them, and repo-authority mapping that trusts `coilyco-flight-deck`. * Assert the fake broker received a `launch` request for role `engineer`, target `coilyco-flight-deck/ward#<fixture>`, harness `codex` or the configured default, and no forbidden local-Docker error occurred. * Add a negative companion test: same read-only environment but no broker address/token should fail with an explicit broker-unavailable error, not silently fall through to Docker. * Add a config regression test: with the fleet config fixture loaded, `coilyco-flight-deck` is trusted; with the neutral example config, it is rejected. This makes the `WARD_CONFIG_REF`/smart-default bug visible in CI. The important assertion is behavioral: read-only + broker configured means broker launch request recorded. Docker must not be consulted on that path. Related: ward#931, ward#900, ward#876.

coilyco-ops commented

2026-07-10 03:50:11 +00:00

Author

Member

Design correction from Kai: the valuable smoke needs to exercise a real Docker socket connection, not only a fake broker.

Revised shape:

Keep the fake-broker unit test as a cheap routing guard, but do not treat it as the smoke. It catches read-only surface forwarding regressions only.
Add a CI job that runs on a runner with Docker available and starts the real host dispatch broker wired to that real Docker socket.
Launch the surface-side command from a subprocess with WARD_READONLY=1, WARD_DISPATCH_BROKER_ADDR, WARD_DISPATCH_BROKER_TOKEN, and no direct Docker socket mounted in the simulated surface environment.
Use a no-op/smoke agent image or a purpose-built WARD_SMOKE_TEST_SKIP mode that still forces the broker to call Docker create/start, but makes the launched container exit quickly without invoking an LLM, mutating Forgejo, or pushing git.
Assert from the host side that Docker saw a container create/start for the expected target labels: ward.role=engineer, ward.repo=coilyco-flight-deck/ward, carried issue number, branch, and harness.
Assert from the surface side that the command reports broker forwarding and never opens /var/run/docker.sock locally.
Negative companion: with broker env missing, the same read-only surface fails with broker-unavailable before any Docker attempt.
Optional deeper check: run the smoke against a local disposable Forgejo fixture or fixture issue provider. If that is too heavy for every PR, run the real-Docker smoke in release CI and keep the fake-broker unit in normal PR CI.

This makes the CI signal cover the important boundary: read-only surface -> broker -> host Docker socket -> short-lived engineer container. A fake broker alone would still miss host Docker wiring regressions.

Design correction from Kai: the valuable smoke needs to exercise a real Docker socket connection, not only a fake broker. Revised shape: * Keep the fake-broker unit test as a cheap routing guard, but do not treat it as the smoke. It catches read-only surface forwarding regressions only. * Add a CI job that runs on a runner with Docker available and starts the real host dispatch broker wired to that real Docker socket. * Launch the surface-side command from a subprocess with `WARD_READONLY=1`, `WARD_DISPATCH_BROKER_ADDR`, `WARD_DISPATCH_BROKER_TOKEN`, and no direct Docker socket mounted in the simulated surface environment. * Use a no-op/smoke agent image or a purpose-built `WARD_SMOKE_TEST_SKIP` mode that still forces the broker to call Docker create/start, but makes the launched container exit quickly without invoking an LLM, mutating Forgejo, or pushing git. * Assert from the host side that Docker saw a container create/start for the expected target labels: `ward.role=engineer`, `ward.repo=coilyco-flight-deck/ward`, carried issue number, branch, and harness. * Assert from the surface side that the command reports broker forwarding and never opens `/var/run/docker.sock` locally. * Negative companion: with broker env missing, the same read-only surface fails with broker-unavailable before any Docker attempt. * Optional deeper check: run the smoke against a local disposable Forgejo fixture or fixture issue provider. If that is too heavy for every PR, run the real-Docker smoke in release CI and keep the fake-broker unit in normal PR CI. This makes the CI signal cover the important boundary: read-only surface -> broker -> host Docker socket -> short-lived engineer container. A fake broker alone would still miss host Docker wiring regressions.

coilyco-ops commented

2026-07-10 03:50:43 +00:00

Author

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Recommendation

Implement the smoke as a small set of hermetic Go tests in cmd/ward, not as a live CI script. The core acceptance should be: WARD_READONLY=1 plus valid dispatch-broker coordinates causes ward agent engineer coilyco-flight-deck/ward#<fixture> to emit one real broker-protocol launch request and return without consulting local Docker.

Decisions

Keep this in ward only. The requested work is single-repo test coverage, so no child issues are needed.
Exercise the command path, not only maybeForwardAgentDispatchToHostBroker. Existing tests already cover the helper and token gate, but ward#931 needs coverage closer to ward agent engineer ... so regressions in dispatch wiring, harness inheritance, config selection, or read-only fallback are caught.
Use an in-process TCP fake broker that speaks the existing dispatchBrokerRequest / dispatchBrokerResponse JSON protocol. It should bind 127.0.0.1:0, require a production-shaped token, accept exactly one connection, record the request, and reply {ok:true}.
Split broker behavior from config trust behavior. A full owner/repo#N ref can forward before fetching issue metadata, so forcing an issue-provider fixture into the broker smoke would either be unused or would make the test larger than needed. Put the owner trust regression in a separate config test.

Recommended Test Shape

TestReadOnlyEngineerDispatchUsesBroker: set WARD_READONLY=1, WARD_DISPATCH_BROKER_ADDR=<fake>, WARD_DISPATCH_BROKER_TOKEN=<64-hex-token>, WARD_CONTAINER_NAME=director-codex-host, and WARD_AGENT / WARD_MODE=codex. Run the engineer command path for coilyco-flight-deck/ward#<fixture>. Assert the fake broker sees Action empty or launch, Role=engineer, argv containing engineer, the target ref, and --harness codex unless an explicit harness override is under test. Assert no Docker error appears.
TestReadOnlyEngineerDispatchWithoutBrokerFailsLoud: set WARD_READONLY=1 but no broker addr/token. The expected behavior should be an explicit broker-unavailable/read-only-surface error. This is important because the current helper returns (false, nil) when the broker address is empty, which lets the path fall through toward local Docker.
TestDispatchTrustConfigSelectsFleetBundle: load a committed or synthesized config-bundle fixture whose repo-authority trusts coilyco-flight-deck, then assert ownerAllowed("coilyco-flight-deck") passes and authority maps coilyco-flight-deck/* to Forgejo. In the neutral baked/example config, assert the same owner is rejected. Do not depend on /substrate in CI, but the substrate aos bundle is a useful model: it trusts coilysiren, coilyco-bridge, coilyco-flight-deck, and coilyco-gaming.

Open Questions

Should read-only-without-broker be modeled as errDispatchBrokerUnavailable, or a more specific dispatch-contract error? I recommend wrapping or reusing errDispatchBrokerUnavailable while naming WARD_READONLY=1 in the message.
Should the broker smoke run through agentEngineerCommand().Run or directly through runAgentEngineer with a parsed command? Prefer the full command if it stays stable and fast. Use runAgentEngineer if audit/global CLI setup makes the test brittle.
If fixture issue metadata is still desired, add it only to a separate resolve/trust test. The broker smoke itself should not require live Forgejo or a fake tracker unless the command path changes to fetch issue metadata before forwarding.

Risks

A fake broker that only calls a stubbed Go function will not catch the actual regression. It must decode the TCP JSON request.
A test that returns success from the broker without asserting the recorded request could still pass while forwarding the wrong role, harness, or target.
A smoke that allows Docker in the environment may pass accidentally on a developer host while still failing in the director surface. Poisoning or isolating Docker for the test is useful, but the stronger assertion is that the broker receives the request before Docker can matter.
Loading the real remote config in CI would make the test flaky and network-dependent. Use a local fixture that represents the trusted fleet bundle instead.

Net: make ward#931 add three focused regressions: broker-forwarded engineer dispatch, read-only no-broker fail-loud behavior, and config trust selection. That directly covers the failures from ward#900 and ward#876 without turning CI into a live dispatch exercise.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- **Recommendation** Implement the smoke as a small set of hermetic Go tests in `cmd/ward`, not as a live CI script. The core acceptance should be: `WARD_READONLY=1` plus valid dispatch-broker coordinates causes `ward agent engineer coilyco-flight-deck/ward#<fixture>` to emit one real broker-protocol `launch` request and return without consulting local Docker. **Decisions** * Keep this in `ward` only. The requested work is single-repo test coverage, so no child issues are needed. * Exercise the command path, not only `maybeForwardAgentDispatchToHostBroker`. Existing tests already cover the helper and token gate, but ward#931 needs coverage closer to `ward agent engineer ...` so regressions in dispatch wiring, harness inheritance, config selection, or read-only fallback are caught. * Use an in-process TCP fake broker that speaks the existing `dispatchBrokerRequest` / `dispatchBrokerResponse` JSON protocol. It should bind `127.0.0.1:0`, require a production-shaped token, accept exactly one connection, record the request, and reply `{ok:true}`. * Split broker behavior from config trust behavior. A full `owner/repo#N` ref can forward before fetching issue metadata, so forcing an issue-provider fixture into the broker smoke would either be unused or would make the test larger than needed. Put the owner trust regression in a separate config test. **Recommended Test Shape** * `TestReadOnlyEngineerDispatchUsesBroker`: set `WARD_READONLY=1`, `WARD_DISPATCH_BROKER_ADDR=<fake>`, `WARD_DISPATCH_BROKER_TOKEN=<64-hex-token>`, `WARD_CONTAINER_NAME=director-codex-host`, and `WARD_AGENT` / `WARD_MODE=codex`. Run the engineer command path for `coilyco-flight-deck/ward#<fixture>`. Assert the fake broker sees `Action` empty or `launch`, `Role=engineer`, argv containing `engineer`, the target ref, and `--harness codex` unless an explicit harness override is under test. Assert no Docker error appears. * `TestReadOnlyEngineerDispatchWithoutBrokerFailsLoud`: set `WARD_READONLY=1` but no broker addr/token. The expected behavior should be an explicit broker-unavailable/read-only-surface error. This is important because the current helper returns `(false, nil)` when the broker address is empty, which lets the path fall through toward local Docker. * `TestDispatchTrustConfigSelectsFleetBundle`: load a committed or synthesized config-bundle fixture whose `repo-authority` trusts `coilyco-flight-deck`, then assert `ownerAllowed("coilyco-flight-deck")` passes and authority maps `coilyco-flight-deck/*` to Forgejo. In the neutral baked/example config, assert the same owner is rejected. Do not depend on `/substrate` in CI, but the substrate aos bundle is a useful model: it trusts `coilysiren`, `coilyco-bridge`, `coilyco-flight-deck`, and `coilyco-gaming`. **Open Questions** * Should read-only-without-broker be modeled as `errDispatchBrokerUnavailable`, or a more specific dispatch-contract error? I recommend wrapping or reusing `errDispatchBrokerUnavailable` while naming `WARD_READONLY=1` in the message. * Should the broker smoke run through `agentEngineerCommand().Run` or directly through `runAgentEngineer` with a parsed command? Prefer the full command if it stays stable and fast. Use `runAgentEngineer` if audit/global CLI setup makes the test brittle. * If fixture issue metadata is still desired, add it only to a separate resolve/trust test. The broker smoke itself should not require live Forgejo or a fake tracker unless the command path changes to fetch issue metadata before forwarding. **Risks** * A fake broker that only calls a stubbed Go function will not catch the actual regression. It must decode the TCP JSON request. * A test that returns success from the broker without asserting the recorded request could still pass while forwarding the wrong role, harness, or target. * A smoke that allows Docker in the environment may pass accidentally on a developer host while still failing in the director surface. Poisoning or isolating Docker for the test is useful, but the stronger assertion is that the broker receives the request before Docker can matter. * Loading the real remote config in CI would make the test flaky and network-dependent. Use a local fixture that represents the trusted fleet bundle instead. Net: make ward#931 add three focused regressions: broker-forwarded engineer dispatch, read-only no-broker fail-loud behavior, and config trust selection. That directly covers the failures from ward#900 and ward#876 without turning CI into a live dispatch exercise. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilyco-ops commented

2026-07-10 03:51:42 +00:00

Author

Member

Detailed design for the real-Docker broker smoke:

Goal:

Prove the dispatch boundary that keeps breaking: read-only surface -> host dispatch broker -> host Docker socket -> short-lived engineer container.

Primary CI test:

Add a Docker-capable CI job in ward. Run it on the same runner class that can build or run ward containers today.
Build a tiny local smoke image inside the job, for example ward-dispatch-smoke:${GITHUB_SHA:-local}. The image should include /opt/ward/entrypoint.sh and make that entrypoint print a sentinel line, write a marker file if mounted, then exit 0. It should not invoke an LLM, Forgejo mutation, git push, or pre-commit.
Override the agent image for the launch with a test-only knob, preferably WARD_AGENT_IMAGE=ward-dispatch-smoke:<tag> if ward already supports it. If ward only has WARD_SMOKE_TEST_SKIP, tighten that path so it still performs Docker create/start and only short-circuits inside the launched container.
Start the real host dispatch broker in the test process or as a child process. It must be the real broker implementation, not a fake listener. Configure it with a random token and with access to the runner Docker socket.
Run the surface-side command from a subprocess with a deliberately read-only surface environment: WARD_READONLY=1, WARD_DISPATCH_BROKER_ADDR=<real broker>, WARD_DISPATCH_BROKER_TOKEN=<token>, WARD_CONTAINER_NAME=director-codex-ci-smoke, WARD_AGENT=codex, and no direct Docker socket available to that subprocess.
Use a fixture issue ref such as coilyco-flight-deck/ward#1 or a local fixture resolver. The test must not call live Forgejo. The fixture only needs enough data to resolve an open trusted issue and choose the engineer workflow.
Invoke the real CLI path, not an internal helper: ward agent engineer coilyco-flight-deck/ward#<fixture> --harness codex --skip-preflight plus the image/smoke overrides needed to keep it short.

Assertions:

The surface subprocess exits 0 and its output says it forwarded through the dispatch broker.
The surface subprocess does not emit Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Docker on the host runner shows a container was created and started by the broker path.
The container labels include at least ward.role=engineer, ward.driver=codex, ward.repo=coilyco-flight-deck/ward, and the carried issue number.
The container env includes the expected carried context: WARD_ROLE=engineer, WARD_TARGET_REPO=coilyco-flight-deck/ward, WARD_TARGET_OWNER=coilyco-flight-deck, and the selected branch name.
The smoke container exits 0 and emits the sentinel line.
Cleanup removes the smoke container, image, temp config, temp gitcache, and broker process.

Negative tests:

Read-only surface with no broker env fails before Docker with a specific broker-unavailable/read-only-surface error.
Read-only surface with a bad broker token reaches the broker and is rejected before Docker create.
Read-only surface with broker configured must never fall through to local Docker. This should be asserted by running the surface subprocess in an environment where /var/run/docker.sock is absent or poisoned.

Config fixture:

Add a local fleet-config fixture that trusts coilyco-flight-deck and maps coilyco-flight-deck/* to Forgejo.
Add a separate fast test that proves the neutral/example config rejects coilyco-flight-deck while the fleet fixture accepts it.
The real-Docker smoke should use the fleet fixture directly. It should not fetch WARD_CONFIG_REF from the network.

CI placement:

Keep cheap fake-broker routing tests in normal unit CI.
Run this real-Docker smoke in the main PR CI if the ward runner already has Docker.
If PR runners are too constrained, run it in release CI and in a scheduled daily workflow. The important rule is that a ward release cannot go green without this smoke passing.

Why this design is valuable:

A fake broker proves the surface chose the broker branch. The real-Docker smoke proves the whole operational chain works: broker auth, broker request decoding, host launch policy, Docker socket access, container create/start, labels/env wiring, and short-lived container completion. That is the class of regression Kai just hit.

Detailed design for the real-Docker broker smoke: Goal: Prove the dispatch boundary that keeps breaking: read-only surface -> host dispatch broker -> host Docker socket -> short-lived engineer container. Primary CI test: * Add a Docker-capable CI job in ward. Run it on the same runner class that can build or run ward containers today. * Build a tiny local smoke image inside the job, for example `ward-dispatch-smoke:${GITHUB_SHA:-local}`. The image should include `/opt/ward/entrypoint.sh` and make that entrypoint print a sentinel line, write a marker file if mounted, then exit 0. It should not invoke an LLM, Forgejo mutation, git push, or pre-commit. * Override the agent image for the launch with a test-only knob, preferably `WARD_AGENT_IMAGE=ward-dispatch-smoke:<tag>` if ward already supports it. If ward only has `WARD_SMOKE_TEST_SKIP`, tighten that path so it still performs Docker create/start and only short-circuits inside the launched container. * Start the real host dispatch broker in the test process or as a child process. It must be the real broker implementation, not a fake listener. Configure it with a random token and with access to the runner Docker socket. * Run the surface-side command from a subprocess with a deliberately read-only surface environment: `WARD_READONLY=1`, `WARD_DISPATCH_BROKER_ADDR=<real broker>`, `WARD_DISPATCH_BROKER_TOKEN=<token>`, `WARD_CONTAINER_NAME=director-codex-ci-smoke`, `WARD_AGENT=codex`, and no direct Docker socket available to that subprocess. * Use a fixture issue ref such as `coilyco-flight-deck/ward#1` or a local fixture resolver. The test must not call live Forgejo. The fixture only needs enough data to resolve an open trusted issue and choose the engineer workflow. * Invoke the real CLI path, not an internal helper: `ward agent engineer coilyco-flight-deck/ward#<fixture> --harness codex --skip-preflight` plus the image/smoke overrides needed to keep it short. Assertions: * The surface subprocess exits 0 and its output says it forwarded through the dispatch broker. * The surface subprocess does not emit `Cannot connect to the Docker daemon at unix:///var/run/docker.sock`. * Docker on the host runner shows a container was created and started by the broker path. * The container labels include at least `ward.role=engineer`, `ward.driver=codex`, `ward.repo=coilyco-flight-deck/ward`, and the carried issue number. * The container env includes the expected carried context: `WARD_ROLE=engineer`, `WARD_TARGET_REPO=coilyco-flight-deck/ward`, `WARD_TARGET_OWNER=coilyco-flight-deck`, and the selected branch name. * The smoke container exits 0 and emits the sentinel line. * Cleanup removes the smoke container, image, temp config, temp gitcache, and broker process. Negative tests: * Read-only surface with no broker env fails before Docker with a specific broker-unavailable/read-only-surface error. * Read-only surface with a bad broker token reaches the broker and is rejected before Docker create. * Read-only surface with broker configured must never fall through to local Docker. This should be asserted by running the surface subprocess in an environment where `/var/run/docker.sock` is absent or poisoned. Config fixture: * Add a local fleet-config fixture that trusts `coilyco-flight-deck` and maps `coilyco-flight-deck/*` to Forgejo. * Add a separate fast test that proves the neutral/example config rejects `coilyco-flight-deck` while the fleet fixture accepts it. * The real-Docker smoke should use the fleet fixture directly. It should not fetch `WARD_CONFIG_REF` from the network. CI placement: * Keep cheap fake-broker routing tests in normal unit CI. * Run this real-Docker smoke in the main PR CI if the ward runner already has Docker. * If PR runners are too constrained, run it in release CI and in a scheduled daily workflow. The important rule is that a ward release cannot go green without this smoke passing. Why this design is valuable: A fake broker proves the surface chose the broker branch. The real-Docker smoke proves the whole operational chain works: broker auth, broker request decoding, host launch policy, Docker socket access, container create/start, labels/env wiring, and short-lived container completion. That is the class of regression Kai just hit.

coilyco-ops commented

2026-07-10 04:19:55 +00:00

Author

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Recommendation

Keep ward#931 as a single-repo coilyco-flight-deck/ward test-design/implementation track. The corrected target should be a two-layer smoke suite: keep the cheap fake-broker routing test, but make the release-blocking signal exercise the real host dispatch broker against a real Docker socket and a short-lived smoke container. The behavioral contract is: WARD_READONLY=1 plus valid broker env forwards through the broker, the broker performs Docker create/start on the host, and the surface never falls through to local Docker.

Decisions

Scope stays in ward. No child issues are needed because the code, tests, workflow, config fixtures, and docs live in this repo.
The fake broker is a unit/regression guard, not the smoke. Current code already has helper-level and fake-broker coverage around maybeForwardAgentDispatchToHostBroker, harness inheritance, token forwarding, and a freeform read-only broker route. That is valuable, but it cannot catch broken broker startup, host Docker socket access, Docker labels/env wiring, or create/start failures.
The smoke should start the real broker implementation (startHostDispatchBroker path), not a JSON listener, then launch the surface-side command in a subprocess with WARD_READONLY=1, WARD_DISPATCH_BROKER_ADDR, WARD_DISPATCH_BROKER_TOKEN, WARD_AGENT=codex, WARD_MODE=codex, and WARD_CONTAINER_NAME=director-codex-ci-smoke.
The launched container should be harmless. Prefer overriding WARD_AGENT_IMAGE to a locally built smoke image whose entrypoint emits a sentinel and exits 0. If using --skip-preflight / WARD_SMOKE_TEST_SKIP, keep that as an in-container auth/preflight bypass only. It must not skip Docker create/start.
Use a local fixture config and issue resolver. Do not fetch live Forgejo and do not fetch WARD_CONFIG_REF from the network. The fixture should trust coilyco-flight-deck, map coilyco-flight-deck/* to Forgejo, and provide just enough issue metadata to resolve an open engineer target.
Add a fail-loud negative path: read-only surface with missing broker addr/token must fail with an explicit broker-unavailable/read-only-surface error before Docker is attempted. The current helper returns (false, nil) when broker addr is empty, so this is a real contract to pin, not just a test assertion.
Add a bad-token negative path if the real smoke can afford it: the surface reaches the broker, the broker rejects the token, and Docker create/start does not happen.
Add a separate config regression test: the neutral baked defaults reject coilyco-flight-deck, while a fleet fixture accepts it. That covers the WARD_CONFIG_REF / smart-defaults class without making the Docker smoke network-dependent.

Recommended Test Shape

Fast unit CI: keep the fake TCP broker tests that decode dispatchBrokerRequest, require a realistic token, assert Role=engineer, target argv, inherited or explicit --harness codex, requester, and no local Docker lookup.
Real-Docker CI: build ward-dispatch-smoke:<sha> locally, start the real broker with a random token, run ward agent engineer coilyco-flight-deck/ward#<fixture> --harness codex --skip-preflight --skip-review from a poisoned surface environment, and assert broker forwarding plus Docker create/start.
Host-side assertions: a container was created/started with labels ward.role=engineer, ward.driver=codex, ward.repo=coilyco-flight-deck/ward, ward.issue=<fixture>, and the expected workflow label when non-default.
Container/env assertions: the smoke container exits 0, emits the sentinel, and carries WARD_ROLE=engineer, WARD_TARGET_REPO=coilyco-flight-deck/ward, WARD_TARGET_OWNER=coilyco-flight-deck, WARD_TARGET_ISSUE=<fixture>, and the selected branch.
Surface assertions: output reports dispatch-broker forwarding, does not include Docker daemon/socket errors, and does not open /var/run/docker.sock locally. Poisoning local Docker in the subprocess is useful, but the strongest check is that host Docker activity is observed only behind the broker.

Open Questions

Where should the real-Docker smoke run? If the normal PR runner exposes a usable Docker socket to the job, run it there. If not, put it in release CI and a scheduled workflow, but make releases unable to go green without it.
Should the CI job run inside the existing containerized workflow or as a host-level job? The current workflows run tests inside agentic-os:latest; the smoke needs a verified path to the host Docker socket, so the implementation should first prove that runner shape.
Should missing broker env reuse errDispatchBrokerUnavailable or introduce a more specific read-only surface requires dispatch broker error? Reusing errDispatchBrokerUnavailable is fine if the message names WARD_READONLY=1 and the missing env var.
Should the smoke use the full CLI binary or agentEngineerCommand().Run? Prefer a subprocess running the built ward binary because this issue is about integration wiring. Fall back to command-runner only if audit/global setup makes the subprocess brittle.

Risks

A fake-broker-only solution will pass while the real broker cannot bind, authenticate, decode, or reach Docker.
A Docker smoke that uses the normal agent image can accidentally invoke LLM auth, Forgejo mutation, git push, or pre-commit. The smoke image or skip mode must be purpose-built and side-effect free.
A surface subprocess with Docker still available may hide the regression by falling through to local Docker. The negative tests need to poison or remove local Docker and assert broker-unavailable happens before any Docker attempt.
Live Forgejo or live config fetches would make the smoke flaky and credential-dependent. Keep issue metadata and fleet config local.
Docker cleanup has to be strict. Remove the smoke container, image, temp config, temp gitcache, and broker process even on failure.

Net: ward#931 should land three concrete guards: fake-broker unit coverage for read-only forwarding, a real-Docker broker smoke for the host boundary, and config-trust regression coverage. The real-Docker smoke is the acceptance test Kai corrected toward because it covers the actual failure boundary: read-only surface -> broker -> host Docker socket -> short-lived engineer container.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- **Recommendation** Keep ward#931 as a single-repo `coilyco-flight-deck/ward` test-design/implementation track. The corrected target should be a two-layer smoke suite: keep the cheap fake-broker routing test, but make the release-blocking signal exercise the real host dispatch broker against a real Docker socket and a short-lived smoke container. The behavioral contract is: `WARD_READONLY=1` plus valid broker env forwards through the broker, the broker performs Docker create/start on the host, and the surface never falls through to local Docker. **Decisions** * Scope stays in ward. No child issues are needed because the code, tests, workflow, config fixtures, and docs live in this repo. * The fake broker is a unit/regression guard, not the smoke. Current code already has helper-level and fake-broker coverage around `maybeForwardAgentDispatchToHostBroker`, harness inheritance, token forwarding, and a freeform read-only broker route. That is valuable, but it cannot catch broken broker startup, host Docker socket access, Docker labels/env wiring, or create/start failures. * The smoke should start the real broker implementation (`startHostDispatchBroker` path), not a JSON listener, then launch the surface-side command in a subprocess with `WARD_READONLY=1`, `WARD_DISPATCH_BROKER_ADDR`, `WARD_DISPATCH_BROKER_TOKEN`, `WARD_AGENT=codex`, `WARD_MODE=codex`, and `WARD_CONTAINER_NAME=director-codex-ci-smoke`. * The launched container should be harmless. Prefer overriding `WARD_AGENT_IMAGE` to a locally built smoke image whose entrypoint emits a sentinel and exits 0. If using `--skip-preflight` / `WARD_SMOKE_TEST_SKIP`, keep that as an in-container auth/preflight bypass only. It must not skip Docker create/start. * Use a local fixture config and issue resolver. Do not fetch live Forgejo and do not fetch `WARD_CONFIG_REF` from the network. The fixture should trust `coilyco-flight-deck`, map `coilyco-flight-deck/*` to Forgejo, and provide just enough issue metadata to resolve an open engineer target. * Add a fail-loud negative path: read-only surface with missing broker addr/token must fail with an explicit broker-unavailable/read-only-surface error before Docker is attempted. The current helper returns `(false, nil)` when broker addr is empty, so this is a real contract to pin, not just a test assertion. * Add a bad-token negative path if the real smoke can afford it: the surface reaches the broker, the broker rejects the token, and Docker create/start does not happen. * Add a separate config regression test: the neutral baked defaults reject `coilyco-flight-deck`, while a fleet fixture accepts it. That covers the `WARD_CONFIG_REF` / smart-defaults class without making the Docker smoke network-dependent. **Recommended Test Shape** * Fast unit CI: keep the fake TCP broker tests that decode `dispatchBrokerRequest`, require a realistic token, assert `Role=engineer`, target argv, inherited or explicit `--harness codex`, requester, and no local Docker lookup. * Real-Docker CI: build `ward-dispatch-smoke:<sha>` locally, start the real broker with a random token, run `ward agent engineer coilyco-flight-deck/ward#<fixture> --harness codex --skip-preflight --skip-review` from a poisoned surface environment, and assert broker forwarding plus Docker create/start. * Host-side assertions: a container was created/started with labels `ward.role=engineer`, `ward.driver=codex`, `ward.repo=coilyco-flight-deck/ward`, `ward.issue=<fixture>`, and the expected workflow label when non-default. * Container/env assertions: the smoke container exits 0, emits the sentinel, and carries `WARD_ROLE=engineer`, `WARD_TARGET_REPO=coilyco-flight-deck/ward`, `WARD_TARGET_OWNER=coilyco-flight-deck`, `WARD_TARGET_ISSUE=<fixture>`, and the selected branch. * Surface assertions: output reports dispatch-broker forwarding, does not include Docker daemon/socket errors, and does not open `/var/run/docker.sock` locally. Poisoning local Docker in the subprocess is useful, but the strongest check is that host Docker activity is observed only behind the broker. **Open Questions** * Where should the real-Docker smoke run? If the normal PR runner exposes a usable Docker socket to the job, run it there. If not, put it in release CI and a scheduled workflow, but make releases unable to go green without it. * Should the CI job run inside the existing containerized workflow or as a host-level job? The current workflows run tests inside `agentic-os:latest`; the smoke needs a verified path to the host Docker socket, so the implementation should first prove that runner shape. * Should missing broker env reuse `errDispatchBrokerUnavailable` or introduce a more specific `read-only surface requires dispatch broker` error? Reusing `errDispatchBrokerUnavailable` is fine if the message names `WARD_READONLY=1` and the missing env var. * Should the smoke use the full CLI binary or `agentEngineerCommand().Run`? Prefer a subprocess running the built `ward` binary because this issue is about integration wiring. Fall back to command-runner only if audit/global setup makes the subprocess brittle. **Risks** * A fake-broker-only solution will pass while the real broker cannot bind, authenticate, decode, or reach Docker. * A Docker smoke that uses the normal agent image can accidentally invoke LLM auth, Forgejo mutation, git push, or pre-commit. The smoke image or skip mode must be purpose-built and side-effect free. * A surface subprocess with Docker still available may hide the regression by falling through to local Docker. The negative tests need to poison or remove local Docker and assert broker-unavailable happens before any Docker attempt. * Live Forgejo or live config fetches would make the smoke flaky and credential-dependent. Keep issue metadata and fleet config local. * Docker cleanup has to be strict. Remove the smoke container, image, temp config, temp gitcache, and broker process even on failure. Net: ward#931 should land three concrete guards: fake-broker unit coverage for read-only forwarding, a real-Docker broker smoke for the host boundary, and config-trust regression coverage. The real-Docker smoke is the acceptance test Kai corrected toward because it covers the actual failure boundary: read-only surface -> broker -> host Docker socket -> short-lived engineer container. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilyco-ops added the

consult

labels

2026-07-10 09:00:33 +00:00

coilyco-ops added

and removed

labels

2026-07-14 06:50:22 +00:00