coilyco-flight-deck/ward

Fork 0

Surface broker/pre-flight dispatch pathology by default: capability-detected status-line injection + out-of-band alerts #1076

New issue

Closed

opened 2026-07-10 19:00:48 +00:00 by coilyco-ops · 6 comments

coilyco-ops commented

2026-07-10 19:00:48 +00:00

Member

Surface broker/pre-flight dispatch pathology by default (child of ward#1069)

Ward already emits dispatch pathology - deferred/failed dispatches, the reservation double-dispatch race (ward#996), /goal runaways opening unbounded branches, backpressure engaged - into run envelopes, reservation comments, and agent logs. But nothing pushes it. It is all pull: a human (or an in-loop agent) only finds out by running docker ps / reading reservation threads by hand. Tonight the #996 double-dispatch and the /goal branch flood were invisible until stumbled on. The observability arm of the anti-churn epic is unowned: the problem is not missing telemetry, it is missing surfacing.

Two consumers need the push, plus an optional operator HUD.

Do

Agent-context injection via ward capability detection. Ward already detects per-harness/agent capabilities. Use it to inject a live dispatch-health status line into agents that support one (Claude Code supports a status line; codex/opencode/goose may not - gate on the detected capability, never inject into an agent that cannot render it). The status line carries the current pathology signal: running / deferred / failed dispatch counts, any active reservation conflict or double-dispatch, open-PR-vs-cap (backpressure state, ward#1073), and /goal runaway indicators. Effect: the in-loop director agent surfaces "3 deferred, double-dispatch on ward#996, over the PR cap" to the operator unprompted, instead of needing to be asked.
Out-of-band alerts. Add alert rules (not new telemetry) on the existing ward agent-run envelope stream into SigNoz (deploy#23) and the existing Telegram main-failure / ntfy channels, firing on dispatch pathology: a deferred-dispatch storm, a reservation race, a runaway dispatch rate, backpressure engaging. This reaches the human when no agent is in the seat, the same way a red main already alerts.
Optional operator HUD. A warded doctor-style dispatch-health watch view suitable for a zellij floating pane or a status bar, for glanceable operator awareness. Nice-to-have on top of 1 and 2, not a substitute - a pane you must watch does not solve "nothing told the agent to tell me."

Acceptance

A deferred-dispatch storm, a reservation double-dispatch, or a /goal runaway surfaces (a) in the director agent's capability-detected status line within a turn, and (b) as a Telegram/ntfy alert, with no one running docker ps.
Status-line injection is gated on ward capability detection and is a no-op for agents that do not support it.
The alert rules ride the existing SigNoz envelope stream + Telegram/ntfy plumbing, not a new pipeline.

Context

Directed by Kai. This is the observability half of ward#1069 (detect the jam, surface it) - the missing signal that would have turned tonight's spelunk into a 5-minute heads-up. Capability detection is the mechanism Kai flagged for the status-line injection (claude supports it today). Refs: ward#1069 (epic), ward#996 (the race that was invisible), ward#1073 (backpressure state to surface), deploy#23 (SigNoz agent-run envelope stream), the Telegram main-failure + ntfy channels.

## Surface broker/pre-flight dispatch pathology by default (child of ward#1069) Ward already *emits* dispatch pathology - deferred/failed dispatches, the reservation double-dispatch race (ward#996), `/goal` runaways opening unbounded branches, backpressure engaged - into run envelopes, reservation comments, and agent logs. But nothing **pushes** it. It is all pull: a human (or an in-loop agent) only finds out by running `docker ps` / reading reservation threads by hand. Tonight the #996 double-dispatch and the `/goal` branch flood were invisible until stumbled on. The observability arm of the anti-churn epic is unowned: **the problem is not missing telemetry, it is missing surfacing.** Two consumers need the push, plus an optional operator HUD. ## Do 1. **Agent-context injection via ward capability detection.** Ward already detects per-harness/agent capabilities. Use it to inject a live **dispatch-health status line** into agents that support one (Claude Code supports a status line; codex/opencode/goose may not - gate on the detected capability, never inject into an agent that cannot render it). The status line carries the current pathology signal: running / deferred / failed dispatch counts, any active reservation conflict or double-dispatch, open-PR-vs-cap (backpressure state, ward#1073), and `/goal` runaway indicators. Effect: the in-loop director agent surfaces "3 deferred, double-dispatch on ward#996, over the PR cap" to the operator **unprompted**, instead of needing to be asked. 2. **Out-of-band alerts.** Add alert *rules* (not new telemetry) on the existing ward agent-run envelope stream into SigNoz (deploy#23) and the existing Telegram main-failure / ntfy channels, firing on dispatch pathology: a deferred-dispatch storm, a reservation race, a runaway dispatch rate, backpressure engaging. This reaches the human when no agent is in the seat, the same way a red main already alerts. 3. **Optional operator HUD.** A `warded doctor`-style dispatch-health watch view suitable for a zellij floating pane or a status bar, for glanceable operator awareness. Nice-to-have on top of 1 and 2, not a substitute - a pane you must watch does not solve "nothing told the agent to tell me." ## Acceptance - A deferred-dispatch storm, a reservation double-dispatch, or a `/goal` runaway surfaces (a) in the director agent's capability-detected status line within a turn, and (b) as a Telegram/ntfy alert, with no one running `docker ps`. - Status-line injection is gated on ward capability detection and is a no-op for agents that do not support it. - The alert rules ride the existing SigNoz envelope stream + Telegram/ntfy plumbing, not a new pipeline. ## Context Directed by Kai. This is the observability half of ward#1069 (detect the jam, surface it) - the missing signal that would have turned tonight's spelunk into a 5-minute heads-up. Capability detection is the mechanism Kai flagged for the status-line injection (claude supports it today). Refs: ward#1069 (epic), ward#996 (the race that was invisible), ward#1073 (backpressure state to surface), deploy#23 (SigNoz agent-run envelope stream), the Telegram main-failure + ntfy channels.

coilyco-ops added the

headless

labels

2026-07-10 19:00:48 +00:00

coilyco-ops commented

2026-07-10 19:06:18 +00:00

Author

Member

Alert channel refinement (Kai): use OS-native notification APIs for the interactive operator, out-of-band for headless.

The out-of-band alert layer should prefer the operator's OS-native notification mechanism when the broker/pre-flight runs on an interactive desktop (which it does - ward runs on Kai's Mac under Warp, where the containers spawn), because a Notification Center toast reaches Kai instantly and in-context, more immediate than a Telegram round-trip:

macOS - built-in via osascript -e 'display notification ...' (or the UserNotifications framework / terminal-notifier for richer toasts).
Windows - built-in WinRT toast (Windows.UI.Notifications, e.g. via PowerShell / BurntToast).
Linux desktop - the equivalent is notify-send (libnotify over the D-Bus org.freedesktop.Notifications daemon), but fleet Linux here is headless (servers / control node / containers) with no desktop session or daemon, so this path is a no-op there.

Selection = capability detection (same mechanism as the status-line injection): detect OS + presence of an interactive desktop/notification daemon, and route accordingly - native toast on an interactive macOS/Windows (and desktop-Linux) operator, and fall back to the existing out-of-band channels (Telegram / ntfy / SigNoz alert rules) on headless Linux where no local user is watching. The two are complementary, not either-or: the native toast catches the operator at the machine, the out-of-band catches them when away.

**Alert channel refinement (Kai): use OS-native notification APIs for the interactive operator, out-of-band for headless.** The out-of-band alert layer should prefer the operator's **OS-native notification mechanism** when the broker/pre-flight runs on an interactive desktop (which it does - ward runs on Kai's Mac under Warp, where the containers spawn), because a Notification Center toast reaches Kai instantly and in-context, more immediate than a Telegram round-trip: - **macOS** - built-in via `osascript -e 'display notification ...'` (or the UserNotifications framework / terminal-notifier for richer toasts). - **Windows** - built-in WinRT toast (`Windows.UI.Notifications`, e.g. via PowerShell / BurntToast). - **Linux desktop** - the equivalent is `notify-send` (libnotify over the D-Bus `org.freedesktop.Notifications` daemon), **but** fleet Linux here is headless (servers / control node / containers) with no desktop session or daemon, so this path is a no-op there. **Selection = capability detection** (same mechanism as the status-line injection): detect OS + presence of an interactive desktop/notification daemon, and route accordingly - native toast on an interactive macOS/Windows (and desktop-Linux) operator, and fall back to the existing out-of-band channels (Telegram / ntfy / SigNoz alert rules) on headless Linux where no local user is watching. The two are complementary, not either-or: the native toast catches the operator at the machine, the out-of-band catches them when away.

coilyco-ops commented

2026-07-10 22:58:03 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-1076 on host KAI-DESKTOP-TOWER.

Reserved by ward agent --harness codex (reserved 2026-07-10T22:58:04Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1076 · branch issue-1076 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-ward-1076 · ward v0.605.0 · dispatched 2026-07-10T22:58:04Z
Comment thread: 1 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T19:06:18Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.605.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-1076` on host `KAI-DESKTOP-TOWER`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T22:58:04Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1076` · branch `issue-1076` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-ward-1076` · ward `v0.605.0` · dispatched `2026-07-10T22:58:04Z` - **Comment thread:** 1 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T19:06:18Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.605.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 23:26:40 +00:00

Surface broker/pre-flight dispatch pathology by default: capability-detected status-line injection + out-of-band alerts #1101

coilyco-ops commented

2026-07-10 23:28:14 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-request-and-merge; review summary: review gate intentionally skipped because the temporary ward default pending brokered QA.

felt: straightforward, but the health tally needed a lint-driven split.
confidence: high.
surprises: Forgejo exposed the PR immediately, while mergeability lagged behind the status check.
follow-ups: none.

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-request-and-merge; review summary: review gate intentionally skipped because the temporary ward default pending brokered QA. felt: straightforward, but the health tally needed a lint-driven split. confidence: high. surprises: Forgejo exposed the PR immediately, while mergeability lagged behind the status check. follow-ups: none. </details>

coilyco-ops commented

2026-07-10 23:30:41 +00:00

Author

Member

WARD-DISPATCH: failed ❌

failure details

This forwarded dispatch failed after the issue was already reserved.

Attempted harness: codex
Attempted run: ward agent engineer coilyco-flight-deck/ward#1076 --harness codex --ward-version v0.605.0 --details PR #1101 is green but Forgejo reports mergeable=false for head issue-1076. Refresh or replace the branch, preserve the broker/pre-flight dispatch pathology status-line and alerting intent, get checks green, and land the issue end to end.
Container: engineer-codex-ward-1076
Container created: no running engineer was observed.
Host log: C:\Users\firem\.ward\agent-logs\dispatch\20260710T233026Z-director-codex-yq57-coilyco-flight-deck-ward-1076.log
Failure: ward agent engineer --harness codex: issue coilyco-flight-deck/ward#1076 is already reserved remotely (by @coilyco-ops at 2026-07-10T22:58:03Z); wait for it to finish or pass --force to override

Retry: choose another harness if the first one is down, or rerun with --force if the reservation is stale.

— Codex, via ward agent

WARD-DISPATCH: failed ❌ <details><summary>failure details</summary> This forwarded dispatch failed after the issue was already reserved. Attempted harness: `codex` Attempted run: `ward agent engineer coilyco-flight-deck/ward#1076 --harness codex --ward-version v0.605.0 --details PR #1101 is green but Forgejo reports mergeable=false for head issue-1076. Refresh or replace the branch, preserve the broker/pre-flight dispatch pathology status-line and alerting intent, get checks green, and land the issue end to end.` Container: `engineer-codex-ward-1076` Container created: no running engineer was observed. Host log: `C:\Users\firem\.ward\agent-logs\dispatch\20260710T233026Z-director-codex-yq57-coilyco-flight-deck-ward-1076.log` Failure: `ward agent engineer --harness codex: issue coilyco-flight-deck/ward#1076 is already reserved remotely (by @coilyco-ops at 2026-07-10T22:58:03Z); wait for it to finish or pass --force to override` Retry: choose another harness if the first one is down, or rerun with `--force` if the reservation is stale. </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-11 00:14:03 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-1076 on host KAI-DESKTOP-TOWER.

Reserved by ward agent --harness codex (reserved 2026-07-11T00:14:03Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#1076 · branch issue-1076 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-ward-1076 · ward v0.605.0 · dispatched 2026-07-11T00:14:03Z
Comment thread: 3 included in the pre-flight read, 1 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T19:06:18Z), @coilyco-ops (2026-07-10T23:28:14Z), @coilyco-ops (2026-07-10T23:30:41Z)
- stripped: @coilyco-ops (2026-07-10T22:58:03Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.605.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-1076` on host `KAI-DESKTOP-TOWER`. Reserved by `ward agent --harness codex` (reserved 2026-07-11T00:14:03Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#1076` · branch `issue-1076` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-ward-1076` · ward `v0.605.0` · dispatched `2026-07-11T00:14:03Z` - **Comment thread:** 3 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T19:06:18Z), @coilyco-ops (2026-07-10T23:28:14Z), @coilyco-ops (2026-07-10T23:30:41Z) - stripped: @coilyco-ops (2026-07-10T22:58:03Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.605.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops closed this issue

2026-07-11 00:23:19 +00:00

coilyco-ops commented

2026-07-11 00:23:35 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-request-and-merge; review summary: review gate intentionally skipped because the temporary ward default pending brokered QA.

felt: straightforward. The merge only needed a docs conflict fix plus a small dispatch-health API swap after main advanced.
confidence: high.
surprises: main had moved the workflow and list wording, so I had to reconcile that before the branch could go green again.
follow-ups: none.

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-request-and-merge; review summary: review gate intentionally skipped because the temporary ward default pending brokered QA. felt: straightforward. The merge only needed a docs conflict fix plus a small dispatch-health API swap after main advanced. confidence: high. surprises: `main` had moved the workflow and list wording, so I had to reconcile that before the branch could go green again. follow-ups: none. </details>