coilyco-flight-deck/agentic-os

Fork 0

chore(goose): decide disposition of scripts/goose-health.sh probe #268

New issue

Closed

opened 2026-06-23 08:29:35 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-06-23 08:29:35 +00:00

Member

An uncommitted helper scripts/goose-health.sh was created during a goose-slowness investigation and is sitting in the working tree. It is a one-shot health probe of the Goose serving stack (Ollama on the tower over tailnet): reachability + latency via /api/version, and resident-model + expires_at via /api/ps. It sits next to the existing goose-* family (goose-ask, goose-json, goose-triage).

Decide its disposition:

Commit as a standing helper - add a goose-health ward verb and a docs/FEATURES.md line, matching the goose-* family pattern, OR
Fold its probe logic into the agent-health heartbeat (see the infrastructure heartbeat issue) and drop the standalone script.

Context: the goose "slowness" root cause was VRAM contention on the tower - the bound model qwen3-coder:30b gets evicted by another resident model (deepseek-v2:16b), so every goose call cold-loads. The probe surfaces exactly that: which model is resident, and a marching expires_at meaning an active job is re-warming it.

Related: the agent-health heartbeat issue in infrastructure (the always-on version of this probe, emitting to SigNoz).

An uncommitted helper `scripts/goose-health.sh` was created during a goose-slowness investigation and is sitting in the working tree. It is a one-shot health probe of the Goose serving stack (Ollama on the tower over tailnet): reachability + latency via `/api/version`, and resident-model + `expires_at` via `/api/ps`. It sits next to the existing `goose-*` family (`goose-ask`, `goose-json`, `goose-triage`). **Decide its disposition:** - **Commit as a standing helper** - add a `goose-health` ward verb and a `docs/FEATURES.md` line, matching the `goose-*` family pattern, OR - **Fold its probe logic into the agent-health heartbeat** (see the infrastructure heartbeat issue) and drop the standalone script. **Context:** the goose "slowness" root cause was VRAM contention on the tower - the bound model `qwen3-coder:30b` gets evicted by another resident model (`deepseek-v2:16b`), so every goose call cold-loads. The probe surfaces exactly that: which model is resident, and a marching `expires_at` meaning an active job is re-warming it. **Related:** the agent-health heartbeat issue in `infrastructure` (the always-on version of this probe, emitting to SigNoz).

coilyco-ops referenced this issue from coilyco-flight-deck/infrastructure

2026-06-23 08:30:19 +00:00

feat(observability): agent-health heartbeat emitting OTLP to SigNoz #389

coilyco-ops added the

consult

labels

2026-06-24 06:12:57 +00:00

coilyco-ops commented

2026-06-24 06:16:31 +00:00

Author

Member

Preserving the working-tree probe here verbatim before deleting it from disk - as written it likely won't land (it would need a goose-health ward verb + a docs/FEATURES.md line to match the goose-* family, or folding into the agent-health heartbeat per the options above). Captured so the disposition decision still has the artifact after the file is gone.

#!/usr/bin/env bash
# goose-health: one-shot health snapshot of the Goose serving stack. Goose binds
# to a model served by Ollama on the tower over tailnet, so "goose is slow/down"
# is almost always this layer, not the harness. Probes reachability, latency, and
# - the diagnostic signal - which model is actually resident in VRAM right now.
# No SSH: hits Ollama's HTTP API, reachable from any tailnet host.
set -euo pipefail

# Tailnet hostname (meaningful name, not an opaque id) so it hardcodes fine.
host="${OLLAMA_HOST:-http://kai-tower-3026:11434}"
bound_model="${GOOSE_MODEL:-qwen3-coder:30b}"

echo "host=$host  bound=$bound_model"
echo

echo "== reachability + latency (/api/version) =="
curl -s -o /tmp/goose-health.ver -w 'connect=%{time_connect}s total=%{time_total}s http=%{http_code}\n' \
  --max-time 8 "$host/api/version" || { echo "UNREACHABLE - tower or tailnet down"; exit 1; }
echo "  version: $(cat /tmp/goose-health.ver)"
echo

echo "== resident models (/api/ps) =="
echo "   bound model resident + warm = fast. Absent = next goose call cold-loads."
echo "   expires_at marching forward on re-run = something is actively hitting it."
curl -s --max-time 8 "$host/api/ps" 2>/dev/null | python3 -c '
import sys, json
d = json.load(sys.stdin)
ms = d.get("models", [])
if not ms:
    print("   (nothing resident - VRAM idle)")
for m in ms:
    name = m.get("name", "?")
    gb = m.get("size_vram", 0) / 1e9
    exp = m.get("expires_at", "?")
    print("   %-22s %5.1fGB vram  expires_at=%s" % (name, gb, exp))
' || echo "   /api/ps unavailable"
echo

echo "Recent goose call timings: ~/.cache/agentic-os/goose-ask/  goose-triage/"
echo "Deeper (which client is re-warming, raw VRAM): ssh the tower, 'ollama ps' / 'nvidia-smi' / server logs."

— claude-macos-kais-macbook-pro-2-df00-she-her

Preserving the working-tree probe here verbatim before deleting it from disk - as written it likely won't land (it would need a `goose-health` ward verb + a `docs/FEATURES.md` line to match the `goose-*` family, or folding into the agent-health heartbeat per the options above). Captured so the disposition decision still has the artifact after the file is gone. ```bash #!/usr/bin/env bash # goose-health: one-shot health snapshot of the Goose serving stack. Goose binds # to a model served by Ollama on the tower over tailnet, so "goose is slow/down" # is almost always this layer, not the harness. Probes reachability, latency, and # - the diagnostic signal - which model is actually resident in VRAM right now. # No SSH: hits Ollama's HTTP API, reachable from any tailnet host. set -euo pipefail # Tailnet hostname (meaningful name, not an opaque id) so it hardcodes fine. host="${OLLAMA_HOST:-http://kai-tower-3026:11434}" bound_model="${GOOSE_MODEL:-qwen3-coder:30b}" echo "host=$host bound=$bound_model" echo echo "== reachability + latency (/api/version) ==" curl -s -o /tmp/goose-health.ver -w 'connect=%{time_connect}s total=%{time_total}s http=%{http_code}\n' \ --max-time 8 "$host/api/version" || { echo "UNREACHABLE - tower or tailnet down"; exit 1; } echo " version: $(cat /tmp/goose-health.ver)" echo echo "== resident models (/api/ps) ==" echo " bound model resident + warm = fast. Absent = next goose call cold-loads." echo " expires_at marching forward on re-run = something is actively hitting it." curl -s --max-time 8 "$host/api/ps" 2>/dev/null | python3 -c ' import sys, json d = json.load(sys.stdin) ms = d.get("models", []) if not ms: print(" (nothing resident - VRAM idle)") for m in ms: name = m.get("name", "?") gb = m.get("size_vram", 0) / 1e9 exp = m.get("expires_at", "?") print(" %-22s %5.1fGB vram expires_at=%s" % (name, gb, exp)) ' || echo " /api/ps unavailable" echo echo "Recent goose call timings: ~/.cache/agentic-os/goose-ask/ goose-triage/" echo "Deeper (which client is re-warming, raw VRAM): ssh the tower, 'ollama ps' / 'nvidia-smi' / server logs." ``` — claude-macos-kais-macbook-pro-2-df00-she-her

coilysiren added

interactive

and removed

consult

labels

2026-07-01 04:29:20 +00:00

coilyco-ops commented

2026-07-09 21:46:23 +00:00

Author

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Recommendation: do not commit scripts/goose-health.sh as a standing agentic-os helper. Treat it as a prototype whose logic has already been successfully absorbed into the canonical infrastructure surface, then close this issue with the disposition "folded into agent-health heartbeat; no standalone goose-health verb."

Why this is the better disposition

The original script answered an infrastructure question, not a Goose-harness question. goose-ask, goose-json, and goose-triage are harness-facing tools. goose-health probes remote Ollama reachability, residency, and VRAM state on the tower over tailnet. That is the serving layer below Goose, so it fits infrastructure, not the goose-* family.
The always-on replacement is no longer hypothetical. coilyco-flight-deck/infrastructure#389 is closed and says the heartbeat landed on main with OTLP-to-SigNoz, a dashboard, a timer rollout, and the probe logic explicitly lifted from this prototype.
The landed heartbeat is strictly more capable than the shell helper. The prototype exposed /api/version, /api/ps, resident model names, size_vram, and expires_at. The shipped heartbeat now emits those signals continuously, adds spill detection (size_vram < size), turns expires_at into expires_in_seconds, and includes best-effort GPU/VRAM telemetry and a glance view.
Keeping both surfaces would create ownership drift. The same parsing and diagnosis logic would now live in two repos with two maintenance paths. That is exactly the kind of duplicated operator logic that goes stale under incident pressure.
Landing it as a verb in agentic-os has non-trivial carrying cost for weak payoff: ward verb, docs/FEATURES.md, likely a doc page, and ongoing expectation that agentic-os is the right place to debug tower/Ollama health.

Decision framing

Option A: commit as goose-health
- Pros: fast ad hoc CLI from an agentic-os checkout; human-readable one-shot output.
- Cons: wrong ownership layer, duplicates shipped infra logic, expands the goose-* surface with something that is not really a Goose harness verb, and invites drift between the ad hoc script and the canonical heartbeat.
Option B: fold into agent-health heartbeat and drop the script
- Pros: one source of truth, always-on telemetry, historical visibility in SigNoz, better fit to the serving-layer problem, and already implemented.
- Cons: a one-shot terminal snapshot is less direct unless the operator uses the infrastructure script's dry-run path or the dashboard.

Advisory conclusion

Choose Option B. The issue's underlying need was "answer goose slowness caused by tower/Ollama residency and VRAM contention quickly." The infrastructure heartbeat now does that in the right repo and at the right abstraction level. Shipping a second helper in agentic-os would mostly preserve an older debugging shape after the canonical solution already exists.

Open question worth calling out in the comment

The only real remaining gap is ergonomic, not architectural: whether Kai still wants a human-readable ad hoc snapshot in addition to SigNoz. If yes, that should be satisfied by the canonical infrastructure surface, for example by documenting or polishing the existing dry-run path (python3 scripts/agent-health-heartbeat.py --dry-run) there, not by resurrecting a second agentic-os script.

Risks / caveats

If SigNoz is down or the dashboard is stale, losing the shell helper can make first-hop debugging feel slower.
That risk is mitigated because the infrastructure repo already ships the canonical probe script and README verification path.
Reintroducing goose-health in agentic-os would create a larger long-term risk: two near-identical probes that can silently disagree.

Suggested close-out wording

Recommendation: drop scripts/goose-health.sh as a standalone agentic-os surface. Its logic has already landed in the canonical serving-layer heartbeat in infrastructure#389, which now emits the same core signals plus spill/GPU telemetry to SigNoz. goose-health would be the wrong ownership layer and would duplicate an already-shipped operator path. If a one-shot human-readable probe is still desired, add or document that as a dry-run/view mode of the infrastructure heartbeat, not as a new goose-* verb here.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- **Recommendation:** do **not** commit `scripts/goose-health.sh` as a standing `agentic-os` helper. Treat it as a prototype whose logic has already been successfully absorbed into the canonical infrastructure surface, then close this issue with the disposition "folded into agent-health heartbeat; no standalone `goose-health` verb." **Why this is the better disposition** - The original script answered an infrastructure question, not a Goose-harness question. `goose-ask`, `goose-json`, and `goose-triage` are harness-facing tools. `goose-health` probes remote Ollama reachability, residency, and VRAM state on the tower over tailnet. That is the serving layer below Goose, so it fits `infrastructure`, not the `goose-*` family. - The always-on replacement is no longer hypothetical. `coilyco-flight-deck/infrastructure#389` is closed and says the heartbeat landed on `main` with OTLP-to-SigNoz, a dashboard, a timer rollout, and the probe logic explicitly lifted from this prototype. - The landed heartbeat is strictly more capable than the shell helper. The prototype exposed `/api/version`, `/api/ps`, resident model names, `size_vram`, and `expires_at`. The shipped heartbeat now emits those signals continuously, adds spill detection (`size_vram < size`), turns `expires_at` into `expires_in_seconds`, and includes best-effort GPU/VRAM telemetry and a glance view. - Keeping both surfaces would create ownership drift. The same parsing and diagnosis logic would now live in two repos with two maintenance paths. That is exactly the kind of duplicated operator logic that goes stale under incident pressure. - Landing it as a verb in `agentic-os` has non-trivial carrying cost for weak payoff: ward verb, `docs/FEATURES.md`, likely a doc page, and ongoing expectation that `agentic-os` is the right place to debug tower/Ollama health. **Decision framing** - **Option A: commit as `goose-health`** - Pros: fast ad hoc CLI from an `agentic-os` checkout; human-readable one-shot output. - Cons: wrong ownership layer, duplicates shipped infra logic, expands the `goose-*` surface with something that is not really a Goose harness verb, and invites drift between the ad hoc script and the canonical heartbeat. - **Option B: fold into agent-health heartbeat and drop the script** - Pros: one source of truth, always-on telemetry, historical visibility in SigNoz, better fit to the serving-layer problem, and already implemented. - Cons: a one-shot terminal snapshot is less direct unless the operator uses the infrastructure script's dry-run path or the dashboard. **Advisory conclusion** Choose **Option B**. The issue's underlying need was "answer goose slowness caused by tower/Ollama residency and VRAM contention quickly." The infrastructure heartbeat now does that in the right repo and at the right abstraction level. Shipping a second helper in `agentic-os` would mostly preserve an older debugging shape after the canonical solution already exists. **Open question worth calling out in the comment** The only real remaining gap is ergonomic, not architectural: whether Kai still wants a **human-readable ad hoc snapshot** in addition to SigNoz. If yes, that should be satisfied by the canonical infrastructure surface, for example by documenting or polishing the existing dry-run path (`python3 scripts/agent-health-heartbeat.py --dry-run`) there, not by resurrecting a second `agentic-os` script. **Risks / caveats** - If SigNoz is down or the dashboard is stale, losing the shell helper can make first-hop debugging feel slower. - That risk is mitigated because the infrastructure repo already ships the canonical probe script and README verification path. - Reintroducing `goose-health` in `agentic-os` would create a larger long-term risk: two near-identical probes that can silently disagree. **Suggested close-out wording** > Recommendation: drop `scripts/goose-health.sh` as a standalone `agentic-os` surface. Its logic has already landed in the canonical serving-layer heartbeat in `infrastructure#389`, which now emits the same core signals plus spill/GPU telemetry to SigNoz. `goose-health` would be the wrong ownership layer and would duplicate an already-shipped operator path. If a one-shot human-readable probe is still desired, add or document that as a dry-run/view mode of the infrastructure heartbeat, not as a new `goose-*` verb here. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilysiren referenced this issue from coilyco-flight-deck/ward

2026-07-10 04:03:12 +00:00

ward release race condition #935

coilyco-ops added

headless

and removed

interactive

labels

2026-07-10 08:59:43 +00:00

coilyco-ops commented

2026-07-10 16:21:47 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-agentic-os-268 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-10T16:21:46Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/agentic-os#268 · branch issue-268 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-agentic-os-268 · ward v0.583.0 · dispatched 2026-07-10T16:21:46Z
Comment thread: 2 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-06-24T06:16:31Z), @coilyco-ops (2026-07-09T21:46:23Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.583.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-agentic-os-268` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T16:21:46Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/agentic-os#268` · branch `issue-268` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-agentic-os-268` · ward `v0.583.0` · dispatched `2026-07-10T16:21:46Z` - **Comment thread:** 2 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-06-24T06:16:31Z), @coilyco-ops (2026-07-09T21:46:23Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.583.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 16:25:26 +00:00

chore(goose): decide disposition of scripts/goose-health.sh probe #443

coilyco-ops commented

2026-07-10 16:30:43 +00:00

Author

Member

WARD-OUTCOME: merge-ready

details

workflow: pull-requests-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default
retrospective: this felt like fixing stale repo contracts more than changing Goose behavior.
confidence: high
surprises: the first CI failure came from stale ward-spec expectations, then the shell path seeding needed to stop assuming one checkout root.
follow-ups: none.

WARD-OUTCOME: merge-ready <details><summary>details</summary> workflow: pull-requests-and-merge; review summary: review gate skipped by ~/.ward/config.yaml default retrospective: this felt like fixing stale repo contracts more than changing Goose behavior. confidence: high surprises: the first CI failure came from stale ward-spec expectations, then the shell path seeding needed to stop assuming one checkout root. follow-ups: none. </details>

coilysiren closed this issue

2026-07-10 18:01:55 +00:00

coilysiren referenced this issue from coilyco-flight-deck/ward

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283