Make gh available in the dev-base surface used by warded directors #395

New issue

Closed

opened 2026-07-09 23:08:28 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-09 23:08:28 +00:00

Member

Goal

Make gh available by default in the aos dev-base surface that warded director/Codex director containers actually run, so GitHub auth can be debugged without ad hoc per-session installs.

Why

During coilysiren/website#70 triage, the director needed to work through GitHub auth. Plain GitHub git reads worked, and the GitHub connector had API access, but local gh was missing:

/bin/bash: line 1: gh: command not found

The agent installed gh into /scratch/bin as a temporary workaround, but Kai clarified the desired fix is in the aos dev-base image.

Evidence from the read-only surface

Environment:

Repo surface: /workspace/website
Container: WARD_CONTAINER_NAME=director-codex-hd68
Role: WARD_ROLE=director
Harness: WARD_AGENT=codex
Ward: ward version v0.540.0
PATH: /home/ubuntu/.codex/tmp/...:/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin
/usr/local/bin/docker exists, but Docker daemon/socket was unavailable from this surface.
/usr/local/bin/gh and /usr/bin/gh were absent.

Aos source already appears to expect gh in dev-base:

docker/dev-base/Dockerfile has ARG GH_VERSION=2.96.0.
dev-base-ops maps GH_ARCH for amd64/arm64.
The fleet CI utilities block downloads gh_${GH_VERSION}_linux_${GH_ARCH}.tar.gz, extracts bin/gh to /usr/local/bin/gh, and runs gh --version.
docs/dev-base-image.md documents gh + helm + kubectl + yq.
scripts/dep-bump.py and tests/test_dep_bump.py assign GH_VERSION to dev-base-ops.

Likely shape

This may not be a missing Dockerfile install anymore. It may be one of:

the published dev-base image used by warded director is stale or not the expected target,
the Codex director surface uses a lighter image than dev-base-ops/dev-base-full,
ward image selection points at a tag/ref that does not include the current aos dev-base contract,
or the Dockerfile install path is correct but not present in the actually launched image.

Acceptance criteria

A fresh warded director Codex surface can run gh --version without installing anything at runtime.
If the fix belongs in aos, land the Dockerfile/docs/test change and let the publish-image workflow rebuild.
If the fix belongs in ward image selection, file the ward follow-up with exact evidence and link it here.
Do not bake GitHub credentials into the image. This issue is only about the gh binary being present.

Temporary workaround used in the blocked session

The agent downloaded GitHub CLI v2.96.0 for linux_arm64, verified the official checksum, and symlinked it into the session PATH from /scratch/bin/gh. That proved the binary itself works in the current container architecture, but it is ephemeral and not a fix.

## Goal Make `gh` available by default in the aos dev-base surface that `warded director`/Codex director containers actually run, so GitHub auth can be debugged without ad hoc per-session installs. ## Why During `coilysiren/website#70` triage, the director needed to work through GitHub auth. Plain GitHub git reads worked, and the GitHub connector had API access, but local `gh` was missing: ```text /bin/bash: line 1: gh: command not found ``` The agent installed `gh` into `/scratch/bin` as a temporary workaround, but Kai clarified the desired fix is in the aos dev-base image. ## Evidence from the read-only surface Environment: * Repo surface: `/workspace/website` * Container: `WARD_CONTAINER_NAME=director-codex-hd68` * Role: `WARD_ROLE=director` * Harness: `WARD_AGENT=codex` * Ward: `ward version v0.540.0` * PATH: `/home/ubuntu/.codex/tmp/...:/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin` * `/usr/local/bin/docker` exists, but Docker daemon/socket was unavailable from this surface. * `/usr/local/bin/gh` and `/usr/bin/gh` were absent. Aos source already appears to expect `gh` in dev-base: * `docker/dev-base/Dockerfile` has `ARG GH_VERSION=2.96.0`. * `dev-base-ops` maps `GH_ARCH` for amd64/arm64. * The fleet CI utilities block downloads `gh_${GH_VERSION}_linux_${GH_ARCH}.tar.gz`, extracts `bin/gh` to `/usr/local/bin/gh`, and runs `gh --version`. * `docs/dev-base-image.md` documents `gh + helm + kubectl + yq`. * `scripts/dep-bump.py` and `tests/test_dep_bump.py` assign `GH_VERSION` to `dev-base-ops`. ## Likely shape This may not be a missing Dockerfile install anymore. It may be one of: * the published dev-base image used by `warded director` is stale or not the expected target, * the Codex director surface uses a lighter image than `dev-base-ops`/`dev-base-full`, * ward image selection points at a tag/ref that does not include the current aos dev-base contract, * or the Dockerfile install path is correct but not present in the actually launched image. ## Acceptance criteria * A fresh `warded director` Codex surface can run `gh --version` without installing anything at runtime. * If the fix belongs in aos, land the Dockerfile/docs/test change and let the publish-image workflow rebuild. * If the fix belongs in ward image selection, file the ward follow-up with exact evidence and link it here. * Do not bake GitHub credentials into the image. This issue is only about the `gh` binary being present. ## Temporary workaround used in the blocked session The agent downloaded GitHub CLI `v2.96.0` for `linux_arm64`, verified the official checksum, and symlinked it into the session PATH from `/scratch/bin/gh`. That proved the binary itself works in the current container architecture, but it is ephemeral and not a fix.

coilyco-ops commented

2026-07-09 23:08:38 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-agentic-os-395 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T23:08:36Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/agentic-os#395 · branch issue-395 · harness codex · workflow direct-main
Run: engineer-codex-agentic-os-395 · ward v0.540.0 · dispatched 2026-07-09T23:08:36Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.540.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-agentic-os-395` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T23:08:36Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/agentic-os#395` · branch `issue-395` · harness `codex` · workflow `direct-main` - **Run:** `engineer-codex-agentic-os-395` · ward `v0.540.0` · dispatched `2026-07-09T23:08:36Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.540.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-09 23:19:00 +00:00

fix(dev-base): refresh pinned image manifest for gh

coilyco-ops closed this issue

2026-07-09 23:19:00 +00:00

coilyco-ops commented

2026-07-09 23:19:37 +00:00

Author

Member

WARD-OUTCOME: done

details

workflow: direct-to-main; review summary: review gate skipped by ~/.ward/config.yaml default retrospective: this felt like a stale pin refresh, not a Dockerfile chase confidence: high surprises: gh was already baked into the image contract, but the consumed tag had drifted follow-ups: none

WARD-OUTCOME: done <details><summary>details</summary> workflow: direct-to-main; review summary: review gate skipped by ~/.ward/config.yaml default retrospective: this felt like a stale pin refresh, not a Dockerfile chase confidence: high surprises: gh was already baked into the image contract, but the consumed tag had drifted follow-ups: none </details>