coilyco-flight-deck/agentic-os

Fork 0

Right-size director/advisor container surface tiers after the 2026-07-10 permissions rollback #447

New issue

Closed

opened 2026-07-10 18:03:22 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-10 18:03:22 +00:00

Member

Problem

A permissions rollback on 2026-07-10 shrank the director/advisor container surface tiers well below what those roles operationally need (Kai: "I accidentally requested a permissions rollback much greater than the current operational needs"). Probed live from a running director surface during the infrastructure#538 recovery, the surface was missing capabilities its role is supposed to carry:

No kubectl - the ward ops kubectl guardfile is compiled in, but the binary is absent (/usr/local/bin/kubectl not found) and there is no kubeconfig, so no cluster reach even though guardfile.kubectl.kdl exists.
No tailnet - tailscaled is not running in the surface, so tailscale status/MagicDNS fail. The advisor role is documented to carry the live-observe set (tailnet + ~/.aws) by default, and the advisor dispatched for infra#538 produced a runbook-for-a-human rather than live findings - consistent with the same strip hitting the advisor role.
No ~/.aws dir (SSM still authenticated via env, but the documented creds path is gone).
generate-runner-token not surfaced - ward ops forgejo actions exposed only logs, so the operator token-restore path (infrastructure#539) could not run from the surface.
No commit-status read / no merge - covered by the two dedicated issues below, not this one.

The net effect: the roster's stated role tiers (engineer/director/advisor/qa/ops in .ward/roles.kdl, plus the ward-embedded role presets and the container image + WARD_CONTEXT_LEVEL provisioning) drifted from intent, and nothing flagged it.

Do

Re-establish the intended per-role surface tiers as a deliberate, reviewed definition, not an accidental residue. For each role (director, advisor, engineer, qa, ops) state what it should carry: forgejo read/write, actions read, aws/ssm, kubectl, tailnet live-observe, admin verbs.
Restore the specifically-stripped, operationally-needed reach: the advisor live-observe set (tailnet + ~/.aws) per its documented role, and ward ops forgejo actions generate-runner-token on the operator/ops surface (its absence blocked the infra#539 token restore).
Reconcile the split across layers: .ward/roles.kdl guardfile bindings (dialect-1 surface), the ward-embedded role presets, and the container image / WARD_CONTEXT_LEVEL provisioning that decides which binaries (kubectl, tailscale) and creds land in a given surface. Note which layer owns each capability so a future rollback of one layer cannot silently disable a role.
Add a surface self-check the role can run to assert its expected capabilities are present (fail loud on drift), so an over-broad rollback is caught at bring-up, not mid-incident.

Acceptance

Each role's intended surface tier is written down and the live container matches it (an advisor surface has tailnet + ~/.aws; an ops/director surface can mint a runner token).
A capability drift (a role missing an expected verb/binary) is detectable at surface bring-up rather than discovered by hand during an incident.
Merge + commit-status are explicitly deferred to their own issues (foundational: the ward-native PR-workflow issue; convenience: agentic-os#446) and not duplicated here.

Context

Filed at Kai's request after the 2026-07-10 rollback surfaced during the infrastructure#538 recovery. This is the tier right-size half; the merge/status half splits into the ward-native foundational capability and agentic-os#446's specgen also-grant. Refs: .ward/roles.kdl, .ward/guardfile.kubectl.kdl, .ward/guardfile.aws.kdl, .ward/guardfile.forgejo.admin.kdl, docker/dev-base/ops/Dockerfile (kubectl install), infrastructure#538, infrastructure#539, agentic-os#446.

## Problem A permissions rollback on 2026-07-10 shrank the director/advisor container surface tiers well below what those roles operationally need (Kai: "I accidentally requested a permissions rollback much greater than the current operational needs"). Probed live from a running director surface during the infrastructure#538 recovery, the surface was missing capabilities its role is supposed to carry: - **No `kubectl`** - the `ward ops kubectl` guardfile is compiled in, but the binary is absent (`/usr/local/bin/kubectl` not found) and there is no kubeconfig, so no cluster reach even though `guardfile.kubectl.kdl` exists. - **No tailnet** - `tailscaled` is not running in the surface, so `tailscale status`/MagicDNS fail. The `advisor` role is documented to carry the live-observe set (tailnet + `~/.aws`) by default, and the advisor dispatched for infra#538 produced a runbook-for-a-human rather than live findings - consistent with the same strip hitting the advisor role. - **No `~/.aws` dir** (SSM still authenticated via env, but the documented creds path is gone). - **`generate-runner-token` not surfaced** - `ward ops forgejo actions` exposed only `logs`, so the operator token-restore path (infrastructure#539) could not run from the surface. - **No commit-status read / no merge** - covered by the two dedicated issues below, not this one. The net effect: the roster's stated role tiers (`engineer`/`director`/`advisor`/`qa`/`ops` in `.ward/roles.kdl`, plus the ward-embedded role presets and the container image + `WARD_CONTEXT_LEVEL` provisioning) drifted from intent, and nothing flagged it. ## Do - Re-establish the intended per-role surface tiers as a **deliberate, reviewed** definition, not an accidental residue. For each role (`director`, `advisor`, `engineer`, `qa`, `ops`) state what it should carry: forgejo read/write, actions read, aws/ssm, kubectl, tailnet live-observe, admin verbs. - Restore the specifically-stripped, operationally-needed reach: the `advisor` live-observe set (tailnet + `~/.aws`) per its documented role, and `ward ops forgejo actions generate-runner-token` on the operator/ops surface (its absence blocked the infra#539 token restore). - Reconcile the split across layers: `.ward/roles.kdl` guardfile bindings (dialect-1 surface), the ward-embedded role presets, and the container image / `WARD_CONTEXT_LEVEL` provisioning that decides which binaries (`kubectl`, `tailscale`) and creds land in a given surface. Note which layer owns each capability so a future rollback of one layer cannot silently disable a role. - Add a surface self-check the role can run to assert its expected capabilities are present (fail loud on drift), so an over-broad rollback is caught at bring-up, not mid-incident. ## Acceptance - Each role's intended surface tier is written down and the live container matches it (an `advisor` surface has tailnet + `~/.aws`; an `ops`/director surface can mint a runner token). - A capability drift (a role missing an expected verb/binary) is detectable at surface bring-up rather than discovered by hand during an incident. - Merge + commit-status are explicitly deferred to their own issues (foundational: the ward-native PR-workflow issue; convenience: agentic-os#446) and not duplicated here. ## Context Filed at Kai's request after the 2026-07-10 rollback surfaced during the infrastructure#538 recovery. This is the **tier right-size** half; the **merge/status** half splits into the ward-native foundational capability and agentic-os#446's specgen also-grant. Refs: `.ward/roles.kdl`, `.ward/guardfile.kubectl.kdl`, `.ward/guardfile.aws.kdl`, `.ward/guardfile.forgejo.admin.kdl`, `docker/dev-base/ops/Dockerfile` (kubectl install), infrastructure#538, infrastructure#539, agentic-os#446.

coilyco-ops added the

headless

labels

2026-07-10 18:03:22 +00:00

coilyco-ops commented

2026-07-10 18:03:54 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-claude-agentic-os-447 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness claude (reserved 2026-07-10T18:03:53Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/agentic-os#447 · branch issue-447 · harness claude · workflow pull-requests-and-merge
Run: engineer-claude-agentic-os-447 · ward v0.584.0 · dispatched 2026-07-10T18:03:53Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0).

— Claude (she/her), via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-claude-agentic-os-447` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness claude` (reserved 2026-07-10T18:03:53Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/agentic-os#447` · branch `issue-447` · harness `claude` · workflow `pull-requests-and-merge` - **Run:** `engineer-claude-agentic-os-447` · ward `v0.584.0` · dispatched `2026-07-10T18:03:53Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.584.0). </details> </details>  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-10 18:18:06 +00:00

feat(ward): right-size role surface tiers after the 2026-07-10 rollback

coilyco-ops referenced this issue from a commit

2026-07-10 18:24:34 +00:00

feat(ward): add tailnet live-observe guardfile and bind it per role

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-10 18:25:12 +00:00

Right-size director/advisor container surface tiers after the 2026-07-10 rollback #448

coilyco-ops referenced this issue from a commit

2026-07-10 18:31:51 +00:00

fix: merge main into issue-447; package new bundle files in the explicit ward-specs tar

coilyco-ops commented

2026-07-10 18:33:38 +00:00

Author

Member

WARD-OUTCOME: merge-ready

details

workflow: pull-requests-and-merge; review summary: skipped - the in-container review gate was intentionally skipped by the ~/.ward/config.yaml default.

PR: #448 (branch issue-447, head 72727fd, ci / gate green, mergeable). The director merges it and records the final done outcome.

What landed

docs/role-surface-tiers.md - the deliberate per-role tier definition plus the per-layer capability ownership map (roles.kdl bindings / ward-embedded presets / dev-base image binaries / ward container bring-up).
roles.kdl rebind - advisor regains its live-observe set (aws + new tailscale guardfile), director gains live-observe + kubectl + runner-token mint, ops gains runner-token mint.
guardfile.forgejo.runnertoken.kdl + forgejo-runner-token.sh - ward ops forgejo actions generate-runner-token (global/org/repo scopes), the infrastructure#539 restore path, kept out of the fleet-wide readactions overlay.
guardfile.tailscale.kdl - read-only tailnet observe surface (status/ping/netcheck/ip/whois/dns status/version), state-changing verbs denied.
surface-check.sh + ward exec surface-check - bring-up self-check keyed on WARD_ROLE, exits non-zero listing every miss. Run as director from this engineer container it reproduced the exact drift this issue described (5 misses).
release.yml tar list + check_ward_specs_bundle drift-guard extended so the four new bundle files ship in ward-specs and cannot silently drop.

Verified - ward doctor PASS (fleet + exec bundle) against the branch bundle via WARD_CONFIG_REF, pre-commit all-files green, pytest 455 passed, CI green after the engineer merged main in and resolved the #437 ward-specs packaging conflicts.

How it felt - the archaeology was the work: the rollback turned out to be layered (roles.kdl preset strip at 09:10, plus ward-side bring-up loss), and the shipped ward binary's strings were the only source for the ward#578 per-role guardfile-set behavior. The aos-side authoring was straightforward once the layer map existed.

Surprises

Shipped ward (v0.584.0) mounts only the hardcoded guardfile.forgejo.kdl monolith for ward ops forgejo (actions logs is ward-native Go), so the runnertoken overlay - like the existing readactions runs/tasks leaves - is authored intent until ward mounts role-bound overlays. Filed ward#1072 for that wiring plus invoking surface-check at bring-up.
ward doctor on current main already fails its ops-bundle check with this ward version (#437 removed the committed swagger lock the shipped binary still wants). Pre-existing on main, not from this branch.
The host /gitcache disk sits at 100%. Config-bundle clones failed mid-run until the agent removed its own cache dirs. Worth a look.

Confidence - high on the aos-side definition, bindings, and self-check (all exercised live). Medium on how ward bring-up consumes the tailscale/aws bindings for daemon/creds provisioning - that half of the acceptance closes via ward#1072.

Follow-ups - ward#1072 (runtime overlay mounts, bring-up self-check, provisioning reconcile). Merge + commit-status stay deferred to the ward-native PR-workflow issue and #446, per this issue.

— Claude (she/her), via ward agent

WARD-OUTCOME: merge-ready <details><summary>details</summary> workflow: pull-requests-and-merge; review summary: skipped - the in-container review gate was intentionally skipped by the `~/.ward/config.yaml` default. **PR:** #448 (branch `issue-447`, head `72727fd`, `ci / gate` green, mergeable). The director merges it and records the final done outcome. **What landed** * `docs/role-surface-tiers.md` - the deliberate per-role tier definition plus the per-layer capability ownership map (roles.kdl bindings / ward-embedded presets / dev-base image binaries / ward container bring-up). * `roles.kdl` rebind - advisor regains its live-observe set (aws + new tailscale guardfile), director gains live-observe + kubectl + runner-token mint, ops gains runner-token mint. * `guardfile.forgejo.runnertoken.kdl` + `forgejo-runner-token.sh` - `ward ops forgejo actions generate-runner-token` (global/org/repo scopes), the infrastructure#539 restore path, kept out of the fleet-wide readactions overlay. * `guardfile.tailscale.kdl` - read-only tailnet observe surface (status/ping/netcheck/ip/whois/dns status/version), state-changing verbs denied. * `surface-check.sh` + `ward exec surface-check` - bring-up self-check keyed on `WARD_ROLE`, exits non-zero listing every miss. Run as director from this engineer container it reproduced the exact drift this issue described (5 misses). * `release.yml` tar list + `check_ward_specs_bundle` drift-guard extended so the four new bundle files ship in ward-specs and cannot silently drop. **Verified** - `ward doctor` PASS (fleet + exec bundle) against the branch bundle via `WARD_CONFIG_REF`, pre-commit all-files green, pytest 455 passed, CI green after the engineer merged main in and resolved the #437 ward-specs packaging conflicts. **How it felt** - the archaeology was the work: the rollback turned out to be layered (roles.kdl preset strip at 09:10, plus ward-side bring-up loss), and the shipped ward binary's strings were the only source for the ward#578 per-role guardfile-set behavior. The aos-side authoring was straightforward once the layer map existed. **Surprises** * Shipped ward (v0.584.0) mounts only the hardcoded `guardfile.forgejo.kdl` monolith for `ward ops forgejo` (`actions logs` is ward-native Go), so the runnertoken overlay - like the existing readactions runs/tasks leaves - is authored intent until ward mounts role-bound overlays. Filed ward#1072 for that wiring plus invoking surface-check at bring-up. * `ward doctor` on current main already fails its ops-bundle check with this ward version (#437 removed the committed swagger lock the shipped binary still wants). Pre-existing on main, not from this branch. * The host `/gitcache` disk sits at 100%. Config-bundle clones failed mid-run until the agent removed its own cache dirs. Worth a look. **Confidence** - high on the aos-side definition, bindings, and self-check (all exercised live). Medium on how ward bring-up consumes the tailscale/aws bindings for daemon/creds provisioning - that half of the acceptance closes via ward#1072. **Follow-ups** - ward#1072 (runtime overlay mounts, bring-up self-check, provisioning reconcile). Merge + commit-status stay deferred to the ward-native PR-workflow issue and #446, per this issue. </details>  — Claude (she/her), via `ward agent`

coilyco-ops referenced this issue

2026-07-10 18:51:05 +00:00

Replace .ward/forgejo-runner-token.sh raw-curl exec-bridge with a first-class specverb fetch overlay (cli-guard#216) #453

coilyco-ops commented

2026-07-10 23:05:51 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-agentic-os-447 on host KAI-DESKTOP-TOWER.

Reserved by ward agent --harness codex (reserved 2026-07-10T23:05:51Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/agentic-os#447 · branch issue-447 · harness codex · workflow pull-requests-and-merge
Run: engineer-codex-agentic-os-447 · ward v0.605.0 · dispatched 2026-07-10T23:05:51Z
Comment thread: 1 included in the pre-flight read, 1 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-10T18:33:38Z)
- stripped: @coilyco-ops (2026-07-10T18:03:54Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.605.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-agentic-os-447` on host `KAI-DESKTOP-TOWER`. Reserved by `ward agent --harness codex` (reserved 2026-07-10T23:05:51Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/agentic-os#447` · branch `issue-447` · harness `codex` · workflow `pull-requests-and-merge` - **Run:** `engineer-codex-agentic-os-447` · ward `v0.605.0` · dispatched `2026-07-10T23:05:51Z` - **Comment thread:** 1 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-10T18:33:38Z) - stripped: @coilyco-ops (2026-07-10T18:03:54Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.605.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops closed this issue

2026-07-10 23:18:35 +00:00

coilyco-ops commented

2026-07-10 23:20:04 +00:00

Author

Member

WARD-OUTCOME: submitted

details

workflow: pull-request-and-merge; review summary: skipped - the in-container review gate was intentionally skipped by the temporary ward default pending brokered QA.

PR #448 is green and mergeable after the branch refresh.

What landed felt layered, not broad. The merge had to preserve the existing merge overlay from main while refreshing the director/advisor surface-tier fix and updating the bundle contract tests.

confidence: high

surprises: the flattened ward bundle had to keep the merge overlay in the packaged set once the role binding stayed on director.

follow-ups: merge PR #448 when the director lane takes it.

WARD-OUTCOME: submitted <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped - the in-container review gate was intentionally skipped by the temporary ward default pending brokered QA. PR #448 is green and mergeable after the branch refresh. What landed felt layered, not broad. The merge had to preserve the existing merge overlay from main while refreshing the director/advisor surface-tier fix and updating the bundle contract tests. confidence: high surprises: the flattened ward bundle had to keep the merge overlay in the packaged set once the role binding stayed on director. follow-ups: merge PR #448 when the director lane takes it. </details>