coilyco-flight-deck/agentic-os

Watch

#1349 breaks 65 existing aosguard ops kubectl call sites at rebuild, including two live-mutation scripts #1352

New issue

Open

opened 2026-08-28 19:02:52 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-08-28 19:02:52 +00:00

Owner

What #1349 did

--context is now mandatory on every aosguard ops kubectl leaf. Nothing changes until aosguard is rebuilt on a host, so this is latent, not live. At rebuild every call written without the flag starts failing closed:

aosguard: `get` denied: context did not match any allowed pattern [kai-server ser8] (fail-closed)

The extent

65 call sites across 13 files in coilyco-flight-deck/infrastructure, none carrying --context.

Two live-mutation scripts, which are the urgent part:

scripts/bootstrap-ser8-forgejo-runner-canary.sh - three aosguard ops kubectl apply -f - calls, executed on a remote host through a remote helper.
scripts/forgejo-external-account-lockdown-rollout.sh - aosguard ops kubectl diff -f and two apply -f calls.

Eleven ops-investigation runbooks, under .agents/skills/ops-investigation-*: disk-pressure, forgejo-runner-failure, k3s-connectivity, k3s-deploy-failure, k3s-pod-eviction. These carry concrete copy-paste recipes. They are loaded during an incident, which is exactly when a command failing for an unrelated-looking reason costs the most.

scripts/cleanup-orphaned-cluster-objects.sh mentions the wrap in a comment as an optional KUBECTL= value; its actual verb is delete, which the wrap never exposed, so it is unaffected in practice.

Not affected, contrary to the first guess

forgejo_storage_measure.py was flagged as the likely victim. It is not. It calls subprocess.run(("kubectl", ...)), spawning the binary directly from inside the forgejo-storage wrap, so it never crosses aosguard ops kubectl and no wrap-level guard reaches it. That is #1350, a different defect, and the two should not be conflated.

The breakage is mostly the guard working

Worth stating before anyone reaches for a revert. bootstrap-ser8-forgejo-runner-canary.sh targets ser8 by name and applies through remote onto a host whose active kubeconfig context is not guaranteed to be ser8. An unnamed target on a remote mutation is precisely the ambiguity #1349 exists to catch, so these two scripts are the strongest argument for naming the cluster rather than against it.

The runbooks are a genuine cost with no upside: they are read by an operator who already knows which cluster they are on.

Fixing it

Do not hardcode a context into runbook examples. Most read -A or take a namespace, and stamping --context kai-server into a recipe would be actively wrong during a ser8 incident.

Scripts - explicit context, since each targets a known cluster.
Runbooks - one line per skill stating the flag is required and the caller supplies it, plus a placeholder in the examples rather than a guessed value.

Alternative, if the bill is judged too high

Revisit #1349's shape: default the context instead of requiring it, e.g. argv-prefix --context kai-server with deny-flag --context, which pins a target unoverridably per wrap and needs a second wrap for ser8. That restores the invariant without touching a single caller. It was considered and set aside in #1348 in favour of the mandatory form, before this call-site count was known.

Sequencing

The rebuild is the trigger. Whatever lands here should land before aosguard is next rebuilt and shipped in the dev-base image, or the first person to hit this debugs a fail-closed deny with no connection to the change that caused it.

A regression I introduced in #1349, found before it bit anyone. Vera flagged the shape of it; the specific extent is worse than either of us assumed, and her example was the one case that is **not** affected. ## What #1349 did `--context` is now mandatory on every `aosguard ops kubectl` leaf. Nothing changes until aosguard is rebuilt on a host, so this is latent, not live. At rebuild every call written without the flag starts failing closed: ``` aosguard: `get` denied: context did not match any allowed pattern [kai-server ser8] (fail-closed) ``` ## The extent **65 call sites** across 13 files in `coilyco-flight-deck/infrastructure`, none carrying `--context`. **Two live-mutation scripts**, which are the urgent part: * `scripts/bootstrap-ser8-forgejo-runner-canary.sh` - three `aosguard ops kubectl apply -f -` calls, executed **on a remote host** through a `remote` helper. * `scripts/forgejo-external-account-lockdown-rollout.sh` - `aosguard ops kubectl diff -f` and two `apply -f` calls. **Eleven ops-investigation runbooks**, under `.agents/skills/ops-investigation-*`: disk-pressure, forgejo-runner-failure, k3s-connectivity, k3s-deploy-failure, k3s-pod-eviction. These carry concrete copy-paste recipes. They are loaded *during an incident*, which is exactly when a command failing for an unrelated-looking reason costs the most. `scripts/cleanup-orphaned-cluster-objects.sh` mentions the wrap in a comment as an optional `KUBECTL=` value; its actual verb is `delete`, which the wrap never exposed, so it is unaffected in practice. ## Not affected, contrary to the first guess `forgejo_storage_measure.py` was flagged as the likely victim. It is not. It calls `subprocess.run(("kubectl", ...))`, spawning the **binary** directly from inside the `forgejo-storage` wrap, so it never crosses `aosguard ops kubectl` and no wrap-level guard reaches it. That is #1350, a different defect, and the two should not be conflated. ## The breakage is mostly the guard working Worth stating before anyone reaches for a revert. `bootstrap-ser8-forgejo-runner-canary.sh` targets ser8 by name and applies through `remote` onto a host whose *active* kubeconfig context is not guaranteed to be ser8. An unnamed target on a remote mutation is precisely the ambiguity #1349 exists to catch, so these two scripts are the strongest argument for naming the cluster rather than against it. The runbooks are a genuine cost with no upside: they are read by an operator who already knows which cluster they are on. ## Fixing it Do not hardcode a context into runbook examples. Most read `-A` or take a namespace, and stamping `--context kai-server` into a recipe would be actively wrong during a ser8 incident. * **Scripts** - explicit context, since each targets a known cluster. * **Runbooks** - one line per skill stating the flag is required and the caller supplies it, plus a placeholder in the examples rather than a guessed value. ## Alternative, if the bill is judged too high Revisit #1349's shape: default the context instead of requiring it, e.g. `argv-prefix --context kai-server` with `deny-flag --context`, which pins a target unoverridably per wrap and needs a second wrap for ser8. That restores the invariant without touching a single caller. It was considered and set aside in #1348 in favour of the mandatory form, before this call-site count was known. ## Sequencing The rebuild is the trigger. Whatever lands here should land **before** aosguard is next rebuilt and shipped in the dev-base image, or the first person to hit this debugs a fail-closed deny with no connection to the change that caused it.

coilyco-ops added the

labels

2026-08-29 07:02:47 +00:00

coilyco-ops commented

2026-08-29 07:55:14 +00:00

Author

Owner

Decided by Kai, 2026-08-29: do the survey, land nothing. This issue stays out of the executing half of the overnight lane.

Scope of the overnight pass

Read-only. Enumerate all 65 aosguard ops kubectl call sites, classify each by whether #1349 breaks it and how, and post the inventory here. No edits, no rebuild, no sweep.

The two live-mutation scripts get named explicitly and separated from the rest, because they are the reason this cannot be a mechanical sweep: a wrong edit there is discovered by a broken deploy rather than by a failing test.

Why not the sweep

I recommended keeping it out entirely and Kai took the middle option, which is better than either thing I weighted. The survey is the part that is genuinely safe unattended and genuinely useful in the morning: it converts "65 call sites, unknown shape" into a reviewed list that a person can decide against in one sitting.

The sweep itself stays out. Beyond the risk, kubectl is the Systems Administrator seat's surface rather than the Platform seat's, so a fleet edit to live-mutation scripts is a handover rather than a task I should take unattended even if the survey says it is uniform.

What the survey should answer

The inventory is only worth reading if it settles these, so each row carries them:

the call site, and whether it mutates or reads
whether #1349 breaks it, and the exact reason rather than a yes
whether the fix is mechanical or needs judgement
which seat owns the file it lives in

That last column is what makes the result actionable rather than a list: it splits the 65 into the ones a platform run can take and the ones that need sysadmin.

#1351 is the adjacent defect: an argN or any-arg guard on a kubectl grant would silently not bind, because umbra's valueFlags table is AWS-shaped. It is in the executing half of the lane. Anyone reading this survey should read that too, since a guard that does not bind changes what "breaks" means for some of these 65.

**Decided by Kai, 2026-08-29: do the survey, land nothing.** This issue stays out of the executing half of the overnight lane. ## Scope of the overnight pass Read-only. Enumerate all 65 `aosguard ops kubectl` call sites, classify each by whether `#1349` breaks it and how, and post the inventory here. No edits, no rebuild, no sweep. The two live-mutation scripts get named explicitly and separated from the rest, because they are the reason this cannot be a mechanical sweep: a wrong edit there is discovered by a broken deploy rather than by a failing test. ## Why not the sweep I recommended keeping it out entirely and Kai took the middle option, which is better than either thing I weighted. The survey is the part that is genuinely safe unattended and genuinely useful in the morning: it converts "65 call sites, unknown shape" into a reviewed list that a person can decide against in one sitting. The sweep itself stays out. Beyond the risk, `kubectl` is the Systems Administrator seat's surface rather than the Platform seat's, so a fleet edit to live-mutation scripts is a handover rather than a task I should take unattended even if the survey says it is uniform. ## What the survey should answer The inventory is only worth reading if it settles these, so each row carries them: * the call site, and whether it mutates or reads * whether `#1349` breaks it, and the exact reason rather than a yes * whether the fix is mechanical or needs judgement * which seat owns the file it lives in That last column is what makes the result actionable rather than a list: it splits the 65 into the ones a platform run can take and the ones that need sysadmin. ## Related `#1351` is the adjacent defect: an `argN` or any-arg guard on a kubectl grant would silently not bind, because umbra's `valueFlags` table is AWS-shaped. It is in the executing half of the lane. Anyone reading this survey should read that too, since a guard that does not bind changes what "breaks" means for some of these 65.

coilyco-ops added this to the Rules that do not hold: silent guards, full budgets, second writers milestone

2026-08-29 07:55:39 +00:00

coilyco-ops commented

2026-08-29 10:57:14 +00:00

Author

Owner

Surveyed on disk, read-only. This is already fixed.

infrastructure at b37caa0 (main), measured 2026-08-29:

70  total `aosguard ops kubectl` mentions
61  carry --context
 9  do not

All nine remaining are prose, not call sites:

5 are the fix's own instruction line - "Every aosguard ops kubectl call names its cluster. Substitute <cluster> with kai-server or ser8; the wrap fails closed without it (agentic-os#1349)" - in the five ops-investigation-* SKILL.md files.
2 are prose in raw-du-guardrails.md about what the wrap does and does not expose.
1 is aosguard ops kubectl top pod -A in a sentence about metrics-server availability.
1 is the KUBECTL= comment in cleanup-orphaned-cluster-objects.sh, which this issue already identified as unaffected.

The urgent part - the two live-mutation scripts - is done:

bootstrap-ser8-forgejo-runner-canary.sh - all 19 calls carry --context ser8, including the three apply -f - through the remote helper.
forgejo-external-account-lockdown-rollout.sh - the diff -f and both apply -f carry --context "${context}".

Landed in 929c806, infrastructure#977, "fix(ops): name the cluster on every guarded kubectl call before the rebuild", 2026-08-28. Before the rebuild, as the title says, so the latent failure never fired.

Surveyed only, per the milestone run order. I made no edits to the infrastructure checkout. Closing this needs someone to confirm no other consumer repo carries aosguard ops kubectl call sites - I can only see infrastructure and agentic-os from this host, and agentic-os carries none.

Surveyed on disk, read-only. **This is already fixed.** `infrastructure` at `b37caa0` (main), measured 2026-08-29: ``` 70 total `aosguard ops kubectl` mentions 61 carry --context 9 do not ``` All nine remaining are **prose, not call sites**: * 5 are the fix's own instruction line - "**Every `aosguard ops kubectl` call names its cluster.** Substitute `<cluster>` with `kai-server` or `ser8`; the wrap fails closed without it (agentic-os#1349)" - in the five `ops-investigation-*` SKILL.md files. * 2 are prose in `raw-du-guardrails.md` about what the wrap does and does not expose. * 1 is `aosguard ops kubectl top pod -A` in a sentence about metrics-server availability. * 1 is the `KUBECTL=` comment in `cleanup-orphaned-cluster-objects.sh`, which this issue already identified as unaffected. The urgent part - the two live-mutation scripts - is done: * `bootstrap-ser8-forgejo-runner-canary.sh` - all 19 calls carry `--context ser8`, including the three `apply -f -` through the `remote` helper. * `forgejo-external-account-lockdown-rollout.sh` - the `diff -f` and both `apply -f` carry `--context "${context}"`. Landed in `929c806`, infrastructure#977, "fix(ops): name the cluster on every guarded kubectl call before the rebuild", 2026-08-28. Before the rebuild, as the title says, so the latent failure never fired. Surveyed only, per the milestone run order. I made no edits to the infrastructure checkout. Closing this needs someone to confirm no other consumer repo carries `aosguard ops kubectl` call sites - I can only see `infrastructure` and `agentic-os` from this host, and agentic-os carries none.

coilyco-ops commented

2026-08-29 11:12:10 +00:00

Author

Owner

Milestone status, so it survives the session.

Closed this run (13): #1111, #1119, #1180, #1192, #1199, #1205, #1206, #1208, #1212, #1274, #1303, #1312, #1337, #1341, #1366, #1373, #1374, #1375, #1378.

In flight: #1119 via PR #1398 (green on gate and aos-cli-tests, waiting on the dev-base image build, which failed once on a runner network flake - six curl: (35) Connection reset by peer retries over 15 minutes - and is rebuilding). #1317 via PR #1407.

Filed as follow-ups: #1393 (landed and closed), #1406 (a describe on a spec-resolved can leaf is silently ignored, found while building aosguard for #1375).

The nine still open, and what each actually needs:

#1352 (this one) - already fixed in infrastructure by 929c806 / infrastructure#977. All 70 call sites surveyed; the 9 without --context are prose, not calls. Needs someone to confirm no other consumer repo has call sites - I can only see infrastructure and agentic-os from this host.
#1380 - the state/ambient instance is closed. Measured drift now: role/exec and role/qa live in all three orgs undeclared, plus coherence-core, qa-fixture, c#. Whether those get declared or deleted is a taxonomy decision. The script-uses-aosguard half is untouched.
#1377 - blocked on shape, not effort. restrict is wrap-level, so a create label scoped to coilysiren needs a separate guardfile tier inheriting forgejo.kdl with override can create label, not a leaf edit. Doing it as a plain can would reopen the ward#107 shadowing this repo cleaned up across 26 repos.
#1365, #1376 - guardfile refactor and a cross-repo push to deploy.
#1351 - needs a umbra-side answer about argN binding on exec grants.
#1299, #1329 - decisions Kai owns, not work.

Everything above is pushed. No local-only state.

Milestone status, so it survives the session. **Closed this run (13):** #1111, #1119, #1180, #1192, #1199, #1205, #1206, #1208, #1212, #1274, #1303, #1312, #1337, #1341, #1366, #1373, #1374, #1375, #1378. **In flight:** #1119 via PR #1398 (green on gate and aos-cli-tests, waiting on the dev-base image build, which failed once on a runner network flake - six `curl: (35) Connection reset by peer` retries over 15 minutes - and is rebuilding). #1317 via PR #1407. **Filed as follow-ups:** #1393 (landed and closed), #1406 (a `describe` on a spec-resolved `can` leaf is silently ignored, found while building aosguard for #1375). **The nine still open, and what each actually needs:** * **#1352** (this one) - already fixed in infrastructure by `929c806` / infrastructure#977. All 70 call sites surveyed; the 9 without `--context` are prose, not calls. Needs someone to confirm no other consumer repo has call sites - I can only see `infrastructure` and `agentic-os` from this host. * **#1380** - the `state/ambient` instance is closed. Measured drift now: `role/exec` and `role/qa` live in all three orgs undeclared, plus `coherence-core`, `qa-fixture`, `c#`. Whether those get declared or deleted is a taxonomy decision. The script-uses-aosguard half is untouched. * **#1377** - blocked on shape, not effort. `restrict` is wrap-level, so a `create label` scoped to `coilysiren` needs a separate guardfile tier inheriting forgejo.kdl with `override can create label`, not a leaf edit. Doing it as a plain `can` would reopen the ward#107 shadowing this repo cleaned up across 26 repos. * **#1365, #1376** - guardfile refactor and a cross-repo push to `deploy`. * **#1351** - needs a umbra-side answer about argN binding on exec grants. * **#1299, #1329** - decisions Kai owns, not work. Everything above is pushed. No local-only state.

coilyco-ops commented

2026-08-29 12:05:16 +00:00

Author

Owner

Did the cross-repo confirmation this issue asked for, and found a live one it missed. infrastructure was surveyed; deploy was not.

Swept every repository on this host:

coilyco-flight-deck/infrastructure  19 files   all executable calls carry --context
coilyco-bridge/deploy                4 files   ONE executable call does not
coilyco-flight-deck/agentic-os       3 files   guardfile + prose
coilyco-bridge/agentic-os-kai        1 file    a digest doc

The live break

coilyco-bridge/deploy/scripts/kubectl-diff.sh:21

aosguard ops kubectl diff -f - <"${tmp}"

No --context. At rebuild that fails closed exactly as this issue predicts. It is not dead code - three justfile targets invoke it:

justfile:140  diff-factory-game
justfile:148  diff-galaxy-gen
justfile:156  diff-website

So just diff-* in deploy stops working the moment the host rebuilds aosguard. It is a diff rather than an apply, so nothing mutates, but the three drift checks go dark.

Two copy-paste recipes, also in deploy

Lower stakes and both apply, so worth fixing in the same pass:

services/teable/README.md:69
services/teable/deploy/tailnet-rbac.yml:32 (a comment recipe)

The one thing I cannot answer

Which cluster those three targets run against. deploy distinguishes a default and ser8 throughout its justfile, and kubectl-diff.sh is generic across three services, so --context there needs either a default or a parameter. Diffing the wrong cluster reports spurious drift, which is worse than the current failure because it is quiet.

That is a deploy decision, and I have not touched the checkout. Everything above is read-only.

infrastructure's half is genuinely done (929c806, infrastructure#977, before the rebuild). This issue should stay open for deploy's half rather than close on the infrastructure evidence alone - which is what my earlier comment would have led to.

Did the cross-repo confirmation this issue asked for, and **found a live one it missed**. `infrastructure` was surveyed; `deploy` was not. Swept every repository on this host: ``` coilyco-flight-deck/infrastructure 19 files all executable calls carry --context coilyco-bridge/deploy 4 files ONE executable call does not coilyco-flight-deck/agentic-os 3 files guardfile + prose coilyco-bridge/agentic-os-kai 1 file a digest doc ``` ## The live break `coilyco-bridge/deploy/scripts/kubectl-diff.sh:21` ```sh aosguard ops kubectl diff -f - <"${tmp}" ``` No `--context`. At rebuild that fails closed exactly as this issue predicts. It is not dead code - three justfile targets invoke it: ``` justfile:140 diff-factory-game justfile:148 diff-galaxy-gen justfile:156 diff-website ``` So `just diff-*` in deploy stops working the moment the host rebuilds aosguard. It is a `diff` rather than an `apply`, so nothing mutates, but the three drift checks go dark. ## Two copy-paste recipes, also in deploy Lower stakes and both `apply`, so worth fixing in the same pass: * `services/teable/README.md:69` * `services/teable/deploy/tailnet-rbac.yml:32` (a comment recipe) ## The one thing I cannot answer Which cluster those three targets run against. deploy distinguishes a default and `ser8` throughout its justfile, and `kubectl-diff.sh` is generic across three services, so `--context` there needs either a default or a parameter. Diffing the wrong cluster reports spurious drift, which is worse than the current failure because it is quiet. That is a deploy decision, and I have not touched the checkout. Everything above is read-only. **infrastructure's half is genuinely done** (`929c806`, infrastructure#977, before the rebuild). This issue should stay open for deploy's half rather than close on the infrastructure evidence alone - which is what my earlier comment would have led to.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#1352

No description provided.

Rows
Columns