CI red on main: cli/sandbox security-claim tests fail on the runner; lint job red despite 0 local issues #180

New issue

Closed

opened 2026-07-01 16:24:16 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-07-01 16:24:16 +00:00

Owner

Symptom

main is red at c8da301 (and the prior b951993): both the test and lint jobs fail (release and secrets are green). Diagnosed read-only from the aos director surface (substrate clone at c8da301, ops surface grants CI status only - no raw job logs - so confirm the rest in a writable clone).

`test` job - three cli/sandbox tests (reproduced locally)

--- FAIL: TestSetupDenied
    sandbox_degrade_linux_test.go:43: precondition: Wrap should have jailed the cmd
--- FAIL: TestSecurityClaim_GrandchildRoutesThroughGate
    sandbox_linux_test.go:111/114: name/abs-path grandchild call was not rerouted to the gate; log: REAL spawn / REAL byname / REAL byabspath
--- FAIL: TestSecurityClaim_SeccompDeniesPtrace
    sandbox_linux_test.go:136: ptrace probe exited non-zero: seccomp denylist not enforced (exit status 9)

These assert kernel sandbox enforcement (namespace jail, gate rerouting, seccomp ptrace denial). Key facts:

Not caused by the recent merges. git log -- cli/sandbox/ shows nothing since c154e1d (2026-06-17, the degrade gracefully feature) and 5d09088. The July-01 merges (b951993 value-chain, c8da301 fleetconfig) touch neither sandbox nor lint config. So the runner regressed, not the code.
Robustness gap. The two SecurityClaim tests get past their isUserNSUnavailable skip guard (sandbox_linux_test.go:101, 132) - unprivileged namespaces are nominally available on the runner - but then fail on the deeper enforcement (gate preload / seccomp) that the runner can no longer deliver, so Wrap degrades (c154e1d) and the tests observe un-enforced behavior instead of skipping. TestSetupDenied (sandbox_degrade_linux_test.go:43) has no skip guard at all - its precondition fatal-errors the moment the jail can't engage.

`lint` job - not lint rules (does NOT reproduce)

golangci-lint run --timeout=5m ./... at the exact CI-pinned v2.12.2 gives 0 issues locally. godoc-current.txt is fresh (check-godoc-current.sh exit 0). So lint is not failing on lint findings - suspect the job's install/env step (Go or golangci-lint install, network, runner image). Do not fabricate a code fix for a green linter - pull the actual CI job log first, and if it is a transient install flake a re-run may clear it.

Fix direction

Harden the sandbox security-claim tests (the real code fix, in cli-guard). Make TestSecurityClaim_GrandchildRoutesThroughGate, TestSecurityClaim_SeccompDeniesPtrace, and TestSetupDenied skip when the runner cannot actually enforce the sandbox - detect the missing capability (extend the isUserNSUnavailable pattern to cover the gate-preload and seccomp-enforcement preconditions, and add a guard to TestSetupDenied, which has none). Preserve the guarantee where the capability exists: skip ONLY on a genuine can't-enforce runner, never blanket-skip - these tests must still catch a real enforcement regression on a capable runner. A Skipf naming the missing capability, mirroring the existing lines, is the shape.
Investigate the runner capability loss (infra). Determine what the docker CI runner lost (unprivileged-userns config, seccomp, mount perms) between the last green run and now. If it is a runner/host regression, the rollout fix lives in infrastructure/ansible, not here - file/point at that separately. The cli-guard-side change is the test hardening in (1).
Diagnose lint from the real job log. 0 issues locally at v2.12.2 means the failure is env/install. Confirm from the log; re-run if transient.

Verify in the writable clone (couldn't from read-only substrate)

go mod tidy && git diff --exit-code -- go.mod go.sum (the test job's tidy gate - I could not write go.mod to check drift).
Pull the test and lint job logs to confirm the exact failing step (my repro is a container that may not match the runner).

Done condition

test job green: sandbox security-claim tests skip cleanly on a can't-enforce runner and still assert enforcement on a capable one.
lint job green (root cause identified - env fix or re-run, not a bogus code change).
If a runner capability regressed, an infrastructure/ansible follow-up is filed.

Filed from the aos director surface after Kai flagged cli-guard CI red.

## Symptom `main` is red at c8da301 (and the prior b951993): both the `test` and `lint` jobs fail (`release` and `secrets` are green). Diagnosed read-only from the aos director surface (substrate clone at c8da301, ops surface grants CI status only - no raw job logs - so confirm the rest in a writable clone). ## `test` job - three cli/sandbox tests (reproduced locally) ``` --- FAIL: TestSetupDenied sandbox_degrade_linux_test.go:43: precondition: Wrap should have jailed the cmd --- FAIL: TestSecurityClaim_GrandchildRoutesThroughGate sandbox_linux_test.go:111/114: name/abs-path grandchild call was not rerouted to the gate; log: REAL spawn / REAL byname / REAL byabspath --- FAIL: TestSecurityClaim_SeccompDeniesPtrace sandbox_linux_test.go:136: ptrace probe exited non-zero: seccomp denylist not enforced (exit status 9) ``` These assert kernel sandbox **enforcement** (namespace jail, gate rerouting, seccomp ptrace denial). Key facts: - **Not caused by the recent merges.** `git log -- cli/sandbox/` shows nothing since c154e1d (2026-06-17, the `degrade gracefully` feature) and 5d09088. The July-01 merges (b951993 value-chain, c8da301 fleetconfig) touch neither sandbox nor lint config. So the runner regressed, not the code. - **Robustness gap.** The two `SecurityClaim` tests get **past** their `isUserNSUnavailable` skip guard (sandbox_linux_test.go:101, 132) - unprivileged namespaces are nominally available on the runner - but then fail on the deeper enforcement (gate preload / seccomp) that the runner can no longer deliver, so `Wrap` degrades (c154e1d) and the tests observe un-enforced behavior instead of **skipping**. `TestSetupDenied` (sandbox_degrade_linux_test.go:43) has **no skip guard at all** - its precondition fatal-errors the moment the jail can't engage. ## `lint` job - not lint rules (does NOT reproduce) `golangci-lint run --timeout=5m ./...` at the **exact CI-pinned v2.12.2** gives **`0 issues`** locally. `godoc-current.txt` is fresh (`check-godoc-current.sh` exit 0). So `lint` is not failing on lint findings - suspect the job's install/env step (Go or golangci-lint install, network, runner image). **Do not fabricate a code fix for a green linter** - pull the actual CI job log first, and if it is a transient install flake a re-run may clear it. ## Fix direction 1. **Harden the sandbox security-claim tests (the real code fix, in cli-guard).** Make `TestSecurityClaim_GrandchildRoutesThroughGate`, `TestSecurityClaim_SeccompDeniesPtrace`, and `TestSetupDenied` **skip** when the runner cannot actually enforce the sandbox - detect the missing capability (extend the `isUserNSUnavailable` pattern to cover the gate-preload and seccomp-enforcement preconditions, and add a guard to `TestSetupDenied`, which has none). **Preserve the guarantee where the capability exists**: skip ONLY on a genuine can't-enforce runner, never blanket-skip - these tests must still catch a real enforcement regression on a capable runner. A `Skipf` naming the missing capability, mirroring the existing lines, is the shape. 2. **Investigate the runner capability loss (infra).** Determine what the docker CI runner lost (unprivileged-userns config, seccomp, mount perms) between the last green run and now. If it is a runner/host regression, the rollout fix lives in infrastructure/ansible, not here - file/point at that separately. The cli-guard-side change is the test hardening in (1). 3. **Diagnose `lint` from the real job log.** 0 issues locally at v2.12.2 means the failure is env/install. Confirm from the log; re-run if transient. ## Verify in the writable clone (couldn't from read-only substrate) - `go mod tidy && git diff --exit-code -- go.mod go.sum` (the test job's tidy gate - I could not write go.mod to check drift). - Pull the `test` and `lint` job logs to confirm the exact failing step (my repro is a container that may not match the runner). ## Done condition - `test` job green: sandbox security-claim tests skip cleanly on a can't-enforce runner and still assert enforcement on a capable one. - `lint` job green (root cause identified - env fix or re-run, not a bogus code change). - If a runner capability regressed, an infrastructure/ansible follow-up is filed. Filed from the aos director surface after Kai flagged cli-guard CI red.

coilyco-ops commented

2026-07-01 16:24:24 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-cli-guard-180 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-01T16:24:21Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-cli-guard-180` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-01T16:24:21Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-01 16:32:45 +00:00

fix(sandbox): skip security-claim tests on a can't-enforce runner; pin lint installer

coilysiren closed this issue

2026-07-01 16:32:45 +00:00

coilysiren referenced this issue from a commit

2026-07-01 16:32:45 +00:00

Merge branch 'issue-180': skip sandbox security-claim tests on can't-enforce runner (closes #180)

coilysiren commented

2026-07-01 16:33:37 +00:00

Author

Owner

WARD-OUTCOME: done - sandbox security-claim tests now skip on a can't-enforce runner (positive JAILED marker + distinct probe exit codes); lint installer pinned to its release tag.

The test half was the satisfying kind of bug: the tests weren't wrong about what to assert, they just had no positive signal that the jail actually engaged, so once auto-degrade (c154e1d) started silently falling back to an unsandboxed run they read that as an enforcement failure. Threading CLIGUARD_JAILED into the reroute log line and giving the ptrace probe three distinct exit codes (7 setup-failed / 8 not-jailed / 9 filter-missing) cleanly separates "runner can't deliver the jail" from "the guarantee broke" — so it skips on the former and still fails hard on the latter. I verified both directions here: opt-out and clone-denied both skip, and TestSetupDenied still runs fully when the sandbox isn't opted out, so it's not a blanket skip.

What fought back a little: this container can't create an unprivileged userns at all, so I couldn't exercise the fully-capable path end-to-end — I'm confident by construction (a probe exit 9 inside the jail still hits Fatalf) but that branch is unverified on real hardware. The lint red I couldn't confirm from a raw job log (no log grant from this clone), so I pinned the golangci-lint install.sh from a moving master to the GOLANGCI_LINT_VERSION tag — a genuine determinism fix for the exact step the issue flagged, and 0 issues reproduce locally at v2.12.2 — but if the real cause was a transient network/runner flake, the fresh CI run on this push is the real test.

Follow-up worth filing: the underlying runner capability loss (unprivileged-userns / seccomp / mount perms on the docker CI runner) is an infra concern for infrastructure/ansible, out of scope for this repo and not a granted repo here, so I left it for a separate infra issue rather than filing across the wall.

WARD-OUTCOME: done - sandbox security-claim tests now skip on a can't-enforce runner (positive JAILED marker + distinct probe exit codes); lint installer pinned to its release tag. The `test` half was the satisfying kind of bug: the tests weren't wrong about *what* to assert, they just had no positive signal that the jail actually engaged, so once auto-degrade (c154e1d) started silently falling back to an unsandboxed run they read that as an enforcement failure. Threading `CLIGUARD_JAILED` into the reroute log line and giving the ptrace probe three distinct exit codes (7 setup-failed / 8 not-jailed / 9 filter-missing) cleanly separates "runner can't deliver the jail" from "the guarantee broke" — so it skips on the former and still fails hard on the latter. I verified both directions here: opt-out and clone-denied both skip, and `TestSetupDenied` still runs fully when the sandbox isn't opted out, so it's not a blanket skip. What fought back a little: this container can't create an unprivileged userns at all, so I couldn't exercise the *fully-capable* path end-to-end — I'm confident by construction (a probe exit 9 inside the jail still hits Fatalf) but that branch is unverified on real hardware. The `lint` red I couldn't confirm from a raw job log (no log grant from this clone), so I pinned the golangci-lint `install.sh` from a moving `master` to the `GOLANGCI_LINT_VERSION` tag — a genuine determinism fix for the exact step the issue flagged, and 0 issues reproduce locally at v2.12.2 — but if the real cause was a transient network/runner flake, the fresh CI run on this push is the real test. Follow-up worth filing: the underlying runner capability loss (unprivileged-userns / seccomp / mount perms on the docker CI runner) is an infra concern for infrastructure/ansible, out of scope for this repo and not a granted repo here, so I left it for a separate infra issue rather than filing across the wall.