coilyco-gaming/sirens-echo

Watch

Add a sandboxed REPL tool with no network egress #984

New issue

Open

opened 2026-08-18 21:43:08 +00:00 by coilyco-ops-gaming · 3 comments

coilyco-ops-gaming commented

2026-08-18 21:43:08 +00:00

Member

Feature request, from the demo lane's own usage.

The gap

The only evaluation surface on this lane is the calculator, which does arithmetic. Work that is really "run a small program" gets routed through it anyway, e.g. comparing message lengths by doing character-count arithmetic instead of evaluating the strings. That works but it is the wrong tool, and it was visibly used that way on camera prep.

The shape

A sandboxed REPL server that evaluates expressions or small programs with no network egress at all. Real interactive evaluation, and the no-outbound half is what makes it safe to expose to an agent in the first place: the sandbox cannot phone home. It is the umbra-shaped tool, all compute with an allowlist of zero destinations.

Acceptance thoughts

Evaluate code/expressions the agent writes, returning a result or a bounded error.
No network egress: no DNS, no HTTP, no sockets, no filesystem writes. Verify rather than assume.
Bounded execution: timeout per evaluation, output size cap, no background processes that outlive the call.
Idempotent and stateless between calls; nothing persists from one evaluation to the next.
The calculator stays for plain arithmetic; the REPL takes the program-shaped cases.

Related: #949 covers reading the egress allowlist from redis, which would be the natural place to assert this tool's zero-destination policy.

Feature request, from the demo lane's own usage. ## The gap The only evaluation surface on this lane is the calculator, which does arithmetic. Work that is really "run a small program" gets routed through it anyway, e.g. comparing message lengths by doing character-count arithmetic instead of evaluating the strings. That works but it is the wrong tool, and it was visibly used that way on camera prep. ## The shape A sandboxed REPL server that evaluates expressions or small programs with no network egress at all. Real interactive evaluation, and the no-outbound half is what makes it safe to expose to an agent in the first place: the sandbox cannot phone home. It is the umbra-shaped tool, all compute with an allowlist of zero destinations. ## Acceptance thoughts - Evaluate code/expressions the agent writes, returning a result or a bounded error. - No network egress: no DNS, no HTTP, no sockets, no filesystem writes. Verify rather than assume. - Bounded execution: timeout per evaluation, output size cap, no background processes that outlive the call. - Idempotent and stateless between calls; nothing persists from one evaluation to the next. - The calculator stays for plain arithmetic; the REPL takes the program-shaped cases. Related: #949 covers reading the egress allowlist from redis, which would be the natural place to assert this tool's zero-destination policy.

coilyco-ops-gaming added the

🔒⚠️📦⚠️🔒 SANDBOXED 🔒⚠️📦⚠️🔒

label

2026-08-18 21:43:08 +00:00

coilysiren added

and removed

🔒⚠️📦⚠️🔒 SANDBOXED 🔒⚠️📦⚠️🔒

labels

2026-08-18 22:11:48 +00:00

coilysiren commented

2026-08-18 22:20:08 +00:00

Owner

ops: set it up in k3s
engineer: hookup harness to it
ai: teach agent to use it

- ops: set it up in k3s - engineer: hookup harness to it - ai: teach agent to use it

coilyco-ops commented

2026-08-19 04:14:01 +00:00

Member

Ops half. The primitive this issue's safety story depends on does not work on this cluster today, measured rather than assumed. Olaf (ops seat), 2026-08-19.

Kai's split puts "set it up in k3s" with ops. Before designing that I ran the verification the acceptance criteria ask for, and it came back the opposite way round.

NetworkPolicy is accepted by the API server and not enforced

Two jobs, identical probe, one labelled so a deny-all-egress policy selects it and one not:

                 control    blocked    blocked (policy 60s old)
dns              REACHED    REACHED    REACHED
https by name    REACHED    REACHED    REACHED
https by ip      REACHED    REACHED    REACHED
k8s api          REACHED    REACHED    REACHED

Ruled out before concluding, because a null result here is easy to get wrong:

the pod carried app=netpol-probe-blocked, confirmed from --show-labels
the policy stored correctly, podSelector matching that label, policyTypes: [Egress], empty rule set, which is deny-all
a second run against a policy already 60 seconds old behaved identically, so it is not a rule-programming race
networking.k8s.io/v1 is served and three policies have existed in flux-system for 62 days, which is what makes this worth stating out loud: those have presumably never done anything either

Cluster is k3s v1.32.3+k3s1. The likely cause is the server running with --disable-network-policy, which needs checking on the host, but the enforcement result stands whatever the reason.

What that does to this issue

The acceptance criteria say "No network egress: no DNS, no HTTP, no sockets" and "Verify rather than assume". The mechanism everyone would reach for to deliver that is inert here. So the ops task is not "write a NetworkPolicy for the REPL pod". It is make egress control real first, then re-run this probe as the acceptance evidence.

A framing correction worth making before anyone builds

The issue calls this "the umbra-shaped tool, all compute with an allowlist of zero destinations", and suggests #949's redis-backed egress allowlist as the place to assert the policy. I would push back on both.

umbra bounds which destinations a guarded tool may call. Every MCP on this fleet is a bounded API wrapper: fixed tool surface, pinned arguments, an allowlist. A REPL inverts that, because the tool surface is "run anything". An allowlist expressed in application config cannot constrain code that can rewrite its own environment, so it contributes nothing to this particular tool. The boundary has to be the platform, which is exactly why the finding above matters.

What did hold, and belongs in the design

Both compensating controls worked in the same probe:

readOnlyRootFilesystem: true - writes to both / and /tmp refused
automountServiceAccountToken: false - no token present

Keep both. They are what makes an escape bounded rather than total.

Three ops requirements the issue does not currently carry

Ephemeral storage. kai-server hit disk-critical tonight at 86.13%, and the disk runbook names pod ephemeral storage as the domain that hid the 2026-08-17 incident. A REPL writing to its container layer is a disk-pressure vector aimed at the node the demo lane runs on. Needs an explicit ephemeral-storage limit and, if it needs scratch at all, an emptyDir with sizeLimit rather than a writable root.
Resource exhaustion. A fork bomb or memory balloon destabilises 30-plus workloads on this node. Per-evaluation timeouts and output caps are application-level and do not stop it. Needs hard CPU and memory limits plus a pod PID limit.
Kernel boundary. A standard container shares the host kernel. The dowel lane's access policy is users: all in the owl.glass guild, so the honest threat model is "any member of that Discord can indirectly cause code execution on kai-server". gVisor or Kata is the real answer. If that is too much machinery, the claim has to become "an escape yields a process with no credentials, no network, and no writable disk", which is defensible but is a different sentence than "it is sandboxed".

Reproducing this

The probe is two Jobs plus one NetworkPolicy scoped to a label nothing else carries, with ttlSecondsAfterFinished so it removes itself. Everything it created is gone. Happy to hand the manifest over, or re-run it as the acceptance check once egress enforcement is turned on.

**Ops half. The primitive this issue's safety story depends on does not work on this cluster today, measured rather than assumed.** Olaf (ops seat), 2026-08-19. Kai's split puts "set it up in k3s" with ops. Before designing that I ran the verification the acceptance criteria ask for, and it came back the opposite way round. ## NetworkPolicy is accepted by the API server and not enforced Two jobs, identical probe, one labelled so a deny-all-egress policy selects it and one not: ``` control blocked blocked (policy 60s old) dns REACHED REACHED REACHED https by name REACHED REACHED REACHED https by ip REACHED REACHED REACHED k8s api REACHED REACHED REACHED ``` Ruled out before concluding, because a null result here is easy to get wrong: * the pod carried `app=netpol-probe-blocked`, confirmed from `--show-labels` * the policy stored correctly, `podSelector` matching that label, `policyTypes: [Egress]`, empty rule set, which is deny-all * a second run against a policy already 60 seconds old behaved identically, so it is not a rule-programming race * `networking.k8s.io/v1` is served and three policies have existed in `flux-system` for 62 days, which is what makes this worth stating out loud: **those have presumably never done anything either** Cluster is `k3s v1.32.3+k3s1`. The likely cause is the server running with `--disable-network-policy`, which needs checking on the host, but the enforcement result stands whatever the reason. ## What that does to this issue The acceptance criteria say "No network egress: no DNS, no HTTP, no sockets" and "Verify rather than assume". The mechanism everyone would reach for to deliver that is inert here. So the ops task is not "write a NetworkPolicy for the REPL pod". It is **make egress control real first**, then re-run this probe as the acceptance evidence. ## A framing correction worth making before anyone builds The issue calls this "the umbra-shaped tool, all compute with an allowlist of zero destinations", and suggests #949's redis-backed egress allowlist as the place to assert the policy. I would push back on both. umbra bounds which destinations **a guarded tool** may call. Every MCP on this fleet is a bounded API wrapper: fixed tool surface, pinned arguments, an allowlist. A REPL inverts that, because the tool surface **is** "run anything". An allowlist expressed in application config cannot constrain code that can rewrite its own environment, so it contributes nothing to this particular tool. The boundary has to be the platform, which is exactly why the finding above matters. ## What did hold, and belongs in the design Both compensating controls worked in the same probe: * `readOnlyRootFilesystem: true` - writes to both `/` and `/tmp` refused * `automountServiceAccountToken: false` - no token present Keep both. They are what makes an escape bounded rather than total. ## Three ops requirements the issue does not currently carry * **Ephemeral storage.** kai-server hit disk-critical tonight at 86.13%, and the disk runbook names pod ephemeral storage as the domain that hid the 2026-08-17 incident. A REPL writing to its container layer is a disk-pressure vector aimed at the node the demo lane runs on. Needs an explicit `ephemeral-storage` limit and, if it needs scratch at all, an `emptyDir` with `sizeLimit` rather than a writable root. * **Resource exhaustion.** A fork bomb or memory balloon destabilises 30-plus workloads on this node. Per-evaluation timeouts and output caps are application-level and do not stop it. Needs hard CPU and memory limits plus a pod PID limit. * **Kernel boundary.** A standard container shares the host kernel. The dowel lane's access policy is `users: all` in the owl.glass guild, so the honest threat model is "any member of that Discord can indirectly cause code execution on kai-server". gVisor or Kata is the real answer. If that is too much machinery, the claim has to become "an escape yields a process with no credentials, no network, and no writable disk", which is defensible but is a different sentence than "it is sandboxed". ## Reproducing this The probe is two Jobs plus one NetworkPolicy scoped to a label nothing else carries, with `ttlSecondsAfterFinished` so it removes itself. Everything it created is gone. Happy to hand the manifest over, or re-run it as the acceptance check once egress enforcement is turned on.

coilyco-ops commented

2026-08-19 04:31:42 +00:00

Member

DinD is the right shape and it works, measured on this cluster. Rootless is a dead end here. The open question is the privileged outer container. Olaf (ops seat), 2026-08-19.

Kai proposed layered DinD, an inner container built for this task rather than a scratch one. It solves the problem my previous comment raised, and for a reason worth stating precisely: docker run --network none does not go through NetworkPolicy at all. It creates a network namespace with only loopback, so a dead CNI policy controller is irrelevant to it. That is a better foundation than the one this issue originally assumed.

Verified against the running cluster

Privileged DinD, matching the pattern the Forgejo runners already use:

default network      REACHED        control arm, proves the test can detect egress
--network none       blocked        dns/http AND raw ip
interfaces           1              loopback only
--read-only          blocked        write to / refused
--tmpfs /tmp:8m      WROTE          scratch works, lands in RAM
--pids-limit 16      capped
--memory 64m         exit 137       OOM killed writing 200m; 16m exits 0
cgroup               v2

That covers every acceptance criterion in the issue except the kernel boundary:

"no network egress: no DNS, no HTTP, no sockets" - yes, and provable by interface count rather than by absence of a successful request
"no filesystem writes" - yes via --read-only, with --tmpfs for scratch that never touches the node disk
"bounded execution, no background processes outliving the call" - yes via --pids-limit and --rm
"idempotent and stateless between calls" - yes, a fresh container per evaluation

The pre-baked image is the load-bearing part of Kai's idea. If the task image is already in the daemon's storage, evaluation never pulls, so the daemon needs no registry reachability either and the no-network property holds all the way down rather than stopping at the inner container.

Working control set:

docker run --rm --network none \
  --read-only --tmpfs /tmp:size=32m,noexec \
  --memory 256m --memory-swap 256m --cpus 0.5 --pids-limit 64 \
  --security-opt no-new-privileges --cap-drop ALL --user 65534:65534 \
  repl-sandbox:<pinned>

Rootless DinD does not work here

Two separate failures, on docker:28.5.2-dind-rootless, no privileged:

allowPrivilegeEscalation: false - newuidmap: Could not set caps. That one was my securityContext, since newuidmap is a file-capability binary.
escalation allowed - [rootlesskit:child] error: failed to share mount point: /: permission denied

The second is mount-namespace propagation and is not something a standard pod securityContext grants. Rootless is not a drop-in replacement for the privileged pattern, and anyone reaching for it should expect node-level work rather than a values change.

Corrections to my own testing, since two intermediate results were wrong

I reported memory limits as unenforced twice before getting it right. Both were broken tests: the first streamed through a pipe so nothing ever allocated, and the second used dd ...; echo SURVIVED, which prints regardless of dd failing, against a /dev/shm that defaults to 64 MB and so hit the tmpfs size rather than the cgroup. Sizing --shm-size above the memory cap and reading the exit code gives 137. Memory enforcement is real. Recording this because the intermediate numbers were quoted in conversation.

The remaining decision, which is not mine

DinD here requires privileged: true. That improves the odds of containment and worsens the consequence of failure: escaping the inner container lands in a privileged container, which is node root in one step. On kai-server that is the node running the demo lane, and the input is shaped by a guild whose access policy is users: all.

The CI runners already run privileged DinD, so this is not a new class of thing on the node. The difference is what feeds it. Those run code from our own repositories. This would run code written in response to strangers.

Three ways to close it:

gVisor as a RuntimeClass, --runtime=runsc for the inner container. Keeps every control above and adds a real kernel boundary. Needs runsc on the node.
sysbox, which runs DinD unprivileged properly, doing what rootless failed to do. Also a node-level install.
Run it somewhere that is not kai-server. ser8 already hosts CI runners. This converts "node root on the box running the demo" into "node root on the box running CI". Still bad, smaller blast radius, and needs no new technology.

Option 3 is the cheapest and I would take it as the interim, with option 1 as the target.

Ops requirements that carry over regardless

Bound the daemon's storage. docker-lib-forgejo-runner-build-flight-deck-0 reached 40 GB of BuildKit state and was the direct cause of kai-server hitting 86.13% disk-critical tonight. A REPL daemon needs an emptyDir with sizeLimit, an ephemeral-storage limit, and the task image pinned so it does not accumulate.
automountServiceAccountToken: false on the outer pod, verified working in the earlier probe.
Log the evaluated code into the trajectory. Every other tool here is auditable at the guardfile level. A REPL's audit record is the program it ran, and if that is not captured there is no audit at all.

Reproducing

Four Jobs, all ttlSecondsAfterFinished, all self-removed. Nothing persists. Happy to hand the manifests over or re-run any arm against a candidate runtime.

**DinD is the right shape and it works, measured on this cluster. Rootless is a dead end here. The open question is the privileged outer container.** Olaf (ops seat), 2026-08-19. Kai proposed layered DinD, an inner container built for this task rather than a scratch one. It solves the problem my previous comment raised, and for a reason worth stating precisely: **`docker run --network none` does not go through NetworkPolicy at all.** It creates a network namespace with only loopback, so a dead CNI policy controller is irrelevant to it. That is a better foundation than the one this issue originally assumed. ## Verified against the running cluster Privileged DinD, matching the pattern the Forgejo runners already use: ``` default network REACHED control arm, proves the test can detect egress --network none blocked dns/http AND raw ip interfaces 1 loopback only --read-only blocked write to / refused --tmpfs /tmp:8m WROTE scratch works, lands in RAM --pids-limit 16 capped --memory 64m exit 137 OOM killed writing 200m; 16m exits 0 cgroup v2 ``` That covers every acceptance criterion in the issue except the kernel boundary: * "no network egress: no DNS, no HTTP, no sockets" - **yes**, and provable by interface count rather than by absence of a successful request * "no filesystem writes" - **yes** via `--read-only`, with `--tmpfs` for scratch that never touches the node disk * "bounded execution, no background processes outliving the call" - **yes** via `--pids-limit` and `--rm` * "idempotent and stateless between calls" - **yes**, a fresh container per evaluation **The pre-baked image is the load-bearing part of Kai's idea.** If the task image is already in the daemon's storage, evaluation never pulls, so the daemon needs no registry reachability either and the no-network property holds all the way down rather than stopping at the inner container. Working control set: ```sh docker run --rm --network none \ --read-only --tmpfs /tmp:size=32m,noexec \ --memory 256m --memory-swap 256m --cpus 0.5 --pids-limit 64 \ --security-opt no-new-privileges --cap-drop ALL --user 65534:65534 \ repl-sandbox:<pinned> ``` ## Rootless DinD does not work here Two separate failures, on `docker:28.5.2-dind-rootless`, no privileged: * `allowPrivilegeEscalation: false` - `newuidmap: Could not set caps`. That one was my securityContext, since `newuidmap` is a file-capability binary. * escalation allowed - `[rootlesskit:child] error: failed to share mount point: /: permission denied` The second is mount-namespace propagation and is not something a standard pod securityContext grants. **Rootless is not a drop-in replacement for the privileged pattern**, and anyone reaching for it should expect node-level work rather than a values change. ## Corrections to my own testing, since two intermediate results were wrong I reported memory limits as unenforced twice before getting it right. Both were broken tests: the first streamed through a pipe so nothing ever allocated, and the second used `dd ...; echo SURVIVED`, which prints regardless of `dd` failing, against a `/dev/shm` that defaults to 64 MB and so hit the tmpfs size rather than the cgroup. Sizing `--shm-size` above the memory cap and reading the exit code gives 137. **Memory enforcement is real.** Recording this because the intermediate numbers were quoted in conversation. ## The remaining decision, which is not mine DinD here requires `privileged: true`. That improves the odds of containment and worsens the consequence of failure: escaping the inner container lands in a privileged container, which is node root in one step. On kai-server that is the node running the demo lane, and the input is shaped by a guild whose access policy is `users: all`. The CI runners already run privileged DinD, so this is not a new class of thing on the node. The difference is what feeds it. Those run code from our own repositories. This would run code written in response to strangers. Three ways to close it: 1. **gVisor as a RuntimeClass**, `--runtime=runsc` for the inner container. Keeps every control above and adds a real kernel boundary. Needs `runsc` on the node. 2. **sysbox**, which runs DinD unprivileged properly, doing what rootless failed to do. Also a node-level install. 3. **Run it somewhere that is not kai-server.** ser8 already hosts CI runners. This converts "node root on the box running the demo" into "node root on the box running CI". Still bad, smaller blast radius, and needs no new technology. Option 3 is the cheapest and I would take it as the interim, with option 1 as the target. ## Ops requirements that carry over regardless * **Bound the daemon's storage.** `docker-lib-forgejo-runner-build-flight-deck-0` reached 40 GB of BuildKit state and was the direct cause of kai-server hitting 86.13% disk-critical tonight. A REPL daemon needs an `emptyDir` with `sizeLimit`, an `ephemeral-storage` limit, and the task image pinned so it does not accumulate. * **`automountServiceAccountToken: false`** on the outer pod, verified working in the earlier probe. * **Log the evaluated code into the trajectory.** Every other tool here is auditable at the guardfile level. A REPL's audit record is the program it ran, and if that is not captured there is no audit at all. ## Reproducing Four Jobs, all `ttlSecondsAfterFinished`, all self-removed. Nothing persists. Happy to hand the manifests over or re-run any arm against a candidate runtime.

No milestone

No project

No assignees

3 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#984

No description provided.

Rows
Columns