coilyco-flight-deck/infrastructure

Watch

ser8 gaming runner evicted for exceeding its 24Gi docker-lib EmptyDir, ~1.6h after starting #808

New issue

Open

opened 2026-08-13 06:31:54 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-13 06:31:54 +00:00

Member

What happened

forgejo-runner-ser8-gaming-0 was evicted at 2026-08-13 05:57:30Z:

Evicted — Usage of EmptyDir volume "docker-lib" exceeds the limit "24Gi".

Both containers were killed (Stopping container dind, Stopping container runner) and the pod was rescheduled onto ser8 at 05:57:31, back up by 05:57:35.

The pod it replaced had been created at 04:20:39Z. That is a ~1h37m lifetime before the DinD image/layer cache filled a 24Gi EmptyDir.

Why this matters

This is the gaming-org runner. coilyco-gaming/sirens-echo builds on it, and that repository is under active development right now — a Sirens Deep image rolled at 04:00Z and another push landed at 04:38Z, both inside the evicted pod's lifetime.

An eviction mid-build fails the job with no useful message about cause, and the replacement pod starts with a cold Docker cache, so the next build is slower and fills the volume again. At a ~1.6h fill rate under load, this is a cycle rather than a one-off.

Evidence

From get_k3s_node_health on ser8, 24h window:

Time	Event	Object
05:57:30Z	`Warning Evicted` — EmptyDir `docker-lib` exceeds `24Gi`	`forgejo-runner-ser8-gaming-0`
05:57:30Z	`Killing` — Stopping container `dind`	same
05:57:30Z	`Killing` — Stopping container `runner`	same
05:57:31Z	`Scheduled` → ser8	replacement pod
05:57:35Z	`Started` — `dind`, `runner`	replacement pod

Node itself is healthy and not the constraint: ser8 reports DiskPressure: False (unchanged 61d18h), 931GB allocatable ephemeral storage, 60GB memory. The 24Gi EmptyDir limit is the binding constraint, not the disk.

Other ser8 runners use a PVC for docker-lib rather than an EmptyDir — docker-lib-forgejo-runner-build-0 and docker-lib-forgejo-runner-build-flight-deck-0 are both 10Gi local-path PVCs on kai-server. So the EmptyDir shape is not universal, and the two approaches have drifted.

What I have not established

Whether this is recurrent. The event carries count: 1 and the 24h window shows one occurrence. An eviction that has happened once in 24h is a different problem from one happening hourly. Event retention may also be hiding earlier ones — I would not read "one event" as "one occurrence".
What filled the volume. DinD layer cache is the obvious candidate but I did not inspect the volume contents, and the pod that filled it is gone.
Whether any job actually failed. I did not correlate against Forgejo Actions run history.

Suggested direction, not a prescription

Three options, in rough order of how much they change:

Raise the limit. Cheapest, and ser8 has ~931GB free. Does not stop the growth, just moves the wall.
Recycle the volume on a schedule. There is already a forgejo-runner-recycle CronJob on ser8 (last succeeded 19h30m ago) and an init-wipe-docker-lib init container on the flight-deck build runner. Whether the gaming runner is in scope for either is worth checking — it may simply have been missed.
Match the PVC shape the build runners use, so the volume is sized and observable rather than an unbounded scratch area.

Option 2 looks most likely to be the actual gap, since the machinery already exists.

Acceptance

The gaming runner does not evict itself during ordinary build load.
If evictions remain possible, they are visible — right now this was found by reading node events during an unrelated investigation, not by an alert.

coilyco-gaming/sirens-echo#190 — nothing alerts on this class of failure either.

Next owner

Ops.

## What happened `forgejo-runner-ser8-gaming-0` was evicted at **2026-08-13 05:57:30Z**: > `Evicted` — Usage of EmptyDir volume "docker-lib" exceeds the limit "24Gi". Both containers were killed (`Stopping container dind`, `Stopping container runner`) and the pod was rescheduled onto ser8 at 05:57:31, back up by 05:57:35. The pod it replaced had been created at **04:20:39Z**. That is a **~1h37m lifetime** before the DinD image/layer cache filled a 24Gi EmptyDir. ## Why this matters This is the gaming-org runner. `coilyco-gaming/sirens-echo` builds on it, and that repository is under active development right now — a Sirens Deep image rolled at 04:00Z and another push landed at 04:38Z, both inside the evicted pod's lifetime. An eviction mid-build fails the job with no useful message about cause, and the replacement pod starts with a cold Docker cache, so the next build is slower and fills the volume again. At a ~1.6h fill rate under load, this is a cycle rather than a one-off. ## Evidence From `get_k3s_node_health` on ser8, 24h window: | Time | Event | Object | | --- | --- | --- | | 05:57:30Z | `Warning Evicted` — EmptyDir `docker-lib` exceeds `24Gi` | `forgejo-runner-ser8-gaming-0` | | 05:57:30Z | `Killing` — Stopping container `dind` | same | | 05:57:30Z | `Killing` — Stopping container `runner` | same | | 05:57:31Z | `Scheduled` → ser8 | replacement pod | | 05:57:35Z | `Started` — `dind`, `runner` | replacement pod | Node itself is healthy and not the constraint: ser8 reports `DiskPressure: False` (unchanged 61d18h), 931GB allocatable ephemeral storage, 60GB memory. **The 24Gi EmptyDir limit is the binding constraint, not the disk.** Other ser8 runners use a PVC for `docker-lib` rather than an EmptyDir — `docker-lib-forgejo-runner-build-0` and `docker-lib-forgejo-runner-build-flight-deck-0` are both 10Gi `local-path` PVCs on kai-server. So the EmptyDir shape is not universal, and the two approaches have drifted. ## What I have not established - **Whether this is recurrent.** The event carries `count: 1` and the 24h window shows one occurrence. An eviction that has happened once in 24h is a different problem from one happening hourly. Event retention may also be hiding earlier ones — I would not read "one event" as "one occurrence". - **What filled the volume.** DinD layer cache is the obvious candidate but I did not inspect the volume contents, and the pod that filled it is gone. - **Whether any job actually failed.** I did not correlate against Forgejo Actions run history. ## Suggested direction, not a prescription Three options, in rough order of how much they change: 1. **Raise the limit.** Cheapest, and ser8 has ~931GB free. Does not stop the growth, just moves the wall. 2. **Recycle the volume on a schedule.** There is already a `forgejo-runner-recycle` CronJob on ser8 (last succeeded 19h30m ago) and an `init-wipe-docker-lib` init container on the flight-deck build runner. Whether the gaming runner is in scope for either is worth checking — it may simply have been missed. 3. **Match the PVC shape** the build runners use, so the volume is sized and observable rather than an unbounded scratch area. Option 2 looks most likely to be the actual gap, since the machinery already exists. ## Acceptance - The gaming runner does not evict itself during ordinary build load. - If evictions remain possible, they are visible — right now this was found by reading node events during an unrelated investigation, not by an alert. ## Related - https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/190 — nothing alerts on this class of failure either. ## Next owner Ops.

coilyco-ops commented

2026-08-13 07:07:55 +00:00

Author

Member

The first step in this issue cannot be run from an ops session

The body asks whether the eviction is recurrent, since the event carries count: 1 and retention may be hiding earlier ones. That check needs kubectl against ser8, and the guarded surface has only one context:

CURRENT   NAME         CLUSTER      AUTHINFO     NAMESPACE
*         kai-server   kai-server   kai-server

So from ops I can reach kai-server and not ser8. The ser8 node-stats MCP fills part of the gap — node conditions, pod inventory, restart counts, container memory, a bounded recent event window — which is how the eviction was spotted at all. What it does not expose is anything historical or per-object: no describe, no container termination reasons, no event history beyond its window.

Concretely, this blocks:

Here — whether forgejo-runner-ser8-gaming-0 has evicted before, and how often.
coilyco-bridge/deploy#436 — whether node-stats-exporter on ser8 is OOMKilled. On kai-server the equivalent question was settled in one kubectl describe (see coilyco-bridge/deploy#433, where it corrected a wrong diagnosis of mine).

That asymmetry is worth naming on its own: ser8 runs Agent Proxy, LiteLLM, SigNoz, and five Forgejo runners, and an ops session can observe it but not interrogate it. Both of the ser8 findings from this morning's sweep end at the same wall, and in the one case where the same wall existed on kai-server, crossing it reversed the conclusion.

Not proposing a specific fix — whether that is a second kubectl context, a widened node-stats surface, or deliberately keeping ser8 out of reach is a design call rather than a gap to patch. Recording it because two open issues now depend on it.

## The first step in this issue cannot be run from an ops session The body asks whether the eviction is recurrent, since the event carries `count: 1` and retention may be hiding earlier ones. That check needs `kubectl` against ser8, and the guarded surface has only one context: ``` CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kai-server kai-server kai-server ``` So from ops I can reach kai-server and not ser8. The ser8 `node-stats` MCP fills part of the gap — node conditions, pod inventory, restart counts, container memory, a bounded recent event window — which is how the eviction was spotted at all. What it does not expose is anything historical or per-object: no `describe`, no container termination reasons, no event history beyond its window. Concretely, this blocks: - **Here** — whether `forgejo-runner-ser8-gaming-0` has evicted before, and how often. - https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/436 — whether `node-stats-exporter` on ser8 is `OOMKilled`. On kai-server the equivalent question was settled in one `kubectl describe` (see https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/433, where it corrected a wrong diagnosis of mine). That asymmetry is worth naming on its own: ser8 runs Agent Proxy, LiteLLM, SigNoz, and five Forgejo runners, and an ops session can observe it but not interrogate it. Both of the ser8 findings from this morning's sweep end at the same wall, and in the one case where the same wall existed on kai-server, crossing it reversed the conclusion. Not proposing a specific fix — whether that is a second kubectl context, a widened node-stats surface, or deliberately keeping ser8 out of reach is a design call rather than a gap to patch. Recording it because two open issues now depend on it.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#808

No description provided.

Rows
Columns