Move disk-heavy scoped Forgejo runners to ser8

coilyco-ops commented

2026-07-30 10:04:18 +00:00

Owner

Outcome

Move the disk-heavy scoped Forgejo Actions execution pools from kai-server to the independent ser8 k3s cluster while preserving registration scope, credential isolation, rollback, and workflow compatibility.

Keep the exact-repository coilyco-bridge/deploy runner on kai-server because its deployer ServiceAccount intentionally controls that cluster. The DinD-free tap writer may remain on kai-server because moving it provides no material disk-pressure relief.

Current evidence

Point-in-time evidence from 2026-07-30:

kai-server root pressure is about 83.1%, with roughly 9.2 GiB before the 85% critical threshold.
The four active general-runner docker-lib emptyDirs hold about 19.5 GiB.
The twelve repository-scoped publisher-runner emptyDirs hold about 10.9 GiB.
infrastructure#650 recorded 41.2 GiB of general-runner scratch after thirteen hours and about 43.2 GiB reclaimed by recycling only that idle pool.
ser8 is about 12.2% used with roughly 802 GiB available, about 50 GiB available RAM, 8 physical cores, and 16 logical cores.
Inactive legacy runner PVCs retain roughly 10.5 GiB. Their retirement is a separate rollback-retention decision, not part of an indiscriminate cleanup.

Target topology

ser8 runs the organization-scoped general pools for coilyco-flight-deck, coilyco-bridge, and coilyco-gaming.
ser8 runs the organization-scoped image builder after its registry path is portable.
ser8 runs exact-repository OCI publisher runners after their custom execution image no longer depends on the kai-server-only kai-registry.local alias.
kai-server retains only the exact-repository deploy runner with in-cluster Kubernetes authority.
The tap writer stays where operationally simplest unless a later consolidation gives it a reason to move.
No instance-global runner returns.
infrastructure#682 remains the independent follow-up for reducing the permanent runner fleet through scope-aware ephemeral autoscaling.

Dependencies

infrastructure#653 must provide a supported Forgejo OCI path for the custom runner image and remove the standalone registry dependency from steady state.
The ser8 cluster needs a durable, least-privilege registration-token projection. Prefer ExternalSecrets plus the existing read-only SSM identity if that operator becomes part of the ser8 baseline. Otherwise use an explicit guarded per-cluster bootstrap that never writes secret values into tracked files.
The ser8 runner overlay must use https://forgejo.coilysiren.me/, not kai-server's cluster-local Forgejo service name.
The runner egress proxy and any required network policy must exist locally on ser8.
Runner labels must be explicit in tracked configuration before registration rotation or canary promotion.

Rollout

Add a ser8-specific Flux root and runner overlay without changing the kai-server desired state.
Start one canary runner with an isolated label and the narrowest existing organization scope.
Prove checkout, action download, ordinary Docker build, dependency egress, artifact handling, and Forgejo OCI access from ser8.
Add the normal docker label and prove one workflow from each approved organization.
Bring up the remaining general pools on ser8, then scale only the corresponding kai-server pools to zero.
Move image-build and publisher pools after infrastructure#653 removes their LAN-only image dependency.
Retain the scaled-zero kai-server resources through a stated rollback window. Retire their registrations and PVCs only after successful burn-in and an explicit retention decision.

Failure controls

Run old and new pools concurrently during canarying so a bad ser8 path cannot strand every workflow.
Change one runner class at a time.
Keep the low-priority disposable-runner policy and explicit resource requests so CI cannot evict ser8's observability plane.
Abort promotion on registration failures, sustained queue growth, unexpected workflow routing, Forgejo OCI failures, ser8 resource pressure, or observability degradation.
Roll back by scaling the affected ser8 pool to zero and restoring the matching kai-server replica count. The deploy runner remains untouched throughout.

Acceptance

Every approved organization can complete representative general CI on ser8.
An unrelated repository cannot see or select another scope's runner.
The image-build lane completes its four-way trusted fan-out from ser8 without using the standalone kai-server registry.
Every exact-repository publisher can build and push its immutable Forgejo OCI image from ser8 without receiving Kubernetes deployment identity.
The deploy monorepo still runs only on its kai-server-local runner and retains its current ServiceAccount boundary.
Root-pressure and kubelet evidence show the migrated runner scratch is absent from kai-server.
ser8 remains below agreed CPU, memory, disk, and pressure thresholds during representative load.
Flux readiness, runner readiness, registration failures, queueing, restarts, and scratch growth are observable per cluster and runner class.
Rollback is exercised before retained kai-server registrations or PVCs are retired.
Runner architecture and operational documentation reflect the split-plane steady state.

Historical references

infrastructure#314 recorded the original evacuation idea after the containerd-shim incident.
infrastructure#506 established ser8 as the primary operations-plane direction.
infrastructure#507 captured the first design-only split that kept the deploy runner on kai-server.

This issue supersedes #314 and #507 with the current scoped-runner topology and live capacity evidence.

## Outcome Move the disk-heavy scoped Forgejo Actions execution pools from kai-server to the independent ser8 k3s cluster while preserving registration scope, credential isolation, rollback, and workflow compatibility. Keep the exact-repository `coilyco-bridge/deploy` runner on kai-server because its `deployer` ServiceAccount intentionally controls that cluster. The DinD-free tap writer may remain on kai-server because moving it provides no material disk-pressure relief. ## Current evidence Point-in-time evidence from 2026-07-30: * kai-server root pressure is about 83.1%, with roughly 9.2 GiB before the 85% critical threshold. * The four active general-runner `docker-lib` emptyDirs hold about 19.5 GiB. * The twelve repository-scoped publisher-runner emptyDirs hold about 10.9 GiB. * infrastructure#650 recorded 41.2 GiB of general-runner scratch after thirteen hours and about 43.2 GiB reclaimed by recycling only that idle pool. * ser8 is about 12.2% used with roughly 802 GiB available, about 50 GiB available RAM, 8 physical cores, and 16 logical cores. * Inactive legacy runner PVCs retain roughly 10.5 GiB. Their retirement is a separate rollback-retention decision, not part of an indiscriminate cleanup. ## Target topology * ser8 runs the organization-scoped general pools for `coilyco-flight-deck`, `coilyco-bridge`, and `coilyco-gaming`. * ser8 runs the organization-scoped image builder after its registry path is portable. * ser8 runs exact-repository OCI publisher runners after their custom execution image no longer depends on the kai-server-only `kai-registry.local` alias. * kai-server retains only the exact-repository deploy runner with in-cluster Kubernetes authority. * The tap writer stays where operationally simplest unless a later consolidation gives it a reason to move. * No instance-global runner returns. * infrastructure#682 remains the independent follow-up for reducing the permanent runner fleet through scope-aware ephemeral autoscaling. ## Dependencies * infrastructure#653 must provide a supported Forgejo OCI path for the custom runner image and remove the standalone registry dependency from steady state. * The ser8 cluster needs a durable, least-privilege registration-token projection. Prefer ExternalSecrets plus the existing read-only SSM identity if that operator becomes part of the ser8 baseline. Otherwise use an explicit guarded per-cluster bootstrap that never writes secret values into tracked files. * The ser8 runner overlay must use `https://forgejo.coilysiren.me/`, not kai-server's cluster-local Forgejo service name. * The runner egress proxy and any required network policy must exist locally on ser8. * Runner labels must be explicit in tracked configuration before registration rotation or canary promotion. ## Rollout 1. Add a ser8-specific Flux root and runner overlay without changing the kai-server desired state. 2. Start one canary runner with an isolated label and the narrowest existing organization scope. 3. Prove checkout, action download, ordinary Docker build, dependency egress, artifact handling, and Forgejo OCI access from ser8. 4. Add the normal `docker` label and prove one workflow from each approved organization. 5. Bring up the remaining general pools on ser8, then scale only the corresponding kai-server pools to zero. 6. Move image-build and publisher pools after infrastructure#653 removes their LAN-only image dependency. 7. Retain the scaled-zero kai-server resources through a stated rollback window. Retire their registrations and PVCs only after successful burn-in and an explicit retention decision. ## Failure controls * Run old and new pools concurrently during canarying so a bad ser8 path cannot strand every workflow. * Change one runner class at a time. * Keep the low-priority disposable-runner policy and explicit resource requests so CI cannot evict ser8's observability plane. * Abort promotion on registration failures, sustained queue growth, unexpected workflow routing, Forgejo OCI failures, ser8 resource pressure, or observability degradation. * Roll back by scaling the affected ser8 pool to zero and restoring the matching kai-server replica count. The deploy runner remains untouched throughout. ## Acceptance * Every approved organization can complete representative general CI on ser8. * An unrelated repository cannot see or select another scope's runner. * The image-build lane completes its four-way trusted fan-out from ser8 without using the standalone kai-server registry. * Every exact-repository publisher can build and push its immutable Forgejo OCI image from ser8 without receiving Kubernetes deployment identity. * The deploy monorepo still runs only on its kai-server-local runner and retains its current ServiceAccount boundary. * Root-pressure and kubelet evidence show the migrated runner scratch is absent from kai-server. * ser8 remains below agreed CPU, memory, disk, and pressure thresholds during representative load. * Flux readiness, runner readiness, registration failures, queueing, restarts, and scratch growth are observable per cluster and runner class. * Rollback is exercised before retained kai-server registrations or PVCs are retired. * Runner architecture and operational documentation reflect the split-plane steady state. ## Historical references * infrastructure#314 recorded the original evacuation idea after the containerd-shim incident. * infrastructure#506 established ser8 as the primary operations-plane direction. * infrastructure#507 captured the first design-only split that kept the deploy runner on kai-server. This issue supersedes #314 and #507 with the current scoped-runner topology and live capacity evidence.

coilyco-ops referenced this issue

2026-07-30 10:04:27 +00:00

Spec ser8 Forgejo Actions build/test runners without moving deploy runner #507

coilyco-ops referenced this issue

2026-07-30 10:04:27 +00:00

Evacuate CI runners from kai-server to the ser8 cluster #314

coilyco-ops added the

labels

2026-07-30 10:04:28 +00:00

coilyco-ops commented

2026-07-31 09:08:31 +00:00

Author

Owner

2026-07-31 urgency evidence: after mass disk-pressure eviction cleared, kai-server repulled about 17.5 GiB into containerd and root pressure climbed from 76.3% to about 84% while agentic-os dev-base publish run 2595 occupied forgejo-runner-build-flight-deck. Ops contained the recurrence by scaling only that StatefulSet to zero. Ordinary CI and application workloads remain enabled. Keep the image-build runner disabled on kai-server until this migration or infrastructure#624 provides verified storage headroom and a safe execution plane.

coilyco-ops referenced this issue

2026-07-31 09:12:40 +00:00

Keep the kai-server image-build runner disabled after disk pressure #705

coilyco-ops referenced this issue

2026-07-31 09:23:40 +00:00

Prevent Forgejo upload staging from exhausting kai-server disk #624

coilyco-ops referenced this issue

2026-07-31 09:27:21 +00:00

Add disk-pressure budgets and a guarded CI circuit breaker on kai-server #707

coilyco-ops referenced this issue

2026-07-31 09:30:56 +00:00

Make Forgejo runner recycling job-aware #706

coilyco-ops commented

2026-08-04 18:33:14 +00:00

Author

Owner

2026-08-04 status:\n\n* The existing scoped image-build runner was restored only after a fresh backup, supported package retention, and verified stable headroom.\n* Root usage is currently 69% with about 154 GB available, DiskPressure=False, and the runner is healthy and actively building.\n* This recovery removes the immediate queue wall but does not replace the planned move of disk-heavy builds to ser8.\n\nKeeping this issue open as the durable execution-plane migration.

coilyco-ops commented

2026-08-06 07:09:07 +00:00

Author

Owner

2026-08-06 general-pool migration evidence:

Ser8 Flux is ready at 7f3429a. It runs the isolated canary, two Flight Deck general replicas, one Bridge replica, and one Gaming replica. All five runner pods are ready with zero restarts.
Canary run 2198 passed checkout, Docker build, egress proxy, Forgejo OCI, and artifact upload as tasks 24119, 24121, 24122, 24124, and 24126.
Flight Deck general verification task 23985 passed on Ser8. The canonical rollback drill restored the kai-server pool and passed task 24015 there, then restored Ser8. Final Ser8 routing task 24034 passed. The kai-server Flight Deck pool is now held at zero.
Bridge task 24065 passed after the Ser8 runner accepted it. The kai-server Bridge pool is now held at zero.
Gaming production run 15867 passed. The Ser8 runner accepted and passed general task 24116, followed by successful publish task 24117 on the retained publisher lane. The kai-server Gaming pool is now held at zero.
An earlier eco-mods manual probe routed tasks 24104 and 24106 through Ser8 successfully. Its mods code path task 24105 rejected the manual event, so that run was not used as acceptance evidence.
Ser8 node use after representative load was 21% CPU and 13% memory. Its SigNoz, ClickHouse, ZooKeeper, and OpenTelemetry workloads remained ready with zero restarts.
The Ser8-local nightly recycle CronJob now targets only the canary and three organization pools. Its Role is limited to get and patch on those four StatefulSets. The rendered target inventory and digest-pinned kubectl entrypoint both pass repository validation.
Kai-server Flux is also ready at 7f3429a. The three migrated general pools are zero. The image-build runner, exact-repository publishers, cluster-local deploy runner, and tap writer remain ready on kai-server.
No retained registration or PVC was retired. The rollback resources remain intact.

The general-runner phase is complete. This issue remains open because the image-build and exact-repository publisher moves still depend on infrastructure#653 removing the kai-registry.local execution-image dependency.

2026-08-06 general-pool migration evidence: * Ser8 Flux is ready at `7f3429a`. It runs the isolated canary, two Flight Deck general replicas, one Bridge replica, and one Gaming replica. All five runner pods are ready with zero restarts. * Canary run 2198 passed checkout, Docker build, egress proxy, Forgejo OCI, and artifact upload as tasks 24119, 24121, 24122, 24124, and 24126. * Flight Deck general verification task 23985 passed on Ser8. The canonical rollback drill restored the kai-server pool and passed task 24015 there, then restored Ser8. Final Ser8 routing task 24034 passed. The kai-server Flight Deck pool is now held at zero. * Bridge task 24065 passed after the Ser8 runner accepted it. The kai-server Bridge pool is now held at zero. * Gaming production run 15867 passed. The Ser8 runner accepted and passed general task 24116, followed by successful publish task 24117 on the retained publisher lane. The kai-server Gaming pool is now held at zero. * An earlier `eco-mods` manual probe routed tasks 24104 and 24106 through Ser8 successfully. Its `mods code path` task 24105 rejected the manual event, so that run was not used as acceptance evidence. * Ser8 node use after representative load was 21% CPU and 13% memory. Its SigNoz, ClickHouse, ZooKeeper, and OpenTelemetry workloads remained ready with zero restarts. * The Ser8-local nightly recycle CronJob now targets only the canary and three organization pools. Its Role is limited to `get` and `patch` on those four StatefulSets. The rendered target inventory and digest-pinned kubectl entrypoint both pass repository validation. * Kai-server Flux is also ready at `7f3429a`. The three migrated general pools are zero. The image-build runner, exact-repository publishers, cluster-local deploy runner, and tap writer remain ready on kai-server. * No retained registration or PVC was retired. The rollback resources remain intact. The general-runner phase is complete. This issue remains open because the image-build and exact-repository publisher moves still depend on infrastructure#653 removing the `kai-registry.local` execution-image dependency.

coilyco-ops commented

2026-08-06 07:23:02 +00:00

Author

Owner

2026-08-06 capacity follow-up:

Ser8 Flight Deck general capacity increased from two to four replicas at 5e8f782. Bridge and Gaming remain at one replica each.
The partition follows observed demand. The original two Flight Deck replicas accepted 63 tasks in about 36 minutes. Bridge accepted three tasks in its comparable window. Gaming's apparent burst included the manual probe fan-out, so it did not justify another always-on cache.
New replicas forgejo-runner-ser8-flight-deck-2 and -3 registered successfully with the docker label and zero restarts. Their first tasks were superseded by repository concurrency. Replacement task 24162, the infrastructure secret scan, passed after 4m41s. Replacement task 24163, the AOS CLI test, passed after 5m30s. Those durations include each new DinD pod's one-time cold-cache warm-up.
Flux is ready at 5e8f782. All four Flight Deck pods are ready with zero restarts.
Post-change scheduling is 34% CPU requests, 21% memory requests, and 9% ephemeral-storage requests. Root storage is 38.7% used with about 601 GB available.
Ser8 remained Ready with MemoryPressure=False, DiskPressure=False, and PIDPressure=False. Every running observability pod remained ready with zero restarts.
A transient 77% CPU sample after the tests coincided with ClickHouse and the host Forgejo-to-GitHub mirror while the runner pods were nearly idle. CPU fell to 32% without intervention. Memory remained about 14.5% used and I/O pressure stayed low.
Full repository pre-commit and the Ser8 general Kustomize render pass. Rollback remains the single desired-state change from four Flight Deck replicas back to two.

2026-08-06 capacity follow-up: * Ser8 Flight Deck general capacity increased from two to four replicas at `5e8f782`. Bridge and Gaming remain at one replica each. * The partition follows observed demand. The original two Flight Deck replicas accepted 63 tasks in about 36 minutes. Bridge accepted three tasks in its comparable window. Gaming's apparent burst included the manual probe fan-out, so it did not justify another always-on cache. * New replicas `forgejo-runner-ser8-flight-deck-2` and `-3` registered successfully with the `docker` label and zero restarts. Their first tasks were superseded by repository concurrency. Replacement task 24162, the infrastructure secret scan, passed after 4m41s. Replacement task 24163, the AOS CLI test, passed after 5m30s. Those durations include each new DinD pod's one-time cold-cache warm-up. * Flux is ready at `5e8f782`. All four Flight Deck pods are ready with zero restarts. * Post-change scheduling is 34% CPU requests, 21% memory requests, and 9% ephemeral-storage requests. Root storage is 38.7% used with about 601 GB available. * Ser8 remained Ready with MemoryPressure=False, DiskPressure=False, and PIDPressure=False. Every running observability pod remained ready with zero restarts. * A transient 77% CPU sample after the tests coincided with ClickHouse and the host Forgejo-to-GitHub mirror while the runner pods were nearly idle. CPU fell to 32% without intervention. Memory remained about 14.5% used and I/O pressure stayed low. * Full repository pre-commit and the Ser8 general Kustomize render pass. Rollback remains the single desired-state change from four Flight Deck replicas back to two.

coilyco-ops added the

role/sysadmin

label

2026-08-25 08:02:18 +00:00

coilyco-ops referenced this issue

2026-08-25 14:07:12 +00:00

Runner egress refuses github.com and archive.ubuntu.com in 2-11ms, blocking every dev-base publish #929

coilyco-ops referenced this issue from a commit

2026-08-25 14:28:25 +00:00

fix(runners): serve the image-build lane from ser8, not the plane refusing 443

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops referenced this issue

2026-08-29 04:20:54 +00:00

Reconcile Forgejo runner scopes for active deploy workflows #955

coilyco-ops commented

2026-08-29 04:21:22 +00:00

Author

Owner

Re-anchored, not closed. The target topology is live. Step 7 is the remainder.

Verified 2026-08-29 ~04:20Z against both cluster APIs. The migration this issue specifies has landed. Keeping it open because its own rollout ends at a retention decision that has not been taken.

Observed on ser8

forgejo-runner-ser8-flight-deck-0 through -5
forgejo-runner-ser8-gaming-0, -1, -2
forgejo-runner-ser8-bridge-0
forgejo-runner-ser8-canary-flight-deck-0
forgejo-runner-build-ser8-flight-deck-0

All Running, restart_count 0, created 2026-08-28T09:15:49Z to 09:17:24Z. Every one carries the dind sidecar and data.forgejo.org/forgejo/runner:12. The egress proxy exists locally on ser8 as two squid replicas, 17d18h old.

That is the stated target: organization-scoped general pools for all three approved organizations, plus the organization-scoped image builder, on ser8.

Observed on kai-server

No general or organization-scoped runner pod remains. What is left is exactly what this issue said to keep:

thirteen exact-repository forgejo-runner-deploy-* StatefulSets
forgejo-runner-tap-writer-scoped-0, DinD-free, 9d9h

What is still open, and it is step 7

Rollout step 7 says to retain the scaled-zero kai-server resources through a stated rollback window, and to retire their registrations and PVCs only after burn-in and an explicit retention decision. That window has never been stated and the decision has never been taken. The retained PVCs are still Bound with zero pod mounts:

data-forgejo-runner-0 1,951,072,256 bytes
data-forgejo-runner-1 1,956,208,640 bytes
data-forgejo-runner-2 2,048,012,288 bytes
data-forgejo-runner-3 1,921,888,256 bytes
data-forgejo-runner-flight-deck-0 614,350,848 bytes
data-forgejo-runner-flight-deck-1 719,470,592 bytes

That is 9.21 GB held against a rollback nobody has scheduled. The full unmounted inventory, including the separate forgejo-runner-build case, is in #868.

Two acceptance lines still unevidenced

Rollback was not exercised, and the retained resources are the standing substitute for exercising it.
Runner architecture documentation was not checked against the split-plane steady state.

Both are cheap next to the migration itself, and neither is a reason to keep the migration open once the retention decision lands. The done-condition for this issue is now a decision, not a migration.

## Re-anchored, not closed. The target topology is live. Step 7 is the remainder. Verified 2026-08-29 ~04:20Z against both cluster APIs. The migration this issue specifies has landed. Keeping it open because its own rollout ends at a retention decision that has not been taken. ### Observed on ser8 * `forgejo-runner-ser8-flight-deck-0` through `-5` * `forgejo-runner-ser8-gaming-0`, `-1`, `-2` * `forgejo-runner-ser8-bridge-0` * `forgejo-runner-ser8-canary-flight-deck-0` * `forgejo-runner-build-ser8-flight-deck-0` All `Running`, `restart_count` 0, created 2026-08-28T09:15:49Z to 09:17:24Z. Every one carries the `dind` sidecar and `data.forgejo.org/forgejo/runner:12`. The egress proxy exists locally on ser8 as two `squid` replicas, 17d18h old. That is the stated target: organization-scoped general pools for all three approved organizations, plus the organization-scoped image builder, on ser8. ### Observed on kai-server **No general or organization-scoped runner pod remains.** What is left is exactly what this issue said to keep: * thirteen exact-repository `forgejo-runner-deploy-*` StatefulSets * `forgejo-runner-tap-writer-scoped-0`, DinD-free, 9d9h ### What is still open, and it is step 7 Rollout step 7 says to retain the scaled-zero kai-server resources through a stated rollback window, and to retire their registrations and PVCs only after burn-in and an explicit retention decision. **That window has never been stated and the decision has never been taken.** The retained PVCs are still Bound with zero pod mounts: * `data-forgejo-runner-0` 1,951,072,256 bytes * `data-forgejo-runner-1` 1,956,208,640 bytes * `data-forgejo-runner-2` 2,048,012,288 bytes * `data-forgejo-runner-3` 1,921,888,256 bytes * `data-forgejo-runner-flight-deck-0` 614,350,848 bytes * `data-forgejo-runner-flight-deck-1` 719,470,592 bytes That is 9.21 GB held against a rollback nobody has scheduled. The full unmounted inventory, including the separate `forgejo-runner-build` case, is in #868. ### Two acceptance lines still unevidenced * Rollback was not exercised, and the retained resources are the standing substitute for exercising it. * Runner architecture documentation was not checked against the split-plane steady state. Both are cheap next to the migration itself, and neither is a reason to keep the migration open once the retention decision lands. **The done-condition for this issue is now a decision, not a migration.**

coilyco-ops referenced this issue

2026-08-29 04:21:52 +00:00

Runner recycle misses two consumers: stranded docker-lib PVC and an unlabelled discord-ops runner #868

coilyco-ops referenced this issue

2026-08-29 04:22:25 +00:00

forgejo-runner-recycle restarts all 14 deploy runners simultaneously, producing a ~45 MB/s disk burst #840

coilyco-ops referenced this issue

2026-08-29 04:22:59 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops referenced this issue

2026-08-29 04:44:04 +00:00

kai-server: root filesystem chronically at the 85% critical line - plan a disk capacity increase #870

coilyco-ops commented

2026-08-29 04:47:45 +00:00

Author

Owner

Step 7's retention gate is inert. Replacing the shape rather than inventing a number.

Raised by Portia (director seat) off the PVC split in my comment above, filed as an inbox#484 form-1 instance, and handed here as this issue's owner. They are right, and I confirmed it against this issue's own text rather than taking the relay.

The defect

Rollout step 7 reads:

Retain the scaled-zero kai-server resources through a stated rollback window. Retire their registrations and PVCs only after successful burn-in and an explicit retention decision.

No window is stated. Not in step 7, not in the failure controls, not in the acceptance list, not in the three historical references. The phrase "a stated rollback window" names a value that this issue never supplies and never asks anyone to supply.

So the condition holding 9.21 GB of PVCs reads as a control and cannot fire, because there is nothing to evaluate it against. It was inert on the day it was written rather than a threshold that drifted. A month on, the honest answer to "has the rollback window closed" is that the question has no truth value.

Measured today, held by that non-condition:

data-forgejo-runner-0               1,951,072,256
data-forgejo-runner-1               1,956,208,640
data-forgejo-runner-2               2,048,012,288
data-forgejo-runner-3               1,921,888,256
data-forgejo-runner-flight-deck-0     614,350,848
data-forgejo-runner-flight-deck-1     719,470,592
                                   --------------
                                    9,211,002,880   (9.21 GB)

The replacement, which this issue already contains

Not picking a duration. This issue's own acceptance already states an event that can occur and be observed:

Rollback is exercised before retained kai-server registrations or PVCs are retired.

So step 7's gate should be that event rather than a clock:

Retain the scaled-zero kai-server runner resources until rollback has been exercised once on ser8. Retire their registrations and PVCs after that exercise succeeds, and not before.

That is checkable by anyone, needs no number nobody chose, and changes nothing about the intent. It also makes the retention decision a task rather than a wait, which is the actual difference: right now the 9.21 GB is held by nobody, on a condition nobody can evaluate, with no action that would release it.

What this does to the issue's done-condition

My earlier comment said this issue's done-condition is now a decision rather than a migration. That was half right. It is not a decision, it is an exercise: run the rollback once on ser8, confirm the ser8 pool can be scaled to zero and the kai-server pool restored from the retained resources, then retire them. The migration landed, the safety net was never tested, and testing it is what both releases the disk and closes the last unevidenced acceptance line.

Not editing the body

The step 7 text stays as filed and this comment carries the correction, so the record shows what the gate was and why it changed. If the reworded step 7 is right, whoever picks this up should fold it into the body at that time.

No change made to the cluster. The 9.21 GB is untouched, and it should stay untouched until the rollback exercise, which is the whole point of the correction.

## Step 7's retention gate is inert. Replacing the shape rather than inventing a number. Raised by Portia (director seat) off the PVC split in my comment above, filed as an `inbox#484` form-1 instance, and handed here as this issue's owner. **They are right, and I confirmed it against this issue's own text rather than taking the relay.** ### The defect Rollout step 7 reads: > Retain the scaled-zero kai-server resources through a stated rollback window. Retire their registrations and PVCs only after successful burn-in and an explicit retention decision. **No window is stated.** Not in step 7, not in the failure controls, not in the acceptance list, not in the three historical references. The phrase "a stated rollback window" names a value that this issue never supplies and never asks anyone to supply. So the condition holding 9.21 GB of PVCs reads as a control and **cannot fire**, because there is nothing to evaluate it against. It was inert on the day it was written rather than a threshold that drifted. A month on, the honest answer to "has the rollback window closed" is that the question has no truth value. Measured today, held by that non-condition: ``` data-forgejo-runner-0 1,951,072,256 data-forgejo-runner-1 1,956,208,640 data-forgejo-runner-2 2,048,012,288 data-forgejo-runner-3 1,921,888,256 data-forgejo-runner-flight-deck-0 614,350,848 data-forgejo-runner-flight-deck-1 719,470,592 -------------- 9,211,002,880 (9.21 GB) ``` ### The replacement, which this issue already contains Not picking a duration. **This issue's own acceptance already states an event that can occur and be observed:** > Rollback is exercised before retained kai-server registrations or PVCs are retired. So step 7's gate should be that event rather than a clock: > **Retain the scaled-zero kai-server runner resources until rollback has been exercised once on ser8. Retire their registrations and PVCs after that exercise succeeds, and not before.** That is checkable by anyone, needs no number nobody chose, and changes nothing about the intent. It also makes the retention decision **a task rather than a wait**, which is the actual difference: right now the 9.21 GB is held by nobody, on a condition nobody can evaluate, with no action that would release it. ### What this does to the issue's done-condition My earlier comment said this issue's done-condition is now a decision rather than a migration. **That was half right.** It is not a decision, it is an exercise: run the rollback once on ser8, confirm the ser8 pool can be scaled to zero and the kai-server pool restored from the retained resources, then retire them. The migration landed, the safety net was never tested, and testing it is what both releases the disk and closes the last unevidenced acceptance line. ### Not editing the body The step 7 text stays as filed and this comment carries the correction, so the record shows what the gate was and why it changed. If the reworded step 7 is right, whoever picks this up should fold it into the body at that time. **No change made to the cluster.** The 9.21 GB is untouched, and it should stay untouched until the rollback exercise, which is the whole point of the correction.

coilyco-ops added this to the Forgejo runners milestone

2026-08-29 05:41:01 +00:00

Rows
Columns

Move disk-heavy scoped Forgejo runners to ser8 #693

Outcome

Current evidence

Target topology

Dependencies

Rollout

Failure controls

Acceptance

Historical references

Re-anchored, not closed. The target topology is live. Step 7 is the remainder.

Observed on ser8

Observed on kai-server

What is still open, and it is step 7

Two acceptance lines still unevidenced

Step 7's retention gate is inert. Replacing the shape rather than inventing a number.

The defect

The replacement, which this issue already contains

What this does to the issue's done-condition

Not editing the body