coilyco-flight-deck/agentic-os

Watch

dev-base release blocked: publish-lang-rust builds but cannot push, and kai-server root is at 87% critical #1060

New issue

Open

opened 2026-08-14 07:06:53 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-08-14 07:06:53 +00:00

Member

Release run for 864e3a08 (#1058, the Rust toolchain bake) failed in publish-lang-rust. The image built. The push failed.

What the log shows

Both architectures built through every stage, then:

07:01:49  #24 exporting manifest list sha256:89032ba2... 0.1s done
07:01:49  #24 pushing layers
07:03:09  #24 pushing layers 80.0s done
07:03:09  #24 ERROR: failed to push
          forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:lang-rust-draft-864e3a08...: unknown

So this is not a Dockerfile fault. publish-full skipped, release skipped, and no release tag was cut - v0.273.0 at 06:32 predates the merge at 06:41, so the current :release image does not contain the fix.

Node disk is critical

root filesystem: 87.19% used   status: critical
  total 480 GiB | available 61 GiB
  11.3 GiB past its own 85% critical threshold
  37.0 GiB past its 80% warn threshold
inodes: 10.5% - not the constraint

Registry PVC: 20 GiB capacity, at least 7.2 GiB used (scan timed out, so that is a lower bound). The forgejo namespace holds at least 17.7 GiB across 35 volumes; several runner docker-lib PVCs are multi-GiB.

What I can and cannot conclude

Circumstantial, and I want to be precise about it. The four smaller language payloads - dotnet, go, node, python - pushed successfully in the same run. Only lang-rust failed, and it is both the largest payload and the one my change just enlarged, by an extra toolchain with clippy, rustfmt, and a wasm32 target across two architectures.

That is consistent with running out of room, but unknown is a generic registry error and I have read-only observability. I have not proven the registry write failed for lack of space, only that the node is critically full and the largest push is the one that failed.

Ops actions, and the evidence each would produce

Reclaim space on kai-server root. Expected evidence: get_filesystem_pressure returns status below critical. Candidates visible from here are registry garbage collection and the runner docker-lib PVCs, one of which already has an init-wipe-docker-lib init container.
Read the registry pod's own logs during a retried push. That is what separates "disk full" from an unrelated registry fault, and it needs in-cluster log access I do not have.
Re-run the release once space is reclaimed. Expected evidence: publish-lang-rust succeeds and a release tag appears with a timestamp after 06:41:54Z.

Blast radius

This blocks more than my change. No dev-base release can ship while this stands, because every release publishes all five payloads. It also blocks the downstream chain it was meant to unblock: coilyco-gaming/galaxy-gen#84 needs a :release image carrying the baked toolchain before its CI stops failing on a rustup fetch, and its publish has not run since, leaving that site four commits behind its own main.

If the size increase is judged unacceptable

The bake is one line to narrow. RUST_PINNED_VERSIONS is a space-separated list and could be emptied, or scoped to a single architecture, rather than reverted. But that returns galaxy-gen to failing on the network path, so it trades one outage for another.

Release run for `864e3a08` (#1058, the Rust toolchain bake) failed in `publish-lang-rust`. **The image built. The push failed.** ## What the log shows Both architectures built through every stage, then: ``` 07:01:49 #24 exporting manifest list sha256:89032ba2... 0.1s done 07:01:49 #24 pushing layers 07:03:09 #24 pushing layers 80.0s done 07:03:09 #24 ERROR: failed to push forgejo.coilysiren.me/coilyco-flight-deck/agentic-os:lang-rust-draft-864e3a08...: unknown ``` So this is not a Dockerfile fault. `publish-full` skipped, `release` skipped, and no release tag was cut - `v0.273.0` at 06:32 predates the merge at 06:41, so **the current `:release` image does not contain the fix**. ## Node disk is critical ``` root filesystem: 87.19% used status: critical total 480 GiB | available 61 GiB 11.3 GiB past its own 85% critical threshold 37.0 GiB past its 80% warn threshold inodes: 10.5% - not the constraint ``` Registry PVC: 20 GiB capacity, at least 7.2 GiB used (scan timed out, so that is a lower bound). The `forgejo` namespace holds at least 17.7 GiB across 35 volumes; several runner `docker-lib` PVCs are multi-GiB. ## What I can and cannot conclude **Circumstantial, and I want to be precise about it.** The four smaller language payloads - dotnet, go, node, python - pushed successfully in the same run. Only `lang-rust` failed, and it is both the largest payload and the one my change just enlarged, by an extra toolchain with clippy, rustfmt, and a wasm32 target across two architectures. That is consistent with running out of room, but `unknown` is a generic registry error and I have read-only observability. **I have not proven the registry write failed for lack of space**, only that the node is critically full and the largest push is the one that failed. ## Ops actions, and the evidence each would produce 1. **Reclaim space on kai-server root.** Expected evidence: `get_filesystem_pressure` returns `status` below `critical`. Candidates visible from here are registry garbage collection and the runner `docker-lib` PVCs, one of which already has an `init-wipe-docker-lib` init container. 2. **Read the registry pod's own logs** during a retried push. That is what separates "disk full" from an unrelated registry fault, and it needs in-cluster log access I do not have. 3. **Re-run the release** once space is reclaimed. Expected evidence: `publish-lang-rust` succeeds and a release tag appears with a timestamp after 06:41:54Z. ## Blast radius This blocks more than my change. **No dev-base release can ship while this stands**, because every release publishes all five payloads. It also blocks the downstream chain it was meant to unblock: coilyco-gaming/galaxy-gen#84 needs a `:release` image carrying the baked toolchain before its CI stops failing on a rustup fetch, and its `publish` has not run since, leaving that site four commits behind its own main. ## If the size increase is judged unacceptable The bake is one line to narrow. `RUST_PINNED_VERSIONS` is a space-separated list and could be emptied, or scoped to a single architecture, rather than reverted. But that returns galaxy-gen to failing on the network path, so it trades one outage for another.

coilyco-ops added the

labels

2026-08-17 02:35:43 +00:00

coilyco-ops commented

2026-08-17 06:10:19 +00:00

Author

Member

Status recheck, and the disk half is materially worse than this issue records. Darren (director seat), 2026-08-17 06:10Z.

The trend

2026-08-14  (filed)   87.19%
2026-08-17 05:20Z     88.72%
2026-08-17 06:05Z     90.48%

Now 28.2 GiB past its own critical threshold, with 49.0 GiB available against 480 GiB total. It moved 1.76 points in forty minutes. At that rate the remaining headroom is hours rather than days, and both Sirens lanes are pinned to this node.

Where it is going, from a complete scan rather than a truncated one

An earlier scan timed out and reported lower bounds that badly understated this. A full 76-second scan of /var/lib/rancher/k3s/storage completed at 05:30 and totals 174.3 GiB across 43 volumes:

forgejo-data                                    114.0 GiB     (20 GiB PVC request)
docker-lib-forgejo-runner-build-flight-deck-0    30.3 GiB     (10 GiB request, mounted)
registry-data                                    13.1 GiB     (20 GiB request)
docker-lib-forgejo-runner-build-0                 6.5 GiB     no pod mount
data-forgejo-runner-2                             2.0 GiB     no pod mount
data-forgejo-runner-1                             1.96 GiB    no pod mount
data-forgejo-runner-0                             1.95 GiB    no pod mount
data-forgejo-runner-3                             1.92 GiB    no pod mount

forgejo-data is 5.7x its PVC request and is the whole problem. local-path does not enforce the quota, so nothing stopped it. The runner build cache at 30.3 GiB is 3x its request and second.

The orphaned PVCs total roughly 14.5 GiB. Still worth reclaiming, and now clearly secondary rather than the fix. I had earlier suggested otherwise on the strength of the truncated scan, and that was wrong.

The release half

Unchanged. services/agent-proxy/chart/values.yaml still pins 349dd56, and no dev-base release has been cut past the failure. The registry PVC at 13.1 GiB of 20 GiB is not itself full, so the push failure is more likely the node's root filesystem than the PVC quota - which makes the disk half and the release half the same problem rather than two.

What this needs

An operator, and quickly. The two candidate actions differ a lot in risk:

Reclaim the ~14.5 GiB of unmounted runner PVCs. Safe, reversible in the sense that they are caches, and buys back roughly half the overshoot. reclaimPolicy: Delete, so confirm the owning StatefulSets are gone rather than scaled to zero first.
Understand forgejo-data. 114 GiB is where the disk actually went, and it is live Forgejo state rather than cache, so it wants inspection before anything is removed. Host path is /var/lib/rancher/k3s/storage/pvc-e0167dde-7421-4ae2-aff0-44a6e5e92eb2_forgejo_forgejo-data.

Labels unchanged at priority/P0 autonomy/live-collab role/ops. Changing a live system is not a director's to do, and this needs judgement about what in Forgejo's data is disposable.

**Status recheck, and the disk half is materially worse than this issue records. Darren (director seat), 2026-08-17 06:10Z.** ## The trend ``` 2026-08-14 (filed) 87.19% 2026-08-17 05:20Z 88.72% 2026-08-17 06:05Z 90.48% ``` Now **28.2 GiB past its own critical threshold**, with 49.0 GiB available against 480 GiB total. It moved 1.76 points in forty minutes. At that rate the remaining headroom is hours rather than days, and both Sirens lanes are pinned to this node. ## Where it is going, from a complete scan rather than a truncated one An earlier scan timed out and reported lower bounds that badly understated this. A full 76-second scan of `/var/lib/rancher/k3s/storage` completed at 05:30 and totals **174.3 GiB across 43 volumes**: ``` forgejo-data 114.0 GiB (20 GiB PVC request) docker-lib-forgejo-runner-build-flight-deck-0 30.3 GiB (10 GiB request, mounted) registry-data 13.1 GiB (20 GiB request) docker-lib-forgejo-runner-build-0 6.5 GiB no pod mount data-forgejo-runner-2 2.0 GiB no pod mount data-forgejo-runner-1 1.96 GiB no pod mount data-forgejo-runner-0 1.95 GiB no pod mount data-forgejo-runner-3 1.92 GiB no pod mount ``` **`forgejo-data` is 5.7x its PVC request and is the whole problem.** `local-path` does not enforce the quota, so nothing stopped it. The runner build cache at 30.3 GiB is 3x its request and second. The orphaned PVCs total roughly 14.5 GiB. Still worth reclaiming, and now clearly secondary rather than the fix. I had earlier suggested otherwise on the strength of the truncated scan, and that was wrong. ## The release half Unchanged. `services/agent-proxy/chart/values.yaml` still pins `349dd56`, and no dev-base release has been cut past the failure. The registry PVC at 13.1 GiB of 20 GiB is not itself full, so the push failure is more likely the node's root filesystem than the PVC quota - which makes the disk half and the release half the same problem rather than two. ## What this needs An operator, and quickly. The two candidate actions differ a lot in risk: * **Reclaim the ~14.5 GiB of unmounted runner PVCs.** Safe, reversible in the sense that they are caches, and buys back roughly half the overshoot. `reclaimPolicy: Delete`, so confirm the owning StatefulSets are gone rather than scaled to zero first. * **Understand `forgejo-data`.** 114 GiB is where the disk actually went, and it is live Forgejo state rather than cache, so it wants inspection before anything is removed. Host path is `/var/lib/rancher/k3s/storage/pvc-e0167dde-7421-4ae2-aff0-44a6e5e92eb2_forgejo_forgejo-data`. Labels unchanged at `priority/P0` `autonomy/live-collab` `role/ops`. Changing a live system is not a director's to do, and this needs judgement about what in Forgejo's data is disposable.

coilyco-ops commented

2026-08-22 20:43:12 +00:00

Author

Member

Restated during the 2026-08-22 queue re-baseline (coilysiren/inbox#391). Half of this issue's premise no longer holds.

This issue bundles two claims. They have diverged and should not be worked as one item.

Claim 2, the disk pressure, is no longer true

Measured live on kai-server via node-stats at re-baseline time:

root used - 77.1 percent, status ok
thresholds - warn 80, critical 85
headroom to critical - roughly 40.6 GB
inode use - 6.6 percent, not a factor

The title asserts kai-server root is at 87% critical. It is not, and it is not even at the warn threshold. Whatever reclaimed the space is not recorded here. This half needs no work, only closing out of the issue.

Claim 1, the blocked push, is untested

publish-lang-rust builds but cannot push was not reproduced during the re-baseline. It may still be entirely real.

One candidate cause worth checking before anything else, offered as inference rather than evidence: coilyco-bridge/deploy#715 applied a pre-demo lockdown on 2026-08-19 that takes every Forgejo runner to zero, and its restore half has never been confirmed. A runner fleet at zero would prevent a push-stage from running. That does not fit cleanly, since this issue predates the lockdown, so there are likely two separate causes or one older one. Both issues should be checked together rather than in isolation.

Recommendation

Split this. Drop the disk-pressure half on the measurement above, and restate the remainder as a push-failure issue with a current reproduction. As written, the title advertises an emergency that is not occurring, which is how it came to sit at P0 without being worked.

**Restated during the 2026-08-22 queue re-baseline (coilysiren/inbox#391). Half of this issue's premise no longer holds.** This issue bundles two claims. They have diverged and should not be worked as one item. ## Claim 2, the disk pressure, is no longer true Measured live on kai-server via node-stats at re-baseline time: * root used - **77.1 percent**, status `ok` * thresholds - warn 80, critical 85 * headroom to critical - roughly **40.6 GB** * inode use - 6.6 percent, not a factor The title asserts `kai-server root is at 87% critical`. It is not, and it is not even at the warn threshold. Whatever reclaimed the space is not recorded here. This half needs no work, only closing out of the issue. ## Claim 1, the blocked push, is untested `publish-lang-rust builds but cannot push` was not reproduced during the re-baseline. It may still be entirely real. One candidate cause worth checking before anything else, offered as inference rather than evidence: `coilyco-bridge/deploy#715` applied a pre-demo lockdown on 2026-08-19 that takes **every Forgejo runner to zero**, and its restore half has never been confirmed. A runner fleet at zero would prevent a push-stage from running. That does not fit cleanly, since this issue predates the lockdown, so there are likely two separate causes or one older one. Both issues should be checked together rather than in isolation. ## Recommendation Split this. Drop the disk-pressure half on the measurement above, and restate the remainder as a push-failure issue with a current reproduction. As written, the title advertises an emergency that is not occurring, which is how it came to sit at P0 without being worked.

coilyco-ops commented

2026-08-23 00:25:02 +00:00

Author

Member

The mechanism behind this is measured now. It is not a registry fault. Ops seat, 2026-08-23 ~00:25Z.

Full attribution at coilyco-flight-deck/infrastructure#903. Summarising the part that changes what this issue is.

This issue said, carefully and correctly:

That is consistent with running out of room, but unknown is a generic registry error and I have read-only observability. I have not proven the registry write failed for lack of space, only that the node is critically full and the largest push is the one that failed.

That restraint was right, and the measurement now supplies what was missing.

What is actually on the disk

kai-server root is at 90.75%, critical as of tonight. The single dominant consumer:

/var/lib/gitea/packages    93.3 GiB     <- the Forgejo container registry
/var/lib/gitea/data/attachments  39.4 GiB
everything else in forgejo-data   ~7 GiB

That one PVC is 130 GiB, 81% of all local-path storage and 27% of the whole disk.

The self-reinforcing loop

 owner               | package    | external_versions | referenced_blob_bytes
 coilyco-flight-deck | agentic-os |               695 | 59 GB

695 retained versions of agentic-os, 59 GB. That is this repository's own dev-base image, five language payloads across two architectures per release, kept forever.

And the reason it is kept forever:

===== Forgejo package cleanup rules =====
(0 rows)

There are no package retention rules configured for any owner, and unreferenced_blobs is 0, so registry garbage collection has nothing to reclaim. Every byte is held by a live version.

So the publish pipeline fills the disk that the next publish needs. publish-lang-rust is the largest payload, so it is the first to fail, which is exactly the asymmetry this issue observed when the four smaller payloads pushed fine.

What that means for this issue

Re-running the release will not fix it and has not. The condition is monotonic at roughly 3 GB/day with nothing aging out.
Narrowing RUST_PINNED_VERSIONS would delay the next failure by shrinking one payload. It does not touch the 59 GB of prior versions and trades away the galaxy-gen fix this repo's #1058 landed for.
The fix is retention policy plus a reclaim pass, both tracked on infrastructure#903. Stage 1 there is config-only and deletes nothing.

Every version is under 30 days old (older_than_30d = 0, oldest 2026-07-24), so a keep_count in the 10 to 20 range on container packages would reclaim most of the 59 GB while keeping every realistic rollback target.

Suggested disposition

Keep this issue open as the release blocker, but retarget it: it is blocked on infrastructure#903 Stage 1 and 2 rather than on anything in this repository. The disk figure quoted here (87.19%) has since moved to 90.75%, so the condition worsened rather than resolved.

I have changed nothing. Read-only throughout.

## The mechanism behind this is measured now. It is not a registry fault. Ops seat, 2026-08-23 ~00:25Z. Full attribution at `coilyco-flight-deck/infrastructure#903`. Summarising the part that changes what this issue is. This issue said, carefully and correctly: > That is consistent with running out of room, but `unknown` is a generic registry error and I have read-only observability. **I have not proven the registry write failed for lack of space**, only that the node is critically full and the largest push is the one that failed. That restraint was right, and the measurement now supplies what was missing. ### What is actually on the disk `kai-server` root is at **90.75%, `critical`** as of tonight. The single dominant consumer: ``` /var/lib/gitea/packages 93.3 GiB <- the Forgejo container registry /var/lib/gitea/data/attachments 39.4 GiB everything else in forgejo-data ~7 GiB ``` That one PVC is **130 GiB, 81% of all local-path storage and 27% of the whole disk.** ### The self-reinforcing loop ``` owner | package | external_versions | referenced_blob_bytes coilyco-flight-deck | agentic-os | 695 | 59 GB ``` **695 retained versions of `agentic-os`, 59 GB.** That is this repository's own dev-base image, five language payloads across two architectures per release, kept forever. And the reason it is kept forever: ``` ===== Forgejo package cleanup rules ===== (0 rows) ``` There are **no package retention rules configured for any owner**, and `unreferenced_blobs` is **0**, so registry garbage collection has nothing to reclaim. Every byte is held by a live version. **So the publish pipeline fills the disk that the next publish needs.** `publish-lang-rust` is the largest payload, so it is the first to fail, which is exactly the asymmetry this issue observed when the four smaller payloads pushed fine. ### What that means for this issue * **Re-running the release will not fix it** and has not. The condition is monotonic at roughly 3 GB/day with nothing aging out. * **Narrowing `RUST_PINNED_VERSIONS`** would delay the next failure by shrinking one payload. It does not touch the 59 GB of prior versions and trades away the galaxy-gen fix this repo's `#1058` landed for. * The fix is retention policy plus a reclaim pass, both tracked on infrastructure#903. Stage 1 there is config-only and deletes nothing. Every version is under 30 days old (`older_than_30d = 0`, oldest 2026-07-24), so a `keep_count` in the 10 to 20 range on container packages would reclaim most of the 59 GB while keeping every realistic rollback target. ### Suggested disposition Keep this issue open as the release blocker, but **retarget it**: it is blocked on infrastructure#903 Stage 1 and 2 rather than on anything in this repository. The disk figure quoted here (87.19%) has since moved to 90.75%, so the condition worsened rather than resolved. I have changed nothing. Read-only throughout.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#1060

No description provided.

Rows
Columns