Incident: kai-server root filesystem is at 94.1% pressure

coilyco-ops commented

2026-07-23 06:35:47 +00:00

Member

Live evidence

A read-only node-stats MCP call on 2026-07-23 reported the kai-server root filesystem at critical pressure:

total: 514,909,585,408 bytes
available: 30,327,468,032 bytes
pressure used: 484,582,117,376 bytes
used: 94.1101%
configured critical threshold: 85%
bytes over critical: 46,908,969,780
inode use: 11.26%

This is byte pressure, not inode exhaustion. It strengthens the storage/I/O hypothesis in #592 but is acute enough to track separately.

Required outcome

Use live, read-only inspection to identify the largest safe-to-classify consumers and correlate them with containerd, k3s, Forgejo Actions, observability retention, logs, and game-server data.
Produce a ranked pressure-relief plan with ownership and recoverability for each candidate.
Do not delete images, volumes, logs, databases, or game data blindly.
Apply only cleanup already covered by an established safe operator path; capture any new cleanup automation as a separate code issue.
Verify node-stats reports below the critical threshold after attended remediation.
Add or verify alerting before the node reaches this state again.

Related: #592 and #560.

## Live evidence A read-only node-stats MCP call on 2026-07-23 reported the kai-server root filesystem at critical pressure: - total: 514,909,585,408 bytes - available: 30,327,468,032 bytes - pressure used: 484,582,117,376 bytes - used: 94.1101% - configured critical threshold: 85% - bytes over critical: 46,908,969,780 - inode use: 11.26% This is byte pressure, not inode exhaustion. It strengthens the storage/I/O hypothesis in #592 but is acute enough to track separately. ## Required outcome - Use live, read-only inspection to identify the largest safe-to-classify consumers and correlate them with containerd, k3s, Forgejo Actions, observability retention, logs, and game-server data. - Produce a ranked pressure-relief plan with ownership and recoverability for each candidate. - Do not delete images, volumes, logs, databases, or game data blindly. - Apply only cleanup already covered by an established safe operator path; capture any new cleanup automation as a separate code issue. - Verify node-stats reports below the critical threshold after attended remediation. - Add or verify alerting before the node reaches this state again. Related: #592 and #560.

coilyco-ops referenced this issue from a commit

2026-07-23 06:50:08 +00:00

feat(o11y): add kai disk pressure watcher

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-23 06:50:20 +00:00

feat(o11y): add kai disk pressure watcher #597

coilyco-ops commented

2026-07-23 06:56:44 +00:00

Author

Member

WARDED_WORKFLOW: #597

details

Review summary: skipped. The in-container review gate was intentionally skipped because the temporary Ward default is pending brokered QA.

workflow: pull-request; review summary: skipped pending brokered QA

Retrospective: focused and bounded. The work stayed on read-only diagnosis and alerting, with the attended-remediation checkpoint explicit.

Confidence: high for the checked configuration and offline alert/recovery paths.

Surprises: the existing rootfs metric used 90%, so this change aligned it to the incident's 85% critical threshold.

Follow-up: an operator must install/converge the role, perform any approved remediation, and verify live node-stats below 85%.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/597 <details><summary>details</summary> Review summary: skipped. The in-container review gate was intentionally skipped because the temporary Ward default is pending brokered QA. workflow: pull-request; review summary: skipped pending brokered QA Retrospective: focused and bounded. The work stayed on read-only diagnosis and alerting, with the attended-remediation checkpoint explicit. Confidence: high for the checked configuration and offline alert/recovery paths. Surprises: the existing rootfs metric used 90%, so this change aligned it to the incident's 85% critical threshold. Follow-up: an operator must install/converge the role, perform any approved remediation, and verify live node-stats below 85%. </details>

coilyco-ops commented

2026-07-23 06:56:57 +00:00

Author

Member

WARDED_WORKFLOW: reservation-released

release details

Run finished with WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/597.

ward container reap released container engineer-codex-infrastructure-596 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

Outcome summary: #597

— Codex, via ward agent

WARDED_WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/597`. `ward container reap` released container `engineer-codex-infrastructure-596` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. **Outcome summary:** https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/597 </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-23 07:25:30 +00:00

Author

Member

Recurrence: Forgejo/private LFS still blocking the agent lane

Live recovery evidence from 2026-07-23 UTC:

The earlier bootstrap failures clustered at 05:52-06:02. A second event hit at 07:05.
Forgejo returned HTTP 503 while refreshing the eco-ops mirror during the second event.
The Forgejo API is responding again.
A direct anonymous pull of public eco-mods LFS object a9127c7... now succeeds.
A fresh brokered retry of private eco-ops#19 at 07:24 still failed before the agent launched because LFS object d2baa6a... is reported as missing.
These are bootstrap deaths, not OOM kills. No engineer reached the harness and there is no in-container edit state to rescue.

Affected unfinished work from the two events:

coilyco-gaming/eco-ops#19
coilyco-gaming/eco-ops#54
coilyco-gaming/eco-ops#56
coilyco-gaming/eco-ops#57
coilyco-gaming/eco-mods#2
coilyco-gaming/eco-mods#7
coilyco-gaming/factory-game-v3#27

Immediate recovery outcome required:

Restore reliable private-repository LFS reads and determine whether the missing-object response is storage pressure, mount availability, authorization, or actual object loss.
Reduce rootfs below the 85% critical threshold through an established safe operator path only. Do not blindly delete images, volumes, logs, databases, LFS data, or game state.
Repair and converge the failed forgejo-runner-recycle CronJob image/command pairing, and verify it clears the intended runner scratch without touching durable repository data.
Verify Forgejo API, private git clone, and the exact private LFS object read before re-queuing the affected suite.
Record live before/after disk and workload evidence on this issue.

PR #597 adds alerting but does not perform the attended remediation or restore the blocked LFS path.

## Recurrence: Forgejo/private LFS still blocking the agent lane Live recovery evidence from 2026-07-23 UTC: - The earlier bootstrap failures clustered at 05:52-06:02. A second event hit at 07:05. - Forgejo returned HTTP 503 while refreshing the `eco-ops` mirror during the second event. - The Forgejo API is responding again. - A direct anonymous pull of public `eco-mods` LFS object `a9127c7...` now succeeds. - A fresh brokered retry of private `eco-ops#19` at 07:24 still failed before the agent launched because LFS object `d2baa6a...` is reported as missing. - These are bootstrap deaths, not OOM kills. No engineer reached the harness and there is no in-container edit state to rescue. Affected unfinished work from the two events: - `coilyco-gaming/eco-ops#19` - `coilyco-gaming/eco-ops#54` - `coilyco-gaming/eco-ops#56` - `coilyco-gaming/eco-ops#57` - `coilyco-gaming/eco-mods#2` - `coilyco-gaming/eco-mods#7` - `coilyco-gaming/factory-game-v3#27` Immediate recovery outcome required: 1. Restore reliable private-repository LFS reads and determine whether the missing-object response is storage pressure, mount availability, authorization, or actual object loss. 2. Reduce rootfs below the 85% critical threshold through an established safe operator path only. Do not blindly delete images, volumes, logs, databases, LFS data, or game state. 3. Repair and converge the failed `forgejo-runner-recycle` CronJob image/command pairing, and verify it clears the intended runner scratch without touching durable repository data. 4. Verify Forgejo API, private git clone, and the exact private LFS object read before re-queuing the affected suite. 5. Record live before/after disk and workload evidence on this issue. PR #597 adds alerting but does not perform the attended remediation or restore the blocked LFS path.

coilyco-ops commented

2026-07-23 07:37:08 +00:00

Author

Member

WARDED_WORKFLOW: #597

details

Review summary: skipped. The in-container review gate was intentionally skipped because the temporary Ward default is pending brokered QA.

workflow: pull-request; review summary: skipped pending brokered QA

Retrospective: focused and bounded. The shell-free native kubectl arguments fix the recycle regression without extending cleanup authority.

Confidence: high for the checked configuration and green CI.

Surprises: the recurrence exposed that the selected kubectl image is distroless and cannot run the old shell wrapper.

Follow-up: an attended operator must converge the watcher and CronJob, perform owner-approved remediation, then verify private LFS reads and node-stats below 85%.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/597 <details><summary>details</summary> Review summary: skipped. The in-container review gate was intentionally skipped because the temporary Ward default is pending brokered QA. workflow: pull-request; review summary: skipped pending brokered QA Retrospective: focused and bounded. The shell-free native kubectl arguments fix the recycle regression without extending cleanup authority. Confidence: high for the checked configuration and green CI. Surprises: the recurrence exposed that the selected kubectl image is distroless and cannot run the old shell wrapper. Follow-up: an attended operator must converge the watcher and CronJob, perform owner-approved remediation, then verify private LFS reads and node-stats below 85%. </details>

coilyco-ops commented

2026-07-23 07:37:15 +00:00

Author

Member

WARDED_WORKFLOW: reservation-released

release details

Run finished with WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/597.

ward container reap released container engineer-codex-infrastructure-596 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

Outcome summary: #597

— Codex, via ward agent

WARDED_WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/597`. `ward container reap` released container `engineer-codex-infrastructure-596` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. **Outcome summary:** https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/597 </details>  — Codex, via `ward agent`