coilyco-flight-deck/infrastructure

Watch

Add durable guardrails for kai-server disk pressure from xrdp logs and Forgejo PVC growth #488

New issue

Closed

opened 2026-07-09 16:54:00 +00:00 by coilyco-ops · 5 comments

coilyco-ops commented

2026-07-09 16:54:00 +00:00

Member

Finding

A live read-only disk check on 2026-07-09 found kai-server root at 91% used: 480G total, about 412G used, about 44G available.

Visible contributors:

/var/log is 16G, dominated by /var/log/xrdp.log at 12G. Journald is 4.0G.
/home/kai is 58G, with .local 13G, Steam 12G, .cache 9.2G, .net 7.1G, .npm 4.2G, projects 3.0G.
Kubelet/PVC in-pod du shows forgejo/Deployment/forgejo using 68.4GiB in the forgejo-data PVC even though the PVC request is 20Gi. Registry is next at 3.4GiB.

Because this was a director read-only surface, no live cleanup was performed.

Do

Add or fix log rotation for xrdp so /var/log/xrdp.log cannot grow unbounded again.
Decide whether journald’s 4G retention is intentional or should be capped lower.
Investigate Forgejo data growth and identify the high-growth subtrees inside the PVC using an operator-safe method.
Add a durable disk-pressure check or runbook entry so root filesystem pressure is caught before the node reaches the danger zone.
If live deletion/truncation is needed, leave the exact operator command and risk notes for Kai rather than performing destructive cleanup autonomously.

Acceptance

The repo records the durable fix or runbook for xrdp/journal disk pressure.
The Forgejo PVC growth has a concrete next action or documented explanation.
No destructive live cleanup happens without Kai explicitly authorizing it.

## Finding A live read-only disk check on 2026-07-09 found kai-server root at 91% used: 480G total, about 412G used, about 44G available. Visible contributors: * `/var/log` is 16G, dominated by `/var/log/xrdp.log` at 12G. Journald is 4.0G. * `/home/kai` is 58G, with `.local` 13G, `Steam` 12G, `.cache` 9.2G, `.net` 7.1G, `.npm` 4.2G, `projects` 3.0G. * Kubelet/PVC in-pod du shows `forgejo/Deployment/forgejo` using 68.4GiB in the `forgejo-data` PVC even though the PVC request is 20Gi. Registry is next at 3.4GiB. Because this was a director read-only surface, no live cleanup was performed. ## Do * Add or fix log rotation for xrdp so `/var/log/xrdp.log` cannot grow unbounded again. * Decide whether journald’s 4G retention is intentional or should be capped lower. * Investigate Forgejo data growth and identify the high-growth subtrees inside the PVC using an operator-safe method. * Add a durable disk-pressure check or runbook entry so root filesystem pressure is caught before the node reaches the danger zone. * If live deletion/truncation is needed, leave the exact operator command and risk notes for Kai rather than performing destructive cleanup autonomously. ## Acceptance * The repo records the durable fix or runbook for xrdp/journal disk pressure. * The Forgejo PVC growth has a concrete next action or documented explanation. * No destructive live cleanup happens without Kai explicitly authorizing it.

coilyco-ops commented

2026-07-09 16:56:51 +00:00

Author

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Recommendation

Treat this as one infrastructure issue with three deliverables in this repo:

Host-side log guardrails on kai-server for xrdp and journald.
A read-only disk-pressure watcher that alerts before / gets into the 90% range.
A Forgejo PVC sizing + subtree audit runbook, with no ad-hoc deletion from the PVC.

The repo already points to the right implementation surfaces for all three:

Host-mutating config belongs in Ansible host roles on kai-server, not in app repos or one-off shell fixes.
Read-only alerting already follows the ops-plane watcher pattern on ser8 (git-state-watch, flux-readiness-watch) with Telegram via o11y-telegram.
Forgejo storage is declared here in deploy/forgejo.yml, and the repo already knows how to resolve PVC host paths safely in scripts/restic-backup.sh.

Key findings

The reported forgejo-data PVC request is 20Gi, but the live data is about 68.4GiB. On k3s local-path, that request is not a hard quota. The pod can keep consuming node disk until / fills. So the mismatch is both a documentation problem and an operational risk.
xrdp.log at 12G is the clearest single bad actor. That is not a sizing problem, it is a missing or ineffective rotation boundary.
journald at 4G is not the primary incident driver, but on a single 480G root disk that also backs local-path PVCs, desktop logs, and user home data, it is too large to leave implicit.
Forgejo likely stores growth under the usual app-data subtrees. Official Forgejo docs for v15 show local storage under the app data path for subsystems such as attachments, LFS, packages, repo archives, and the CLI/storage docs also list actions-log and actions-artifacts as separate storage types. Sources: Forgejo storage docs and CLI docs. https://forgejo.org/docs/v15.0/admin/setup/storage/ https://forgejo.org/docs/v15.0/admin/command-line/ https://forgejo.org/docs/latest/user/packages/
This repo enables Forgejo Actions in the manifest, so action artifacts/logs are plausible contributors. The package registry is also a plausible contributor if packages are being published without cleanup rules.

Decisions to make

1. `xrdp` guardrail

Recommendation: manage an explicit logrotate policy in this repo for kai-server, even if the distro package ships one.

Why:

The current state proves the package default is either absent, bypassed, or ineffective.
A repo-owned policy is auditable and survives package drift.
This belongs in the same host-converge layer as the existing docker and node-tuning kai-server roles.

Suggested policy shape:

Rotate by size and time.
Compress old logs.
Keep a bounded count.
Use the reopen/truncate mechanism that matches the installed xrdp service.

Open question:

Is xrdp writing only /var/log/xrdp.log, or are sesman / xorgxrdp logs also bypassing rotation?

Risk note:

Rotation fixes the disk-pressure symptom, but not the underlying spam source. A short pass over recent xrdp errors is still worth doing so the host is not quietly logging a persistent auth/session failure forever.

2. `journald` retention

Recommendation: cap it explicitly. I would set SystemMaxUse=1G and also keep a free-space floor with SystemKeepFree.

Why:

4G is not outrageous in isolation, but this node is a single-disk convergence point for desktop usage, host logs, and local-path PVCs.
journald should not be competing with Forgejo data and kubelet-backed storage for the last few GiB of root.
A 1G cap is enough to keep useful recent history without letting the journal become a silent multi-GiB reservoir.

Alternative:

2G if there is a strong debugging reason to keep more history.

I would not keep the current implicit 4G unless there is a known incident-response need for that much local history.

3. Forgejo PVC growth

Recommendation: do a read-only subtree audit first, then make two decisions separately:

What is consuming the space?
What should the manifest request say?

Those are related, but not the same:

The subtree audit decides whether the problem is expected growth (git/repositories, data/lfs) or avoidable accumulation (data/packages, data/actions-artifacts, stale archives, attachments).
After that, the PVC request in deploy/forgejo.yml should be raised to at least reflect reality, but that is bookkeeping, not containment. On local-path, increasing the request does not protect / from filling.

Most likely high-growth candidates to measure first:

/var/lib/gitea/git/repositories
/var/lib/gitea/data/lfs
/var/lib/gitea/data/packages
/var/lib/gitea/data/actions_artifacts or equivalent actions storage path
/var/lib/gitea/data/actions_log or equivalent actions log path
/var/lib/gitea/data/attachments
/var/lib/gitea/data/repo-archive

Open questions:

Are Forgejo Actions artifact retention defaults acceptable for this instance?
Is the package registry in active use, and if so, do cleanup rules exist for the relevant owners?
Is the dominant growth real repository content, or generated by CI/package churn?

4. Disk-pressure detection path

Recommendation: add a new ops-plane watcher on ser8, modeled on flux-readiness-watch, instead of trying to build a first-ever metrics alert pipeline right now.

Why:

The repo already has the pattern: read-only polling over SSH, Telegram via o11y-telegram, dedup/re-nag/recovery behavior, and no new infra dependencies.
node-exporter and vmagent are present, but this repo does not appear to have an existing VictoriaMetrics-to-Telegram alert path. Building one would widen scope beyond the issue.
The watcher can alert on / usage thresholds and optionally include the top offenders when above threshold.

Suggested behavior:

Warn at 80%.
Alert high at 85%.
Re-nag hourly while still above threshold.
Recovery message when back below threshold.
Include a compact offender digest for /var/log, /home/kai, and the Forgejo PVC host path when available.

Operator-safe inspection commands

These are read-only and suitable for the issue/runbook.

Root filesystem and obvious host offenders

ssh kai-server -- 'df -h /; sudo du -xhd1 /var/log | sort -h; sudo du -xhd1 /home/kai | sort -h'

Resolve the live Forgejo PVC host path, then inspect it on-host

This repo already uses the same hostPath lookup pattern in scripts/restic-backup.sh.

PV=$(ssh kai-server -- "k3s kubectl get pvc -n forgejo forgejo-data -o jsonpath='{.spec.volumeName}'")
PVC_PATH=$(ssh kai-server -- "k3s kubectl get pv ${PV} -o jsonpath='{.spec.hostPath.path}'")
ssh kai-server -- "sudo du -xhd1 ${PVC_PATH} | sort -h"
ssh kai-server -- "sudo du -xhd2 ${PVC_PATH}/data 2>/dev/null | sort -h | tail -50"
ssh kai-server -- "sudo du -xhd3 ${PVC_PATH}/git/repositories 2>/dev/null | sort -h | tail -50"

In-pod read-only view, if the on-host path is less convenient

ssh kai-server -- "k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc 'du -x -d1 /var/lib/gitea 2>/dev/null | sort -h; du -x -d2 /var/lib/gitea/data 2>/dev/null | sort -h'"

If emergency manual cleanup is needed before the durable fix lands

The issue asked for commands and risk notes, not for autonomous cleanup.

`xrdp.log`

Preferred if a logrotate config already exists:

ssh kai-server -- 'sudo logrotate -f /etc/logrotate.d/xrdp'

Fallback if the file must be reclaimed immediately:

ssh kai-server -- 'sudo truncate -s 0 /var/log/xrdp.log'

Risk:

Loses historical xrdp logs.
Does not fix the spam source.

`journald`

ssh kai-server -- 'sudo journalctl --vacuum-size=1G'

Risk:

Deletes older journal history.
Safe for service integrity, but loses forensic depth.

Forgejo PVC

Do not manually rm files from the Forgejo PVC as the first response.

Risk:

Forgejo keeps database metadata for many stored objects.
Ad-hoc deletion can orphan metadata or break downloads/UI references.
Cleanup should happen either through Forgejo-managed retention/cleanup paths or after a subtree audit that identifies a specifically safe target.

Advisory close

My recommendation on the issue would be:

Do not split this into child issues yet. The work is still one-repo infrastructure work.
Land a repo-owned host fix for xrdp rotation and a repo-owned cap for journald.
Add a new read-only disk-pressure watcher on ser8, using the existing watcher + Telegram pattern rather than inventing a new alert stack.
Audit Forgejo storage read-only first, then update the manifest request to match reality and decide whether the real follow-up is retention/cleanup or simply honest sizing.
Avoid destructive Forgejo PVC cleanup until the subtree audit says exactly what is safe to remove.

That gives a durable answer to the immediate disk-pressure finding without pretending that raising the Forgejo PVC request alone would solve node-disk exhaustion.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- ## Recommendation Treat this as one infrastructure issue with three deliverables in this repo: 1. **Host-side log guardrails on `kai-server`** for `xrdp` and `journald`. 2. **A read-only disk-pressure watcher** that alerts before `/` gets into the 90% range. 3. **A Forgejo PVC sizing + subtree audit runbook**, with no ad-hoc deletion from the PVC. The repo already points to the right implementation surfaces for all three: - Host-mutating config belongs in **Ansible host roles** on `kai-server`, not in app repos or one-off shell fixes. - Read-only alerting already follows the **ops-plane watcher** pattern on `ser8` (`git-state-watch`, `flux-readiness-watch`) with Telegram via `o11y-telegram`. - Forgejo storage is declared here in [`deploy/forgejo.yml`](deploy/forgejo.yml), and the repo already knows how to resolve PVC host paths safely in [`scripts/restic-backup.sh`](scripts/restic-backup.sh). ## Key findings - The reported `forgejo-data` PVC request is **20Gi**, but the live data is **about 68.4GiB**. On `k3s` `local-path`, that request is **not a hard quota**. The pod can keep consuming node disk until `/` fills. So the mismatch is both a documentation problem and an operational risk. - `xrdp.log` at **12G** is the clearest single bad actor. That is not a sizing problem, it is a missing or ineffective rotation boundary. - `journald` at **4G** is not the primary incident driver, but on a single 480G root disk that also backs local-path PVCs, desktop logs, and user home data, it is too large to leave implicit. - Forgejo likely stores growth under the usual app-data subtrees. Official Forgejo docs for v15 show local storage under the app data path for subsystems such as **attachments, LFS, packages, repo archives**, and the CLI/storage docs also list **actions-log** and **actions-artifacts** as separate storage types. Sources: Forgejo storage docs and CLI docs. <https://forgejo.org/docs/v15.0/admin/setup/storage/> <https://forgejo.org/docs/v15.0/admin/command-line/> <https://forgejo.org/docs/latest/user/packages/> - This repo enables **Forgejo Actions** in the manifest, so action artifacts/logs are plausible contributors. The package registry is also a plausible contributor if packages are being published without cleanup rules. ## Decisions to make ### 1. `xrdp` guardrail **Recommendation:** manage an explicit `logrotate` policy in this repo for `kai-server`, even if the distro package ships one. Why: - The current state proves the package default is either absent, bypassed, or ineffective. - A repo-owned policy is auditable and survives package drift. - This belongs in the same host-converge layer as the existing `docker` and `node-tuning` kai-server roles. Suggested policy shape: - Rotate by **size and time**. - Compress old logs. - Keep a bounded count. - Use the reopen/truncate mechanism that matches the installed `xrdp` service. Open question: - Is `xrdp` writing only `/var/log/xrdp.log`, or are `sesman` / `xorgxrdp` logs also bypassing rotation? Risk note: - Rotation fixes the disk-pressure symptom, but not the underlying spam source. A short pass over recent `xrdp` errors is still worth doing so the host is not quietly logging a persistent auth/session failure forever. ### 2. `journald` retention **Recommendation:** cap it explicitly. I would set **`SystemMaxUse=1G`** and also keep a free-space floor with **`SystemKeepFree`**. Why: - 4G is not outrageous in isolation, but this node is a single-disk convergence point for desktop usage, host logs, and `local-path` PVCs. - `journald` should not be competing with Forgejo data and kubelet-backed storage for the last few GiB of root. - A 1G cap is enough to keep useful recent history without letting the journal become a silent multi-GiB reservoir. Alternative: - `2G` if there is a strong debugging reason to keep more history. I would **not** keep the current implicit 4G unless there is a known incident-response need for that much local history. ### 3. Forgejo PVC growth **Recommendation:** do a **read-only subtree audit first**, then make two decisions separately: 1. **What is consuming the space?** 2. **What should the manifest request say?** Those are related, but not the same: - The subtree audit decides whether the problem is expected growth (`git/repositories`, `data/lfs`) or avoidable accumulation (`data/packages`, `data/actions-artifacts`, stale archives, attachments). - After that, the PVC request in `deploy/forgejo.yml` should be raised to at least reflect reality, but that is **bookkeeping, not containment**. On `local-path`, increasing the request does not protect `/` from filling. Most likely high-growth candidates to measure first: - `/var/lib/gitea/git/repositories` - `/var/lib/gitea/data/lfs` - `/var/lib/gitea/data/packages` - `/var/lib/gitea/data/actions_artifacts` or equivalent actions storage path - `/var/lib/gitea/data/actions_log` or equivalent actions log path - `/var/lib/gitea/data/attachments` - `/var/lib/gitea/data/repo-archive` Open questions: - Are Forgejo Actions artifact retention defaults acceptable for this instance? - Is the package registry in active use, and if so, do cleanup rules exist for the relevant owners? - Is the dominant growth real repository content, or generated by CI/package churn? ### 4. Disk-pressure detection path **Recommendation:** add a **new ops-plane watcher on `ser8`**, modeled on `flux-readiness-watch`, instead of trying to build a first-ever metrics alert pipeline right now. Why: - The repo already has the pattern: read-only polling over SSH, Telegram via `o11y-telegram`, dedup/re-nag/recovery behavior, and no new infra dependencies. - `node-exporter` and `vmagent` are present, but this repo does **not** appear to have an existing VictoriaMetrics-to-Telegram alert path. Building one would widen scope beyond the issue. - The watcher can alert on **`/` usage thresholds** and optionally include the top offenders when above threshold. Suggested behavior: - Warn at **80%**. - Alert high at **85%**. - Re-nag hourly while still above threshold. - Recovery message when back below threshold. - Include a compact offender digest for `/var/log`, `/home/kai`, and the Forgejo PVC host path when available. ## Operator-safe inspection commands These are read-only and suitable for the issue/runbook. ### Root filesystem and obvious host offenders ```bash ssh kai-server -- 'df -h /; sudo du -xhd1 /var/log | sort -h; sudo du -xhd1 /home/kai | sort -h' ``` ### Resolve the live Forgejo PVC host path, then inspect it on-host This repo already uses the same `hostPath` lookup pattern in `scripts/restic-backup.sh`. ```bash PV=$(ssh kai-server -- "k3s kubectl get pvc -n forgejo forgejo-data -o jsonpath='{.spec.volumeName}'") PVC_PATH=$(ssh kai-server -- "k3s kubectl get pv ${PV} -o jsonpath='{.spec.hostPath.path}'") ssh kai-server -- "sudo du -xhd1 ${PVC_PATH} | sort -h" ssh kai-server -- "sudo du -xhd2 ${PVC_PATH}/data 2>/dev/null | sort -h | tail -50" ssh kai-server -- "sudo du -xhd3 ${PVC_PATH}/git/repositories 2>/dev/null | sort -h | tail -50" ``` ### In-pod read-only view, if the on-host path is less convenient ```bash ssh kai-server -- "k3s kubectl -n forgejo exec deploy/forgejo -- sh -lc 'du -x -d1 /var/lib/gitea 2>/dev/null | sort -h; du -x -d2 /var/lib/gitea/data 2>/dev/null | sort -h'" ``` ## If emergency manual cleanup is needed before the durable fix lands The issue asked for commands and risk notes, not for autonomous cleanup. ### `xrdp.log` Preferred if a logrotate config already exists: ```bash ssh kai-server -- 'sudo logrotate -f /etc/logrotate.d/xrdp' ``` Fallback if the file must be reclaimed immediately: ```bash ssh kai-server -- 'sudo truncate -s 0 /var/log/xrdp.log' ``` Risk: - Loses historical `xrdp` logs. - Does **not** fix the spam source. ### `journald` ```bash ssh kai-server -- 'sudo journalctl --vacuum-size=1G' ``` Risk: - Deletes older journal history. - Safe for service integrity, but loses forensic depth. ### Forgejo PVC **Do not manually `rm` files from the Forgejo PVC as the first response.** Risk: - Forgejo keeps database metadata for many stored objects. - Ad-hoc deletion can orphan metadata or break downloads/UI references. - Cleanup should happen either through Forgejo-managed retention/cleanup paths or after a subtree audit that identifies a specifically safe target. ## Advisory close My recommendation on the issue would be: - **Do not split this into child issues yet.** The work is still one-repo infrastructure work. - Land a repo-owned host fix for `xrdp` rotation and a repo-owned cap for `journald`. - Add a new read-only disk-pressure watcher on `ser8`, using the existing watcher + Telegram pattern rather than inventing a new alert stack. - Audit Forgejo storage read-only first, then update the manifest request to match reality and decide whether the real follow-up is retention/cleanup or simply honest sizing. - Avoid destructive Forgejo PVC cleanup until the subtree audit says exactly what is safe to remove. That gives a durable answer to the immediate disk-pressure finding without pretending that raising the Forgejo PVC request alone would solve node-disk exhaustion. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`

coilysiren commented

2026-07-09 17:09:27 +00:00

Owner

Forgejo is the most important thing we run, lets start with sizing it up

The full disk is... 400GB? Thats small, I need to buy more.

In the meantime, forgejo has a bunch of content types that are more durable than I would like. the biggest offender is branches.

Forgejo is the most important thing we run, lets start with sizing it up The full disk is... 400GB? Thats small, I need to buy more. In the meantime, forgejo has a bunch of content types that are more durable than I would like. the biggest offender is branches.

coilyco-ops commented

2026-07-09 17:40:04 +00:00

Author

Member

Director handoff for the next run:

Kai's latest direction narrows the first slice: Forgejo is the critical service, so start by sizing it honestly and understanding branch-heavy growth. Treat xrdp/journald guardrails as still valid follow-ups, but do not let them distract from the Forgejo sizing pass.

Requested first slice:

inspect deploy/forgejo.yml, backup scripts, and any existing Forgejo retention/config docs in infrastructure
add a durable, operator-safe Forgejo storage audit runbook or script if the repo does not already have one
update the Forgejo PVC request/config to reflect current live usage plus reasonable near-term headroom, if the manifest owns that value
document that local-path PVC requests are not hard quotas, so this is honest scheduling/bookkeeping, not containment
identify branch/content retention knobs or cleanup candidates, but do not delete live Forgejo data
leave exact operator commands and risk notes for any live measurement or cleanup Kai must run

No destructive cleanup. No ad-hoc PVC rm. If a safe branch/artifact/package retention task emerges, file a separate follow-up with the exact Forgejo-managed mechanism.

Director handoff for the next run: Kai's latest direction narrows the first slice: Forgejo is the critical service, so start by sizing it honestly and understanding branch-heavy growth. Treat xrdp/journald guardrails as still valid follow-ups, but do not let them distract from the Forgejo sizing pass. Requested first slice: * inspect `deploy/forgejo.yml`, backup scripts, and any existing Forgejo retention/config docs in infrastructure * add a durable, operator-safe Forgejo storage audit runbook or script if the repo does not already have one * update the Forgejo PVC request/config to reflect current live usage plus reasonable near-term headroom, if the manifest owns that value * document that local-path PVC requests are not hard quotas, so this is honest scheduling/bookkeeping, not containment * identify branch/content retention knobs or cleanup candidates, but do not delete live Forgejo data * leave exact operator commands and risk notes for any live measurement or cleanup Kai must run No destructive cleanup. No ad-hoc PVC `rm`. If a safe branch/artifact/package retention task emerges, file a separate follow-up with the exact Forgejo-managed mechanism.

coilyco-ops commented

2026-07-09 17:40:14 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-infrastructure-488 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T17:40:12Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/infrastructure#488 · branch issue-488 · harness codex · workflow direct-to-main
Run: engineer-codex-infrastructure-488 · ward v0.501.0 · dispatched 2026-07-09T17:40:12Z
Comment thread: 3 included in the pre-flight read, 0 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-09T16:56:51Z), @coilysiren (2026-07-09T17:09:27Z), @coilyco-ops (2026-07-09T17:40:04Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.501.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-infrastructure-488` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T17:40:12Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/infrastructure#488` · branch `issue-488` · harness `codex` · workflow `direct-to-main` - **Run:** `engineer-codex-infrastructure-488` · ward `v0.501.0` · dispatched `2026-07-09T17:40:12Z` - **Comment thread:** 3 included in the pre-flight read, 0 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-09T16:56:51Z), @coilysiren (2026-07-09T17:09:27Z), @coilyco-ops (2026-07-09T17:40:04Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.501.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-09 17:50:47 +00:00

feat: add kai-server disk guardrails

coilyco-ops closed this issue

2026-07-09 17:50:47 +00:00

coilyco-ops commented

2026-07-09 17:51:31 +00:00

Author

Member

WARD-OUTCOME: done ✅

details

workflow: direct-to-main; review summary: skipped by ~/.ward/config.yaml default
retrospective: this landed as a focused guardrail sweep plus telemetry.
confidence: high
surprises: origin/main advanced mid-run, so I merged it before the push.
follow-ups: none required for this issue; any live cleanup stays operator-run.

WARD-OUTCOME: done ✅ <details><summary>details</summary> workflow: direct-to-main; review summary: skipped by ~/.ward/config.yaml default retrospective: this landed as a focused guardrail sweep plus telemetry. confidence: high surprises: origin/main advanced mid-run, so I merged it before the push. follow-ups: none required for this issue; any live cleanup stays operator-run. </details>

coilyco-ops referenced this issue

2026-07-09 18:01:03 +00:00

Audit Forgejo branch-heavy storage growth and define safe retention cleanup #490

coilyco-ops referenced this issue

2026-07-23 09:41:38 +00:00

Define and automate Forgejo package and attachment retention after the disk-pressure purge #600

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#488

No description provided.

Rows
Columns

Add durable guardrails for kai-server disk pressure from xrdp logs and Forgejo PVC growth #488

Finding

Do

Acceptance

🔎 ward agent advisor

Recommendation

Key findings

Decisions to make

1. xrdp guardrail

2. journald retention

3. Forgejo PVC growth

4. Disk-pressure detection path

Operator-safe inspection commands

Root filesystem and obvious host offenders

Resolve the live Forgejo PVC host path, then inspect it on-host

In-pod read-only view, if the on-host path is less convenient

If emergency manual cleanup is needed before the durable fix lands

xrdp.log

journald

Forgejo PVC

Advisory close

1. `xrdp` guardrail

2. `journald` retention

`xrdp.log`

`journald`