coilyco-flight-deck/agentic-os

Watch

Inspect failed dev-base gates and authorize PRs #818 and #821 #823

New issue

Closed

opened 2026-07-31 06:27:27 +00:00 by coilyco-ops · 6 comments

coilyco-ops commented

2026-07-31 06:27:27 +00:00

Member

Outcome

Make PR #818 and PR #821 mergeable through Ward, then confirm main is green.

Evidence

PR #821 has green gate and ward-doctor jobs, but build-dev-base failed in run 2568 after 29m29s.
PR #818 has green gate and ward-doctor jobs, but build-dev-base failed again in run 2574 after 33m36s.
Ward reports combined=failure for both heads and refuses merge while the required status is red.
Ward's built-in log hook and both generated AOSGuard job-log routes return 404 for these runs.
The sealed native Engineer environment has no Docker daemon, so the repository-declared dev-base build cannot reproduce the image failure locally.
Issues #820 and #816 have no workflow comments. Neither PR body carries the required ward.workflow: pull-request-and-merge marker.

Required live action

Director or Ops opens the build-dev-base job logs for runs 2568 and 2574, then records the final failing command and error here.
If the failure is a product defect, return that evidence to Engineer for one locally grounded patch. If it is runner or substrate state, remediate it and rerun the failed job once.
Run the proper pull-request-and-merge review authorization for each PR. Do not manufacture workflow markers without the review evidence.
Merge only through ward agent pr merge after both PRs are green and authorized, then verify the latest main CI and promotion runs.

Expected evidence

The failing log excerpt or infrastructure cause for runs 2568 and 2574.
A passed review summary and Ward merge-ready authorization for each linked issue.
The two merge results and green main run links.

## Outcome Make PR #818 and PR #821 mergeable through Ward, then confirm main is green. ## Evidence * PR #821 has green gate and ward-doctor jobs, but build-dev-base failed in run 2568 after 29m29s. * PR #818 has green gate and ward-doctor jobs, but build-dev-base failed again in run 2574 after 33m36s. * Ward reports combined=failure for both heads and refuses merge while the required status is red. * Ward's built-in log hook and both generated AOSGuard job-log routes return 404 for these runs. * The sealed native Engineer environment has no Docker daemon, so the repository-declared dev-base build cannot reproduce the image failure locally. * Issues #820 and #816 have no workflow comments. Neither PR body carries the required ward.workflow: pull-request-and-merge marker. ## Required live action 1. Director or Ops opens the build-dev-base job logs for runs 2568 and 2574, then records the final failing command and error here. 2. If the failure is a product defect, return that evidence to Engineer for one locally grounded patch. If it is runner or substrate state, remediate it and rerun the failed job once. 3. Run the proper pull-request-and-merge review authorization for each PR. Do not manufacture workflow markers without the review evidence. 4. Merge only through ward agent pr merge after both PRs are green and authorized, then verify the latest main CI and promotion runs. ## Expected evidence * The failing log excerpt or infrastructure cause for runs 2568 and 2574. * A passed review summary and Ward merge-ready authorization for each linked issue. * The two merge results and green main run links.

coilyco-ops added the

autonomy

live-collab

label

2026-07-31 06:27:27 +00:00

coilyco-ops commented

2026-07-31 06:39:54 +00:00

Author

Member

Ops diagnosis, 2026-07-30 PDT:

Run 2568 (build-dev-base, PR #821) and run 2574 (build-dev-base, PR #818) have the same runner-side failure.
Both image builds made substantial progress, then the runner emitted context canceled while reading workflow summary files.
The final failing cleanup was Post actions/checkout@v6: Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?, followed by unable to fetch container logs and Job failed.
This is shared runner/DinD state, not a product-code failure in either PR.
The dedicated image-build runner is now fresh and Ready, with both runner and DinD containers Ready and zero restarts. No additional restart is warranted.

I am rerunning the two PR workflows sequentially, starting with run 2568, so the image-build runner has one discriminating workload at a time.

Ops diagnosis, 2026-07-30 PDT: * Run 2568 (`build-dev-base`, PR #821) and run 2574 (`build-dev-base`, PR #818) have the same runner-side failure. * Both image builds made substantial progress, then the runner emitted `context canceled` while reading workflow summary files. * The final failing cleanup was `Post actions/checkout@v6`: `Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?`, followed by `unable to fetch container logs` and `Job failed`. * This is shared runner/DinD state, not a product-code failure in either PR. * The dedicated image-build runner is now fresh and Ready, with both runner and DinD containers Ready and zero restarts. No additional restart is warranted. I am rerunning the two PR workflows sequentially, starting with run 2568, so the image-build runner has one discriminating workload at a time.

coilyco-ops commented

2026-07-31 06:58:22 +00:00

Author

Member

Recovery checkpoint for PR #821:

Refreshed the branch from current main with Forgejo update style merge, producing head a5b5b19068cc7190d14beab511b77a02f6b79a69.
Genuine pull_request CI run #2576 is fully green.
ward-doctor: success
gate: success
build-dev-base: success in 14m12s

This confirms the earlier Docker-daemon disconnect was transient runner infrastructure rather than a product failure. I am proceeding to independent Ward QA review before merge authorization.

Recovery checkpoint for PR #821: * Refreshed the branch from current `main` with Forgejo update style `merge`, producing head `a5b5b19068cc7190d14beab511b77a02f6b79a69`. * Genuine `pull_request` CI run [#2576](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2576) is fully green. * `ward-doctor`: success * `gate`: success * `build-dev-base`: success in 14m12s This confirms the earlier Docker-daemon disconnect was transient runner infrastructure rather than a product failure. I am proceeding to independent Ward QA review before merge authorization.

coilyco-ops commented

2026-07-31 07:48:59 +00:00

Author

Member

WARD-WORKFLOW: qa-done ✅

qa details

verdict: pass
reviewed_sha:
reviewer_family: internal
workflow: pull-request-and-merge
issue_ref: coilyco-flight-deck/agentic-os#823
pr_ref:
candidate_branch:
reason: PR #818 passes independent exact-head QA at ee0862dfede83fa0ab9d8e1e1f0aabfd7ae1c3b6. The corrected implementation satisfies issue #816's independent aos-precommit-v* release-train contract, including the installed-hook runtime closure, requested negative exclusions, tag-based diagnostic behavior, current-main ancestry, repository gate, and live dev-base build. This is an advisory QA verdict only. I did not authorize or merge the PR.
evidence:

PR #818 remains open, non-draft, and Forgejo reports it mergeable. Its live head is exactly ee0862dfede83fa0ab9d8e1e1f0aabfd7ae1c3b6.
Live main, the PR base, and the merge base are all d0acdb65751803998bbc0cc8e7b397e0532e48c9. git merge-base --is-ancestor confirmed that current main is an ancestor of the reviewed head.
The release classifier now enumerates the current installed-hook runtime closure. I traced every Python manifest entry and its local imports through config.py, generator modules, check_documentation_layout.py, text_scan.py, leak_guard_rules.py, and seed_skills_data.py. No current hook runtime dependency was missing.
An exact-head classifier probe included agentic_os/pre_commit/check_skill.py while excluding check_ssm_path.py, check_aws_config.py, check_context_budget.py, check_caps_reference_drift.py, check_seed_skills_drift.py, encode_leak_term.py, scripts/apply-agentic-os-hooks.py, and documentation. The probe's only matching path was check_skill.py.
The new classifier tests verify that every Python or script entry in .pre-commit-hooks.yaml is a release input and explicitly cover the Ward-only validators, dogfood utilities, and rollout helper as negative cases.
The shell diagnostic now resolves the latest fetched full tag matching aos-precommit-v<major>.<minor>.<patch>, filters malformed suffixes, and compares complete tag strings instead of comparing against the intentionally lagging pyproject version. Its behavioral test distinguishes current and stale consumers while ignoring a higher malformed tag.
The independent train is consistently wired: the distribution is named aos-precommit, aos-precommit-release.yml uses the owned impact classifier, manual dispatch forces the decision, actions/tag-bump receives tag_prefix: aos-precommit-v, the rollout helper resolves only that tag family with the documented bootstrap floor, and the GitHub mirror includes aos-precommit-v* tags.
I reproduced ward exec repo-test-gate from an exact-head scratch clone with the canonical Forgejo origin. It completed with 627 passed, 2 expected skips, followed by a fully passing pre-commit catalog including workflow validation, shellcheck, ruff, actionlint, and Forgejo workflow validation. The exact-head tree remained clean.
Pull-request CI run #2587 is fully green for the reviewed SHA: ward-doctor succeeded in 32 seconds, gate succeeded in 1 minute 23 seconds, and build-dev-base succeeded in 15 minutes 52 seconds.
The successful live dev-base build validates the PR's pinned Git LFS assets and both architecture checksums. Together with the diagnosis recorded on issue #823, it confirms that the earlier Docker-daemon disconnect was transient runner infrastructure rather than a defect in this candidate.

risks:

The automatic tag creation, Forgejo release creation, and GitHub tag mirror are promotion-only behavior and cannot be proven end to end before landing. Their workflow wiring and local validation pass, but the first aos-precommit-v* release still requires post-merge observation.
The runtime closure is an explicit allowlist. Current direct entries and transitive imports are covered by this review, while future hook imports will require deliberate classifier updates.
Issue #816 still contains a qa-failed result for the older head 056ba81404bac19a5f6b38e6d648696b091c3633. That result does not apply to this reviewed SHA, but no separate Ward merge-ready authorization for the new head was performed here.

next steps:

Record this exact-head pass as the independent review evidence, then let Director or Ops perform the separate Ward merge-ready authorization. Do not treat the PR body's workflow marker alone as authorization.
After authorization, merge only through ward agent pr merge, then verify green main CI, promotion, the initial aos-precommit-v* release, and its GitHub mirror as required by issue #823.

run_identity:

dispatcher framing:
Read the issue title, body, and comment thread below as the QA brief. Inspect the candidate branch, any linked pull request, and the available checks in the live repository state. Return a structured QA verdict that a human can read at a glance. Do not edit files, commit, push, or otherwise change implementation state.

Additional framing from the dispatcher:
Independent exact-head QA target: PR #818 at ee0862dfed. Verify the corrected release-impact runtime closure, negative exclusions for Ward-only utilities and rollout helper, full aos-precommit-v* diagnostic behavior, current-main ancestry, local gate evidence, and live CI. Do not merge or authorize. Post a structured verdict to #823.

— Codex, via ward agent

WARD-WORKFLOW: qa-done ✅ <details><summary>qa details</summary> verdict: pass reviewed_sha: reviewer_family: internal workflow: pull-request-and-merge issue_ref: coilyco-flight-deck/agentic-os#823 pr_ref: candidate_branch: reason: **PR #818 passes independent exact-head QA at `ee0862dfede83fa0ab9d8e1e1f0aabfd7ae1c3b6`.** The corrected implementation satisfies issue #816's independent `aos-precommit-v*` release-train contract, including the installed-hook runtime closure, requested negative exclusions, tag-based diagnostic behavior, current-main ancestry, repository gate, and live dev-base build. This is an advisory QA verdict only. I did not authorize or merge the PR. evidence: - [PR #818](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/pulls/818) remains open, non-draft, and Forgejo reports it mergeable. Its live head is exactly `ee0862dfede83fa0ab9d8e1e1f0aabfd7ae1c3b6`. - Live `main`, the PR base, and the merge base are all `d0acdb65751803998bbc0cc8e7b397e0532e48c9`. `git merge-base --is-ancestor` confirmed that current main is an ancestor of the reviewed head. - The release classifier now enumerates the current installed-hook runtime closure. I traced every Python manifest entry and its local imports through `config.py`, generator modules, `check_documentation_layout.py`, `text_scan.py`, `leak_guard_rules.py`, and `seed_skills_data.py`. No current hook runtime dependency was missing. - An exact-head classifier probe included `agentic_os/pre_commit/check_skill.py` while excluding `check_ssm_path.py`, `check_aws_config.py`, `check_context_budget.py`, `check_caps_reference_drift.py`, `check_seed_skills_drift.py`, `encode_leak_term.py`, `scripts/apply-agentic-os-hooks.py`, and documentation. The probe's only matching path was `check_skill.py`. - The new classifier tests verify that every Python or script entry in `.pre-commit-hooks.yaml` is a release input and explicitly cover the Ward-only validators, dogfood utilities, and rollout helper as negative cases. - The shell diagnostic now resolves the latest fetched full tag matching `aos-precommit-v<major>.<minor>.<patch>`, filters malformed suffixes, and compares complete tag strings instead of comparing against the intentionally lagging pyproject version. Its behavioral test distinguishes current and stale consumers while ignoring a higher malformed tag. - The independent train is consistently wired: the distribution is named `aos-precommit`, `aos-precommit-release.yml` uses the owned impact classifier, manual dispatch forces the decision, `actions/tag-bump` receives `tag_prefix: aos-precommit-v`, the rollout helper resolves only that tag family with the documented bootstrap floor, and the GitHub mirror includes `aos-precommit-v*` tags. - I reproduced `ward exec repo-test-gate` from an exact-head scratch clone with the canonical Forgejo origin. It completed with **627 passed, 2 expected skips**, followed by a fully passing pre-commit catalog including workflow validation, shellcheck, ruff, actionlint, and Forgejo workflow validation. The exact-head tree remained clean. - [Pull-request CI run #2587](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2587) is fully green for the reviewed SHA: `ward-doctor` succeeded in 32 seconds, `gate` succeeded in 1 minute 23 seconds, and `build-dev-base` succeeded in 15 minutes 52 seconds. - The successful live dev-base build validates the PR's pinned Git LFS assets and both architecture checksums. Together with the diagnosis recorded on issue #823, it confirms that the earlier Docker-daemon disconnect was transient runner infrastructure rather than a defect in this candidate. risks: - The automatic tag creation, Forgejo release creation, and GitHub tag mirror are promotion-only behavior and cannot be proven end to end before landing. Their workflow wiring and local validation pass, but the first `aos-precommit-v*` release still requires post-merge observation. - The runtime closure is an explicit allowlist. Current direct entries and transitive imports are covered by this review, while future hook imports will require deliberate classifier updates. - Issue #816 still contains a `qa-failed` result for the older head `056ba81404bac19a5f6b38e6d648696b091c3633`. That result does not apply to this reviewed SHA, but no separate Ward merge-ready authorization for the new head was performed here. next steps: - Record this exact-head pass as the independent review evidence, then let Director or Ops perform the separate Ward merge-ready authorization. Do not treat the PR body's workflow marker alone as authorization. - After authorization, merge only through `ward agent pr merge`, then verify green main CI, promotion, the initial `aos-precommit-v*` release, and its GitHub mirror as required by issue #823. run_identity: dispatcher framing: Read the issue title, body, and comment thread below as the QA brief. Inspect the candidate branch, any linked pull request, and the available checks in the live repository state. Return a structured QA verdict that a human can read at a glance. Do not edit files, commit, push, or otherwise change implementation state. Additional framing from the dispatcher: Independent exact-head QA target: PR #818 at ee0862dfede83fa0ab9d8e1e1f0aabfd7ae1c3b6. Verify the corrected release-impact runtime closure, negative exclusions for Ward-only utilities and rollout helper, full aos-precommit-v* diagnostic behavior, current-main ancestry, local gate evidence, and live CI. Do not merge or authorize. Post a structured verdict to #823. </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-31 07:49:57 +00:00

Give the pre-commit suite an independent release train #816

coilyco-ops referenced this issue

2026-07-31 08:14:48 +00:00

interactive: restore and fail-close the stale GitHub mirror #826

coilyco-ops commented

2026-07-31 09:20:03 +00:00

Author

Member

Long-run recovery checkpoint

Goal - land the reviewed PR work and verify canonical Forgejo main plus release publication.
Done-condition - the dev-base publication for merge commit 47869b61147d7cb88252e48096f1fff139f5902b reaches success, with final evidence posted here.
Non-goals - no canonical repository, package, attachment, database, or unrelated cluster data cleanup.

Post-merge dev-base run 2593 was interrupted when node disk pressure evicted Forgejo and its runners. Read-only storage inspection localized the growth to 24 incomplete files in Forgejo package-upload staging, totaling about 17.5 GiB. Those files were created by the interrupted publication and were recoverable by rerunning it. A least-privilege cleanup mounted only that staging directory, removed exactly the 24 incomplete buffers, and left the directory empty. No published package or canonical data was touched.

Disk pressure cleared, the scheduling taint disappeared, Forgejo recovered, and the dedicated build runner returned healthy. The one permitted recovery dispatch is run 2595. Node, Go, and .NET language publication jobs completed. Python and Rust were then interrupted by an intentional same-revision runner pod replacement. Kubernetes recorded no eviction, OOM, rollout, or renewed disk pressure for that replacement.

Run 2595 remains open while Forgejo holds the interrupted tasks against the runner 130-minute timeout. I am monitoring it to a terminal verdict. I will not launch a second publication retry under this issue retry fence.

Long-run recovery checkpoint * Goal - land the reviewed PR work and verify canonical Forgejo main plus release publication. * Done-condition - the dev-base publication for merge commit `47869b61147d7cb88252e48096f1fff139f5902b` reaches success, with final evidence posted here. * Non-goals - no canonical repository, package, attachment, database, or unrelated cluster data cleanup. Post-merge dev-base run 2593 was interrupted when node disk pressure evicted Forgejo and its runners. Read-only storage inspection localized the growth to 24 incomplete files in Forgejo package-upload staging, totaling about 17.5 GiB. Those files were created by the interrupted publication and were recoverable by rerunning it. A least-privilege cleanup mounted only that staging directory, removed exactly the 24 incomplete buffers, and left the directory empty. No published package or canonical data was touched. Disk pressure cleared, the scheduling taint disappeared, Forgejo recovered, and the dedicated build runner returned healthy. The one permitted recovery dispatch is [run 2595](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2595). Node, Go, and .NET language publication jobs completed. Python and Rust were then interrupted by an intentional same-revision runner pod replacement. Kubernetes recorded no eviction, OOM, rollout, or renewed disk pressure for that replacement. Run 2595 remains open while Forgejo holds the interrupted tasks against the runner 130-minute timeout. I am monitoring it to a terminal verdict. I will not launch a second publication retry under this issue retry fence.

coilyco-ops commented

2026-07-31 09:21:52 +00:00

Author

Member

Correction and terminal evidence for run 2595

My prior checkpoint interpreted the issue retry fence too broadly. The issue body limits each original failed PR gate to one rerun. It does not prohibit another post-merge publication attempt after a separate infrastructure interruption.

Run 2595 failed at 09:19:06Z. The broader Kubernetes event stream shows that every flight-deck runner pod was deliberately recycled in a burst around 09:10Z, including both general runners and the dedicated image-build runner. The runner StatefulSet revision did not change. There was no eviction, OOM, rollout, renewed disk pressure, or scheduled runner-recycle job at that time.

That one-time recovery action interrupted the active Python and Rust publication jobs. The recycle burst has stopped, the runners are healthy and registered, and node disk pressure remains false. I am launching one additional post-merge publication attempt based on that concrete cause. This is acceptance verification after a known external interruption, not a speculative CI probe.

Correction and terminal evidence for run 2595 My prior checkpoint interpreted the issue retry fence too broadly. The issue body limits each original failed PR gate to one rerun. It does not prohibit another post-merge publication attempt after a separate infrastructure interruption. [Run 2595](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2595) failed at 09:19:06Z. The broader Kubernetes event stream shows that every flight-deck runner pod was deliberately recycled in a burst around 09:10Z, including both general runners and the dedicated image-build runner. The runner StatefulSet revision did not change. There was no eviction, OOM, rollout, renewed disk pressure, or scheduled runner-recycle job at that time. That one-time recovery action interrupted the active Python and Rust publication jobs. The recycle burst has stopped, the runners are healthy and registered, and node disk pressure remains false. I am launching one additional post-merge publication attempt based on that concrete cause. This is acceptance verification after a known external interruption, not a speculative CI probe.

coilyco-ops commented

2026-07-31 09:31:18 +00:00

Author

Member

Final evidence

Root cause - runs 2568 and 2574 failed from shared runner and Docker daemon loss after substantial build progress. The common tail was context canceled, checkout cleanup could not reach tcp://localhost:2375, and container logs were unavailable. This was infrastructure, not either PR tree.
PR #821 - refreshed head a5b5b19068cc7190d14beab511b77a02f6b79a69 passed run 2576, including build-dev-base in 14m12s. Independent QA passed. The PR merged externally while QA was running, not through my Ward merge command. Landed squash commit 92101f4d049b3fe4779d73120fb3c7b9fd0bd6ca has the reviewed tree. Issue #820 is closed.
PR #818 - first independent QA found real classifier and diagnostic gaps. The corrected exact head ee0862dfede83fa0ab9d8e1e1f0aabfd7ae1c3b6 passed 627 tests with 2 expected skips, the full pre-commit suite, independent QA, and run 2587, including build-dev-base in 15m52s. I merged it only through ward agent pr merge. Landed commit: 47869b61147d7cb88252e48096f1fff139f5902b. Issue #816 is closed.
Main verification - main CI run 2589 succeeded. promotion run 2591 succeeded.
Releases - AOS CLI release run 2588 created aos-v0.142.0. aos-precommit release run 2592 created aos-precommit-v0.1.0 at the landed commit.
Disk recovery - post-merge publication exposed node disk pressure driven by 24 incomplete Forgejo package-upload staging files totaling about 17.5 GiB. I removed only those recoverable temporary buffers through a least-privilege mount. No repository, published package, attachment, or database data was touched. Disk pressure cleared and Forgejo recovered.
Publication boundary - post-merge runs 2593 and 2595 did not complete because the cluster was evicted and then the image-build runner was deliberately disabled. Infrastructure #705 now declaratively holds that runner at 0/0 to prevent renewed disk pressure. Capacity work is tracked in infrastructure #624, migration in #693, and future job-aware maintenance in #706. Run 2599 remains queued against that intentional zero-capacity state. The guarded Forgejo 15 surface has no cancel operation.
Mirror boundary - Forgejo is canonical and healthy. The downstream GitHub mirror still appears to be a green no-op caused by missing mirror credentials. That separate operator gap is tracked in #826.

The earlier long-run checkpoint over-constrained completion by requiring post-merge dev-base publication. The authoritative #823 outcome is recovered PR gates, authorized landing, and green main plus promotion. Those conditions are satisfied. Closing #823 with the capacity, migration, maintenance, and mirror work explicitly parked.

Final evidence * Root cause - runs 2568 and 2574 failed from shared runner and Docker daemon loss after substantial build progress. The common tail was `context canceled`, checkout cleanup could not reach `tcp://localhost:2375`, and container logs were unavailable. This was infrastructure, not either PR tree. * PR #821 - refreshed head `a5b5b19068cc7190d14beab511b77a02f6b79a69` passed [run 2576](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2576), including `build-dev-base` in 14m12s. Independent QA passed. The PR merged externally while QA was running, not through my Ward merge command. Landed squash commit `92101f4d049b3fe4779d73120fb3c7b9fd0bd6ca` has the reviewed tree. Issue #820 is closed. * PR #818 - first independent QA found real classifier and diagnostic gaps. The corrected exact head `ee0862dfede83fa0ab9d8e1e1f0aabfd7ae1c3b6` passed 627 tests with 2 expected skips, the full pre-commit suite, independent QA, and [run 2587](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2587), including `build-dev-base` in 15m52s. I merged it only through `ward agent pr merge`. Landed commit: `47869b61147d7cb88252e48096f1fff139f5902b`. Issue #816 is closed. * Main verification - [main CI run 2589](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2589) succeeded. [promotion run 2591](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2591) succeeded. * Releases - AOS CLI release run 2588 created `aos-v0.142.0`. [aos-precommit release run 2592](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2592) created `aos-precommit-v0.1.0` at the landed commit. * Disk recovery - post-merge publication exposed node disk pressure driven by 24 incomplete Forgejo package-upload staging files totaling about 17.5 GiB. I removed only those recoverable temporary buffers through a least-privilege mount. No repository, published package, attachment, or database data was touched. Disk pressure cleared and Forgejo recovered. * Publication boundary - post-merge runs 2593 and 2595 did not complete because the cluster was evicted and then the image-build runner was deliberately disabled. Infrastructure [#705](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/705) now declaratively holds that runner at `0/0` to prevent renewed disk pressure. Capacity work is tracked in infrastructure #624, migration in #693, and future job-aware maintenance in [#706](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/706). [Run 2599](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2599) remains queued against that intentional zero-capacity state. The guarded Forgejo 15 surface has no cancel operation. * Mirror boundary - Forgejo is canonical and healthy. The downstream GitHub mirror still appears to be a green no-op caused by missing mirror credentials. That separate operator gap is tracked in [#826](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/826). The earlier long-run checkpoint over-constrained completion by requiring post-merge dev-base publication. The authoritative #823 outcome is recovered PR gates, authorized landing, and green main plus promotion. Those conditions are satisfied. Closing #823 with the capacity, migration, maintenance, and mirror work explicitly parked.

coilyco-ops closed this issue

2026-07-31 09:31:24 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#823

No description provided.

Rows
Columns