Fix current agentic-os main and release CI red state #548

New issue

Closed

opened 2026-07-15 04:26:41 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-15 04:26:41 +00:00

Owner

The director observed agentic-os CI red on both main and release while babysitting ward / aos CI on 2026-07-15. Ward is currently green on both branches, so this issue is scoped to agentic-os.

Evidence:

main head 55d308f7ee has failed promote and mirror runs.
main run 1337, promote.yml, failed in gate. Tests passed, then pre-commit env setup started and the job ended with runner context canceled / deadline exceeded while the alert step also failed.
main run 1336, mirror-to-github.yml, failed in test with the same pattern: 469 tests passed, pre-commit env setup started, then runner context deadline exceeded.
main run 1335, ci.yml, was cancelled on the same head. ward-doctor passed, gate was cancelled.
release head 24980330c2 has failed release run 1277. publish-lang-go and publish-lang-dotnet failed, publish-full and release were skipped.
release run 1277 job list shows publish-lang-go and publish-lang-dotnet failed. publish-lang-go built and pushed lang-go-v0.253.0 / lang-go-release, then Alert Telegram on main failure failed the job. publish-lang-dotnet was interrupted during arm64 .NET SDK install and ended with runner context canceled.

Expected outcome:

agentic-os main branch latest CI for the current head is green.
agentic-os release branch latest CI for the current head is green.
A follow-up comment on this issue names the final green run URLs.

Notes:

The failure mode looks like runner timeout / cancellation plus alert hard-fail behavior around long jobs, not a unit-test regression.
Do not debug a live deployment from a sealed engineer container. If the fix requires live runner or cluster observation, file an interactive-labeled follow-up with the exact live verification needed.

The director observed agentic-os CI red on both main and release while babysitting ward / aos CI on 2026-07-15. Ward is currently green on both branches, so this issue is scoped to agentic-os. Evidence: * main head 55d308f7ee0726a56a249ddb6ea07b4cc88b57e2 has failed promote and mirror runs. * main run 1337, promote.yml, failed in gate. Tests passed, then pre-commit env setup started and the job ended with runner context canceled / deadline exceeded while the alert step also failed. * main run 1336, mirror-to-github.yml, failed in test with the same pattern: 469 tests passed, pre-commit env setup started, then runner context deadline exceeded. * main run 1335, ci.yml, was cancelled on the same head. ward-doctor passed, gate was cancelled. * release head 24980330c233d1338be428a16a0a2750f24a3749 has failed release run 1277. publish-lang-go and publish-lang-dotnet failed, publish-full and release were skipped. * release run 1277 job list shows publish-lang-go and publish-lang-dotnet failed. publish-lang-go built and pushed lang-go-v0.253.0 / lang-go-release, then Alert Telegram on main failure failed the job. publish-lang-dotnet was interrupted during arm64 .NET SDK install and ended with runner context canceled. Expected outcome: * agentic-os main branch latest CI for the current head is green. * agentic-os release branch latest CI for the current head is green. * A follow-up comment on this issue names the final green run URLs. Notes: * The failure mode looks like runner timeout / cancellation plus alert hard-fail behavior around long jobs, not a unit-test regression. * Do not debug a live deployment from a sealed engineer container. If the fix requires live runner or cluster observation, file an interactive-labeled follow-up with the exact live verification needed.

coilyco-ops commented

2026-07-15 04:27:13 +00:00

Author

Owner

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-agentic-os-548 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-15T04:27:12Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/agentic-os#548 · branch issue-548 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-agentic-os-548 · ward v0.697.0 · dispatched 2026-07-15T04:27:02Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.697.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-agentic-os-548` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-15T04:27:12Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/agentic-os#548` · branch `issue-548` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-agentic-os-548` · ward `v0.697.0` · dispatched `2026-07-15T04:27:02Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.697.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a commit

2026-07-15 04:36:27 +00:00

fix(ci): warm pre-commit cache and soften alert failures

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-15 04:36:47 +00:00

fix(ci): warm pre-commit cache and soften alert failures #550

coilyco-ops commented

2026-07-15 04:40:16 +00:00

Author

Owner

WARDED_WORKFLOW: #550

details

workflow: pull-request-and-merge; review summary: skipped because the temporary ward default pending brokered QA

The cache hit removed the pre-commit cold-start drag, and the alert step no longer poisons a healthy job when Telegram flakes.

Confidence: medium-high.
Surprises: the main gate was spending its time in pre-commit env setup, not in pytest.
Follow-ups: watch the merge-triggered main and release runs, then add the final green run URLs after landing.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/pulls/550 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped because the temporary ward default pending brokered QA The cache hit removed the pre-commit cold-start drag, and the alert step no longer poisons a healthy job when Telegram flakes. Confidence: medium-high. Surprises: the main gate was spending its time in pre-commit env setup, not in pytest. Follow-ups: watch the merge-triggered main and release runs, then add the final green run URLs after landing. </details>

coilyco-ops referenced this issue

2026-07-15 04:48:34 +00:00

Make release branch a promote-only last-known-good ref #551

coilyco-ops closed this issue

2026-07-15 05:13:21 +00:00