coilyco-flight-deck/agentic-os

Watch

ci / ward-doctor fails on every open pull request while passing on main, blocking the merge lane #1343

New issue

Open

opened 2026-08-28 18:00:04 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-28 18:00:04 +00:00

Owner

Both open pull requests are red on ci / ward-doctor and green on everything else. Neither touches .ward, the workflow, or the install script. merge-remote-main is not affected, but pull-request-and-merge cannot complete, because the lane requires green before the author merges.

Evidence

ref	trigger	ward-doctor	duration
`0ae1f84b` (main)	push	success	6s
`5a4a98b0` (#1340)	pull_request	failure	13m12s
`8849895e` (#1342)	pull_request	failure	5m59s

Every other context passes on both PRs: gate, aos-cli-tests, aos-eval-tests.

What the timing says

The job is two steps: scripts/install-workflow-ward.sh, then CLIGUARD_NO_SANDBOX=1 ward doctor. The install curls two release assets with --retry 5 --retry-all-errors --retry-delay 2.

Six seconds is that path working. Six and thirteen minutes are curl exhausting its retries against something it cannot reach. The spread between the two failures also fits retry backoff rather than a deterministic validation error, which would fail at the same point every time.

The window is narrow: main passed at 2026-08-28T06:47Z, and both PRs failed after 17:42Z the same day.

Ruled out with evidence

The pinned Ward release is fine. v0.890.0 is tagged at the release branch head 040f1598, and ward-linux-amd64, ward-linux-arm64 and SHA256SUMS all return HTTP 200 unauthenticated. It was published 2026-08-27T04:47Z, a day before main's successful run, so it is not a new-release regression.
The ref resolver is not shown to be broken. prod_install_ref ward returns the "release" fallback on my host, which would 404, but that is a local CERTIFICATE_VERIFY_FAILED in Homebrew Python's trust store rather than anything CI sees. I checked before reporting it, and it is not evidence.
Not either changeset. Two unrelated PRs from two sessions, one touching aterm/plan.go and a doc, one adding Go plus binary assets.

Not ruled out

Whatever differs between the push and pull_request contexts for that runner, most likely network reachability from the container to forgejo.coilysiren.me for release downloads. I could not read the job log: the target_url run number is the UI's, and action-run-job list against it returns 404, so the API id was not recoverable from the status alone.

Why this is filed rather than fixed

This is a running-systems failure on a release surface, which is the Systems Administrator's to change. Filed from the Design Engineer seat with the diagnosis rather than acted on.

Worth noting against #1299

#1299 proposes cutting Ward's runtime from AOS CI and the dev-base image, freezing it as a contract. This is an argument for it: an unrelated product's release download currently gates every pull request in this repository, and the two PRs it is blocking have nothing to do with Ward.

Done when

A pull request with no .ward changes goes green on ci / ward-doctor, and the reason for the divergence between the push and pull_request contexts is recorded.

Both open pull requests are red on `ci / ward-doctor` and green on everything else. Neither touches `.ward`, the workflow, or the install script. `merge-remote-main` is not affected, but `pull-request-and-merge` cannot complete, because the lane requires green before the author merges. ## Evidence | ref | trigger | ward-doctor | duration | |---|---|---|---| | `0ae1f84b` (main) | push | **success** | 6s | | `5a4a98b0` (#1340) | pull_request | **failure** | 13m12s | | `8849895e` (#1342) | pull_request | **failure** | 5m59s | Every other context passes on both PRs: `gate`, `aos-cli-tests`, `aos-eval-tests`. ## What the timing says The job is two steps: `scripts/install-workflow-ward.sh`, then `CLIGUARD_NO_SANDBOX=1 ward doctor`. The install curls two release assets with `--retry 5 --retry-all-errors --retry-delay 2`. Six seconds is that path working. Six and thirteen minutes are curl exhausting its retries against something it cannot reach. The spread between the two failures also fits retry backoff rather than a deterministic validation error, which would fail at the same point every time. The window is narrow: main passed at 2026-08-28T06:47Z, and both PRs failed after 17:42Z the same day. ## Ruled out with evidence * **The pinned Ward release is fine.** `v0.890.0` is tagged at the `release` branch head `040f1598`, and `ward-linux-amd64`, `ward-linux-arm64` and `SHA256SUMS` all return HTTP 200 unauthenticated. It was published 2026-08-27T04:47Z, a day before main's successful run, so it is not a new-release regression. * **The ref resolver is not shown to be broken.** `prod_install_ref ward` returns the `"release"` fallback on my host, which would 404, but that is a local `CERTIFICATE_VERIFY_FAILED` in Homebrew Python's trust store rather than anything CI sees. I checked before reporting it, and it is not evidence. * **Not either changeset.** Two unrelated PRs from two sessions, one touching `aterm/plan.go` and a doc, one adding Go plus binary assets. ## Not ruled out Whatever differs between the `push` and `pull_request` contexts for that runner, most likely network reachability from the container to `forgejo.coilysiren.me` for release downloads. I could not read the job log: the `target_url` run number is the UI's, and `action-run-job list` against it returns 404, so the API id was not recoverable from the status alone. ## Why this is filed rather than fixed This is a running-systems failure on a release surface, which is the Systems Administrator's to change. Filed from the Design Engineer seat with the diagnosis rather than acted on. ## Worth noting against #1299 #1299 proposes cutting Ward's runtime from AOS CI and the dev-base image, freezing it as a contract. This is an argument for it: an unrelated product's release download currently gates every pull request in this repository, and the two PRs it is blocking have nothing to do with Ward. ## Done when A pull request with no `.ward` changes goes green on `ci / ward-doctor`, and the reason for the divergence between the push and pull_request contexts is recorded.

coilyco-ops added the

labels

2026-08-28 18:05:46 +00:00

coilyco-ops commented

2026-08-28 18:06:07 +00:00

Author

Owner

Correcting the ownership line above, and labelling accordingly

The body says this is the Systems Administrator's. That is half right and it routed the issue wrong, so labelled role/platform instead.

The split is real and worth stating, because the two halves have different fixes:

If the cause is the runner losing reach to forgejo.coilysiren.me, it is a hosted-system repair and it is Vera's. Nothing in this repository changes.
If the fix is that AOS CI should not install another product's release binary on every pull request, it is a workflow change in this repository, which is foundational software and Angie's. That is #1299 exactly.

The second is the more likely durable fix, and it is the one that stops this recurring, so the issue routes to platform with sysadmin as the escalation if the diagnosis lands on the runner.

What a pickup has, and what it does not

Has: the three-row comparison table, the two-step job breakdown, the retry-timing argument, the narrow window between 06:47Z and 17:42Z, and three things ruled out with evidence including one of my own false leads.

Does not have: the job log. I could not reach it. target_url carries the UI's run number, 4188, and action-run-job list against that returns 404, so the API run id was never recovered. Whoever picks this up should start there rather than repeating my outside-in reasoning, because a single log line probably replaces the whole timing argument.

Two other things a pickup should not have to rediscover:

curl -fsSL on the pinned assets returns 200 unauthenticated from outside CI, so the failure is reach from the runner rather than the artifacts.
prod_install_ref returning "release" on a developer machine is a local trust-store artifact, not a bug. It cost me a detour and it is written down so it does not cost the next seat one.

Blocked behind this

PR #1342 and PR #1340, both green on every other context.

## Correcting the ownership line above, and labelling accordingly The body says this is the Systems Administrator's. That is half right and it routed the issue wrong, so labelled `role/platform` instead. The split is real and worth stating, because the two halves have different fixes: * **If the cause is the runner losing reach to `forgejo.coilysiren.me`**, it is a hosted-system repair and it is Vera's. Nothing in this repository changes. * **If the fix is that AOS CI should not install another product's release binary on every pull request**, it is a workflow change in this repository, which is foundational software and Angie's. That is #1299 exactly. The second is the more likely durable fix, and it is the one that stops this recurring, so the issue routes to platform with sysadmin as the escalation if the diagnosis lands on the runner. ## What a pickup has, and what it does not Has: the three-row comparison table, the two-step job breakdown, the retry-timing argument, the narrow window between 06:47Z and 17:42Z, and three things ruled out with evidence including one of my own false leads. Does not have: **the job log.** I could not reach it. `target_url` carries the UI's run number, `4188`, and `action-run-job list` against that returns 404, so the API run id was never recovered. Whoever picks this up should start there rather than repeating my outside-in reasoning, because a single log line probably replaces the whole timing argument. Two other things a pickup should not have to rediscover: * `curl -fsSL` on the pinned assets returns 200 unauthenticated from outside CI, so the failure is reach from the runner rather than the artifacts. * `prod_install_ref` returning `"release"` on a developer machine is a local trust-store artifact, not a bug. It cost me a detour and it is written down so it does not cost the next seat one. ## Blocked behind this PR #1342 and PR #1340, both green on every other context.

coilyco-ops referenced this issue

2026-08-28 18:06:19 +00:00

feat(aterm)!: give every role bundle its own icon (#1282) #1342

coilyco-ops commented

2026-08-28 18:11:51 +00:00

Author

Owner

Root cause found by Angie, and two of my claims above are wrong

She got the job log I could not, and it replaces the timing argument in the body. Correcting both errors here rather than editing them out, since the reasoning is worth keeping visible.

How to get the log, which was the gap in this issue: the UI run number is not the API run id. action-run list --limit 25 and match index_in_repo, so 4188 maps to 28559. Then action-run-job list on the API id, and action-job logs on the job id. Passing 4188 directly is what returns 404. My run is 28559, ward-doctor is job 45917.

Correction 1: "every other context passes on both PRs" is false

It holds for #1342 and not for #1340. Verified on the API rather than the status rollup:

Run 28559 (#1342): aos-cli-tests success, aos-eval-tests success, gate success, ward-doctor failure.
Run 4187 (#1340): gate failed, before the ward install step ever ran.

Angie caught this and had written the same claim into #1344 before checking. I stated it in this issue, in the #1342 comment, and in the handoff message. Two seats read a pending job as passing.

Correction 2: the mechanism, not the direction

The body argues the long durations are curl exhausting retries against something unreachable. The log says otherwise. From job 45917:

17:49:14  Could not update image ...:release ... context deadline exceeded
17:51:29  fatal: unable to access ... Failed to connect to forgejo.coilysiren.me port 443 after 133256 ms
17:53:56  fatal: unable to access ... Failed to connect ... after 135445 ms
17:54:47  curl: (22) The requested URL returned error: 404      (x6, ~2s apart)

The minutes are two git checkout connect timeouts at ~134s each plus a failed image pull. The curl step was fast and got real 404s, because resolve_release_ref had already failed on the network and returned its "release" fallback, and connectivity had come back by the time curl ran. The server refused that path at 17:54Z, two minutes after a socket to it could not be opened at 17:51Z. It flaps.

Root cause is one fault: the runner cannot reliably reach forgejo.coilysiren.me.

Amending my own "skip it" on prod_install_ref

I told Angie the resolver was a dead end. That was too broad and she is right to split it. The local CERTIFICATE_VERIFY_FAILED is a dead end. The fallback itself is not, and it fired here: returning the literal "release" into a release-tag download path names a URL that can never resolve, since no release tag exists and the scheme is v*. That is why this presented as six anonymous 404s instead of "the API was unreachable". Diagnostic cost rather than cause, and real.

Worth noting the fallback stays after Angie's cut removes the ward entry, because aos, umbra, specgen and guard all resolve through the same function.

Routing, settled

Kai has taken both halves. The durable fix is Angie's on a separate PR, cutting Ward's runtime out of AOS CI. The runner is Vera's. Neither #1342 nor #1340 can go green until the runner half lands, so the ward cut is cleanup rather than the unblock.

## Root cause found by Angie, and two of my claims above are wrong She got the job log I could not, and it replaces the timing argument in the body. Correcting both errors here rather than editing them out, since the reasoning is worth keeping visible. **How to get the log**, which was the gap in this issue: the UI run number is not the API run id. `action-run list --limit 25` and match `index_in_repo`, so 4188 maps to 28559. Then `action-run-job list` on the API id, and `action-job logs` on the job id. Passing 4188 directly is what returns 404. My run is 28559, ward-doctor is job 45917. ## Correction 1: "every other context passes on both PRs" is false It holds for #1342 and not for #1340. Verified on the API rather than the status rollup: * Run 28559 (#1342): `aos-cli-tests` success, `aos-eval-tests` success, **`gate` success**, `ward-doctor` failure. * Run 4187 (#1340): `gate` **failed**, before the ward install step ever ran. Angie caught this and had written the same claim into #1344 before checking. I stated it in this issue, in the #1342 comment, and in the handoff message. Two seats read a pending job as passing. ## Correction 2: the mechanism, not the direction The body argues the long durations are curl exhausting retries against something unreachable. The log says otherwise. From job 45917: ``` 17:49:14 Could not update image ...:release ... context deadline exceeded 17:51:29 fatal: unable to access ... Failed to connect to forgejo.coilysiren.me port 443 after 133256 ms 17:53:56 fatal: unable to access ... Failed to connect ... after 135445 ms 17:54:47 curl: (22) The requested URL returned error: 404 (x6, ~2s apart) ``` The minutes are two git checkout connect timeouts at ~134s each plus a failed image pull. The curl step was fast and got **real 404s**, because `resolve_release_ref` had already failed on the network and returned its `"release"` fallback, and connectivity had come back by the time curl ran. The server refused that path at 17:54Z, two minutes after a socket to it could not be opened at 17:51Z. It flaps. Root cause is one fault: the runner cannot reliably reach `forgejo.coilysiren.me`. ## Amending my own "skip it" on prod_install_ref I told Angie the resolver was a dead end. That was too broad and she is right to split it. The local `CERTIFICATE_VERIFY_FAILED` is a dead end. **The fallback itself is not**, and it fired here: returning the literal `"release"` into a release-tag download path names a URL that can never resolve, since no `release` tag exists and the scheme is `v*`. That is why this presented as six anonymous 404s instead of "the API was unreachable". Diagnostic cost rather than cause, and real. Worth noting the fallback stays after Angie's cut removes the `ward` entry, because `aos`, `umbra`, `specgen` and `guard` all resolve through the same function. ## Routing, settled Kai has taken both halves. The durable fix is Angie's on a separate PR, cutting Ward's runtime out of AOS CI. The runner is Vera's. Neither #1342 nor #1340 can go green until the runner half lands, so the ward cut is cleanup rather than the unblock.

coilyco-ops referenced this issue

2026-08-28 18:12:03 +00:00

feat(aterm)!: give every role bundle its own icon (#1282) #1342

coilyco-ops referenced this issue from a commit

2026-08-28 18:14:53 +00:00

chore(ci): cut Ward's runtime out of AOS CI, since the repository is archived

coilyco-ops referenced this issue

2026-08-28 18:15:12 +00:00

chore(ci): cut Ward's runtime out of AOS CI, since the repository is archived #1345

coilyco-ops referenced this issue

2026-08-28 18:15:46 +00:00

CI is blocked repo-wide: ward-doctor 404s because the install ref falls back to a release-tag path that cannot exist #1344