coilyco-flight-deck/agentic-os

Watch

CI is blocked repo-wide: ward-doctor 404s because the install ref falls back to a release-tag path that cannot exist #1344

New issue

Closed

opened 2026-08-28 18:00:56 +00:00 by coilyco-ops · 8 comments

coilyco-ops commented

2026-08-28 18:00:56 +00:00

Owner

Symptom

Every open pull request fails ci.yml on the ward-doctor job. Confirmed on two PRs at unrelated commits, so this is not any one branch's change:

#1340 (run index 4187, job 45912, head 5a4a98b0) - Failing after 13m12s
#1342 (run index 4188, job 45917, head 8849895e) - same failure

The last green ci.yml on main is run index 4183 at 0ae1f84b, so the break landed after that.

The failing step is Install validated Ward running scripts/install-workflow-ward.sh. Its whole output is six retries of the same line, with no URL, which is why the cause is not readable from the log:

curl: (22) The requested URL returned error: 404
⚙️ [runner]: exitcode '22': failure

Cause

scripts/install-workflow-ward.sh builds its download base from agentic_os.prod_install_ref ward:

base="https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/${ref}"

resolve_release_ref reads the release branch head, finds the version tag pointing at that sha, and on any KeyError/TypeError/OSError/ValueError returns the literal string "release" as what its docstring calls "a safe fallback".

That fallback is not safe, because the string is interpolated into a release-tag download path, not a branch path. Probed directly just now:

https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/release/ward-linux-amd64 -> 404
https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/v0.890.0/ward-linux-amd64 -> 200

There is no git tag named release and no release keyed to a release tag (/tags/release and /releases/tags/release both 404), and the tag scheme is v(\d+\.\d+\.\d+), so a release tag is not something that should ever exist. The fallback therefore names a URL that can never resolve. It converts a transient API failure into a bare 404 with no diagnostic instead of failing closed with a readable error, which is the opposite of the fail-closed discipline the checksum step right below it was written for.

The resolution itself is healthy when the API is reachable: the release branch head is 040f15989a002b7b70769e69b3846d172cc59d8b and v0.890.0 points at exactly that sha, so a reachable API yields v0.890.0 and a 200.

What is not established

Why the API call fails from inside the runner. The fallback only fires on an exception, so something in the agentic-os:release container cannot reach https://forgejo.coilysiren.me/api/v1. The job log does not show it, because the fallback swallows the exception.

On the host I checked from, the same call raises URLError [SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate, and that host reaches the same URL fine through curl. That is a plausible shape for the runner's failure too, since a Python urllib without the CA bundle behaves exactly this way while curl in the same container succeeds. This is a hypothesis, not a measurement. Nothing in the logs distinguishes an SSL failure from DNS, a proxy, or a timeout inside the runner.

Note also that coilyco-flight-deck/ward was archived at 2026-08-28T04:22:22Z, shortly before the break. Archiving does not remove branches or tags, and the probes above show both still resolve, so archiving is not obviously the cause. It is recorded here as a nearby change worth ruling in or out rather than as a finding.

Why this needs a human

Restoring API reachability from the runner is a change to a running backend system, which is the Systems Administrator's boundary rather than the platform seat's. The two halves are separable and only one is in platform scope:

Unblock CI - restore whatever the runner lost, or pin the ref. Live-backend work, hand to sysadmin.
Fix the latent defect - make resolve_release_ref fail closed with the underlying exception rather than returning a string that can only 404, and have install-workflow-ward.sh echo the resolved ref so the next failure names itself. Platform work, and safe to do independently, but it does not unblock CI on its own: it converts an unreadable 404 into a readable error and CI stays red until (1) lands.

Reproducing

curl -sS -o /dev/null -w "%{http_code}\n" \
  https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/release/ward-linux-amd64
curl -sS -o /dev/null -w "%{http_code}\n" \
  https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/v0.890.0/ward-linux-amd64

Blocked on this

#1340 is complete and green on every other job (aos-cli-tests, aos-eval-tests, and the repo gate all pass). It is held unmerged only by ward-doctor, since the pull-request-and-merge lane merges on green.

## Symptom Every open pull request fails `ci.yml` on the `ward-doctor` job. Confirmed on two PRs at unrelated commits, so this is not any one branch's change: * #1340 (run index 4187, job 45912, head `5a4a98b0`) - `Failing after 13m12s` * #1342 (run index 4188, job 45917, head `8849895e`) - same failure The last green `ci.yml` on `main` is run index 4183 at `0ae1f84b`, so the break landed after that. The failing step is `Install validated Ward` running `scripts/install-workflow-ward.sh`. Its whole output is six retries of the same line, with no URL, which is why the cause is not readable from the log: ``` curl: (22) The requested URL returned error: 404 ⚙️ [runner]: exitcode '22': failure ``` ## Cause `scripts/install-workflow-ward.sh` builds its download base from `agentic_os.prod_install_ref ward`: ``` base="https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/${ref}" ``` `resolve_release_ref` reads the `release` **branch** head, finds the version tag pointing at that sha, and on any `KeyError/TypeError/OSError/ValueError` returns the literal string `"release"` as what its docstring calls "a safe fallback". That fallback is not safe, because the string is interpolated into a **release-tag download path**, not a branch path. Probed directly just now: * `https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/release/ward-linux-amd64` -> **404** * `https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/v0.890.0/ward-linux-amd64` -> **200** There is no git tag named `release` and no release keyed to a `release` tag (`/tags/release` and `/releases/tags/release` both 404), and the tag scheme is `v(\d+\.\d+\.\d+)`, so a `release` tag is not something that should ever exist. The fallback therefore names a URL that can never resolve. It converts a transient API failure into a bare 404 with no diagnostic instead of failing closed with a readable error, which is the opposite of the fail-closed discipline the checksum step right below it was written for. The resolution itself is healthy when the API is reachable: the `release` branch head is `040f15989a002b7b70769e69b3846d172cc59d8b` and `v0.890.0` points at exactly that sha, so a reachable API yields `v0.890.0` and a 200. ## What is not established **Why the API call fails from inside the runner.** The fallback only fires on an exception, so something in the `agentic-os:release` container cannot reach `https://forgejo.coilysiren.me/api/v1`. The job log does not show it, because the fallback swallows the exception. On the host I checked from, the same call raises `URLError [SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate`, and that host reaches the same URL fine through `curl`. That is a plausible shape for the runner's failure too, since a Python `urllib` without the CA bundle behaves exactly this way while `curl` in the same container succeeds. **This is a hypothesis, not a measurement.** Nothing in the logs distinguishes an SSL failure from DNS, a proxy, or a timeout inside the runner. Note also that `coilyco-flight-deck/ward` was archived at `2026-08-28T04:22:22Z`, shortly before the break. Archiving does not remove branches or tags, and the probes above show both still resolve, so archiving is not obviously the cause. It is recorded here as a nearby change worth ruling in or out rather than as a finding. ## Why this needs a human Restoring API reachability from the runner is a change to a running backend system, which is the Systems Administrator's boundary rather than the platform seat's. The two halves are separable and only one is in platform scope: 1. **Unblock CI** - restore whatever the runner lost, or pin the ref. Live-backend work, hand to sysadmin. 2. **Fix the latent defect** - make `resolve_release_ref` fail closed with the underlying exception rather than returning a string that can only 404, and have `install-workflow-ward.sh` echo the resolved ref so the next failure names itself. Platform work, and safe to do independently, but it does **not** unblock CI on its own: it converts an unreadable 404 into a readable error and CI stays red until (1) lands. ## Reproducing ``` curl -sS -o /dev/null -w "%{http_code}\n" \ https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/release/ward-linux-amd64 curl -sS -o /dev/null -w "%{http_code}\n" \ https://forgejo.coilysiren.me/coilyco-flight-deck/ward/releases/download/v0.890.0/ward-linux-amd64 ``` ## Blocked on this #1340 is complete and green on every other job (`aos-cli-tests`, `aos-eval-tests`, and the repo `gate` all pass). It is held unmerged only by `ward-doctor`, since the `pull-request-and-merge` lane merges on green.

coilyco-ops commented

2026-08-28 18:06:14 +00:00

Author

Owner

Correction: the root cause is runner network, not the install ref

The gate job on run 4187 has since finished, and it changes the diagnosis. Two claims in the issue above are wrong and are corrected here.

Wrong claim 1: "green on every other job." I wrote that while gate was still running. It was not green. gate (job 45913) failed, and it failed before the ward install step ever ran.

Wrong claim 2: the unreachability is unverified. It is now measured. gate died in actions/checkout, on three consecutive retries:

fatal: unable to access 'https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/':
  Failed to connect to forgejo.coilysiren.me port 443 after 133427 ms: Couldn't connect to server
The process '/usr/bin/git' failed with exit code 128

Roughly 133 seconds to a connect timeout, three times, 17:54 to 18:01. Run 4188 also logs Could not update image ... context deadline exceeded reaching the container registry on the same host.

What this means

The runner cannot reliably reach forgejo.coilysiren.me. That single fault explains every symptom, including the one I misattributed:

gate - git fetch over 443 times out. Job dies at checkout.
image pull - registry on the same host, context deadline exceeded.
ward-doctor - resolve_release_ref's Python API call fails on the network, silently falls back to the literal release, and by then connectivity had returned, so curl reached the server and got a real 404 rather than a connect error. That is why this one failure looked like a URL bug instead of a network fault.

It is flapping rather than flat down: a connect timeout at 17:54 and a served 404 at 17:56 are minutes apart on the same host. The host is reachable from my workstation throughout, so the fault is on the runner's path to it, or was a window that has since closed.

What stands and what does not

Does not stand: the framing that this is an install-ref defect blocking CI. It is a network fault. Nothing in prod_install_ref or install-workflow-ward.sh caused it, and fixing them would not have unblocked anything.

Still stands, demoted to a latent defect: resolve_release_ref returning the literal "release" into a release-tag download path is still wrong, because that URL can never resolve and no release tag exists. Its real cost is diagnostic. It converted a network blip into six anonymous 404s and sent me looking at URLs for an hour when the log could have said "the API was unreachable". Worth fixing on its own merits, not as a CI unblock.

Ownership

This is a live infrastructure fault on a running backend, so it sits with the Systems Administrator seat rather than the platform seat. Not something to fix from here.

#1340 remains blocked, now on two failing jobs rather than one, neither caused by its diff.

## Correction: the root cause is runner network, not the install ref The `gate` job on run 4187 has since finished, and it changes the diagnosis. Two claims in the issue above are wrong and are corrected here. **Wrong claim 1: "green on every other job."** I wrote that while `gate` was still running. It was not green. `gate` (job 45913) **failed**, and it failed *before* the ward install step ever ran. **Wrong claim 2: the unreachability is unverified.** It is now measured. `gate` died in `actions/checkout`, on three consecutive retries: ``` fatal: unable to access 'https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/': Failed to connect to forgejo.coilysiren.me port 443 after 133427 ms: Couldn't connect to server The process '/usr/bin/git' failed with exit code 128 ``` Roughly 133 seconds to a connect timeout, three times, 17:54 to 18:01. Run 4188 also logs `Could not update image ... context deadline exceeded` reaching the container registry on the same host. ## What this means **The runner cannot reliably reach `forgejo.coilysiren.me`.** That single fault explains every symptom, including the one I misattributed: * `gate` - git fetch over 443 times out. Job dies at checkout. * image pull - registry on the same host, `context deadline exceeded`. * `ward-doctor` - `resolve_release_ref`'s Python API call fails on the network, silently falls back to the literal `release`, and *by then connectivity had returned*, so curl reached the server and got a real 404 rather than a connect error. That is why this one failure looked like a URL bug instead of a network fault. It is flapping rather than flat down: a connect timeout at 17:54 and a served 404 at 17:56 are minutes apart on the same host. The host is reachable from my workstation throughout, so the fault is on the runner's path to it, or was a window that has since closed. ## What stands and what does not **Does not stand:** the framing that this is an install-ref defect blocking CI. It is a network fault. Nothing in `prod_install_ref` or `install-workflow-ward.sh` caused it, and fixing them would not have unblocked anything. **Still stands, demoted to a latent defect:** `resolve_release_ref` returning the literal `"release"` into a release-tag download path is still wrong, because that URL can never resolve and no `release` tag exists. Its real cost is diagnostic. It converted a network blip into six anonymous 404s and sent me looking at URLs for an hour when the log could have said "the API was unreachable". Worth fixing on its own merits, not as a CI unblock. ## Ownership This is a live infrastructure fault on a running backend, so it sits with the Systems Administrator seat rather than the platform seat. Not something to fix from here. #1340 remains blocked, now on two failing jobs rather than one, neither caused by its diff.

coilyco-ops referenced this issue

2026-08-28 18:11:51 +00:00

ci / ward-doctor fails on every open pull request while passing on main, blocking the merge lane #1343

coilyco-ops referenced this issue

2026-08-28 18:12:03 +00:00

feat(aterm)!: give every role bundle its own icon (#1282) #1342

coilyco-ops referenced this issue from a commit

2026-08-28 18:14:53 +00:00

chore(ci): cut Ward's runtime out of AOS CI, since the repository is archived

coilyco-ops referenced this issue

2026-08-28 18:15:12 +00:00

chore(ci): cut Ward's runtime out of AOS CI, since the repository is archived #1345

coilyco-ops referenced this issue

2026-08-28 18:15:31 +00:00

resolve_release_ref falls back to a branch name inside a release-tag URL, so an unreachable API becomes an anonymous 404 #1346

coilyco-ops commented

2026-08-28 18:15:46 +00:00

Author

Owner

Cross-links, and what has moved

Duplicate: #1343 is the same break, filed independently by Delphi from the main-vs-PR contrast that this issue lacks. Both stay open and neither is the canonical one. #1343 has the "green on main, red on every PR" framing and the timing window; this one has the job logs and the measurement.

Sysadmin has it. The measurements were sent to the Systems Administrator seat directly at Kai's request. Nothing in the repo changes for that half, and no one from the platform seat is touching runner or host networking.

Repo-side half is cut: #1345 removes Ward's runtime from AOS CI entirely - five install steps across four workflows, the ward-doctor job, the ward doctor step in promote, install-workflow-ward.sh, and the ward product in prod_install_ref. That is Kai's call under #1299 and it is not a fix for this issue. It removes five network dependencies on an archived repository that made this outage worse and much harder to read. Every PR stays red until the runner half lands.

The diagnostic defect is split out: #1346 covers resolve_release_ref returning a branch name into a release-tag path, which is what turned "the API was unreachable" into six anonymous 404s here. It survives #1345 for aos, umbra, specgen, and guard.

One thing worth keeping from this

Both Delphi and I read a pending job as passing, straight off the combined status rollup, and both wrote "everything else is green" into an issue. On run 4187 gate was still running and then failed; on run 4188 it genuinely passed. Same rollup, opposite truths.

The rollup answers "is anything red yet", not "did everything pass". For a claim about which jobs passed, read per-job state (action-run-job list on the API run id) and check that nothing is still running. That mistake is what made this look like a single-job ward problem for the first hour, when a second job was failing on the real cause the whole time.

## Cross-links, and what has moved **Duplicate:** #1343 is the same break, filed independently by Delphi from the main-vs-PR contrast that this issue lacks. Both stay open and neither is the canonical one. #1343 has the "green on main, red on every PR" framing and the timing window; this one has the job logs and the measurement. **Sysadmin has it.** The measurements were sent to the Systems Administrator seat directly at Kai's request. Nothing in the repo changes for that half, and no one from the platform seat is touching runner or host networking. **Repo-side half is cut:** #1345 removes Ward's runtime from AOS CI entirely - five install steps across four workflows, the `ward-doctor` job, the `ward doctor` step in promote, `install-workflow-ward.sh`, and the `ward` product in `prod_install_ref`. That is Kai's call under #1299 and it is **not** a fix for this issue. It removes five network dependencies on an archived repository that made this outage worse and much harder to read. Every PR stays red until the runner half lands. **The diagnostic defect is split out:** #1346 covers `resolve_release_ref` returning a branch name into a release-tag path, which is what turned "the API was unreachable" into six anonymous 404s here. It survives #1345 for aos, umbra, specgen, and guard. ## One thing worth keeping from this Both Delphi and I read a **pending** job as passing, straight off the combined status rollup, and both wrote "everything else is green" into an issue. On run 4187 `gate` was still running and then failed; on run 4188 it genuinely passed. Same rollup, opposite truths. The rollup answers "is anything red yet", not "did everything pass". For a claim about which jobs passed, read per-job state (`action-run-job list` on the API run id) and check that nothing is still `running`. That mistake is what made this look like a single-job ward problem for the first hour, when a second job was failing on the real cause the whole time.

coilyco-ops commented

2026-08-28 18:21:36 +00:00

Author

Owner

Sysadmin diagnosis: the network half

Taking the live-backend half Angie handed over. This is the network path, not the install ref.

Root cause

Runner job containers reach Forgejo over a double-NAT hairpin, and that path degrades under concurrency instead of failing cleanly.

forgejo.coilysiren.me resolves to 99.110.50.213 - measured - this is the current WAN address, confirmed against checkip.amazonaws.com, so DNS is correct and not stale
The runner daemon does not use it - measured - FORGEJO_INSTANCE_URL is http://forgejo.forgejo.svc.cluster.local, so the control-plane connection stays in-cluster. That is why runners keep claiming jobs while the jobs themselves die
The job does use it - actions/checkout clones Forgejo's advertised public ROOT_URL, so the clone leaves the cluster, transits the router, and comes back in to reach a service that never needed to be public
Jobs run under docker-in-docker with dnsPolicy: ClusterFirst, so CoreDNS is in the resolution path and a split-horizon entry does reach them

Hairpinned flows are masqueraded twice, at flannel and again at the gateway. That is the failure mode that produces three consecutive 133s connect timeouts while ward-doctor gets a clean HTTP 404 from the same host two minutes later. It flaps rather than sitting down, which matches the reported signature exactly.

Correction to my own first pass

I initially scoped this to the kai-server cluster and was wrong. Kai pointed out that most runners live on ser8, which is a separate k3s cluster with its own CoreDNS and its own Traefik. The flight-deck runners that run this repo's CI are forgejo-runner-ser8-flight-deck-0 through -5 on ser8, so a fix applied only to kai-server would have unblocked a minority of runners while appearing to resolve the incident.

That correction also rules out the target I was going to recommend. A Traefik ClusterIP is meaningful only inside its own cluster and resolves to nothing from ser8.

Proven fix target

The tailnet address is the only one that works from both clusters, and it is verified rather than assumed:

curl --resolve forgejo.coilysiren.me:443:100.69.164.66 https://forgejo.coilysiren.me/api/v1/version
http=200  connect=0.003s  ssl_verify_result=0  {"version":"16.0.2+gitea-1.22.0"}

Traefik's LoadBalancer already carries 100.69.164.66 as its external IP. SNI is preserved, so the existing certificate validates unchanged and no TLS work is needed.

The change is one coredns-custom key per cluster:

forgejo.server: |
  forgejo.coilysiren.me:53 {
      hosts {
          100.69.164.66 forgejo.coilysiren.me
      }
  }

coredns-custom is the supported k3s extension point. This matters, because the live kai-server Corefile has already lost api.coilysiren.me and eco.coilysiren.me hosts blocks that survive only in its stale last-applied-configuration annotation. That ConfigMap is owned by a k3s Addon and reverts hand-edits, so an earlier customization went into the wrong place and was silently taken back. Scope for this change is forgejo.coilysiren.me only, per Kai.

Not established

What changed at 17:42Z. The hairpin is chronically fragile, so I can explain why it breaks but not why it broke then. Ruled out so far:

Runner recycle - ser8 runners came up 09:17Z and kai-server's at 16:17Z, both before the last green run at 17:42Z
Stale DNS - the record matches the live WAN address
Node exhaustion - kai-server shows CPU 7.4 percent, memory 43.7 percent, PSI low, no active pressure
Forgejo itself - healthy, returns 200 over both the public and tailnet paths right now

Treat the trigger as open. The structural fault is real and worth fixing regardless.

Blocked on

I cannot apply the ser8 half. The guarded kubectl surface exposes only current-context and get-contexts, rejects --context, and is pinned to kai-server. ser8 is reachable read-only through its node-stats MCP but not writable through any approved verb I hold. Handing that back to Kai rather than routing around the guard.

## Sysadmin diagnosis: the network half Taking the live-backend half Angie handed over. This is the network path, not the install ref. ### Root cause Runner job containers reach Forgejo over a double-NAT hairpin, and that path degrades under concurrency instead of failing cleanly. * `forgejo.coilysiren.me` resolves to `99.110.50.213` - measured - this is the current WAN address, confirmed against `checkip.amazonaws.com`, so DNS is correct and not stale * The runner daemon does not use it - measured - `FORGEJO_INSTANCE_URL` is `http://forgejo.forgejo.svc.cluster.local`, so the control-plane connection stays in-cluster. That is why runners keep claiming jobs while the jobs themselves die * The job does use it - `actions/checkout` clones Forgejo's advertised public `ROOT_URL`, so the clone leaves the cluster, transits the router, and comes back in to reach a service that never needed to be public * Jobs run under docker-in-docker with `dnsPolicy: ClusterFirst`, so CoreDNS is in the resolution path and a split-horizon entry does reach them Hairpinned flows are masqueraded twice, at flannel and again at the gateway. That is the failure mode that produces three consecutive 133s connect timeouts while `ward-doctor` gets a clean HTTP 404 from the same host two minutes later. It flaps rather than sitting down, which matches the reported signature exactly. ### Correction to my own first pass I initially scoped this to the kai-server cluster and was wrong. Kai pointed out that most runners live on ser8, which is a separate k3s cluster with its own CoreDNS and its own Traefik. The flight-deck runners that run this repo's CI are `forgejo-runner-ser8-flight-deck-0` through `-5` on ser8, so a fix applied only to kai-server would have unblocked a minority of runners while appearing to resolve the incident. That correction also rules out the target I was going to recommend. A Traefik ClusterIP is meaningful only inside its own cluster and resolves to nothing from ser8. ### Proven fix target The tailnet address is the only one that works from both clusters, and it is verified rather than assumed: ``` curl --resolve forgejo.coilysiren.me:443:100.69.164.66 https://forgejo.coilysiren.me/api/v1/version http=200 connect=0.003s ssl_verify_result=0 {"version":"16.0.2+gitea-1.22.0"} ``` Traefik's LoadBalancer already carries `100.69.164.66` as its external IP. SNI is preserved, so the existing certificate validates unchanged and no TLS work is needed. The change is one `coredns-custom` key per cluster: ``` forgejo.server: | forgejo.coilysiren.me:53 { hosts { 100.69.164.66 forgejo.coilysiren.me } } ``` `coredns-custom` is the supported k3s extension point. This matters, because the live kai-server Corefile has already lost `api.coilysiren.me` and `eco.coilysiren.me` hosts blocks that survive only in its stale `last-applied-configuration` annotation. That ConfigMap is owned by a k3s Addon and reverts hand-edits, so an earlier customization went into the wrong place and was silently taken back. Scope for this change is `forgejo.coilysiren.me` only, per Kai. ### Not established **What changed at 17:42Z.** The hairpin is chronically fragile, so I can explain why it breaks but not why it broke then. Ruled out so far: * Runner recycle - ser8 runners came up 09:17Z and kai-server's at 16:17Z, both before the last green run at 17:42Z * Stale DNS - the record matches the live WAN address * Node exhaustion - kai-server shows CPU 7.4 percent, memory 43.7 percent, PSI low, no active pressure * Forgejo itself - healthy, returns 200 over both the public and tailnet paths right now Treat the trigger as open. The structural fault is real and worth fixing regardless. ### Blocked on I cannot apply the ser8 half. The guarded kubectl surface exposes only `current-context` and `get-contexts`, rejects `--context`, and is pinned to `kai-server`. ser8 is reachable read-only through its node-stats MCP but not writable through any approved verb I hold. Handing that back to Kai rather than routing around the guard.

coilyco-ops referenced this issue

2026-08-28 18:24:55 +00:00

kubectl guardfile declares an exec-target invariant it does not enforce: --context selects any kubeconfig cluster #1348

coilysiren referenced this issue from a commit

2026-08-28 18:26:55 +00:00

chore(ci): cut Ward's runtime out of AOS CI, since the repository is archived (#1345)

coilyco-ops commented

2026-08-28 18:32:40 +00:00

Author

Owner

Correction: the "Blocked on" section of my previous comment is wrong

The diagnosis comment above ends by claiming I cannot apply the ser8 half because aosguard ops kubectl is pinned to kai-server and rejects --context. That is false. Angie falsified it and I reproduced her result.

--context works. My invocation was malformed: I put the flag before the verb, where aosguard's own CLI parser rejects it as an unknown global flag. After the verb it passes through to kubectl.

$ aosguard ops kubectl get nodes --context ser8
ser8   Ready   control-plane   77d   v1.35.5+k3s1

I ran a single malformed command and promoted its parse error into an architectural claim about the guard, without trying the other spelling. Everything else in that comment stands. The hairpin diagnosis, the verified tailnet fix target, and the open question about the 17:42Z trigger are all unaffected. Only the blocking claim was wrong.

Nothing is blocked. The ser8 half is applicable today with aosguard ops kubectl apply -f <manifest> --context ser8.

#1348 has been rewritten accordingly. It is no longer a dependency of this issue. It now records the real finding, which Angie identified: the kubectl guardfile declares "the exec target is fixed at parse, so the caller can never substitute it," and that does not hold. exec kubectl fixes the binary, not the cluster. An unknown context does fail closed, so the reachable set is the kubeconfig's contexts rather than anywhere, but a valid wrong context is accepted silently on apply, scale, and rollout.

## Correction: the "Blocked on" section of my previous comment is wrong The [diagnosis comment above](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/1344#issuecomment-80019) ends by claiming I cannot apply the ser8 half because `aosguard ops kubectl` is pinned to `kai-server` and rejects `--context`. That is false. Angie falsified it and I reproduced her result. `--context` works. My invocation was malformed: I put the flag **before** the verb, where aosguard's own CLI parser rejects it as an unknown global flag. After the verb it passes through to kubectl. ``` $ aosguard ops kubectl get nodes --context ser8 ser8 Ready control-plane 77d v1.35.5+k3s1 ``` I ran a single malformed command and promoted its parse error into an architectural claim about the guard, without trying the other spelling. Everything else in that comment stands. The hairpin diagnosis, the verified tailnet fix target, and the open question about the 17:42Z trigger are all unaffected. Only the blocking claim was wrong. **Nothing is blocked.** The ser8 half is applicable today with `aosguard ops kubectl apply -f <manifest> --context ser8`. #1348 has been rewritten accordingly. It is no longer a dependency of this issue. It now records the real finding, which Angie identified: the kubectl guardfile declares "the exec target is fixed at parse, so the caller can never substitute it," and that does not hold. `exec kubectl` fixes the binary, not the cluster. An unknown context does fail closed, so the reachable set is the kubeconfig's contexts rather than anywhere, but a valid wrong context is accepted silently on `apply`, `scale`, and `rollout`.

coilyco-ops commented

2026-08-28 18:44:25 +00:00

Author

Owner

Applied to both clusters. Causation not established.

What landed

ser8 at 18:38Z. coredns-custom did not exist, so this created it. Server-side diff confirmed a pure create beforehand.

kai-server at 18:42Z. coredns-custom already held ser8-observability.server and carried no last-applied-configuration annotation, so I included the existing key verbatim in the manifest rather than rely on apply's merge semantics against an unrecorded object. Diff confirmed addition only.

Both now carry:

forgejo.server: |
  forgejo.coilysiren.me:53 {
      errors
      hosts {
          100.69.164.66 forgejo.coilysiren.me
      }
  }

CoreDNS restarted on both and loaded the block. ser8 serves .:53 and forgejo.coilysiren.me.:53. kai-server serves those plus the pre-existing tail09a41b.ts.net.:53, so nothing was displaced. Both pods 1/1 Running, 0 restarts, no load errors.

The change did not fix this incident

CI recovered on its own before I touched anything:

4189 - 18:15Z - success
4190, 4191, 4192 - 18:26Z to 18:28Z - success
ser8 change - 18:38Z
kai-server change - 18:42Z

All four green runs predate the first change by roughly twenty minutes. Had I applied earlier and then seen them, I would have reported a confirmed fix and been wrong.

Run 4193, a rerun of the failed 4187 on the same ref, went green at 18:40Z with all four jobs passing, gate and ward-doctor included. That is post-change evidence that the change is safe and that the previously failing jobs pass. It is not evidence the change caused anything, because the path was already working when it was applied.

What the change is worth: it removes the hairpin permanently, so the intermittent failure should not recur. The acute episode ended without intervention, which is what an intermittent double-NAT hairpin does.

Still not established

What changed at 17:42Z, and what changed back around 18:15Z. Both ends of the window are unexplained. Ruled out: the runner recycles, stale DNS, node exhaustion on kai-server, and Forgejo itself. A confound worth naming, since the green runs at 18:15Z onward were all on #1345, which cuts five install round-trips per run. Fewer hairpinned connections could mask a still-degraded path rather than prove a recovered one. Run 4193 is the counterweight, since it ran the pre-#1345 ref and passed, but it also ran after the DNS change. The two explanations are not cleanly separable from what I have.

Durability gap

Both clusters run Flux. These applies are imperative, so both are drift relative to git. Neither ConfigMap sits inside an existing Flux kustomization, so nothing will prune them, but neither is declared anywhere either. A cluster rebuild loses both silently and the hairpin returns.

The durable commit belongs in the GitOps repo. Per the estate rule that repository-proven landing in a GitOps flow is the platform seat's, that is a handoff rather than something I close here.

Health check after the change

No new failures on kai-server. The non-running pods there are 8 to 28 days old and predate this. The coilysiren-eco-app-discord backoff has 740 restarts over 2d15h and is long-standing. A sirens-deep readiness 503 appeared in the same window and I have not established whether it is related, so it is recorded here rather than dismissed.

## Applied to both clusters. Causation not established. ### What landed **ser8** at 18:38Z. `coredns-custom` did not exist, so this created it. Server-side `diff` confirmed a pure create beforehand. **kai-server** at 18:42Z. `coredns-custom` already held `ser8-observability.server` and carried no `last-applied-configuration` annotation, so I included the existing key verbatim in the manifest rather than rely on apply's merge semantics against an unrecorded object. Diff confirmed addition only. Both now carry: ``` forgejo.server: | forgejo.coilysiren.me:53 { errors hosts { 100.69.164.66 forgejo.coilysiren.me } } ``` CoreDNS restarted on both and loaded the block. ser8 serves `.:53` and `forgejo.coilysiren.me.:53`. kai-server serves those plus the pre-existing `tail09a41b.ts.net.:53`, so nothing was displaced. Both pods 1/1 Running, 0 restarts, no load errors. ### The change did not fix this incident CI recovered on its own before I touched anything: * 4189 - 18:15Z - success * 4190, 4191, 4192 - 18:26Z to 18:28Z - success * ser8 change - 18:38Z * kai-server change - 18:42Z All four green runs predate the first change by roughly twenty minutes. Had I applied earlier and then seen them, I would have reported a confirmed fix and been wrong. Run 4193, a rerun of the failed 4187 on the same ref, went green at 18:40Z with all four jobs passing, `gate` and `ward-doctor` included. That is post-change evidence that the change is safe and that the previously failing jobs pass. It is not evidence the change caused anything, because the path was already working when it was applied. What the change is worth: it removes the hairpin permanently, so the intermittent failure should not recur. The acute episode ended without intervention, which is what an intermittent double-NAT hairpin does. ### Still not established **What changed at 17:42Z, and what changed back around 18:15Z.** Both ends of the window are unexplained. Ruled out: the runner recycles, stale DNS, node exhaustion on kai-server, and Forgejo itself. A confound worth naming, since the green runs at 18:15Z onward were all on #1345, which cuts five install round-trips per run. Fewer hairpinned connections could mask a still-degraded path rather than prove a recovered one. Run 4193 is the counterweight, since it ran the pre-#1345 ref and passed, but it also ran after the DNS change. The two explanations are not cleanly separable from what I have. ### Durability gap Both clusters run Flux. These applies are imperative, so both are drift relative to git. Neither ConfigMap sits inside an existing Flux kustomization, so nothing will prune them, but neither is declared anywhere either. A cluster rebuild loses both silently and the hairpin returns. The durable commit belongs in the GitOps repo. Per the estate rule that repository-proven landing in a GitOps flow is the platform seat's, that is a handoff rather than something I close here. ### Health check after the change No new failures on kai-server. The non-running pods there are 8 to 28 days old and predate this. The `coilysiren-eco-app-discord` backoff has 740 restarts over 2d15h and is long-standing. A `sirens-deep` readiness 503 appeared in the same window and I have not established whether it is related, so it is recorded here rather than dismissed.

coilyco-ops referenced this issue

2026-08-28 18:46:43 +00:00

forgejo-storage reaches kubectl outside the kubectl allowlist, and runs against whatever context is active #1350

coilyco-ops commented

2026-08-28 19:00:37 +00:00

Author

Owner

Correction: the manifest in comment 80048 is incomplete for kai-server

Angie caught this while writing the GitOps commit, and it is worth fixing in place rather than leaving for the next reader.

Comment 80048 says "Both now carry" above a fenced block containing only forgejo.server. The prose immediately above it does say I included the pre-existing key verbatim, but the fenced block is the part someone copies. Anyone writing a declarative manifest from that block alone produces a one-key ConfigMap for kai-server. Flux then takes ownership and deletes ser8-observability.server on the next reconcile, removing a working tailnet forwarder.

The block was accurate about what I added and misleading about what the object contains. Those are not the same thing, and on a shared object only the second one is safe to act on.

Complete content, per cluster

ser8 - one key, this is the whole object:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  forgejo.server: |
    forgejo.coilysiren.me:53 {
        errors
        hosts {
            100.69.164.66 forgejo.coilysiren.me
        }
    }

kai-server - two keys. ser8-observability.server predates this work, is unrelated to the hairpin, and must survive:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  ser8-observability.server: |
    tail09a41b.ts.net:53 {
        errors
        cache 30
        forward . 100.100.100.100
    }
  forgejo.server: |
    forgejo.coilysiren.me:53 {
        errors
        hosts {
            100.69.164.66 forgejo.coilysiren.me
        }
    }

Live state confirmed after Flux adoption

Checked directly rather than assumed, since coilyco-flight-deck/infrastructure#976 adopts objects in both clusters:

kai-server coredns-custom keys - forgejo.server, ser8-observability.server
ser8 coredns-custom keys - forgejo.server
kai-server CoreDNS serving - .:53, forgejo.coilysiren.me.:53, tail09a41b.ts.net.:53
ser8 CoreDNS serving - .:53, forgejo.coilysiren.me.:53
Both pods 1/1 Running, 0 restarts

Adoption was a no-op on content, matching what Angie measured with kubectl diff -k against both clusters.

The general shape

I was careful about apply's merge semantics against an object with no last-applied-configuration annotation, and I wrote up the fix without carrying that same caution into the record. The imperative hazard and the declarative hazard are the same underlying fact, that this object has more than one owner's content in it, and I only guarded one of them. Worth remembering for any shared ConfigMap: a record that shows what changed is not sufficient for someone who has to declare what exists.

## Correction: the manifest in comment 80048 is incomplete for kai-server Angie caught this while writing the GitOps commit, and it is worth fixing in place rather than leaving for the next reader. [Comment 80048](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/1344#issuecomment-80048) says "Both now carry" above a fenced block containing only `forgejo.server`. The prose immediately above it does say I included the pre-existing key verbatim, but the fenced block is the part someone copies. **Anyone writing a declarative manifest from that block alone produces a one-key ConfigMap for kai-server. Flux then takes ownership and deletes `ser8-observability.server` on the next reconcile, removing a working tailnet forwarder.** The block was accurate about what I added and misleading about what the object contains. Those are not the same thing, and on a shared object only the second one is safe to act on. ### Complete content, per cluster **ser8** - one key, this is the whole object: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: coredns-custom namespace: kube-system data: forgejo.server: | forgejo.coilysiren.me:53 { errors hosts { 100.69.164.66 forgejo.coilysiren.me } } ``` **kai-server** - two keys. `ser8-observability.server` predates this work, is unrelated to the hairpin, and must survive: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: coredns-custom namespace: kube-system data: ser8-observability.server: | tail09a41b.ts.net:53 { errors cache 30 forward . 100.100.100.100 } forgejo.server: | forgejo.coilysiren.me:53 { errors hosts { 100.69.164.66 forgejo.coilysiren.me } } ``` ### Live state confirmed after Flux adoption Checked directly rather than assumed, since coilyco-flight-deck/infrastructure#976 adopts objects in both clusters: * kai-server `coredns-custom` keys - `forgejo.server`, `ser8-observability.server` * ser8 `coredns-custom` keys - `forgejo.server` * kai-server CoreDNS serving - `.:53`, `forgejo.coilysiren.me.:53`, `tail09a41b.ts.net.:53` * ser8 CoreDNS serving - `.:53`, `forgejo.coilysiren.me.:53` * Both pods 1/1 Running, 0 restarts Adoption was a no-op on content, matching what Angie measured with `kubectl diff -k` against both clusters. ### The general shape I was careful about apply's merge semantics against an object with no `last-applied-configuration` annotation, and I wrote up the fix without carrying that same caution into the record. The imperative hazard and the declarative hazard are the same underlying fact, that this object has more than one owner's content in it, and I only guarded one of them. Worth remembering for any shared ConfigMap: a record that shows what changed is not sufficient for someone who has to declare what exists.

coilyco-ops referenced this issue

2026-08-28 19:07:23 +00:00

Add an aosguard ops netlify wrap so agents can manage site domain aliases without holding the token #1353

coilyco-ops referenced this issue

2026-08-28 19:56:50 +00:00

feat(aosguard): a guarded Netlify domain-alias surface, so the token stays inside #1355

coilyco-ops commented

2026-08-28 20:02:17 +00:00

Author

Owner

Correction: ser8 runners were hairpinning their control plane too

My diagnosis said the runner daemon was safe and only the job hairpinned:

The runner daemon does not use it - measured - FORGEJO_INSTANCE_URL is http://forgejo.forgejo.svc.cluster.local, so the control-plane connection stays in-cluster. That is why runners keep claiming jobs while the jobs themselves die

That was measured on forgejo-runner-deploy-scoped, a kai-server deploy runner. It does not hold for the runners that actually run this repo's CI. On ser8:

$ kubectl get statefulset -n forgejo forgejo-runner-ser8-flight-deck -o yaml --context ser8
  - name: FORGEJO_INSTANCE_URL
    value: https://forgejo.coilysiren.me/

The public hostname. So on ser8 both the daemon control plane and the job checkout crossed the hairpin, not just the checkout. The fault surface was larger than I described, and my explanation for why runners kept claiming jobs while jobs died was wrong for the cluster it mattered on.

Same root error as before, one cluster over: I measured on the cluster I could see and generalised to the fleet. That is twice on the same axis in one incident.

It does not change the fix or the target. Both paths resolve the same name, so the override covers both.

Why the ingress logs cannot verify this, and what does

Traefik access logging is enabled and detailed, but it cannot discriminate origin. externalTrafficPolicy is Local, yet every request arrives with ClientHost: 10.42.0.22, which is svclb-traefik-bcc27bfa-474s5, the klipper-lb pod. Requests from an external crawler and from a runner are indistinguishable in the log because klipper-lb SNATs all of them. Worth recording so nobody else spends time there expecting client IPs.

The verification that does hold is structural, and each link is measured:

ser8 CoreDNS serves forgejo.coilysiren.me.:53 as its own server block with a hosts stanza and no fallthrough, so it is authoritative for that name and 100.69.164.66 is the only answer it can return. There is no path by which a ser8 pod resolves the public address any more
Job containers reach that resolver. dockerd runs with no --dns, dnsPolicy: ClusterFirst, dnsConfig unset, hostNetwork unset, so DinD inherits the pod resolver
Therefore every job container created after 18:38Z resolved to the tailnet address

CI traffic has been flowing over that path since. Runs 4197 through 4205 all succeeded between 18:51Z and 18:55Z, and 4206 started 19:56Z. Each of those cleared actions/checkout, which is the step that was failing, on a resolution path that can only produce the tailnet IP.

That is a stronger claim than I could make earlier and it is still not a claim about the original incident. It establishes that the in-cluster path carries real CI traffic successfully. It does not establish that the hairpin caused the 17:42Z failures, because CI recovered on its own at 18:15Z before any of this was applied. Both statements can be true at once and only the first one is proven.

## Correction: ser8 runners were hairpinning their control plane too My diagnosis said the runner daemon was safe and only the job hairpinned: > The runner daemon does not use it - measured - `FORGEJO_INSTANCE_URL` is `http://forgejo.forgejo.svc.cluster.local`, so the control-plane connection stays in-cluster. That is why runners keep claiming jobs while the jobs themselves die That was measured on `forgejo-runner-deploy-scoped`, a **kai-server** deploy runner. It does not hold for the runners that actually run this repo's CI. On ser8: ``` $ kubectl get statefulset -n forgejo forgejo-runner-ser8-flight-deck -o yaml --context ser8 - name: FORGEJO_INSTANCE_URL value: https://forgejo.coilysiren.me/ ``` The public hostname. So on ser8 both the daemon control plane and the job checkout crossed the hairpin, not just the checkout. The fault surface was larger than I described, and my explanation for why runners kept claiming jobs while jobs died was wrong for the cluster it mattered on. Same root error as before, one cluster over: I measured on the cluster I could see and generalised to the fleet. That is twice on the same axis in one incident. It does not change the fix or the target. Both paths resolve the same name, so the override covers both. ### Why the ingress logs cannot verify this, and what does Traefik access logging is enabled and detailed, but it cannot discriminate origin. `externalTrafficPolicy` is `Local`, yet every request arrives with `ClientHost: 10.42.0.22`, which is `svclb-traefik-bcc27bfa-474s5`, the klipper-lb pod. Requests from an external crawler and from a runner are indistinguishable in the log because klipper-lb SNATs all of them. Worth recording so nobody else spends time there expecting client IPs. The verification that does hold is structural, and each link is measured: * ser8 CoreDNS serves `forgejo.coilysiren.me.:53` as its own server block with a `hosts` stanza and **no `fallthrough`**, so it is authoritative for that name and `100.69.164.66` is the only answer it can return. There is no path by which a ser8 pod resolves the public address any more * Job containers reach that resolver. `dockerd` runs with no `--dns`, `dnsPolicy: ClusterFirst`, `dnsConfig` unset, `hostNetwork` unset, so DinD inherits the pod resolver * Therefore every job container created after 18:38Z resolved to the tailnet address CI traffic has been flowing over that path since. Runs 4197 through 4205 all succeeded between 18:51Z and 18:55Z, and 4206 started 19:56Z. Each of those cleared `actions/checkout`, which is the step that was failing, on a resolution path that can only produce the tailnet IP. That is a stronger claim than I could make earlier and it is still not a claim about the original incident. It establishes that the in-cluster path carries real CI traffic successfully. It does not establish that the hairpin caused the 17:42Z failures, because CI recovered on its own at 18:15Z before any of this was applied. Both statements can be true at once and only the first one is proven.

coilyco-ops referenced this issue from coilyco-flight-deck/infrastructure

2026-08-28 21:11:48 +00:00

Forgejo Actions runners hairpin through the public edge, 59% of all ingress traffic #912

coilyco-ops commented

2026-08-28 22:43:37 +00:00

Author

Owner

Resolved on both halves. Closing.

CI is green and has been for hours. Ten consecutive successful runs, 4235 through 4244, spanning 21:52Z to 22:33Z:

4244 success  22:33:36Z    4239 success  21:54:42Z
4243 success  22:32:18Z    4238 success  21:54:42Z
4242 success  22:31:36Z    4237 success  21:54:34Z
4241 success  22:31:02Z    4236 success  21:54:06Z
4240 success  21:57:32Z    4235 success  21:52:08Z

ward-doctor no longer appears in the task list at all. #1345 cut Ward's runtime out of AOS CI, so the job that this issue is named after has been removed rather than fixed. That is the durable resolution of the reported symptom: an install step that made five network round-trips to an archived repository on every run does not exist any more.

The two halves, and which did what

Repo half, Angie. #1345 removed the Ward install steps. That eliminates the 404-on-a-release-tag-path failure mode permanently, and it is why ward-doctor is absent above rather than green.

Network half, mine. Split-horizon DNS on both clusters so forgejo.coilysiren.me resolves in-cluster to the tailnet address instead of hairpinning through the router. Landed declaratively in coilyco-flight-deck/infrastructure#976 and closed coilyco-flight-deck/infrastructure#912.

What I am still not claiming

The DNS change did not end this incident. CI recovered on its own at 18:15Z, roughly twenty minutes before the first apply, and I never explained either end of that window: not the 17:42Z break, not the recovery. The fix removes the structural fragility. It is not the reason that day's outage stopped.

That distinction survives into this close deliberately. If checkout starts timing out again, the hairpin is no longer the explanation, and the 17:42Z trigger remains unidentified.

Corrections recorded on this issue, worth carrying forward

Three of my own claims here were wrong and are corrected in comments above rather than silently left:

I reported the runner daemon used an in-cluster URL and only the job hairpinned. True for kai-server's deploy runners, false for the ser8 flight-deck runners that run this repo's CI, which used the public hostname for both
I reported ser8 as unreachable through the guarded kubectl. It was a flag-position error in a single malformed command, not an architectural limit
The manifest in comment 80048 is incomplete for kai-server and would delete a working tailnet forwarder if copied literally. Complete per-cluster manifests are in comment 80069

#1343 - Delphi's filing of the same break from the install-ref angle. resolve_release_ref returning the literal string "release" on any exception remains a real latent defect worth fixing on its own merit, independently of this
coilyco-flight-deck/infrastructure#304 - its revisit trigger, "complete the Ward deprovisioning sweep before a stale entry causes a red repository", fired here. Recording that explicitly so the trigger is not read as still ahead

## Resolved on both halves. Closing. CI is green and has been for hours. Ten consecutive successful runs, 4235 through 4244, spanning 21:52Z to 22:33Z: ``` 4244 success 22:33:36Z 4239 success 21:54:42Z 4243 success 22:32:18Z 4238 success 21:54:42Z 4242 success 22:31:36Z 4237 success 21:54:34Z 4241 success 22:31:02Z 4236 success 21:54:06Z 4240 success 21:57:32Z 4235 success 21:52:08Z ``` **`ward-doctor` no longer appears in the task list at all.** #1345 cut Ward's runtime out of AOS CI, so the job that this issue is named after has been removed rather than fixed. That is the durable resolution of the reported symptom: an install step that made five network round-trips to an archived repository on every run does not exist any more. ### The two halves, and which did what **Repo half, Angie.** #1345 removed the Ward install steps. That eliminates the 404-on-a-release-tag-path failure mode permanently, and it is why `ward-doctor` is absent above rather than green. **Network half, mine.** Split-horizon DNS on both clusters so `forgejo.coilysiren.me` resolves in-cluster to the tailnet address instead of hairpinning through the router. Landed declaratively in coilyco-flight-deck/infrastructure#976 and closed coilyco-flight-deck/infrastructure#912. ### What I am still not claiming The DNS change did not end this incident. CI recovered on its own at 18:15Z, roughly twenty minutes before the first apply, and I never explained either end of that window: not the 17:42Z break, not the recovery. The fix removes the structural fragility. It is not the reason that day's outage stopped. That distinction survives into this close deliberately. If checkout starts timing out again, the hairpin is no longer the explanation, and the 17:42Z trigger remains unidentified. ### Corrections recorded on this issue, worth carrying forward Three of my own claims here were wrong and are corrected in comments above rather than silently left: * I reported the runner daemon used an in-cluster URL and only the job hairpinned. True for kai-server's deploy runners, false for the ser8 flight-deck runners that run this repo's CI, which used the public hostname for both * I reported ser8 as unreachable through the guarded kubectl. It was a flag-position error in a single malformed command, not an architectural limit * The manifest in comment 80048 is incomplete for kai-server and would delete a working tailnet forwarder if copied literally. Complete per-cluster manifests are in comment 80069 ### Related * #1343 - Delphi's filing of the same break from the install-ref angle. `resolve_release_ref` returning the literal string `"release"` on any exception remains a real latent defect worth fixing on its own merit, independently of this * coilyco-flight-deck/infrastructure#304 - its revisit trigger, "complete the Ward deprovisioning sweep before a stale entry causes a red repository", fired here. Recording that explicitly so the trigger is not read as still ahead

coilyco-ops closed this issue

2026-08-28 22:43:42 +00:00

coilyco-ops referenced this issue from coilyco-flight-deck/infrastructure

2026-08-29 00:15:57 +00:00

Unexplained: CI lost its path to Forgejo at 17:42Z on 2026-08-28 and recovered on its own at 18:15Z #982