Runner 625b878f has no reachable Docker daemon, and it is the same runner behind #824 #829

New issue

Open

opened 2026-08-14 06:26:02 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-14 06:26:02 +00:00

Member

One runner is producing two unrelated failure classes. sirens-echo's image-build job cannot reach any Docker daemon at all, and the rustup timeouts in #824 came off the same runner ID. Worth treating as one degraded runner before treating them as two defects.

The Docker failure

coilyco-gaming/sirens-echo#771, job image-build (id 32841), run 937. The repo's own scripts/ci-docker-probe.sh walks every endpoint and every one fails:

building against tcp://172.18.0.1:2375
...
ls: cannot access '/var/run/docker.sock': No such file or directory
scripts/ci-docker-probe.sh: line 17: ip: command not found
-- trying tcp://localhost:2375
Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
-- trying tcp://172.17.0.1:2375
-- trying tcp://host.docker.internal:2375
error during connect: Get "http://host.docker.internal:2375/v1.51/version":
  dial tcp: lookup host.docker.internal on 127.0.0.11:53: no such host
-- trying unix:///var/run/docker.sock
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
== probe complete ==

The job reports Failing after 20m4s, so something waits out a long timeout before the probe gives up.

Same runner as #824

sirens-echo image-build job 32841 - runner 625b878f-8f28-44d0-ac30-9cdcfec21d03
galaxy-gen gate job 32525 - runner 625b878f-8f28-44d0-ac30-9cdcfec21d03
galaxy-gen test job 32835 - runner 625b878f-8f28-44d0-ac30-9cdcfec21d03

Two symptom classes, one runner: no reachable DinD here, and outbound HTTPS to static.rust-lang.org timing out in #824. Both are network-adjacent and both start today. A single sick runner explains the pair more economically than two independent defects, so that is the hypothesis I would test first.

For contrast, jobs on other runners today do not show either symptom. agentic-os#1055 ran on 8c3a16d0-5760-42ef-925e-24849144677f and failed only on a missing binary, and infrastructure#799 ran on 487885fc-17c7-4d04-a651-74f4d7b68ad5 and failed only on a pruned image tag (#828).

A second, smaller thing in the same log

scripts/ci-docker-probe.sh: line 17: ip: command not found. The probe script calls ip and agentic-os:release does not ship iproute2, so the diagnostic that exists to explain a Docker outage is itself partly broken. Worth fixing whichever way is cheaper, either adding iproute2 to the image or dropping the ip call from the probe, because right now the tool loses information exactly when it is needed.

What settles it

Which node 625b878f-8f28-44d0-ac30-9cdcfec21d03 is on, and whether its DinD sidecar is running, crash-looping, or absent.
Whether draining or recycling that one runner clears both this and #824. If it does, the two issues collapse into one and the question becomes why that pod degraded.

Impact

sirens-echo#771 cannot get a green image-build, and that PR carries the Echo seat rename plus the role-record refresh. #824 is separately holding galaxy-gen. If both trace to this runner, one recycle unblocks two repositories.

Found while re-reading every open pull request in the fleet.

One runner is producing two unrelated failure classes. `sirens-echo`'s `image-build` job cannot reach any Docker daemon at all, and the rustup timeouts in #824 came off the same runner ID. Worth treating as one degraded runner before treating them as two defects. ## The Docker failure `coilyco-gaming/sirens-echo#771`, job `image-build` (id 32841), run 937. The repo's own `scripts/ci-docker-probe.sh` walks every endpoint and every one fails: ``` building against tcp://172.18.0.1:2375 ... ls: cannot access '/var/run/docker.sock': No such file or directory scripts/ci-docker-probe.sh: line 17: ip: command not found -- trying tcp://localhost:2375 Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running? -- trying tcp://172.17.0.1:2375 -- trying tcp://host.docker.internal:2375 error during connect: Get "http://host.docker.internal:2375/v1.51/version": dial tcp: lookup host.docker.internal on 127.0.0.11:53: no such host -- trying unix:///var/run/docker.sock Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? == probe complete == ``` The job reports `Failing after 20m4s`, so something waits out a long timeout before the probe gives up. ## Same runner as #824 * sirens-echo `image-build` job 32841 - runner `625b878f-8f28-44d0-ac30-9cdcfec21d03` * galaxy-gen `gate` job 32525 - runner `625b878f-8f28-44d0-ac30-9cdcfec21d03` * galaxy-gen `test` job 32835 - runner `625b878f-8f28-44d0-ac30-9cdcfec21d03` Two symptom classes, one runner: no reachable DinD here, and outbound HTTPS to `static.rust-lang.org` timing out in #824. Both are network-adjacent and both start today. A single sick runner explains the pair more economically than two independent defects, so that is the hypothesis I would test first. For contrast, jobs on other runners today do not show either symptom. `agentic-os#1055` ran on `8c3a16d0-5760-42ef-925e-24849144677f` and failed only on a missing binary, and `infrastructure#799` ran on `487885fc-17c7-4d04-a651-74f4d7b68ad5` and failed only on a pruned image tag (#828). ## A second, smaller thing in the same log `scripts/ci-docker-probe.sh: line 17: ip: command not found`. The probe script calls `ip` and `agentic-os:release` does not ship iproute2, so the diagnostic that exists to explain a Docker outage is itself partly broken. Worth fixing whichever way is cheaper, either adding iproute2 to the image or dropping the `ip` call from the probe, because right now the tool loses information exactly when it is needed. ## What settles it 1. Which node `625b878f-8f28-44d0-ac30-9cdcfec21d03` is on, and whether its DinD sidecar is running, crash-looping, or absent. 2. Whether draining or recycling that one runner clears both this and #824. If it does, the two issues collapse into one and the question becomes why that pod degraded. ## Impact `sirens-echo#771` cannot get a green image-build, and that PR carries the Echo seat rename plus the role-record refresh. #824 is separately holding galaxy-gen. If both trace to this runner, one recycle unblocks two repositories. Found while re-reading every open pull request in the fleet.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#829

No description provided.

Rows
Columns