coilyco-gaming/sirens-echo

Watch

The docker probe cannot explain the failure it is attached to #787

New issue

Closed

opened 2026-08-14 22:59:03 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-14 22:59:03 +00:00

Member

scripts/ci-docker-probe.sh runs only when image-build fails, and exists to explain that failure. Two things stop it doing so.

It calls a command the image does not have

Line 17 is ip route 2>&1 || cat /proc/net/route 2>&1 || true. ip is not in agentic-os:release, so every run prints:

scripts/ci-docker-probe.sh: line 17: ip: command not found

The fallback fires and the reader gets the raw table instead:

Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT
eth0	00000000	010012AC	0003	0	0	0	00000000	0	0	0

That gateway is little-endian hex. A reader has to decode 010012AC to 172.18.0.1 by hand, which is exactly the arithmetic default_gateway in scripts/ci-image-build.sh already does.

It does not test the address the build actually uses

The probe's candidate list is hardcoded:

"${DOCKER_HOST:-}" tcp://localhost:2375 tcp://172.17.0.1:2375 tcp://host.docker.internal:2375 unix:///var/run/docker.sock

ci-image-build.sh inserts the derived bridge gateway ahead of the hardcoded 172.17.0.1. The probe never tests that derived address. Run 941 built against tcp://172.18.0.1:2375, and the probe would not have tested it. When daemon resolution is the real fault, the probe can report every address it knows as unreachable while the one the build chose is fine, or the reverse.

Evidence

Run 945 job 1 attempt 2, the probe step:

-- trying tcp://localhost:2375
Cannot connect to the Docker daemon at tcp://localhost:2375.
unreachable
-- trying tcp://172.17.0.1:2375
28.5.2
-- trying tcp://host.docker.internal:2375
... no such host
unreachable

The daemon was reachable and the build proceeded to step 26 of 40. Nothing here was a daemon fault, but the report reads like a partial outage.

Worth deciding

Both scripts derive or guess the same thing separately. Sharing default_gateway between them, or having the probe print the resolved host ci-image-build.sh settled on, would make the report answer the question it is attached to. Dropping the ip call costs nothing since the fallback is already the path that runs.

The header calls the script temporary, for #91. If that is still the plan, closing it out may be better than fixing it.

`scripts/ci-docker-probe.sh` runs only when `image-build` fails, and exists to explain that failure. Two things stop it doing so. ## It calls a command the image does not have Line 17 is `ip route 2>&1 || cat /proc/net/route 2>&1 || true`. `ip` is not in `agentic-os:release`, so every run prints: ``` scripts/ci-docker-probe.sh: line 17: ip: command not found ``` The fallback fires and the reader gets the raw table instead: ``` Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT eth0 00000000 010012AC 0003 0 0 0 00000000 0 0 0 ``` That gateway is little-endian hex. A reader has to decode `010012AC` to `172.18.0.1` by hand, which is exactly the arithmetic `default_gateway` in `scripts/ci-image-build.sh` already does. ## It does not test the address the build actually uses The probe's candidate list is hardcoded: ``` "${DOCKER_HOST:-}" tcp://localhost:2375 tcp://172.17.0.1:2375 tcp://host.docker.internal:2375 unix:///var/run/docker.sock ``` `ci-image-build.sh` inserts the derived bridge gateway ahead of the hardcoded `172.17.0.1`. The probe never tests that derived address. Run 941 built against `tcp://172.18.0.1:2375`, and the probe would not have tested it. When daemon resolution is the real fault, the probe can report every address it knows as unreachable while the one the build chose is fine, or the reverse. ## Evidence Run 945 job 1 attempt 2, the probe step: ``` -- trying tcp://localhost:2375 Cannot connect to the Docker daemon at tcp://localhost:2375. unreachable -- trying tcp://172.17.0.1:2375 28.5.2 -- trying tcp://host.docker.internal:2375 ... no such host unreachable ``` The daemon was reachable and the build proceeded to step 26 of 40. Nothing here was a daemon fault, but the report reads like a partial outage. ## Worth deciding Both scripts derive or guess the same thing separately. Sharing `default_gateway` between them, or having the probe print the resolved host `ci-image-build.sh` settled on, would make the report answer the question it is attached to. Dropping the `ip` call costs nothing since the fallback is already the path that runs. The header calls the script temporary, for #91. If that is still the plan, closing it out may be better than fixing it.

coilyco-ops added the

priority

autonomy

headless

labels

2026-08-15 15:17:29 +00:00

coilyco-ops referenced this issue from a commit

2026-08-15 15:27:48 +00:00

fix(ci): make the docker probe report on the daemon the build chose

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-15 15:28:20 +00:00

Make the docker probe report on the daemon the build chose #798

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-08-15 15:35:54 +00:00

aosguard ops forgejo action-job logs JSON-decodes a plaintext body, so CI logs are unreachable through the guarded surface #1044

coilysiren closed this issue

2026-08-15 15:36:20 +00:00

coilyco-ops commented

2026-08-15 15:36:23 +00:00

Author

Member

Fixed in #798, awaiting merge.

Both defects, plus the one underneath them:

The ip call is gone. The probe prints the decoded gateway instead of a command-not-found line followed by the raw table.
One candidate list. New scripts/lib/docker-host.sh holds default_gateway and docker_host_candidates, read by both scripts. The probe can no longer skip the derived gateway the build uses, nor spend lines on localhost and host.docker.internal, which the build never tries.
The misleading report. The build records the address it resolved, and the probe leads with it. On the run 945 shape the first two lines now read "the build reached a daemon at ..., so resolution succeeded and the fault is later in the build."

On closing it out instead

Kept it. #91 is closed, so "temporary" no longer holds either way, but the probe is still the only diagnostic image-build has when the daemon is genuinely unreachable, and the bridge has renumbered under it once already. The header stops calling it temporary, and the reasoning moved to docs/sirens-echo-image-build.md.

What is verified, and what is not

Verified: both resolution paths with a stubbed docker, candidate ordering, the little-endian decode against a fixture route table, pre-commit run --all-files, ward exec test-skips, and CI run 990 green on the real runner with image-build passing.

Not verified: the address ci-image-build.sh actually chose on that run. #91's acceptance asks for the first run to be checked against real logs, and that check is currently unsatisfiable. Both aosguard ops forgejo action-job logs and action-run logs fail on every call, JSON-decoding a plaintext and a ZIP body respectively. Tracked at coilyco-flight-deck/agentic-os#1044, where I added the ZIP case and this context.

The probe's own on-runner behaviour is also unexercised, because it only runs when image-build fails and image-build passed. The next genuine failure is its first real outing.

Fixed in #798, awaiting merge. Both defects, plus the one underneath them: * **The `ip` call is gone.** The probe prints the decoded gateway instead of a command-not-found line followed by the raw table. * **One candidate list.** New `scripts/lib/docker-host.sh` holds `default_gateway` and `docker_host_candidates`, read by both scripts. The probe can no longer skip the derived gateway the build uses, nor spend lines on `localhost` and `host.docker.internal`, which the build never tries. * **The misleading report.** The build records the address it resolved, and the probe leads with it. On the run 945 shape the first two lines now read "the build reached a daemon at ..., so resolution succeeded and the fault is later in the build." ## On closing it out instead Kept it. #91 is closed, so "temporary" no longer holds either way, but the probe is still the only diagnostic `image-build` has when the daemon is genuinely unreachable, and the bridge has renumbered under it once already. The header stops calling it temporary, and the reasoning moved to `docs/sirens-echo-image-build.md`. ## What is verified, and what is not Verified: both resolution paths with a stubbed `docker`, candidate ordering, the little-endian decode against a fixture route table, `pre-commit run --all-files`, `ward exec test-skips`, and CI run 990 green on the real runner with `image-build` passing. Not verified: the address `ci-image-build.sh` actually chose on that run. #91's acceptance asks for the first run to be checked against real logs, and that check is currently unsatisfiable. Both `aosguard ops forgejo action-job logs` and `action-run logs` fail on every call, JSON-decoding a plaintext and a ZIP body respectively. Tracked at coilyco-flight-deck/agentic-os#1044, where I added the ZIP case and this context. The probe's own on-runner behaviour is also unexercised, because it only runs when `image-build` fails and `image-build` passed. The next genuine failure is its first real outing.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#787

No description provided.

Rows
Columns