coilyco-gaming/sirens-echo

Watch

ci: build the image on pull requests, not only after a merge to main #91

New issue

Closed

opened 2026-08-10 23:38:06 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-10 23:38:06 +00:00

Member

Problem

publish-echo-image is gated on github.event_name == 'push' && github.ref == 'refs/heads/main', so nothing exercises the Docker build context on a pull request. Any build-context breakage is invisible until after the merge that causes it.

This is not hypothetical. #82 added a docs/ read to sirens-echo-policy-check, which the Dockerfile runs during the build stage, while .dockerignore excluded docs. The test job passed because it runs in a full checkout. The image build failed only after the merge, and b906d83 has no published artifact. #85 fixes the specific bug and adds a Go-level guard, but the structural gap is still open.

Runner findings

A build-only job needs a Docker daemon, not a registry credential, so it does not belong on the publisher runner.

forgejo-runner.yml backs runs-on: docker, already serves the test job, and has a privileged dind sidecar plus DOCKER_HOST.
forgejo-runner-build.yml offers a docker-build label but is scoped org/coilyco-flight-deck. This repo is in coilyco-gaming, so it is unreachable from here.
deploy comes from forgejo-runner-publisher.yml, repo-scoped to this repo and holding REGISTRY_TOKEN.

What was tried

An image-build job on runs-on: docker with container: agentic-os:release, running docker build with no push. It failed on its first run (run 86, task 26180) and was removed from #85 rather than debugged blind, because it was blocking a fix for a red main.

Why it could not be diagnosed

This Forgejo instance predates the Actions log endpoints. Both return 404:

GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs
GET /repos/{owner}/{repo}/actions/runs/{run_id}/logs

Job logs are readable in the web UI but not through the API, so an agent cannot self-diagnose a CI failure in this repo. That is worth solving on its own.

Leading hypothesis

The job declared a container:, so the step ran inside that container and never saw the runner process's DOCKER_HOST pointing at the dind sidecar. Candidate fixes, in order of preference:

Drop container: from the job so steps run where the daemon is reachable.
Set DOCKER_HOST explicitly in the job env to match the runner manifest.
Add a coilyco-gaming-scoped overlay of forgejo-runner-build.yml in infrastructure, giving this org a docker-build runner.

Acceptance

A pull request that breaks the Docker build context fails on that pull request.
No change to the publisher runner's exposure, and no registry credential in the new job.
The first run of the new job is checked against real logs before it is relied on.

## Problem `publish-echo-image` is gated on `github.event_name == 'push' && github.ref == 'refs/heads/main'`, so nothing exercises the Docker build context on a pull request. Any build-context breakage is invisible until after the merge that causes it. This is not hypothetical. #82 added a `docs/` read to `sirens-echo-policy-check`, which the Dockerfile runs during the build stage, while `.dockerignore` excluded `docs`. The `test` job passed because it runs in a full checkout. The image build failed only after the merge, and `b906d83` has no published artifact. #85 fixes the specific bug and adds a Go-level guard, but the structural gap is still open. ## Runner findings A build-only job needs a Docker daemon, not a registry credential, so it does not belong on the publisher runner. - `forgejo-runner.yml` backs `runs-on: docker`, already serves the `test` job, and has a privileged dind sidecar plus `DOCKER_HOST`. - `forgejo-runner-build.yml` offers a `docker-build` label but is scoped `org/coilyco-flight-deck`. This repo is in `coilyco-gaming`, so it is unreachable from here. - `deploy` comes from `forgejo-runner-publisher.yml`, repo-scoped to this repo and holding `REGISTRY_TOKEN`. ## What was tried An `image-build` job on `runs-on: docker` with `container: agentic-os:release`, running `docker build` with no push. It failed on its first run (run 86, task 26180) and was removed from #85 rather than debugged blind, because it was blocking a fix for a red `main`. ## Why it could not be diagnosed This Forgejo instance predates the Actions log endpoints. Both return 404: - `GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs` - `GET /repos/{owner}/{repo}/actions/runs/{run_id}/logs` Job logs are readable in the web UI but not through the API, so an agent cannot self-diagnose a CI failure in this repo. That is worth solving on its own. ## Leading hypothesis The job declared a `container:`, so the step ran inside that container and never saw the runner process's `DOCKER_HOST` pointing at the dind sidecar. Candidate fixes, in order of preference: 1. Drop `container:` from the job so steps run where the daemon is reachable. 2. Set `DOCKER_HOST` explicitly in the job env to match the runner manifest. 3. Add a `coilyco-gaming`-scoped overlay of `forgejo-runner-build.yml` in `infrastructure`, giving this org a `docker-build` runner. ## Acceptance - A pull request that breaks the Docker build context fails on that pull request. - No change to the publisher runner's exposure, and no registry credential in the new job. - The first run of the new job is checked against real logs before it is relied on.

coilyco-ops referenced this issue

2026-08-12 10:22:41 +00:00

Catch a Dockerfile and .dockerignore disagreement on pull requests, without a daemon #140

coilyco-ops referenced this issue from a commit

2026-08-12 10:23:17 +00:00

test(image): fail a pull request when the Dockerfile and .dockerignore disagree

coilyco-ops referenced this issue

2026-08-12 10:34:22 +00:00

Pull-request CI never builds the image, so Dockerfile faults only surface on main #129

coilyco-ops commented

2026-08-12 10:34:35 +00:00

Author

Member

Narrowed, not closed

#140 landed the daemon-free half. A Go test in the existing test job parses the Dockerfile's context COPY sources and checks each against .dockerignore with Docker's real semantics, so the specific fault in this issue's description now fails on the pull request:

#82 added a docs/ read to sirens-echo-policy-check, which the Dockerfile runs during the build stage, while .dockerignore excluded docs.

That case is reproduced in the test suite and was confirmed to fail against this checkout before being restored.

What is still yours

Everything in the "Runner findings" and "Leading hypothesis" sections. A build-only job that reaches a Docker daemon, and the acceptance criterion this issue is careful about:

The first run of the new job is checked against real logs before it is relied on.

That is not satisfiable headlessly here. The API returns 404 for both job and run logs on this instance, which this issue already documents, so an agent cannot tell a misconfigured job from a real build failure. I have not pushed a speculative job to main for that reason.

The three candidate fixes remain in the right order. Dropping container: is the cheapest to try and matches the hypothesis that the step never saw the runner process's DOCKER_HOST.

Worth noting separately

Job logs are readable in the web UI but not through the API, so an agent cannot self-diagnose a CI failure in this repo. That is worth solving on its own.

Agreed, and it is the thing gating this issue rather than a side observation. Every attempt at the build job costs a human log read until it is fixed. If that gets its own issue in infrastructure, this one becomes an agent-completable task instead of a supervised one.

Next owner

Ops or Kai, one supervised attempt.

## Narrowed, not closed #140 landed the daemon-free half. A Go test in the existing `test` job parses the Dockerfile's context `COPY` sources and checks each against `.dockerignore` with Docker's real semantics, so the specific fault in this issue's description now fails on the pull request: > #82 added a `docs/` read to `sirens-echo-policy-check`, which the Dockerfile runs during the build stage, while `.dockerignore` excluded `docs`. That case is reproduced in the test suite and was confirmed to fail against this checkout before being restored. ## What is still yours Everything in the "Runner findings" and "Leading hypothesis" sections. A build-only job that reaches a Docker daemon, and the acceptance criterion this issue is careful about: > The first run of the new job is checked against real logs before it is relied on. That is not satisfiable headlessly here. The API returns 404 for both job and run logs on this instance, which this issue already documents, so an agent cannot tell a misconfigured job from a real build failure. I have not pushed a speculative job to `main` for that reason. The three candidate fixes remain in the right order. Dropping `container:` is the cheapest to try and matches the hypothesis that the step never saw the runner process's `DOCKER_HOST`. ## Worth noting separately > Job logs are readable in the web UI but not through the API, so an agent cannot self-diagnose a CI failure in this repo. That is worth solving on its own. Agreed, and it is the thing gating this issue rather than a side observation. Every attempt at the build job costs a human log read until it is fixed. If that gets its own issue in `infrastructure`, this one becomes an agent-completable task instead of a supervised one. ## Next owner Ops or Kai, one supervised attempt.

coilyco-ops referenced this issue from a commit

2026-08-12 10:53:02 +00:00

ci: probe docker reachability from inside a job container

coilyco-ops referenced this issue from a commit

2026-08-12 10:53:57 +00:00

ci: move the probe body into a tracked script

~~coilyco-ops referenced this issue 2026-08-12 10:54:30 +00:00~~

ci: build the image on pull requests #142

coilyco-ops referenced this issue from a commit

2026-08-12 10:57:22 +00:00

ci: build the image on pull requests

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-12 11:00:52 +00:00

ci: build the image on pull requests #142

coilyco-ops commented

2026-08-12 11:02:05 +00:00

Author

Member

Done, and checked against real logs

The Forgejo 16 upgrade serves the Actions log endpoints that returned 404 here, which is what made this diagnosable. Merged as #142, fedeb997.

The failure this issue could not read

Run 17155, job 26996, retrieved from /actions/jobs/26996/logs:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
⚙️ [runner]: exitcode '1': failure

The leading hypothesis was right about the cause and would not have been fixed by either of the first two candidate remedies as written.

A probe job reported what a job container actually sees:

agentic-os:release carries a docker client, 28.5.2, at /usr/local/bin/docker
no socket is mounted at /var/run/docker.sock
DOCKER_HOST is unset, so the client fell back to the socket
tcp://localhost:2375 is unreachable, tcp://172.17.0.1:2375 answers 28.5.2

So dropping container: was not needed, and copying the runner's DOCKER_HOST would have failed: tcp://localhost:2375 is the runner container's loopback and a job container has its own. dockerd listens on 0.0.0.0:2375 in the runner pod, so the daemon answers on the job container's default gateway.

What landed

scripts/ci-image-build.sh derives the address from /proc/net/route, falls back to the historic bridge and then the socket, and verifies each candidate with a version call before using it. Renumbering the bridge changes nothing.

Acceptance, each with its evidence:

A pull request that breaks the Docker build context fails on that pull request. Reproduced by removing the compose stage's COPY --from=build of the expander. Build fails, exit 127 at Dockerfile line 33. The #140 static guard passes on the same fault, so the two checks are complementary rather than overlapping.
No change to the publisher runner's exposure and no registry credential in the new job. The job runs on docker, builds with no --push, and tags sirens-echo:pr-check, which nothing ships. publish-echo-image is untouched on deploy.
The first run of the new job is checked against real logs before it is relied on. Run 17781 on the pull request: building against tcp://172.18.0.1:2375, all 35 Dockerfile steps including the compose stage clone, image build succeeded. The derived address was used, not the hardcoded fallback. Run 17782 on main after the merge is green across test, image-build, and publish-echo-image.

The probe stays as a failure()-only step, so the next red run reports what it can see without anyone adding a commit to ask.

The observation that is now obsolete

Job logs are readable in the web UI but not through the API, so an agent cannot self-diagnose a CI failure in this repo.

Fixed by the upgrade. Both /actions/runs/{id}/logs and /actions/jobs/{id}/logs return 200, the job endpoint as plain text. That is what turned this from a supervised task into a completable one.

## Done, and checked against real logs The Forgejo 16 upgrade serves the Actions log endpoints that returned 404 here, which is what made this diagnosable. Merged as #142, `fedeb997`. ## The failure this issue could not read Run 17155, job 26996, retrieved from `/actions/jobs/26996/logs`: ``` Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? ⚙️ [runner]: exitcode '1': failure ``` The leading hypothesis was right about the cause and would not have been fixed by either of the first two candidate remedies as written. A probe job reported what a job container actually sees: * `agentic-os:release` carries a docker client, 28.5.2, at `/usr/local/bin/docker` * no socket is mounted at `/var/run/docker.sock` * `DOCKER_HOST` is unset, so the client fell back to the socket * `tcp://localhost:2375` is unreachable, `tcp://172.17.0.1:2375` answers 28.5.2 So dropping `container:` was not needed, and copying the runner's `DOCKER_HOST` would have failed: `tcp://localhost:2375` is the runner container's loopback and a job container has its own. `dockerd` listens on `0.0.0.0:2375` in the runner pod, so the daemon answers on the job container's default gateway. ## What landed `scripts/ci-image-build.sh` derives the address from `/proc/net/route`, falls back to the historic bridge and then the socket, and verifies each candidate with a version call before using it. Renumbering the bridge changes nothing. Acceptance, each with its evidence: * **A pull request that breaks the Docker build context fails on that pull request.** Reproduced by removing the compose stage's `COPY --from=build` of the expander. Build fails, exit 127 at Dockerfile line 33. The #140 static guard passes on the same fault, so the two checks are complementary rather than overlapping. * **No change to the publisher runner's exposure and no registry credential in the new job.** The job runs on `docker`, builds with no `--push`, and tags `sirens-echo:pr-check`, which nothing ships. `publish-echo-image` is untouched on `deploy`. * **The first run of the new job is checked against real logs before it is relied on.** Run 17781 on the pull request: `building against tcp://172.18.0.1:2375`, all 35 Dockerfile steps including the compose stage clone, `image build succeeded`. The derived address was used, not the hardcoded fallback. Run 17782 on `main` after the merge is green across `test`, `image-build`, and `publish-echo-image`. The probe stays as a `failure()`-only step, so the next red run reports what it can see without anyone adding a commit to ask. ## The observation that is now obsolete > Job logs are readable in the web UI but not through the API, so an agent cannot self-diagnose a CI failure in this repo. Fixed by the upgrade. Both `/actions/runs/{id}/logs` and `/actions/jobs/{id}/logs` return 200, the job endpoint as plain text. That is what turned this from a supervised task into a completable one.

coilyco-ops referenced this issue

2026-08-12 11:02:19 +00:00

Pull-request CI never builds the image, so Dockerfile faults only surface on main #129

coilyco-ops closed this issue

2026-08-12 11:02:24 +00:00

coilyco-ops referenced this issue

2026-08-14 22:59:03 +00:00

The docker probe cannot explain the failure it is attached to #787

coilyco-ops referenced this issue from a commit

2026-08-15 15:27:48 +00:00

fix(ci): make the docker probe report on the daemon the build chose

coilyco-ops referenced this issue