interactive: inspect repeated CI image-build failure

coilyco-ops commented

2026-07-22 06:37:43 +00:00

Member

The new per-mod matrix is landed in commit 9154779 and locally passes all 523 tests, lint, mypy, pre-commit, and Ward validation. Forgejo run 105 failed in build-image task 14518 after 11m02s, before the four publish-mod matrix tasks started: https://forgejo.coilysiren.me/coilyco-gaming/eco-app/actions/runs/105\n\nThe preceding combined workflow failed in the same image-build phase in run 104, task 14491: https://forgejo.coilysiren.me/coilyco-gaming/eco-app/actions/runs/104\n\nThe sanctioned Actions log endpoint returns HTTP 404 for both completed jobs using either the run index or ActionRun id, so this surface cannot inspect the failing Docker step. An attended operator should open run 105 in Forgejo, capture the exact failing build-image log, and verify whether the runner hit storage, network, build timeout, or a Docker build error. Do not probe this with another push. The four package jobs are correctly dependency-gated and did not run.

The new per-mod matrix is landed in commit 9154779 and locally passes all 523 tests, lint, mypy, pre-commit, and Ward validation. Forgejo run 105 failed in build-image task 14518 after 11m02s, before the four publish-mod matrix tasks started: https://forgejo.coilysiren.me/coilyco-gaming/eco-app/actions/runs/105\n\nThe preceding combined workflow failed in the same image-build phase in run 104, task 14491: https://forgejo.coilysiren.me/coilyco-gaming/eco-app/actions/runs/104\n\nThe sanctioned Actions log endpoint returns HTTP 404 for both completed jobs using either the run index or ActionRun id, so this surface cannot inspect the failing Docker step. An attended operator should open run 105 in Forgejo, capture the exact failing build-image log, and verify whether the runner hit storage, network, build timeout, or a Docker build error. Do not probe this with another push. The four package jobs are correctly dependency-gated and did not run.

coilyco-ops added the

autonomy

live-collab

label

2026-07-22 06:37:53 +00:00

coilyco-ops commented

2026-07-22 06:43:05 +00:00

Author

Member

Root cause confirmed from run 105 task 14518. The image build reaches mods/jobs/src, then dotnet restore fails with NU1301 after timing out on the public https://api.nuget.org/v3/index.json service index. This is the same known runner-egress blocker from #69, not an image tag, package token, or per-mod matrix failure. The matrix never starts because build-image fails first.

Application-side retries were already tried and retired because they turned outages into 20 to 40 minute sinks. The durable infrastructure prerequisite is coilyco-flight-deck/infrastructure#571. Keep this issue open until an operator verifies the internal NuGet feed from both runner and DinD build paths, then rerun the image build and four independent package publishers.

Root cause confirmed from run 105 task 14518. The image build reaches `mods/jobs/src`, then `dotnet restore` fails with NU1301 after timing out on the public `https://api.nuget.org/v3/index.json` service index. This is the same known runner-egress blocker from #69, not an image tag, package token, or per-mod matrix failure. The matrix never starts because `build-image` fails first. Application-side retries were already tried and retired because they turned outages into 20 to 40 minute sinks. The durable infrastructure prerequisite is coilyco-flight-deck/infrastructure#571. Keep this issue open until an operator verifies the internal NuGet feed from both runner and DinD build paths, then rerun the image build and four independent package publishers.

coilyco-ops commented

2026-07-22 06:51:19 +00:00

Author

Member

Canonical infrastructure blocker corrected to coilyco-flight-deck/infrastructure#563. The earlier Telegram curl-container test already showed destination-sensitive failure inside the Forgejo DinD path while normal Kubernetes pods could reach the same external service. NuGet is now a second affected destination. infrastructure#571 is closed as a narrower duplicate.

The image and four package jobs remain correctly blocked pending live verification of both api.telegram.org and api.nuget.org from the general DinD and dedicated Docker-build paths.

Canonical infrastructure blocker corrected to coilyco-flight-deck/infrastructure#563. The earlier Telegram curl-container test already showed destination-sensitive failure inside the Forgejo DinD path while normal Kubernetes pods could reach the same external service. NuGet is now a second affected destination. infrastructure#571 is closed as a narrower duplicate. The image and four package jobs remain correctly blocked pending live verification of both `api.telegram.org` and `api.nuget.org` from the general DinD and dedicated Docker-build paths.

coilyco-ops referenced this issue

2026-07-22 07:14:02 +00:00

ci: route the Eco image build through the runner egress proxy #142

coilyco-ops commented

2026-07-22 07:46:11 +00:00

Author

Member

Resolved. Infrastructure main now deploys a two-replica allowlisted Forgejo runner HTTPS proxy and injects FORGEJO_EGRESS_PROXY into both runner pools. Flux applied revision b2ec6ec with both proxy replicas ready, and the operator recycled both runner StatefulSets. Eco commit 58ab8bd passes the proxy into Docker's predefined HTTP_PROXY and HTTPS_PROXY build arguments. Workflow-dispatch run 107 completed successfully from 07:34:05Z to 07:45:08Z. Test, frontend, build-image, publish-eco-jobs-tracker, publish-eco-replay, publish-eco-store-exporter, and publish-eco-telemetry all succeeded. This verifies the in-container NuGet restore and all four independent Forgejo package publishers.

Resolved. Infrastructure main now deploys a two-replica allowlisted Forgejo runner HTTPS proxy and injects FORGEJO_EGRESS_PROXY into both runner pools. Flux applied revision b2ec6ec with both proxy replicas ready, and the operator recycled both runner StatefulSets. Eco commit 58ab8bd passes the proxy into Docker's predefined HTTP_PROXY and HTTPS_PROXY build arguments. Workflow-dispatch run 107 completed successfully from 07:34:05Z to 07:45:08Z. Test, frontend, build-image, publish-eco-jobs-tracker, publish-eco-replay, publish-eco-store-exporter, and publish-eco-telemetry all succeeded. This verifies the in-container NuGet restore and all four independent Forgejo package publishers.

coilyco-ops closed this issue

2026-07-22 07:46:12 +00:00

Rows
Columns

interactive: inspect repeated CI image-build failure #141