coilyco-flight-deck/agentic-os

Watch

Diagnose concurrent dev-base export stall on live image runner #692

New issue

Closed

opened 2026-07-24 05:53:50 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-07-24 05:53:50 +00:00

Member

Engineer handoff after run 1815 stalled four concurrent language jobs at layer export.

Repository evidence corrected the initial diagnosis:

Infrastructure config gives the dedicated image-build runner runner.capacity: 4 specifically to preserve the AOS tier fan-out.
The shared DinD and BuildKit daemon are intentional. Concurrent jobs reach that daemon through its API.
Each language tier writes a distinct registry cache ref, so the repository does not show a cache-destination collision.
Commit b8d1a6f incorrectly serialized the workflow. Corrective commit 7754c1e restores max-parallel: 4.

Live verification needed:

Confirm a replacement run schedules up to four publish-lang-* matrix rows concurrently.
While the rows show exporting layers, inspect the forgejo-runner-build DinD and BuildKit container disk, memory, Kubernetes events, and registry upload activity.
Determine whether exporting layers is slow forward progress or a deadlock by comparing layer-upload activity over time.
Confirm all five language draft manifests and the full draft manifest publish successfully.

The engineer must not use another code push as a live probe. Ops should record the observed bottleneck and remediation here.

Engineer handoff after run 1815 stalled four concurrent language jobs at layer export. Repository evidence corrected the initial diagnosis: * Infrastructure config gives the dedicated image-build runner `runner.capacity: 4` specifically to preserve the AOS tier fan-out. * The shared DinD and BuildKit daemon are intentional. Concurrent jobs reach that daemon through its API. * Each language tier writes a distinct registry cache ref, so the repository does not show a cache-destination collision. * Commit b8d1a6f incorrectly serialized the workflow. Corrective commit 7754c1e restores `max-parallel: 4`. Live verification needed: * Confirm a replacement run schedules up to four `publish-lang-*` matrix rows concurrently. * While the rows show `exporting layers`, inspect the `forgejo-runner-build` DinD and BuildKit container disk, memory, Kubernetes events, and registry upload activity. * Determine whether `exporting layers` is slow forward progress or a deadlock by comparing layer-upload activity over time. * Confirm all five language draft manifests and the full draft manifest publish successfully. The engineer must not use another code push as a live probe. Ops should record the observed bottleneck and remediation here.

coilyco-ops added the

autonomy

live-collab

label

2026-07-24 05:53:58 +00:00

coilyco-ops referenced this issue from a commit

2026-07-24 06:01:59 +00:00

fix(dev-base): honor image runner concurrency

coilyco-ops changed title from ~~Verify serialized dev-base publishing on the live image runner~~ to Diagnose concurrent dev-base export stall on live image runner

2026-07-24 06:02:55 +00:00

coilyco-ops referenced this issue

2026-07-24 06:47:11 +00:00

Publish the refactored dev-base family for Ward native MCP bootstrap #665

coilyco-ops referenced this issue

2026-07-24 06:48:14 +00:00

Releases are blocked: a dotnet image-push flake skips the release job, so no tag and no ward-specs bundle (stuck at v0.248.0) #490

coilyco-ops commented

2026-07-24 06:48:22 +00:00

Author

Member

The replacement dev-base-publish.yml run is currently https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/1840. Forgejo still reports the run as running. The issue remains the live gate for #665, and the operator must record builder-side progress evidence before closing it.

The replacement `dev-base-publish.yml` run is currently https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/1840. Forgejo still reports the run as running. The issue remains the live gate for #665, and the operator must record builder-side progress evidence before closing it.

coilyco-ops commented

2026-07-24 10:04:38 +00:00

Author

Member

Ops update from the Specgen v0.119.0 rollout:

The agent observed three consecutive publish-full jobs cancelled by newer main pushes while each job still showed fresh progress.
The Specgen pin's job reached multi-architecture image export and registry layer push before a newer commit cancelled it.
Each superseding main revision retained ARG SPECGEN_VERSION=0.119.0.
The current publisher is for feat(skills): add second principal method batch, and its source still carries the fixed pin.

This reinforces the live gate. A full draft manifest still needs one terminal successful publisher through a quiet main window. The operator should verify builder and registry progress if the job stalls instead of receiving another superseding cancellation.

Ops update from the Specgen v0.119.0 rollout: * The agent observed three consecutive publish-full jobs cancelled by newer main pushes while each job still showed fresh progress. * The Specgen pin's job reached multi-architecture image export and registry layer push before a newer commit cancelled it. * Each superseding main revision retained ARG SPECGEN_VERSION=0.119.0. * The current publisher is for feat(skills): add second principal method batch, and its source still carries the fixed pin. This reinforces the live gate. A full draft manifest still needs one terminal successful publisher through a quiet main window. The operator should verify builder and registry progress if the job stalls instead of receiving another superseding cancellation.

coilyco-ops referenced this issue

2026-07-25 05:55:08 +00:00

Bound dev-base package churn before it fills kai-server again #739

coilyco-ops commented

2026-07-25 18:47:43 +00:00

Author

Member

Current rollout evidence for the agent-compose v0.57.0 pin: dev-base publish run 2114 on commit 21bc338e89 has remained running for more than 25 minutes. The preceding successful publish run 2105 completed in 160 seconds. The dedicated forgejo-runner-build-0 pod is Ready with no Kubernetes events, but both runner and dind containers report about 1m CPU, the runner log shows task 17908 starting at 18:21:26Z with no cleanup event, and the dind log shows no activity after 18:21:55Z. Repository CI, promotion, mirroring, and native release run 2115 are green. The engineer made no live mutation and no probe push. Ops should compare registry upload and BuildKit progress for task 17908, then record whether the run is making slow progress or is deadlocked before any remediation.

Current rollout evidence for the agent-compose v0.57.0 pin: dev-base publish run 2114 on commit 21bc338e89d3d051c37545484af95822be3cca9e has remained running for more than 25 minutes. The preceding successful publish run 2105 completed in 160 seconds. The dedicated forgejo-runner-build-0 pod is Ready with no Kubernetes events, but both runner and dind containers report about 1m CPU, the runner log shows task 17908 starting at 18:21:26Z with no cleanup event, and the dind log shows no activity after 18:21:55Z. Repository CI, promotion, mirroring, and native release run 2115 are green. The engineer made no live mutation and no probe push. Ops should compare registry upload and BuildKit progress for task 17908, then record whether the run is making slow progress or is deadlocked before any remediation.

coilyco-ops added the

priority

label

2026-07-29 03:44:55 +00:00

coilyco-ops closed this issue

2026-07-29 06:04:42 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#692

No description provided.

Rows
Columns