coilyco-flight-deck/agentic-os

Watch

Retry dev-base publish after stalled registry upload #754

New issue

Closed

opened 2026-07-25 19:25:37 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-25 19:25:37 +00:00

Member

Live-only failure

dev-base-publish run 2114 for release commit 21bc338e89d3d051c37545484af95822be3cca9e completed both architecture builds and passed in-image ward doctor, then stalled while exporting image and cache layers.

Run: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2114/jobs/1

Evidence

The build runner remains Ready with zero restarts and no pod events.
Forgejo remains Ready with no namespace Warning events.
Registry requests included successful 201 and 202 responses, followed by repeated 500 responses after 60 seconds.
Registry errors report unexpected EOF and offset mismatch between file and model.
The last registry request from this publish arrived at 18:43:43 UTC. The task heartbeat continues, but no further upload request or build log has appeared.
The AOS CLI release for the same commit is green and independently verified as aos-v0.92.0.

Operator action

Confirm release still points at the exact commit above. Replace the stalled publish with one documented guarded same-ref retry, preserving the original draft tag and commit. Verify the resulting multi-architecture image publication. Do not change code, secrets, runner configuration, registry configuration, or deployment configuration. If the retry reproduces the registry failure, preserve both runs and escalate the registry incident instead of dispatching again.

## Live-only failure `dev-base-publish` run 2114 for release commit `21bc338e89d3d051c37545484af95822be3cca9e` completed both architecture builds and passed in-image `ward doctor`, then stalled while exporting image and cache layers. Run: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2114/jobs/1 ## Evidence * The build runner remains Ready with zero restarts and no pod events. * Forgejo remains Ready with no namespace Warning events. * Registry requests included successful 201 and 202 responses, followed by repeated 500 responses after 60 seconds. * Registry errors report `unexpected EOF` and `offset mismatch between file and model`. * The last registry request from this publish arrived at 18:43:43 UTC. The task heartbeat continues, but no further upload request or build log has appeared. * The AOS CLI release for the same commit is green and independently verified as `aos-v0.92.0`. ## Operator action Confirm `release` still points at the exact commit above. Replace the stalled publish with one documented guarded same-ref retry, preserving the original draft tag and commit. Verify the resulting multi-architecture image publication. Do not change code, secrets, runner configuration, registry configuration, or deployment configuration. If the retry reproduces the registry failure, preserve both runs and escalate the registry incident instead of dispatching again.

coilyco-ops added the

autonomy

live-collab

label

2026-07-25 19:25:42 +00:00

coilyco-ops commented

2026-07-25 19:30:13 +00:00

Author

Member

Ops confirmed release still points at 21bc338e89 and resolved the original target as draft-21bc338e89d3d051c37545484af95822be3cca9e. Ops invoked the documented guarded rerun bridge once. Workflow concurrency cancelled stalled run 2114 and started retry run 2116 on the same release ref and commit: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2116. Retry 2116 rebuilt from the existing cache, reached image and cache export, then reproduced the registry upload fault. The cache export returned 499 Client Closed Request after roughly 60 seconds while the image export remained in flight. This matches the earlier server-side unexpected EOF and offset mismatch failure family. Run 2116 still reports running and its task heartbeat is current. Ops will not dispatch a second retry. The next operator action is a registry incident investigation that preserves both runs and determines why Forgejo package blob finalization drops or desynchronizes 60-second uploads before any further publish attempt.

Ops confirmed release still points at 21bc338e89d3d051c37545484af95822be3cca9e and resolved the original target as draft-21bc338e89d3d051c37545484af95822be3cca9e. Ops invoked the documented guarded rerun bridge once. Workflow concurrency cancelled stalled run 2114 and started retry run 2116 on the same release ref and commit: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2116. Retry 2116 rebuilt from the existing cache, reached image and cache export, then reproduced the registry upload fault. The cache export returned 499 Client Closed Request after roughly 60 seconds while the image export remained in flight. This matches the earlier server-side unexpected EOF and offset mismatch failure family. Run 2116 still reports running and its task heartbeat is current. Ops will not dispatch a second retry. The next operator action is a registry incident investigation that preserves both runs and determines why Forgejo package blob finalization drops or desynchronizes 60-second uploads before any further publish attempt.

coilyco-ops referenced this issue from coilyco-flight-deck/infrastructure

2026-07-25 19:38:31 +00:00

Raise the Forgejo registry request-body timeout above Traefik's 60-second default #646

coilyco-ops commented

2026-07-25 19:38:44 +00:00

Author

Member

Run 2116 succeeded after 9m41s on the exact release commit and draft tag.

Verified evidence:

OCI index contains linux/amd64 and linux/arm64.
Both platform pulls succeeded.
In-image tool checks succeeded.
The final job step succeeded.

The optional cache export still reproduced the confirmed 60-second Traefik request-body cutoff, but it was non-fatal to the required image publication. Durable remediation is tracked in infrastructure#646.

Run [2116](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2116) succeeded after 9m41s on the exact release commit and draft tag. Verified evidence: * OCI index contains `linux/amd64` and `linux/arm64`. * Both platform pulls succeeded. * In-image tool checks succeeded. * The final job step succeeded. The optional cache export still reproduced the confirmed 60-second Traefik request-body cutoff, but it was non-fatal to the required image publication. Durable remediation is tracked in [infrastructure#646](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/646).

coilyco-ops closed this issue

2026-07-25 19:38:50 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#754

No description provided.

Rows
Columns