Add bounded retries for transient dev-base publish operations #596

New issue

Closed

opened 2026-07-15 13:18:43 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-15 13:18:43 +00:00

Member

Context

During the July 15 release recovery, release progress depended on re-running full publish workflows after transient-looking Docker or registry failures. Kai called out the lack of retries as a release hardening gap.

Requested work

Add bounded retry/backoff around the narrow external operations in dev-base publish that are known to flake, such as registry probes, cache probes, manifest checks, and image pulls.
Keep Docker builds themselves honest. Do not hide deterministic build failures behind unbounded retries.
Make retry attempts visible in existing step output or summaries so a later director can tell when a run succeeded after retry.
Keep this separate from broad ::error:: message rewriting.

Acceptance

Transient probe/pull/push failures get a small bounded retry with clear attempt logging.
Deterministic build failures still fail promptly.
The repo's normal validation path passes.

Context During the July 15 release recovery, release progress depended on re-running full publish workflows after transient-looking Docker or registry failures. Kai called out the lack of retries as a release hardening gap. Requested work * Add bounded retry/backoff around the narrow external operations in dev-base publish that are known to flake, such as registry probes, cache probes, manifest checks, and image pulls. * Keep Docker builds themselves honest. Do not hide deterministic build failures behind unbounded retries. * Make retry attempts visible in existing step output or summaries so a later director can tell when a run succeeded after retry. * Keep this separate from broad ::error:: message rewriting. Acceptance * Transient probe/pull/push failures get a small bounded retry with clear attempt logging. * Deterministic build failures still fail promptly. * The repo's normal validation path passes.

coilyco-ops commented

2026-07-15 13:34:07 +00:00

Author

Member

WARDED_WORKFLOW: reservation-held

reservation details

Holder: launch intent for container engineer-codex-agentic-os-596 on host kais-macbook-pro-2.local.

Accepted by ward agent --harness codex (reserved 2026-07-15T13:34:06Z). Concurrent ward agent runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). --override-reservation overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/agentic-os#596 · branch issue-596 · harness codex · workflow pull-request-and-merge
Run: engineer-codex-agentic-os-596 · ward v0.697.0 · dispatched 2026-07-15T13:34:01Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.697.0).

— Codex, via ward agent

WARDED_WORKFLOW: reservation-held <details><summary>reservation details</summary> Holder: launch intent for container `engineer-codex-agentic-os-596` on host `kais-macbook-pro-2.local`. Accepted by `ward agent --harness codex` (reserved 2026-07-15T13:34:06Z). Concurrent `ward agent` runs are blocked until this intent becomes visible or the intent is released. The stale-intent fallback is still TTL-bounded (3h TTL). `--override-reservation` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/agentic-os#596` · branch `issue-596` · harness `codex` · workflow `pull-request-and-merge` - **Run:** `engineer-codex-agentic-os-596` · ward `v0.697.0` · dispatched `2026-07-15T13:34:01Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.697.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-15 13:41:24 +00:00

Add bounded retries for dev-base publish probes #599

coilyco-ops closed this issue

2026-07-15 13:45:36 +00:00

coilyco-ops commented

2026-07-15 13:46:16 +00:00

Author

Member

WARDED_WORKFLOW: #599

details

workflow: pull-request-and-merge; review summary: skipped, in-container review gate intentionally skipped because temporary ward default pending brokered QA.
felt: the retry boundary was straightforward once the narrow probe and wait calls were isolated.
confidence: high.
surprises: Forgejo kept the PR run pending for a while before both checks landed green.
follow-ups: none.

WARDED_WORKFLOW: https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/pulls/599 <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped, in-container review gate intentionally skipped because temporary ward default pending brokered QA. felt: the retry boundary was straightforward once the narrow probe and wait calls were isolated. confidence: high. surprises: Forgejo kept the PR run pending for a while before both checks landed green. follow-ups: none. </details>