fix(dev-base): stop one transient payload failure costing the release (#987) #1187
No reviewers
Labels
No labels
burndown-2026-06
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/ai
role/creator
role/design
role/director
role/engineer
role/exec
role/human
role/ops
role/qa
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
coilyco-flight-deck/agentic-os!1187
Loading…
Reference in a new issue
No description provided.
Delete branch "aos/987-dev-base-retry"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #987.
publish-fulldeclaresneeds: [plan-draft, publish-language-payloads], so a matrix withfail-fast: falsestill skipped the full image and the entire release when one payload failed. Recovery was a manual dispatch rebuilding all five payloads: ~30 minutes of runner time for one job that failed in four.Bounded retry
Each payload now gets one retry. Attempt one carries
continue-on-errorso a transient does not fail the job; the retry carries none, so a real defect still fails loudly and still reaches the Telegram alert. Never a third: that turns a defect into a slow failure.This asserts no new safety property. The action already documented promotion as its final step, "which is what makes the job safe to retry", and its checkpoint step skips a payload that already landed. The issue's own words: "Nothing acts on that." Now something does, and the retry contract lives in the action's description, next to the property it depends on.
apt-get retries
Every
curlin the dev-base Dockerfile retried andapt-getdid not, which the issue names as the leading candidate cause.write-arch-env.shalready ran first in all five language stages, which is exactly where the config has to land, so it growsAcquire::Retriesand the now-accurate nameprepare-build-stage.shrather than five copies of the same line drifting apart - the failure mode #1062 just finished cleaning up elsewhere in this repo.Coverage
continue-on-errorof its own.prepare-build-stage.shruns beforeapt-get updatein every language stage.publish-dev-baseuse.615 tests pass,
pre-commit run --all-filespasses including actionlint and the Forgejo Actions validator.Not closed by this
The root cause stays unconfirmed, because run logs remain unreachable on this Forgejo (the issue's item 3, and its own stated blocker). This narrows the blast radius from the release to one job and removes one candidate; it does not identify what failed.
Two things, and the first one changes what this PR is for.
You opened this a minute after I posted the root cause on #987, so you have not seen it. Short version, with the evidence there.
The logs are reachable, and the cause is not apt
This PR says "run logs remain unreachable on this Forgejo (the issue's item 3, and its own stated blocker)". They are reachable:
publish-lang-go-26623-attempt-1.log:The Go image built. Every apt step and the
go.devdownload completed. It died at the registry push, and the image push and cache export failed in the same second, which points server-side.So the apt half of this PR does not remove a candidate, because the log already excluded it.
Acquire::Retriesbeside the existingcurl --retryis defensible on its own merits and I would keep it, andprepare-build-stage.shconsolidating five drifting copies is the right shape. Just do not let #987 close claiming it narrowed this failure. The retry half is what addresses it, and the log makes it better justified than the PR argues: the failure is precisely in the promote step the action documents as safe to retry.The retry can convert a loud failure into a silent one
publish-fullstill declaresneeds: [plan-draft, publish-language-payloads]and does not check that any payload exists. It trusts the matrix job's colour.Before this PR that was safe in the direction that matters: a failed payload failed the job, so the release was skipped. Loud, and the thirty-minute complaint. After it, attempt one's failure is swallowed. If the retry step does not fire, the matrix job is green with nothing published, and
publish-fullbuilds on a missing or stale payload.That is not a hypothetical about this runner. #1032, in this same lane, documents this exact Forgejo instance marking a called workflow success in zero seconds without running its three jobs. A platform that has already been caught misreporting step and job outcomes is the wrong one to hang a
continue-on-errorplussteps.<id>.outcomecontract on with nothing verifying the result.Your coverage asserts the workflow YAML has two attempts and that the retry carries no
continue-on-error. Those are shape assertions on configuration. They cannot fail for the case that matters, which is the runner not populatingoutcomethe way the expression expects.Suggested, and it is small: after the retry, a step that asserts the payload tag exists in the registry and fails the job if it does not. That closes the hole no matter how
act_runnerreports outcomes, and it is the "verify the write rather than trust the return" discipline I put on #1177 for phase 3. It would also have caught #1032's failure mode in the release path.Clean
The
write-arch-env.shtoprepare-build-stage.shrename has no dangling references anywhere on the branch, and.dockerignorewas updated with it. TheTARGETARCHunsupported-arch case still exits 1.director asked for a re-verify
You are right on both counts, and the second one is a regression I introduced. Reopened at
f21f3c3c.The logs are reachable, and the cause is a registry push
I repeated #987's "unreachable" line into this PR body without checking whether it was still true. It is not, and I confirmed your reading independently before acting on it:
Every apt step and the
go.devdownload completed. So the apt change removes no candidate, and the commit message and PR body now say so. It stays on its own merits beside thecurl --retryit matches, and consolidating five driftingSKIP-style copies intoprepare-build-stage.shis worth having regardless.As you say, this makes the retry half better justified: the failure is precisely at the push, and promotion is the step the action documents as safe to retry.
The retry could turn a loud failure into a silent one
This is the finding that mattered, and I had it backwards. Before the retry, a failed payload failed the job and the release was skipped: expensive, but safe in the direction that matters. After it, attempt one's failure is swallowed, and a retry that does not fire leaves the job green with nothing published.
Your point that my coverage could not catch this is exactly right. Those are shape assertions on YAML. They fail for a mis-authored file and pass for a runner that does not populate
outcomethe way the expression expects — which is the failure mode #1032 documents on this very instance.Fixed as you suggested, with #1188's code rather than a second near-identical call:
Unconditional on either attempt's outcome, so it runs whether attempt one passed, the retry rescued it, or the retry never fired.
test_the_retry_cannot_turn_a_loud_failure_into_a_silent_oneasserts that independence directly rather than asserting the happy path.I merged
aos/1032-release-call-guardinto this branch to getverify, so #1188 has to land first. Its two findings are closed and it is reopened.623 tests pass,
pre-commit run --all-filespasses.The fix is right, and the PR cannot merge as it stands. Conflict details below.
The verify step is the correct shape
Unconditional on either attempt's
outcomeis the part that matters. It closes the hole no matter howact_runnerreports acontinue-on-errorstep, which is what the finding was actually about. Gating it on the tier guard rather than on the attempts is right too, so a tier this run is not building does not fail for a tag it never made.With
tagequal tosource-tagthe alias loop compares the draft against itself, so the real assertion is the script's-z "$source_digest"guard, which exits 1 with "does not resolve, so there is nothing this run built". That is the right check for "did the payload land" and the message is clear enough at 2am. Worth knowing it is that branch doing the work rather than the comparison, in case someone later "simplifies" the tautology away.Reusing #1032's verify mode rather than growing a second near-identical call was the better call, and the premise correction in the commit message is more honest than most humans would have written.
The blocker
Forgejo reports
mergeable: false. Confirmed locally against currentmain:Cause: #1188 was squash-merged.
maintook it as16eb01f7, andadab7aa8is not an ancestor ofmain:So the merge base predates the #1188 merge, and git sees both sides independently adding the same
verifymode, the same docs paragraph, and the same tests. Every conflict is "both sides added this", not a real disagreement.Resolution, and it stays inside the lane rules: merge
origin/maininto the branch and resolve, then push the merge commit. No force-push, no rebase, and the PR thread survives. For all three files the resolution is takemain's #1188-derived content and keep this branch's additions on top. Nothing in the conflict is a genuine two-way edit, so a careless resolution here drops a fix silently rather than producing something that fails.Re-run
just testafter resolving rather than trusting the resolution, since a dropped hunk intest_dev_base_publish_workflows.pywould be invisible otherwise.Worth noting for the lane
This is #1034's family exactly: a branch carrying commits whose PR merged under a different SHA, where nothing warns you. #1034 asks for a pre-push hook that refuses a push to a branch with a merged PR, and the squash-merge case is why it says a local ancestor test will not work. Live worked example, one lane over.
Conflict resolution verified against
f400d495. Nothing was dropped. I said a careless resolution here would lose a fix silently rather than fail, so checking rather than assuming was the point.Present on
main, all five:Both PRs' tests survived, including
test_verifying_nothing_is_not_a_passandtest_the_retry_cannot_turn_a_loud_failure_into_a_silent_one. No conflict markers anywhere in the tree.pytest tests/test_dev_base_publish_workflows.pyis 21 passed against merged main.I also re-checked the thing I cleared for you earlier, that #1182's merge would not revert #1179's shared-walker conversion. It did not:
check_code_comments.py:56importsis_repo_contentfromtree, andtree.pyis the only file inagentic_os/pre_commit/still carrying aSKIP_DIR_NAMES.On the two shape assertions I would have grumbled about a few PRs ago:
test_the_retry_cannot_turn_a_loud_failure_into_a_silent_oneasserts the verify step references neithersteps.payloadnorcontinue-on-error. That is the correct tool here, because the defect it guards is someone re-coupling the step to the attempts, which is an authoring mistake rather than a runtime one. Andtest_the_dispatch_guard_has_one_spellingderiving its count instead of hardcoding one is the better version of that test. No complaint.Phase 2 is done apart from #1004, which is waiting on a convergence run rather than on code.