coilyco-flight-deck/agentic-os

Watch

Nothing reaps un-checked-out local branches, so 253 fully-pushed branches have accumulated across the fleet #1084

New issue

Closed

opened 2026-08-16 03:41:39 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-16 03:41:39 +00:00

Member

Filed by Olaf (OPS) after Kai asked why her ~/projects checkout was sitting on a stale branch rather than main. The answer turned out not to be the thing she suspected, so the diagnosis is worth recording before the fix.

The sweep is not broken

Worth saying first, because it was my initial hypothesis and it is wrong.

sweep.json shows last_sweep: 2026-08-16T01:54:13Z with candidates: {}, ninety minutes before I looked. Of the dead leases still holding artifacts, every one is inside its 24-hour grace — uv95 dead at 23:24Z and wy58 dead at 01:54Z. Those will release on their own. cleanDeadNativeSessions is doing exactly what it says.

Gap 1: ten leases are stuck permanently, and nothing says so

cleanNativeArtifact handles a purged worktree correctly at the top: stat fails, prune, then deleteNativeBranchIfRemote. That last call is where it stops.

output, err := nativeGit(repository, "rev-list", "refs/heads/"+branch, "--not", "--remotes=origin")
if err != nil || output != "" {
    return false, err
}

A branch holding a local-only commit returns false, the artifact goes into remaining, and len(remaining) > 0 preserves the lease. Forever. There is no later pass that revisits it.

Ten leases are in this state right now, every one in agentic-os, every one with a purged worktree:

20260804t215251z-2355-1bc13047   1 unpushed   dead 11 days
20260805t213122z-72648-631e03e5  1 unpushed
20260805t230857z-96529-e6773e0b  1 unpushed
20260805t233604z-38148-85946804  1 unpushed
20260806t155911z-31254-5122a724  1 unpushed
ck67                             1 unpushed
fx48                             1 unpushed
td77                             1 unpushed
uz86                             1 unpushed
wm54                             2 unpushed

The guard is right and should not be relaxed. The worktree is gone, so that commit exists only as the local branch ref. Deleting it would destroy the only copy — precisely the remote-checkpoint failure the rule exists to catch, showing up as evidence that the rule works.

What is missing is that nothing surfaces them. Eleven days of silence on a commit with no second copy. A launch that printed "10 dead sessions hold unpushed commits" would turn a permanent leak into a decision.

Gap 2: the actual source of the litter

The fleet pass switches a clean, inactive, remotely-recoverable checked-out branch to main and deletes that one branch. cleanDeadNativeSessions releases branches held by dead leases. Neither reaches a local branch that is simply sitting there un-checked-out.

So every PR branch ever created accumulates for the life of the clone. Measured across 11 repositories:


local branches total	429
fully pushed, not worktree-held, safe to delete	253
holding local-only commits	55

deploy carries 119 branches, agentic-os 106, infrastructure 63. That is the frustration, and it is unrelated to leases.

Why Kai's checkout stayed on a branch

Incidental but it is what started this. Her infrastructure reflog shows she checked out aos/claude/uz86-session-name on 2026-08-13, committed to it, and moved herself back on 08-15. The tree was clean and the tip reachable from origin/main, so the protective guards did not apply — but the pass only acts on an inactive checkout, and eleven live sessions hold linked worktrees on that repository. A canonical checkout that reads as active is never eligible.

Proposed

Surface stuck leases at launch. One line naming the count and the repositories. Cheap, and it is the difference between a protected commit and a lost one.
Reap fully-pushed local branches. Same rev-list --not --remotes safety test the sweep already uses, extended to branches that are not checked out and not worktree-held. Deliberately stricter than git branch --merged, which would delete a branch whose commits are merged into main but never pushed anywhere.
Consider whether the inactive condition should mean "no session holds a worktree on this branch" rather than "on this repository". With eleven concurrent sessions the current reading makes the pass close to inert on busy repos.

scripts/reap-merged-branches.sh ships the second one as an operator tool now, dry-run by default. The durable answer belongs in the sweep.

**Filed by Olaf (OPS)** after Kai asked why her `~/projects` checkout was sitting on a stale branch rather than `main`. The answer turned out not to be the thing she suspected, so the diagnosis is worth recording before the fix. ## The sweep is not broken Worth saying first, because it was my initial hypothesis and it is wrong. `sweep.json` shows `last_sweep: 2026-08-16T01:54:13Z` with `candidates: {}`, ninety minutes before I looked. Of the dead leases still holding artifacts, **every one is inside its 24-hour grace** — `uv95` dead at 23:24Z and `wy58` dead at 01:54Z. Those will release on their own. `cleanDeadNativeSessions` is doing exactly what it says. ## Gap 1: ten leases are stuck permanently, and nothing says so `cleanNativeArtifact` handles a purged worktree correctly at the top: stat fails, prune, then `deleteNativeBranchIfRemote`. That last call is where it stops. ```go output, err := nativeGit(repository, "rev-list", "refs/heads/"+branch, "--not", "--remotes=origin") if err != nil || output != "" { return false, err } ``` A branch holding a local-only commit returns `false`, the artifact goes into `remaining`, and `len(remaining) > 0` preserves the lease. Forever. There is no later pass that revisits it. **Ten leases are in this state right now**, every one in `agentic-os`, every one with a purged worktree: ``` 20260804t215251z-2355-1bc13047 1 unpushed dead 11 days 20260805t213122z-72648-631e03e5 1 unpushed 20260805t230857z-96529-e6773e0b 1 unpushed 20260805t233604z-38148-85946804 1 unpushed 20260806t155911z-31254-5122a724 1 unpushed ck67 1 unpushed fx48 1 unpushed td77 1 unpushed uz86 1 unpushed wm54 2 unpushed ``` **The guard is right and should not be relaxed.** The worktree is gone, so that commit exists only as the local branch ref. Deleting it would destroy the only copy — precisely the remote-checkpoint failure the rule exists to catch, showing up as evidence that the rule works. What is missing is that **nothing surfaces them.** Eleven days of silence on a commit with no second copy. A launch that printed "10 dead sessions hold unpushed commits" would turn a permanent leak into a decision. ## Gap 2: the actual source of the litter The fleet pass switches a clean, inactive, remotely-recoverable **checked-out** branch to `main` and deletes that one branch. `cleanDeadNativeSessions` releases branches held by dead **leases**. Neither reaches a local branch that is simply sitting there un-checked-out. So every PR branch ever created accumulates for the life of the clone. Measured across 11 repositories: | | | | --- | --- | | local branches total | **429** | | fully pushed, not worktree-held, safe to delete | **253** | | holding local-only commits | **55** | `deploy` carries 119 branches, `agentic-os` 106, `infrastructure` 63. That is the frustration, and it is unrelated to leases. ## Why Kai's checkout stayed on a branch Incidental but it is what started this. Her `infrastructure` reflog shows she checked out `aos/claude/uz86-session-name` on 2026-08-13, committed to it, and moved herself back on 08-15. The tree was clean and the tip reachable from `origin/main`, so the protective guards did not apply — but the pass only acts on an **inactive** checkout, and eleven live sessions hold linked worktrees on that repository. A canonical checkout that reads as active is never eligible. ## Proposed 1. **Surface stuck leases at launch.** One line naming the count and the repositories. Cheap, and it is the difference between a protected commit and a lost one. 2. **Reap fully-pushed local branches.** Same `rev-list --not --remotes` safety test the sweep already uses, extended to branches that are not checked out and not worktree-held. Deliberately stricter than `git branch --merged`, which would delete a branch whose commits are merged into main but never pushed anywhere. 3. Consider whether the **inactive** condition should mean "no session holds a worktree on this branch" rather than "on this repository". With eleven concurrent sessions the current reading makes the pass close to inert on busy repos. `scripts/reap-merged-branches.sh` ships the second one as an operator tool now, dry-run by default. The durable answer belongs in the sweep.

coilyco-ops referenced this issue from a commit

2026-08-16 03:42:36 +00:00

feat(scripts): reap fully-pushed local branches, which nothing does today

coilyco-ops commented

2026-08-16 03:55:24 +00:00

Author

Member

A second checkout in the same state, and it makes proposal 3 the main one rather than a footnote. Olaf (ops, claude seat).

Kai found another repository off main, so I swept every canonical checkout under ~/projects rather than reasoning from one case.

Fleet state

Eleven aos-managed checkouts. Ten are on clean main. One is not:

deploy   ops/chart-pins-are-watched   tip e51268d   dirty=0   local-only=0
         behind origin/main: 147
         merged into origin/main: YES
         live worktrees on the repo: 13

Eco is on release with one dirty file, which reads as deliberate for a game checkout rather than drift, so I have left it alone.

Why this one matters more than the first

The infrastructure case had an explanation that could be waved at: Kai checked the branch out herself and committed to it, so a reader could call it operator behaviour.

This one has no such story. ops/chart-pins-are-watched is clean, carries zero local-only commits, and is fully merged into origin/main. It satisfies every safety condition the fleet pass tests — clean, non-main, remotely recoverable — and it has been sitting 147 commits and three days stale.

The single condition it fails is inactive, and it fails it because thirteen worktrees are linked off that repository.

That is the whole bug, isolated. Not a guard protecting work, not a grace period, not a timing miss. A checkout that qualifies on every dimension the pass cares about, permanently ineligible because unrelated sessions have worktrees on the same repository.

What it implies

On a busy repository the fleet pass is effectively dead code. deploy and agentic-os are the two most-worked repositories in the estate and therefore the two most likely to always have a live worktree, which makes them exactly the ones the pass will never service. The feature degrades precisely where it is needed most.

So proposal 3 is the fix, and proposals 1 and 2 are cleanup around it. Scoping inactive to "no session holds a worktree on this branch" rather than "on this repository" would have swapped this checkout on the next launch. A linked worktree on aos/claude/gq98 says nothing about whether the canonical checkout sitting on ops/chart-pins-are-watched is safe to move.

Worth checking whether the same repository-scoped reading also gates the fast-forward of a clean main, since that would mean busy repositories go stale as well as unswapped. Ten of eleven checkouts being on main here does not prove they are current.

I have not touched the deploy checkout. It is Kai's, and switching branches in a checkout an agent did not create is outside what the foreign-checkout rule permits, even when the move is provably safe.

**A second checkout in the same state, and it makes proposal 3 the main one rather than a footnote.** Olaf (ops, claude seat). Kai found another repository off `main`, so I swept every canonical checkout under `~/projects` rather than reasoning from one case. ## Fleet state Eleven aos-managed checkouts. Ten are on clean `main`. One is not: ``` deploy ops/chart-pins-are-watched tip e51268d dirty=0 local-only=0 behind origin/main: 147 merged into origin/main: YES live worktrees on the repo: 13 ``` `Eco` is on `release` with one dirty file, which reads as deliberate for a game checkout rather than drift, so I have left it alone. ## Why this one matters more than the first The `infrastructure` case had an explanation that could be waved at: Kai checked the branch out herself and committed to it, so a reader could call it operator behaviour. **This one has no such story.** `ops/chart-pins-are-watched` is clean, carries zero local-only commits, and is **fully merged into `origin/main`**. It satisfies every safety condition the fleet pass tests — clean, non-main, remotely recoverable — and it has been sitting **147 commits and three days stale**. The single condition it fails is **inactive**, and it fails it because thirteen worktrees are linked off that repository. That is the whole bug, isolated. Not a guard protecting work, not a grace period, not a timing miss. A checkout that qualifies on every dimension the pass cares about, permanently ineligible because unrelated sessions have worktrees on the same repository. ## What it implies **On a busy repository the fleet pass is effectively dead code.** `deploy` and `agentic-os` are the two most-worked repositories in the estate and therefore the two most likely to always have a live worktree, which makes them exactly the ones the pass will never service. The feature degrades precisely where it is needed most. So **proposal 3 is the fix**, and proposals 1 and 2 are cleanup around it. Scoping `inactive` to "no session holds a worktree **on this branch**" rather than "on this repository" would have swapped this checkout on the next launch. A linked worktree on `aos/claude/gq98` says nothing about whether the canonical checkout sitting on `ops/chart-pins-are-watched` is safe to move. Worth checking whether the same repository-scoped reading also gates the fast-forward of a clean `main`, since that would mean busy repositories go stale as well as unswapped. Ten of eleven checkouts being on `main` here does not prove they are *current*. I have not touched the `deploy` checkout. It is Kai's, and switching branches in a checkout an agent did not create is outside what the foreign-checkout rule permits, even when the move is provably safe.

coilyco-ops referenced this issue

2026-08-16 04:01:51 +00:00

A session worktree can check out the default branch and lock the canonical checkout out of it #1086

coilyco-ops referenced this issue from a commit

2026-08-16 05:08:35 +00:00

fix(native): the fleet pass has been a no-op since 2026-08-05

coilyco-ops referenced this issue

2026-08-16 05:09:14 +00:00

fix(native): the fleet pass has been a no-op since 2026-08-05 #1088

coilysiren referenced this issue from a commit

2026-08-16 05:18:14 +00:00

fix(native): the fleet pass has been a no-op since 2026-08-05 (#1088)

coilyco-ops referenced this issue

2026-08-16 05:28:39 +00:00

A session worktree can check out the default branch and lock the canonical checkout out of it #1086

coilyco-ops added the

labels

2026-08-17 02:36:00 +00:00

coilyco-ops referenced this issue

2026-08-22 20:54:50 +00:00

Lane: headless engineer burn-down in agentic-os, 28 issues at P1 and P2, ordered so the toolchain unblocks itself first #1177

coilyco-ops added this to the Headless engineer lane: toolchain first milestone

2026-08-22 20:58:28 +00:00

coilyco-ops commented

2026-08-22 23:12:30 +00:00

Author

Member

Director prep before phase 4 opens this (#1177). Measured on kais-macbook-pro today, and the local-only rule this needs cannot work. There is a forge key that does, and I verified it.

Current fleet state

Nineteen resident checkouts, -workdir excluded:

local branches, fleet                                    361
local branches whose remote counterpart is gone          247

agentic-os alone holds 103 local branches, 94 with no remote. Splitting those 94 by whether their tip is reachable from any origin ref:

tip IS on some origin ref (ordinary merge)                15
tip is NOWHERE on origin                                  79

The 79 are the whole problem, and #1088's reaper cannot see them

reapNativeMergedBranches takes "local branches already wholly on origin". That is an ancestor test, and it is blind to squash and rebase merges for exactly the reason #1034 gives for the pre-push guard. This repo squash-merges, so a merged branch's commits never land on main and its tip is unreachable.

The list is not stale history. It is today's lane:

aos/1089-agents-headroom      merged as PR #1178
aos/1062-build-output-hooks   merged as PR #1179
aos/993-code-comments-managed merged as PR #1182
aos/987-dev-base-retry        merged as PR #1187
aos/985-created-then-unchanged merged as PR #1191

Every one merged, every one has a local branch whose tip exists nowhere on origin, and the reaper will leave all of them behind forever. That is how 103 accumulates in one repo.

And "tip is nowhere on origin" must never be read as "safe to delete". It is exactly as consistent with unpushed work, which the checkpoint rules exist to protect. Locally the two are indistinguishable. That is why this issue is not a one-line git branch -d sweep and should not become one.

The forge key that works, verified

I expected the #1193 approach to transfer and it does not: pr-guard matches on head.ref, and Forgejo degrades head.ref to refs/pull/N/head once a branch is deleted, which is precisely this issue's population.

Two other fields survive deletion. PR #1178, branch long gone:

head.ref  : refs/pull/1178/head      <- useless here
head.label: aos/1089-agents-headroom <- the original branch name
head.sha  : 86b16f0bcbaec1adf95bc6fed1daa7bf1d590e88

And locally:

$ git rev-parse aos/1089-agents-headroom
86b16f0bcbaec1adf95bc6fed1daa7bf1d590e88

head.sha is the exact local branch tip. That is the strong key: a local branch whose tip equals some merged PR's head.sha was merged, whatever the merge strategy, and no ancestor relationship is needed. head.label is the weaker second key, useful when a branch was pushed to more than once.

Matching only the newest 49 merged PRs against the 94 already resolves 18 of them. Paginating resolves the rest.

Suggested acceptance

Reap a local branch when its tip matches the head.sha of a merged PR, or when it is an ancestor of the default branch. The second is the existing rule and stays.
Never reap on "no remote counterpart" alone. State that as the invariant, because it is the tempting shortcut and it deletes unpushed work.
Report the residue: branches matching neither rule are the ones a human should look at, and the count is the useful output rather than a silent skip.
Keep the existing aos/ namespace skip if session-ID uniqueness still reads it, or state why it no longer needs to.
The head.ref versus head.label distinction deserves a comment wherever the lookup lands. It cost me a false finding on #1193 and it will cost the next reader the same.

Also worth knowing for the rest of phase 4

Every one of the nineteen checkouts is currently on its default branch, so #1086's squatting hazard and #1033's pinning requirement are both satisfied on this host right now. Whatever those two land, they are hardening a state that is presently correct rather than repairing a broken one. Worktree counts run 1 to 12 per repo, about 110 live.

**Director prep before phase 4 opens this (#1177). Measured on `kais-macbook-pro` today, and the local-only rule this needs cannot work. There is a forge key that does, and I verified it.** ## Current fleet state Nineteen resident checkouts, `-workdir` excluded: ``` local branches, fleet 361 local branches whose remote counterpart is gone 247 ``` agentic-os alone holds **103 local branches, 94 with no remote**. Splitting those 94 by whether their tip is reachable from any `origin` ref: ``` tip IS on some origin ref (ordinary merge) 15 tip is NOWHERE on origin 79 ``` ## The 79 are the whole problem, and #1088's reaper cannot see them `reapNativeMergedBranches` takes "local branches already wholly on origin". That is an ancestor test, and it is blind to squash and rebase merges for exactly the reason **#1034** gives for the pre-push guard. This repo squash-merges, so a merged branch's commits never land on `main` and its tip is unreachable. The list is not stale history. It is today's lane: ``` aos/1089-agents-headroom merged as PR #1178 aos/1062-build-output-hooks merged as PR #1179 aos/993-code-comments-managed merged as PR #1182 aos/987-dev-base-retry merged as PR #1187 aos/985-created-then-unchanged merged as PR #1191 ``` Every one merged, every one has a local branch whose tip exists nowhere on `origin`, and the reaper will leave all of them behind forever. That is how 103 accumulates in one repo. **And "tip is nowhere on origin" must never be read as "safe to delete".** It is exactly as consistent with unpushed work, which the checkpoint rules exist to protect. Locally the two are indistinguishable. That is why this issue is not a one-line `git branch -d` sweep and should not become one. ## The forge key that works, verified I expected the #1193 approach to transfer and it does not: `pr-guard` matches on `head.ref`, and Forgejo degrades `head.ref` to `refs/pull/N/head` once a branch is deleted, which is precisely this issue's population. Two other fields survive deletion. PR #1178, branch long gone: ``` head.ref : refs/pull/1178/head <- useless here head.label: aos/1089-agents-headroom <- the original branch name head.sha : 86b16f0bcbaec1adf95bc6fed1daa7bf1d590e88 ``` And locally: ``` $ git rev-parse aos/1089-agents-headroom 86b16f0bcbaec1adf95bc6fed1daa7bf1d590e88 ``` **`head.sha` is the exact local branch tip.** That is the strong key: a local branch whose tip equals some merged PR's `head.sha` was merged, whatever the merge strategy, and no ancestor relationship is needed. `head.label` is the weaker second key, useful when a branch was pushed to more than once. Matching only the newest 49 merged PRs against the 94 already resolves **18** of them. Paginating resolves the rest. ## Suggested acceptance * Reap a local branch when its tip matches the `head.sha` of a **merged** PR, or when it is an ancestor of the default branch. The second is the existing rule and stays. * **Never** reap on "no remote counterpart" alone. State that as the invariant, because it is the tempting shortcut and it deletes unpushed work. * Report the residue: branches matching neither rule are the ones a human should look at, and the count is the useful output rather than a silent skip. * Keep the existing `aos/` namespace skip if session-ID uniqueness still reads it, or state why it no longer needs to. * The `head.ref` versus `head.label` distinction deserves a comment wherever the lookup lands. It cost me a false finding on #1193 and it will cost the next reader the same. ## Also worth knowing for the rest of phase 4 Every one of the nineteen checkouts is currently **on its default branch**, so #1086's squatting hazard and #1033's pinning requirement are both satisfied on this host right now. Whatever those two land, they are hardening a state that is presently correct rather than repairing a broken one. Worktree counts run 1 to 12 per repo, about 110 live.

coilyco-ops referenced this issue from a commit

2026-08-22 23:27:31 +00:00

feat(native): surface dead-session branches nothing will ever release (#1084)

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-22 23:27:49 +00:00

feat(native): surface dead-session branches nothing will ever release (#1084) #1201

coilyco-ops referenced this issue

2026-08-22 23:31:45 +00:00

feat(native): surface dead-session branches nothing will ever release (#1084) #1201

coilyco-ops closed this issue

2026-08-22 23:34:07 +00:00

coilyco-ops referenced this issue from a commit