FJ<->GH sync: replace force-mirror with a safe reconcile, relocate to Forgejo Actions #365

New issue

Open

opened 2026-06-17 22:16:38 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-06-17 22:16:38 +00:00

Owner

Spun out of the o11y rollout: the git-state observer empirically confirmed the GitHub mirror is dead fleet-wide (infra#329 - the kai-server coilysiren-github-mirror timer was never installed; checkouts 54-58 commits ahead of GitHub on active repos). Decided home for the relocated sync: a Forgejo Actions scheduled workflow (server-side, declarative, off kai-server).

Critical finding - the current scripts are NOT a merge

Both scripts/coilysiren-github-mirror.sh (FJ->GH, 04:15) and scripts/coilysiren-forgejo-mirror.sh (GH->FJ, 03:45) do git push --mirror, which force-updates and deletes non-shared refs. They force-reset one side to the other in sequence.

GitHub is a real PR/contribution surface (the GitHub mirror stays PR-gated), so both sides diverge: a PR merged on GitHub advances GH main while Kai's work advances FJ main. Under that concurrent divergence, the force-push clobbers the other side's commits (and --mirror deletes the PR branch). This is latent data-loss; it only "works" today because changes rarely diverge in the same window. So there is no safe logic to port - the relocation must also fix the safety model.

Required: a safe reconcile (pick the model)

(A) FF-push + divergence-alert (recommended, zero-clobber): fetch both; fast-forward-push FJ->GH per branch with NO --force. If a branch cannot FF (GH ahead = un-ingested PR), do NOT force - emit an ntfy alert 'GH PR needs merging into Forgejo'. Never force-pushes either side. External PRs get a human merge into canonical FJ, which is the correct review gate anyway.
(B) true auto-merge: fetch GH, merge GH's PR commits into FJ (merge commit), push the unified state to both. No human step, but merges can conflict -> must stop + alert, never force.

Build outline (after model chosen)

.forgejo/workflows/mirror-sync.yml: schedule: cron + workflow_dispatch, runs-on: docker. Iterate Forgejo repos via the search API (existing pattern), reconcile each per the chosen model.
Creds: FJ token (/forgejo/api-token) + a repo-scoped GitHub PAT - PREREQUISITE: the current /github/pat is read:packages only, so a new push-scoped PAT must be provisioned in SSM (own secret).
Failure / divergence / conflict -> ntfy (reuse the o11y-ntfy base-url cache pattern; topic e.g. fleet-mirror).
Runner caveat: the in-cluster runner has had multi-day outages, so the workflow MUST alert on its own non-run/failure. Also teach the git-state observer to flag behind-github (un-ingested PRs), which it does not compute today.

Cutover (after the workflow is proven)

Delete systemd/coilysiren-github-mirror.{service,timer}, systemd/coilysiren-forgejo-mirror.{service,timer}, and the two scripts. Removes 2 kai-server timers and closes infra#329 properly.

Related: infra#319, infra#329, infra#334.

Spun out of the o11y rollout: the git-state observer empirically confirmed the GitHub mirror is dead fleet-wide (infra#329 - the kai-server `coilysiren-github-mirror` timer was never installed; checkouts 54-58 commits ahead of GitHub on active repos). Decided home for the relocated sync: a **Forgejo Actions scheduled workflow** (server-side, declarative, off kai-server). ## Critical finding - the current scripts are NOT a merge Both `scripts/coilysiren-github-mirror.sh` (FJ->GH, 04:15) and `scripts/coilysiren-forgejo-mirror.sh` (GH->FJ, 03:45) do `git push --mirror`, which **force-updates and deletes** non-shared refs. They force-reset one side to the other in sequence. GitHub is a real PR/contribution surface (`the GitHub mirror stays PR-gated`), so both sides diverge: a PR merged on GitHub advances GH main while Kai's work advances FJ main. Under that concurrent divergence, the force-push **clobbers** the other side's commits (and `--mirror` deletes the PR branch). This is latent data-loss; it only "works" today because changes rarely diverge in the same window. So there is no safe logic to port - the relocation must also fix the safety model. ## Required: a safe reconcile (pick the model) - **(A) FF-push + divergence-alert (recommended, zero-clobber):** fetch both; fast-forward-push FJ->GH per branch with NO `--force`. If a branch cannot FF (GH ahead = un-ingested PR), do NOT force - emit an ntfy alert 'GH PR needs merging into Forgejo'. Never force-pushes either side. External PRs get a human merge into canonical FJ, which is the correct review gate anyway. - **(B) true auto-merge:** fetch GH, merge GH's PR commits into FJ (merge commit), push the unified state to both. No human step, but merges can conflict -> must stop + alert, never force. ## Build outline (after model chosen) - `.forgejo/workflows/mirror-sync.yml`: `schedule:` cron + `workflow_dispatch`, `runs-on: docker`. Iterate Forgejo repos via the search API (existing pattern), reconcile each per the chosen model. - Creds: FJ token (`/forgejo/api-token`) + a **repo-scoped GitHub PAT** - PREREQUISITE: the current `/github/pat` is `read:packages` only, so a new push-scoped PAT must be provisioned in SSM (own secret). - Failure / divergence / conflict -> ntfy (reuse the `o11y-ntfy` base-url cache pattern; topic e.g. `fleet-mirror`). - Runner caveat: the in-cluster runner has had multi-day outages, so the workflow MUST alert on its own non-run/failure. Also teach the git-state observer to flag *behind*-github (un-ingested PRs), which it does not compute today. ## Cutover (after the workflow is proven) - Delete `systemd/coilysiren-github-mirror.{service,timer}`, `systemd/coilysiren-forgejo-mirror.{service,timer}`, and the two scripts. Removes 2 kai-server timers and closes infra#329 properly. Related: infra#319, infra#329, infra#334.

coilysiren referenced this issue from a commit

2026-06-17 22:24:26 +00:00

feat(mirror): safe FJ<->GitHub auto-merge reconcile (dispatch-only, dry-run)

coilysiren added the

interactive

labels

2026-06-18 06:17:31 +00:00