feat(mirrors): verify push mirrors from the receiving side, on a schedule #967

Merged

coilysiren merged 1 commit from aos/claude/wz68-mirror-watch into main

2026-08-27 10:22:10 +00:00

coilysiren commented

2026-08-27 10:20:10 +00:00

Owner

The obvious check is the wrong one

Forgejo exposes last_error and last_update per mirror, and a monitor built on them reports healthy through the exact failure it exists to catch:

last_update timestamps the attempt, not the outcome. Observed on 2026-08-27: all three mirrors advanced it while every push was being rejected for an unregistered key.
last_error can stay empty through total data loss. A knot running an outdated version drops pushes, pull requests, issues, and invites without reporting an error. paulvall.dev/fid exists because of this.

So this asks the receiving side instead: compare refs/heads/main on both forges. Only the receiver knows whether anything arrived.

What landed

scripts/check_mirror_sync.py - anonymous git ls-remote on both sides, compares, exits 1 on divergence or unreadability.
scripts/mirror-targets.json - the three mirrors.
.forgejo/workflows/mirror-sync-watch.yml - every six hours, plus workflow_dispatch, plus on PRs that touch the checker itself.
just mirror-sync-check and just mirror-sync-check-test.

No credentials anywhere. Both forges are public, so the check reads them anonymously, and alerting goes through the existing in-cluster signoz-telegram mapper which holds the Telegram identity itself. The job carries no secret at all.

Proven in both directions

A monitor that has only ever passed is unproven, so the failure path was exercised against live endpoints before landing.

$ just mirror-sync-check
ok   umbra                    match       8e46ec6c5676
ok   mcp-beaver               match       eebbe24e40c0
ok   agent-compose            match       5c37622422f9
3/3 mirrors in sync                                        exit=0

$ ... --targets <crossed and missing pair>
FAIL deliberately-crossed     diverged    source 8e46ec6c5676 != mirror eebbe24e40c0
FAIL mirror-does-not-exist    unreadable  source: ok; mirror: fatal: repository '...' not found
0/2 mirrors in sync                                        exit=1

12 unit tests, all green, covering divergence, both unreadable directions, timeout, and malformed target lists.

Two deliberate choices worth reviewing

The target list is static rather than derived. Forgejo's push-mirror API is the owning source, but reading it needs an admin token, and a credential-free check was judged worth more than a derived list. The cost is that a new mirror goes unwatched until listed, which mirror-targets.json states in its own comment. Reverse this if the drift risk outweighs the credential.

It refuses when ls-remote answers with more than one ref. git ls-remote <repo> main matches any ref ending in main, and a stray mirror ref did exactly that on mcp-beaver, doubling the answer and corrupting 25 files downstream (#963). This asks for the full ref name and refuses a multi-ref answer rather than taking the first line.

Not included

No automatic repair. #440 kept the read-only-detector split deliberately, and repair stays an operator action.

Closes coilysiren/inbox#441

Nothing watched the Tangled mirrors. A push mirror can fail and keep failing with no outward sign, which is the failure coilysiren/inbox#440 named as the whole reason the server-side shape was chosen. ## The obvious check is the wrong one Forgejo exposes `last_error` and `last_update` per mirror, and a monitor built on them reports healthy through the exact failure it exists to catch: * **`last_update` timestamps the attempt, not the outcome.** Observed on 2026-08-27: all three mirrors advanced it while every push was being rejected for an unregistered key. * **`last_error` can stay empty through total data loss.** A knot running an outdated version drops pushes, pull requests, issues, and invites without reporting an error. `paulvall.dev/fid` exists because of this. So this asks the receiving side instead: compare `refs/heads/main` on both forges. Only the receiver knows whether anything arrived. ## What landed * `scripts/check_mirror_sync.py` - anonymous `git ls-remote` on both sides, compares, exits 1 on divergence or unreadability. * `scripts/mirror-targets.json` - the three mirrors. * `.forgejo/workflows/mirror-sync-watch.yml` - every six hours, plus `workflow_dispatch`, plus on PRs that touch the checker itself. * `just mirror-sync-check` and `just mirror-sync-check-test`. **No credentials anywhere.** Both forges are public, so the check reads them anonymously, and alerting goes through the existing in-cluster `signoz-telegram` mapper which holds the Telegram identity itself. The job carries no secret at all. ## Proven in both directions A monitor that has only ever passed is unproven, so the failure path was exercised against live endpoints before landing. ``` $ just mirror-sync-check ok umbra match 8e46ec6c5676 ok mcp-beaver match eebbe24e40c0 ok agent-compose match 5c37622422f9 3/3 mirrors in sync exit=0 ``` ``` $ ... --targets <crossed and missing pair> FAIL deliberately-crossed diverged source 8e46ec6c5676 != mirror eebbe24e40c0 FAIL mirror-does-not-exist unreadable source: ok; mirror: fatal: repository '...' not found 0/2 mirrors in sync exit=1 ``` 12 unit tests, all green, covering divergence, both unreadable directions, timeout, and malformed target lists. ## Two deliberate choices worth reviewing **The target list is static rather than derived.** Forgejo's push-mirror API is the owning source, but reading it needs an admin token, and a credential-free check was judged worth more than a derived list. The cost is that a new mirror goes unwatched until listed, which `mirror-targets.json` states in its own comment. Reverse this if the drift risk outweighs the credential. **It refuses when `ls-remote` answers with more than one ref.** `git ls-remote <repo> main` matches any ref ending in `main`, and a stray mirror ref did exactly that on mcp-beaver, doubling the answer and corrupting 25 files downstream (#963). This asks for the full ref name and refuses a multi-ref answer rather than taking the first line. ## Not included No automatic repair. #440 kept the read-only-detector split deliberately, and repair stays an operator action. Closes coilysiren/inbox#441

coilysiren added 1 commit

2026-08-27 10:20:10 +00:00

feat(mirrors): verify push mirrors from the receiving side, on a schedule

Mirror sync / Compare mirrors against their source (pull_request) Successful in 21s

Details

TruffleHog / Scan for secrets (pull_request) Successful in 6s

Details

CI / lint (pull_request) Successful in 59s

Details

9e3380a337

Nothing watched the Tangled mirrors. A push mirror can fail and keep
failing with no outward sign, which is the failure coilysiren/inbox#440
named as the reason the server-side shape was chosen at all.

The obvious check is the wrong one. Forgejo's `last_update` timestamps the
attempt rather than the outcome: on 2026-08-27 all three mirrors advanced
it while every push was being rejected for an unregistered key. And a knot
running an outdated version drops pushes without reporting an error at
all, so `last_error` can stay empty through total data loss. A monitor
built on either field reports healthy through the exact failure it exists
to catch, which is worse than no monitor.

So this asks the receiving side instead, comparing refs/heads/main on both
forges with anonymous ls-remote. Both are public, so the job carries no
credential. It runs every six hours and alerts through the existing
signoz-telegram mapper, which needs no token either.

Proven in both directions before landing: 3/3 green against the live
mirrors, and a deliberately crossed pair plus a missing mirror both
detected with exit 1. A monitor that has only ever passed is unproven.

The target list is a static file rather than derived from Forgejo's
push-mirror API, because that API needs an admin token and the whole point
is a credential-free check. The trade is that a new mirror goes unwatched
until listed, which the file says in its own comment.

Closes coilysiren/inbox#441

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Agent-Role: sysadmin

coilysiren merged commit cc9a6296d5 into main

2026-08-27 10:22:10 +00:00

coilysiren referenced this pull request from a commit

2026-08-27 10:22:11 +00:00

feat(mirrors): verify push mirrors from the receiving side, on a schedule (#967)

coilyco-ops referenced this pull request

2026-08-27 10:34:52 +00:00

24 of 25 test verbs never run in CI, so most of this repo's tests only rot #969