coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

caddy-shortcuts regenerate job fails on every daily run since 2026-05-25 #380

New issue
Closed
opened 2026-06-19 07:55:05 +00:00 by coilyco-ops · 3 comments
coilyco-ops commented 2026-06-19 07:55:05 +00:00
Member
Copy link

Symptom

The regenerate job in .forgejo/workflows/caddy-shortcuts.yml (daily cron 17 6 * * *) has failed on every scheduled run since 2026-05-25. The only success on record is the very first run (task id 7, 2026-05-25 05:19Z). 12+ consecutive daily failures since.

Recent failing runs: 1265 (eb6e5de), 1080 (b390eb0), 581, 487, 382, 352, 332, 256, 230, 207, 138, 28, 10.

Impact

Tailnet shortcut snippets under caddy/sites/ never regenerate. There are real pending changes stuck behind this: a local dry-run shows two stale snippets that should be deleted but never land:

delete caddy/sites/eco-jobs-tracker.caddy
delete caddy/sites/eco-mcp.caddy
shortcuts: 2 desired / 2 changes

Diagnosis so far

scripts/generate-caddy-shortcuts.py runs clean locally (exit 0) both with and without FORGEJO_TOKEN, so the failure is in the CI environment / workflow, not the generator. Forgejo's task-log API returned 404 for the api-token, so step-level logs were not pulled programmatically.

Likely suspects (in order)

  1. Final git push origin HEAD:main step. Checkout authenticates with secrets.GITHUB_TOKEN; pushing back to main is likely denied. Because changes are always pending (see above), the commit if changed step always reaches the push. This is exactly the commit-pushback fragility explored in #129.
  2. yq install. curl -sSfL ... -o /usr/local/bin/yq may hit permission-denied if the runner container is non-root.
  3. Missing FORGEJO_TOKEN env on the regenerate step. The step never receives the token, so the script scans unauthenticated (15 repos vs 25 with a token). Degraded visibility, not fatal on its own, but worth fixing alongside.

Next steps

Filed from a session that was enabling Forgejo push-to-create; the failure surfaced incidentally while verifying CI on commit ce36173.

## Symptom The `regenerate` job in `.forgejo/workflows/caddy-shortcuts.yml` (daily cron `17 6 * * *`) has failed on **every** scheduled run since 2026-05-25. The only success on record is the very first run (task id 7, 2026-05-25 05:19Z). 12+ consecutive daily failures since. Recent failing runs: 1265 (eb6e5de), 1080 (b390eb0), 581, 487, 382, 352, 332, 256, 230, 207, 138, 28, 10. ## Impact Tailnet shortcut snippets under `caddy/sites/` never regenerate. There are real pending changes stuck behind this: a local dry-run shows two stale snippets that should be deleted but never land: ``` delete caddy/sites/eco-jobs-tracker.caddy delete caddy/sites/eco-mcp.caddy shortcuts: 2 desired / 2 changes ``` ## Diagnosis so far `scripts/generate-caddy-shortcuts.py` runs clean **locally** (exit 0) both with and without `FORGEJO_TOKEN`, so the failure is in the CI environment / workflow, not the generator. Forgejo's task-log API returned 404 for the api-token, so step-level logs were not pulled programmatically. ## Likely suspects (in order) 1. **Final `git push origin HEAD:main` step.** Checkout authenticates with `secrets.GITHUB_TOKEN`; pushing back to `main` is likely denied. Because changes are always pending (see above), the `commit if changed` step always reaches the push. This is exactly the commit-pushback fragility explored in #129. 2. **`yq` install.** `curl -sSfL ... -o /usr/local/bin/yq` may hit permission-denied if the runner container is non-root. 3. **Missing `FORGEJO_TOKEN` env on the regenerate step.** The step never receives the token, so the script scans unauthenticated (15 repos vs 25 with a token). Degraded visibility, not fatal on its own, but worth fixing alongside. ## Next steps - Pull the run logs from the web UI to confirm the failing step: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/actions/runs/1265 - Fix the confirmed step. If it is the pushback, revisit the #129 no-pushback approach or push with a scoped deploy token instead of `GITHUB_TOKEN`. - Once green, the two stale snippet deletions should land automatically. Filed from a session that was enabling Forgejo push-to-create; the failure surfaced incidentally while verifying CI on commit ce36173.
coilyco-ops commented 2026-06-24 05:38:29 +00:00
Author
Member
Copy link

Triage: P2 / headless - caddy-shortcuts regenerate job failing every daily run since 2026-05-25; self-contained bug to fix.

(goose-style pass; claude as the judgment engine, applied 2026-06-23)

<!-- goose-triage --> **Triage:** `P2` / `headless` - caddy-shortcuts regenerate job failing every daily run since 2026-05-25; self-contained bug to fix. (goose-style pass; claude as the judgment engine, applied 2026-06-23)
coilyco-ops commented 2026-06-24 05:39:43 +00:00
Author
Member
Copy link

🔒 Reserved by ward agent claude — container ward-infrastructure-issue-380-claude-45628d06 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-06-24T05:39:42Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

<!-- ward-agent-reservation --> 🔒 Reserved by `ward agent claude` — container `ward-infrastructure-issue-380-claude-45628d06` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-06-24T05:39:42Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`
coilysiren commented 2026-06-24 06:38:24 +00:00
Owner
Copy link

Fixed and verified green (workflow_dispatch run #542). The actual cause was not the push (suspect #1) but the regenerate step exiting non-zero, masked by the runner having no API log endpoint.

Root causes (confirmed from runner logs):

  1. Stale owner. After the coilysiren → coilyco-* split, the script enumerated repo names across all orgs (the search owner filter wants a uid, so it's ignored) but read each at the hardcoded coilysiren/<name>, which only 301-redirects to coilyco-*. The runner doesn't follow that redirect, so every config read missed. Fixed: list_repos now returns (owner, name) and reads each repo at its real owner.
  2. In-cluster contents reads are blocked. From inside the cluster the Forgejo API serves anonymous /repos/search (200) but 404s anonymous /contents/.... The repo-scoped auto GITHUB_TOKEN can't read siblings either, and the org CI_RELEASE_TOKEN secret is currently rejected (HTTP 401) — its value is invalid/expired. So the runner cannot enumerate sibling shortcuts today.
  3. Data-loss gap in the guardrail. With desired=0 and only 2 snippets, the mass-delete guard (len(to_delete) > max(2, len//2)) didn't fire (2 > 2 is false), so it deleted both and was only saved by the pre-receive hook rejecting the push to main.

What landed (main, commits 99e0f44 → 6fa3946):

  • Read repos at their real owner; drop the coilysiren/ 301 dependency.
  • Reorder reconcile so writes always apply and a suspicious/empty-desired delete is a non-destructive no-op (exit 0) instead of sys.exit(2) — the job stays green and never wipes valid snippets on a failed/unauthenticated scan.
  • Log non-404 fetch failures (the silent swallow hid this for weeks).
  • Point the scan at the in-cluster FORGEJO_URL; checkout stays on the auto GITHUB_TOKEN (the only token the in-cluster runner authenticates for git — a custom PAT fails checkout with could not read Username).
  • Land the two stale snippet deletes (eco-jobs-tracker, eco-mcp) and correct the remaining headers to their real owner.

Run #542 is green: scans, refuses to wipe, 0 changes, succeeds.

Residual (needs operator action): the job is currently a safe no-op — it can't actually regenerate until a valid read-capable scan token is wired in as FORGEJO_TOKEN (the existing CI_RELEASE_TOKEN 401s and needs re-provisioning via scripts/provision-ci-release-token.sh). The snippets are correct and preserved in the meantime. Pushing regenerated changes to main from CI also still hits the pre-receive wall (the #129 fragility). Filed/decided separately.

Fixed and verified green (workflow_dispatch run #542). The actual cause was **not** the push (suspect #1) but the `regenerate` step exiting non-zero, masked by the runner having no API log endpoint. **Root causes (confirmed from runner logs):** 1. **Stale owner.** After the coilysiren → coilyco-* split, the script enumerated repo *names* across all orgs (the search `owner` filter wants a uid, so it's ignored) but read each at the hardcoded `coilysiren/<name>`, which only 301-redirects to `coilyco-*`. The runner doesn't follow that redirect, so every config read missed. Fixed: `list_repos` now returns `(owner, name)` and reads each repo at its real owner. 2. **In-cluster contents reads are blocked.** From inside the cluster the Forgejo API serves anonymous `/repos/search` (200) but **404s anonymous `/contents/...`**. The repo-scoped auto `GITHUB_TOKEN` can't read siblings either, and the org `CI_RELEASE_TOKEN` secret is currently **rejected (HTTP 401)** — its value is invalid/expired. So the runner cannot enumerate sibling shortcuts today. 3. **Data-loss gap in the guardrail.** With `desired=0` and only 2 snippets, the mass-delete guard (`len(to_delete) > max(2, len//2)`) didn't fire (2 > 2 is false), so it deleted both and was only saved by the `pre-receive` hook rejecting the push to `main`. **What landed (`main`, commits 99e0f44 → 6fa3946):** - Read repos at their real owner; drop the `coilysiren/` 301 dependency. - Reorder `reconcile` so writes always apply and a suspicious/empty-`desired` delete is a **non-destructive no-op (exit 0)** instead of `sys.exit(2)` — the job stays green and never wipes valid snippets on a failed/unauthenticated scan. - Log non-404 fetch failures (the silent swallow hid this for weeks). - Point the scan at the in-cluster `FORGEJO_URL`; checkout stays on the auto `GITHUB_TOKEN` (the only token the in-cluster runner authenticates for git — a custom PAT fails checkout with `could not read Username`). - Land the two stale snippet deletes (`eco-jobs-tracker`, `eco-mcp`) and correct the remaining headers to their real owner. Run #542 is green: scans, refuses to wipe, `0 changes`, succeeds. **Residual (needs operator action):** the job is currently a safe **no-op** — it can't actually regenerate until a *valid* read-capable scan token is wired in as `FORGEJO_TOKEN` (the existing `CI_RELEASE_TOKEN` 401s and needs re-provisioning via `scripts/provision-ci-release-token.sh`). The snippets are correct and preserved in the meantime. Pushing regenerated changes to `main` from CI also still hits the `pre-receive` wall (the #129 fragility). Filed/decided separately.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#380
No description provided.