coilyco-flight-deck/infrastructure

Watch

forgejo-runner-recycle comment says 09:15 UTC, the job actually fires 09:15 local #869

New issue

Open

opened 2026-08-19 05:22:14 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-19 05:22:14 +00:00

Owner

deploy/forgejo-runner-recycle.yml sets schedule: "15 9 * * *" with no spec.timeZone, so kube-controller-manager interprets it in the controller's local zone rather than UTC.

Observed in-cluster:

kubectl get cronjob forgejo-runner-recycle -n forgejo \
  -o custom-columns='SCHEDULE:.spec.schedule,TZ:.spec.timeZone,LAST:.status.lastScheduleTime'

SCHEDULE     TZ       LAST
15 9 * * *   <none>   2026-08-18T16:15:00Z

16:15Z is 09:15 PDT, confirming local interpretation. Job pod names agree: the last three recycle runs are all stamped 16:15:01Z.

The header comment claims:

09:15 UTC is overnight local with no scheduled workloads nearby.

Both halves are off. The schedule is not UTC, and 09:15 local is mid-morning rather than overnight, so the stated reason for choosing the slot does not hold either way. Someone reading the comment to decide whether a change is safe would reason from the wrong window.

This is a documentation and intent bug, not a functional one. The job works. But the slot was chosen for a property it does not have, so the choice should be made deliberately rather than left where a wrong comment put it.

Worth noting for scheduling: a 09:15 local fire lands 1 hour 45 minutes before an 11:00 local start. That is fine for the runners themselves, since they re-register in well under a minute, but it means CI is briefly rolling during any late morning prep window.

Acceptance

Either set spec.timeZone: Etc/UTC and keep 09:15 UTC if overnight-local was the real intent, or keep local time and correct the comment to say which zone it means and why that slot. Either way the comment and the behavior agree afterward.

`deploy/forgejo-runner-recycle.yml` sets `schedule: "15 9 * * *"` with no `spec.timeZone`, so kube-controller-manager interprets it in the controller's local zone rather than UTC. Observed in-cluster: kubectl get cronjob forgejo-runner-recycle -n forgejo \ -o custom-columns='SCHEDULE:.spec.schedule,TZ:.spec.timeZone,LAST:.status.lastScheduleTime' SCHEDULE TZ LAST 15 9 * * * <none> 2026-08-18T16:15:00Z 16:15Z is 09:15 PDT, confirming local interpretation. Job pod names agree: the last three recycle runs are all stamped 16:15:01Z. The header comment claims: > 09:15 UTC is overnight local with no scheduled workloads nearby. Both halves are off. The schedule is not UTC, and 09:15 local is mid-morning rather than overnight, so the stated reason for choosing the slot does not hold either way. Someone reading the comment to decide whether a change is safe would reason from the wrong window. This is a documentation and intent bug, not a functional one. The job works. But the slot was chosen for a property it does not have, so the choice should be made deliberately rather than left where a wrong comment put it. Worth noting for scheduling: a 09:15 local fire lands 1 hour 45 minutes before an 11:00 local start. That is fine for the runners themselves, since they re-register in well under a minute, but it means CI is briefly rolling during any late morning prep window. ## Acceptance Either set `spec.timeZone: Etc/UTC` and keep 09:15 UTC if overnight-local was the real intent, or keep local time and correct the comment to say which zone it means and why that slot. Either way the comment and the behavior agree afterward.

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:07 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:58 +00:00

coilyco-ops referenced this issue

2026-08-28 21:32:38 +00:00

Backlog burn-down 2026-08-28: what was closed and why #981

coilyco-ops referenced this issue

2026-08-29 04:22:25 +00:00

forgejo-runner-recycle restarts all 14 deploy runners simultaneously, producing a ~45 MB/s disk burst #840

coilyco-ops commented

2026-08-29 04:22:27 +00:00

Author

Owner

Confirmed at the object, 2026-08-29 ~04:20Z. The manifest and the wall clock disagree by seven hours.

Read directly from the kai-server CronJob:

namespace:        forgejo
cronjob:          forgejo-runner-recycle
schedule:         15 9 * * *
time_zone:        null
concurrency:      Forbid
last_schedule_at: 2026-08-28T16:15:00+00:00
last_successful:  2026-08-28T16:15:05+00:00

The schedule says 09:15 and the controller fired it at 16:15Z. With time_zone: null the schedule resolves against the controller's local zone, so kai-server at UTC-7 turns 15 9 * * * into 16:15Z.

ser8 runs the same manifest and its recycle Jobs land at 09:15:00Z exactly, three consecutive days checked (2026-08-26, -27, -28). So ser8 resolves the same expression in UTC.

That is the bug this issue names, confirmed from both ends rather than inferred from one. Two clusters, one tracked schedule, seven hours apart, and neither wall-clock time is the one written in the file.

Why it matters more than a cosmetic drift

Both recycles are thundering herds, per #840. The seven-hour offset is currently the only thing keeping the two herds off the same clock, and it is an accident of an unpinned timezone rather than a chosen stagger. Setting time_zone to fix this issue, without touching #840 first, would converge both clusters onto one instant and make the herd problem worse. The two issues have to move in that order.

Stale objects noticed alongside

Three forgejo-runner-recycle Jobs from 2026-07-21, -22 and -23 persist in forgejo with failed: 2 and BackoffLimitExceeded, 36 to 38 days old. Not related to the timezone, and not worth its own issue, but they are why a Job list for this CronJob reads as failing at a glance when the last four runs all succeeded.

## Confirmed at the object, 2026-08-29 ~04:20Z. The manifest and the wall clock disagree by seven hours. Read directly from the kai-server CronJob: ``` namespace: forgejo cronjob: forgejo-runner-recycle schedule: 15 9 * * * time_zone: null concurrency: Forbid last_schedule_at: 2026-08-28T16:15:00+00:00 last_successful: 2026-08-28T16:15:05+00:00 ``` **The schedule says 09:15 and the controller fired it at 16:15Z.** With `time_zone: null` the schedule resolves against the controller's local zone, so kai-server at UTC-7 turns `15 9 * * *` into 16:15Z. ser8 runs the same manifest and its recycle Jobs land at **09:15:00Z** exactly, three consecutive days checked (2026-08-26, -27, -28). So ser8 resolves the same expression in UTC. That is the bug this issue names, confirmed from both ends rather than inferred from one. Two clusters, one tracked schedule, seven hours apart, and neither wall-clock time is the one written in the file. ### Why it matters more than a cosmetic drift Both recycles are thundering herds, per #840. The seven-hour offset is currently the only thing keeping the two herds off the same clock, and **it is an accident of an unpinned timezone rather than a chosen stagger.** Setting `time_zone` to fix this issue, without touching #840 first, would converge both clusters onto one instant and make the herd problem worse. The two issues have to move in that order. ### Stale objects noticed alongside Three `forgejo-runner-recycle` Jobs from 2026-07-21, -22 and -23 persist in `forgejo` with `failed: 2` and `BackoffLimitExceeded`, 36 to 38 days old. Not related to the timezone, and not worth its own issue, but they are why a Job list for this CronJob reads as failing at a glance when the last four runs all succeeded.

coilyco-ops added this to the Forgejo runners milestone

2026-08-29 05:40:50 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#869

No description provided.

Rows
Columns