Forgejo daily logout: 7d session TTL not rolled out + kubectl kai-server.local Bad Gateway #350

New issue

Closed

opened 2026-06-17 06:11:16 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-06-17 06:11:16 +00:00

Owner

Summary

Forgejo logs Kai out roughly once a day, per browser. The "once a day" cadence is the exact signature of Forgejo's default [session] SESSION_LIFE_TIME = 86400 (1 day) still being live, i.e. the 7-day fix from 1da5986 was committed but never rolled out to the running pod.

This issue covers two distinct problems found while diagnosing:

1. Forgejo 7-day session TTL committed but (likely) not deployed

Symptom - logged out ~daily per browser. Login is via external OAuth (SSO), but Forgejo still holds its own i_like_gitea web session after the OAuth dance, so its SESSION_LIFE_TIME governs the daily bounce-back to the IdP.
Fix already in git - deploy/forgejo.yml:226 sets FORGEJO__session__SESSION_LIFE_TIME = "604800" and pins FORGEJO__session__PROVIDER = db. Landed in 1da5986 (2026-05-25, "fix(forgejo): pin session provider to postgres with 7d TTL", closing the old #127).
Why it persists - per this repo's rule "a commit to main is not a deploy", the manifest only reaches kai-server on apply. The 3-week-old fix plus a still-daily symptom points to the live pod running the default 86400.
To confirm + fix (blocked today by problem 2):
- coily ops kubectl -n forgejo get deploy forgejo -o yaml | grep -A1 -iE "SESSION_LIFE_TIME|session__PROVIDER" - empty or 86400 confirms the bug.
- coily ops kubectl apply -f deploy/forgejo.yml then coily ops kubectl -n forgejo rollout status deploy/forgejo.
Follow-up if it recurs after rollout - check the upstream OAuth IdP session/token lifetime. A ~1-day IdP session would re-prompt regardless of Forgejo's TTL. Secondary, since 86400 matches the symptom exactly.

2. kubectl hits `Bad Gateway` because kai-server is sometimes `kai-server.local`

Symptom - coily ops kubectl ... against https://kai-server.local:6443/api returns Bad Gateway on every retry from kais-macbook-pro. Forgejo itself is up, so this is the control-plane API route, not the app.
Root cause (Kai) - kai-server resolves as kai-server.local (mDNS) sometimes, which is flaky / lands on the wrong endpoint. Same class of bug already hit and fixed for Tailscale.
Existing ansible config - ansible/roles/lan-hosts/defaults/main.yml pins { name: kai-server, ip: 192.168.0.194 }. The fix is to make the kubeconfig server use the stable kai-server name (or the LAN IP) rather than the .local mDNS name, mirroring the Tailscale resolution fix.
Impact - blocks confirming/applying problem 1 from the macbook until resolved.

Acceptance

Live forgejo pod runs SESSION_LIFE_TIME=604800 with PROVIDER=db, verified via kubectl.
Kai stays logged in across days (within the 7-day window) in each browser.
coily ops kubectl reaches the cluster reliably from the macbook without .local-induced Bad Gateway.

## Summary Forgejo logs Kai out roughly **once a day, per browser**. The "once a day" cadence is the exact signature of Forgejo's default `[session] SESSION_LIFE_TIME = 86400` (1 day) still being live, i.e. the 7-day fix from `1da5986` was committed but never rolled out to the running pod. This issue covers two distinct problems found while diagnosing: ## 1. Forgejo 7-day session TTL committed but (likely) not deployed * **Symptom** - logged out ~daily per browser. Login is via external OAuth (SSO), but Forgejo still holds its own `i_like_gitea` web session after the OAuth dance, so its `SESSION_LIFE_TIME` governs the daily bounce-back to the IdP. * **Fix already in git** - `deploy/forgejo.yml:226` sets `FORGEJO__session__SESSION_LIFE_TIME = "604800"` and pins `FORGEJO__session__PROVIDER = db`. Landed in `1da5986` (2026-05-25, "fix(forgejo): pin session provider to postgres with 7d TTL", closing the old #127). * **Why it persists** - per this repo's rule "a commit to main is not a deploy", the manifest only reaches kai-server on apply. The 3-week-old fix plus a still-daily symptom points to the live pod running the default 86400. * **To confirm + fix** (blocked today by problem 2): * `coily ops kubectl -n forgejo get deploy forgejo -o yaml | grep -A1 -iE "SESSION_LIFE_TIME|session__PROVIDER"` - empty or `86400` confirms the bug. * `coily ops kubectl apply -f deploy/forgejo.yml` then `coily ops kubectl -n forgejo rollout status deploy/forgejo`. * **Follow-up if it recurs after rollout** - check the upstream OAuth IdP session/token lifetime. A ~1-day IdP session would re-prompt regardless of Forgejo's TTL. Secondary, since 86400 matches the symptom exactly. ## 2. kubectl hits `Bad Gateway` because kai-server is sometimes `kai-server.local` * **Symptom** - `coily ops kubectl ...` against `https://kai-server.local:6443/api` returns `Bad Gateway` on every retry from kais-macbook-pro. Forgejo itself is up, so this is the control-plane API route, not the app. * **Root cause (Kai)** - kai-server resolves as `kai-server.local` (mDNS) sometimes, which is flaky / lands on the wrong endpoint. Same class of bug already hit and fixed for Tailscale. * **Existing ansible config** - `ansible/roles/lan-hosts/defaults/main.yml` pins `{ name: kai-server, ip: 192.168.0.194 }`. The fix is to make the kubeconfig server use the stable `kai-server` name (or the LAN IP) rather than the `.local` mDNS name, mirroring the Tailscale resolution fix. * **Impact** - blocks confirming/applying problem 1 from the macbook until resolved. ## Acceptance * Live forgejo pod runs `SESSION_LIFE_TIME=604800` with `PROVIDER=db`, verified via kubectl. * Kai stays logged in across days (within the 7-day window) in each browser. * `coily ops kubectl` reaches the cluster reliably from the macbook without `.local`-induced Bad Gateway.

coilysiren commented

2026-06-17 06:21:05 +00:00

Author

Owner

Update after restoring cluster access

Part 2 (kubectl Bad Gateway) - root-caused and fixed.

The mac is currently remote, not on the home LAN. kai-server.local is mDNS (LAN-only) and was not resolving at all from off-LAN, while the kubeconfig pointed at https://kai-server.local:6443 - that is the failure.
Bare kai-server resolves via Tailscale MagicDNS to the tailnet IP and is reachable when remote. On the home LAN that same tailnet IP is blackholed (infrastructure#294), so there the only reliable address is the LAN IP 192.168.0.194.
Fix applied to ~/.kube/config: server changed from https://kai-server.local:6443 to https://kai-server:6443. tls-server-name: kai-server was already set, so the API cert validates with no --insecure-skip-tls-verify. Verified: kubectl get nodes returns kai-server Ready through the plain context.
Durable home support: enabled the lan-hosts role (commit b0e6846 on main) so /etc/hosts pins kai-server -> 192.168.0.194 when home and falls back to MagicDNS away. One kubeconfig now works in both locations.

Part 1 (daily logout) - my "not rolled out" theory was wrong.

With kubectl restored I read the live deploy: it already has FORGEJO__session__SESSION_LIFE_TIME=604800 and FORGEJO__session__PROVIDER=db. The 7-day fix from 1da5986 is deployed.
So the daily logout is not Forgejo's session TTL. Since login is external OAuth (SSO), the prime remaining suspect is the upstream IdP's session/token lifetime (~1 day), which would bounce her back to re-auth even while Forgejo's own session is still valid.
Next step: identify the OAuth source (coily ops forgejo admin auth list) and check its session length / max_age.

infrastructure#303 note. The kai-server.local + --insecure-skip-tls-verify workaround in fleet-readout.sh can be replaced with kai-server + tls-server-name=kai-server (cert-valid) once lan-hosts converges at home. The "add .local to the k3s tls-san" premise in #303 is now moot - use the cert-valid name instead.

### Update after restoring cluster access **Part 2 (kubectl Bad Gateway) - root-caused and fixed.** * The mac is currently remote, not on the home LAN. `kai-server.local` is mDNS (LAN-only) and was not resolving at all from off-LAN, while the kubeconfig pointed at `https://kai-server.local:6443` - that is the failure. * Bare `kai-server` resolves via Tailscale MagicDNS to the tailnet IP and is reachable when remote. On the home LAN that same tailnet IP is blackholed (infrastructure#294), so there the only reliable address is the LAN IP `192.168.0.194`. * Fix applied to `~/.kube/config`: `server` changed from `https://kai-server.local:6443` to `https://kai-server:6443`. `tls-server-name: kai-server` was already set, so the API cert validates with no `--insecure-skip-tls-verify`. Verified: `kubectl get nodes` returns `kai-server Ready` through the plain context. * Durable home support: enabled the `lan-hosts` role (commit `b0e6846` on main) so `/etc/hosts` pins `kai-server -> 192.168.0.194` when home and falls back to MagicDNS away. One kubeconfig now works in both locations. **Part 1 (daily logout) - my "not rolled out" theory was wrong.** * With kubectl restored I read the live deploy: it already has `FORGEJO__session__SESSION_LIFE_TIME=604800` and `FORGEJO__session__PROVIDER=db`. The 7-day fix from `1da5986` **is** deployed. * So the daily logout is **not** Forgejo's session TTL. Since login is external OAuth (SSO), the prime remaining suspect is the **upstream IdP's session/token lifetime** (~1 day), which would bounce her back to re-auth even while Forgejo's own session is still valid. * Next step: identify the OAuth source (`coily ops forgejo admin auth list`) and check its session length / `max_age`. **infrastructure#303 note.** The `kai-server.local` + `--insecure-skip-tls-verify` workaround in `fleet-readout.sh` can be replaced with `kai-server` + `tls-server-name=kai-server` (cert-valid) once `lan-hosts` converges at home. The "add `.local` to the k3s tls-san" premise in #303 is now moot - use the cert-valid name instead.

coilysiren commented

2026-06-17 06:28:41 +00:00

Author

Owner

Part 1 (daily logout) - root cause found, it is an upstream Forgejo limitation

Not a misconfiguration. Two facts combine:

The Forgejo session cookie carries no Max-Age - it is browser-session scoped. Confirmed live against the OAuth-start endpoint:
set-cookie: session=...; Path=/; HttpOnly; SameSite=Lax (no Max-Age, no Expires).
SESSION_LIFE_TIME=604800 only sets how long the server keeps the session row (verified: the session table holds rows whose last-activity is 27h+ old, so server retention is fine). It does not put an expiry on the cookie. See go-gitea/gitea#33403.
Login is GitHub OAuth, and OAuth2/SSO logins never set the persistent remember-me cookie (gitea_incredible). That cookie - the only thing that survives a browser-session end and silently re-establishes login - is set solely by the username/password form with "Remember Me" ticked. The login page actively clears it (persistent=; Max-Age=0). Upstream bug: forgejo/forgejo#6007 (closed, no code fix; maintainer "not an easy fix"), go-gitea/gitea#28394.

Net: server keeps the session 7 days, but the browser drops the session-scoped cookie when it ends its session (quit / overnight), and SSO gives no persistent cookie to fall back on -> re-auth ~daily, per browser. Exactly the reported symptom.

Local password + "Remember Me" (recommended, reliable). Set a Forgejo password on the account and log in via the username/password form with Remember Me. That sets gitea_incredible for LOGIN_REMEMBER_DAYS. The GitHub link can stay for identity. Pair with pinning FORGEJO__security__LOGIN_REMEMBER_DAYS (currently unset -> default) in deploy/forgejo.yml for clarity.
Keep SSO, rewrite the cookie at the ingress. Have Traefik add a Max-Age/Expires to the Forgejo session Set-Cookie, converting the session cookie to persistent. Keeps the SSO UX but means tampering with auth cookies at the proxy (SameSite/Secure care needed). More fragile.
Browser-side: keep the browser from clearing session cookies on quit. Not a real fix, browser-dependent.
Upstream: #6007 is closed without a fix; would need a contribution or a future Forgejo version.

Also spotted while reading the running config

[session] COOKIE_SECURE = false is still live (the running app.ini confirms it), even though the public HTTPS ingress is up. docs/forgejo-deploy-plan.md flags this env block as phase-1-only and says to remove it in phase 2. Worth dropping FORGEJO__session__COOKIE_SECURE=false from deploy/forgejo.yml now that the public surface exists - separate from the logout issue, but a live cookie-security weakening.

### Part 1 (daily logout) - root cause found, it is an upstream Forgejo limitation Not a misconfiguration. Two facts combine: 1. **The Forgejo session cookie carries no `Max-Age`** - it is browser-session scoped. Confirmed live against the OAuth-start endpoint: `set-cookie: session=...; Path=/; HttpOnly; SameSite=Lax` (no Max-Age, no Expires). `SESSION_LIFE_TIME=604800` only sets how long the **server** keeps the session row (verified: the `session` table holds rows whose last-activity is 27h+ old, so server retention is fine). It does **not** put an expiry on the cookie. See go-gitea/gitea#33403. 2. **Login is GitHub OAuth, and OAuth2/SSO logins never set the persistent remember-me cookie** (`gitea_incredible`). That cookie - the only thing that survives a browser-session end and silently re-establishes login - is set solely by the username/password form with "Remember Me" ticked. The login page actively clears it (`persistent=; Max-Age=0`). Upstream bug: forgejo/forgejo#6007 (closed, no code fix; maintainer "not an easy fix"), go-gitea/gitea#28394. Net: server keeps the session 7 days, but the browser drops the session-scoped cookie when it ends its session (quit / overnight), and SSO gives no persistent cookie to fall back on -> re-auth ~daily, per browser. Exactly the reported symptom. ### Fix options (a login-UX decision, not a pure config change) 1. **Local password + "Remember Me"** (recommended, reliable). Set a Forgejo password on the account and log in via the username/password form with Remember Me. That sets `gitea_incredible` for `LOGIN_REMEMBER_DAYS`. The GitHub link can stay for identity. Pair with pinning `FORGEJO__security__LOGIN_REMEMBER_DAYS` (currently unset -> default) in `deploy/forgejo.yml` for clarity. 2. **Keep SSO, rewrite the cookie at the ingress.** Have Traefik add a `Max-Age`/`Expires` to the Forgejo session `Set-Cookie`, converting the session cookie to persistent. Keeps the SSO UX but means tampering with auth cookies at the proxy (SameSite/Secure care needed). More fragile. 3. **Browser-side**: keep the browser from clearing session cookies on quit. Not a real fix, browser-dependent. 4. **Upstream**: #6007 is closed without a fix; would need a contribution or a future Forgejo version. ### Also spotted while reading the running config `[session] COOKIE_SECURE = false` is still live (the running `app.ini` confirms it), even though the public HTTPS ingress is up. `docs/forgejo-deploy-plan.md` flags this env block as phase-1-only and says to remove it in phase 2. Worth dropping `FORGEJO__session__COOKIE_SECURE=false` from `deploy/forgejo.yml` now that the public surface exists - separate from the logout issue, but a live cookie-security weakening.

coilysiren commented

2026-06-17 06:34:30 +00:00

Author

Owner

Resolution summary - closing

1. Daily logout (the original report) - root-caused to an upstream Forgejo limitation (forgejo/forgejo#6007, go-gitea/gitea#33403): the session cookie is browser-session scoped (no Max-Age) and OAuth2/SSO logins never set the persistent remember-me cookie. SESSION_LIFE_TIME=604800 is live and correct but only governs server-side retention. Decision: leave as-is (re-login is a one-click GitHub redirect), documented here. Not fixable by config; would need the upstream change or switching to local-password + Remember Me login.

2. kubectl Bad Gateway - fixed. ~/.kube/config now targets https://kai-server:6443 (was kai-server.local, flaky mDNS) with the existing tls-server-name: kai-server keeping TLS valid. lan-hosts ansible role enabled (b0e6846) so /etc/hosts pins kai-server -> 192.168.0.194 at home (bypassing the same-LAN tailnet blackhole, #294) and falls back to MagicDNS away. Verified kubectl get nodes works from the macbook.

3. COOKIE_SECURE=false leftover - fixed. The phase-1 env had been removed but the value persisted in the PVC-resident app.ini (env-to-ini only upserts). Set FORGEJO__session__COOKIE_SECURE=auto explicitly (1f755a9), applied + rolled out. Verified: app.ini now COOKIE_SECURE = auto and the live Set-Cookie over HTTPS now carries Secure. Corrected the misleading phase-2 step in docs/forgejo-deploy-plan.md (a00889b).

All three addressed. Closing.

### Resolution summary - closing **1. Daily logout (the original report)** - root-caused to an upstream Forgejo limitation (forgejo/forgejo#6007, go-gitea/gitea#33403): the session cookie is browser-session scoped (no `Max-Age`) and OAuth2/SSO logins never set the persistent remember-me cookie. `SESSION_LIFE_TIME=604800` is live and correct but only governs server-side retention. Decision: **leave as-is** (re-login is a one-click GitHub redirect), documented here. Not fixable by config; would need the upstream change or switching to local-password + Remember Me login. **2. kubectl `Bad Gateway`** - fixed. `~/.kube/config` now targets `https://kai-server:6443` (was `kai-server.local`, flaky mDNS) with the existing `tls-server-name: kai-server` keeping TLS valid. `lan-hosts` ansible role enabled (`b0e6846`) so `/etc/hosts` pins `kai-server -> 192.168.0.194` at home (bypassing the same-LAN tailnet blackhole, #294) and falls back to MagicDNS away. Verified `kubectl get nodes` works from the macbook. **3. `COOKIE_SECURE=false` leftover** - fixed. The phase-1 env had been removed but the value persisted in the PVC-resident `app.ini` (env-to-ini only upserts). Set `FORGEJO__session__COOKIE_SECURE=auto` explicitly (`1f755a9`), applied + rolled out. Verified: `app.ini` now `COOKIE_SECURE = auto` and the live `Set-Cookie` over HTTPS now carries `Secure`. Corrected the misleading phase-2 step in `docs/forgejo-deploy-plan.md` (`a00889b`). All three addressed. Closing.

coilysiren closed this issue

2026-06-17 06:34:31 +00:00