coilyco-flight-deck/infrastructure

Watch

Raise the Forgejo registry request-body timeout above Traefik's 60-second default #646

New issue

Open

opened 2026-07-25 19:38:31 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-25 19:38:31 +00:00

Member

Problem

Large Forgejo OCI uploads are nondeterministic because the public Forgejo Ingress inherits Traefik's 60-second request-body read timeout.

The AOS dev-base publish for commit 21bc338e89d3d051c37545484af95822be3cca9e built both architectures successfully, then its first run stalled after registry upload failures. One guarded same-ref retry eventually published and verified the required multi-architecture image, but its optional cache export still hit the same timeout.

Source runs:

Evidence

Live Traefik is v3.3.2.
Neither web nor websecure configures transport.respondingTimeouts.readTimeout.
Helm values add no read-timeout override.
The Forgejo Ingress routes through Traefik to service port 80 with rate-limit and HSTS middlewares. No ServersTransport is present.
Traefik documents the absent readTimeout default as 60 seconds for reading the entire request body.
Forgejo recorded failed blob PUTs at 60,000.0 to 60,000.1 milliseconds with unexpected EOF, followed by upload offset mismatches.
Forgejo had about 70.75 GiB free. Capacity was not the cause.
Recent failed-session scratch under the exact package-upload path reached about 33.27 GiB. The bounded cleanup CronJob was healthy, but those files were younger than its 180-minute safety window.

Official behavior: https://doc.traefik.io/traefik/v3.3/routing/entrypoints/

Completion

Choose and document a bounded request-body timeout that admits the largest expected Forgejo registry blob without disabling slow-client protection globally.
Author the setting at the infrastructure layer that owns Traefik configuration.
Apply it through the established operator path.
Verify a large multi-architecture Forgejo registry push and cache export complete without a 60-second EOF, 499, or offset mismatch.
Confirm failed-upload scratch returns to its normal bounded level after the cleanup window.

## Problem Large Forgejo OCI uploads are nondeterministic because the public Forgejo Ingress inherits Traefik's 60-second request-body read timeout. The AOS dev-base publish for commit `21bc338e89d3d051c37545484af95822be3cca9e` built both architectures successfully, then its first run stalled after registry upload failures. One guarded same-ref retry eventually published and verified the required multi-architecture image, but its optional cache export still hit the same timeout. Source runs: * https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2114 * https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2116 * https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/754 ## Evidence * Live Traefik is v3.3.2. * Neither `web` nor `websecure` configures `transport.respondingTimeouts.readTimeout`. * Helm values add no read-timeout override. * The Forgejo Ingress routes through Traefik to service port 80 with rate-limit and HSTS middlewares. No ServersTransport is present. * Traefik documents the absent `readTimeout` default as 60 seconds for reading the entire request body. * Forgejo recorded failed blob PUTs at 60,000.0 to 60,000.1 milliseconds with `unexpected EOF`, followed by upload offset mismatches. * Forgejo had about 70.75 GiB free. Capacity was not the cause. * Recent failed-session scratch under the exact package-upload path reached about 33.27 GiB. The bounded cleanup CronJob was healthy, but those files were younger than its 180-minute safety window. Official behavior: https://doc.traefik.io/traefik/v3.3/routing/entrypoints/ ## Completion * Choose and document a bounded request-body timeout that admits the largest expected Forgejo registry blob without disabling slow-client protection globally. * Author the setting at the infrastructure layer that owns Traefik configuration. * Apply it through the established operator path. * Verify a large multi-architecture Forgejo registry push and cache export complete without a 60-second EOF, 499, or offset mismatch. * Confirm failed-upload scratch returns to its normal bounded level after the cleanup window.

coilyco-ops added the

autonomy

live-collab

label

2026-07-25 19:38:37 +00:00

coilyco-ops referenced this issue from coilyco-flight-deck/agentic-os

2026-07-25 19:38:44 +00:00

Retry dev-base publish after stalled registry upload #754

coilyco-ops referenced this issue

2026-07-26 05:55:54 +00:00

Make Forgejo runner scratch classification and recycle busy-aware #650

coilyco-ops referenced this issue

2026-07-27 03:02:14 +00:00

Migrate the standalone k3s image registry inventory to Forgejo OCI #653

coilyco-ops commented

2026-07-27 03:08:54 +00:00

Author

Member

Fresh recurrence after the aos manifest-inspection timeout fix landed:

https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2161
The checkpoint probe returned immediately, and BuildKit completed the real dual-architecture build.
The initial export and all three cached retries failed uploading the same large blob. Forgejo returned 499 Client Closed Request for each PUT.
The workflow exhausted its four-attempt build budget and failed after 31m43s.
Repository CI, branch promotion, the AOS CLI release, and the GitHub mirror all passed. The registry upload was the only failed surface.

This run cleanly separates the open Traefik request-body timeout from the repaired Buildx manifest probe and provides a fresh verification target for this issue.

Fresh recurrence after the aos manifest-inspection timeout fix landed: * https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/actions/runs/2161 * The checkpoint probe returned immediately, and BuildKit completed the real dual-architecture build. * The initial export and all three cached retries failed uploading the same large blob. Forgejo returned `499 Client Closed Request` for each PUT. * The workflow exhausted its four-attempt build budget and failed after 31m43s. * Repository CI, branch promotion, the AOS CLI release, and the GitHub mirror all passed. The registry upload was the only failed surface. This run cleanly separates the open Traefik request-body timeout from the repaired Buildx manifest probe and provides a fresh verification target for this issue.

coilyco-ops commented

2026-07-28 08:06:53 +00:00

Author

Member

Timeout remediation is live.

Infrastructure commit 9fcef40 authors a bounded 300-second websecure request-body timeout in the tracked k3s-config role and preserves the operator-local load-balancer address through SSM. The guarded live apply rolled Traefik successfully.
Live Traefik reports --entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s. The Service retained its load-balancer setting and external ingress, and Forgejo remained ready.
AOS dev-base-publish run 2251 completed both plan-draft and publish-full successfully on dc37f74d. Two formerly failing cache layers each uploaded for 140.0 seconds, the multi-architecture image and cache manifests published, and the post-publish pull verified their checksums.
Traefik and Forgejo recorded zero matching 499, 500, unexpected EOF, or offset-mismatch errors during the verification window.
Infrastructure commit a32d8bb aligns the repo with the released AOS catalog hooks. Lint run 1665 and secret-scan run 1666 both passed.

The cleanup criterion is still time-bound. The 08:00 UTC cleanup job succeeded with eligible_files=0 and remaining_kib=18687328 because the staging files are younger than the configured 180-minute safety window. This issue stays open until the first conclusive cleanup cycle at or after 11:00 UTC confirms that failed-upload scratch returns to its bounded level.

Timeout remediation is live. * Infrastructure commit 9fcef40 authors a bounded 300-second websecure request-body timeout in the tracked k3s-config role and preserves the operator-local load-balancer address through SSM. The guarded live apply rolled Traefik successfully. * Live Traefik reports --entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s. The Service retained its load-balancer setting and external ingress, and Forgejo remained ready. * AOS dev-base-publish run 2251 completed both plan-draft and publish-full successfully on dc37f74d. Two formerly failing cache layers each uploaded for 140.0 seconds, the multi-architecture image and cache manifests published, and the post-publish pull verified their checksums. * Traefik and Forgejo recorded zero matching 499, 500, unexpected EOF, or offset-mismatch errors during the verification window. * Infrastructure commit a32d8bb aligns the repo with the released AOS catalog hooks. Lint run 1665 and secret-scan run 1666 both passed. The cleanup criterion is still time-bound. The 08:00 UTC cleanup job succeeded with eligible_files=0 and remaining_kib=18687328 because the staging files are younger than the configured 180-minute safety window. This issue stays open until the first conclusive cleanup cycle at or after 11:00 UTC confirms that failed-upload scratch returns to its bounded level.

coilyco-ops added the

priority

label

2026-07-29 03:46:33 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#646

No description provided.

Rows
Columns