A 400 is classified as a transport error, retried three times, and returned to the caller as 502 #114

New issue

Closed

opened 2026-08-13 17:17:28 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 17:17:28 +00:00

Member

🤖 Filed by Claude Code on Kai's behalf.

Split out of #113 so the fixes stay separable. Same trace: 1a49200c3bebaed778ef2ac5b79d3d99, 2026-08-13T17:07:07Z (SigNoz: http://ser8:30808/trace/1a49200c3bebaed778ef2ac5b79d3d99).

The payload in that trace was malformed — that is #113's problem. This issue is about what the dispatcher did with the resulting 400, which would be wrong regardless of how the payload got that way.

1. A deterministic client error is retried three times

17:07:08.088  event=dispatch.transport_error  attempt=0  outcome=retry
17:07:09.267  event=dispatch.transport_error  attempt=1  outcome=retry
17:07:10.972  event=dispatch.transport_error  attempt=2

Each attempt carried error: litellm: Client error '400 Bad Request' for url 'http://100.105.159.124:4000/v1/chat/completions'.

A 400 is a statement about the request body. The body is byte-identical on every attempt, so attempts 1 and 2 were guaranteed to fail before they were sent. They burned ~2.9s of the caller's budget and three round-trips of backend capacity to re-learn a settled fact.

The retry is downstream of the classification: dispatch.transport_error is the wrong bucket. Nothing about the transport failed — litellm answered promptly each time, and answered a different request successfully 7.5s later in the same trace.

2. The 400 is reported to the caller as 502

sirens-deep received:

502  {"error":{"message":"sirens-echo/deepseek: all backends failed (litellm: Client
     error '400 Bad Request' ...)","type":"upstream_error"}}

and surfaced model backend unavailable, retry shortly.

The backend was not unavailable. It served rounds 0–4 of this same turn, and served another request 7.5s after this one failed. Collapsing "your request was invalid" into "all backends failed" points the operator at capacity and availability — see #108, #111, #112, all of which are about genuinely unavailable backends — when the actual defect was in the request agent-proxy itself had just constructed.

The AllBackendsFailed wrapper is also literally inaccurate here: litellm's own log records No fallback model group found for original model_group=sirens-echo/deepseek, so no fallback was attempted. One backend rejected one payload.

Why the two halves belong together

Retrying is only sensible for errors that might resolve on their own; reporting 502 is only sensible when the backend is the problem. Both stem from the same missing distinction between a request defect and an infrastructure defect. Fixing one without the other leaves either a fast-but-mislabeled failure or a correctly-labeled one that still costs 3×.

Acceptance

Upstream 4xx is classified as a request error, not dispatch.transport_error, and is not retried. 408, 425, and 429 stay retryable.
The caller receives the upstream status and the upstream error body, not a synthesized 502 / all backends failed.
AllBackendsFailed is raised only when more than one backend was actually attempted.
A malformed-payload rejection is distinguishable from a saturated backend in both the caller's response and the span record.

#113 — the trimmer defect that produced this particular 400; independent fix.
#106 — upstream status not recorded on the agent-proxy span; the same status-fidelity theme, and #106's errorRate concern applies here in reverse (this failure is counted, but as the wrong kind).
#108 — a genuinely saturated backend, which is what 502 backend unavailable should be reserved for.

🤖 Filed by Claude Code on Kai's behalf.

> 🤖 Filed by Claude Code on Kai's behalf. Split out of #113 so the fixes stay separable. Same trace: `1a49200c3bebaed778ef2ac5b79d3d99`, 2026-08-13T17:07:07Z (SigNoz: `http://ser8:30808/trace/1a49200c3bebaed778ef2ac5b79d3d99`). The payload in that trace was malformed — that is #113's problem. This issue is about what the dispatcher did with the resulting 400, which would be wrong regardless of how the payload got that way. ## 1. A deterministic client error is retried three times ``` 17:07:08.088 event=dispatch.transport_error attempt=0 outcome=retry 17:07:09.267 event=dispatch.transport_error attempt=1 outcome=retry 17:07:10.972 event=dispatch.transport_error attempt=2 ``` Each attempt carried `error: litellm: Client error '400 Bad Request' for url 'http://100.105.159.124:4000/v1/chat/completions'`. A 400 is a statement about the request body. The body is byte-identical on every attempt, so attempts 1 and 2 were guaranteed to fail before they were sent. They burned ~2.9s of the caller's budget and three round-trips of backend capacity to re-learn a settled fact. The retry is downstream of the classification: `dispatch.transport_error` is the wrong bucket. Nothing about the transport failed — litellm answered promptly each time, and answered a *different* request successfully 7.5s later in the same trace. ## 2. The 400 is reported to the caller as 502 `sirens-deep` received: ``` 502 {"error":{"message":"sirens-echo/deepseek: all backends failed (litellm: Client error '400 Bad Request' ...)","type":"upstream_error"}} ``` and surfaced `model backend unavailable, retry shortly`. The backend was not unavailable. It served rounds 0–4 of this same turn, and served another request 7.5s after this one failed. Collapsing "your request was invalid" into "all backends failed" points the operator at capacity and availability — see #108, #111, #112, all of which are about genuinely unavailable backends — when the actual defect was in the request agent-proxy itself had just constructed. The `AllBackendsFailed` wrapper is also literally inaccurate here: litellm's own log records `No fallback model group found for original model_group=sirens-echo/deepseek`, so no fallback was attempted. One backend rejected one payload. ## Why the two halves belong together Retrying is only sensible for errors that might resolve on their own; reporting 502 is only sensible when the backend is the problem. Both stem from the same missing distinction between a request defect and an infrastructure defect. Fixing one without the other leaves either a fast-but-mislabeled failure or a correctly-labeled one that still costs 3×. ## Acceptance - Upstream 4xx is classified as a request error, not `dispatch.transport_error`, and is not retried. 408, 425, and 429 stay retryable. - The caller receives the upstream status and the upstream error body, not a synthesized 502 / `all backends failed`. - `AllBackendsFailed` is raised only when more than one backend was actually attempted. - A malformed-payload rejection is distinguishable from a saturated backend in both the caller's response and the span record. ## Related - #113 — the trimmer defect that produced this particular 400; independent fix. - #106 — upstream status not recorded on the agent-proxy span; the same status-fidelity theme, and #106's `errorRate` concern applies here in reverse (this failure *is* counted, but as the wrong kind). - #108 — a genuinely saturated backend, which is what `502 backend unavailable` should be reserved for. > 🤖 Filed by Claude Code on Kai's behalf.

coilyco-ops referenced this issue

2026-08-13 17:25:38 +00:00

The context budget is 47,104 tokens against a 1M-token model, because num_ctx is a local-inference parameter applied to a hosted route #115

coilyco-ops referenced this issue from a commit

2026-08-13 17:54:36 +00:00

Classify an upstream 4xx as a request error, not a transport failure

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 17:55:00 +00:00

Classify an upstream 4xx as a request error, not a transport failure #118