Bound one request total upstream time and record what the caller got #119

Merged

coilysiren merged 1 commit from cancel-abandoned-work into main

2026-08-13 18:11:15 +00:00

coilyco-ops commented

2026-08-13 18:05:00 +00:00

Member

Bug

Fixes #112. Fixes #106.

The timeout ladder is inverted at every layer:

Layer	Ceiling observed
`sirens-echo` turn	180 s
`agent-proxy` `POST /v1/chat/completions`	240 s
`litellm` `litellm_request`	600 s
`litellm` `Received Proxy Server Request`	~1004 s

The caller gives up first and the abandoned request runs on for another 7 to 13 minutes with nobody to receive its answer. #106 records the same shape from the far end: litellm generated for 207 seconds after agent-proxy stopped waiting, then returned 500, while every agent-proxy span in that trace read has_error: false with an empty response_status_code.

The caller is not at fault - sirens-echo builds its request with http.NewRequestWithContext on its turn deadline and does disconnect at 180s.

Root cause

PROXY_REQUEST_TIMEOUT is per attempt. With max_retries and a fallback chain, nothing bounded the request as a whole. And POST /v1/completions had no disconnect watcher at all, while POST /v1/chat/completions has had one.

Fix

PROXY_REQUEST_DEADLINE - a wall clock for the whole request: queue wait, every retry, every fallback. Past it no further attempt starts, and the in-flight attempt is cut by asyncio.wait_for, which closes the httpx request and with it the upstream connection. Answers 504, not a 502 folded into backend-unavailable. Defaults to 0 (off), so a deployment that does not set it is unchanged.
X-Request-Deadline-Ms (or X-Request-Timeout-Ms) lets a caller declare its own budget. It may only shorten the configured deadline. Set below the caller's own timeout, it closes the connection while the caller is still listening.
/v1/completions gets the disconnect watcher. Both surfaces now share _until_disconnect. The body is read to completion first, since the watcher and request.json() would otherwise race for the same ASGI receive channel.
http.response.status_code on the request span for every terminal response. That is the measurement half of #106.

How to verify

ward exec test - 8 new cases in tests/test_cancellation.py, 1 in tests/test_error_spans.py.
test_deadline_cuts_the_attempt_and_stops_upstream_work asserts the mock transport observes cancellation.
test_expired_deadline_starts_no_attempt asserts an already-spent budget reaches no upstream.
test_caller_deadline_only_shortens_the_configured_one asserts 1000ms against a 100s ceiling still yields 100s.
test_completions_disconnect_cancels_upstream covers the surface that had no watcher.

Test plan

Regression tests for both halves
Existing tests still pass (327 passed)
ward exec format-check, lint, typecheck, pre-commit all clean

Risk

Low by default: the deadline is off unless configured, so behaviour is unchanged out of the box. The /v1/completions watcher is a behaviour change on that surface - a caller that disconnects mid-request now has its work cancelled, which is what the chat surface already did.

Four test doubles gained a deadline=None / signature update. Two new closed-taxonomy entries and one new RequestOutcome value.

What this does not do

It does not make litellm stop. Closing the connection is the only cancellation signal HTTP offers; whether the far end acts on it is litellm's own behaviour and needs verifying there. The deadline is what makes agent-proxy stop asking.

Note

Branched off upstream-error-classification (#118), which carries the upstream-span half of #106. Review #118 first; this branch contains its commit.

## Bug Fixes #112. Fixes #106. The timeout ladder is inverted at every layer: | Layer | Ceiling observed | | --- | --- | | `sirens-echo` turn | 180 s | | `agent-proxy` `POST /v1/chat/completions` | 240 s | | `litellm` `litellm_request` | 600 s | | `litellm` `Received Proxy Server Request` | ~1004 s | The caller gives up first and the abandoned request runs on for another 7 to 13 minutes with nobody to receive its answer. #106 records the same shape from the far end: litellm generated for 207 seconds after agent-proxy stopped waiting, then returned 500, while every agent-proxy span in that trace read `has_error: false` with an empty `response_status_code`. The caller is not at fault - `sirens-echo` builds its request with `http.NewRequestWithContext` on its turn deadline and does disconnect at 180s. ## Root cause `PROXY_REQUEST_TIMEOUT` is **per attempt**. With `max_retries` and a fallback chain, nothing bounded the request as a whole. And `POST /v1/completions` had no disconnect watcher at all, while `POST /v1/chat/completions` has had one. ## Fix - **`PROXY_REQUEST_DEADLINE`** - a wall clock for the whole request: queue wait, every retry, every fallback. Past it no further attempt starts, and the in-flight attempt is cut by `asyncio.wait_for`, which closes the httpx request and with it the upstream connection. Answers **504**, not a 502 folded into backend-unavailable. **Defaults to 0 (off)**, so a deployment that does not set it is unchanged. - **`X-Request-Deadline-Ms`** (or `X-Request-Timeout-Ms`) lets a caller declare its own budget. It may only **shorten** the configured deadline. Set below the caller's own timeout, it closes the connection while the caller is still listening. - **`/v1/completions` gets the disconnect watcher.** Both surfaces now share `_until_disconnect`. The body is read to completion first, since the watcher and `request.json()` would otherwise race for the same ASGI receive channel. - **`http.response.status_code` on the request span** for every terminal response. That is the measurement half of #106. ## How to verify 1. `ward exec test` - 8 new cases in `tests/test_cancellation.py`, 1 in `tests/test_error_spans.py`. 2. `test_deadline_cuts_the_attempt_and_stops_upstream_work` asserts the mock transport observes cancellation. 3. `test_expired_deadline_starts_no_attempt` asserts an already-spent budget reaches no upstream. 4. `test_caller_deadline_only_shortens_the_configured_one` asserts 1000ms against a 100s ceiling still yields 100s. 5. `test_completions_disconnect_cancels_upstream` covers the surface that had no watcher. ## Test plan - [x] Regression tests for both halves - [x] Existing tests still pass (327 passed) - [x] `ward exec format-check`, `lint`, `typecheck`, `pre-commit` all clean ## Risk Low by default: the deadline is off unless configured, so behaviour is unchanged out of the box. The `/v1/completions` watcher is a behaviour change on that surface - a caller that disconnects mid-request now has its work cancelled, which is what the chat surface already did. Four test doubles gained a `deadline=None` / signature update. Two new closed-taxonomy entries and one new `RequestOutcome` value. ## What this does not do It does not make litellm stop. Closing the connection is the only cancellation signal HTTP offers; whether the far end acts on it is litellm's own behaviour and needs verifying there. The deadline is what makes agent-proxy stop *asking*. ## Note Branched off `upstream-error-classification` (#118), which carries the upstream-span half of #106. Review #118 first; this branch contains its commit.

coilyco-ops added 1 commit

2026-08-13 18:05:01 +00:00

Bound one request's total upstream time and record what the caller got

ci / quality (pull_request) Successful in 22s

Details

ci / smoke (pull_request) Successful in 8s

Details

2bfd472b26

Issue #112 measured an inverted timeout ladder: sirens-echo gives up at 180s,
agent-proxy at 240s, litellm's request span at 600s, its server span at ~1004s.
The caller goes first and the abandoned upstream request runs on for another 7
to 13 minutes with nobody to receive its answer. Issue #106 records the same
shape from the far end: litellm generated for 207 seconds after agent-proxy
stopped waiting, then returned 500.

The caller is not the problem and raising its deadline does not fix this.
sirens-echo builds its request with http.NewRequestWithContext on its turn
deadline, so it does disconnect at 180s. Whatever the caller's number is, a
caller that has gone away should stop costing inference.

PROXY_REQUEST_TIMEOUT was per attempt, so with retries and a fallback chain
nothing bounded the request as a whole. PROXY_REQUEST_DEADLINE is a wall clock
for all of it: queue wait, every retry, every fallback. Past it no further
attempt starts, and the in-flight attempt is cut by asyncio.wait_for, which
closes the httpx request and with it the upstream connection. The route answers
504 rather than folding a timeout into 502 backend-unavailable. It defaults to
0, off, so nothing changes for a deployment that does not set it.

A caller that knows its own budget sends X-Request-Deadline-Ms. It may only
shorten the configured deadline, never lengthen it, because an operator ceiling
a caller can talk past is not a ceiling. Set below the caller's own timeout it
closes the connection while the caller is still listening, rather than leaving
the upstream generating for someone who has already left.

POST /v1/completions had no disconnect watcher at all, while the chat surface
has had one. Same resilience path, none of the cancellation. Both now share
_until_disconnect. The body is read to completion before the watcher starts,
since the two would otherwise race for the same ASGI receive channel.

For the measurement half of #106: every terminal response now stamps
http.response.status_code on the request span. The trace in that issue carried
an empty status and has_error false on all six agent-proxy spans against an
upstream 500, which is the reason its 0.74% error rate could not be trusted. The
upstream-span half of that landed with #114.

What this does not do is make litellm stop. Closing the connection is the only
cancellation signal HTTP offers, and whether the far end acts on it is litellm's
own behaviour to verify.

closes #112
closes #106

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>