An abandoned completion keeps running for 7 to 13 minutes after the client disconnects #112

New issue

Closed

opened 2026-08-13 11:30:42 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 11:30:42 +00:00

Member

Filed from coilyco-gaming/sirens-echo, where the symptom was measured but the cause is not. Cross-reference: coilyco-gaming/sirens-echo#171

The measurement

QA measured a 24h window of SigNoz traces. The timeout ladder is inverted, every layer below the caller allowed to run longer than the caller will wait:

Layer	Ceiling observed
`sirens-echo` turn	180 s
`agent-proxy` `POST /v1/chat/completions`	240 s
`litellm` `litellm_request`	600 s
`litellm` `Received Proxy Server Request`	~1004 s

sirens-echo gives up first, and the abandoned upstream request continues for a further 7 to 13 minutes with nobody to receive its answer. A retry then lands on a backend still busy with the abandoned work, which is a plausible mechanism for the failure clustering in coilyco-gaming/sirens-echo#190 - 33 failures in one hour rather than an even spread. That causal link is consistent with the shape and is not proven.

The caller is not the problem

I checked this before filing, because "the client did not hang up" is the cheaper explanation and it is wrong.

sirens-echo builds the completion request with http.NewRequestWithContext, on a context descending from its turn deadline. When that deadline fires the context cancels and Go's HTTP client closes the connection. The caller does disconnect at 180 s.

So the continued work is downstream of a real client disconnect. Either Agent Proxy is not observing the disconnect on its own request context, or it observes it and does not propagate cancellation to LiteLLM.

Why it is worth fixing here rather than by raising the caller's deadline

The obvious mitigation is to reorder the ladder so sirens-echo waits longer than its dependency. That is being considered on 171 and it is a product decision, because it means a member waits more than four minutes.

But it does not fix this. Whatever the caller's deadline is, a caller that goes away should stop costing inference. A cancelled request that runs to completion is capacity spent on an answer nobody can receive, and it is spent on kai-tower-3026 and the hosted route alike.

What would settle it

Whether the FastAPI handler observes await request.is_disconnected() or an equivalent, and whether that cancellation reaches the LiteLLM call. I cannot see either from here - this repository is not in my checkout and I have no live access.

What I am not claiming

Any of it. I am a sealed engineer on sirens-echo with no ability to run or observe this service. Filing so the half that lives here is not carried indefinitely in another repository's issue, where it would survive every fix made there.

Filed from `coilyco-gaming/sirens-echo`, where the symptom was measured but the cause is not. Cross-reference: https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/171 ## The measurement QA measured a 24h window of SigNoz traces. The timeout ladder is inverted, every layer below the caller allowed to run longer than the caller will wait: | Layer | Ceiling observed | | --- | --- | | `sirens-echo` turn | 180 s | | `agent-proxy` `POST /v1/chat/completions` | 240 s | | `litellm` `litellm_request` | 600 s | | `litellm` `Received Proxy Server Request` | ~1004 s | `sirens-echo` gives up first, and the abandoned upstream request continues for a further 7 to 13 minutes with nobody to receive its answer. A retry then lands on a backend still busy with the abandoned work, which is a plausible mechanism for the failure clustering in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/190 - 33 failures in one hour rather than an even spread. That causal link is consistent with the shape and is **not** proven. ## The caller is not the problem I checked this before filing, because "the client did not hang up" is the cheaper explanation and it is wrong. `sirens-echo` builds the completion request with `http.NewRequestWithContext`, on a context descending from its turn deadline. When that deadline fires the context cancels and Go's HTTP client closes the connection. The caller does disconnect at 180 s. So the continued work is downstream of a real client disconnect. Either Agent Proxy is not observing the disconnect on its own request context, or it observes it and does not propagate cancellation to LiteLLM. ## Why it is worth fixing here rather than by raising the caller's deadline The obvious mitigation is to reorder the ladder so `sirens-echo` waits longer than its dependency. That is being considered on 171 and it is a product decision, because it means a member waits more than four minutes. But it does not fix this. Whatever the caller's deadline is, a caller that goes away should stop costing inference. A cancelled request that runs to completion is capacity spent on an answer nobody can receive, and it is spent on `kai-tower-3026` and the hosted route alike. ## What would settle it Whether the FastAPI handler observes `await request.is_disconnected()` or an equivalent, and whether that cancellation reaches the LiteLLM call. I cannot see either from here - this repository is not in my checkout and I have no live access. ## What I am not claiming Any of it. I am a sealed engineer on `sirens-echo` with no ability to run or observe this service. Filing so the half that lives here is not carried indefinitely in another repository's issue, where it would survive every fix made there.

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-13 11:31:07 +00:00

Move the model call to an idle timeout and consume SSE heartbeats, so a queued turn is not killed as a hung one #171

coilyco-ops referenced this issue

2026-08-13 17:17:29 +00:00

A 400 is classified as a transport error, retried three times, and returned to the caller as 502 #114

coilyco-ops referenced this issue from a commit

2026-08-13 18:04:38 +00:00

Bound one request's total upstream time and record what the caller got

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 18:05:00 +00:00

Bound one request total upstream time and record what the caller got #119