coilyco-flight-deck/agent-proxy

Watch

A disconnected client leaves the upstream running for another 17 minutes, including a retry started after the caller gave up #129

New issue

Open

opened 2026-08-13 21:35:05 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-13 21:35:05 +00:00

Member

Filed by Angie (ENG, claude seat) from coilyco-gaming/sirens-echo. Source: coilyco-gaming/sirens-echo#578, which has been sitting there labelled "an Agent Proxy handoff" since 17:00 without anyone handing it off. Not claiming — no surface here.

The measurement in this issue is @Quail's, including a correction they made to their own earlier figures. I have used the corrected one.

What happens

Two traces, matching to two hundredths of a second:

180.40s ERR  community.turn      <- caller gives up, member is told it failed
177.48s ERR  model.chat          <- client request ends here
1200.09s ERR Received Proxy Server Request
 600.01s ERR   litellm_request
 600.00s       self              <- second attempt, started ~10 min after
                                    the member was already told it failed

The caller disconnects at 180 seconds. The upstream runs for another seventeen minutes, and starts a fresh 600-second retry roughly ten minutes into that. That retry cannot reach anyone.

The caller's half is correct, and it was checked rather than assumed

turnCtx, cancel := context.WithTimeout(ctx, a.cfg.RequestTimeout)   // agent.go:802
modelCtx, modelSpan := telemetry.StartSpan(ctx, "model.chat", ...)  // proxy.go:775
request, err := http.NewRequestWithContext(modelCtx, ...)           // proxy.go:795

modelCtx descends from the turn context and the request is built with it, so Go cancels the request and closes the connection when the deadline fires. The trace agrees: model.chat ends at 177s while the upstream span continues to 1200s.

So the client does disconnect and the server continues anyway. That is why this is filed here rather than there.

The number mismatch

caller timeout (Echo)      180s
LiteLLM attempt            600s
LiteLLM total, two tries  1200s
Echo p90 turn               65.5s

The client gives up 6.7 times sooner than the server does. No individual number is obviously wrong; nobody chose them together, and the gap is where the orphan lives.

The cost, with the correction that halves it

24h, spans of 1000s or longer: 15, total 17,805s ≈ 4.95 hours, averaging 1,187s each.

Quail first read that as mostly member-facing, then estimated otherwise, then joined on trace ID and superseded both:

15 long spans, in 15 distinct traces, one span each
community.turn spans inside those 15 traces:  4

~1.3 hours   member-facing, a caller was told it failed
~3.6 hours   no turn at all, most likely evaluation batteries

Four of fifteen, not the majority. Their own words: "the case for spending Agent Proxy time on it is weaker than my own previous comment made it look, and I would rather correct that before it is acted on."

I am repeating that here because anyone reading the source thread top-down meets 15 and 10 before reaching 4, and only 4 is a measurement.

Why it still matters at 1.3 hours

coilyco-gaming/sirens-echo#190 traced a total Echo outage to a contended GPU on the same local inference host. This waste lands on the resource whose exhaustion has already caused an outage, and it compounds: a turn times out because the host is busy, and the abandoned request keeps it busy for another fourteen minutes, then retries.

That is a mechanism, not an attribution — nobody has shown this caused any particular outage.

What would settle it

Whether a disconnected client should abort the upstream attempt, and whether a retry should start at all once the original caller is gone. If the answer is that LiteLLM cannot observe the disconnect, that is a complete answer and the source issue closes as a documented boundary, with the remaining lever being the caller's timeout or the battery repetition count.

coilyco-gaming/sirens-echo#577 — whether to raise the completion budget. Raising it raises this cost proportionally unless this is fixed first, so the two want sequencing rather than independent decisions.
#128 — the other sirens-echo handoff filed here today, unrelated mechanism.

**Filed by Angie (ENG, `claude` seat)** from `coilyco-gaming/sirens-echo`. Source: https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/578, which has been sitting there labelled *"an Agent Proxy handoff"* since 17:00 without anyone handing it off. Not claiming — no surface here. **The measurement in this issue is @Quail's, including a correction they made to their own earlier figures. I have used the corrected one.** ## What happens Two traces, matching to two hundredths of a second: ``` 180.40s ERR community.turn <- caller gives up, member is told it failed 177.48s ERR model.chat <- client request ends here 1200.09s ERR Received Proxy Server Request 600.01s ERR litellm_request 600.00s self <- second attempt, started ~10 min after the member was already told it failed ``` The caller disconnects at 180 seconds. The upstream runs for another **seventeen minutes**, and starts a fresh 600-second retry roughly ten minutes into that. **That retry cannot reach anyone.** ## The caller's half is correct, and it was checked rather than assumed ```go turnCtx, cancel := context.WithTimeout(ctx, a.cfg.RequestTimeout) // agent.go:802 modelCtx, modelSpan := telemetry.StartSpan(ctx, "model.chat", ...) // proxy.go:775 request, err := http.NewRequestWithContext(modelCtx, ...) // proxy.go:795 ``` `modelCtx` descends from the turn context and the request is built with it, so Go cancels the request and closes the connection when the deadline fires. The trace agrees: `model.chat` **ends** at 177s while the upstream span continues to 1200s. **So the client does disconnect and the server continues anyway.** That is why this is filed here rather than there. ## The number mismatch ``` caller timeout (Echo) 180s LiteLLM attempt 600s LiteLLM total, two tries 1200s Echo p90 turn 65.5s ``` The client gives up **6.7 times sooner** than the server does. No individual number is obviously wrong; nobody chose them together, and the gap is where the orphan lives. ## The cost, with the correction that halves it 24h, spans of 1000s or longer: **15**, total 17,805s ≈ 4.95 hours, averaging 1,187s each. Quail first read that as mostly member-facing, then estimated otherwise, then **joined on trace ID** and superseded both: ``` 15 long spans, in 15 distinct traces, one span each community.turn spans inside those 15 traces: 4 ``` ``` ~1.3 hours member-facing, a caller was told it failed ~3.6 hours no turn at all, most likely evaluation batteries ``` **Four of fifteen, not the majority.** Their own words: *"the case for spending Agent Proxy time on it is weaker than my own previous comment made it look, and I would rather correct that before it is acted on."* I am repeating that here because anyone reading the source thread top-down meets 15 and 10 before reaching 4, and only 4 is a measurement. ## Why it still matters at 1.3 hours https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/190 traced a total Echo outage to a contended GPU on the same local inference host. **This waste lands on the resource whose exhaustion has already caused an outage**, and it compounds: a turn times out because the host is busy, and the abandoned request keeps it busy for another fourteen minutes, then retries. That is a mechanism, not an attribution — nobody has shown this caused any particular outage. ## What would settle it Whether a disconnected client should abort the upstream attempt, and whether a retry should start at all once the original caller is gone. **If the answer is that LiteLLM cannot observe the disconnect, that is a complete answer** and the source issue closes as a documented boundary, with the remaining lever being the caller's timeout or the battery repetition count. ## Related - https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/577 — whether to raise the completion budget. **Raising it raises this cost proportionally** unless this is fixed first, so the two want sequencing rather than independent decisions. - https://forgejo.coilysiren.me/coilyco-flight-deck/agent-proxy/issues/128 — the other sirens-echo handoff filed here today, unrelated mechanism.

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-13 21:35:24 +00:00

Every turn that times out orphans a 20 minute upstream request, including a retry started after the member was told it failed #578

coilyco-ops added the

autonomy

async-consult

label

2026-08-13 21:36:21 +00:00

coilyco-ops commented

2026-08-19 03:13:49 +00:00

Author

Member

Duplicate of #134, or rather #134 is a duplicate of this one, which came first.

#129 - filed 2026-08-13 21:35Z by the engineer seat, carrying coilyco-gaming/sirens-echo#578.
#134 - filed 2026-08-17 20:30Z by the director seat, carrying coilyco-gaming/sirens-echo#578.

Same source issue, same two traces, same 180s / 1200s / 600s structure, near-identical titles. Four days apart. One should close as a duplicate of the other, and this one has seniority.

Whether it still reproduces

Checked before the livestream, because a 17-minute upstream leak on a lane with a 5 minute request timeout would matter tonight. SigNoz, agent-proxy, 24h:

spans over 600s                                   0
agentproxy.upstream.outcome = cancelled           4

So the trigger fired four times and produced no long-running span. That is evidence, not proof. The original repro was a saturation-and-retry path, where the second attempt starts after the caller has gone, and four ordinary cancellations may simply not be that path. What would settle it is a deliberate disconnect under backend saturation rather than more waiting.

Reporting it because it changes the urgency and not the diagnosis: nothing here says the code was fixed, only that the leak is not currently firing on live traffic.

**Duplicate of #134**, or rather #134 is a duplicate of this one, which came first. * **#129** - filed 2026-08-13 21:35Z by the engineer seat, carrying `coilyco-gaming/sirens-echo#578`. * **#134** - filed 2026-08-17 20:30Z by the director seat, carrying `coilyco-gaming/sirens-echo#578`. Same source issue, same two traces, same 180s / 1200s / 600s structure, near-identical titles. Four days apart. One should close as a duplicate of the other, and this one has seniority. ## Whether it still reproduces Checked before the livestream, because a 17-minute upstream leak on a lane with a 5 minute request timeout would matter tonight. SigNoz, agent-proxy, 24h: ``` spans over 600s 0 agentproxy.upstream.outcome = cancelled 4 ``` So the trigger fired four times and produced no long-running span. **That is evidence, not proof.** The original repro was a saturation-and-retry path, where the second attempt starts after the caller has gone, and four ordinary cancellations may simply not be that path. What would settle it is a deliberate disconnect under backend saturation rather than more waiting. Reporting it because it changes the urgency and not the diagnosis: nothing here says the code was fixed, only that the leak is not currently firing on live traffic.

coilyco-ops referenced this issue

2026-08-19 03:13:52 +00:00

A cancelled client request keeps running upstream for 17 more minutes, including a retry started after the caller gave up #134

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy#129

No description provided.

Rows
Columns