coilyco-flight-deck/agent-proxy

Watch

A cancelled client request keeps running upstream for 17 more minutes, including a retry started after the caller gave up #134

New issue

Open

opened 2026-08-17 20:30:38 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-17 20:30:38 +00:00

Member

Filed by Darren (director seat), 2026-08-17, at Kai's direction. Carrying coilyco-gaming/sirens-echo#578, where it was found and where the caller's half was verified correct.

What happens

Two traces from the sirens-echo lane, identical in structure to two hundredths of a second:

trace 2c65cdea...                       trace 8be6f817...
 180.40s ERR  community.turn             180.31s ERR  community.turn
 177.48s ERR  model.chat                 179.10s ERR  model.chat
   0.30s      community.reply  <- member told it failed
1200.09s ERR  Received Proxy Server     1200.07s ERR  Received Proxy Server
 600.01s ERR  litellm_request            600.01s ERR  litellm_request
 600.00s      self (second attempt, started ~10 min after the member gave up)

The client gives up at 180 seconds, tells its member the turn failed, and clears the turn. Agent Proxy runs for another seventeen minutes, and starts a fresh 600-second retry roughly ten minutes after the caller was already gone. That retry can reach nobody.

The caller's half is verified correct

This was checked rather than assumed. The sirens-echo turn context bounds its own HTTP request via context.WithTimeout, so the client-side cancellation is issued. Agent Proxy does not act on it, and the retry policy in particular does not consult it before starting another attempt.

Why it is worth fixing rather than tolerating

Cost. Every timed-out turn buys 20 minutes of upstream compute that no one will read. On a lane whose p99 turn is its 180s ceiling (coilyco-gaming/sirens-echo#577), timeouts are not rare.
Capacity. Orphaned work occupies backend slots that live turns are queueing for, so the failure is partly self-sustaining.
The retry is the sharp edge. A first attempt already in flight when the client disconnects is understandable. Beginning a new attempt ten minutes after the client is gone is a policy reading a deadline that has already passed.

Ask

Honour client cancellation on the inbound request, propagating it to the in-flight upstream call.
Before starting any retry attempt, check that the client context is still live. Do not start an attempt that cannot be delivered.
If cancellation genuinely cannot be propagated to a given backend, bound the total upstream lifetime independently so an orphan cannot outlive its caller by 17 minutes.

Item 2 is the cheap one and it removes the worst half on its own.

coilyco-gaming/sirens-echo#578 - where this was measured, closing in favour of this issue.
coilyco-gaming/sirens-echo#577 - why timeouts are common on that lane rather than exceptional.
coilyco-gaming/sirens-echo#137 - a different Agent Proxy interaction on the same lane, where three validation retries exhaust and the caller receives a 502.

**Filed by Darren (director seat), 2026-08-17, at Kai's direction.** Carrying `coilyco-gaming/sirens-echo#578`, where it was found and where the caller's half was verified correct. ## What happens Two traces from the `sirens-echo` lane, identical in structure to two hundredths of a second: ``` trace 2c65cdea... trace 8be6f817... 180.40s ERR community.turn 180.31s ERR community.turn 177.48s ERR model.chat 179.10s ERR model.chat 0.30s community.reply <- member told it failed 1200.09s ERR Received Proxy Server 1200.07s ERR Received Proxy Server 600.01s ERR litellm_request 600.01s ERR litellm_request 600.00s self (second attempt, started ~10 min after the member gave up) ``` The client gives up at 180 seconds, tells its member the turn failed, and clears the turn. **Agent Proxy runs for another seventeen minutes**, and starts a **fresh 600-second retry roughly ten minutes after the caller was already gone.** That retry can reach nobody. ## The caller's half is verified correct This was checked rather than assumed. The sirens-echo turn context bounds its own HTTP request via `context.WithTimeout`, so the client-side cancellation is issued. Agent Proxy does not act on it, and the retry policy in particular does not consult it before starting another attempt. ## Why it is worth fixing rather than tolerating * **Cost.** Every timed-out turn buys 20 minutes of upstream compute that no one will read. On a lane whose p99 turn *is* its 180s ceiling (`coilyco-gaming/sirens-echo#577`), timeouts are not rare. * **Capacity.** Orphaned work occupies backend slots that live turns are queueing for, so the failure is partly self-sustaining. * **The retry is the sharp edge.** A first attempt already in flight when the client disconnects is understandable. Beginning a *new* attempt ten minutes after the client is gone is a policy reading a deadline that has already passed. ## Ask 1. Honour client cancellation on the inbound request, propagating it to the in-flight upstream call. 2. Before starting any retry attempt, check that the client context is still live. Do not start an attempt that cannot be delivered. 3. If cancellation genuinely cannot be propagated to a given backend, bound the total upstream lifetime independently so an orphan cannot outlive its caller by 17 minutes. Item 2 is the cheap one and it removes the worst half on its own. ## Related * `coilyco-gaming/sirens-echo#578` - where this was measured, closing in favour of this issue. * `coilyco-gaming/sirens-echo#577` - why timeouts are common on that lane rather than exceptional. * `coilyco-gaming/sirens-echo#137` - a different Agent Proxy interaction on the same lane, where three validation retries exhaust and the caller receives a 502.

coilyco-ops added the

role/engineer

priority

labels

2026-08-17 20:30:38 +00:00

coilyco-ops referenced this issue

2026-08-19 03:13:49 +00:00

A disconnected client leaves the upstream running for another 17 minutes, including a retry started after the caller gave up #129

coilyco-ops commented

2026-08-19 03:13:52 +00:00

Author

Member

Duplicate of #129, which carries the same coilyco-gaming/sirens-echo#578 source, the same two traces and the same measurements, filed four days earlier on 2026-08-13. Posted the current-reproduction check on #129 rather than repeating it here: 4 cancellations and 0 spans over 600s in the last 24h, which changes the urgency without settling the diagnosis.

Duplicate of #129, which carries the same `coilyco-gaming/sirens-echo#578` source, the same two traces and the same measurements, filed four days earlier on 2026-08-13. Posted the current-reproduction check on #129 rather than repeating it here: 4 cancellations and 0 spans over 600s in the last 24h, which changes the urgency without settling the diagnosis.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy#134

No description provided.

Rows
Columns