Emit SSE heartbeats carrying queue and attempt state, so callers can tell a queued turn from a hung one #104

New issue

Closed

opened 2026-08-12 20:16:38 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-12 20:16:38 +00:00

Member

Problem

A caller waiting on POST /v1/chat/completions cannot distinguish four states that look identical from outside: admitted and queued, retrying after a validation failure, generating, and hung. It gets silence, then either a response or a deadline.

sirens-echo currently resolves that ambiguity by giving up at ~179.5s and telling the user the turn timed out — including in cases where the proxy was still doing legitimate work.

Evidence, 24h window

Span	p50	p99
`litellm_request`	3.42s	233.71s
`queue.wait`	20.09s	147.13s
`upstream.chat`	3.43s	145.56s

litellm's p99 sits 54s above the caller's ~179.5s deadline, so the slowest turns are guaranteed to be cut regardless of what was asked. Measured from the Discord side (coilyco-gaming/sirens-echo#160), four independent user-visible failures landed at 180.3s, 180.5s, 180.5s, 180.6s — a 26% failure rate over 19 exchanges, with the caller unable to report anything more useful than "timed out."

Proposal

When the request is streaming, emit SSE comment lines during the pre-generation phase:

: {"state":"queued","position":4}
: {"state":"attempt","n":2,"of":3}
: {"state":"upstream_started"}

SSE comments — lines beginning with : — are ignored by every spec-compliant SSE client, so existing consumers are unaffected without any change on their side. Consumers that want the signal parse them; consumers that don't, drop them. That property is why I would prefer comments over empty-delta chunks ({"choices":[{"delta":{}}]}), which some OpenAI-compatible clients mishandle.

Cadence: on state transition, plus a keepalive every N seconds while a state persists.

Retry visibility is the part I would not skip

coilyco-gaming/sirens-echo#137 documents three resilience.attempt spans burning ~9s before a turn 502s. From outside, that is indistinguishable from one slow attempt. Surfacing attempt n of 3 turns a silent retry storm into something a caller can log, display, and alert on — and it is the cheapest observability win in this proposal.

Non-streaming requests

Heartbeats have nowhere to go on a non-streaming response without breaking the response contract. Options, in order of preference:

Callers that want progress opt into stream: true. Cleanest; no new surface.
Retry-After plus a queue-depth header on the 503 path when admission would exceed a threshold — complements rather than replaces this.

This does nothing on its own

A total request deadline ignores heartbeats by construction. sirens-echo currently wraps the call in a total context deadline (context deadline exceeded at 179.456s), which fires on schedule no matter how many bytes arrive. The caller must move to an idle/read timeout with a larger overall ceiling for any of this to change behaviour.

Filing that half separately against sirens-echo. This issue is only the emitting side, and it should be understood as inert until the consuming side lands — worth stating plainly so nobody ships this and expects the timeouts to stop.

Acceptance

A streaming request that waits in admission receives at least one heartbeat before the first content delta.
Heartbeats carry queue state and attempt number.
A consumer that ignores comment lines sees byte-identical output to today.
Heartbeat emission is visible in traces, so a silent heartbeat path cannot regress unnoticed.

coilyco-gaming/sirens-echo#160 — the 180s deadline, measured from both sides
coilyco-gaming/sirens-echo#162 — 53 KB uncached prefix, and the p50/p99 data above
coilyco-gaming/sirens-echo#137 — the three-attempt retry path this would surface
coilyco-gaming/sirens-echo#111 — the Discord-side progress element that would consume this

## Problem A caller waiting on `POST /v1/chat/completions` cannot distinguish four states that look identical from outside: admitted and queued, retrying after a validation failure, generating, and hung. It gets silence, then either a response or a deadline. `sirens-echo` currently resolves that ambiguity by giving up at ~179.5s and telling the user the turn timed out — including in cases where the proxy was still doing legitimate work. ## Evidence, 24h window | Span | p50 | p99 | | --- | --- | --- | | `litellm_request` | 3.42s | **233.71s** | | `queue.wait` | 20.09s | 147.13s | | `upstream.chat` | 3.43s | 145.56s | litellm's p99 sits **54s above** the caller's ~179.5s deadline, so the slowest turns are guaranteed to be cut regardless of what was asked. Measured from the Discord side (`coilyco-gaming/sirens-echo#160`), four independent user-visible failures landed at 180.3s, 180.5s, 180.5s, 180.6s — a 26% failure rate over 19 exchanges, with the caller unable to report anything more useful than "timed out." ## Proposal When the request is streaming, emit **SSE comment lines** during the pre-generation phase: ``` : {"state":"queued","position":4} : {"state":"attempt","n":2,"of":3} : {"state":"upstream_started"} ``` SSE comments — lines beginning with `:` — are ignored by every spec-compliant SSE client, so existing consumers are unaffected without any change on their side. Consumers that want the signal parse them; consumers that don't, drop them. That property is why I would prefer comments over empty-delta chunks (`{"choices":[{"delta":{}}]}`), which some OpenAI-compatible clients mishandle. Cadence: on state transition, plus a keepalive every N seconds while a state persists. ### Retry visibility is the part I would not skip `coilyco-gaming/sirens-echo#137` documents three `resilience.attempt` spans burning ~9s before a turn 502s. From outside, that is indistinguishable from one slow attempt. Surfacing `attempt n of 3` turns a silent retry storm into something a caller can log, display, and alert on — and it is the cheapest observability win in this proposal. ### Non-streaming requests Heartbeats have nowhere to go on a non-streaming response without breaking the response contract. Options, in order of preference: 1. Callers that want progress opt into `stream: true`. Cleanest; no new surface. 2. `Retry-After` plus a queue-depth header on the 503 path when admission would exceed a threshold — complements rather than replaces this. ## This does nothing on its own **A total request deadline ignores heartbeats by construction.** `sirens-echo` currently wraps the call in a total context deadline (`context deadline exceeded` at 179.456s), which fires on schedule no matter how many bytes arrive. The caller must move to an idle/read timeout with a larger overall ceiling for any of this to change behaviour. Filing that half separately against `sirens-echo`. This issue is only the emitting side, and it should be understood as inert until the consuming side lands — worth stating plainly so nobody ships this and expects the timeouts to stop. ## Acceptance - A streaming request that waits in admission receives at least one heartbeat before the first content delta. - Heartbeats carry queue state and attempt number. - A consumer that ignores comment lines sees byte-identical output to today. - Heartbeat emission is visible in traces, so a silent heartbeat path cannot regress unnoticed. ## Related - `coilyco-gaming/sirens-echo#160` — the 180s deadline, measured from both sides - `coilyco-gaming/sirens-echo#162` — 53 KB uncached prefix, and the p50/p99 data above - `coilyco-gaming/sirens-echo#137` — the three-attempt retry path this would surface - `coilyco-gaming/sirens-echo#111` — the Discord-side progress element that would consume this

coilyco-ops referenced this issue

2026-08-12 20:34:59 +00:00

Median turn spends 20s to do 3.4s of work — establish what queue.wait actually measures, then fix it #105

coilyco-ops referenced this issue

2026-08-12 20:49:58 +00:00

Median turn spends 20s to do 3.4s of work — establish what queue.wait actually measures, then fix it #105

coilyco-ops referenced this issue

2026-08-12 20:55:33 +00:00

Median turn spends 20s to do 3.4s of work — establish what queue.wait actually measures, then fix it #105

coilyco-ops referenced this issue

2026-08-12 21:19:13 +00:00

Concurrency burst test: does agent-proxy queue or shed, and should heartbeats carry queue position #107