Treat a slow backend as unavailable and advance the chain

coilyco-ops commented

2026-08-13 18:27:46 +00:00

Member

Summary

Closes #108, options 2 and 4 - the pair you recommended.

The 2026-08-12 outage: a game on the local GPU host starved local inference. Echo failed 100% of turns for ~2.5 hours at ~180s each while Deep answered ping in 2.18s through the same proxy.

Two gaps, either one sufficient:

Health is not capacity. GET /api/tags returns 200 in 6.7ms while the GPU is fully occupied.
A hang is not an error. The machinery fires on validation failures, 500s, and refused connections. A saturated backend produces none of those - it accepts the request and goes quiet, and the proxy waited until the caller's deadline killed it.

Fix

PROXY_BACKEND_SLOW_AFTER - seconds an attempt may run before its backend counts as saturated. Defaults to 0 (off), so nothing changes until Deploy sets it.

Past it: the attempt is abandoned, the chain advances to the next tier rather than retrying the same backend, and the breaker records a failure so later requests skip it for the cooldown. A backend nobody can get a token out of is unavailable, whatever it says about itself.

This is what makes the cloud failover in sirens-echo#81 reachable. It could not help before, because from the proxy's point of view nothing had failed yet.

Only the wait is bounded. On the streaming path the threshold covers time to the first chunk - a stream that has started is making progress, and cutting it for being long would be the opposite of what you asked for. The non-streaming path has no first-token signal, so it bounds the whole attempt; the doc says plainly that the value must sit above the slowest legitimate completion on that route.

Saturation and a spent budget stay separable. Budget exhausted answers 504 and does not fail over, because there is no time to fail over into. Backend slow with budget remaining advances the chain.

Saying so (option 4)

Through #104's channel, as it happens:

: {"state":"backend_saturated","backend":"tower-3026","failing_over":true}

Plus agentproxy.outcome=saturated and agentproxy.backend.regime=saturated on the attempt span, llm_backend_saturated_total{logical_model,backend}, and dispatch.backend_saturated.

How to verify

ward exec test - tests/test_saturation.py, 7 cases, no test sleeps longer than 50ms.
test_a_quiet_backend_fails_over_instead_of_being_waited_on is the outage shape: the tower goes quiet, the hosted tier answers.
test_a_saturated_backend_stops_receiving_work asserts the breaker opens.
test_a_spent_budget_is_a_deadline_not_a_saturation pins the distinction.
test_a_stream_already_generating_is_not_cut_for_being_long pins that only the wait is bounded.

Test plan

New suite for the behaviour
Existing tests still pass (345 passed)
ward exec format-check, lint, typecheck, pre-commit all clean

What this does not do

It reacts rather than prevents. Options 1 and 3 in the issue are what would prevent:

nothing here probes capacity - /api/tags still reports presence
nothing derives the regime automatically - the operator-supplied field from #109 is still the only source

For August 19, this makes the failure legible and recoverable within one turn instead of silent for three minutes. It does not replace either mitigation you named: pinning Echo to a non-local tier, or dedicating the GPU. Both are still worth doing, and neither is automatic.

Follow-up for Deploy

PROXY_BACKEND_SLOW_AFTER needs a value to do anything. Something comfortably above the slowest normal completion on the local tier and comfortably below the caller's own deadline is the shape; I have not measured a number and would rather you pick it than guess.

Note

Branched off sse-heartbeats (#123), tip of the chain #118 -> #119 -> #120 -> #122 -> #123. The backend_saturated heartbeat needs #123's channel.

## Summary Closes #108, options 2 and 4 - the pair you recommended. The 2026-08-12 outage: a game on the local GPU host starved local inference. Echo failed **100% of turns for ~2.5 hours** at ~180s each while Deep answered `ping` in 2.18s through the same proxy. Two gaps, either one sufficient: - **Health is not capacity.** `GET /api/tags` returns 200 in 6.7ms while the GPU is fully occupied. - **A hang is not an error.** The machinery fires on validation failures, 500s, and refused connections. A saturated backend produces none of those - it accepts the request and goes quiet, and the proxy waited until the caller's deadline killed it. ## Fix `PROXY_BACKEND_SLOW_AFTER` - seconds an attempt may run before its backend counts as saturated. **Defaults to 0 (off)**, so nothing changes until Deploy sets it. Past it: the attempt is abandoned, the chain **advances to the next tier** rather than retrying the same backend, and the breaker records a failure so later requests skip it for the cooldown. A backend nobody can get a token out of is unavailable, whatever it says about itself. This is what makes the cloud failover in `sirens-echo#81` reachable. It could not help before, because from the proxy's point of view nothing had failed yet. **Only the wait is bounded.** On the streaming path the threshold covers time to the *first chunk* - a stream that has started is making progress, and cutting it for being long would be the opposite of what you asked for. The non-streaming path has no first-token signal, so it bounds the whole attempt; the doc says plainly that the value must sit above the slowest legitimate completion on that route. **Saturation and a spent budget stay separable.** Budget exhausted answers 504 and does not fail over, because there is no time to fail over into. Backend slow with budget remaining advances the chain. ## Saying so (option 4) Through #104's channel, as it happens: ``` : {"state":"backend_saturated","backend":"tower-3026","failing_over":true} ``` Plus `agentproxy.outcome=saturated` and `agentproxy.backend.regime=saturated` on the attempt span, `llm_backend_saturated_total{logical_model,backend}`, and `dispatch.backend_saturated`. ## How to verify 1. `ward exec test` - `tests/test_saturation.py`, 7 cases, no test sleeps longer than 50ms. 2. `test_a_quiet_backend_fails_over_instead_of_being_waited_on` is the outage shape: the tower goes quiet, the hosted tier answers. 3. `test_a_saturated_backend_stops_receiving_work` asserts the breaker opens. 4. `test_a_spent_budget_is_a_deadline_not_a_saturation` pins the distinction. 5. `test_a_stream_already_generating_is_not_cut_for_being_long` pins that only the wait is bounded. ## Test plan - [x] New suite for the behaviour - [x] Existing tests still pass (345 passed) - [x] `ward exec format-check`, `lint`, `typecheck`, `pre-commit` all clean ## What this does not do **It reacts rather than prevents.** Options 1 and 3 in the issue are what would prevent: - nothing here probes capacity - `/api/tags` still reports presence - nothing derives the regime automatically - the operator-supplied field from #109 is still the only source For **August 19**, this makes the failure legible and recoverable within one turn instead of silent for three minutes. It does **not** replace either mitigation you named: pinning Echo to a non-local tier, or dedicating the GPU. Both are still worth doing, and neither is automatic. ## Follow-up for Deploy `PROXY_BACKEND_SLOW_AFTER` needs a value to do anything. Something comfortably above the slowest normal completion on the local tier and comfortably below the caller's own deadline is the shape; I have not measured a number and would rather you pick it than guess. ## Note Branched off `sse-heartbeats` (#123), tip of the chain #118 -> #119 -> #120 -> #122 -> #123. The `backend_saturated` heartbeat needs #123's channel.

coilyco-ops added 2 commits

2026-08-13 18:27:46 +00:00

Emit SSE heartbeats carrying attempt and backend state

ci / quality (pull_request) Failing after 21s

Details

ci / smoke (pull_request) Successful in 6s

Details

d8277f41c9

Issue #104: a caller waiting on a streaming completion cannot tell four states
apart. Admitted and queued, retrying after a failure, generating, and hung all
look like silence followed by either a response or a deadline. sirens-echo
resolved that by giving up at ~179.5s and reporting a timeout, including in
cases where the proxy was still doing legitimate work.

Lines beginning with a colon are SSE comments, which every spec-compliant client
ignores, so a consumer that does not parse them sees the same data frames it saw
before. That is why the issue preferred comments to empty-delta chunks, which
some OpenAI-compatible clients mishandle.

dispatch_stream yields a state marker before each chain entry and once the first
real chunk arrives, and the streaming surface turns those into comment lines
rather than deltas. attempt carries n, of, backend, and the backend's regime from
#109. A keepalive repeats the current state every PROXY_HEARTBEAT_INTERVAL
seconds, default 10, so a state that persists stays visible without a
transition.

Retry visibility is what earns this. sirens-echo#137 records three attempts
burning ~9s before a turn 502s, and from outside that is indistinguishable from
one slow attempt. attempt n of N turns a silent fallback into something a caller
can log, display, and alert on.

Queue position is not carried. Issue #105 measured admission delay between
0.7ms and 4.5ms across 29 traces while queue.wait itself varied 25-fold, so
there is nothing to report at current traffic. It is also moot on this path: the
streaming surface calls dispatch_stream directly and never touches the queue.
Whether position is worth carrying under concurrency is #107's question, and the
payload has room for it when that is answered.

_with_keepalives shields the in-flight read while it waits, so a keepalive tick
never cancels the chunk it was waiting for. Setting the interval to 0 removes
the keepalives and leaves the state markers.

Each emission increments llm_stream_heartbeats_total and adds a stream.heartbeat
event to the request span, because the issue asked that a silent heartbeat path
not be able to regress unnoticed.

This changes nothing a user sees on its own, and the issue says so plainly. A
total request deadline ignores heartbeats by construction, so sirens-echo must
move to an idle or read timeout before any of this alters an outcome. That half
lives in that repository. This is the emitting side.

closes #104

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Treat a slow backend as unavailable and advance the chain

ci / quality (pull_request) Failing after 19s

Details

ci / smoke (pull_request) Successful in 7s

Details

2b993f213e

Issue #108, on the 2026-08-12 Echo outage: a game on the local GPU host starved
local inference, Echo failed 100% of turns for about two and a half hours at
~180s each, and Deep answered ping in 2.18s through the same proxy. Not a
service outage. A resource conflict the proxy could not see, route around, or
describe.

Two gaps, either one sufficient. Health is not capacity: GET /api/tags returns
200 in 6.7ms while the GPU is fully occupied, which says the model is installed,
not that inference can proceed. And a hang is not an error: the resilience
machinery fires on validation failures, 500s, and refused connections, and a
saturated backend produces none of those. It accepts the request and goes quiet,
and the proxy waited until the caller's deadline killed it.

PROXY_BACKEND_SLOW_AFTER makes slowness a failure condition, which for a
self-hosted GPU tier sharing hardware with a human is the primary one. Past the
threshold the attempt is abandoned, the chain advances to the next tier rather
than retrying the same backend, and the breaker records a failure so later
requests skip it for the cooldown. It defaults to 0, off, so a deployment that
does not set it is unchanged.

This is what makes the cloud failover in sirens-echo#81 reachable. It could not
help before, because from the proxy's point of view nothing had failed yet.

Only the wait is bounded. On the streaming path the threshold covers time to the
first chunk, because a stream that has started is making progress and cutting it
for being long would be the opposite of what the issue asks for. The
non-streaming path has no first-token signal to observe, so it bounds the whole
attempt, and the doc says plainly that the value must sit above the slowest
legitimate completion on that route.

A saturated backend and a spent request budget are different facts and stay
separable. Budget exhausted answers 504 and does not fail over, because there is
no time left to fail over into. Backend slow with budget remaining advances the
chain.

A streaming caller is told as it happens, through #104's channel:

    : {"state":"backend_saturated","backend":"tower-3026","failing_over":true}

which is the literal content of "aproxy doesn't know how to communicate that
limitation / state". The attempt span carries outcome=saturated and
regime=saturated, beside llm_backend_saturated_total and
dispatch.backend_saturated.

This reacts rather than prevents, and options 1 and 3 of that issue are what
would prevent. Nothing here probes capacity, so /api/tags still reports
presence, and nothing derives the regime automatically, so the operator-supplied
field from #109 is still the only source.

closes #108

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 18:27:46 +00:00

coilyco-ops added 3 commits

2026-08-13 18:34:22 +00:00

Close the wrapped stream cleanly when a caller stops reading

ci / quality (pull_request) Successful in 28s

Details

ci / smoke (pull_request) Successful in 8s

Details

0bbf45932d

The keepalive wrapper advances its source through a task so a tick can fire
while a read is outstanding. When the consumer stops early, that task was
cancelled and abandoned, which leaves the source async generator mid-step. The
generator's own close then raises from a state it cannot unwind, and Python
reports it as an unraisable error at collection time - so whether it surfaces
depends on when garbage collection runs, which is exactly the kind of failure
that reproduces on a loaded runner and not on a laptop.

The wrapper now awaits the cancelled read before dropping it, then closes the
source explicitly rather than leaving it to the collector.

The keepalive test also waited on a fixed sleep for a tick that is only
guaranteed to be prompt on an idle machine. It waits for the behaviour now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Merge the stream teardown fix from sse-heartbeats be12461cc5

Give the saturation tests real headroom

ci / smoke (pull_request) Successful in 8s

Details

ci / quality (pull_request) Successful in 32s

Details

ae6a9ef9cc

The thresholds were 50ms against sleeps measured in seconds, which is a clear
contrast for the slow side and almost none for the fast one: a runner that
stalls for 50ms between scheduling the healthy backend and its stubbed answer
would see that backend marked saturated too. Half a second keeps the contrast
and stops the fast path racing the scheduler.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops referenced this pull request

2026-08-13 18:47:41 +00:00

Give saturation its own threshold and cooldown, and ship the burst probe #126

coilyco-ops referenced this pull request

2026-08-13 18:52:44 +00:00

Let the caller name which tier it would rather use #127

coilysiren closed this pull request

2026-08-13 19:41:50 +00:00