Give saturation its own threshold and cooldown, and ship the burst probe

coilyco-ops commented

2026-08-13 18:47:41 +00:00

Member

Summary

The parts of #111 and #107 that did not actually need a decision from you.

Refs #111. Refs #107. Neither is closed - each keeps the half that is genuinely yours.

#111: the two definitions you asked for

You wrote that "persistently" and "stick" both needed defining. Those are numbers, not a product decision, and the circuit breaker's existing ones are wrong for your case in the same direction:

	failure default	saturation default	why
count	5	2	each saturation costs a full slow-path wait, so five degraded turns before the switch sticks is too many
cooldown	30s	900s	a game runs for hours; 30s re-probes a busy GPU ~120 times an hour, each probe a turn that pays the full wait before failing over again

PROXY_SATURATION_THRESHOLD and PROXY_SATURATION_COOLDOWN. The count is consecutive, so one good turn clears it, and half-open probing still recovers with no human involved.

What stays open on #111: the explicit signal. "I would like sirens echo to be able to inform aproxy" is not something a threshold provides, and the two shapes for it - a request header or a small admin pin endpoint - differ enough to be your call. The admin endpoint in particular is a new authenticated write surface on a service that has had none, and AGENTS.md says Ward owns authorization. My reasoning is in the issue comment.

#107: the probe, not the run

scripts/burst_probe.py plus ward exec burst-probe. It runs the method from that issue: concurrency 1/5/10/30, three repeats, shortest possible identical prompt, max_tokens: 1.

It refuses to start without --yes and names the exact call count in the refusal:

error: this would issue 138 real model calls against http://ser8:8080.
Re-run with --yes once the GPU is idle and the rate limiter is off.

The verdict is deliberately weak where the evidence is weak. A flat run cannot separate "no admission control" from "the backend absorbed the burst", and it says so rather than picking. A shedding run warns that #110 ships the limiter on at 1/s with burst 1 - which would answer 429 to 29 of 30 requests and read as "the proxy sheds" when it measured the limiter I just added. That is the likeliest way to get a confidently wrong answer, so it is in the verdict text, the doc, and a test.

The client side cannot see admission delay at all. docs/burst-probe.md says where the answer lives: queue.wait per trace, which since #105 closes at dequeue and reports admission delay as its own duration.

What stays open on #107: the run. Idle GPU, not during an incident, and your decision to spend the tokens.

How to verify

ward exec test - 3 new cases in tests/test_saturation.py, 7 in tests/test_burst_probe.py (355 passed).
test_one_saturation_does_not_stick_by_default and test_saturation_sticks_for_its_own_cooldown pin both numbers, including that it does reopen at 901s rather than sticking forever.
test_a_recovered_backend_forgets_its_saturations pins that the count is consecutive.
test_it_refuses_to_spend_tokens_without_consent and test_shedding_verdict_warns_about_the_rate_limiter pin the guards.

Risk

Low. The thresholds only apply on the saturation path, which is itself off unless PROXY_BACKEND_SLOW_AFTER is set. The probe is a script that will not run without an explicit flag.

Note

Branched off saturation-failover (#124), tip of #118 → #119 → #120 → #122 → #123 → #124.

## Summary The parts of #111 and #107 that did not actually need a decision from you. Refs #111. Refs #107. Neither is closed - each keeps the half that is genuinely yours. ## #111: the two definitions you asked for You wrote that "persistently" and "stick" both needed defining. Those are numbers, not a product decision, and the circuit breaker's existing ones are wrong for your case in the same direction: | | failure default | saturation default | why | | --- | --- | --- | --- | | count | 5 | **2** | each saturation costs a full slow-path wait, so five degraded turns before the switch sticks is too many | | cooldown | 30s | **900s** | a game runs for hours; 30s re-probes a busy GPU ~120 times an hour, each probe a turn that pays the full wait before failing over again | `PROXY_SATURATION_THRESHOLD` and `PROXY_SATURATION_COOLDOWN`. The count is consecutive, so one good turn clears it, and half-open probing still recovers with no human involved. **What stays open on #111:** the explicit signal. "I would like sirens echo to be able to inform aproxy" is not something a threshold provides, and the two shapes for it - a request header or a small admin pin endpoint - differ enough to be your call. The admin endpoint in particular is a new authenticated write surface on a service that has had none, and `AGENTS.md` says Ward owns authorization. My reasoning is in the issue comment. ## #107: the probe, not the run `scripts/burst_probe.py` plus `ward exec burst-probe`. It runs the method from that issue: concurrency 1/5/10/30, three repeats, shortest possible identical prompt, `max_tokens: 1`. It **refuses to start without `--yes`** and names the exact call count in the refusal: ``` error: this would issue 138 real model calls against http://ser8:8080. Re-run with --yes once the GPU is idle and the rate limiter is off. ``` **The verdict is deliberately weak where the evidence is weak.** A flat run cannot separate "no admission control" from "the backend absorbed the burst", and it says so rather than picking. A shedding run warns that #110 ships the limiter on at 1/s with burst 1 - which would answer 429 to 29 of 30 requests and read as "the proxy sheds" when it measured the limiter I just added. That is the likeliest way to get a confidently wrong answer, so it is in the verdict text, the doc, and a test. The client side cannot see admission delay at all. `docs/burst-probe.md` says where the answer lives: `queue.wait` per trace, which since #105 closes at dequeue and reports admission delay as its own duration. **What stays open on #107:** the run. Idle GPU, not during an incident, and your decision to spend the tokens. ## How to verify 1. `ward exec test` - 3 new cases in `tests/test_saturation.py`, 7 in `tests/test_burst_probe.py` (355 passed). 2. `test_one_saturation_does_not_stick_by_default` and `test_saturation_sticks_for_its_own_cooldown` pin both numbers, including that it does reopen at 901s rather than sticking forever. 3. `test_a_recovered_backend_forgets_its_saturations` pins that the count is consecutive. 4. `test_it_refuses_to_spend_tokens_without_consent` and `test_shedding_verdict_warns_about_the_rate_limiter` pin the guards. ## Risk Low. The thresholds only apply on the saturation path, which is itself off unless `PROXY_BACKEND_SLOW_AFTER` is set. The probe is a script that will not run without an explicit flag. ## Note Branched off `saturation-failover` (#124), tip of #118 → #119 → #120 → #122 → #123 → #124.

coilyco-ops added 6 commits

2026-08-13 18:47:41 +00:00

Emit SSE heartbeats carrying attempt and backend state

ci / quality (pull_request) Failing after 21s

Details

ci / smoke (pull_request) Successful in 6s

Details

d8277f41c9

Issue #104: a caller waiting on a streaming completion cannot tell four states
apart. Admitted and queued, retrying after a failure, generating, and hung all
look like silence followed by either a response or a deadline. sirens-echo
resolved that by giving up at ~179.5s and reporting a timeout, including in
cases where the proxy was still doing legitimate work.

Lines beginning with a colon are SSE comments, which every spec-compliant client
ignores, so a consumer that does not parse them sees the same data frames it saw
before. That is why the issue preferred comments to empty-delta chunks, which
some OpenAI-compatible clients mishandle.

dispatch_stream yields a state marker before each chain entry and once the first
real chunk arrives, and the streaming surface turns those into comment lines
rather than deltas. attempt carries n, of, backend, and the backend's regime from
#109. A keepalive repeats the current state every PROXY_HEARTBEAT_INTERVAL
seconds, default 10, so a state that persists stays visible without a
transition.

Retry visibility is what earns this. sirens-echo#137 records three attempts
burning ~9s before a turn 502s, and from outside that is indistinguishable from
one slow attempt. attempt n of N turns a silent fallback into something a caller
can log, display, and alert on.

Queue position is not carried. Issue #105 measured admission delay between
0.7ms and 4.5ms across 29 traces while queue.wait itself varied 25-fold, so
there is nothing to report at current traffic. It is also moot on this path: the
streaming surface calls dispatch_stream directly and never touches the queue.
Whether position is worth carrying under concurrency is #107's question, and the
payload has room for it when that is answered.

_with_keepalives shields the in-flight read while it waits, so a keepalive tick
never cancels the chunk it was waiting for. Setting the interval to 0 removes
the keepalives and leaves the state markers.

Each emission increments llm_stream_heartbeats_total and adds a stream.heartbeat
event to the request span, because the issue asked that a silent heartbeat path
not be able to regress unnoticed.

This changes nothing a user sees on its own, and the issue says so plainly. A
total request deadline ignores heartbeats by construction, so sirens-echo must
move to an idle or read timeout before any of this alters an outcome. That half
lives in that repository. This is the emitting side.

closes #104

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Treat a slow backend as unavailable and advance the chain

ci / quality (pull_request) Failing after 19s

Details

ci / smoke (pull_request) Successful in 7s

Details

2b993f213e

Issue #108, on the 2026-08-12 Echo outage: a game on the local GPU host starved
local inference, Echo failed 100% of turns for about two and a half hours at
~180s each, and Deep answered ping in 2.18s through the same proxy. Not a
service outage. A resource conflict the proxy could not see, route around, or
describe.

Two gaps, either one sufficient. Health is not capacity: GET /api/tags returns
200 in 6.7ms while the GPU is fully occupied, which says the model is installed,
not that inference can proceed. And a hang is not an error: the resilience
machinery fires on validation failures, 500s, and refused connections, and a
saturated backend produces none of those. It accepts the request and goes quiet,
and the proxy waited until the caller's deadline killed it.

PROXY_BACKEND_SLOW_AFTER makes slowness a failure condition, which for a
self-hosted GPU tier sharing hardware with a human is the primary one. Past the
threshold the attempt is abandoned, the chain advances to the next tier rather
than retrying the same backend, and the breaker records a failure so later
requests skip it for the cooldown. It defaults to 0, off, so a deployment that
does not set it is unchanged.

This is what makes the cloud failover in sirens-echo#81 reachable. It could not
help before, because from the proxy's point of view nothing had failed yet.

Only the wait is bounded. On the streaming path the threshold covers time to the
first chunk, because a stream that has started is making progress and cutting it
for being long would be the opposite of what the issue asks for. The
non-streaming path has no first-token signal to observe, so it bounds the whole
attempt, and the doc says plainly that the value must sit above the slowest
legitimate completion on that route.

A saturated backend and a spent request budget are different facts and stay
separable. Budget exhausted answers 504 and does not fail over, because there is
no time left to fail over into. Backend slow with budget remaining advances the
chain.

A streaming caller is told as it happens, through #104's channel:

    : {"state":"backend_saturated","backend":"tower-3026","failing_over":true}

which is the literal content of "aproxy doesn't know how to communicate that
limitation / state". The attempt span carries outcome=saturated and
regime=saturated, beside llm_backend_saturated_total and
dispatch.backend_saturated.

This reacts rather than prevents, and options 1 and 3 of that issue are what
would prevent. Nothing here probes capacity, so /api/tags still reports
presence, and nothing derives the regime automatically, so the operator-supplied
field from #109 is still the only source.

closes #108

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Close the wrapped stream cleanly when a caller stops reading

ci / quality (pull_request) Successful in 28s

Details

ci / smoke (pull_request) Successful in 8s

Details

0bbf45932d

The keepalive wrapper advances its source through a task so a tick can fire
while a read is outstanding. When the consumer stops early, that task was
cancelled and abandoned, which leaves the source async generator mid-step. The
generator's own close then raises from a state it cannot unwind, and Python
reports it as an unraisable error at collection time - so whether it surfaces
depends on when garbage collection runs, which is exactly the kind of failure
that reproduces on a loaded runner and not on a laptop.

The wrapper now awaits the cancelled read before dropping it, then closes the
source explicitly rather than leaving it to the collector.

The keepalive test also waited on a fixed sleep for a tick that is only
guaranteed to be prompt on an idle machine. It waits for the behaviour now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Merge the stream teardown fix from sse-heartbeats be12461cc5

Give the saturation tests real headroom

ci / smoke (pull_request) Successful in 8s

Details

ci / quality (pull_request) Successful in 32s

Details

ae6a9ef9cc

The thresholds were 50ms against sleeps measured in seconds, which is a clear
contrast for the slow side and almost none for the fast one: a runner that
stalls for 50ms between scheduling the healthy backend and its stubbed answer
would see that backend marked saturated too. Half a second keeps the contrast
and stops the fast path racing the scheduler.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Give saturation its own threshold and cooldown, and ship the burst probe

ci / smoke (pull_request) Successful in 8s

Details

ci / quality (pull_request) Successful in 30s

Details

e27cebf5bf

Two halves of two issues that were sitting behind decisions, where only part of
each actually needed one.

Issue #111 asked for an automatic fallback when a model is persistently
unavailable, and for that choice to stick, and said both words needed defining.
The explicit signal it also asks for - sirens-echo telling the proxy to switch -
is a product decision between a request header and an admin pin endpoint, and
that is still open on the issue. The thresholds are not a decision, and they are
what makes the feature work without one.

Saturation now counts and cools separately from failure, because a busy backend
is not a broken one. PROXY_SATURATION_THRESHOLD defaults to 2 rather than the
failure threshold's 5: each saturation costs a full slow-path wait, so five
degraded turns before the switch sticks is too many. PROXY_SATURATION_COOLDOWN
defaults to 900 rather than 30: a game runs for hours, and a thirty-second
cooldown re-probes a busy GPU about 120 times an hour, each probe a turn that
pays the full wait before failing over again. The count is consecutive, so one
good turn clears it, and half-open probing still recovers without a human.

Issue #107 needs an idle GPU, a not-during-an-incident window, and a decision to
spend tokens, none of which are mine. Writing the probe is not any of those.
scripts/burst_probe.py runs the method from that issue - concurrency 1, 5, 10,
30, three repeats, shortest possible identical prompt, max_tokens 1 - and
refuses to start without --yes, naming the exact call count in the refusal.

Its verdict is deliberately weak where the evidence is weak. A flat run cannot
separate "no admission control" from "the backend absorbed the burst", and it
says so instead of picking. A shedding run warns that #110's rate limiter ships
on at 1/s with a burst of 1, which would answer 429 to 29 of 30 requests and
read as "the proxy sheds" when it measured the limiter. That is the likeliest
way to get a confidently wrong answer out of this run, so it is in the verdict
text, the doc, and a test.

The client side cannot see admission delay at all. docs/burst-probe.md says
where the answer actually lives: queue.wait per trace, which since #105 closes
at dequeue and so reports admission delay as its own duration.

Refs #111
Refs #107

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>