Let the caller name which tier it would rather use

coilyco-ops commented

2026-08-13 18:52:44 +00:00

Member

Summary

Refs #111 — the "inform aproxy" half, which I had wrongly bundled with the part that needs your decision.

I would like instead for sirens echo to be able to inform aproxy that future requests should "stick" to the fallback model

X-Prefer-Backend: hosted

Moves the named backend to the front of that route's chain for the request.

Why this one did not need you

I offered you a choice between a request header and an admin pin endpoint and then built neither. The endpoint genuinely needs your call — it is a new authenticated write surface on a service that has had none, and AGENTS.md says Ward owns authorization. The header needs none of that. It is the shape your sentence literally describes, it asks the proxy for no authority it did not already have, and it does not foreclose the endpoint.

The safety property

Reorder, never filter. A hint from a caller cannot empty the chain, cannot disarm the fallback, and cannot turn a recoverable turn into a hard failure. Prefer the hosted tier and the tower stays behind it — if hosted is down, the request falls back exactly as before. test_a_preferred_backend_that_fails_still_falls_back pins that.

An unrecognised name leaves the order untouched rather than 400ing, because a routing hint should not fail a request that would otherwise have worked. Value is trimmed and bounded to 64 chars before it reaches a routing decision.

Stickiness is Echo's

A header is per-request, so "future requests should stick" means Echo keeps sending it — a variable it already owns. That is what keeps this free of shared state that two callers could disagree over.

request.backend_preference logs the requested name and whether it applied, so a typo or a backend absent from that route's chain is visible rather than mysterious.

How to verify

ward exec test — tests/test_prefer_backend.py, 9 cases (364 passed overall).
test_the_caller_can_stick_to_the_fallback is your sentence end to end.
test_the_other_tiers_stay_behind_it and test_an_unknown_name_changes_nothing pin the safety property.

Risk

Low. Additive, no-op when the header is absent, and every existing route order is unchanged without it.

Still open on #111

The shared version: an admin endpoint pinning a route for every caller until an expiry. Better fit for you flipping a switch before starting a game; worse fit for the authorization boundary. Still yours to call — but #111 no longer needs it to be useful.

Note

Branched off saturation-stickiness (#126), tip of the #118 → … → #124 → #126 chain.

## Summary Refs #111 — the "inform aproxy" half, which I had wrongly bundled with the part that needs your decision. > I would like instead for sirens echo to be able to inform aproxy that future requests should "stick" to the fallback model ``` X-Prefer-Backend: hosted ``` Moves the named backend to the **front** of that route's chain for the request. ## Why this one did not need you I offered you a choice between a request header and an admin pin endpoint and then built neither. The endpoint genuinely needs your call — it is a new authenticated write surface on a service that has had none, and `AGENTS.md` says Ward owns authorization. **The header needs none of that.** It is the shape your sentence literally describes, it asks the proxy for no authority it did not already have, and it does not foreclose the endpoint. ## The safety property **Reorder, never filter.** A hint from a caller cannot empty the chain, cannot disarm the fallback, and cannot turn a recoverable turn into a hard failure. Prefer the hosted tier and the tower stays behind it — if hosted is down, the request falls back exactly as before. `test_a_preferred_backend_that_fails_still_falls_back` pins that. An unrecognised name leaves the order untouched rather than 400ing, because a routing hint should not fail a request that would otherwise have worked. Value is trimmed and bounded to 64 chars before it reaches a routing decision. ## Stickiness is Echo's A header is per-request, so "future requests should stick" means Echo keeps sending it — a variable it already owns. That is what keeps this free of shared state that two callers could disagree over. `request.backend_preference` logs the requested name and whether it applied, so a typo or a backend absent from that route's chain is visible rather than mysterious. ## How to verify 1. `ward exec test` — `tests/test_prefer_backend.py`, 9 cases (364 passed overall). 2. `test_the_caller_can_stick_to_the_fallback` is your sentence end to end. 3. `test_the_other_tiers_stay_behind_it` and `test_an_unknown_name_changes_nothing` pin the safety property. ## Risk Low. Additive, no-op when the header is absent, and every existing route order is unchanged without it. ## Still open on #111 The **shared** version: an admin endpoint pinning a route for every caller until an expiry. Better fit for you flipping a switch before starting a game; worse fit for the authorization boundary. Still yours to call — but #111 no longer needs it to be useful. ## Note Branched off `saturation-stickiness` (#126), tip of the #118 → … → #124 → #126 chain.

coilyco-ops added 7 commits

2026-08-13 18:52:44 +00:00

Emit SSE heartbeats carrying attempt and backend state

ci / quality (pull_request) Failing after 21s

Details

ci / smoke (pull_request) Successful in 6s

Details

d8277f41c9

Issue #104: a caller waiting on a streaming completion cannot tell four states
apart. Admitted and queued, retrying after a failure, generating, and hung all
look like silence followed by either a response or a deadline. sirens-echo
resolved that by giving up at ~179.5s and reporting a timeout, including in
cases where the proxy was still doing legitimate work.

Lines beginning with a colon are SSE comments, which every spec-compliant client
ignores, so a consumer that does not parse them sees the same data frames it saw
before. That is why the issue preferred comments to empty-delta chunks, which
some OpenAI-compatible clients mishandle.

dispatch_stream yields a state marker before each chain entry and once the first
real chunk arrives, and the streaming surface turns those into comment lines
rather than deltas. attempt carries n, of, backend, and the backend's regime from
#109. A keepalive repeats the current state every PROXY_HEARTBEAT_INTERVAL
seconds, default 10, so a state that persists stays visible without a
transition.

Retry visibility is what earns this. sirens-echo#137 records three attempts
burning ~9s before a turn 502s, and from outside that is indistinguishable from
one slow attempt. attempt n of N turns a silent fallback into something a caller
can log, display, and alert on.

Queue position is not carried. Issue #105 measured admission delay between
0.7ms and 4.5ms across 29 traces while queue.wait itself varied 25-fold, so
there is nothing to report at current traffic. It is also moot on this path: the
streaming surface calls dispatch_stream directly and never touches the queue.
Whether position is worth carrying under concurrency is #107's question, and the
payload has room for it when that is answered.

_with_keepalives shields the in-flight read while it waits, so a keepalive tick
never cancels the chunk it was waiting for. Setting the interval to 0 removes
the keepalives and leaves the state markers.

Each emission increments llm_stream_heartbeats_total and adds a stream.heartbeat
event to the request span, because the issue asked that a silent heartbeat path
not be able to regress unnoticed.

This changes nothing a user sees on its own, and the issue says so plainly. A
total request deadline ignores heartbeats by construction, so sirens-echo must
move to an idle or read timeout before any of this alters an outcome. That half
lives in that repository. This is the emitting side.

closes #104

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Treat a slow backend as unavailable and advance the chain

ci / quality (pull_request) Failing after 19s

Details

ci / smoke (pull_request) Successful in 7s

Details

2b993f213e

Issue #108, on the 2026-08-12 Echo outage: a game on the local GPU host starved
local inference, Echo failed 100% of turns for about two and a half hours at
~180s each, and Deep answered ping in 2.18s through the same proxy. Not a
service outage. A resource conflict the proxy could not see, route around, or
describe.

Two gaps, either one sufficient. Health is not capacity: GET /api/tags returns
200 in 6.7ms while the GPU is fully occupied, which says the model is installed,
not that inference can proceed. And a hang is not an error: the resilience
machinery fires on validation failures, 500s, and refused connections, and a
saturated backend produces none of those. It accepts the request and goes quiet,
and the proxy waited until the caller's deadline killed it.

PROXY_BACKEND_SLOW_AFTER makes slowness a failure condition, which for a
self-hosted GPU tier sharing hardware with a human is the primary one. Past the
threshold the attempt is abandoned, the chain advances to the next tier rather
than retrying the same backend, and the breaker records a failure so later
requests skip it for the cooldown. It defaults to 0, off, so a deployment that
does not set it is unchanged.

This is what makes the cloud failover in sirens-echo#81 reachable. It could not
help before, because from the proxy's point of view nothing had failed yet.

Only the wait is bounded. On the streaming path the threshold covers time to the
first chunk, because a stream that has started is making progress and cutting it
for being long would be the opposite of what the issue asks for. The
non-streaming path has no first-token signal to observe, so it bounds the whole
attempt, and the doc says plainly that the value must sit above the slowest
legitimate completion on that route.

A saturated backend and a spent request budget are different facts and stay
separable. Budget exhausted answers 504 and does not fail over, because there is
no time left to fail over into. Backend slow with budget remaining advances the
chain.

A streaming caller is told as it happens, through #104's channel:

    : {"state":"backend_saturated","backend":"tower-3026","failing_over":true}

which is the literal content of "aproxy doesn't know how to communicate that
limitation / state". The attempt span carries outcome=saturated and
regime=saturated, beside llm_backend_saturated_total and
dispatch.backend_saturated.

This reacts rather than prevents, and options 1 and 3 of that issue are what
would prevent. Nothing here probes capacity, so /api/tags still reports
presence, and nothing derives the regime automatically, so the operator-supplied
field from #109 is still the only source.

closes #108

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Close the wrapped stream cleanly when a caller stops reading

ci / quality (pull_request) Successful in 28s

Details

ci / smoke (pull_request) Successful in 8s

Details

0bbf45932d

The keepalive wrapper advances its source through a task so a tick can fire
while a read is outstanding. When the consumer stops early, that task was
cancelled and abandoned, which leaves the source async generator mid-step. The
generator's own close then raises from a state it cannot unwind, and Python
reports it as an unraisable error at collection time - so whether it surfaces
depends on when garbage collection runs, which is exactly the kind of failure
that reproduces on a loaded runner and not on a laptop.

The wrapper now awaits the cancelled read before dropping it, then closes the
source explicitly rather than leaving it to the collector.

The keepalive test also waited on a fixed sleep for a tick that is only
guaranteed to be prompt on an idle machine. It waits for the behaviour now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Merge the stream teardown fix from sse-heartbeats be12461cc5

Give the saturation tests real headroom

ci / smoke (pull_request) Successful in 8s

Details

ci / quality (pull_request) Successful in 32s

Details

ae6a9ef9cc

The thresholds were 50ms against sleeps measured in seconds, which is a clear
contrast for the slow side and almost none for the fast one: a runner that
stalls for 50ms between scheduling the healthy backend and its stubbed answer
would see that backend marked saturated too. Half a second keeps the contrast
and stops the fast path racing the scheduler.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Give saturation its own threshold and cooldown, and ship the burst probe

ci / smoke (pull_request) Successful in 8s

Details

ci / quality (pull_request) Successful in 30s

Details

e27cebf5bf

Two halves of two issues that were sitting behind decisions, where only part of
each actually needed one.

Issue #111 asked for an automatic fallback when a model is persistently
unavailable, and for that choice to stick, and said both words needed defining.
The explicit signal it also asks for - sirens-echo telling the proxy to switch -
is a product decision between a request header and an admin pin endpoint, and
that is still open on the issue. The thresholds are not a decision, and they are
what makes the feature work without one.

Saturation now counts and cools separately from failure, because a busy backend
is not a broken one. PROXY_SATURATION_THRESHOLD defaults to 2 rather than the
failure threshold's 5: each saturation costs a full slow-path wait, so five
degraded turns before the switch sticks is too many. PROXY_SATURATION_COOLDOWN
defaults to 900 rather than 30: a game runs for hours, and a thirty-second
cooldown re-probes a busy GPU about 120 times an hour, each probe a turn that
pays the full wait before failing over again. The count is consecutive, so one
good turn clears it, and half-open probing still recovers without a human.

Issue #107 needs an idle GPU, a not-during-an-incident window, and a decision to
spend tokens, none of which are mine. Writing the probe is not any of those.
scripts/burst_probe.py runs the method from that issue - concurrency 1, 5, 10,
30, three repeats, shortest possible identical prompt, max_tokens 1 - and
refuses to start without --yes, naming the exact call count in the refusal.

Its verdict is deliberately weak where the evidence is weak. A flat run cannot
separate "no admission control" from "the backend absorbed the burst", and it
says so instead of picking. A shedding run warns that #110's rate limiter ships
on at 1/s with a burst of 1, which would answer 429 to 29 of 30 requests and
read as "the proxy sheds" when it measured the limiter. That is the likeliest
way to get a confidently wrong answer out of this run, so it is in the verdict
text, the doc, and a test.

The client side cannot see admission delay at all. docs/burst-probe.md says
where the answer actually lives: queue.wait per trace, which since #105 closes
at dequeue and so reports admission delay as its own duration.

Refs #111
Refs #107

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Let the caller name which tier it would rather use

ci / smoke (pull_request) Successful in 8s

Details

ci / quality (pull_request) Successful in 37s

Details

554372874d

Issue #111 asks for sirens-echo to be able to inform aproxy that requests should
stick to the fallback while ornith is behind a game. I had treated the whole of
that as needing a decision from Kai, between a request header and an admin pin
endpoint. Only one of those needed the decision.

X-Prefer-Backend moves the named backend to the front of the route's chain for
that request. It is exactly the shape the issue describes, the caller informing
the proxy, and it needs no new authority: no authenticated write surface, no
shared state two callers could disagree over, and nothing that makes Agent Proxy
an execution authority against the boundary AGENTS.md draws around Ward.

Reordering rather than filtering is the safety property that makes this safe to
accept from a caller at all. A hint cannot empty the chain, cannot disarm the
fallback, and cannot turn a recoverable turn into a hard failure. If the
preferred backend is down the request falls back exactly as it would have. An
unrecognised name leaves the order untouched rather than erroring, because a
routing hint has no business failing a request that would otherwise have
worked, and the value is trimmed and bounded before it reaches a routing
decision.

Stickiness is the caller's to keep, since a header is per-request. For Echo that
is a variable it already owns. It is also what keeps this free of shared state.

Every request carrying the header logs request.backend_preference with the
requested name and whether it applied, so a preference that silently does
nothing - a typo, or a backend absent from that route's chain - is visible
rather than mysterious.

The admin pin endpoint is still open on #111 and this does not foreclose it. It
remains the better fit for an operator flipping a switch before starting a game,
and the worse fit for the authorization boundary, which is why it is still a
question rather than a branch.

Refs #111

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 18:52:44 +00:00

coilysiren merged commit d5e40b2f64 into main

2026-08-13 19:41:20 +00:00

coilysiren referenced this pull request from a commit

2026-08-13 19:41:21 +00:00

Merge pull request 'Let the caller name which tier it would rather use' (#127) from prefer-backend-header into main