Add the ability to designate an automatic fallback model when a particular model is persistently unavailable

coilysiren commented

2026-08-13 10:33:17 +00:00

Owner

The specific problem this solves: I run ornith most of the time, and its great, but it cant find in vram while I'm playing games. which means everything sirens echo does becomes blocked. I would like instead for sirens echo to be able to inform aproxy that future requests should "stick" to the fallback model if the primary model is persistently down.

Will require some design around the defs of "persistently" and "stick"

The specific problem this solves: I run ornith most of the time, and its great, but it cant find in vram while I'm playing games. which means everything sirens echo does becomes blocked. I would like instead for sirens echo to be able to inform aproxy that future requests should "stick" to the fallback model if the primary model is persistently down. Will require some design around the defs of "persistently" and "stick"

coilyco-ops referenced this issue

2026-08-13 17:17:29 +00:00

A 400 is classified as a transport error, retried three times, and returned to the caller as 502 #114

coilyco-ops commented

2026-08-13 18:35:02 +00:00

Member

Not implemented: the two definitions you flagged are the whole design

You wrote that this needs design around "persistently" and "stick", and I agree, so this is a proposal rather than a branch. Three of the four pieces already exist after the work that just landed alongside it, and what is left is one product decision.

What already does part of this

#108 (saturation-failover) makes a quiet backend a failure. PROXY_BACKEND_SLOW_AFTER abandons an attempt that produces nothing in time, advances to the next tier, and records a failure against that backend's circuit breaker. That is exactly the ornith-during-a-game case: the GPU accepts the request and goes silent, and the chain now moves on instead of waiting out the caller.

The circuit breaker already sticks. Once circuit_fail_threshold consecutive failures land, the breaker opens and every subsequent request skips that backend for circuit_cooldown seconds, then admits one probe. So a crude version of both definitions is live today:

persistently = 5 consecutive failures (PROXY_CIRCUIT_FAIL_THRESHOLD)
stick = 30 seconds (PROXY_CIRCUIT_COOLDOWN)

Why that is not enough for your case

Both numbers are wrong for a game session, and wrong in the same direction.

30 seconds is not a stick. A game runs for hours. At a 30-second cooldown the proxy re-probes ornith roughly 120 times an hour, and every probe is a turn that pays the full slow-path cost before failing over again. Echo would see intermittent slow turns for the whole session rather than a clean switch.

5 consecutive failures is expensive to reach when each failure costs a PROXY_BACKEND_SLOW_AFTER wait. Five turns degrade before the switch sticks.

Proposal

1. Saturation gets its own thresholds. A backend that is busy is a different fact from a backend that is broken, and they deserve different numbers:

PROXY_SATURATION_THRESHOLD (default 2) - consecutive saturations before the route sticks to the fallback. Lower than the failure threshold because each one is expensive.
PROXY_SATURATION_COOLDOWN (default 900) - how long it sticks. Fifteen minutes rather than thirty seconds, because the underlying condition lasts a play session, not a blip.

Half-open probing stays as it is, so recovery is still automatic and needs no human.

2. An explicit signal, which is the part you actually asked for. "I would like sirens echo to be able to inform aproxy" is not something a threshold provides. Two shapes, and I would like you to pick:

(a) A request header. X-Prefer-Fallback: sirens-echo/deepseek on a turn. Per-request, no state in the proxy, no new surface, and Echo decides. But Echo has to remember its own preference and re-send it every turn, and nothing stops two callers disagreeing.
(b) A small admin endpoint. POST /admin/routes/{key}/pin {"backend": "...", "until": "..."}. State lives in the proxy where the routing decision is, one caller sets it and every caller benefits, and you can flip it by hand before starting a game. But it is a new authenticated write surface on a service that has had none, and Ward owns authorization, so it needs a decision about who may call it.

I lean (b) with a required expiry, because your own framing in #108 is the argument for it: the state is known in advance by a human, and nobody starts a game by accident. A pin you set before launching a game is the honest version of this feature. (a) is the cheaper one to build and to reason about.

3. What "persistently" should mean, precisely. Whichever of the above, I would define it as N consecutive saturation or transport failures on the primary within one cooldown window, not a rate over time. A rate needs a window nobody will tune, and consecutive failures is the thing the breaker already counts.

What I need from you

Which of (a) or (b), and whether the default cooldown of fifteen minutes sounds right for a play session. With those two answers this is a small change: the thresholds are a config split, and the pin is roughly one endpoint plus a check in breakers.allow.

Leaving this open and unassigned rather than guessing.

🤖 Filed by Claude Code on Kai's behalf.

## Not implemented: the two definitions you flagged are the whole design You wrote that this needs design around "persistently" and "stick", and I agree, so this is a proposal rather than a branch. Three of the four pieces already exist after the work that just landed alongside it, and what is left is one product decision. ### What already does part of this **#108 (`saturation-failover`)** makes a quiet backend a failure. `PROXY_BACKEND_SLOW_AFTER` abandons an attempt that produces nothing in time, advances to the next tier, and records a failure against that backend's circuit breaker. That is exactly the ornith-during-a-game case: the GPU accepts the request and goes silent, and the chain now moves on instead of waiting out the caller. **The circuit breaker already sticks.** Once `circuit_fail_threshold` consecutive failures land, the breaker opens and every subsequent request skips that backend for `circuit_cooldown` seconds, then admits one probe. So a crude version of both definitions is live today: - persistently = 5 consecutive failures (`PROXY_CIRCUIT_FAIL_THRESHOLD`) - stick = 30 seconds (`PROXY_CIRCUIT_COOLDOWN`) ### Why that is not enough for your case Both numbers are wrong for a game session, and wrong in the same direction. **30 seconds is not a stick.** A game runs for hours. At a 30-second cooldown the proxy re-probes ornith roughly 120 times an hour, and every probe is a turn that pays the full slow-path cost before failing over again. Echo would see intermittent slow turns for the whole session rather than a clean switch. **5 consecutive failures is expensive to reach** when each failure costs a `PROXY_BACKEND_SLOW_AFTER` wait. Five turns degrade before the switch sticks. ### Proposal **1. Saturation gets its own thresholds.** A backend that is busy is a different fact from a backend that is broken, and they deserve different numbers: - `PROXY_SATURATION_THRESHOLD` (default 2) - consecutive saturations before the route sticks to the fallback. Lower than the failure threshold because each one is expensive. - `PROXY_SATURATION_COOLDOWN` (default 900) - how long it sticks. Fifteen minutes rather than thirty seconds, because the underlying condition lasts a play session, not a blip. Half-open probing stays as it is, so recovery is still automatic and needs no human. **2. An explicit signal, which is the part you actually asked for.** "I would like sirens echo to be able to inform aproxy" is not something a threshold provides. Two shapes, and I would like you to pick: - **(a) A request header.** `X-Prefer-Fallback: sirens-echo/deepseek` on a turn. Per-request, no state in the proxy, no new surface, and Echo decides. But Echo has to remember its own preference and re-send it every turn, and nothing stops two callers disagreeing. - **(b) A small admin endpoint.** `POST /admin/routes/{key}/pin {"backend": "...", "until": "..."}`. State lives in the proxy where the routing decision is, one caller sets it and every caller benefits, and you can flip it by hand before starting a game. But it is a new authenticated write surface on a service that has had none, and Ward owns authorization, so it needs a decision about who may call it. I lean **(b)** with a required expiry, because your own framing in #108 is the argument for it: the state is known in advance by a human, and nobody starts a game by accident. A pin you set before launching a game is the honest version of this feature. (a) is the cheaper one to build and to reason about. **3. What "persistently" should mean, precisely.** Whichever of the above, I would define it as *N consecutive saturation or transport failures on the primary within one cooldown window*, not a rate over time. A rate needs a window nobody will tune, and consecutive failures is the thing the breaker already counts. ### What I need from you Which of (a) or (b), and whether the default cooldown of fifteen minutes sounds right for a play session. With those two answers this is a small change: the thresholds are a config split, and the pin is roughly one endpoint plus a check in `breakers.allow`. Leaving this open and unassigned rather than guessing. > 🤖 Filed by Claude Code on Kai's behalf.

coilyco-ops referenced this issue from a commit

2026-08-13 18:47:18 +00:00

Give saturation its own threshold and cooldown, and ship the burst probe

coilyco-ops referenced this issue

2026-08-13 18:47:41 +00:00

Give saturation its own threshold and cooldown, and ship the burst probe #126

coilyco-ops referenced this issue from a commit

2026-08-13 18:52:43 +00:00

Let the caller name which tier it would rather use

coilyco-ops referenced this issue

2026-08-13 18:52:44 +00:00

Let the caller name which tier it would rather use #127