coilyco-flight-deck/agent-proxy

Watch

A saturated backend is not a failed backend, so failover never fires and the caller is told nothing #108

New issue

Closed

opened 2026-08-12 22:17:52 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-12 22:17:52 +00:00

Member

Root cause of the 2026-08-12 Echo outage

Kai:

echo failed its turns b/c I had a game running on the 3026 and aproxy doesn't know how to communicate that limitation / state

A game on the local GPU host starved local inference. Echo routes to that backend; Deep does not. Echo failed 100% of turns for ~2.5 hours at ~180s each while Deep answered ping in 2.18s through the same proxy.

Not a service outage. A resource conflict the proxy cannot see, cannot route around, and cannot describe.

Why the existing machinery did not save it

Two independent gaps, and each one alone would have been enough.

1. Health is not capacity

agent-proxy issues GET http://100.118.232.24:11434/api/tags on the request path — an Ollama model listing, returning 200 in 6.7ms even while the GPU is fully occupied.

A 200 from /api/tags means the model is installed. It says nothing about whether inference can proceed now. The proxy is checking the wrong property and getting a confident, fast, useless answer.

2. A hang is not an error, so failover never triggers

The resilience machinery — resilience.attempt, AllBackendsFailed — fires on errors: validation failures, 500s, refused connections. A saturated backend produces none of those. It accepts the request and goes quiet.

coilyco-gaming/sirens-echo#81 records that cloud-hosted model variants exist as failover. That failover could not help here, because from the proxy's point of view nothing had failed yet. It was still waiting, and it waited until the caller's deadline killed it.

Slowness is not currently a failure condition. For a self-hosted GPU tier sharing hardware with a human, it is the primary one.

This also explains the latency distribution

coilyco-gaming/sirens-echo#162 measured litellm_request p50 3.42s against p99 233.71s — a 68× spread. I attributed it to "contention or a stall" and could not do better from telemetry.

It is not a tail. It is bimodal, and the modes are GPU free and GPU shared with a game. Percentiles were the wrong tool: they smear two distinct operating regimes into one distribution and make a predictable state look like random variance.

The part that makes this unusually fixable

The state is known in advance by a human. Nobody starts a game by accident. This is not an unpredictable tail — it is a scheduled condition that the operator knows about before the first turn fails, and that no component in the path can currently be told.

That is what makes an operator-supplied signal viable here, where for genuine hardware faults it would not be.

Options

Probe capacity, not presence. A tiny real completion with a short deadline, rather than /api/tags. Measures the property that matters. Costs a little inference per health check.
Treat slow as failed. If first token does not arrive within N seconds, abandon and fail over to the next tier. This is the change that makes #81's cloud failover actually reachable, and it is probably the highest value item here.
Operator state signal. A "local tier unavailable" switch — set by hand, or derived from GPU utilisation via the fleet's existing node-stats telemetry. The elegant version, and the one that matches how the failure actually originates.
Communicate it. Say so. #104's SSE heartbeats are exactly the channel — {"state":"backend_saturated","failing_over":true} turns three hours of silence into a legible condition. This is the literal content of "aproxy doesn't know how to communicate that limitation / state."

Recommend 2 + 4 as the pair that fixes the outage and makes it visible, with 3 as the follow-on that prevents it rather than reacting to it.

August 19

Per coilyco-gaming/sirens-echo#178, dead air is the demo's worst outcome, and this is the most likely route to it. Two mitigations, not mutually exclusive:

Pin Echo to a non-local tier for the demo, or
Guarantee the local GPU host is dedicated for the duration

Either is sufficient. Neither is automatic today, and the failure mode is silent for three minutes per turn.

Acceptance

A saturated backend is detected as unavailable rather than waited on until the caller's deadline.
Failover to another tier occurs on slowness, not only on error.
The caller receives a signal distinguishing "backend saturated" from "still working" and from "failed."

#104 — heartbeats, the channel this state should travel on
#106 — upstream running long after the caller left; same class of missing state propagation
#107 — burst test; note that a saturated local backend will confound it, so run it against a known-idle GPU
coilyco-gaming/sirens-echo#162 — the bimodal latency this explains
coilyco-gaming/sirens-echo#171 — the caller-side deadline, which was doing the only sane thing available to it
coilyco-gaming/sirens-echo#81 — the tier failover this makes reachable

## Root cause of the 2026-08-12 Echo outage Kai: > echo failed its turns b/c I had a game running on the 3026 and aproxy doesn't know how to communicate that limitation / state A game on the local GPU host starved local inference. Echo routes to that backend; Deep does not. Echo failed **100% of turns for ~2.5 hours** at ~180s each while Deep answered `ping` in 2.18s through the same proxy. Not a service outage. A **resource conflict the proxy cannot see, cannot route around, and cannot describe.** ## Why the existing machinery did not save it Two independent gaps, and each one alone would have been enough. ### 1. Health is not capacity agent-proxy issues `GET http://100.118.232.24:11434/api/tags` on the request path — an Ollama model listing, returning **200 in 6.7ms** even while the GPU is fully occupied. A 200 from `/api/tags` means *the model is installed*. It says nothing about whether inference can proceed now. The proxy is checking the wrong property and getting a confident, fast, useless answer. ### 2. A hang is not an error, so failover never triggers The resilience machinery — `resilience.attempt`, `AllBackendsFailed` — fires on **errors**: validation failures, 500s, refused connections. A saturated backend produces none of those. It accepts the request and goes quiet. `coilyco-gaming/sirens-echo#81` records that cloud-hosted model variants exist as **failover**. That failover could not help here, because from the proxy's point of view nothing had failed yet. It was still waiting, and it waited until the caller's deadline killed it. **Slowness is not currently a failure condition.** For a self-hosted GPU tier sharing hardware with a human, it is the *primary* one. ## This also explains the latency distribution `coilyco-gaming/sirens-echo#162` measured `litellm_request` p50 **3.42s** against p99 **233.71s** — a 68× spread. I attributed it to "contention or a stall" and could not do better from telemetry. It is not a tail. It is **bimodal**, and the modes are *GPU free* and *GPU shared with a game*. Percentiles were the wrong tool: they smear two distinct operating regimes into one distribution and make a predictable state look like random variance. ## The part that makes this unusually fixable **The state is known in advance by a human.** Nobody starts a game by accident. This is not an unpredictable tail — it is a scheduled condition that the operator knows about before the first turn fails, and that no component in the path can currently be told. That is what makes an operator-supplied signal viable here, where for genuine hardware faults it would not be. ## Options 1. **Probe capacity, not presence.** A tiny real completion with a short deadline, rather than `/api/tags`. Measures the property that matters. Costs a little inference per health check. 2. **Treat slow as failed.** If first token does not arrive within N seconds, abandon and fail over to the next tier. This is the change that makes #81's cloud failover actually reachable, and it is probably the highest value item here. 3. **Operator state signal.** A "local tier unavailable" switch — set by hand, or derived from GPU utilisation via the fleet's existing node-stats telemetry. The elegant version, and the one that matches how the failure actually originates. 4. **Communicate it.** Say so. `#104`'s SSE heartbeats are exactly the channel — `{"state":"backend_saturated","failing_over":true}` turns three hours of silence into a legible condition. This is the literal content of "aproxy doesn't know how to communicate that limitation / state." Recommend **2 + 4** as the pair that fixes the outage and makes it visible, with **3** as the follow-on that prevents it rather than reacting to it. ## August 19 Per `coilyco-gaming/sirens-echo#178`, dead air is the demo's worst outcome, and this is the most likely route to it. Two mitigations, not mutually exclusive: - Pin Echo to a non-local tier for the demo, or - Guarantee the local GPU host is dedicated for the duration Either is sufficient. Neither is automatic today, and the failure mode is silent for three minutes per turn. ## Acceptance - A saturated backend is detected as unavailable rather than waited on until the caller's deadline. - Failover to another tier occurs on slowness, not only on error. - The caller receives a signal distinguishing "backend saturated" from "still working" and from "failed." ## Related - #104 — heartbeats, the channel this state should travel on - #106 — upstream running long after the caller left; same class of missing state propagation - #107 — burst test; note that a saturated local backend will confound it, so run it against a known-idle GPU - `coilyco-gaming/sirens-echo#162` — the bimodal latency this explains - `coilyco-gaming/sirens-echo#171` — the caller-side deadline, which was doing the only sane thing available to it - `coilyco-gaming/sirens-echo#81` — the tier failover this makes reachable

coilyco-ops referenced this issue

2026-08-12 22:35:09 +00:00

Tag spans with backend operating regime — percentiles are unreadable across a shared GPU #109

coilyco-ops referenced this issue

2026-08-12 22:35:25 +00:00

Concurrency burst test: does agent-proxy queue or shed, and should heartbeats carry queue position #107

coilyco-ops commented

2026-08-12 23:38:23 +00:00

Author

Member

Correction: Deep was not unaffected, and there are two deadlines

This issue leans on the contrast "Echo failed 100% of turns while Deep answered ping in 2.18s through the same proxy." That contrast is real but overstated, and I built it from a single well-timed sample.

Spans for sirens-deep POST /v1/turn, 21:21–21:36 UTC — the same window in which Echo was failing continuously:

Status	Count	Durations
200	45	mostly 2–8s
502	4	30.00s, 30.00s, 30.00s, 66.04s
429	1	0.00s

Deep was taking failures throughout the contended-GPU period. It was degraded, not healthy — I sampled it during a good moment and generalised.

Two deadlines, two messages, one cause

Agent	Deadline	User-visible reply
Echo	180s	`> \`turn timed out, retry shortly``
Deep	30s	`> \`busy, retry shortly``

Three of Deep's four failures land at exactly 30.00s, which is as sharp a deadline signature as Echo's 180s cluster. Same underlying condition — a starved backend — surfacing at two different limits with two different phrasings, neither of which says anything about capacity.

What this changes

It strengthens the case rather than weakening it. The argument in this issue was that a saturated backend is invisible to the proxy and unroutable-around. Deep demonstrates the same defect at a shorter timescale: it also could not tell the caller why, it also did not fail over, and its failure text (busy) is arguably more misleading than Echo's, because it implies transient load rather than a backend that will be unavailable for hours.

It also corrects a downstream claim: coilyco-gaming/sirens-echo#190 proposes alerting on divergence between agents on the grounds that Deep stayed healthy while Echo did not. That signal is weaker than I described — the divergence was one of degree, not a clean healthy/failed split. Divergence is still worth alerting on, but the threshold has to account for both agents degrading together at different rates.

Not corrected

The core finding stands: Deep's failure rate was ~8% in that window against Echo's 100%, because Deep's primary tier is elsewhere. The tier difference is real. What is not real is the clean binary I drew from it.

## Correction: Deep was not unaffected, and there are two deadlines This issue leans on the contrast *"Echo failed 100% of turns while Deep answered `ping` in 2.18s through the same proxy."* That contrast is real but overstated, and I built it from a single well-timed sample. Spans for `sirens-deep` `POST /v1/turn`, 21:21–21:36 UTC — the same window in which Echo was failing continuously: | Status | Count | Durations | | --- | --- | --- | | 200 | 45 | mostly 2–8s | | **502** | **4** | **30.00s, 30.00s, 30.00s**, 66.04s | | 429 | 1 | 0.00s | Deep was taking failures throughout the contended-GPU period. It was **degraded, not healthy** — I sampled it during a good moment and generalised. ### Two deadlines, two messages, one cause | Agent | Deadline | User-visible reply | | --- | --- | --- | | Echo | **180s** | `> \`turn timed out, retry shortly\`` | | Deep | **30s** | `> \`busy, retry shortly\`` | Three of Deep's four failures land at exactly 30.00s, which is as sharp a deadline signature as Echo's 180s cluster. Same underlying condition — a starved backend — surfacing at two different limits with two different phrasings, neither of which says anything about capacity. ### What this changes **It strengthens the case rather than weakening it.** The argument in this issue was that a saturated backend is invisible to the proxy and unroutable-around. Deep demonstrates the same defect at a shorter timescale: it also could not tell the caller why, it also did not fail over, and its failure text (`busy`) is arguably *more* misleading than Echo's, because it implies transient load rather than a backend that will be unavailable for hours. It also corrects a downstream claim: `coilyco-gaming/sirens-echo#190` proposes alerting on **divergence between agents** on the grounds that Deep stayed healthy while Echo did not. That signal is weaker than I described — the divergence was one of degree, not a clean healthy/failed split. Divergence is still worth alerting on, but the threshold has to account for both agents degrading together at different rates. ### Not corrected The core finding stands: Deep's failure rate was ~8% in that window against Echo's 100%, because Deep's primary tier is elsewhere. The tier difference is real. What is not real is the clean binary I drew from it.

coilyco-ops referenced this issue

2026-08-13 17:17:03 +00:00

Prompt trimming orphans tool messages, so every trimmed request is rejected 400 by the backend #113

coilyco-ops referenced this issue

2026-08-13 17:17:29 +00:00

A 400 is classified as a transport error, retried three times, and returned to the caller as 502 #114

coilyco-ops referenced this issue

2026-08-13 17:25:38 +00:00

The context budget is 47,104 tokens against a 1M-token model, because num_ctx is a local-inference parameter applied to a hosted route #115

coilyco-ops referenced this issue

2026-08-13 18:15:38 +00:00

Tag spans with backend identity and operating regime #122