coilyco-flight-deck/agent-proxy

Watch

A fallback is invisible: nothing records which backend actually served a turn #136

New issue

Closed

opened 2026-08-18 21:32:55 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-18 21:32:55 +00:00

Member

Found reviewing coilyco-bridge/deploy#681, which adds a Baseten fallback behind sirens-echo/deepseek. The deploy side of that change is complete. This is the half that lives here.

The gap

sirens-echo/deepseek is a hosted route, so its entry in the proxy route registry is:

{"direct": null, "key": "sirens-echo/deepseek", "upstream_alias": "sirens-echo/deepseek"}

Agent Proxy forwards it and records the route key. LiteLLM picks the backend, and once deploy#681 lands that may be first-party DeepSeek or Baseten. Nothing on this side distinguishes them, so a turn served by the fallback and a turn served by the primary are indistinguishable in Agent Proxy's data after the fact.

Why it is worth fixing rather than inferring

The two backends do not perform alike, and deploy#681 records the measurement: 1.23s time to first token on first-party DeepSeek against 8.44s on Baseten, from the Artificial Analysis figures weighed on deploy#619. Output throughput is 2.3x, so long generations may finish comparably, but the sirens lanes make many short tool-call turns and those are TTFT-dominated.

The observable consequence is that a fallback reads exactly like a regression. Turn latency jumps roughly sevenfold at first token, and the only honest answer available from Agent Proxy's own data is "something got slower". Distinguishing "we failed over, as designed" from "the primary degraded" would mean correlating against LiteLLM logs by hand, per turn.

That matters most in the case the fallback exists for: an incident, where the person looking has the least time to correlate anything.

What would close it

The response carries the answer. An OpenAI-compatible completion names the model that served it, and Baseten's surface is OpenAI-compatible, so the served model is already on the wire. Recording it beside the route key is the whole fix:

a span attribute or trajectory field naming the backend that answered, distinct from the requested route key
enough to answer "what fraction of turns in this window went to the fallback" without leaving Agent Proxy

Scoping the metric or dashboard is out of scope here. The ask is that the fact stops being discarded.

Not urgent for the 19th, and worth saying why it might be

coilyco-gaming/sirens-echo#929 and coilyco-bridge/deploy#644 hold a scope freeze through August 19, and this is post-freeze work by default.

The argument for the exception: the livestream is the highest-consequence window this fallback will ever see, it is the window where a latency change is most likely to be read live as a bug, and it is the one where nobody will have time to correlate logs. If any part of deploy#681's follow-up gets done before the stream, this is the one I would take over scoping the fallback classes, because it is the difference between a fallback you can see and one you can only infer.

Found reviewing coilyco-bridge/deploy#681, which adds a Baseten fallback behind `sirens-echo/deepseek`. The deploy side of that change is complete. This is the half that lives here. ## The gap `sirens-echo/deepseek` is a hosted route, so its entry in the proxy route registry is: ```json {"direct": null, "key": "sirens-echo/deepseek", "upstream_alias": "sirens-echo/deepseek"} ``` Agent Proxy forwards it and records the **route key**. LiteLLM picks the backend, and once deploy#681 lands that may be first-party DeepSeek or Baseten. Nothing on this side distinguishes them, so a turn served by the fallback and a turn served by the primary are indistinguishable in Agent Proxy's data after the fact. ## Why it is worth fixing rather than inferring The two backends do not perform alike, and deploy#681 records the measurement: **1.23s time to first token on first-party DeepSeek against 8.44s on Baseten**, from the Artificial Analysis figures weighed on deploy#619. Output throughput is 2.3x, so long generations may finish comparably, but the sirens lanes make many short tool-call turns and those are TTFT-dominated. The observable consequence is that **a fallback reads exactly like a regression**. Turn latency jumps roughly sevenfold at first token, and the only honest answer available from Agent Proxy's own data is "something got slower". Distinguishing "we failed over, as designed" from "the primary degraded" would mean correlating against LiteLLM logs by hand, per turn. That matters most in the case the fallback exists for: an incident, where the person looking has the least time to correlate anything. ## What would close it The response carries the answer. An OpenAI-compatible completion names the model that served it, and Baseten's surface is OpenAI-compatible, so the served model is already on the wire. Recording it beside the route key is the whole fix: * a span attribute or trajectory field naming the backend that answered, distinct from the requested route key * enough to answer "what fraction of turns in this window went to the fallback" without leaving Agent Proxy Scoping the metric or dashboard is out of scope here. The ask is that the fact stops being discarded. ## Not urgent for the 19th, and worth saying why it might be coilyco-gaming/sirens-echo#929 and coilyco-bridge/deploy#644 hold a scope freeze through August 19, and this is post-freeze work by default. The argument for the exception: the livestream is the highest-consequence window this fallback will ever see, it is the window where a latency change is most likely to be read live as a bug, and it is the one where nobody will have time to correlate logs. If any part of deploy#681's follow-up gets done before the stream, this is the one I would take over scoping the fallback classes, because it is the difference between a fallback you can see and one you can only infer. Related: coilyco-bridge/deploy#681, coilyco-bridge/deploy#344, coilyco-bridge/deploy#670, coilyco-bridge/deploy#619.

coilyco-ops referenced this issue from a commit

2026-08-18 21:42:10 +00:00

feat(obs): record the model that answered, not only the backend dialled

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-18 21:42:26 +00:00

feat(obs): record the model that answered, not only the backend dialled #137

coilyco-ops commented

2026-08-18 21:42:39 +00:00

Author

Member

Correcting this issue before it gets read as written, then fixed in #137.

"Nothing records which backend served" was wrong. agentproxy.backend already exists and dispatch stamps it from result.served_by, alongside agentproxy.backend.regime. I filed this from the route registry shape without reading resilience.py, which was the mistake.

The real gap is narrower and still worth closing. agentproxy.backend names this proxy's chain entry, which is the right answer for a route it load-balances. sirens-echo/deepseek is direct: null with an upstream_alias, so the chain is one hop and that attribute names the upstream no matter which model ran. LiteLLM's choice between first-party DeepSeek and Baseten happens beyond it.

The fix is correspondingly smaller than this issue implied. upstream.py already parses the response's model into result.model and never emits it, while gen_ai.request.model is emitted. So it is one attribute under the same OTel GenAI convention rather than new plumbing:

gen_ai.request.model    what was asked for
gen_ai.response.model   what answered

Everything this issue says about why it matters stands: the 1.23s to 8.44s TTFT gap means a failover reads as a sevenfold latency regression, and separating "failed over as designed" from "primary degraded" should not require correlating LiteLLM logs by hand mid-incident.

The scope-freeze paragraph also stands, and the exception argument is now cheaper to accept than when I wrote it, since the change is one line plus two tests rather than new instrumentation.

Correcting this issue before it gets read as written, then fixed in #137. **"Nothing records which backend served" was wrong.** `agentproxy.backend` already exists and dispatch stamps it from `result.served_by`, alongside `agentproxy.backend.regime`. I filed this from the route registry shape without reading `resilience.py`, which was the mistake. The real gap is narrower and still worth closing. `agentproxy.backend` names **this proxy's chain entry**, which is the right answer for a route it load-balances. `sirens-echo/deepseek` is `direct: null` with an `upstream_alias`, so the chain is one hop and that attribute names the upstream no matter which model ran. LiteLLM's choice between first-party DeepSeek and Baseten happens beyond it. The fix is correspondingly smaller than this issue implied. `upstream.py` already parses the response's `model` into `result.model` and never emits it, while `gen_ai.request.model` is emitted. So it is one attribute under the same OTel GenAI convention rather than new plumbing: ``` gen_ai.request.model what was asked for gen_ai.response.model what answered ``` Everything this issue says about **why** it matters stands: the 1.23s to 8.44s TTFT gap means a failover reads as a sevenfold latency regression, and separating "failed over as designed" from "primary degraded" should not require correlating LiteLLM logs by hand mid-incident. The scope-freeze paragraph also stands, and the exception argument is now cheaper to accept than when I wrote it, since the change is one line plus two tests rather than new instrumentation.

coilysiren closed this issue

2026-08-18 21:46:55 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy#136

No description provided.

Rows
Columns