feat(obs): record the model that answered, not only the backend dialled #137

Merged

coilysiren merged 1 commit from task/served-model-attribute into main

2026-08-18 21:46:55 +00:00

coilyco-ops commented

2026-08-18 21:42:26 +00:00

Member

Closes #136.

The correction that made this a one-liner

I filed #136 saying nothing records which backend served. That was imprecise, and reading the code fixed it: agentproxy.backend already exists and dispatch stamps it from result.served_by.

It is the right answer for a route this proxy load-balances across its own chain. It cannot answer for a route it forwards. sirens-echo/deepseek is direct: null with an upstream_alias, so the chain is one hop and agentproxy.backend names the upstream regardless of which model actually ran.

coilyco-bridge/deploy#681 puts a Baseten fallback behind that route inside LiteLLM. From here those two backends are the same hop, so a turn served by the fallback and one served by the primary were indistinguishable in this proxy's telemetry.

The fix

The response already carries the answer, and upstream.py already parses it into result.model. It was simply never emitted, while gen_ai.request.model is. So this is the sibling attribute under the same OTel GenAI convention, not a new concept:

gen_ai.request.model    what was asked for
gen_ai.response.model   what answered

Omitted when upstream names no model, so an absent attribute stays a fact rather than an empty string.

Why it earns the line

deploy#681 measures 1.23s time to first token on first-party DeepSeek against 8.44s on Baseten, and the sirens lanes make short, TTFT-dominated turns. Without this a failover reads as a sevenfold latency regression, with nothing in this proxy's data separating "we failed over, as designed" from "the primary degraded" — at exactly the moment nobody has time to correlate LiteLLM logs by hand.

Tests

Two, in the existing set_result_span_attributes idiom:

the served model is recorded alongside agentproxy.backend, asserting both, because they are different facts and a LiteLLM fallback moves only the second
the attribute is absent when upstream named no model

just test 402 passed, just lint clean.

Scope

One attribute. No metric, no dashboard, no change to what is dialled or how failover behaves. Deciding what to chart from it is separate work.

Closes #136. ## The correction that made this a one-liner I filed #136 saying nothing records which backend served. That was imprecise, and reading the code fixed it: **`agentproxy.backend` already exists** and dispatch stamps it from `result.served_by`. It is the right answer for a route this proxy load-balances across its own chain. It cannot answer for a route it forwards. `sirens-echo/deepseek` is `direct: null` with an `upstream_alias`, so the chain is one hop and `agentproxy.backend` names the upstream regardless of which model actually ran. coilyco-bridge/deploy#681 puts a Baseten fallback behind that route inside LiteLLM. From here those two backends are the same hop, so a turn served by the fallback and one served by the primary were indistinguishable in this proxy's telemetry. ## The fix The response already carries the answer, and `upstream.py` already parses it into `result.model`. It was simply never emitted, while `gen_ai.request.model` is. So this is the sibling attribute under the same OTel GenAI convention, not a new concept: ``` gen_ai.request.model what was asked for gen_ai.response.model what answered ``` Omitted when upstream names no model, so an absent attribute stays a fact rather than an empty string. ## Why it earns the line deploy#681 measures **1.23s time to first token on first-party DeepSeek against 8.44s on Baseten**, and the sirens lanes make short, TTFT-dominated turns. Without this a failover reads as a sevenfold latency regression, with nothing in this proxy's data separating "we failed over, as designed" from "the primary degraded" — at exactly the moment nobody has time to correlate LiteLLM logs by hand. ## Tests Two, in the existing `set_result_span_attributes` idiom: - the served model is recorded **alongside** `agentproxy.backend`, asserting both, because they are different facts and a LiteLLM fallback moves only the second - the attribute is absent when upstream named no model `just test` 402 passed, `just lint` clean. ## Scope One attribute. No metric, no dashboard, no change to what is dialled or how failover behaves. Deciding what to chart from it is separate work. Related: coilyco-bridge/deploy#681, coilyco-bridge/deploy#684, coilyco-bridge/deploy#685.

coilyco-ops added 1 commit

2026-08-18 21:42:26 +00:00

feat(obs): record the model that answered, not only the backend dialled

ci / quality (pull_request) Successful in 29s

Details

ci / smoke (pull_request) Successful in 6s

Details

0fe0f3cacd

Closes #136.

agentproxy.backend already names which chain entry served, stamped by dispatch.
That is the right answer for a route this proxy load-balances. It cannot answer
for a route it forwards: sirens-echo/deepseek is `direct: null` with an
upstream_alias, so the chain is one hop and agentproxy.backend says "the
upstream" no matter which model actually ran.

coilyco-bridge/deploy#681 puts a Baseten fallback behind that route in LiteLLM.
From here the two backends are one hop, so a turn served by the fallback and a
turn served by the primary were indistinguishable in this proxy's telemetry.

The response already carries the answer and this already parses it into
result.model. It was simply never emitted, while gen_ai.request.model is. So
this is the sibling attribute under the same OTel GenAI convention rather than a
new concept:

    gen_ai.request.model   what was asked for
    gen_ai.response.model  what answered

Omitted when upstream names no model, so an absent attribute stays a fact rather
than an empty string.

WHY IT IS WORTH THE LINE. deploy#681 measures 1.23s time to first token on
first-party DeepSeek against 8.44s on Baseten, and the sirens lanes make short
TTFT-dominated turns. Without this a failover reads as a sevenfold latency
regression with nothing in this proxy's data to distinguish "we failed over, as
designed" from "the primary degraded", at exactly the moment nobody has time to
correlate LiteLLM logs by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-18 21:42:26 +00:00

coilyco-ops referenced this pull request

2026-08-18 21:42:39 +00:00

A fallback is invisible: nothing records which backend actually served a turn #136