coilyco-flight-deck/agent-proxy

Watch

Token usage stops at LiteLLM: agent-proxy records 0 and the harness records nothing #135

New issue

Open

opened 2026-08-18 01:37:02 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-18 01:37:02 +00:00

Member

Filed by Olaf (ops seat) from the Owl Deep critical-failure investigation on 2026-08-18. Kai approved fixing this. Read-only observation, no configuration opened.

Sibling of coilyco-gaming/sirens-echo#932, which had to measure its cache-read finding off LiteLLM spans for exactly the reason below.

What is measured

SigNoz on ser8, 3h window ending 2026-08-18 ~01:10Z, sum(gen_ai.usage.input_tokens) grouped by service.name:

litellm - 3,905,072
agent-proxy - 0
sirens-deep-owl-glass - null, the attribute is absent rather than zero

The trajectory ledger agrees. All 92 model.response.captured events in a 6h window carry agentproxy.prompt_tokens: 0 and agentproxy.completion_tokens: 0. Sample: request_id 1539073268550737965, logical_model sirens-echo/deepseek, upstream_mode litellm, outcome ok, finish_reason stop. Not a sampling artifact, and not confined to failures.

So the number exists at the bottom hop, is zeroed at the middle hop, and never reaches the top hop.

Why it matters

Nothing above LiteLLM can bound or observe context growth in tokens. During the same burst, harness-side model.request grew from 132,111 bytes at round 0 to 217,079 bytes at round 5 on sirens-deep-owl-glass. No component in the path could express that in tokens, so no budget, alarm, or admission check could act on it. Bytes are the only unit anyone upstream has.
The ledger is the audit record and it under-reports to zero. A trajectory ledger whose usage fields are structurally 0 cannot answer what a turn cost, which is the question it exists to answer.
It forced this investigation sideways. #932's decisive cache finding was only reachable by querying LiteLLM spans directly. An operator asking agent-proxy what a request consumed gets 0.

Same shape as coilyco-gaming/sirens-echo#675: the information is present one hop down and structured, and it does not propagate up. #675 tracks the error instance of that pattern. This is the usage instance.

Ask

agent-proxy copies upstream usage onto its own spans and into model.response.captured: prompt, completion and total tokens, plus cache_read_input_tokens where the upstream reports it.

Done when

sum(gen_ai.usage.input_tokens) on agent-proxy spans is non-zero and tracks LiteLLM's for the same window, and a model.response.captured event for a successful chat carries non-zero prompt and completion tokens.

Not established, stated rather than papered over

Whether LiteLLM reports usage on every route or only on sirens-echo/deepseek. I measured the aggregate across the window, not per route.
Whether agent-proxy never reads the field, reads and drops it, or writes a default. I did not open the response-handling path, so cause is inference from behaviour.
No labels applied, because I did not enumerate this repo's label IDs.

**Filed by Olaf (ops seat)** from the Owl Deep critical-failure investigation on 2026-08-18. Kai approved fixing this. Read-only observation, no configuration opened. Sibling of `coilyco-gaming/sirens-echo#932`, which had to measure its cache-read finding off LiteLLM spans for exactly the reason below. ## What is measured SigNoz on ser8, 3h window ending 2026-08-18 ~01:10Z, `sum(gen_ai.usage.input_tokens)` grouped by `service.name`: * `litellm` - **3,905,072** * `agent-proxy` - **0** * `sirens-deep-owl-glass` - **null**, the attribute is absent rather than zero The trajectory ledger agrees. All **92** `model.response.captured` events in a 6h window carry `agentproxy.prompt_tokens: 0` and `agentproxy.completion_tokens: 0`. Sample: `request_id 1539073268550737965`, `logical_model sirens-echo/deepseek`, `upstream_mode litellm`, `outcome ok`, `finish_reason stop`. Not a sampling artifact, and not confined to failures. So the number exists at the bottom hop, is zeroed at the middle hop, and never reaches the top hop. ## Why it matters * **Nothing above LiteLLM can bound or observe context growth in tokens.** During the same burst, harness-side `model.request` grew from **132,111 bytes at round 0 to 217,079 bytes at round 5** on `sirens-deep-owl-glass`. No component in the path could express that in tokens, so no budget, alarm, or admission check could act on it. Bytes are the only unit anyone upstream has. * **The ledger is the audit record and it under-reports to zero.** A trajectory ledger whose usage fields are structurally 0 cannot answer what a turn cost, which is the question it exists to answer. * **It forced this investigation sideways.** #932's decisive cache finding was only reachable by querying LiteLLM spans directly. An operator asking agent-proxy what a request consumed gets 0. Same shape as `coilyco-gaming/sirens-echo#675`: the information is present one hop down and structured, and it does not propagate up. #675 tracks the error instance of that pattern. This is the usage instance. ## Ask agent-proxy copies upstream usage onto its own spans and into `model.response.captured`: prompt, completion and total tokens, plus `cache_read_input_tokens` where the upstream reports it. ## Done when `sum(gen_ai.usage.input_tokens)` on `agent-proxy` spans is non-zero and tracks LiteLLM's for the same window, and a `model.response.captured` event for a successful chat carries non-zero prompt and completion tokens. ## Not established, stated rather than papered over * Whether LiteLLM reports usage on every route or only on `sirens-echo/deepseek`. I measured the aggregate across the window, not per route. * Whether agent-proxy never reads the field, reads and drops it, or writes a default. I did not open the response-handling path, so cause is inference from behaviour. * No labels applied, because I did not enumerate this repo's label IDs.

coilyco-ops commented

2026-08-19 03:13:46 +00:00

Author

Member

The middle hop is fixed. The top hop is not, and it is not this repository's.

Same three-hop measurement as this issue, SigNoz, 3h window ending 2026-08-19 ~07:00Z:

                    this issue (2026-08-18)      now
litellm                     3,905,072       7,538,431
agent-proxy                         0         530,152
sirens-deep / sirens-dowel       null            null

The agent-proxy zero is closed by 50af3de and f4a59d3, both on main and both deployed: streaming was never asking the provider for the usage block, and once it did, the normalizer read usage only in the finish_reason branch while the usage arrives in a chunk of its own after it. #138 and #139 carried those halves and are now closed with the verification.

What remains is the harness hop, where gen_ai.usage.input_tokens is still absent rather than zero. That is coilyco-gaming/sirens-echo reading what agent-proxy now reports and not recording it, so the remaining work does not live here.

Two things worth knowing before anyone picks that up:

The trajectory-ledger symptom in this issue - all 92 model.response.captured events carrying agentproxy.prompt_tokens: 0 - was downstream of the agent-proxy zero. Those should now carry real numbers, and re-measuring is the cheap first step rather than assuming they still do not.
agent-proxy's 530,152 is well below litellm's 7,538,431. I have not established why, and I am not going to guess: the two count different span sets, and litellm's own internal spans may double-count. Naming it as unexplained rather than as a second defect. What would settle it is one request traced end to end with both services' spans read side by side.

Leaving this open for the harness half, and worth retitling to say so, since the title as written now describes a state that is two thirds repaired.

**The middle hop is fixed. The top hop is not, and it is not this repository's.** Same three-hop measurement as this issue, SigNoz, 3h window ending 2026-08-19 ~07:00Z: ``` this issue (2026-08-18) now litellm 3,905,072 7,538,431 agent-proxy 0 530,152 sirens-deep / sirens-dowel null null ``` The agent-proxy zero is closed by `50af3de` and `f4a59d3`, both on `main` and both deployed: streaming was never asking the provider for the usage block, and once it did, the normalizer read usage only in the `finish_reason` branch while the usage arrives in a chunk of its own after it. #138 and #139 carried those halves and are now closed with the verification. **What remains is the harness hop**, where `gen_ai.usage.input_tokens` is still absent rather than zero. That is `coilyco-gaming/sirens-echo` reading what agent-proxy now reports and not recording it, so the remaining work does not live here. Two things worth knowing before anyone picks that up: * The trajectory-ledger symptom in this issue - all 92 `model.response.captured` events carrying `agentproxy.prompt_tokens: 0` - was downstream of the agent-proxy zero. Those should now carry real numbers, and re-measuring is the cheap first step rather than assuming they still do not. * agent-proxy's 530,152 is well below litellm's 7,538,431. I have not established why, and I am not going to guess: the two count different span sets, and litellm's own internal spans may double-count. **Naming it as unexplained rather than as a second defect.** What would settle it is one request traced end to end with both services' spans read side by side. Leaving this open for the harness half, and worth retitling to say so, since the title as written now describes a state that is two thirds repaired.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy#135

No description provided.

Rows
Columns