Measure the prompt cache instead of inferring it #103

Merged

coilysiren merged 1 commit from prompt-cache-accounting into main

2026-08-12 18:42:09 +00:00

coilyco-ops commented

2026-08-12 18:38:57 +00:00

Member

Closes #101 and #102.

What #101 actually found

The ticket read a 53 KB byte-identical system prefix as evidence the sirens-echo/deepseek route was uncached. It is not, and it was not then. LiteLLM's gen_ai.cost.cache_read_cost span attribute was already nonzero for deepseek-v4-flash over the same window and absent on both Ornith groups, so DeepSeek's automatic prefix caching had been working the whole time.

What was missing was the ability to say so. The proxy read prompt_tokens and completion_tokens out of the upstream usage block and discarded every cache field beside them, so a fully cached route and a fully uncached one produced identical evidence. Cost alone cannot give a hit rate either, because LiteLLM prices that model from its own bundled map rather than from Deploy-owned configuration.

What this does

parse_cache_usage normalizes the three shapes that reach this proxy through LiteLLM:

DeepSeek's native prompt_cache_hit_tokens / prompt_cache_miss_tokens
the OpenAI-compatible prompt_tokens_details.cached_tokens, which is also LiteLLM's normalized field
the Anthropic-style cache_read_input_tokens with its separate creation charge

The counts surface in the response usage block under the OpenAI-canonical spelling, on request and upstream spans, in three Prometheus counters, and on the terminal trajectory event.

Reported zero is not silence

A provider that reports nothing is an unmeasured route, not a cache miss. An Ollama backend reuses its KV cache without ever saying so, and publishing that silence as a 100% miss would invent a regression it never had. UpstreamResult carries cache_usage_reported beside the counts and every surface stays absent without it. A reported zero, which is what the turn that populates a cache returns, is a real measurement and is published.

Metrics record once per served response rather than once per upstream attempt, so a turn that retried counts the tokens the caller was actually billed for. Misses derive from prompt_tokens - cached_tokens rather than from a provider's own miss field, which keeps the counters summing to the billed prompt on every provider.

Request shape

The request span also carries gen_ai.request.system_bytes, gen_ai.request.tool_count, and gen_ai.request.tool_bytes. The second half of #101 asks whether a narrower default tool roster is worth the tool calls it would lose, and that trade needs the roster's share of the request measured at the one point every governed route passes through. These are byte counts, so they carry no model-visible content and emit with body capture off.

Not done, on purpose

Nothing here injects cache_control breakpoints. No configured backend needs one, and an unexercised injector would be dead policy on the live routes.

#102

The MCP surface already declares tools.listChanged: false, which is what makes a client-side roster cache correct against Agent Proxy. Stateless HTTP gives the server no channel for an unsolicited notification, so a client that cached the roster and waited for one would wait forever. That is now a tested contract rather than an accident, because a silent flip to true would break every caching client.

The cache itself belongs to the MCP client. For the Discord path that is Sirens Echo, tracked in sirens-echo#163, and the four round-trips per turn are its two roster servers initializing.

Verification

ward exec test (307 passed), format-check, lint, typecheck, and pre-commit over all files.

Deploy's half is coilyco-bridge/deploy#404.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

Closes #101 and #102. ## What #101 actually found The ticket read a 53 KB byte-identical system prefix as evidence the `sirens-echo/deepseek` route was uncached. It is not, and it was not then. LiteLLM's `gen_ai.cost.cache_read_cost` span attribute was already nonzero for `deepseek-v4-flash` over the same window and absent on both Ornith groups, so DeepSeek's automatic prefix caching had been working the whole time. What was missing was the ability to say so. The proxy read `prompt_tokens` and `completion_tokens` out of the upstream usage block and discarded every cache field beside them, so a fully cached route and a fully uncached one produced identical evidence. Cost alone cannot give a hit rate either, because LiteLLM prices that model from its own bundled map rather than from Deploy-owned configuration. ## What this does `parse_cache_usage` normalizes the three shapes that reach this proxy through LiteLLM: - DeepSeek's native `prompt_cache_hit_tokens` / `prompt_cache_miss_tokens` - the OpenAI-compatible `prompt_tokens_details.cached_tokens`, which is also LiteLLM's normalized field - the Anthropic-style `cache_read_input_tokens` with its separate creation charge The counts surface in the response usage block under the OpenAI-canonical spelling, on request and upstream spans, in three Prometheus counters, and on the terminal trajectory event. ## Reported zero is not silence A provider that reports nothing is an unmeasured route, not a cache miss. An Ollama backend reuses its KV cache without ever saying so, and publishing that silence as a 100% miss would invent a regression it never had. `UpstreamResult` carries `cache_usage_reported` beside the counts and every surface stays absent without it. A reported zero, which is what the turn that populates a cache returns, is a real measurement and is published. Metrics record once per **served** response rather than once per upstream attempt, so a turn that retried counts the tokens the caller was actually billed for. Misses derive from `prompt_tokens - cached_tokens` rather than from a provider's own miss field, which keeps the counters summing to the billed prompt on every provider. ## Request shape The request span also carries `gen_ai.request.system_bytes`, `gen_ai.request.tool_count`, and `gen_ai.request.tool_bytes`. The second half of #101 asks whether a narrower default tool roster is worth the tool calls it would lose, and that trade needs the roster's share of the request measured at the one point every governed route passes through. These are byte counts, so they carry no model-visible content and emit with body capture off. ## Not done, on purpose Nothing here injects `cache_control` breakpoints. No configured backend needs one, and an unexercised injector would be dead policy on the live routes. ## #102 The MCP surface already declares `tools.listChanged: false`, which is what makes a client-side roster cache correct against Agent Proxy. Stateless HTTP gives the server no channel for an unsolicited notification, so a client that cached the roster and waited for one would wait forever. That is now a tested contract rather than an accident, because a silent flip to true would break every caching client. The cache itself belongs to the MCP client. For the Discord path that is Sirens Echo, tracked in sirens-echo#163, and the four round-trips per turn are its two roster servers initializing. ## Verification `ward exec test` (307 passed), `format-check`, `lint`, `typecheck`, and `pre-commit` over all files. Deploy's half is coilyco-bridge/deploy#404. --- Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

coilyco-ops added 1 commit

2026-08-12 18:38:57 +00:00

Measure the prompt cache instead of inferring it

ci / smoke (pull_request) Successful in 8s

Details

ci / quality (pull_request) Successful in 22s

Details

3901097425

Issue #101 measured a 53 KB system block sent byte-identical on all 46 turns of
a window against a four-byte user turn, and concluded the prefix was uncached.
The proxy could not settle that. It read prompt_tokens and completion_tokens out
of the upstream usage block and discarded every cache field beside them, so a
fully cached route and a fully uncached one produced identical evidence.

parse_cache_usage normalizes the three shapes that reach this proxy through
LiteLLM: DeepSeek's native prompt_cache_hit_tokens, the OpenAI-compatible
prompt_tokens_details.cached_tokens, and the Anthropic-style
cache_read_input_tokens with its separate creation charge. The counts surface in
the response usage block under the OpenAI-canonical spelling, on request and
upstream spans, in three Prometheus counters, and on the terminal trajectory
event.

A provider that reports nothing is an unmeasured route rather than a cache miss.
An Ollama backend reuses its KV cache without ever saying so, and publishing
that silence as a 100% miss would invent a regression it never had, so
UpstreamResult carries cache_usage_reported beside the counts and every surface
stays absent without it. A reported zero, which is what the turn that populates
a cache returns, is a real measurement and is published.

Metrics record once per served response rather than once per upstream attempt,
so a turn that retried counts the tokens the caller was actually billed for.
Misses derive from prompt_tokens minus cached_tokens rather than from a
provider's own miss field, which keeps the counters summing to the billed prompt
everywhere.

The request span also carries system_bytes, tool_count, and tool_bytes. The
second half of #101 asks whether a narrower default roster is worth the tool
calls it would lose, and that trade needs the roster's share of the request
measured at the one point every governed route passes through. These are byte
counts, so they carry no model-visible content and emit with body capture off.

Nothing here injects cache_control breakpoints. No configured backend needs one,
and an unexercised injector would be dead policy on the live routes.

For #102, the MCP surface already declares tools.listChanged false, which is
what makes a client-side roster cache correct against Agent Proxy. Stateless
HTTP gives the server no channel for an unsolicited notification, so a client
that cached the roster and waited for one would wait forever. That is now a
tested contract rather than an accident, because a silent flip to true would
break every caching client. The cache itself belongs to the MCP client, which
for the Discord path is sirens-echo#163.

closes #101
closes #102

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>