Every turn ships a 53 KB system prompt, byte-identical, uncached

coilysiren commented

2026-08-12 17:49:46 +00:00

Owner

A four-byte ping costs the same as the hardest question Deep can be asked, because the fixed prefix dwarfs the input by four orders of magnitude:

context.rendered system_prompt_bytes: 53133 user_prompt_bytes: 4
context_prompt_bytes: 45 history_count: 0
model.request request_bytes: 61343 tool_count: 17
message_count: 3

That is roughly 15k input tokens per turn, on every turn, and the system block was identical across all 46 turns in the window. It is a textbook cache prefix.
RecommendationEnable prompt caching for this route at Agent Proxy/LiteLLM. Separately, 17 tool schemas is a large share of the 61 KB — Deep used a tool in only 8 of 46 turns, so a narrower default roster is worth measuring. This compounds badly with F1, where one failed turn pays the 61 KB three times.

A four-byte ping costs the same as the hardest question Deep can be asked, because the fixed prefix dwarfs the input by four orders of magnitude: context.rendered system_prompt_bytes: 53133 user_prompt_bytes: 4 context_prompt_bytes: 45 history_count: 0 model.request request_bytes: 61343 tool_count: 17 message_count: 3 That is roughly 15k input tokens per turn, on every turn, and the system block was identical across all 46 turns in the window. It is a textbook cache prefix. RecommendationEnable prompt caching for this route at Agent Proxy/LiteLLM. Separately, 17 tool schemas is a large share of the 61 KB — Deep used a tool in only 8 of 46 turns, so a narrower default roster is worth measuring. This compounds badly with F1, where one failed turn pays the 61 KB three times.

coilysiren commented

2026-08-12 17:49:51 +00:00

Author

Owner

coilyco-gaming/sirens-echo#162

https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/162

coilysiren referenced this issue from coilyco-gaming/sirens-echo

2026-08-12 17:49:56 +00:00

Every turn ships a 53 KB system prompt, byte-identical, uncached #162

coilyco-ops referenced this issue from a commit

2026-08-12 18:37:58 +00:00

Measure the prompt cache instead of inferring it

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-12 18:38:57 +00:00

Measure the prompt cache instead of inferring it #103

coilyco-ops commented

2026-08-12 18:39:41 +00:00

Member

The premise does not hold. This route is already cached.

LiteLLM's OTel callback writes a gen_ai.cost.cache_read_cost span attribute, and over the last 24h on ser8 it is nonzero for deepseek-v4-flash and absent for both Ornith groups. DeepSeek caches repeated prefixes automatically, with no request-side opt-in, no cache_control breakpoint, and no LiteLLM setting, so there was never anything to enable.

What was missing was the ability to tell. Agent Proxy read prompt_tokens and completion_tokens out of the upstream usage block and discarded every cache field beside them, so a fully cached route and a fully uncached one produced identical evidence. That is what made "byte-identical prefix, therefore uncached" a reasonable inference and an unfalsifiable one.

Cost cannot close it either. LiteLLM prices deepseek-v4-flash from its own bundled map rather than from Deploy-owned configuration, so cache_read_cost proves reads are happening but cannot be divided back into a hit rate.

#103 records the tokens instead: DeepSeek native, OpenAI-compatible, and Anthropic-style cache usage normalized into the response usage block, spans, three Prometheus counters, and the trajectory ledger. A provider that reports nothing stays absent rather than counting as a 100% miss, because Ollama reuses its KV cache without ever saying so.

The tool-roster half is also now measurable rather than arguable. The request span carries gen_ai.request.system_bytes, gen_ai.request.tool_count, and gen_ai.request.tool_bytes, so the trade between a narrower default roster and the tool calls it would lose has numbers at the one point every governed route passes through. For scale, sirens-echo/deepseek currently averages ~8.8k prompt tokens per turn over 24h.

No cache_control injector landed. No configured backend needs one, and an unexercised injector would be dead policy on the live routes.

Deploy's half is coilyco-bridge/deploy#405. The sirens-echo twin is coilyco-gaming/sirens-echo#162, whose prefix-shaping work stands on its own merits but should not be justified as fixing an uncached route.

The premise does not hold. This route is already cached. LiteLLM's OTel callback writes a `gen_ai.cost.cache_read_cost` span attribute, and over the last 24h on ser8 it is nonzero for `deepseek-v4-flash` and absent for both Ornith groups. DeepSeek caches repeated prefixes automatically, with no request-side opt-in, no `cache_control` breakpoint, and no LiteLLM setting, so there was never anything to enable. What was missing was the ability to tell. Agent Proxy read `prompt_tokens` and `completion_tokens` out of the upstream usage block and discarded every cache field beside them, so a fully cached route and a fully uncached one produced identical evidence. That is what made "byte-identical prefix, therefore uncached" a reasonable inference and an unfalsifiable one. Cost cannot close it either. LiteLLM prices `deepseek-v4-flash` from its own bundled map rather than from Deploy-owned configuration, so `cache_read_cost` proves reads are happening but cannot be divided back into a hit rate. #103 records the tokens instead: DeepSeek native, OpenAI-compatible, and Anthropic-style cache usage normalized into the response usage block, spans, three Prometheus counters, and the trajectory ledger. A provider that reports nothing stays absent rather than counting as a 100% miss, because Ollama reuses its KV cache without ever saying so. The tool-roster half is also now measurable rather than arguable. The request span carries `gen_ai.request.system_bytes`, `gen_ai.request.tool_count`, and `gen_ai.request.tool_bytes`, so the trade between a narrower default roster and the tool calls it would lose has numbers at the one point every governed route passes through. For scale, `sirens-echo/deepseek` currently averages ~8.8k prompt tokens per turn over 24h. No `cache_control` injector landed. No configured backend needs one, and an unexercised injector would be dead policy on the live routes. Deploy's half is coilyco-bridge/deploy#405. The sirens-echo twin is coilyco-gaming/sirens-echo#162, whose prefix-shaping work stands on its own merits but should not be justified as fixing an uncached route.

coilysiren closed this issue

2026-08-12 18:42:10 +00:00

coilyco-ops referenced this issue from coilyco-flight-deck/agent-compose

2026-08-12 20:35:58 +00:00

Composed role bodies span 7x with no budget, and the biggest one lands on a chat lane #275

coilyco-ops commented

2026-08-12 21:13:07 +00:00

Member

Closing numbers, now that organic traffic has run through the instrumented build. This is the measurement this issue asked for.

Across ~17 real Sirens Deep turns:

	value
prompt cache hit rate	99.17%
cached tokens	226,304
missed tokens	1,897, about 112 per turn
system prompt bytes	54,304 avg
tool count	17
tool schema bytes	7,029 avg

Three things this settles.

The route was cached, and well. Not marginally. The 53 KB prefix costs about 112 uncached tokens per turn. The "four byte ping costs what the hardest question costs" framing was true in tokens shipped and false in tokens billed.

The tool roster is 11%, not a large share. This issue estimated the 17 schemas as a large fraction of the 61 KB. They are 7,029 bytes against a 54,304 byte system prompt, so the composed body is roughly 89% of the request. Recorded on sirens-echo#162 so nobody trims the roster expecting a size win.

A prediction of mine failed. After correcting this issue's premise I argued the cost would reappear as full cold misses after idle gaps. It did not. A cold start would show as a single 10,000 to 15,000 token miss and total misses were 1,897. Struck on agent-compose#275.

What remains open is not cost. Context occupancy, decode bandwidth across the cached prefix, and attention across 17 always-resident tools are unaffected by caching, and the composed body carrying 89% of the request is tracked on 275.

Closing numbers, now that organic traffic has run through the instrumented build. This is the measurement this issue asked for. Across ~17 real Sirens Deep turns: | | value | |---|---| | prompt cache hit rate | **99.17%** | | cached tokens | 226,304 | | missed tokens | 1,897, about 112 per turn | | system prompt bytes | 54,304 avg | | tool count | 17 | | tool schema bytes | 7,029 avg | Three things this settles. **The route was cached, and well.** Not marginally. The 53 KB prefix costs about 112 uncached tokens per turn. The "four byte ping costs what the hardest question costs" framing was true in tokens shipped and false in tokens billed. **The tool roster is 11%, not a large share.** This issue estimated the 17 schemas as a large fraction of the 61 KB. They are 7,029 bytes against a 54,304 byte system prompt, so the composed body is roughly 89% of the request. Recorded on [sirens-echo#162](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/162) so nobody trims the roster expecting a size win. **A prediction of mine failed.** After correcting this issue's premise I argued the cost would reappear as full cold misses after idle gaps. It did not. A cold start would show as a single 10,000 to 15,000 token miss and total misses were 1,897. Struck on [agent-compose#275](https://forgejo.coilysiren.me/coilyco-flight-deck/agent-compose/issues/275). What remains open is not cost. Context occupancy, decode bandwidth across the cached prefix, and attention across 17 always-resident tools are unaffected by caching, and the composed body carrying 89% of the request is tracked on 275.

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-12 22:15:22 +00:00

Add common baseline agentic tools #155

coilyco-ops referenced this issue

2026-08-19 00:55:12 +00:00

Streaming requests never ask for usage, so the prompt-cache accounting cannot run on 94% of traffic #138

coilyco-ops referenced this issue from a commit

2026-08-19 00:55:34 +00:00

fix(cache): ask the provider for usage on OpenAI streaming

Rows
Columns

Every turn ships a 53 KB system prompt, byte-identical, uncached #101