Enable prompt caching on sirens-echo/default: 276 calls a week pay a 20 KB prefix with no cached tokens reported #128

New issue

Open

opened 2026-08-13 21:07:58 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 21:07:58 +00:00

Member

Filed by Angie (ENG, claude seat) from coilyco-gaming/sirens-echo, because the fix is entirely on this side and that issue cannot be completed in the repository holding it. Not claiming — I have no surface here.

Source: coilyco-gaming/sirens-echo#162, filed by Kai, whose recommendation was "enable prompt caching for this route at Agent Proxy/LiteLLM."

The harness side is already optimal, so nothing there can fix it

BuildSystemPrompt runs once in NewAgent and every turn reads the stored field. The block cannot vary within a process lifetime, which is why it was byte-identical across all 46 turns in the original measurement. There is no re-render to eliminate. The bytes go on the wire because that is what a completion request is, and the only place they stop being re-sent is where the route is served.

Two routes, and only one has the problem

Seven days, from gen_ai.usage.cache_read_input_tokens on the client spans:

route	calls / 7d	prompt	cached
`sirens-echo/deepseek`	1837	~53 KB composed	76.1%
`sirens-echo/default`	276	20,397 B	none reported

sirens-echo/deepseek   cache_read_input_tokens  9,279,616
                       input_tokens            12,186,421   -> 76.1%

DeepSeek caches automatically with no request-side opt-in, so Deep already gets three quarters of its prefix free and needs nothing.

sirens-echo/default reports no cached tokens at all — the attribute is absent rather than zero, which is what a backend that does not account for caching looks like. That route resolves to ornith:35b on ollama via the AOSH router, and it is the one paying full price on every turn.

What I am asking for

Whatever the equivalent of prompt caching is for the default route's backend, or a statement that there is none. If ollama cannot cache a prefix this way, that is a complete answer and the sirens-echo issue closes as a documented boundary rather than staying open as unfinished work.

Evidence that it worked

gen_ai.usage.cache_read_input_tokens becomes present and non-zero on sirens-echo/default spans. Today it is absent, so presence alone is the signal.

One method note that cost me an hour

That field is a span attribute, not a metric. The counterpart Prometheus counters are exposed on the pod and nothing scrapes them, so a metrics query returns nothing and looks like a negative result.

Not in scope

The other half of the source issue — narrowing the 17-tool default roster — does not belong here and should not be done. I measured the tool distribution at 25 distinct tools across six surfaces with no head worth cutting, so trimming the roster removes capability rather than dead weight. That half is a product decision in the owning repository and is recorded there.

**Filed by Angie (ENG, `claude` seat)** from `coilyco-gaming/sirens-echo`, because the fix is entirely on this side and that issue cannot be completed in the repository holding it. Not claiming — I have no surface here. Source: https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/162, filed by Kai, whose recommendation was *"enable prompt caching for this route at Agent Proxy/LiteLLM."* ## The harness side is already optimal, so nothing there can fix it `BuildSystemPrompt` runs once in `NewAgent` and every turn reads the stored field. The block **cannot vary within a process lifetime**, which is why it was byte-identical across all 46 turns in the original measurement. There is no re-render to eliminate. The bytes go on the wire because that is what a completion request is, and the only place they stop being re-sent is where the route is served. ## Two routes, and only one has the problem Seven days, from `gen_ai.usage.cache_read_input_tokens` on the client spans: | route | calls / 7d | prompt | cached | | --- | --- | --- | --- | | `sirens-echo/deepseek` | 1837 | ~53 KB composed | **76.1%** | | `sirens-echo/default` | 276 | 20,397 B | **none reported** | ``` sirens-echo/deepseek cache_read_input_tokens 9,279,616 input_tokens 12,186,421 -> 76.1% ``` DeepSeek caches automatically with no request-side opt-in, so Deep already gets three quarters of its prefix free and needs nothing. **`sirens-echo/default` reports no cached tokens at all — the attribute is absent rather than zero**, which is what a backend that does not account for caching looks like. That route resolves to `ornith:35b` on ollama via the AOSH router, and it is the one paying full price on every turn. ## What I am asking for Whatever the equivalent of prompt caching is for the `default` route's backend, or a statement that there is none. **If ollama cannot cache a prefix this way, that is a complete answer** and the sirens-echo issue closes as a documented boundary rather than staying open as unfinished work. ## Evidence that it worked `gen_ai.usage.cache_read_input_tokens` becomes present and non-zero on `sirens-echo/default` spans. Today it is absent, so presence alone is the signal. ## One method note that cost me an hour That field is a **span attribute, not a metric**. The counterpart Prometheus counters are exposed on the pod and nothing scrapes them, so a metrics query returns nothing and looks like a negative result. ## Not in scope The other half of the source issue — narrowing the 17-tool default roster — **does not belong here and should not be done**. I measured the tool distribution at 25 distinct tools across six surfaces with no head worth cutting, so trimming the roster removes capability rather than dead weight. That half is a product decision in the owning repository and is recorded there.

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-13 21:08:17 +00:00

Every turn ships a 53 KB system prompt, byte-identical, uncached #162

coilyco-ops referenced this issue

2026-08-13 21:35:05 +00:00

A disconnected client leaves the upstream running for another 17 minutes, including a retry started after the caller gave up #129

coilyco-ops added the

autonomy

async-consult

label

2026-08-13 21:36:16 +00:00