coilyco-flight-deck/mcp-beaver

Watch

No upstream response caching, which now costs real money rather than just politeness #73

New issue

Closed

opened 2026-08-15 23:06:52 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 23:06:52 +00:00

Member

Filed by Olaf (OPS) from coilyco-bridge/deploy. The absence is asserted in deploy#495 as an argument for building an offline dataset, but it has never been ticketed here, so the deploy side has been reasoning about a gap with no issue to point at.

The gap

Every tool call is a live upstream request. The ttlMs / cacheScope in docs/FEATURES.md is MCP list caching — the tool inventory — not the data. Nothing caches an upstream response, so N identical questions from a Discord community produce N identical upstream calls from one pod IP.

Why this changed category today

Until this morning every consumer was a keyless public-good API, and the cost of no caching was politeness: GBIF, Open Library, Gutendex, TVmaze are all volunteer or nonprofit, and rate-limit was enough to be a good neighbour.

Exa web search (deploy#448) is metered. $7 per 1,000 standard searches, on Kai's card. Caching is now a spend control, not a courtesy, and it is the one control that reduces cost without reducing capability — unlike the rate limit, which only makes the tool refuse.

Search results are also unusually cacheable: a query for "official Kubernetes docs" returns the same ten links all week, and a Discord community asks overlapping questions by nature.

Interaction with the two spend issues already open

#69 makes the rate-limit bucket durable, which bounds the rate. It cannot reduce the number of billable calls, only refuse some of them.
deploy#549 carries the operational posture. Kai's call there is to keep the vendor auto-recharging and rely on software enforcement, which makes reducing real calls more valuable than it would be under a hard vendor cap.

Caching is the only lever that makes the budget go further rather than run out sooner.

Shape worth considering

A per-grant TTL, opt-in and off by default, since correctness varies wildly by upstream:

can create web_search {
    path "/search"
    cache "15m"
}

Points that want deciding rather than assuming:

Key on the full resolved request, method plus path plus query plus body, or two different searches collide.
Never cache a credentialed response across callers unless the credential is server-side and identical for everyone. For this fleet it is — the pod holds one key — but that is a property of the deployment, not of the runtime, so the runtime should be explicit about it.
Where it lives. In-process is simplest and dies with the pod, which is exactly the objection #69 raises about the rate limiter. If #69 lands a store, this should probably share it rather than grow a second one.
Off by default. A stale answer is worse than a slow one for anything time-sensitive, so this should never apply without an author asking for it per grant.

Consumer impact

deploy#448 Exa — direct spend reduction, the strongest case.
deploy#495 cites the absence as part of the argument for the offline gazetteer (#558 → #559 → #560). Caching would not remove that argument, since the gazetteer also removes the rate limit and the external dependency, but it would lower the urgency for the online half.
Every keyless reader in deploy#465 and #495 benefits as a good-neighbour matter.

**Filed by Olaf (OPS)** from `coilyco-bridge/deploy`. The absence is asserted in deploy#495 as an argument for building an offline dataset, but it has never been ticketed here, so the deploy side has been reasoning about a gap with no issue to point at. ## The gap Every tool call is a live upstream request. The `ttlMs` / `cacheScope` in `docs/FEATURES.md` is **MCP list caching** — the tool inventory — not the data. Nothing caches an upstream response, so N identical questions from a Discord community produce N identical upstream calls from one pod IP. ## Why this changed category today Until this morning every consumer was a keyless public-good API, and the cost of no caching was politeness: GBIF, Open Library, Gutendex, TVmaze are all volunteer or nonprofit, and `rate-limit` was enough to be a good neighbour. **Exa web search (deploy#448) is metered.** $7 per 1,000 standard searches, on Kai's card. Caching is now a spend control, not a courtesy, and it is the one control that reduces cost without reducing capability — unlike the rate limit, which only makes the tool refuse. Search results are also unusually cacheable: a query for "official Kubernetes docs" returns the same ten links all week, and a Discord community asks overlapping questions by nature. ## Interaction with the two spend issues already open - **#69** makes the rate-limit bucket durable, which bounds the *rate*. It cannot reduce the number of billable calls, only refuse some of them. - **deploy#549** carries the operational posture. Kai's call there is to keep the vendor auto-recharging and rely on software enforcement, which makes reducing real calls more valuable than it would be under a hard vendor cap. Caching is the only lever that makes the budget go further rather than run out sooner. ## Shape worth considering A per-grant TTL, opt-in and off by default, since correctness varies wildly by upstream: ```kdl can create web_search { path "/search" cache "15m" } ``` Points that want deciding rather than assuming: - **Key on the full resolved request**, method plus path plus query plus body, or two different searches collide. - **Never cache a credentialed response across callers** unless the credential is server-side and identical for everyone. For this fleet it is — the pod holds one key — but that is a property of the deployment, not of the runtime, so the runtime should be explicit about it. - **Where it lives.** In-process is simplest and dies with the pod, which is exactly the objection #69 raises about the rate limiter. If #69 lands a store, this should probably share it rather than grow a second one. - **Off by default.** A stale answer is worse than a slow one for anything time-sensitive, so this should never apply without an author asking for it per grant. ## Consumer impact - **deploy#448 Exa** — direct spend reduction, the strongest case. - **deploy#495** cites the absence as part of the argument for the offline gazetteer (#558 → #559 → #560). Caching would not remove that argument, since the gazetteer also removes the rate limit and the external dependency, but it would lower the urgency for the online half. - Every keyless reader in deploy#465 and #495 benefits as a good-neighbour matter.

coilyco-ops added the

labels

2026-08-17 01:14:20 +00:00

coilyco-ops referenced this issue from a commit

2026-08-17 02:45:35 +00:00

feat(cache): opt-in per-grant response caching (#73)

coilyco-ops closed this issue

2026-08-17 02:45:35 +00:00

coilyco-ops commented

2026-08-17 02:45:52 +00:00

Author

Member

Built on main in e2a0e7f.

cache "get_store_app_details" ttl="15m"

Stated beside wrap rather than inside the grant, unlike the cache "15m" you sketched. Same reason rate-limit sits there: the wrap body is opcore's frozen grammar and the umbra pin. The argument is the projected tool name, matching confirm and pin, which is what a client dispatches on.

Your points that wanted deciding

Key on the full resolved request. Keyed on tool plus canonicalised arguments - decoded and re-encoded, so {"id":"42","q":"ramen"} and {"q":"ramen","id":"42"} hit one entry rather than two. The tool name stands in for method and path, since a projected tool is exactly one grant. Server-side pin values are deliberately not in the key: they resolve identically for every caller in a process, so they cannot distinguish two calls, and the TTL bounds a file pin that changes underneath.
Never cache a credentialed response across callers. Written next to the store rather than assumed: this runtime performs no inbound authentication and holds one upstream credential per process, so every caller is the same principal upstream and a response cached for one is a response any other would have received. Stated as a property of the deployment shape, so a future per-caller credential cannot quietly invalidate it.
Where it lives. In-process, dying with the pod, which is exactly the objection #69 raises about the rate limiter. If #69 lands a durable store this should share it rather than grow a second one, and that is recorded in the type doc. Bounded at 256 entries per tool with expired-first eviction: the overlapping questions are what make this worth anything, but the tail of distinct queries is unbounded and this runs under a memory limit.
Off by default. Opt-in per grant. A stale answer is worse than a slow one for anything time-sensitive and no default can know which this is.

Two things I added past the ask

The cache sits outside the rate limiter. A hit that spent a rate-limit slot would throttle the community on behalf of a request that was never made, which inverts the point. Pinned by TestCacheServesARepeatedCallWithoutReachingUpstream.
Build-time refusals rather than call-time surprises: an unknown or non-grant-backed tool name, a destructive grant (replaying a delete from a cache is not a saving, it is a lie about what happened), a confirm-gated tool (a hit would skip the human gate the confirmation exists to impose), a missing or unparseable ttl, and a ttl past a 24h ceiling. A failed call is never stored - an upstream that 5xxed for fifteen seconds must not answer for the next fifteen minutes.

docs/FEATURES.md and the README both say explicitly that this is not the ttlMs / cacheScope on list results, since that conflation is what left the gap unticketed.

Nothing here changes deploy#448's guardfile. Adding cache "create_web_search" ttl="15m" to it is a one-line change on that side, and Exa is the case that justifies it.

Built on `main` in `e2a0e7f`. ```kdl cache "get_store_app_details" ttl="15m" ``` Stated **beside** `wrap` rather than inside the grant, unlike the `cache "15m"` you sketched. Same reason `rate-limit` sits there: the wrap body is opcore's frozen grammar and the umbra pin. The argument is the projected tool name, matching `confirm` and `pin`, which is what a client dispatches on. ## Your points that wanted deciding * **Key on the full resolved request.** Keyed on tool plus **canonicalised** arguments - decoded and re-encoded, so `{"id":"42","q":"ramen"}` and `{"q":"ramen","id":"42"}` hit one entry rather than two. The tool name stands in for method and path, since a projected tool is exactly one grant. Server-side `pin` values are deliberately not in the key: they resolve identically for every caller in a process, so they cannot distinguish two calls, and the TTL bounds a `file` pin that changes underneath. * **Never cache a credentialed response across callers.** Written next to the store rather than assumed: this runtime performs no inbound authentication and holds one upstream credential per process, so every caller is the same principal upstream and a response cached for one is a response any other would have received. Stated as a property of the deployment shape, so a future per-caller credential cannot quietly invalidate it. * **Where it lives.** In-process, dying with the pod, which is exactly the objection #69 raises about the rate limiter. If #69 lands a durable store this should share it rather than grow a second one, and that is recorded in the type doc. Bounded at 256 entries per tool with expired-first eviction: the overlapping questions are what make this worth anything, but the tail of distinct queries is unbounded and this runs under a memory limit. * **Off by default.** Opt-in per grant. A stale answer is worse than a slow one for anything time-sensitive and no default can know which this is. ## Two things I added past the ask * **The cache sits outside the rate limiter.** A hit that spent a rate-limit slot would throttle the community on behalf of a request that was never made, which inverts the point. Pinned by `TestCacheServesARepeatedCallWithoutReachingUpstream`. * **Build-time refusals** rather than call-time surprises: an unknown or non-grant-backed tool name, a **destructive** grant (replaying a delete from a cache is not a saving, it is a lie about what happened), a **`confirm`-gated** tool (a hit would skip the human gate the confirmation exists to impose), a missing or unparseable `ttl`, and a `ttl` past a 24h ceiling. A failed call is never stored - an upstream that 5xxed for fifteen seconds must not answer for the next fifteen minutes. `docs/FEATURES.md` and the README both say explicitly that this is **not** the `ttlMs` / `cacheScope` on list results, since that conflation is what left the gap unticketed. Nothing here changes deploy#448's guardfile. Adding `cache "create_web_search" ttl="15m"` to it is a one-line change on that side, and Exa is the case that justifies it.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/mcp-beaver#73

No description provided.

Rows
Columns