coilyco-flight-deck/mcp-beaver

Watch

rate-limit resets on every restart, so it cannot express a spend budget #69

New issue

Open

opened 2026-08-15 20:36:05 +00:00 by coilyco-ops · 5 comments

coilyco-ops commented

2026-08-15 20:36:05 +00:00

Member

Requested by Kai. rate-limit is currently a rate shape. Making it a budget needs the bucket to survive a process restart, and that needs a backing store. Filing with the measurements rather than the fix, since this is a runtime change and Engineering owns it.

Driver: coilyco-bridge/deploy now runs a keyed, metered upstream for the first time (Exa web search, deploy#448). Every prior consumer was a keyless public-good API where the bucket existed to be a polite neighbour, and losing it on restart cost nothing. With a metered upstream the same bucket is the only software bound on spend, and it is not one.

What it does today

internal/mcpserver/ratelimit.go:

return rate.NewLimiter(rate.Every(d/time.Duration(n)), n), nil

Constructed in memory when the spec is parsed at process start. Nothing persists it.

Measured against a build of main on 2026-08-15 with rate-limit "2/24h":

Calls one and two returned real results immediately. Burst equals count, so the whole window's allowance is available at once.
Calls three and four failed in 0.19s with ward-mcp: rate limit wait: rate: Wait(n=1) would exceed context deadline, isError: true.
Budget fully spent, process restarted, next call succeeded immediately on a refilled burst.

The 0.19s behaviour is good and should be preserved: limiter.Wait(ctx) errors as soon as the projected wait exceeds the request deadline instead of blocking to it, so a spent budget is a fast reportable tool error rather than a hung turn.

Why the restart matters more than it sounds

An MCP pod rolls whenever its guardfile or values change. The two Exa pods rolled twice on their first day. So a 72/24h budget intended as $0.50/day returns to full on each roll, several times a day.

And there are two buckets, not one. Echo and Deep run separate pods from the same guardfile with separate limiters, so every figure doubles. A shared store would fix this too, and that is arguably the larger win: it is the difference between "$0.50/day per lane" and "$0.50/day".

What is wanted

Durable — the bucket survives a process restart.
Optionally shared — two releases of the same guardfile can charge one bucket, so a per-lane figure is not silently doubled. Worth deciding whether shared is the default or opt-in, and what the sharing key is: the wrap ward mcp <name> server name is the obvious candidate, but two lanes deliberately running independent budgets is also legitimate, so it probably needs to be explicit in the spec rather than inferred.
Fail-open or fail-closed on store outage — needs a deliberate answer. Fail-open means a Redis blip removes the spend bound; fail-closed means it removes the tool. For a metered upstream fail-closed is probably right, and it is the opposite of what a politeness bucket would want, so this may need to be per-spec.
Optional — a keyless consumer with no store configured must keep working exactly as it does now, in memory. This cannot become a hard dependency for the ten-odd keyless servers already deployed.

Store choice

Kai suggested Redis and asked for whatever is most amenable to very small k8s deployments. Recording the alternatives so the choice is deliberate:

Redis — the natural fit. INCR plus EXPIRE, or a Lua script for a proper token bucket, is a well-trodden pattern with atomicity for free. Cost is a new component in every namespace that wants a budget.
Postgres — already deployed in both sirens namespaces as the harness job store, so it adds no component. Cost is that the MCP pods currently hold no database credential, and a row-per-bucket with SELECT FOR UPDATE is heavier than this needs.
A Kubernetes Lease or ConfigMap — no new component at all, and coordination.k8s.io/v1 exists for exactly this class of tiny shared state. Cost is API-server write traffic per call and RBAC the MCP pods do not have today.
A file on a PVC — simplest, but does not share across pods, and a PVC per MCP pod on a single-node homelab is worse than the problem.

My read is Redis if a budget is expected to be common, Postgres if it stays rare, because reusing a running database beats standing up a component for one counter. Engineering's call.

Acceptance

A spec can name a durable bucket, and a restart mid-window does not restore the allowance.
Two releases sharing a bucket charge it once, demonstrably.
A spec with no store configured behaves exactly as today.
Store unavailability has a documented and tested behaviour.
The fast-fail on exhaustion is preserved: no change to the 0.19s error path.

Related: deploy#549 carries the operational side and Kai's decision to keep Exa auto-recharging and rely on software enforcement rather than a vendor cap.

**Requested by Kai.** `rate-limit` is currently a rate shape. Making it a budget needs the bucket to survive a process restart, and that needs a backing store. Filing with the measurements rather than the fix, since this is a runtime change and Engineering owns it. Driver: `coilyco-bridge/deploy` now runs a **keyed, metered** upstream for the first time (Exa web search, deploy#448). Every prior consumer was a keyless public-good API where the bucket existed to be a polite neighbour, and losing it on restart cost nothing. With a metered upstream the same bucket is the only software bound on spend, and it is not one. ## What it does today `internal/mcpserver/ratelimit.go`: ```go return rate.NewLimiter(rate.Every(d/time.Duration(n)), n), nil ``` Constructed in memory when the spec is parsed at process start. Nothing persists it. Measured against a build of main on 2026-08-15 with `rate-limit "2/24h"`: - Calls one and two returned real results immediately. **Burst equals count**, so the whole window's allowance is available at once. - Calls three and four failed in **0.19s** with `ward-mcp: rate limit wait: rate: Wait(n=1) would exceed context deadline`, `isError: true`. - Budget fully spent, **process restarted, next call succeeded immediately** on a refilled burst. The 0.19s behaviour is good and should be preserved: `limiter.Wait(ctx)` errors as soon as the projected wait exceeds the request deadline instead of blocking to it, so a spent budget is a fast reportable tool error rather than a hung turn. ## Why the restart matters more than it sounds An MCP pod rolls whenever its guardfile or values change. The two Exa pods rolled twice on their first day. So a `72/24h` budget intended as $0.50/day returns to full on each roll, several times a day. **And there are two buckets, not one.** Echo and Deep run separate pods from the same guardfile with separate limiters, so every figure doubles. A shared store would fix this too, and that is arguably the larger win: it is the difference between "$0.50/day per lane" and "$0.50/day". ## What is wanted 1. **Durable** — the bucket survives a process restart. 2. **Optionally shared** — two releases of the same guardfile can charge one bucket, so a per-lane figure is not silently doubled. Worth deciding whether shared is the default or opt-in, and what the sharing key is: the `wrap ward mcp <name>` server name is the obvious candidate, but two lanes deliberately running independent budgets is also legitimate, so it probably needs to be explicit in the spec rather than inferred. 3. **Fail-open or fail-closed on store outage** — needs a deliberate answer. Fail-open means a Redis blip removes the spend bound; fail-closed means it removes the tool. For a metered upstream fail-closed is probably right, and it is the opposite of what a politeness bucket would want, so this may need to be per-spec. 4. **Optional** — a keyless consumer with no store configured must keep working exactly as it does now, in memory. This cannot become a hard dependency for the ten-odd keyless servers already deployed. ## Store choice Kai suggested Redis and asked for whatever is most amenable to very small k8s deployments. Recording the alternatives so the choice is deliberate: - **Redis** — the natural fit. `INCR` plus `EXPIRE`, or a Lua script for a proper token bucket, is a well-trodden pattern with atomicity for free. Cost is a new component in every namespace that wants a budget. - **Postgres** — **already deployed in both sirens namespaces** as the harness job store, so it adds no component. Cost is that the MCP pods currently hold no database credential, and a row-per-bucket with `SELECT FOR UPDATE` is heavier than this needs. - **A Kubernetes Lease or ConfigMap** — no new component at all, and `coordination.k8s.io/v1` exists for exactly this class of tiny shared state. Cost is API-server write traffic per call and RBAC the MCP pods do not have today. - **A file on a PVC** — simplest, but does not share across pods, and a PVC per MCP pod on a single-node homelab is worse than the problem. My read is Redis if a budget is expected to be common, Postgres if it stays rare, because reusing a running database beats standing up a component for one counter. Engineering's call. ## Acceptance - A spec can name a durable bucket, and a restart mid-window does **not** restore the allowance. - Two releases sharing a bucket charge it once, demonstrably. - A spec with no store configured behaves exactly as today. - Store unavailability has a documented and tested behaviour. - The fast-fail on exhaustion is preserved: no change to the 0.19s error path. Related: deploy#549 carries the operational side and Kai's decision to keep Exa auto-recharging and rely on software enforcement rather than a vendor cap.

coilyco-ops referenced this issue from a commit

2026-08-15 20:37:21 +00:00

feat(sirens-echo): tighten Exa to Kai's 72/24h spend target

coilyco-ops commented

2026-08-15 20:38:52 +00:00

Author

Member

One thing about the Redis option that would quietly reintroduce the bug, worth flagging before the store is chosen.

A default Redis in Kubernetes is memory-only. No PVC, no AOF, no RDB. Restarting the Redis pod loses every key, which resets the bucket exactly the way restarting ward-mcp does today. The failure would just move one layer down and get harder to see, because the MCP pod would look healthy and the budget would silently be full again.

So Redis only satisfies acceptance criterion one if it is deployed with a PVC and AOF enabled, ideally appendfsync everysec. That is a stateful component with a volume, a backup question, and an upgrade path, which is a materially bigger ask than "add a tiny Redis" sounds.

That changes the comparison in the issue body:

Postgres already has durable storage in both sirens namespaces, because it is the harness job store and was chosen over a volume on deploy#464 precisely so state survives a roll. Reusing it means the durability requirement is met by something already load-bearing and already backed up, and the only new work is a credential for the MCP pods and one small table.
Redis needs that durability configured from scratch, per namespace, for one counter.
A Kubernetes Lease is durable by construction, since etcd is the cluster's own store, and needs no new component or volume at all. The cost is API-server writes on the hot path and RBAC the MCP pods do not have.

Not overriding the store choice, which is Engineering's. But "Redis is the lightweight option" is only true if the bucket is allowed to be lossy, and the entire point of this issue is that it is not.

Whichever is picked, the acceptance test should be restart the store, not just the MCP pod, or it will pass while the bug survives.

**One thing about the Redis option that would quietly reintroduce the bug, worth flagging before the store is chosen.** A default Redis in Kubernetes is **memory-only**. No PVC, no AOF, no RDB. Restarting the Redis pod loses every key, which resets the bucket exactly the way restarting ward-mcp does today. The failure would just move one layer down and get harder to see, because the MCP pod would look healthy and the budget would silently be full again. So Redis only satisfies acceptance criterion one if it is deployed with a PVC and AOF enabled, ideally `appendfsync everysec`. That is a stateful component with a volume, a backup question, and an upgrade path, which is a materially bigger ask than "add a tiny Redis" sounds. That changes the comparison in the issue body: - **Postgres** already has durable storage in both sirens namespaces, because it is the harness job store and was chosen over a volume on deploy#464 precisely so state survives a roll. Reusing it means the durability requirement is met by something already load-bearing and already backed up, and the only new work is a credential for the MCP pods and one small table. - **Redis** needs that durability configured from scratch, per namespace, for one counter. - **A Kubernetes Lease** is durable by construction, since etcd is the cluster's own store, and needs no new component or volume at all. The cost is API-server writes on the hot path and RBAC the MCP pods do not have. Not overriding the store choice, which is Engineering's. But "Redis is the lightweight option" is only true if the bucket is allowed to be lossy, and the entire point of this issue is that it is not. Whichever is picked, the acceptance test should be **restart the store, not just the MCP pod**, or it will pass while the bug survives.

coilyco-ops commented

2026-08-15 20:46:34 +00:00

Author

Member

Kai's call: Redis-shaped, and Postgres is out. Recording the requirement that drives it, because it is sharper than "which database".

The requirement is schema-free growth: adding or removing an MCP on a deployment must need no change to the datastore at all. A new server starts writing a new key and that is the whole of it.

Redis satisfies that by construction. This decision stands and the rest of this comment does not reopen it.

One premise worth correcting, so nobody argues Postgres back in on it later

"Postgres needs data migrations" is right about the mechanism and not about the per-MCP case. A bucket table would be roughly (bucket_key text primary key, tokens double precision, updated_at timestamptz), and a new MCP is a new row, not a new column. Adding one would need no migration.

The real Postgres cost is elsewhere and is still disqualifying:

mcp-beaver has no migration mechanism at all today, so the one-time schema means adding one to a small Go binary for a single counter, plus every future evolution of it.
The MCP pods hold no database credential and would each need one.
The harness job store would become a shared dependency of the whole MCP fleet, so a job-store incident becomes an MCP-fleet incident.

So: right conclusion, and the reason is the migration mechanism rather than per-MCP migrations.

An argument for Redis that strengthens the case

TTL means removed MCPs garbage-collect themselves. Set an expiry a little past the window and a decommissioned server's key simply vanishes. The Postgres shape accumulates dead rows forever or needs a reaper nobody will write. That is the flexibility requirement holding on the removal side as well as the addition side, and it is the part a relational store genuinely cannot match cheaply.

The persistence caveat from my earlier comment still applies

Restating because it is the one way this lands and still fails: a default Redis in Kubernetes is memory-only. It needs a PVC and AOF, ideally appendfsync everysec, or restarting Redis resets every bucket exactly the way restarting ward-mcp does now, one layer down and harder to see.

The acceptance test must therefore restart the store, not just the MCP pod.

Worth considering Valkey rather than Redis proper: drop-in protocol compatibility, and it avoids the post-2024 Redis licence question for a self-hosted estate. Engineering's call, and it changes nothing about the design.

Design shape this implies

Key like ratelimit:<server-name>, created on demand, no registration step and no fixed inventory anywhere.
A Lua script for the token-bucket read-modify-write, so the refill is atomic without a round trip per token.
TTL on every key so removal is automatic.
Store address from the environment, absent means today's in-memory limiter, which keeps the keyless servers working untouched.

**Kai's call: Redis-shaped, and Postgres is out. Recording the requirement that drives it, because it is sharper than "which database".** The requirement is **schema-free growth**: adding or removing an MCP on a deployment must need no change to the datastore at all. A new server starts writing a new key and that is the whole of it. Redis satisfies that by construction. **This decision stands** and the rest of this comment does not reopen it. ## One premise worth correcting, so nobody argues Postgres back in on it later "Postgres needs data migrations" is right about the mechanism and not about the per-MCP case. A bucket table would be roughly `(bucket_key text primary key, tokens double precision, updated_at timestamptz)`, and **a new MCP is a new row, not a new column**. Adding one would need no migration. The real Postgres cost is elsewhere and is still disqualifying: - mcp-beaver has **no migration mechanism at all** today, so the one-time schema means adding one to a small Go binary for a single counter, plus every future evolution of it. - The MCP pods hold no database credential and would each need one. - The harness job store would become a shared dependency of the whole MCP fleet, so a job-store incident becomes an MCP-fleet incident. So: right conclusion, and the reason is the migration **mechanism** rather than per-MCP migrations. ## An argument for Redis that strengthens the case **TTL means removed MCPs garbage-collect themselves.** Set an expiry a little past the window and a decommissioned server's key simply vanishes. The Postgres shape accumulates dead rows forever or needs a reaper nobody will write. That is the flexibility requirement holding on the removal side as well as the addition side, and it is the part a relational store genuinely cannot match cheaply. ## The persistence caveat from my earlier comment still applies Restating because it is the one way this lands and still fails: **a default Redis in Kubernetes is memory-only.** It needs a PVC and AOF, ideally `appendfsync everysec`, or restarting Redis resets every bucket exactly the way restarting ward-mcp does now, one layer down and harder to see. The acceptance test must therefore **restart the store, not just the MCP pod.** Worth considering **Valkey** rather than Redis proper: drop-in protocol compatibility, and it avoids the post-2024 Redis licence question for a self-hosted estate. Engineering's call, and it changes nothing about the design. ## Design shape this implies - Key like `ratelimit:<server-name>`, created on demand, no registration step and no fixed inventory anywhere. - A Lua script for the token-bucket read-modify-write, so the refill is atomic without a round trip per token. - TTL on every key so removal is automatic. - Store address from the environment, absent means today's in-memory limiter, which keeps the keyless servers working untouched.

coilyco-ops referenced this issue from a commit

2026-08-15 21:04:10 +00:00

feat: durable Valkey store for mcp-beaver rate limits, and raise Echo's turn budget

coilyco-ops commented

2026-08-15 21:08:14 +00:00

Author

Member

The store is up and waiting. Nothing to stand up when you pick this off. Olaf (DevOps, claude seat), 21:05Z.

Deployed as coilyco-bridge/deploy services/mcp-ratelimit, commit bb29130 on main, rolled by CD.

redis://mcp-ratelimit.mcp-ratelimit.svc.cluster.local:6379

Valkey 8.1.9, ClusterIP, no NodePort and no ingress. Its own namespace rather than a lane's, because the point is that every mcp-beaver deployment charges one bucket.

What is verified

Pod 1/1 Running, PVC mcp-ratelimit-data Bound, 1Gi, local-path.
ExternalSecret mcp-ratelimit-auth SecretSynced.
AOF is on and writing to the volume, which is the whole point rather than a detail:

* Creating AOF base file appendonly.aof.1.base.rdb on server start
* Creating AOF incr file appendonly.aof.1.incr.aof on server start

Those files are on the host volume rather than the container layer, confirmed from the node rather than from the pod: the mcp-ratelimit namespace reports used_bytes: 16384 with used_bytes_complete: true against /var/lib/rancher/k3s/storage.

What is NOT verified, and it is the important one

I have not done a write, restart, read. aosguard ops kubectl grants no exec and no port-forward, and the Service is ClusterIP, so there is no path from here to valkey-cli. What is established is that AOF is enabled and landing on a Bound PVC; what is not established is that a key written before a restart is readable after one.

That is exactly the acceptance criterion in this issue, and it is yours to run, not a formality I have already covered. Restart this pod, not just an MCP pod.

Auth

Required. /mcp-ratelimit/password in SSM. The ExternalSecret templates two keys into the mcp-ratelimit-auth Secret: VALKEY_PASSWORD for a bare password and URL for a ready-to-dial redis://:pass@host:6379/0.

There is no NetworkPolicy fronting the port, so the password is the only boundary rather than an assumption worth relying on.

A consumer in another namespace needs its own ExternalSecret against the same SSM parameter, since a Secret does not cross namespaces. The two Sirens lanes will each need one. Do not copy the value.

Config the design should not fight

Set as flags rather than a mounted config, so there is one place to read what is running:

--appendonly yes --appendfsync everysec --dir /data
--maxmemory 128mb --maxmemory-policy noeviction

noeviction is deliberate and load-bearing: evicting a bucket key is precisely a silent budget reset, so an lru or volatile policy would reintroduce this bug in its quietest form. With TTL'd keys of a few bytes the limit is unreachable in practice, so its real job is to fail loudly if something ever writes bulk data here.

Full rationale and the per-resource notes are in services/mcp-ratelimit/README.md and the manifest header.

**The store is up and waiting. Nothing to stand up when you pick this off.** Olaf (DevOps, claude seat), 21:05Z. Deployed as `coilyco-bridge/deploy` `services/mcp-ratelimit`, commit `bb29130` on main, rolled by CD. ``` redis://mcp-ratelimit.mcp-ratelimit.svc.cluster.local:6379 ``` Valkey 8.1.9, ClusterIP, no NodePort and no ingress. Its own namespace rather than a lane's, because the point is that every mcp-beaver deployment charges one bucket. ## What is verified - Pod `1/1 Running`, PVC `mcp-ratelimit-data` **Bound**, 1Gi, local-path. - ExternalSecret `mcp-ratelimit-auth` **SecretSynced**. - **AOF is on and writing to the volume**, which is the whole point rather than a detail: ``` * Creating AOF base file appendonly.aof.1.base.rdb on server start * Creating AOF incr file appendonly.aof.1.incr.aof on server start ``` - Those files are on the host volume rather than the container layer, confirmed from the node rather than from the pod: the `mcp-ratelimit` namespace reports `used_bytes: 16384` with `used_bytes_complete: true` against `/var/lib/rancher/k3s/storage`. ## What is NOT verified, and it is the important one **I have not done a write, restart, read.** `aosguard ops kubectl` grants no `exec` and no `port-forward`, and the Service is ClusterIP, so there is no path from here to `valkey-cli`. What is established is that AOF is enabled and landing on a Bound PVC; what is not established is that a key written before a restart is readable after one. That is exactly the acceptance criterion in this issue, and it is **yours to run**, not a formality I have already covered. Restart this pod, not just an MCP pod. ## Auth Required. `/mcp-ratelimit/password` in SSM. The ExternalSecret templates two keys into the `mcp-ratelimit-auth` Secret: `VALKEY_PASSWORD` for a bare password and `URL` for a ready-to-dial `redis://:pass@host:6379/0`. There is no NetworkPolicy fronting the port, so the password is the only boundary rather than an assumption worth relying on. **A consumer in another namespace needs its own ExternalSecret against the same SSM parameter**, since a Secret does not cross namespaces. The two Sirens lanes will each need one. Do not copy the value. ## Config the design should not fight Set as flags rather than a mounted config, so there is one place to read what is running: ``` --appendonly yes --appendfsync everysec --dir /data --maxmemory 128mb --maxmemory-policy noeviction ``` `noeviction` is deliberate and load-bearing: evicting a bucket key is precisely a silent budget reset, so an `lru` or `volatile` policy would reintroduce this bug in its quietest form. With TTL'd keys of a few bytes the limit is unreachable in practice, so its real job is to fail loudly if something ever writes bulk data here. Full rationale and the per-resource notes are in `services/mcp-ratelimit/README.md` and the manifest header.

coilyco-ops referenced this issue from a commit

2026-08-15 21:31:05 +00:00

feat(aosguard): guarded redis-cli surface, and redis-tools in dev-base

coilyco-ops commented

2026-08-15 21:37:14 +00:00

Author

Member

Correction and an upgrade to the evidence: the store is Redis, not Valkey, and the AOF load path is now demonstrated rather than inferred. Olaf (DevOps, claude seat).

Redis, not Valkey

Kai reverted my Valkey suggestion and was right to. The only argument I offered for the fork was the post-2024 licence change, and it does not reach a homelab: Redis 8 is AGPLv3, OSI-approved, and nothing here redistributes it. A fork bought no capability and cost the more familiar name.

Live now on redis@sha256:978f0e01593e65eed801f2402944efcd936d43b5027e4908a7897baf88ed6241, commit 7837039 in coilyco-bridge/deploy. Nothing else moved: redis:8-alpine carries the same uid 999 and gid 1000 as the valkey image, read from its /etc/passwd the same way, so the security context is unchanged. Same flags, same PVC.

The environment variable is REDIS_PASSWORD, not VALKEY_PASSWORD, and the manifest is services/mcp-ratelimit/deploy/redis.yml. Anything written against my earlier comment needs that rename.

The engine swap accidentally proved the load path

My previous comment said durability was inferred from AOF files existing on a Bound PVC, and that a write-restart-read remained owed. The swap improved on that without being the full test. Redis started on the volume Valkey had been writing:

* RDB is base AOF
* Done loading RDB, keys loaded: 0, keys expired: 0.
* DB loaded from base file appendonly.aof.1.base.rdb: 0.001 seconds
* DB loaded from append only file: 0.001 seconds
* Opening AOF incr file appendonly.aof.1.incr.aof on server start

So the PVC survived a pod replacement and an engine replacement, and the new process found and loaded the AOF the old one wrote. The mount, the --dir, the file permissions under uid 999, and the AOF load path are all exercised end to end.

What this still does not prove is key survival. keys loaded: 0, because the store has never held data. The load path works; a non-empty dataset round-trip is untested.

So the acceptance criterion stands unchanged and is still yours: write a key, restart this pod, read it back. What has changed is that everything around the data is now known good, so a failure there would be about the data rather than the plumbing.

An operator surface exists now

aosguard ops redis landed in coilyco-flight-deck/agentic-os at 6fe9d15e: ping, info, dbsize, get, ttl, exists, type, keys, scan, config get, del. redis-tools joined the dev-base apt list, since the image did not carry redis-cli.

Two properties worth knowing before you use it for the acceptance test:

Auth is environment-only. -a and -u are absent from every allow-flag list, so the guard refuses them before the process runs. Use REDISCLI_AUTH. I verified allow-flag is a strict whitelist rather than an additive permit by probing a guarded verb with an unlisted flag, which was rejected without reaching the binary.
set is denied and del is not. A bucket key holds a spend budget, so writing one fabricates budget. del is exposed for unsticking a single wedged bucket. That means the write half of the acceptance test cannot be done through this wrapper by design, and wants a direct redis-cli from an operator shell.

Still not reachable from outside the cluster

ClusterIP only, measured: 10.43.241.182:6379 refuses from a tailnet client, and no peer advertises subnet routes. Reachable from a shell on kai-server through kube-proxy, and nowhere else. A Tailscale sidecar giving it the MagicDNS name redis is queued on Kai minting an auth key, and until then aosguard ops redis has nothing to dial from an agent host.

**Correction and an upgrade to the evidence: the store is Redis, not Valkey, and the AOF load path is now demonstrated rather than inferred.** Olaf (DevOps, claude seat). ## Redis, not Valkey Kai reverted my Valkey suggestion and was right to. The only argument I offered for the fork was the post-2024 licence change, and it does not reach a homelab: Redis 8 is AGPLv3, OSI-approved, and nothing here redistributes it. A fork bought no capability and cost the more familiar name. Live now on `redis@sha256:978f0e01593e65eed801f2402944efcd936d43b5027e4908a7897baf88ed6241`, commit `7837039` in `coilyco-bridge/deploy`. Nothing else moved: `redis:8-alpine` carries the same uid 999 and gid 1000 as the valkey image, read from its `/etc/passwd` the same way, so the security context is unchanged. Same flags, same PVC. **The environment variable is `REDIS_PASSWORD`**, not `VALKEY_PASSWORD`, and the manifest is `services/mcp-ratelimit/deploy/redis.yml`. Anything written against my earlier comment needs that rename. ## The engine swap accidentally proved the load path My previous comment said durability was inferred from AOF files existing on a Bound PVC, and that a write-restart-read remained owed. The swap improved on that without being the full test. Redis started on the volume Valkey had been writing: ``` * RDB is base AOF * Done loading RDB, keys loaded: 0, keys expired: 0. * DB loaded from base file appendonly.aof.1.base.rdb: 0.001 seconds * DB loaded from append only file: 0.001 seconds * Opening AOF incr file appendonly.aof.1.incr.aof on server start ``` So the PVC survived a **pod replacement and an engine replacement**, and the new process found and loaded the AOF the old one wrote. The mount, the `--dir`, the file permissions under uid 999, and the AOF load path are all exercised end to end. **What this still does not prove is key survival.** `keys loaded: 0`, because the store has never held data. The load path works; a non-empty dataset round-trip is untested. So the acceptance criterion stands unchanged and is still yours: write a key, restart this pod, read it back. What has changed is that everything around the data is now known good, so a failure there would be about the data rather than the plumbing. ## An operator surface exists now `aosguard ops redis` landed in `coilyco-flight-deck/agentic-os` at `6fe9d15e`: ping, info, dbsize, get, ttl, exists, type, keys, scan, `config get`, del. `redis-tools` joined the dev-base apt list, since the image did not carry `redis-cli`. Two properties worth knowing before you use it for the acceptance test: - **Auth is environment-only.** `-a` and `-u` are absent from every `allow-flag` list, so the guard refuses them before the process runs. Use `REDISCLI_AUTH`. I verified `allow-flag` is a strict whitelist rather than an additive permit by probing a guarded verb with an unlisted flag, which was rejected without reaching the binary. - **`set` is denied and `del` is not.** A bucket key holds a spend budget, so writing one fabricates budget. `del` is exposed for unsticking a single wedged bucket. That means **the write half of the acceptance test cannot be done through this wrapper** by design, and wants a direct `redis-cli` from an operator shell. ## Still not reachable from outside the cluster ClusterIP only, measured: `10.43.241.182:6379` refuses from a tailnet client, and no peer advertises subnet routes. Reachable from a shell on kai-server through kube-proxy, and nowhere else. A Tailscale sidecar giving it the MagicDNS name `redis` is queued on Kai minting an auth key, and until then `aosguard ops redis` has nothing to dial from an agent host.

coilyco-ops referenced this issue

2026-08-15 23:06:52 +00:00

No upstream response caching, which now costs real money rather than just politeness #73

coilyco-ops referenced this issue from a commit

2026-08-16 01:34:35 +00:00

refactor(redis)!: name the store for what it is, not its first consumer

coilyco-ops added the

labels

2026-08-17 01:14:31 +00:00

coilyco-ops referenced this issue from a commit

2026-08-17 02:45:35 +00:00

feat(cache): opt-in per-grant response caching (#73)

coilyco-ops referenced this issue

2026-08-17 02:45:52 +00:00

No upstream response caching, which now costs real money rather than just politeness #73

coilyco-ops referenced this issue from a commit

2026-08-19 10:09:55 +00:00

feat(rate-limit): a durable shared bucket, so a rate can be a spend budget

coilyco-ops commented

2026-08-19 10:10:17 +00:00

Author

Member

Runtime half landed on main as 80f2075. Angie (engineer, claude seat), with Kai settling the two open calls in a live round.

First, a correction to this issue's own framing

The body's store-choice section reads as open. It is not: coilyco-bridge/deploy#549 settled it and provisioned the store three days ago. One ClusterIP Redis on a PVC, --appendonly yes --appendfsync everysec --maxmemory-policy noeviction, live at redis.redis.svc.cluster.local:6379, pod up 3d8h with 0 restarts. Its README says plainly that nothing reads it yet on purpose and that this issue is the runtime change that will.

I initially read this issue and concluded it was blocked on an ops decision. It was not. Reading the deploy repo rather than the issue describing it is what corrected that.

Redis over the running Postgres, per that decision: this binary carries no migration mechanism at all, so even a one-table counter means adding one plus a database credential in every MCP pod. And TTL means a decommissioned server's key expires on its own rather than waiting for a reaper.

The two calls, and why they are recorded

A store outage refuses. No knob. Every spec naming a store did so because its upstream is metered, so none of them wants the spend bound quietly removed by a blip. Falling back to memory would move this exact bug one layer down and make it harder to see, since the pod would look healthy. The issue sketched a per-spec knob; a knob whose wrong setting is invisible until an outage is worse than the right default.

Shared by default, keyed on the wrap server name, with bucket "<key>" to split deliberately. The un-doubling is worth as much as the durability.

Acceptance

A restart mid-window does not restore the allowance. Both restarts tested, and the one that matters is the store's.
Two releases sharing a bucket charge it once, demonstrably. Two buckets on one key: two calls spend a 2/24h budget and both are then refused.
A spec with no store behaves exactly as today. Asserted on the concrete type.
Store unavailability has a documented and tested behaviour. Unreachable store returns a tool error and the upstream is reached zero times.
The fast-fail is preserved. A spent budget refuses well inside the deadline rather than blocking to it.

The run deploy#549 requires

Its README is explicit that restarting an MCP pod is not enough, "or it passes while the bug survives". So:

phase 1: budget spent, third call refused
         key holds tokens=1.7504449236234095e-07
restart: docker restart, uptime_in_seconds=6
         key survived byte-identical
phase 2: budget STILL spent

Against a Redis configured with the cluster's exact flags. Pinning the key via MCP_BEAVER_TEST_BUCKET_KEY is what makes that two-phase run repeatable rather than a one-off.

What is left, and it is not mine

Nothing points at the cluster store yet. This ships the capability; a guardfile in coilyco-bridge/deploy has to declare store redis env "REDIS_URL" and the namespace needs its own ExternalSecret against /redis/password, since a Secret does not cross namespaces.

I could not verify against the cluster Redis. It is ClusterIP-only and, per its README, not tailnet-reachable: the Tailscale sidecar giving it a MagicDNS name is authored but the terraform apply is outstanding. So the acceptance above ran against an identically-configured local Redis, not the real one. One run against the cluster store is worth doing when it becomes dialable.

Leaving this open for that deploy step rather than closing on the runtime half alone.

**Runtime half landed on `main` as `80f2075`.** Angie (engineer, `claude` seat), with Kai settling the two open calls in a live round. ## First, a correction to this issue's own framing The body's store-choice section reads as open. It is not: **`coilyco-bridge/deploy#549` settled it and provisioned the store three days ago.** One ClusterIP Redis on a PVC, `--appendonly yes --appendfsync everysec --maxmemory-policy noeviction`, live at `redis.redis.svc.cluster.local:6379`, pod up 3d8h with 0 restarts. Its README says plainly that nothing reads it yet on purpose and that this issue is the runtime change that will. I initially read this issue and concluded it was blocked on an ops decision. It was not. Reading the deploy repo rather than the issue describing it is what corrected that. Redis over the running Postgres, per that decision: this binary carries **no migration mechanism at all**, so even a one-table counter means adding one plus a database credential in every MCP pod. And TTL means a decommissioned server's key expires on its own rather than waiting for a reaper. ## The two calls, and why they are recorded **A store outage refuses. No knob.** Every spec naming a store did so because its upstream is metered, so none of them wants the spend bound quietly removed by a blip. Falling back to memory would move this exact bug one layer down and make it *harder* to see, since the pod would look healthy. The issue sketched a per-spec knob; a knob whose wrong setting is invisible until an outage is worse than the right default. **Shared by default**, keyed on the wrap server name, with `bucket "<key>"` to split deliberately. The un-doubling is worth as much as the durability. ## Acceptance * **A restart mid-window does not restore the allowance.** Both restarts tested, and the one that matters is the store's. * **Two releases sharing a bucket charge it once, demonstrably.** Two buckets on one key: two calls spend a `2/24h` budget and both are then refused. * **A spec with no store behaves exactly as today.** Asserted on the concrete type. * **Store unavailability has a documented and tested behaviour.** Unreachable store returns a tool error and the upstream is reached zero times. * **The fast-fail is preserved.** A spent budget refuses well inside the deadline rather than blocking to it. ## The run deploy#549 requires Its README is explicit that restarting an MCP pod is not enough, "or it passes while the bug survives". So: ``` phase 1: budget spent, third call refused key holds tokens=1.7504449236234095e-07 restart: docker restart, uptime_in_seconds=6 key survived byte-identical phase 2: budget STILL spent ``` Against a Redis configured with the cluster's exact flags. Pinning the key via `MCP_BEAVER_TEST_BUCKET_KEY` is what makes that two-phase run repeatable rather than a one-off. ## What is left, and it is not mine **Nothing points at the cluster store yet.** This ships the capability; a guardfile in `coilyco-bridge/deploy` has to declare `store redis env "REDIS_URL"` and the namespace needs its own ExternalSecret against `/redis/password`, since a Secret does not cross namespaces. **I could not verify against the cluster Redis.** It is ClusterIP-only and, per its README, not tailnet-reachable: the Tailscale sidecar giving it a MagicDNS name is authored but the `terraform apply` is outstanding. So the acceptance above ran against an identically-configured local Redis, not the real one. One run against the cluster store is worth doing when it becomes dialable. Leaving this open for that deploy step rather than closing on the runtime half alone.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/mcp-beaver#69

No description provided.

Rows
Columns