coilyco-flight-deck/agent-proxy

Watch

Shed requests above a configured admission rate #121

Merged

coilysiren merged 3 commits from configurable-rate-limits into main

2026-08-13 22:51:02 +00:00

coilyco-ops commented

2026-08-13 18:12:39 +00:00

Member

Summary

Closes #110: configurable rate limit, default 1 per second, anything above it shed with 429.

Design

A token bucket per logical route, so a busy route cannot starve a quiet one.

PROXY_RATE_LIMIT_PER_SECOND - sustained rate. Default 1.0. 0 disables shedding.
PROXY_RATE_LIMIT_BURST - bucket capacity. Default 1.

The check sits after route resolution and before admission:

after resolution, so a caller probing unknown model names answers 404 without spending a token it could otherwise exhaust for a real route
before admission, so a shed request never occupies the queue it was shed to protect
/healthz, /readyz, /metrics are never shed

Response carries Retry-After in whole seconds, minimum 1. The recorded error code is rate_limited, kept apart from rate_limit_error so the queue-full path and the rate path stay separable. Counter: llm_rate_limited_total{logical_model}.

The one decision worth your eye

burst=1 means two requests in the same second sheds the second.

That is the literal reading of "1 per second" and the right sustained number against measured traffic of 0.0047 req/s. It is also the setting most likely to be wrong for a tool-using turn, which issues its model rounds back to back rather than spread over seconds - the turn in #113 ran six rounds inside 97 seconds, several following one another immediately. Under this default, that turn sheds.

I shipped the default you asked for rather than quietly picking a safer one. If sirens-echo should keep working unchanged, Deploy wants PROXY_RATE_LIMIT_BURST at or above the number of rounds a turn can issue, with the per-second rate left alone: the rate bounds sustained load, the burst decides whether one legitimate turn survives. docs/rate-limits.md says this where an operator will find it.

How to verify

ward exec test - tests/test_ratelimit.py, 12 cases.
test_first_request_passes_and_the_next_sheds is the shipped default end to end.
test_an_unknown_model_still_404s_rather_than_spending_a_token pins the ordering.
test_health_routes_are_never_shed pins the exemption.
Bucket refill, capacity, and Retry-After are tested on a controlled clock, so no test sleeps.

Test plan

New suite for the behaviour
Existing tests still pass (319 passed)
ward exec format-check, lint, typecheck, pre-commit all clean

Risk

High until the burst is set for the deployment, entirely because of the default above. The mechanism itself is small and self-contained: one new module, one call site per serving route, no change to the dispatch path.

tests/conftest.py pins the rate off for the rest of the suite, because every other test fires far above 1/s and would otherwise be testing this instead.

Note

Branched off main, independent of the other open PRs.

## Summary Closes #110: configurable rate limit, default 1 per second, anything above it shed with 429. ## Design A token bucket **per logical route**, so a busy route cannot starve a quiet one. - `PROXY_RATE_LIMIT_PER_SECOND` - sustained rate. Default `1.0`. `0` disables shedding. - `PROXY_RATE_LIMIT_BURST` - bucket capacity. Default `1`. The check sits **after route resolution and before admission**: - after resolution, so a caller probing unknown model names answers 404 without spending a token it could otherwise exhaust for a real route - before admission, so a shed request never occupies the queue it was shed to protect - `/healthz`, `/readyz`, `/metrics` are never shed Response carries `Retry-After` in whole seconds, minimum 1. The recorded error code is `rate_limited`, kept apart from `rate_limit_error` so the queue-full path and the rate path stay separable. Counter: `llm_rate_limited_total{logical_model}`. ## The one decision worth your eye **`burst=1` means two requests in the same second sheds the second.** That is the literal reading of "1 per second" and the right sustained number against measured traffic of 0.0047 req/s. It is also the setting most likely to be wrong for a **tool-using turn**, which issues its model rounds back to back rather than spread over seconds - the turn in #113 ran six rounds inside 97 seconds, several following one another immediately. Under this default, that turn sheds. I shipped the default you asked for rather than quietly picking a safer one. If `sirens-echo` should keep working unchanged, Deploy wants `PROXY_RATE_LIMIT_BURST` at or above the number of rounds a turn can issue, with the per-second rate left alone: the rate bounds sustained load, the burst decides whether one legitimate turn survives. `docs/rate-limits.md` says this where an operator will find it. ## How to verify 1. `ward exec test` - `tests/test_ratelimit.py`, 12 cases. 2. `test_first_request_passes_and_the_next_sheds` is the shipped default end to end. 3. `test_an_unknown_model_still_404s_rather_than_spending_a_token` pins the ordering. 4. `test_health_routes_are_never_shed` pins the exemption. 5. Bucket refill, capacity, and `Retry-After` are tested on a controlled clock, so no test sleeps. ## Test plan - [x] New suite for the behaviour - [x] Existing tests still pass (319 passed) - [x] `ward exec format-check`, `lint`, `typecheck`, `pre-commit` all clean ## Risk **High until the burst is set for the deployment**, entirely because of the default above. The mechanism itself is small and self-contained: one new module, one call site per serving route, no change to the dispatch path. `tests/conftest.py` pins the rate off for the rest of the suite, because every other test fires far above 1/s and would otherwise be testing this instead. ## Note Branched off `main`, independent of the other open PRs.

coilyco-ops added 1 commit

2026-08-13 18:12:40 +00:00

Shed requests above a configured admission rate

ci / quality (pull_request) Successful in 24s

Details

ci / smoke (pull_request) Successful in 8s

Details

33695b5b13

Issue #110 asks for a configurable rate, default 1 per second, with anything
above it shed as 429.

The bounded queue already gives backpressure once work is accepted, but
backpressure is not a rate. A caller can fill and drain the queue as fast as the
backend serves, and nothing bounded the arrival rate itself. Shedding rather
than buffering is the point: a caller learns immediately instead of waiting
behind work it cannot see.

A token bucket per logical route, so a busy route cannot starve a quiet one.
PROXY_RATE_LIMIT_PER_SECOND is the sustained rate and defaults to 1.0, with 0
turning shedding off. PROXY_RATE_LIMIT_BURST is the bucket capacity and defaults
to 1. Changing either rebuilds the affected bucket on the next request.

The check sits after route resolution and before admission. After resolution, so
a caller probing unknown model names answers 404 without spending a token it
could otherwise exhaust for a real route. Before admission, so a shed request
never occupies the queue it was shed to protect. Health, readiness, and metrics
are never shed.

The response carries Retry-After in whole seconds, minimum 1, and the recorded
code is rate_limited, kept apart from rate_limit_error so the queue-full path
and the rate path stay separable in the error surface. The counter is
llm_rate_limited_total by logical_model.

Worth naming: burst=1 means two requests in the same second sheds the second.
That is the literal reading of the issue and the right sustained number against
measured traffic of 0.0047 req/s, but it is the setting most likely to be wrong
for a tool-using turn, which issues its rounds back to back. The turn in #113
ran six rounds inside 97 seconds with several following one another
immediately. A deployment serving such a caller should raise the burst to the
number of rounds a turn can issue and leave the rate alone. That is a Deploy
decision, and docs/rate-limits.md says so where an operator will find it.

tests/conftest.py pins the rate off for the suite, because every other test
fires far above 1/s and would otherwise be testing this. tests/test_ratelimit.py
turns it back on and is where the behaviour is exercised, including the bucket
itself on a controlled clock.

closes #110

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 18:12:40 +00:00

coilyco-ops added 1 commit

2026-08-13 18:39:44 +00:00

Merge remote-tracking branch 'origin/main' into configurable-rate-limits

ci / smoke (pull_request) Successful in 7s

Details

ci / quality (pull_request) Successful in 22s

Details

610503cf19

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

# Conflicts:
#	docs/exception-taxonomy.md
#	docs/proxy-request-path.md
#	tests/test_exception_taxonomy.py

coilyco-ops added 1 commit

2026-08-13 19:56:16 +00:00

Merge main into the rate limiter and reconcile the shared surfaces

ci / quality (pull_request) Successful in 31s

Details

ci / smoke (pull_request) Successful in 8s

Details

8dcb5582f4

Everything else in this run landed while this branch sat, so the merge is where
the rate limiter meets the deadline, heartbeat, saturation, and preference work
rather than a rebase artifact.

Three real resolutions. In config, the limiter's two settings sit beside the
saturation and heartbeat ones rather than replacing them. In the chat path,
shedding runs before the caller's backend preference is honoured: a shed request
should cost nothing, including the work of reordering a chain it will never
reach. In the request-path doc, trimming keeps its tool-pairing rule from #113
and shedding takes step 3 ahead of admission, with the later steps renumbered.

That doc crossed its 4000-character cap once both halves were in it, which is
the cap doing its job: the detail it had accumulated belongs in the pages that
own each behaviour, and the step list is meant to be the map rather than the
territory. Trimming, shedding, and the queue span keep one sentence each and
their links.

The exception taxonomy grew on both sides of the merge - rate_limited here,
backend_saturated on main - so the cardinality assertion and the doc's stated
count move to 18. Both directions are now reconciled against ERROR_TAXONOMY
rather than counted by hand: no code missing from the doc, no doc row missing
from the code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>