[proxy 08/13] queue.py: bounded asyncio.Queue + worker pool + 429 backpressure #12

New issue

Closed

opened 2026-06-26 09:48:19 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-06-26 09:48:19 +00:00

Owner

Spec: this implements ONE small step of #2 (the locked reliability-proxy build, aosh leg 04). Do not redesign. Stay tightly scoped to this single step.

Before you start: git checkout main && git pull so you build on the latest landed step.
Fill in app/queue.py and route chat requests through it.

A bounded asyncio.Queue (maxsize from settings, default 100) and a worker-pool started on app startup (N workers, default 4).
A job = {messages, logical_name, future}. The route does queue.put_nowait(job) then await job['future']; a worker pops, calls app.upstream.dispatch_chat, sets the future result/exception.
On asyncio.QueueFull the route returns HTTP 429.
Keep app.obs.llm_queue_depth updated (set to queue.qsize() on put/get).

Acceptance: a pytest enqueues a job (worker dispatching a monkeypatched upstream) and gets a result; a second test fills the queue and asserts 429.

When done:

pre-commit run --all-files (or the repo's gate) must pass, commit your work, and push to canonical main (this is an ephemeral feature container - finish the merge, do not leave it for a human).
Update the matching line in docs/FEATURES.md from planned to landed for this feature, in the same push.
Dispatch the next step: ward agent engineer coilyco-flight-deck/agent-proxy#13 --driver goose

**Spec:** this implements ONE small step of #2 (the locked reliability-proxy build, aosh leg 04). **Do not redesign. Stay tightly scoped to this single step.** **Before you start:** `git checkout main && git pull` so you build on the latest landed step. Fill in `app/queue.py` and route chat requests through it. - A bounded `asyncio.Queue` (maxsize from settings, default 100) and a worker-pool started on app startup (N workers, default 4). - A job = `{messages, logical_name, future}`. The route does `queue.put_nowait(job)` then `await job['future']`; a worker pops, calls `app.upstream.dispatch_chat`, sets the future result/exception. - On `asyncio.QueueFull` the route returns HTTP **429**. - Keep `app.obs.llm_queue_depth` updated (set to `queue.qsize()` on put/get). **Acceptance:** a pytest enqueues a job (worker dispatching a monkeypatched upstream) and gets a result; a second test fills the queue and asserts 429. **When done:** 1. `pre-commit run --all-files` (or the repo's gate) must pass, commit your work, and push to canonical `main` (this is an ephemeral feature container - finish the merge, do not leave it for a human). 2. Update the matching line in `docs/FEATURES.md` from `planned` to `landed` for this feature, in the same push. 3. Dispatch the next step: `ward agent engineer coilyco-flight-deck/agent-proxy#13 --driver goose`

coilyco-ops added the

headless

labels

2026-07-02 16:42:41 +00:00

coilyco-ops commented

2026-07-02 16:44:47 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-agent-proxy-12 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T16:44:44Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agent-proxy-12` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T16:44:44Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-02 16:47:22 +00:00

fix(queue): align default queue_maxsize to spec (100)

coilysiren closed this issue

2026-07-02 16:47:22 +00:00

coilysiren commented

2026-07-02 16:48:16 +00:00

Author

Owner

WARD-OUTCOME: done - queue_maxsize default aligned to spec (100); bounded queue + worker pool + 429 backpressure verified landed on main.

Candid retrospective: this one was mostly already done when I opened the box. The bounded asyncio.Queue, the worker pool, the put-then-await-future route wiring, the QueueFull to 429 path, the llm_queue_depth accounting, and both acceptance tests had all landed earlier as part of the consolidated resilience-core commit, and FEATURES.md already read landed. So instead of building, the work was reading carefully enough to be sure nothing was missing.

The one real drift from the leg 04 step 3 spec was the default queue depth: 64 in code where the step says 100. I aligned it and left workers at the spec default of 4. I deliberately did not touch the worker calling resilience.dispatch (the spec text says upstream.dispatch_chat) because the repo evolved resilience to wrap upstream with retry/fallback/breaker, and reverting that would be a regression, not a fix. Stayed tightly scoped as asked.

What fought back: the shared container disk was at 100 percent, so pre-commit could not even create its cache dir. There is no pre-commit config in this repo anyway, so the real gate is ward test (pytest) plus ruff, both green (41 pass, lint clean). I freed cache space to be safe. Also had to rebase onto two concurrent pushes before mine landed.

Confidence is high: change is a one-line default, tests cover both the enqueue-result and fill-429 cases, tree is green on main. No follow-ups worth filing. Next step #13 (resilience validation) was already reserved remotely, so it is in flight.

WARD-OUTCOME: done - queue_maxsize default aligned to spec (100); bounded queue + worker pool + 429 backpressure verified landed on main. Candid retrospective: this one was mostly already done when I opened the box. The bounded asyncio.Queue, the worker pool, the put-then-await-future route wiring, the QueueFull to 429 path, the llm_queue_depth accounting, and both acceptance tests had all landed earlier as part of the consolidated resilience-core commit, and FEATURES.md already read landed. So instead of building, the work was reading carefully enough to be sure nothing was missing. The one real drift from the leg 04 step 3 spec was the default queue depth: 64 in code where the step says 100. I aligned it and left workers at the spec default of 4. I deliberately did not touch the worker calling resilience.dispatch (the spec text says upstream.dispatch_chat) because the repo evolved resilience to wrap upstream with retry/fallback/breaker, and reverting that would be a regression, not a fix. Stayed tightly scoped as asked. What fought back: the shared container disk was at 100 percent, so pre-commit could not even create its cache dir. There is no pre-commit config in this repo anyway, so the real gate is ward test (pytest) plus ruff, both green (41 pass, lint clean). I freed cache space to be safe. Also had to rebase onto two concurrent pushes before mine landed. Confidence is high: change is a one-line default, tests cover both the enqueue-result and fill-429 cases, tree is green on main. No follow-ups worth filing. Next step #13 (resilience validation) was already reserved remotely, so it is in flight.