coilyco-flight-deck/agent-proxy

Watch

Concurrency burst test: does agent-proxy queue or shed, and should heartbeats carry queue position #107

New issue

Open

opened 2026-08-12 21:19:13 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-08-12 21:19:13 +00:00

Member

Why

#105 established that queue.wait shows sub-5ms admission delay (0.54ms and 4.03ms across two traces) — but at the observed load an empty queue is the expected result whether queueing is live or inert:

arrival rate      = 407 req / 86,400s = 0.0047 req/s
mean service time = 23.7s
average concurrency (Little's law)   ≈ 0.11

Three explanations remain open and span data cannot separate them:

Queueing active, never contended
Queueing configured but not wired to the request path
No queue at all, span emitted unconditionally

One burst decides it, and decides two downstream questions with it.

Method

Reuse #164's methodology, which ran a 30-way concurrent burst against the Echo-side limiter and observed 8 admitted / 22 rejected with 429.

Target: POST http://ser8:8080/v1/chat/completions (confirmed reachable — GET /v1/models returns 200 in 70ms)
Concurrency: sweep 1, 5, 10, 30
Prompt: shortest possible, identical across all requests. This measures admission, not generation, and the prompt should not add variance or cost.
Repeat each level 3 times

Measure

Per trace, from spans:

Admission delay = resilience.attempt start − queue.wait start. Baseline is 0.54–4.03ms.
Count admitted vs rejected, and the rejection status (429 / 503)
Whether Retry-After or any queue-depth information is present on rejections

Per-trace subtraction, not percentile subtraction — #105 documents why comparing p50s across queue.wait and resilience.attempt is invalid (the latter emits one span per attempt, so retries skew its population).

Decision table

Observation	Conclusion	Action
Admission delay rises above the ms floor with concurrency	Queueing is live	#104 carries queue position; rename `queue.wait` to match what it measures
Requests are rejected (429/503) with no delay rise	Proxy sheds, never queues	#104 drops queue position; delete `queue.wait` rather than rename — it can never show a wait by construction, and backpressure headers become the design
Neither — requests just run concurrently	No admission control at this layer	Both above are moot; document that admission control lives only in the caller

Cost and timing

A burst issues real model calls and costs real tokens. Keep prompts minimal, and do not run this during an incident — as of 2026-08-12T21:11Z sirens-echo is failing every turn at ~180s while sirens-deep is healthy, so the current backend state is not representative.

Acceptance

Admission delay at each concurrency level is reported, measured per trace.
The queue-versus-shed question is answered with evidence.
#104's queue-position decision and queue.wait's rename-or-delete decision both follow from the result rather than from inference.

#105 — where the ambiguity was found and narrowed
#104 — SSE heartbeats, whose payload depends on this
coilyco-gaming/sirens-echo#164 — the burst methodology, and evidence that the Echo-side limiter sheds rather than queues

## Why #105 established that `queue.wait` shows sub-5ms admission delay (0.54ms and 4.03ms across two traces) — but at the observed load an empty queue is the expected result whether queueing is live or inert: ``` arrival rate = 407 req / 86,400s = 0.0047 req/s mean service time = 23.7s average concurrency (Little's law) ≈ 0.11 ``` Three explanations remain open and span data cannot separate them: 1. Queueing active, never contended 2. Queueing configured but not wired to the request path 3. No queue at all, span emitted unconditionally One burst decides it, and decides two downstream questions with it. ## Method Reuse #164's methodology, which ran a 30-way concurrent burst against the Echo-side limiter and observed 8 admitted / 22 rejected with 429. - Target: `POST http://ser8:8080/v1/chat/completions` (confirmed reachable — `GET /v1/models` returns 200 in 70ms) - Concurrency: sweep 1, 5, 10, 30 - Prompt: shortest possible, identical across all requests. This measures admission, not generation, and the prompt should not add variance or cost. - Repeat each level 3 times ## Measure Per trace, from spans: - **Admission delay** = `resilience.attempt` start − `queue.wait` start. Baseline is 0.54–4.03ms. - Count admitted vs rejected, and the rejection status (429 / 503) - Whether `Retry-After` or any queue-depth information is present on rejections Per-trace subtraction, not percentile subtraction — #105 documents why comparing p50s across `queue.wait` and `resilience.attempt` is invalid (the latter emits one span per attempt, so retries skew its population). ## Decision table | Observation | Conclusion | Action | | --- | --- | --- | | Admission delay rises above the ms floor with concurrency | Queueing is live | #104 carries queue position; rename `queue.wait` to match what it measures | | Requests are rejected (429/503) with no delay rise | Proxy sheds, never queues | #104 drops queue position; **delete** `queue.wait` rather than rename — it can never show a wait by construction, and backpressure headers become the design | | Neither — requests just run concurrently | No admission control at this layer | Both above are moot; document that admission control lives only in the caller | ## Cost and timing A burst issues real model calls and costs real tokens. Keep prompts minimal, and **do not run this during an incident** — as of 2026-08-12T21:11Z `sirens-echo` is failing every turn at ~180s while `sirens-deep` is healthy, so the current backend state is not representative. ## Acceptance - Admission delay at each concurrency level is reported, measured per trace. - The queue-versus-shed question is answered with evidence. - #104's queue-position decision and `queue.wait`'s rename-or-delete decision both follow from the result rather than from inference. ## Related - #105 — where the ambiguity was found and narrowed - #104 — SSE heartbeats, whose payload depends on this - `coilyco-gaming/sirens-echo#164` — the burst methodology, and evidence that the Echo-side limiter sheds rather than queues

coilyco-ops referenced this issue

2026-08-12 22:17:52 +00:00

A saturated backend is not a failed backend, so failover never fires and the caller is told nothing #108

coilyco-ops referenced this issue

2026-08-12 22:35:09 +00:00

Tag spans with backend operating regime — percentiles are unreadable across a shared GPU #109

coilyco-ops commented

2026-08-12 22:35:25 +00:00

Author

Member

Precondition: this test is invalid on a contended GPU

Per #108, the local inference host's capacity depends on what else is running on it — a game on 2026-08-12 starved Echo for ~2.5 hours while Deep, on a different tier, was unaffected.

That breaks this test's core measurement. The burst is meant to isolate admission delay under concurrency. On a saturated backend, every request is slow for a reason that has nothing to do with admission, and the decision table in this issue would then read:

"Admission delay rises above the ms floor with concurrency → queueing is live"

— and conclude that queueing exists when what actually happened was that the GPU was busy. A false positive that confirms the hypothesis, which is the worst kind.

Required before running

Verify the backend is idle at the start of the run, and confirm it stayed idle afterwards. #109 proposes a regime tag on spans that would make this checkable after the fact; until that lands it is a manual check.
Run against a backend not subject to human co-tenancy, or accept that the run is void if the host was busy.
Record which backend served the run, so the result is interpretable later. Right now backend identity is only inferable from a client URL.

Also worth noting for whoever runs it: Deep's admission policy rejected roughly one in three of my probe requests today with > \busy, retry shortly``. A burst will trip that readily. Rejections are a result, not an error — they are precisely what the "does it shed rather than queue" branch of the decision table is looking for — but the harness must record them as such rather than retrying past them.

## Precondition: this test is invalid on a contended GPU Per #108, the local inference host's capacity depends on what else is running on it — a game on 2026-08-12 starved Echo for ~2.5 hours while Deep, on a different tier, was unaffected. That breaks this test's core measurement. The burst is meant to isolate **admission delay under concurrency**. On a saturated backend, every request is slow for a reason that has nothing to do with admission, and the decision table in this issue would then read: > *"Admission delay rises above the ms floor with concurrency → queueing is live"* — and conclude that queueing exists when what actually happened was that the GPU was busy. A false positive that confirms the hypothesis, which is the worst kind. ### Required before running 1. **Verify the backend is idle** at the start of the run, and confirm it stayed idle afterwards. #109 proposes a regime tag on spans that would make this checkable after the fact; until that lands it is a manual check. 2. **Run against a backend not subject to human co-tenancy**, or accept that the run is void if the host was busy. 3. **Record which backend served the run**, so the result is interpretable later. Right now backend identity is only inferable from a client URL. Also worth noting for whoever runs it: Deep's admission policy rejected roughly one in three of my probe requests today with `> \`busy, retry shortly\``. A burst will trip that readily. **Rejections are a result, not an error** — they are precisely what the "does it shed rather than queue" branch of the decision table is looking for — but the harness must record them as such rather than retrying past them.

coilyco-ops commented

2026-08-12 23:38:05 +00:00

Author

Member

Correction, and measured shed behaviour from traffic that was not mine

My previous comment said "Deep's admission policy rejected roughly one in three of my probe requests" and implied that light probing trips it. Kai pushed on the volume claim; it does not survive checking.

My actual volume was negligible

~9 requests, strictly sequential, spread over ~25 minutes, zero concurrency. That is not pressure, and it should not have shed anything.

The rejections were probably not the admission policy

Spans for sirens-deep POST /v1/turn across my probe window (21:21–21:36 UTC):

Status	Count
200	45
502	4 — durations 30.00s, 30.00s, 30.00s, 66.04s
429	1

Three of the four 502s land at exactly 30.00s. My three busy, retry shortly replies align with those, not with the single admission rejection. So busy, retry shortly is most likely a backend failure at a 30s deadline, not shedding — and my attribution of it to the admission policy was wrong.

Caveat: these queries hit the 50-row limit, so the counts are a floor rather than a total. The 4:1 ratio is what is visible, not necessarily what occurred.

What the window does show, from someone else's traffic

The endpoint was carrying sustained load I did not generate — a request every 4–11 seconds from 21:22 onward, and a ~30-way concurrent burst at 21:39. That burst is effectively the experiment this issue proposes, run against sirens-deep's /v1/turn rather than agent-proxy:

Paced traffic is never shed. 21:37:03 → 21:39:14, 23 consecutive requests at ~4–14s spacing, 100% admitted.

Concurrent traffic is shed hard and instantly. At 21:39:19: 8 admitted, 16 rejected. At 21:39:59: 10 admitted, 14 rejected. Every rejection is 429 at 0.00s duration — no wait, no queue, immediate.

But admitted requests serialize. Durations across the simultaneously-submitted batch climb monotonically:

1.52s → 3.07s → 5.36s → 7.38s → 9.44s → 11.44s → 12.72s → 14.52s

That is the signature of a concurrency limit behind the admission gate. So the shape is hard shed at the door, limited concurrency inside — which is neither of the clean branches in this issue's decision table.

Implication for this issue

Two things carry over to the agent-proxy burst:

The decision table should allow for the hybrid: shedding at admission and serialization among admitted requests. Measuring only admission delay would miss the second half, and the climbing-duration signature is the thing to look for.
The endpoint under test must be quiet. Deep's /v1/turn was taking background traffic every few seconds throughout. If agent-proxy is similarly loaded during the burst, the results describe the combined load, not the burst. Worth confirming who or what generates that cadence before running.

## Correction, and measured shed behaviour from traffic that was not mine My previous comment said *"Deep's admission policy rejected roughly one in three of my probe requests"* and implied that light probing trips it. Kai pushed on the volume claim; it does not survive checking. ### My actual volume was negligible ~9 requests, strictly sequential, spread over ~25 minutes, **zero concurrency**. That is not pressure, and it should not have shed anything. ### The rejections were probably not the admission policy Spans for `sirens-deep` `POST /v1/turn` across my probe window (21:21–21:36 UTC): | Status | Count | | --- | --- | | 200 | 45 | | **502** | **4** — durations 30.00s, 30.00s, 30.00s, 66.04s | | 429 | **1** | Three of the four 502s land at **exactly 30.00s**. My three `busy, retry shortly` replies align with those, not with the single admission rejection. So `busy, retry shortly` is most likely a **backend failure at a 30s deadline**, not shedding — and my attribution of it to the admission policy was wrong. Caveat: these queries hit the 50-row limit, so the counts are a floor rather than a total. The 4:1 ratio is what is visible, not necessarily what occurred. ### What the window does show, from someone else's traffic The endpoint was carrying sustained load I did not generate — a request every **4–11 seconds** from 21:22 onward, and a **~30-way concurrent burst at 21:39**. That burst is effectively the experiment this issue proposes, run against `sirens-deep`'s `/v1/turn` rather than agent-proxy: **Paced traffic is never shed.** 21:37:03 → 21:39:14, **23 consecutive requests at ~4–14s spacing, 100% admitted**. **Concurrent traffic is shed hard and instantly.** At 21:39:19: 8 admitted, 16 rejected. At 21:39:59: 10 admitted, 14 rejected. Every rejection is **429 at 0.00s duration** — no wait, no queue, immediate. **But admitted requests serialize.** Durations across the simultaneously-submitted batch climb monotonically: ``` 1.52s → 3.07s → 5.36s → 7.38s → 9.44s → 11.44s → 12.72s → 14.52s ``` That is the signature of a concurrency limit *behind* the admission gate. So the shape is **hard shed at the door, limited concurrency inside** — which is neither of the clean branches in this issue's decision table. ### Implication for this issue Two things carry over to the agent-proxy burst: 1. The decision table should allow for the hybrid: shedding at admission **and** serialization among admitted requests. Measuring only admission delay would miss the second half, and the climbing-duration signature is the thing to look for. 2. **The endpoint under test must be quiet.** Deep's `/v1/turn` was taking background traffic every few seconds throughout. If agent-proxy is similarly loaded during the burst, the results describe the combined load, not the burst. Worth confirming who or what generates that cadence before running.

coilyco-ops referenced this issue

2026-08-13 18:08:41 +00:00

Close queue.wait at dequeue so it measures a wait #120

coilyco-ops referenced this issue

2026-08-13 18:15:38 +00:00

Tag spans with backend identity and operating regime #122

coilyco-ops referenced this issue from a commit

2026-08-13 18:15:39 +00:00

Tag spans with backend identity and operating regime

coilyco-ops referenced this issue

2026-08-13 18:19:54 +00:00

Emit SSE heartbeats carrying attempt and backend state #123

coilyco-ops referenced this issue from a commit

2026-08-13 18:19:55 +00:00

Emit SSE heartbeats carrying attempt and backend state

coilyco-ops commented

2026-08-13 18:35:24 +00:00

Author

Member

Not run: this one needs your go-ahead and an idle GPU, not a branch

Working through the unblocked issues in this repo, this is the one I deliberately did not do. Three preconditions in the issue body are outside what I can decide:

It spends real tokens against ser8:8080. A sweep of 1/5/10/30 repeated three times is 138 real model calls, and that is a spend decision.
"Do not run this during an incident." I cannot see the current backend state from here.
It needs a known-idle GPU. #109 says a contended GPU would confound the run completely.

So it is waiting on you rather than on code.

Two things landed that change the run

#109 (backend-regime-spans) makes the run verifiable after the fact. That was the third acceptance criterion here and it now has a mechanism: set PROXY_BACKEND_REGIME=idle before the burst, and every span in the run carries agentproxy.backend.regime. If it turns out the GPU was not idle, the result can be discarded on evidence instead of argued about. Without that, "was the box quiet" is a memory.

#105 (queue-wait-measures-the-wait) makes the key measurement direct. queue.wait now closes when a worker claims the job, so admission delay is the span's own duration. The method section here specifies resilience.attempt start − queue.wait start precisely because the span used to stay open for the whole request; that subtraction still works, but the span duration is now the simpler and more honest number.

One thing landed that will confound the run if you forget it

#110 (configurable-rate-limits) ships shedding on by default at 1/s with burst 1. A 30-way burst against that limiter answers 429 to 29 of them in milliseconds, and the decision table here would read that as "proxy sheds, never queues" when what it actually measured is the limiter I just added.

Set PROXY_RATE_LIMIT_PER_SECOND=0 for the duration of the burst, or the run measures the wrong layer. This is the single most likely way to get a confidently wrong answer out of this test now.

Suggested procedure when you want it

Confirm the local GPU is idle. Nothing is running on the tower.
PROXY_BACKEND_REGIME=idle and PROXY_RATE_LIMIT_PER_SECOND=0 on the proxy.
Run the sweep from the method section, shortest possible identical prompt.
Read admission delay per trace as the queue.wait duration, and confirm every span in the window carries regime=idle before believing any of it.

Happy to write the burst script and drive the analysis on your word. I am not going to spend tokens against live infrastructure without it.

🤖 Filed by Claude Code on Kai's behalf.

## Not run: this one needs your go-ahead and an idle GPU, not a branch Working through the unblocked issues in this repo, this is the one I deliberately did not do. Three preconditions in the issue body are outside what I can decide: - **It spends real tokens** against `ser8:8080`. A sweep of 1/5/10/30 repeated three times is 138 real model calls, and that is a spend decision. - **"Do not run this during an incident."** I cannot see the current backend state from here. - **It needs a known-idle GPU.** #109 says a contended GPU would confound the run completely. So it is waiting on you rather than on code. ## Two things landed that change the run **#109 (`backend-regime-spans`) makes the run verifiable after the fact.** That was the third acceptance criterion here and it now has a mechanism: set `PROXY_BACKEND_REGIME=idle` before the burst, and every span in the run carries `agentproxy.backend.regime`. If it turns out the GPU was not idle, the result can be discarded on evidence instead of argued about. Without that, "was the box quiet" is a memory. **#105 (`queue-wait-measures-the-wait`) makes the key measurement direct.** `queue.wait` now closes when a worker claims the job, so admission delay is the span's own duration. The method section here specifies `resilience.attempt start − queue.wait start` precisely because the span used to stay open for the whole request; that subtraction still works, but the span duration is now the simpler and more honest number. ## One thing landed that will confound the run if you forget it **#110 (`configurable-rate-limits`) ships shedding on by default at 1/s with burst 1.** A 30-way burst against that limiter answers 429 to 29 of them in milliseconds, and the decision table here would read that as "proxy sheds, never queues" when what it actually measured is the limiter I just added. Set `PROXY_RATE_LIMIT_PER_SECOND=0` for the duration of the burst, or the run measures the wrong layer. This is the single most likely way to get a confidently wrong answer out of this test now. ## Suggested procedure when you want it 1. Confirm the local GPU is idle. Nothing is running on the tower. 2. `PROXY_BACKEND_REGIME=idle` and `PROXY_RATE_LIMIT_PER_SECOND=0` on the proxy. 3. Run the sweep from the method section, shortest possible identical prompt. 4. Read admission delay per trace as the `queue.wait` duration, and confirm every span in the window carries `regime=idle` before believing any of it. Happy to write the burst script and drive the analysis on your word. I am not going to spend tokens against live infrastructure without it. > 🤖 Filed by Claude Code on Kai's behalf.

coilyco-ops referenced this issue from a commit

2026-08-13 18:47:18 +00:00

Give saturation its own threshold and cooldown, and ship the burst probe

coilyco-ops referenced this issue

2026-08-13 18:47:41 +00:00

Give saturation its own threshold and cooldown, and ship the burst probe #126

coilyco-ops added the

autonomy

async-consult

label

2026-08-13 18:56:20 +00:00