coilyco-flight-deck/agent-proxy

Watch

Derive the context budget from the model, not from VRAM #117

Merged

coilysiren merged 2 commits from context-budget-per-model into main

2026-08-13 19:41:09 +00:00

coilyco-ops commented

2026-08-13 17:46:43 +00:00

Member

Bug

Fixes #115.

Every request.prompt_trimmed event in a seven-day window - all eleven, across every logical model - carried the same triple:

target_num_ctx  = 48128
budget_tokens   = 47104
headroom_tokens = 1024

48128 = 47 x 1024, 47104 = 46 x 1024. That is the shape of a VRAM allocation, not a provider limit. The DeepSeek v4 line advertises a 1M window, so the proxy was enforcing 4.7% of it. Nothing upstream ever objected on length: prompts of 171,203, 119,703, and 83,785 tokens all went out and came back complaining about message structure instead.

Root cause

derive_num_ctx applied min(context_length, PROXY_NUM_CTX_CEILING) - headroom to every route. num_ctx is an Ollama parameter - the window allocated when a model is loaded into VRAM - and it is exactly right for ornith:9b and ornith:35b on the tower. A hosted route reports no context_length, so it fell through to the ceiling, and every hosted budget became the tower's.

Fix

The discriminator was already in the registry: a route with a direct target is served out of the tower, and a route without one is served by a hosted provider.

derive_context_budget(context_length, local=) returns the budget and which of four bounds produced it: model_window, vram_ceiling, cost_ceiling, unbounded. The bound rides request.prompt_trimmed as budget_bound_by, so a trim log can no longer present a cost decision as a hardware limit or the reverse.
A hosted route with no declared window is left unbounded rather than given an invented number. The provider owns its own limit and reports it in an error the caller can read.
New optional registry field context_window lets Deploy declare the real window per route.
New PROXY_CONTEXT_COST_CEILING is how to ask for a smaller budget than the model allows, named for the reason it exists rather than borrowed from VRAM. It binds every route, hosted or local.
A hosted request stops carrying the num_ctx extension at all. Local routes behind LiteLLM keep both the ceiling and the injection.

How to verify

ward exec test - 11 new cases across tests/test_models.py, tests/test_route_registry.py, tests/test_analysis.py.
test_hosted_route_is_not_bounded_by_the_tower_ceiling pins the exact defect: that route used to resolve to 48128.
test_local_route_behind_litellm_keeps_the_tower_ceiling pins that sirens-echo/default is unchanged.

Test plan

Regression tests for the hosted and local cases
Existing tests still pass (331 passed)
ward exec format-check, lint, typecheck, pre-commit all clean

Risk

Medium, and worth naming precisely. Hosted prompts are no longer trimmed at all by default, so a genuinely oversized hosted prompt now reaches the provider and gets the provider's own error rather than a silent local trim. That is the intended trade - the previous behaviour destroyed turns to enforce a limit that did not exist - but it does mean hosted spend is bounded only by max_tokens and the caller until Deploy sets context_window or PROXY_CONTEXT_COST_CEILING. Local routes are untouched.

Follow-up for Deploy

services/agent-proxy/chart/files/route-registry.json can now declare context_window on sirens-echo/deepseek and evaluation/deepseek-v4-flash. Not required for this to be correct, and the loader accepts a registry with the field absent.

Note

Stacked on #116 (issue #113). The two are the same incident from opposite ends: #113 makes trimming correct wherever it runs, this stops it running where nothing asked for it. Review #116 first; this branch contains its commit.

## Bug Fixes #115. Every `request.prompt_trimmed` event in a seven-day window - all eleven, across every logical model - carried the same triple: ``` target_num_ctx = 48128 budget_tokens = 47104 headroom_tokens = 1024 ``` `48128 = 47 x 1024`, `47104 = 46 x 1024`. That is the shape of a VRAM allocation, not a provider limit. The DeepSeek v4 line advertises a 1M window, so the proxy was enforcing 4.7% of it. Nothing upstream ever objected on length: prompts of 171,203, 119,703, and 83,785 tokens all went out and came back complaining about message structure instead. ## Root cause `derive_num_ctx` applied `min(context_length, PROXY_NUM_CTX_CEILING) - headroom` to every route. `num_ctx` is an Ollama parameter - the window allocated when a model is loaded into VRAM - and it is exactly right for `ornith:9b` and `ornith:35b` on the tower. A hosted route reports no `context_length`, so it fell through to the ceiling, and every hosted budget became the tower's. ## Fix The discriminator was already in the registry: a route with a `direct` target is served out of the tower, and a route without one is served by a hosted provider. - `derive_context_budget(context_length, local=)` returns the budget **and** which of four bounds produced it: `model_window`, `vram_ceiling`, `cost_ceiling`, `unbounded`. The bound rides `request.prompt_trimmed` as `budget_bound_by`, so a trim log can no longer present a cost decision as a hardware limit or the reverse. - A hosted route with no declared window is left **unbounded** rather than given an invented number. The provider owns its own limit and reports it in an error the caller can read. - New optional registry field `context_window` lets Deploy declare the real window per route. - New `PROXY_CONTEXT_COST_CEILING` is how to ask for a smaller budget than the model allows, named for the reason it exists rather than borrowed from VRAM. It binds every route, hosted or local. - A hosted request stops carrying the `num_ctx` extension at all. Local routes behind LiteLLM keep both the ceiling and the injection. ## How to verify 1. `ward exec test` - 11 new cases across `tests/test_models.py`, `tests/test_route_registry.py`, `tests/test_analysis.py`. 2. `test_hosted_route_is_not_bounded_by_the_tower_ceiling` pins the exact defect: that route used to resolve to 48128. 3. `test_local_route_behind_litellm_keeps_the_tower_ceiling` pins that `sirens-echo/default` is unchanged. ## Test plan - [x] Regression tests for the hosted and local cases - [x] Existing tests still pass (331 passed) - [x] `ward exec format-check`, `lint`, `typecheck`, `pre-commit` all clean ## Risk Medium, and worth naming precisely. Hosted prompts are no longer trimmed at all by default, so a genuinely oversized hosted prompt now reaches the provider and gets the provider's own error rather than a silent local trim. That is the intended trade - the previous behaviour destroyed turns to enforce a limit that did not exist - but it does mean hosted spend is bounded only by `max_tokens` and the caller until Deploy sets `context_window` or `PROXY_CONTEXT_COST_CEILING`. Local routes are untouched. ## Follow-up for Deploy `services/agent-proxy/chart/files/route-registry.json` can now declare `context_window` on `sirens-echo/deepseek` and `evaluation/deepseek-v4-flash`. Not required for this to be correct, and the loader accepts a registry with the field absent. ## Note Stacked on #116 (issue #113). The two are the same incident from opposite ends: #113 makes trimming correct wherever it runs, this stops it running where nothing asked for it. Review #116 first; this branch contains its commit.

coilyco-ops added 2 commits

2026-08-13 17:46:43 +00:00

Drop whole tool-call groups when trimming a prompt

ci / smoke (pull_request) Successful in 7s

Details

ci / quality (pull_request) Successful in 28s

Details

1cdf8eca9d

Issue #113 measured eleven request.prompt_trimmed events over seven days and a
backend rejection beside every one of them: "Messages with role 'tool' must be a
response to a preceding message with 'tool_calls'". The ratio held at 24 log
lines per trim across three days at three different volumes, and no instance of
that 400 appeared on a day with no trim. The feature had, so far as telemetry
showed, never once produced a request the backend accepted.

The cause is in the selection rule rather than the budget. apply_context_budget
walked the non-system messages from the oldest end and dropped them one at a
time until the running total fit. A tool-heavy round is the first to cross the
budget, and it is exactly the round whose messages cannot be separated: an
assistant message carrying tool_calls and the tool messages answering those
calls are one unit in the OpenAI dialect. Dropping the assistant half and
keeping four of five replies produces a payload the provider rejects on
structure, which is why a 97-second six-round Discord turn delivered nothing.

group_tool_call_turns splits the prompt into the smallest units the trimmer may
drop whole. An assistant message with tool_calls collects the tool messages that
answer its ids; every other message is its own group, so grouping changes
nothing for a prompt without tools. The budget walk now costs and drops groups,
and the rule that the live turn always rides now keeps that turn's whole round
rather than its last message.

Group-wise dropping cannot orphan a reply, so unpaired_tool_message running over
the trimmed list can only fail on a prompt that arrived unpaired. That case
raises PromptPairingError and the route answers 400 naming the offending message
index, which is a fact the caller can act on, rather than spending three retries
and ~2.9 seconds to receive an opaque upstream 400 wrapped as a 502.

The budget that triggers all of this is wrong on hosted routes for a separate
reason, tracked in #115. This change makes trimming correct wherever it runs.

closes #113

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Derive the context budget from the model, not from VRAM

ci / smoke (pull_request) Successful in 7s

Details

ci / quality (pull_request) Successful in 26s

Details

1df7fc6cf1

Issue #115 read every request.prompt_trimmed event in a seven-day window, all
eleven of them, across every logical model, and found one invariant triple:
target_num_ctx 48128, budget_tokens 47104, headroom 1024. Those are 47x1024 and
46x1024, the shape of a VRAM allocation rather than a provider limit. The
DeepSeek v4 line the fleet routes to advertises a 1M window, so the proxy was
enforcing 4.7% of it. Nothing upstream had ever objected: prompts of 171203,
119703, and 83785 tokens all went out and came back complaining about message
structure, never about length.

num_ctx is an Ollama parameter. It is the window allocated when a model is
loaded into VRAM, it is exactly right for ornith:9b and ornith:35b on the tower,
and it means nothing to a hosted OpenAI-compatible provider that has no VRAM to
run out of. derive_num_ctx applied it to every route, and a hosted route reports
no context_length, so it fell through to the ceiling and every hosted budget
became the tower's.

The discriminator was already in the registry. A route with a direct target is
served out of the tower; a route without one is served by a hosted provider.
derive_context_budget takes that and returns both the budget and which of four
bounds produced it: model_window, vram_ceiling, cost_ceiling, or unbounded. The
bound rides request.prompt_trimmed as budget_bound_by, so a trim log can no
longer present a cost decision as a hardware limit or the reverse.

A hosted route with no declared window is left unbounded rather than given an
invented number. The provider owns its own limit and reports it in an error the
caller can read, which beats enforcing a guess silently. Deploy can declare the
real window per route through the new optional registry field context_window,
and PROXY_CONTEXT_COST_CEILING is how to ask for a smaller budget than the model
allows, named for the reason it exists rather than borrowed from VRAM.

A hosted request also stops carrying the num_ctx extension at all. Local routes
behind LiteLLM keep both the ceiling and the injection, because ornith really
does live in the tower's VRAM.

This removes the trigger for the trims in #113 on hosted routes. Both fixes
matter: #113 makes trimming correct wherever it runs, and this stops it running
where nothing asked for it.

closes #115

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 17:46:43 +00:00

coilyco-ops referenced this pull request

2026-08-13 17:55:00 +00:00

Classify an upstream 4xx as a request error, not a transport failure #118

coilyco-ops added 2 commits

2026-08-13 18:38:26 +00:00

Merge remote-tracking branch 'origin/main' into trim-tool-call-groups

ci / quality (pull_request) Successful in 23s

Details

ci / smoke (pull_request) Successful in 6s

Details

8e8baa8267

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

# Conflicts:
#	app/main.py
#	tests/test_api.py

Merge branch 'trim-tool-call-groups' into context-budget-per-model

ci / smoke (pull_request) Successful in 7s

Details

ci / quality (pull_request) Successful in 27s

Details

34e63b7115

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

# Conflicts:
#	app/config.py

coilysiren merged commit 74bf8004aa into main

2026-08-13 19:41:09 +00:00

coilysiren referenced this pull request from a commit

2026-08-13 19:41:10 +00:00

Merge pull request 'Derive the context budget from the model, not from VRAM' (#117) from context-budget-per-model into main

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy!117

No description provided.

Rows
Columns