The context budget is 47,104 tokens against a 1M-token model, because `num_ctx` is a local-inference parameter applied to a hosted route #115

New issue

Closed

opened 2026-08-13 17:25:38 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 17:25:38 +00:00

Member

🤖 Filed by Claude Code on Kai's behalf.

Raised by Kai while reading #113: "doesn't deepseek have a massive context window?" It does. The proxy is using 4.7% of it.

The numbers

Every request.prompt_trimmed event in the last 7 days, all 11 of them, across every logical model:

target_num_ctx = 48128
budget_tokens  = 47104     (= target − headroom)
headroom_tokens = 1024

Invariant. 48128 = 47 × 1024, 47104 = 46 × 1024 — round numbers in the shape of a VRAM allocation, not a provider limit.

Against that, from DeepSeek's published pricing page:

Model	Context length	Max output
`deepseek-v4-flash`	1M	384K
`deepseek-v4-pro`	1M	384K

The fleet is on the v4 line — litellm logs carry the model group evaluation/deepseek-v4-flash. I have not confirmed which upstream model sirens-echo/deepseek resolves to, since that lives in litellm's config and is not visible in telemetry; that is the one thing to check before acting on this. But there is no plausible resolution under which 47,104 is the right number.

`num_ctx` is the tell

num_ctx is an Ollama parameter — the context window you allocate when loading a model into VRAM. It is a real and necessary constraint for the local tier: #108 documents GET http://100.118.232.24:11434/api/tags health checks, and the configured fallbacks are deploy-backend/ornith:9b and deploy-backend/ornith:35b.

For a hosted OpenAI-compatible API it means nothing. The budget appears to be derived once from local-model sizing and applied to every route, including routes served by a provider with a 1M window and no VRAM to run out of.

The provider is not the one objecting

Direct evidence that 47,104 is not an upstream limit, from the 08-10 trim events in #113:

A prompt was trimmed to 171,203 tokens and sent. DeepSeek's response was the tool-pairing error, not a context-length error.
Another went out at 119,703 tokens. Same.
Another at 83,785 tokens. Same.

The provider parsed a 171K-token payload far enough to validate message structure and complained about something else entirely. Nothing upstream has ever rejected these requests for length.

What it costs

It manufactures the failures in #113. Trimming is the only thing that breaks these requests, and the budget is the only reason trimming runs. In trace 1a49200c3bebaed778ef2ac5b79d3d99 the prompt was 48,226 tokens — 1,122 over budget, and roughly 95% under the model's actual window. That turn was destroyed to enforce a limit that does not exist.

It caps every turn's working memory at 47K. coilyco-gaming/sirens-echo#162 records a 53 KB system prompt shipping on every turn; that alone consumes a meaningful fraction of the budget before the member's message is read. Round depth, tool-result retention, and conversation history are all bounded by this number.

Related, same shape: max_tokens: 3600 in the captured request body, against a model advertising 384K max output. coilyco-gaming/sirens-echo#367 is currently deliberating whether to raise 3600 and estimating a 10–22% token cost. Worth knowing what the ceiling actually is before that decision is priced.

What I am not claiming

Which model sirens-echo/deepseek resolves to. Not visible from telemetry. Check litellm's config first.
That the budget should simply become 1M. Cost, latency, and prefill all scale with prompt size, and coilyco-gaming/sirens-echo#162 shows the prompt is uncached, so a large budget has a real bill attached. The defect is that the number is derived from the wrong thing, not that it is small.
That local routes should change. For ornith:9b and ornith:35b, num_ctx is exactly right. This is about applying it where it does not belong.

Acceptance

The context budget is per-model, derived from the resolved upstream model's actual context window rather than from a single num_ctx.
A hosted route's budget is not derived from a local-inference VRAM parameter.
A request that fits the upstream model's window is not trimmed.
If a deliberate budget below the model's window is wanted for cost or latency reasons, it is named and configured as that — a cost ceiling — not as num_ctx, and the log says which of the two bound the request.

#113 — the trimmer defect this budget triggers; the two together are what killed the turn.
#114 — how the resulting 400 was retried and mislabeled.
#108 — the local tier, where num_ctx is legitimate.
#109 — tagging spans with backend identity; the same missing per-backend distinction is the root of both.
coilyco-gaming/sirens-echo#162 — the 53 KB uncached system prompt competing for this budget.
coilyco-gaming/sirens-echo#367 — the max_tokens: 3600 decision, priced without the real ceiling.

Next owner

Engineer.

🤖 Filed by Claude Code on Kai's behalf.

> 🤖 Filed by Claude Code on Kai's behalf. Raised by Kai while reading #113: *"doesn't deepseek have a massive context window?"* It does. The proxy is using 4.7% of it. ## The numbers Every `request.prompt_trimmed` event in the last 7 days, all 11 of them, across every logical model: ``` target_num_ctx = 48128 budget_tokens = 47104 (= target − headroom) headroom_tokens = 1024 ``` Invariant. `48128 = 47 × 1024`, `47104 = 46 × 1024` — round numbers in the shape of a VRAM allocation, not a provider limit. Against that, from DeepSeek's published pricing page: | Model | Context length | Max output | | --- | --- | --- | | `deepseek-v4-flash` | **1M** | 384K | | `deepseek-v4-pro` | **1M** | 384K | The fleet is on the v4 line — litellm logs carry the model group `evaluation/deepseek-v4-flash`. I have not confirmed which upstream model `sirens-echo/deepseek` resolves to, since that lives in litellm's config and is not visible in telemetry; that is the one thing to check before acting on this. But there is no plausible resolution under which 47,104 is the right number. ## `num_ctx` is the tell `num_ctx` is an Ollama parameter — the context window you allocate when loading a model into VRAM. It is a real and necessary constraint for the local tier: #108 documents `GET http://100.118.232.24:11434/api/tags` health checks, and the configured fallbacks are `deploy-backend/ornith:9b` and `deploy-backend/ornith:35b`. For a hosted OpenAI-compatible API it means nothing. The budget appears to be derived once from local-model sizing and applied to every route, including routes served by a provider with a 1M window and no VRAM to run out of. ## The provider is not the one objecting Direct evidence that 47,104 is not an upstream limit, from the 08-10 trim events in #113: - A prompt was trimmed to **171,203 tokens** and sent. DeepSeek's response was the tool-pairing error, not a context-length error. - Another went out at **119,703 tokens**. Same. - Another at **83,785 tokens**. Same. The provider parsed a 171K-token payload far enough to validate message structure and complained about something else entirely. Nothing upstream has ever rejected these requests for length. ## What it costs **It manufactures the failures in #113.** Trimming is the only thing that breaks these requests, and the budget is the only reason trimming runs. In trace `1a49200c3bebaed778ef2ac5b79d3d99` the prompt was 48,226 tokens — 1,122 over budget, and roughly 95% *under* the model's actual window. That turn was destroyed to enforce a limit that does not exist. **It caps every turn's working memory at 47K.** `coilyco-gaming/sirens-echo#162` records a 53 KB system prompt shipping on every turn; that alone consumes a meaningful fraction of the budget before the member's message is read. Round depth, tool-result retention, and conversation history are all bounded by this number. **Related, same shape:** `max_tokens: 3600` in the captured request body, against a model advertising 384K max output. `coilyco-gaming/sirens-echo#367` is currently deliberating whether to raise 3600 and estimating a 10–22% token cost. Worth knowing what the ceiling actually is before that decision is priced. ## What I am not claiming - **Which model `sirens-echo/deepseek` resolves to.** Not visible from telemetry. Check litellm's config first. - **That the budget should simply become 1M.** Cost, latency, and prefill all scale with prompt size, and `coilyco-gaming/sirens-echo#162` shows the prompt is uncached, so a large budget has a real bill attached. The defect is that the number is derived from the wrong thing, not that it is small. - **That local routes should change.** For `ornith:9b` and `ornith:35b`, `num_ctx` is exactly right. This is about applying it where it does not belong. ## Acceptance - The context budget is per-model, derived from the resolved upstream model's actual context window rather than from a single `num_ctx`. - A hosted route's budget is not derived from a local-inference VRAM parameter. - A request that fits the upstream model's window is not trimmed. - If a deliberate budget below the model's window is wanted for cost or latency reasons, it is named and configured as that — a cost ceiling — not as `num_ctx`, and the log says which of the two bound the request. ## Related - #113 — the trimmer defect this budget triggers; the two together are what killed the turn. - #114 — how the resulting 400 was retried and mislabeled. - #108 — the local tier, where `num_ctx` is legitimate. - #109 — tagging spans with backend identity; the same missing per-backend distinction is the root of both. - `coilyco-gaming/sirens-echo#162` — the 53 KB uncached system prompt competing for this budget. - `coilyco-gaming/sirens-echo#367` — the `max_tokens: 3600` decision, priced without the real ceiling. ## Next owner Engineer. > 🤖 Filed by Claude Code on Kai's behalf.

coilyco-ops referenced this issue from a commit

2026-08-13 17:35:59 +00:00

Drop whole tool-call groups when trimming a prompt

coilyco-ops referenced this issue

2026-08-13 17:37:03 +00:00

Drop whole tool-call groups when trimming a prompt #116

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 17:46:43 +00:00

Derive the context budget from the model, not from VRAM #117

coilyco-ops referenced this issue from a commit

2026-08-13 17:46:44 +00:00

Derive the context budget from the model, not from VRAM