num_ctx injection silently halved by OLLAMA_NUM_PARALLEL: verify effective context, fail loud

coilysiren commented

2026-07-03 19:22:08 +00:00

Owner

num_ctx injection is silently halved by `OLLAMA_NUM_PARALLEL`

The flagship fix (inject a big num_ctx so ollama stops silently truncating)
does not deliver the context it asks for when the ollama backend serves more
than one parallel slot. ollama loads num_ctx as the model's total context
and then divides it across OLLAMA_NUM_PARALLEL slots, so per-request usable
context is num_ctx / NUM_PARALLEL. The proxy is blind to this and reports
success.

Measured live (Windows tower, qwen3:4b, host.docker.internal:11434,
OLLAMA_NUM_PARALLEL=2), same ~100k-token prompt, straight to ollama
/api/chat:

num_ctx=49152 -> prompt_eval_count=24578   (= 49152/2 + 2)
num_ctx=32768 -> prompt_eval_count=16386   (= 32768/2 + 2)
num_ctx=65536 -> prompt_eval_count=32770   (= 65536/2 + 2)

/api/ps confirms the model loaded at context_length=65536 while that request's
prompt was capped at 32768 - the other half belongs to the second slot. Through
the proxy, fast-think (injects num_ctx=49152) kept 24578, not the 49151
docs/FEATURES.md claims. The proxy forwards num_ctx unmodified
(upstream.py _inject_options), and the budget guard never trims a single-turn
prompt, so the loss is entirely ollama-side.

Why it matters

This is the exact failure the proxy exists to kill (silent truncation below the
context you asked for), pushed down one layer. On the 3026 tower the FEATURES
number (49151) held, which means that host runs OLLAMA_NUM_PARALLEL=1. Any
backend with parallelism > 1 silently halves (or worse) the effective window, and
nothing surfaces it.

Fix - defense in depth, two loci

Proxy (this repo): treat effective context as the contract, not the
injected number.
- Verify the returned prompt_eval_count against intent: when the prompt was
  larger than what came back AND what came back is below the injected
  num_ctx, the backend truncated - surface it loud (metric + structured error
  / done_reason), do not return a silent short read. This is the "fail loud"
  thesis from #18's origin.
- Compensate: resolve the backend's OLLAMA_NUM_PARALLEL (env on the backend,
  or infer from context_length in /api/ps vs delivered) and inject
  target_ctx * num_parallel so the per-slot window equals the intended
  target_ctx. Ties into the auto-num_ctx derivation in #32.
Deployment (infrastructure/ansible): pin OLLAMA_NUM_PARALLEL=1 on the
proxy's ollama backends so a slot gets the whole window, and document the
coupling. Belongs in the ansible ollama role, not this repo.

Doc correction

docs/FEATURES.md "a 55k-token request to fast-think returns
prompt_eval_count=49151" is true only on a NUM_PARALLEL=1 backend. Qualify it,
and note the parallelism coupling wherever num_ctx is described (docs/proxy.md).

Related: #32 (alias retirement + auto num_ctx, where the compensation math
lives), #18 (fail-loud origin), #19 (baseline numbers depend on the backend's
NUM_PARALLEL).

Found from the agent-proxy read-only director surface on the Windows tower.
Discriminator + evidence are reproducible against any NUM_PARALLEL>1 ollama.

## num_ctx injection is silently halved by `OLLAMA_NUM_PARALLEL` The flagship fix (inject a big `num_ctx` so ollama stops silently truncating) does **not** deliver the context it asks for when the ollama backend serves more than one parallel slot. ollama loads `num_ctx` as the model's **total** context and then **divides it across `OLLAMA_NUM_PARALLEL` slots**, so per-request usable context is `num_ctx / NUM_PARALLEL`. The proxy is blind to this and reports success. **Measured live** (Windows tower, `qwen3:4b`, `host.docker.internal:11434`, `OLLAMA_NUM_PARALLEL=2`), same ~100k-token prompt, straight to ollama `/api/chat`: ``` num_ctx=49152 -> prompt_eval_count=24578 (= 49152/2 + 2) num_ctx=32768 -> prompt_eval_count=16386 (= 32768/2 + 2) num_ctx=65536 -> prompt_eval_count=32770 (= 65536/2 + 2) ``` `/api/ps` confirms the model loaded at `context_length=65536` while that request's prompt was capped at 32768 - the other half belongs to the second slot. Through the proxy, `fast-think` (injects `num_ctx=49152`) kept **24578**, not the 49151 `docs/FEATURES.md` claims. The proxy forwards `num_ctx` unmodified (`upstream.py` `_inject_options`), and the budget guard never trims a single-turn prompt, so the loss is entirely ollama-side. **Why it matters** This is the exact failure the proxy exists to kill (silent truncation below the context you asked for), pushed down one layer. On the 3026 tower the FEATURES number (49151) held, which means that host runs `OLLAMA_NUM_PARALLEL=1`. Any backend with parallelism > 1 silently halves (or worse) the effective window, and nothing surfaces it. **Fix - defense in depth, two loci** 1. **Proxy (this repo):** treat effective context as the contract, not the injected number. - Verify the returned `prompt_eval_count` against intent: when the prompt was larger than what came back AND what came back is below the injected `num_ctx`, the backend truncated - surface it loud (metric + structured error / `done_reason`), do not return a silent short read. This is the "fail loud" thesis from #18's origin. - Compensate: resolve the backend's `OLLAMA_NUM_PARALLEL` (env on the backend, or infer from `context_length` in `/api/ps` vs delivered) and inject `target_ctx * num_parallel` so the per-slot window equals the intended `target_ctx`. Ties into the auto-`num_ctx` derivation in #32. 2. **Deployment (infrastructure/ansible):** pin `OLLAMA_NUM_PARALLEL=1` on the proxy's ollama backends so a slot gets the whole window, and document the coupling. Belongs in the ansible ollama role, not this repo. **Doc correction** `docs/FEATURES.md` "a 55k-token request to fast-think returns `prompt_eval_count=49151`" is true only on a `NUM_PARALLEL=1` backend. Qualify it, and note the parallelism coupling wherever `num_ctx` is described (`docs/proxy.md`). **Related:** #32 (alias retirement + auto `num_ctx`, where the compensation math lives), #18 (fail-loud origin), #19 (baseline numbers depend on the backend's `NUM_PARALLEL`). Found from the agent-proxy read-only director surface on the Windows tower. Discriminator + evidence are reproducible against any `NUM_PARALLEL>1` ollama.

coilysiren commented

2026-07-03 19:25:01 +00:00

Author

Owner

Unblocked: #32 has landed on main (aad957ae). The alias table is gone; models.py now derives num_ctx via derive_num_ctx(context_length) = min(context_length, PROXY_NUM_CTX_CEILING) - PROXY_NUM_CTX_HEADROOM, reading /api/tags details.context_length. So this fix builds directly on derive_num_ctx: make it NUM_PARALLEL-aware (the delivered per-request context is injected num_ctx / NUM_PARALLEL), and add the verify-and-fail-loud check in upstream.py comparing prompt_eval_count against intent. Dispatching an engineer now.

Unblocked: #32 has landed on main (aad957ae). The alias table is gone; `models.py` now derives num_ctx via `derive_num_ctx(context_length) = min(context_length, PROXY_NUM_CTX_CEILING) - PROXY_NUM_CTX_HEADROOM`, reading `/api/tags` `details.context_length`. So this fix builds directly on `derive_num_ctx`: make it NUM_PARALLEL-aware (the delivered per-request context is injected num_ctx / NUM_PARALLEL), and add the verify-and-fail-loud check in `upstream.py` comparing `prompt_eval_count` against intent. Dispatching an engineer now.

coilyco-ops commented

2026-07-03 19:25:18 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-agent-proxy-33 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-03T19:25:13Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agent-proxy-33` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-03T19:25:13Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren commented

2026-07-03 19:30:26 +00:00

Author

Owner

Live-confirmed on current main (post-#32) against the tower ollama with NUM_PARALLEL=2: request model=qwen3:4b, ~130k-token prompt. derive_num_ctx produced 48128 (= min(262144, 49152) - 1024, correct), but delivered prompt_tokens=24066 (= 48128/2 + 2). So the #32 derivation is right and the loss is purely the parallel-slot split. /v1/models correctly lists live tags [qwen3:4b, qwen3:8b]. Fix stands: make derive/inject NUM_PARALLEL-aware and verify prompt_eval_count against intent.

coilysiren referenced this issue

2026-07-03 19:33:30 +00:00

[proxy 07/13] Live num_ctx proof script against the tower #11

coilysiren referenced this issue from a commit

2026-07-03 19:35:07 +00:00

feat(#33): verify effective context and compensate for OLLAMA_NUM_PARALLEL

coilysiren closed this issue

2026-07-03 19:35:07 +00:00

coilysiren commented

2026-07-03 19:35:51 +00:00

Author

Owner

WARD-OUTCOME: done - proxy compensates injected num_ctx by OLLAMA_NUM_PARALLEL and fail-loud verifies delivered context (prompt_eval_count) against intent; docs corrected.

Retrospective: this one was cleaner than I expected because #32 had already done the hard part. derive_num_ctx gave me a single, well-named place to build on, and the pass-through Backend spec meant num_parallel slotted in as one more per-backend field without touching the resolution flow. The compensation is a one-line multiply in _inject_options; the interesting work was the fail-loud discriminator.

What fought back a little: getting the truncation check to not false-positive. The proxy counts prompt tokens with tiktoken but the backend reports ollama's real count, and those drift by tokenizer, so a naive "eval_count < num_ctx" flags every medium prompt. I settled on requiring the delivered count to fall materially below both the sent size and the target window, with a configurable tolerance (default 0.15) to absorb drift - the NUM_PARALLEL halving is a ~50% shortfall, miles past any tokenizer noise, so the two signals separate cleanly. The issue's live numbers (24578 against 49152) became the test fixtures, which made me fairly confident the discriminator matches reality.

One deliberate call worth flagging: I placed the verify in resilience.dispatch (where validation already lives) rather than upstream.py as the thread suggested - dispatch is the layer that owns "is this response usable", and it already had the trace-span plumbing. Default behavior marks the short read (finish_reason=length + llm_context_truncated_total + a structured log) and still returns content; PROXY_FAIL_ON_CONTEXT_TRUNCATION makes it a hard 502. I chose mark-by-default because hard-failing every request on a misconfigured backend felt worse than a loud-but-served answer, and it stays reversible.

Confidence: high on the offline behavior (79 tests green, ruff/black/mypy clean), lower on the live end-to-end since I have no NUM_PARALLEL>1 tower to drive here - the integration tests mock the backend at the issue's exact measured numbers, which is the best I could verify from the container.

Rough edges / follow-ups worth filing: (1) streaming skips the check - ollama's final stream chunk does carry prompt_eval_count, so a streaming fail-loud is a natural follow-up; (2) I did not implement the /api/ps-based inference of NUM_PARALLEL the issue mentioned as an alternative - config-driven num_parallel is deterministic and testable, and inference adds a live call per request, so I left it as a possible enhancement; (3) the ansible half (pinning OLLAMA_NUM_PARALLEL=1) is correctly out of this repo and still needs doing in infrastructure/ansible.

WARD-OUTCOME: done - proxy compensates injected num_ctx by OLLAMA_NUM_PARALLEL and fail-loud verifies delivered context (prompt_eval_count) against intent; docs corrected. Retrospective: this one was cleaner than I expected because #32 had already done the hard part. `derive_num_ctx` gave me a single, well-named place to build on, and the pass-through Backend spec meant `num_parallel` slotted in as one more per-backend field without touching the resolution flow. The compensation is a one-line multiply in `_inject_options`; the interesting work was the fail-loud discriminator. What fought back a little: getting the truncation check to not false-positive. The proxy counts prompt tokens with tiktoken but the backend reports ollama's real count, and those drift by tokenizer, so a naive "eval_count < num_ctx" flags every medium prompt. I settled on requiring the delivered count to fall materially below *both* the sent size and the target window, with a configurable tolerance (default 0.15) to absorb drift - the NUM_PARALLEL halving is a ~50% shortfall, miles past any tokenizer noise, so the two signals separate cleanly. The issue's live numbers (24578 against 49152) became the test fixtures, which made me fairly confident the discriminator matches reality. One deliberate call worth flagging: I placed the verify in `resilience.dispatch` (where validation already lives) rather than `upstream.py` as the thread suggested - dispatch is the layer that owns "is this response usable", and it already had the trace-span plumbing. Default behavior *marks* the short read (finish_reason=length + `llm_context_truncated_total` + a structured log) and still returns content; `PROXY_FAIL_ON_CONTEXT_TRUNCATION` makes it a hard 502. I chose mark-by-default because hard-failing every request on a misconfigured backend felt worse than a loud-but-served answer, and it stays reversible. Confidence: high on the offline behavior (79 tests green, ruff/black/mypy clean), lower on the live end-to-end since I have no NUM_PARALLEL>1 tower to drive here - the integration tests mock the backend at the issue's exact measured numbers, which is the best I could verify from the container. Rough edges / follow-ups worth filing: (1) streaming skips the check - ollama's final stream chunk does carry prompt_eval_count, so a streaming fail-loud is a natural follow-up; (2) I did not implement the `/api/ps`-based inference of NUM_PARALLEL the issue mentioned as an alternative - config-driven `num_parallel` is deterministic and testable, and inference adds a live call per request, so I left it as a possible enhancement; (3) the ansible half (pinning OLLAMA_NUM_PARALLEL=1) is correctly out of this repo and still needs doing in infrastructure/ansible.

coilysiren referenced this issue

2026-07-03 19:41:50 +00:00

Finish leg-05 reliability harness and record the M2 baseline-to-after result #19

Rows
Columns

num_ctx injection silently halved by OLLAMA_NUM_PARALLEL: verify effective context, fail loud #33

num_ctx injection is silently halved by OLLAMA_NUM_PARALLEL

num_ctx injection is silently halved by `OLLAMA_NUM_PARALLEL`