coilyco-flight-deck/agent-proxy

Watch

Drop whole tool-call groups when trimming a prompt #116

Merged

coilysiren merged 2 commits from trim-tool-call-groups into main

2026-08-13 19:41:00 +00:00

coilyco-ops commented

2026-08-13 17:37:03 +00:00

Member

Bug

Fixes #113.

Eleven request.prompt_trimmed events over seven days, and a backend rejection beside every one of them:

DeepseekException - {"error":{"message":"Messages with role 'tool' must be a response
to a preceding message with 'tool_calls'", ...}}

The ratio held at 24 log lines per trim across three days at three different volumes, and no instance of that 400 appeared on a day with no trim. As far as telemetry shows, trimming had never once produced a request the backend accepted.

Root cause

apply_context_budget walked the non-system messages from the oldest end and dropped them one at a time until the running total fit. A tool-heavy round is the first round to cross the budget, and it is exactly the round whose messages cannot be separated: an assistant message carrying tool_calls and the tool messages answering those calls are one unit in the OpenAI dialect. In the trace on the issue, the trimmer dropped 5 messages out of a 23-message round that carried one assistant turn plus five steam__get_store_app_details results, and the resulting payload was rejected on structure.

Fix

group_tool_call_turns splits the prompt into the smallest units the trimmer may drop whole. An assistant message with tool_calls collects the tool messages answering its ids; every other message is its own group, so grouping is a no-op for a prompt without tools.
The budget walk costs and drops groups. The rule that the live turn always rides now keeps that turn's whole round rather than its last message.
unpaired_tool_message checks the trimmed list against the upstream rule before dispatch. Group-wise dropping cannot orphan a reply, so a failure there means the prompt arrived unpaired: that raises PromptPairingError and the route answers 400 naming the offending message, instead of spending three retries to receive an opaque upstream 400 wrapped as a 502.

How to verify

ward exec test - 8 new cases in tests/test_analysis.py and 1 in tests/test_api.py.
test_trim_never_orphans_a_tool_reply builds the trace's shape and asserts neither half of the dropped round survives alone.
test_unpaired_trimmed_prompt_is_rejected_locally asserts the stubbed upstream is never called.

Test plan

Regression tests for the reported shape
Existing tests still pass (316 passed)
ward exec format-check, lint, typecheck, pre-commit all clean

Risk

Low. Grouping is identity for prompts without tool calls, so every non-tool path is byte-identical to today. The one behaviour change beyond correct trimming is that an already-unpaired prompt now fails locally with 400 rather than travelling upstream, and only on the path where trimming ran.

Note

The budget that triggers trimming is itself wrong on hosted routes, for a separate reason tracked in #115. This change makes trimming correct wherever it runs.

## Bug Fixes #113. Eleven `request.prompt_trimmed` events over seven days, and a backend rejection beside every one of them: ``` DeepseekException - {"error":{"message":"Messages with role 'tool' must be a response to a preceding message with 'tool_calls'", ...}} ``` The ratio held at 24 log lines per trim across three days at three different volumes, and no instance of that 400 appeared on a day with no trim. As far as telemetry shows, trimming had never once produced a request the backend accepted. ## Root cause `apply_context_budget` walked the non-system messages from the oldest end and dropped them one at a time until the running total fit. A tool-heavy round is the first round to cross the budget, and it is exactly the round whose messages cannot be separated: an `assistant` message carrying `tool_calls` and the `tool` messages answering those calls are one unit in the OpenAI dialect. In the trace on the issue, the trimmer dropped 5 messages out of a 23-message round that carried one assistant turn plus five `steam__get_store_app_details` results, and the resulting payload was rejected on structure. ## Fix - `group_tool_call_turns` splits the prompt into the smallest units the trimmer may drop whole. An assistant message with `tool_calls` collects the `tool` messages answering its ids; every other message is its own group, so grouping is a no-op for a prompt without tools. - The budget walk costs and drops groups. The rule that the live turn always rides now keeps that turn's whole round rather than its last message. - `unpaired_tool_message` checks the trimmed list against the upstream rule before dispatch. Group-wise dropping cannot orphan a reply, so a failure there means the prompt arrived unpaired: that raises `PromptPairingError` and the route answers 400 naming the offending message, instead of spending three retries to receive an opaque upstream 400 wrapped as a 502. ## How to verify 1. `ward exec test` - 8 new cases in `tests/test_analysis.py` and 1 in `tests/test_api.py`. 2. `test_trim_never_orphans_a_tool_reply` builds the trace's shape and asserts neither half of the dropped round survives alone. 3. `test_unpaired_trimmed_prompt_is_rejected_locally` asserts the stubbed upstream is never called. ## Test plan - [x] Regression tests for the reported shape - [x] Existing tests still pass (316 passed) - [x] `ward exec format-check`, `lint`, `typecheck`, `pre-commit` all clean ## Risk Low. Grouping is identity for prompts without tool calls, so every non-tool path is byte-identical to today. The one behaviour change beyond correct trimming is that an already-unpaired prompt now fails locally with 400 rather than travelling upstream, and only on the path where trimming ran. ## Note The budget that triggers trimming is itself wrong on hosted routes, for a separate reason tracked in #115. This change makes trimming correct wherever it runs.

coilyco-ops added 1 commit

2026-08-13 17:37:03 +00:00

Drop whole tool-call groups when trimming a prompt

ci / smoke (pull_request) Successful in 7s

Details

ci / quality (pull_request) Successful in 28s

Details

1cdf8eca9d

Issue #113 measured eleven request.prompt_trimmed events over seven days and a
backend rejection beside every one of them: "Messages with role 'tool' must be a
response to a preceding message with 'tool_calls'". The ratio held at 24 log
lines per trim across three days at three different volumes, and no instance of
that 400 appeared on a day with no trim. The feature had, so far as telemetry
showed, never once produced a request the backend accepted.

The cause is in the selection rule rather than the budget. apply_context_budget
walked the non-system messages from the oldest end and dropped them one at a
time until the running total fit. A tool-heavy round is the first to cross the
budget, and it is exactly the round whose messages cannot be separated: an
assistant message carrying tool_calls and the tool messages answering those
calls are one unit in the OpenAI dialect. Dropping the assistant half and
keeping four of five replies produces a payload the provider rejects on
structure, which is why a 97-second six-round Discord turn delivered nothing.

group_tool_call_turns splits the prompt into the smallest units the trimmer may
drop whole. An assistant message with tool_calls collects the tool messages that
answer its ids; every other message is its own group, so grouping changes
nothing for a prompt without tools. The budget walk now costs and drops groups,
and the rule that the live turn always rides now keeps that turn's whole round
rather than its last message.

Group-wise dropping cannot orphan a reply, so unpaired_tool_message running over
the trimmed list can only fail on a prompt that arrived unpaired. That case
raises PromptPairingError and the route answers 400 naming the offending message
index, which is a fact the caller can act on, rather than spending three retries
and ~2.9 seconds to receive an opaque upstream 400 wrapped as a 502.

The budget that triggers all of this is wrong on hosted routes for a separate
reason, tracked in #115. This change makes trimming correct wherever it runs.

closes #113

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 17:37:03 +00:00

coilyco-ops referenced this pull request

2026-08-13 17:46:43 +00:00

Derive the context budget from the model, not from VRAM #117

coilyco-ops referenced this pull request

2026-08-13 17:55:00 +00:00

Classify an upstream 4xx as a request error, not a transport failure #118

coilyco-ops added 1 commit

2026-08-13 18:37:13 +00:00

Merge remote-tracking branch 'origin/main' into trim-tool-call-groups

ci / quality (pull_request) Successful in 23s

Details

ci / smoke (pull_request) Successful in 6s

Details

8e8baa8267

Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

# Conflicts:
#	app/main.py
#	tests/test_api.py

coilysiren merged commit 4982780b89 into main

2026-08-13 19:41:00 +00:00

coilysiren referenced this pull request from a commit

2026-08-13 19:41:02 +00:00

Merge pull request 'Drop whole tool-call groups when trimming a prompt' (#116) from trim-tool-call-groups into main

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy!116

No description provided.

Rows
Columns