coilyco-gaming/sirens-echo

Watch

Malformed message history: role 'tool' sent without a preceding assistant tool_calls message, DeepSeek 400s the turn #875

New issue

Closed

opened 2026-08-17 02:10:17 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-08-17 02:10:17 +00:00

Member

What

Sirens is sending the model a conversation history where a role: "tool" message has no preceding assistant message carrying tool_calls. DeepSeek rejects it:

litellm.BadRequestError: DeepseekException -
{"error":{"message":"Messages with role 'tool' must be a response to a preceding message with 'tool_calls'",
"type":"invalid_request_error","param":null,"code":"invalid_request_error"}}

This is a malformed request built by the harness. Not a model failure, not a DeepSeek outage. DeepSeek is correctly refusing an invalid request.

Live and ongoing

Not a historical incident. 12 rejections in a 30-minute sample at 2026-08-17 02:00Z, still firing. Each user-visible failure costs two upstream round trips, because LiteLLM retries once before giving up.

User impact

This is what killed the community-visible Discord turns during the 2026-08-16 incident. Confirmed with Kai: users saw turns hang for a long time and then fail, described as a very annoying experience.

The message they got was misleading. turn.stage.failed reported model backend unavailable, retry shortly when nothing about the backend was unavailable. Retrying does not help, because the same malformed history gets rebuilt.

Two candidate causes, both harness-side

A failed tool call appends its result without the assistant tool_calls message. This would tie the incident together. Tool calls are failing constantly and still are: demo-discord ran 15 failed of 28 in a 2h window (54%), with openlibrary also failing. See coilyco-bridge/deploy#591.
Context-window trimming cuts between an assistant tool_calls message and its tool response. Trimming that does not treat that pair as atomic produces exactly this error and is a common way to reach it.

Not yet distinguished. What would settle it is one captured request body that DeepSeek rejected.

Ask

Ensure the message array is well-formed before dispatch. Every role: "tool" message must be immediately preceded by an assistant message whose tool_calls contains the matching tool_call_id, and history trimming must treat an assistant tool_calls message and its tool responses as one atomic unit that is kept or dropped together.

Two related fixes worth the same pass:

Do not report a 400 as "model backend unavailable." A malformed-request rejection and a backend outage need different user-facing text, because one is worth retrying and the other never is.
This compounds with coilyco-gaming/sirens-echo#873. Failed tool calls do not set span error status and record no reason, which is part of why this chain stayed invisible.

coilyco-gaming/sirens-echo#873 - failed MCP tool calls do not set span error status
coilyco-flight-deck/mcp-beaver#78 - ward-mcp containers ship no logs
coilyco-bridge/deploy#591 - demo-discord 47% tool call failure rate

Filed from an ops investigation into sirens-echo / sirens-deep errors. Found only after a SigNoz log parser was added for LiteLLM, whose logs previously carried no severity at all.

## What Sirens is sending the model a conversation history where a `role: "tool"` message has no preceding assistant message carrying `tool_calls`. DeepSeek rejects it: ``` litellm.BadRequestError: DeepseekException - {"error":{"message":"Messages with role 'tool' must be a response to a preceding message with 'tool_calls'", "type":"invalid_request_error","param":null,"code":"invalid_request_error"}} ``` This is a malformed request built by the harness. Not a model failure, not a DeepSeek outage. DeepSeek is correctly refusing an invalid request. ## Live and ongoing Not a historical incident. 12 rejections in a 30-minute sample at 2026-08-17 02:00Z, still firing. Each user-visible failure costs two upstream round trips, because LiteLLM retries once before giving up. ## User impact This is what killed the community-visible Discord turns during the 2026-08-16 incident. Confirmed with Kai: users saw turns hang for a long time and then fail, described as a very annoying experience. The message they got was misleading. `turn.stage.failed` reported `model backend unavailable, retry shortly` when nothing about the backend was unavailable. Retrying does not help, because the same malformed history gets rebuilt. ## Two candidate causes, both harness-side * **A failed tool call appends its result without the assistant `tool_calls` message.** This would tie the incident together. Tool calls are failing constantly and still are: demo-discord ran 15 failed of 28 in a 2h window (54%), with openlibrary also failing. See coilyco-bridge/deploy#591. * **Context-window trimming cuts between an assistant `tool_calls` message and its tool response.** Trimming that does not treat that pair as atomic produces exactly this error and is a common way to reach it. Not yet distinguished. What would settle it is one captured request body that DeepSeek rejected. ## Ask Ensure the message array is well-formed before dispatch. Every `role: "tool"` message must be immediately preceded by an assistant message whose `tool_calls` contains the matching `tool_call_id`, and history trimming must treat an assistant `tool_calls` message and its tool responses as one atomic unit that is kept or dropped together. Two related fixes worth the same pass: * **Do not report a 400 as "model backend unavailable."** A malformed-request rejection and a backend outage need different user-facing text, because one is worth retrying and the other never is. * This compounds with coilyco-gaming/sirens-echo#873. Failed tool calls do not set span error status and record no reason, which is part of why this chain stayed invisible. ## Related * coilyco-gaming/sirens-echo#873 - failed MCP tool calls do not set span error status * coilyco-flight-deck/mcp-beaver#78 - ward-mcp containers ship no logs * coilyco-bridge/deploy#591 - demo-discord 47% tool call failure rate Filed from an ops investigation into sirens-echo / sirens-deep errors. Found only after a SigNoz log parser was added for LiteLLM, whose logs previously carried no severity at all.

coilyco-ops commented

2026-08-17 02:20:12 +00:00

Author

Member

Cause confirmed: history trimming, not the failed-tool-call path

Captured a rejected request from agent-proxy's normalized request capture. Trace ac9d1729365e676f80926d1da1c20bbf, 2026-08-17 02:01:05Z, route sirens-echo/deepseek.

Only the message role sequence is reproduced below. No message content was extracted, since this capture is restricted model-content storage.

The trim event, immediately before the rejected request

{"event": "request.prompt_trimmed",
 "original_token_count": 47843, "final_token_count": 45659,
 "budget_tokens": 47104, "target_num_ctx": 48128, "headroom_tokens": 1024,
 "dropped_message_count": 3}

The resulting message array, 31 messages, 61 tools

idx  role       tool_calls                              tool_call_id
0    system     -                                       -
1    system     -                                       -
2    system     -                                       -
3    tool       -                                       ...7fc5a7_1   <-- orphan
4    tool       -                                       ...155d76_2   <-- orphan
5    assistant  93397d_0, f9d5ae_1, d7eff6_2, 14ed12_3  -
6    tool       -                                       ...93397d_0
7    tool       -                                       ...f9d5ae_1
...

Messages 3 and 4 are role: "tool" immediately preceded by a system message. There is no assistant tool_calls message before them anywhere, and their ids (7fc5a7_1, 155d76_2) match nothing in message 5's tool_calls. They are orphaned tool responses whose parent assistant message was dropped.

The _1 and _2 suffixes imply a _0 sibling that was dropped alongside the parent, which accounts for dropped_message_count: 3.

The rest of the array is well-formed

Every other group is intact and correctly ordered: 5 to 6/7/8/9, 10 to 11/12, 13 to 14, 15 to 16/17, and so on, with every tool_call_id matching its parent exactly. The harness builds history correctly. Only the trimmer breaks it.

The bug

The trimmer drops messages by token budget without treating an assistant tool_calls message and its tool responses as one atomic unit. It removed a parent while leaving two children behind, producing an array DeepSeek is right to reject.

Note the margin: it was 739 tokens over budget (47,843 against 47,104) and dropped 3 messages to recover 2,184. A trimmer that dropped whole groups would have had ample room.

Fix

Make trimming group-aware. An assistant message carrying tool_calls and all tool responses referencing those ids are kept or dropped together, never split. A validation pass before dispatch, asserting every role: "tool" has a matching preceding tool_calls id, would turn any future regression into a caught assertion rather than a provider 400.

Two adjacent observations from the same capture

circuit.open fired on this backend at failures: 5 and failures: 6. The circuit breaker is counting these malformed-request 400s as backend failures. A 400 is a client error and arguably should not trip a breaker protecting against backend unavailability, since it opens the circuit against a healthy provider.
61 tool definitions ride in each request against a 47k budget. That is what keeps the context near the trim threshold, so the trimmer fires often and this bug is hit often. Worth a look independently of the fix.

Also worth noting: this is a plain sequencing bug that a request-validation pass at the proxy would have caught on the first occurrence.

## Cause confirmed: history trimming, not the failed-tool-call path Captured a rejected request from agent-proxy's normalized request capture. Trace `ac9d1729365e676f80926d1da1c20bbf`, 2026-08-17 02:01:05Z, route `sirens-echo/deepseek`. Only the message role sequence is reproduced below. No message content was extracted, since this capture is restricted model-content storage. ### The trim event, immediately before the rejected request ```json {"event": "request.prompt_trimmed", "original_token_count": 47843, "final_token_count": 45659, "budget_tokens": 47104, "target_num_ctx": 48128, "headroom_tokens": 1024, "dropped_message_count": 3} ``` ### The resulting message array, 31 messages, 61 tools ``` idx role tool_calls tool_call_id 0 system - - 1 system - - 2 system - - 3 tool - ...7fc5a7_1 <-- orphan 4 tool - ...155d76_2 <-- orphan 5 assistant 93397d_0, f9d5ae_1, d7eff6_2, 14ed12_3 - 6 tool - ...93397d_0 7 tool - ...f9d5ae_1 ... ``` Messages 3 and 4 are `role: "tool"` immediately preceded by a `system` message. There is no assistant `tool_calls` message before them anywhere, and their ids (`7fc5a7_1`, `155d76_2`) match nothing in message 5's tool_calls. They are orphaned tool responses whose parent assistant message was dropped. The `_1` and `_2` suffixes imply a `_0` sibling that was dropped alongside the parent, which accounts for `dropped_message_count: 3`. ### The rest of the array is well-formed Every other group is intact and correctly ordered: 5 to 6/7/8/9, 10 to 11/12, 13 to 14, 15 to 16/17, and so on, with every `tool_call_id` matching its parent exactly. The harness builds history correctly. **Only the trimmer breaks it.** ### The bug The trimmer drops messages by token budget without treating an assistant `tool_calls` message and its tool responses as one atomic unit. It removed a parent while leaving two children behind, producing an array DeepSeek is right to reject. Note the margin: it was 739 tokens over budget (47,843 against 47,104) and dropped 3 messages to recover 2,184. A trimmer that dropped whole groups would have had ample room. ### Fix Make trimming group-aware. An assistant message carrying `tool_calls` and all tool responses referencing those ids are kept or dropped together, never split. A validation pass before dispatch, asserting every `role: "tool"` has a matching preceding `tool_calls` id, would turn any future regression into a caught assertion rather than a provider 400. ### Two adjacent observations from the same capture * **`circuit.open` fired** on this backend at `failures: 5` and `failures: 6`. The circuit breaker is counting these malformed-request 400s as backend failures. A 400 is a client error and arguably should not trip a breaker protecting against backend unavailability, since it opens the circuit against a healthy provider. * **61 tool definitions** ride in each request against a 47k budget. That is what keeps the context near the trim threshold, so the trimmer fires often and this bug is hit often. Worth a look independently of the fix. Also worth noting: this is a plain sequencing bug that a request-validation pass at the proxy would have caught on the first occurrence.

coilyco-ops referenced this issue

2026-08-17 03:00:43 +00:00

Echo's p99 turn is the 180s ceiling itself, and raising the completion budget would push more turns into it #577

coilyco-ops added the

labels

2026-08-17 05:20:10 +00:00

coilyco-ops commented

2026-08-17 05:20:35 +00:00

Author

Member

No owner, no labels, and it is the largest live failure class on the 2026-08-19 demo path. Labelled priority/P0 autonomy/headless role/engineer. Darren (director seat), 2026-08-17, from a cross-repo status check.

Kai asked me to find out whether this had an owner. It did not: unassigned, and carrying zero labels in a repo whose org has the full scoped taxonomy. Unlabelled fails closed, so it has been invisible to every queue since it was filed at 02:10Z, while engineers worked that repo on labelled headless work.

Why P0

Measured at litellm, the layer that talks to DeepSeek, over 24h:

400 Bad Request     60
500                  5

This bug is the single largest failure class in the estate right now, by a wide margin over every availability class combined.

Its blast radius reaches two other repos:

coilyco-bridge/deploy#344 - the deepseek fallback. Its constraint is that a fallback must not fire on a 400, because that ships the same malformed history to a second provider which may well accept it, turning a loud bug into a silent quality regression. So the fallback cannot cover this, by design.
coilyco-bridge/deploy#619 - the exec brief choosing that provider. It records this issue as the higher-value fix and sequences it first.

A second-order effect worth stating plainly: agent-proxy logged all backends failed (litellm circuit open) twelve times in the same window. LiteLLM's breaker opens on repeated failures and these 400s count toward it, so this bug manufactures availability failures that then kill otherwise-good requests. Fixing it reduces the failure classes a fallback would have covered.

Why headless

The cause is confirmed rather than hypothesised, in the comment above. The capture from trace ac9d1729365e676f80926d1da1c20bbf is unambiguous:

{"event": "request.prompt_trimmed",
 "original_token_count": 47843, "final_token_count": 45659,
 "dropped_message_count": 3}

and the resulting array opens with orphans:

idx  role       tool_calls    tool_call_id
3    tool       -             ...7fc5a7_1   <-- orphan
4    tool       -             ...155d76_2   <-- orphan
5    assistant  93397d_0, ...

The trimmer drops messages from the front to fit the budget and does not preserve the invariant that a tool message must follow the assistant message carrying its tool_calls. That is a self-contained harness fix with no design fork and no product question, which is what autonomy/headless means.

Worth pinning in a test: trimming must never leave a tool message whose tool_call_id has no preceding assistant tool_calls in the surviving array. Dropping the orphaned tool messages alongside their assistant parent, rather than counting messages, is the shape most likely to be correct.

Two things about user impact that raise it above a routing bug

It is what killed the community-visible Discord turns in the 2026-08-16 incident, confirmed with Kai, described as turns hanging a long time and then failing.
The message users got was wrong: turn.stage.failed reported model backend unavailable, retry shortly when nothing about the backend was unavailable. Retrying rebuilds the same malformed history, so the advice is actively misleading. Whatever fixes the trim should also stop this failure class claiming a backend outage.

Each rejection costs two upstream round trips, since LiteLLM retries once before giving up.

#873, failed MCP tool calls not setting span error status, is the reason the status code is null on model.chat spans and I had to reconstruct these numbers from litellm and agent-proxy instead. It is a diagnosis multiplier for exactly this class of incident and is sitting unlabelled too.

**No owner, no labels, and it is the largest live failure class on the 2026-08-19 demo path. Labelled `priority/P0` `autonomy/headless` `role/engineer`. Darren (director seat), 2026-08-17, from a cross-repo status check.** Kai asked me to find out whether this had an owner. It did not: unassigned, and carrying **zero labels** in a repo whose org has the full scoped taxonomy. Unlabelled fails closed, so it has been invisible to every queue since it was filed at 02:10Z, while engineers worked that repo on labelled headless work. ## Why P0 Measured at `litellm`, the layer that talks to DeepSeek, over 24h: ``` 400 Bad Request 60 500 5 ``` **This bug is the single largest failure class in the estate right now**, by a wide margin over every availability class combined. Its blast radius reaches two other repos: * `coilyco-bridge/deploy#344` - the deepseek fallback. Its constraint is that a fallback must **not** fire on a 400, because that ships the same malformed history to a second provider which may well accept it, turning a loud bug into a silent quality regression. So the fallback cannot cover this, by design. * `coilyco-bridge/deploy#619` - the exec brief choosing that provider. It records this issue as the higher-value fix and sequences it first. **A second-order effect worth stating plainly:** `agent-proxy` logged `all backends failed (litellm circuit open)` twelve times in the same window. LiteLLM's breaker opens on repeated failures and **these 400s count toward it**, so this bug manufactures availability failures that then kill otherwise-good requests. Fixing it reduces the failure classes a fallback *would* have covered. ## Why headless The cause is confirmed rather than hypothesised, in the comment above. The capture from trace `ac9d1729365e676f80926d1da1c20bbf` is unambiguous: ```json {"event": "request.prompt_trimmed", "original_token_count": 47843, "final_token_count": 45659, "dropped_message_count": 3} ``` and the resulting array opens with orphans: ``` idx role tool_calls tool_call_id 3 tool - ...7fc5a7_1 <-- orphan 4 tool - ...155d76_2 <-- orphan 5 assistant 93397d_0, ... ``` The trimmer drops messages from the front to fit the budget and does not preserve the invariant that a `tool` message must follow the assistant message carrying its `tool_calls`. That is a self-contained harness fix with no design fork and no product question, which is what `autonomy/headless` means. Worth pinning in a test: trimming must never leave a `tool` message whose `tool_call_id` has no preceding assistant `tool_calls` in the surviving array. Dropping the orphaned `tool` messages alongside their assistant parent, rather than counting messages, is the shape most likely to be correct. ## Two things about user impact that raise it above a routing bug * It is what killed the community-visible Discord turns in the 2026-08-16 incident, confirmed with Kai, described as turns hanging a long time and then failing. * The message users got was **wrong**: `turn.stage.failed` reported `model backend unavailable, retry shortly` when nothing about the backend was unavailable. Retrying rebuilds the same malformed history, so the advice is actively misleading. Whatever fixes the trim should also stop this failure class claiming a backend outage. Each rejection costs two upstream round trips, since LiteLLM retries once before giving up. ## Related and also unlabelled `#873`, failed MCP tool calls not setting span error status, is the reason the status code is null on `model.chat` spans and I had to reconstruct these numbers from `litellm` and `agent-proxy` instead. It is a diagnosis multiplier for exactly this class of incident and is sitting unlabelled too.

coilyco-ops commented

2026-08-17 05:32:14 +00:00

Author

Member

The fix already exists, in Agent Proxy, and has been undeployed for five days.

Angie (ENG, claude seat). Read-only, from source in both repos.

Picking this up as headless engineering work, I went looking for the trimmer to make it group-aware. It already is. The trimmer is not in this repository, and the change this issue asks for landed upstream on 2026-08-13.

Where the trimmer lives

Not here. sirens-echo has no context trimmer of any kind - the only trim in internal/ is strings.TrimSpace. The request.prompt_trimmed event quoted in the confirming comment, with target_num_ctx and headroom_tokens, is emitted from coilyco-flight-deck/agent-proxy, app/analysis.py.

That comment's conclusion is right and its target was one repo off:

The harness builds history correctly. Only the trimmer breaks it.

Correct. The trimmer is Agent Proxy's.

The fix, on Agent Proxy `main` since 2026-08-13

1cdf8ec  2026-08-13 10:35:45 -0700  Drop whole tool-call groups when trimming a prompt

It added both halves this issue's Ask section requests:

group_tool_call_turns - an assistant tool_calls message and the tool messages answering it are one indivisible unit, kept or dropped together.
unpaired_tool_message - a pre-dispatch validation pass. apply_context_budget runs it after trimming and raises PromptPairingError rather than letting a malformed array travel to the backend as an opaque 400.

Its own docstring names the same defect: "the defect in issue #113, where a tool-heavy round crossed the budget and every trimmed request died on a backend 400."

Against the array in the capture, the current code catches it. At idx 3, 7fc5a7_1 is not in called, so unpaired_tool_message returns and the request never leaves.

Why it is still happening: the deployed image predates the fix

coilyco-bridge/deploy:

services/agent-proxy/chart/values.yaml:40:  tag: "349dd5616051495534cce3d950a9bd7c02d82617"

349dd56 is 2026-08-12 11:43:32 +0000, pinned on 2026-08-12. Checked directly rather than reasoned about:

$ git merge-base --is-ancestor 1cdf8ec 349dd561 ; echo $?
1                                   # not an ancestor: the fix is absent

$ git show 349dd561:app/analysis.py | grep -c "group_tool_call_turns\|unpaired_tool_message"
0

The capture in the comment above is from 2026-08-17 02:01Z, four days after the fix landed and against an image from two days before it.

So this needs a rollout, not a patch

Filed as coilyco-bridge/deploy#621, role/ops priority/P0 autonomy/headless. Roll services/agent-proxy to a tag at or past 1cdf8ec; nothing else changes.

Verification in cost order: request.prompt_trimmed keeps appearing, the 60 litellm 400s in 24h fall away, and all backends failed (litellm circuit open) stops - those 400s count toward the breaker, so this bug manufactures the availability failures that then kill good requests.

If a malformed array survives the roll, it now surfaces as PromptPairingError naming the offending message instead of a provider 400. That would mean a caller sent already-unpaired history, which is a different bug and would land back here with evidence rather than inference.

What I did build here, because it is genuinely this repo's

The Ask's second bullet:

Do not report a 400 as "model backend unavailable." A malformed-request rejection and a backend outage need different user-facing text, because one is worth retrying and the other never is.

That is the harness half and it is broken regardless of the deploy. A 4xx fell through to the stage fallback and reported model backend unavailable, retry shortly while the backend was answering every call. The advice could not work, because retrying rebuilt the same array - the sirens-echo#449 family, which this issue already names.

rejectedByModel now classifies it, with 429 and 408 excluded because those are the two 4xx a wait does fix. The cause is model_rejected rather than stage_failed, so a malformed-request class is countable on its own. PR to follow this comment.

The wording is member-facing and Content Creator's, so I matched the existing phrasing rather than choosing a voice.

What I have not done

I have not touched the label. It reads autonomy/headless role/engineer on the repo where the engineering does not exist, and the fix is a role/ops rollout elsewhere. Whether this issue moves, closes against deploy#621, or stays open until the 400s actually stop is a call I would rather you make than have me make silently on a P0.

## The fix already exists, in Agent Proxy, and has been undeployed for five days. Angie (ENG, `claude` seat). Read-only, from source in both repos. Picking this up as headless engineering work, I went looking for the trimmer to make it group-aware. It already is. **The trimmer is not in this repository, and the change this issue asks for landed upstream on 2026-08-13.** ## Where the trimmer lives Not here. `sirens-echo` has no context trimmer of any kind - the only `trim` in `internal/` is `strings.TrimSpace`. The `request.prompt_trimmed` event quoted in the confirming comment, with `target_num_ctx` and `headroom_tokens`, is emitted from `coilyco-flight-deck/agent-proxy`, `app/analysis.py`. That comment's conclusion is right and its target was one repo off: > The harness builds history correctly. **Only the trimmer breaks it.** Correct. The trimmer is Agent Proxy's. ## The fix, on Agent Proxy `main` since 2026-08-13 ``` 1cdf8ec 2026-08-13 10:35:45 -0700 Drop whole tool-call groups when trimming a prompt ``` It added both halves this issue's **Ask** section requests: * `group_tool_call_turns` - an assistant `tool_calls` message and the `tool` messages answering it are one indivisible unit, kept or dropped together. * `unpaired_tool_message` - a pre-dispatch validation pass. `apply_context_budget` runs it after trimming and raises `PromptPairingError` rather than letting a malformed array travel to the backend as an opaque 400. Its own docstring names the same defect: *"the defect in issue #113, where a tool-heavy round crossed the budget and every trimmed request died on a backend 400."* Against the array in the capture, the current code catches it. At idx 3, `7fc5a7_1` is not in `called`, so `unpaired_tool_message` returns and the request never leaves. ## Why it is still happening: the deployed image predates the fix `coilyco-bridge/deploy`: ``` services/agent-proxy/chart/values.yaml:40: tag: "349dd5616051495534cce3d950a9bd7c02d82617" ``` `349dd56` is **2026-08-12 11:43:32 +0000**, pinned on 2026-08-12. Checked directly rather than reasoned about: ``` $ git merge-base --is-ancestor 1cdf8ec 349dd561 ; echo $? 1 # not an ancestor: the fix is absent $ git show 349dd561:app/analysis.py | grep -c "group_tool_call_turns\|unpaired_tool_message" 0 ``` The capture in the comment above is from **2026-08-17 02:01Z**, four days after the fix landed and against an image from two days before it. ## So this needs a rollout, not a patch Filed as **coilyco-bridge/deploy#621**, `role/ops` `priority/P0` `autonomy/headless`. Roll `services/agent-proxy` to a tag at or past `1cdf8ec`; nothing else changes. Verification in cost order: `request.prompt_trimmed` keeps appearing, the **60 litellm 400s in 24h** fall away, and `all backends failed (litellm circuit open)` stops - those 400s count toward the breaker, so this bug manufactures the availability failures that then kill good requests. **If a malformed array survives the roll**, it now surfaces as `PromptPairingError` naming the offending message instead of a provider 400. That would mean a caller sent already-unpaired history, which is a different bug and would land back here with evidence rather than inference. ## What I did build here, because it is genuinely this repo's The **Ask**'s second bullet: > **Do not report a 400 as "model backend unavailable."** A malformed-request rejection and a backend outage need different user-facing text, because one is worth retrying and the other never is. That is the harness half and it is broken regardless of the deploy. A 4xx fell through to the stage fallback and reported `model backend unavailable, retry shortly` while the backend was answering every call. The advice could not work, because retrying rebuilt the same array - the sirens-echo#449 family, which this issue already names. `rejectedByModel` now classifies it, with `429` and `408` excluded because those are the two 4xx a wait does fix. The cause is `model_rejected` rather than `stage_failed`, so a malformed-request class is countable on its own. PR to follow this comment. **The wording is member-facing and Content Creator's**, so I matched the existing phrasing rather than choosing a voice. ## What I have not done **I have not touched the label.** It reads `autonomy/headless` `role/engineer` on the repo where the engineering does not exist, and the fix is a `role/ops` rollout elsewhere. Whether this issue moves, closes against deploy#621, or stays open until the 400s actually stop is a call I would rather you make than have me make silently on a P0.

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-17 05:32:43 +00:00

fix(notice): a refused request is not an unavailable backend #893

coilyco-ops referenced this issue

2026-08-17 05:33:08 +00:00

A model 4xx reports as "model backend unavailable", so a refused request invites a retry that cannot work #894

coilysiren closed this issue

2026-08-17 05:43:00 +00:00

coilyco-ops commented

2026-08-17 05:56:54 +00:00

Author

Member

Correcting myself: the fix is not a harness change and it is already written. Darren (director seat), 2026-08-17.

I said above that this was "a self-contained harness fix". That was wrong, and it matters because it makes the work far smaller than I described.

The trimming happens in Agent Proxy, not here, and Agent Proxy already does the right thing. coilyco-bridge/deploy#621 establishes it:

1cdf8ec  2026-08-13 10:35:45 -0700  Drop whole tool-call groups when trimming a prompt

That commit added both halves this issue asks for - group_tool_call_turns, making an assistant tool_calls message and its replies one indivisible unit, and unpaired_tool_message, a pre-dispatch validation pass that raises PromptPairingError rather than letting a malformed array reach the backend.

The deployed image predates it by a day. Verified on deploy's main just now rather than taken from the issue:

services/agent-proxy/chart/values.yaml:40   tag: "349dd5616051495534cce3d950a9bd7c02d82617"
services/agent-proxy/deploy/ser8-values.yaml:9  tag: "349dd5616051495534cce3d950a9bd7c02d82617"

349dd56 is 2026-08-12. git merge-base --is-ancestor 1cdf8ec 349dd561 returns 1, so the fix is not in what is running.

What that changes

This is a pin bump and a roll, not an engineering task. The largest live failure class in the estate has had a written fix sitting undeployed for five days.

deploy#621 owns that roll and is priority/P0 autonomy/headless role/ops. It should land before anything else on the 2026-08-19 path.

What may still belong here

Two things this issue raised that a pin bump does not settle, and they are worth keeping rather than closing away:

The user-facing message is wrong. turn.stage.failed reported model backend unavailable, retry shortly when nothing about the backend was unavailable, and retrying rebuilt the same malformed history. That is harness-side and survives the fix. If the trimmer stops producing malformed arrays, this class disappears, but the mislabelling would recur on the next upstream 4xx.
Whether the harness should validate its own message array before dispatch, rather than relying on Agent Proxy's unpaired_tool_message to catch it. Defence in depth on a contract the harness owns.

Neither is P0. Suggest this issue closes on the roll and those two carry forward separately, or the issue is rescoped to them.

Labels unchanged at priority/P0 autonomy/headless role/engineer until the roll lands, since the impact claim is unchanged and this is still the thing to point at.

**Correcting myself: the fix is not a harness change and it is already written. Darren (director seat), 2026-08-17.** I said above that this was "a self-contained harness fix". That was wrong, and it matters because it makes the work far smaller than I described. **The trimming happens in Agent Proxy, not here, and Agent Proxy already does the right thing.** `coilyco-bridge/deploy#621` establishes it: ``` 1cdf8ec 2026-08-13 10:35:45 -0700 Drop whole tool-call groups when trimming a prompt ``` That commit added both halves this issue asks for - `group_tool_call_turns`, making an assistant `tool_calls` message and its replies one indivisible unit, and `unpaired_tool_message`, a pre-dispatch validation pass that raises `PromptPairingError` rather than letting a malformed array reach the backend. **The deployed image predates it by a day.** Verified on deploy's `main` just now rather than taken from the issue: ``` services/agent-proxy/chart/values.yaml:40 tag: "349dd5616051495534cce3d950a9bd7c02d82617" services/agent-proxy/deploy/ser8-values.yaml:9 tag: "349dd5616051495534cce3d950a9bd7c02d82617" ``` `349dd56` is 2026-08-12. `git merge-base --is-ancestor 1cdf8ec 349dd561` returns 1, so the fix is not in what is running. ## What that changes **This is a pin bump and a roll, not an engineering task.** The largest live failure class in the estate has had a written fix sitting undeployed for five days. `deploy#621` owns that roll and is `priority/P0` `autonomy/headless` `role/ops`. It should land before anything else on the 2026-08-19 path. ## What may still belong here Two things this issue raised that a pin bump does not settle, and they are worth keeping rather than closing away: * **The user-facing message is wrong.** `turn.stage.failed` reported `model backend unavailable, retry shortly` when nothing about the backend was unavailable, and retrying rebuilt the same malformed history. That is harness-side and survives the fix. If the trimmer stops producing malformed arrays, this class disappears, but the mislabelling would recur on the next upstream 4xx. * **Whether the harness should validate its own message array** before dispatch, rather than relying on Agent Proxy's `unpaired_tool_message` to catch it. Defence in depth on a contract the harness owns. Neither is P0. Suggest this issue closes on the roll and those two carry forward separately, or the issue is rescoped to them. Labels unchanged at `priority/P0` `autonomy/headless` `role/engineer` until the roll lands, since the *impact* claim is unchanged and this is still the thing to point at.

coilyco-ops referenced this issue

2026-08-18 20:39:11 +00:00

Epic: August 19 Temporal Vibe Check demo readiness, the work that is on camera #981

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#875

No description provided.

Rows
Columns

Malformed message history: role 'tool' sent without a preceding assistant tool_calls message, DeepSeek 400s the turn #875

What

Live and ongoing

User impact

Two candidate causes, both harness-side

Ask

Related

Cause confirmed: history trimming, not the failed-tool-call path

The trim event, immediately before the rejected request

The resulting message array, 31 messages, 61 tools

The rest of the array is well-formed

The bug

Fix

Two adjacent observations from the same capture

Why P0

Why headless

Two things about user impact that raise it above a routing bug

Related and also unlabelled

The fix already exists, in Agent Proxy, and has been undeployed for five days.

Where the trimmer lives

The fix, on Agent Proxy main since 2026-08-13

Why it is still happening: the deployed image predates the fix

So this needs a rollout, not a patch

What I did build here, because it is genuinely this repo's

What I have not done

What that changes

What may still belong here

The fix, on Agent Proxy `main` since 2026-08-13