coilyco-flight-deck/agent-proxy

Watch

Every check agent-proxy makes is structured in its logs and prose in its responses, so no caller can react to any of them #125

New issue

Open

opened 2026-08-13 18:30:22 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-13 18:30:22 +00:00

Member

🤖 Filed by Claude Code on Kai's behalf.

Raised by Kai after a day of tracing: "aproxy and litellm have a bunch of boundaries and checks that they aren't communicating to sirens echo in a structured way." This is the agent-proxy half. The cross-cutting version is a tracker in coilyco-gaming/sirens-echo.

Restating #108's own framing — "aproxy doesn't know how to communicate that limitation / state" — with the specific mechanism: the state is already known and already structured. It goes to SigNoz. It does not go into the response.

The asymmetry

agent-proxy emits richly typed telemetry for every decision it makes:

{"event": "request.prompt_trimmed", "logical_model": "sirens-echo/deepseek",
 "original_token_count": 48226, "final_token_count": 45985,
 "budget_tokens": 47104, "target_num_ctx": 48128,
 "headroom_tokens": 1024, "dropped_message_count": 5}

{"event": "dispatch.transport_error", "backend": "litellm", "attempt": 0,
 "error": "litellm: Client error '400 Bad Request' for url '...'"}

Six typed fields describing a payload mutation. An attempt counter. A backend name. All of it queryable, none of it in the HTTP response.

What the caller got for that same request:

502  {"error": {"message": "sirens-echo/deepseek: all backends failed (litellm:
     Client error '400 Bad Request' for url 'http://100.105.159.124:4000/
     v1/chat/completions'\nFor more information check: https://developer.
     mozilla.org/en-US/docs/Web/HTTP/Status/400)", "type": "upstream_error"}}

A status code and a sentence with an MDN link in it. Nothing a program can branch on.

Structure is destroyed at each hop

DeepSeek answered with a typed object:

{"error": {"message": "Messages with role 'tool' must be a response to a
 preceding message with 'tool_calls'", "type": "invalid_request_error",
 "param": null, "code": "invalid_request_error"}}

litellm preserved it — inside a Python traceback string.
agent-proxy wrapped that string in AllBackendsFailed: ... and returned 502.
sirens-echo turned 502 into error_type: model_failed.
The member read model backend unavailable, retry shortly.

Four hops. A machine-readable code at hop zero; prose pointing at the wrong system by hop four. Every hop had the information and every hop dropped its shape.

What callers currently cannot determine

Question the caller must answer	Available today
Was my prompt modified before dispatch?	no — silent, even on 200
Was this my fault or the backend's?	no — both are 502
Is retrying worth anything?	no — inferred from a string
How many attempts were already spent?	no
Which backend served or failed?	no (#109)
Was a validator the cause? (#137)	only inside a `status_message`
Am I queued or is it hung?	no (#104)

Six of those seven are already known inside agent-proxy at the moment it answers.

The trimming case is the sharpest

request.prompt_trimmed fires on the request path and the caller is never told. In every observed instance the request then failed (#113), so the silence has been academic — but the design is that agent-proxy can silently rewrite the message array a caller constructed and return 200, and the caller would attribute any resulting weirdness to the model. A caller cannot reason about a conversation it does not know was edited.

Proposal

A typed envelope alongside the existing prose, on both paths.

Failure:

{"error": {
  "type": "upstream_error",
  "message": "<unchanged prose, for humans>",
  "agentproxy": {
    "reason": "upstream_invalid_request",
    "stage": "upstream",
    "retryable": false,
    "attempts": 3,
    "upstream_status": 400,
    "upstream_code": "invalid_request_error",
    "upstream_message": "Messages with role 'tool' must be a response to...",
    "backend": "litellm"
  }}}

Success, when the request was altered:

{"agentproxy": {"request_modified": {
   "prompt_trimmed": {"dropped_messages": 5,
                      "original_tokens": 48226, "final_tokens": 45985,
                      "budget_tokens": 47104}}}}

reason should be a closed enumeration, documented, and stable — that is the whole value. retryable should be stated by the proxy, which knows, rather than guessed by the caller from a string.

This composes with #104 rather than competing: heartbeats carry in-flight state on streaming requests, this carries terminal state and applies to non-streaming too.

What I am not claiming

That this fixes the failures. #113, #114 and #115 are real defects and stay real. This makes them legible to the caller instead of only to me in SigNoz.
That the prose should go. Keep it. Humans read it. Add structure beside it.
A specific schema. The shape above is illustrative. The load-bearing parts are: a stable enumerated reason, honest retryable, upstream status/code preserved, and request mutations disclosed.
That agent-proxy is uniquely at fault. litellm loses structure too, and sirens-echo does this to itself (coilyco-gaming/sirens-echo#651, where a harness-internal check rejected two correct answers and logged only attempt: 1). agent-proxy is the highest-leverage place to start because it sits in the middle and holds the most checks.

Acceptance

Error responses carry a stable, enumerated, documented reason code.
Upstream status and error code are propagated, not collapsed into 502.
retryable is asserted by the proxy.
A request mutated before dispatch says so in the response, including on 200.
AllBackendsFailed appears only when more than one backend was attempted (also #114).
A caller can distinguish saturated backend / invalid request / validator rejection / timeout without parsing prose.

#108 — Kai's original statement of the problem; this is its response-path half.
#104 — SSE heartbeats; in-flight state, complementary.
#106 — upstream status not recorded on the span; the same loss, observability side.
#109 — backend identity missing from spans; one of the seven unanswerable questions.
#113, #114, #115 — the three defects from 2026-08-13 that this would have made self-diagnosing.
coilyco-gaming/sirens-echo#137 — a validator rejection reaching the caller as 502.
coilyco-gaming/sirens-echo#651 — the same disease inside the harness.
The cross-cutting tracker in coilyco-gaming/sirens-echo, filed alongside this.

Next owner

Engineer, after the schema call.

🤖 Filed by Claude Code on Kai's behalf.

> 🤖 Filed by Claude Code on Kai's behalf. Raised by Kai after a day of tracing: *"aproxy and litellm have a bunch of boundaries and checks that they aren't communicating to sirens echo in a structured way."* This is the agent-proxy half. The cross-cutting version is a tracker in `coilyco-gaming/sirens-echo`. Restating #108's own framing — *"aproxy doesn't know how to communicate that limitation / state"* — with the specific mechanism: **the state is already known and already structured. It goes to SigNoz. It does not go into the response.** ## The asymmetry agent-proxy emits richly typed telemetry for every decision it makes: ```json {"event": "request.prompt_trimmed", "logical_model": "sirens-echo/deepseek", "original_token_count": 48226, "final_token_count": 45985, "budget_tokens": 47104, "target_num_ctx": 48128, "headroom_tokens": 1024, "dropped_message_count": 5} ``` ```json {"event": "dispatch.transport_error", "backend": "litellm", "attempt": 0, "error": "litellm: Client error '400 Bad Request' for url '...'"} ``` Six typed fields describing a payload mutation. An attempt counter. A backend name. All of it queryable, none of it in the HTTP response. What the caller got for that same request: ``` 502 {"error": {"message": "sirens-echo/deepseek: all backends failed (litellm: Client error '400 Bad Request' for url 'http://100.105.159.124:4000/ v1/chat/completions'\nFor more information check: https://developer. mozilla.org/en-US/docs/Web/HTTP/Status/400)", "type": "upstream_error"}} ``` A status code and a sentence with an MDN link in it. Nothing a program can branch on. ## Structure is destroyed at each hop DeepSeek answered with a typed object: ```json {"error": {"message": "Messages with role 'tool' must be a response to a preceding message with 'tool_calls'", "type": "invalid_request_error", "param": null, "code": "invalid_request_error"}} ``` - **litellm** preserved it — inside a Python traceback string. - **agent-proxy** wrapped that string in `AllBackendsFailed: ...` and returned **502**. - **sirens-echo** turned 502 into `error_type: model_failed`. - **The member** read `model backend unavailable, retry shortly`. Four hops. A machine-readable `code` at hop zero; prose pointing at the wrong system by hop four. Every hop had the information and every hop dropped its shape. ## What callers currently cannot determine | Question the caller must answer | Available today | | --- | --- | | Was my prompt modified before dispatch? | no — silent, even on 200 | | Was this my fault or the backend's? | no — both are 502 | | Is retrying worth anything? | no — inferred from a string | | How many attempts were already spent? | no | | Which backend served or failed? | no (#109) | | Was a validator the cause? (#137) | only inside a `status_message` | | Am I queued or is it hung? | no (#104) | Six of those seven are already known inside agent-proxy at the moment it answers. ## The trimming case is the sharpest `request.prompt_trimmed` fires on the request path and the caller is never told. In every observed instance the request then failed (#113), so the silence has been academic — but the design is that agent-proxy can **silently rewrite the message array a caller constructed** and return 200, and the caller would attribute any resulting weirdness to the model. A caller cannot reason about a conversation it does not know was edited. ## Proposal A typed envelope alongside the existing prose, on both paths. Failure: ```json {"error": { "type": "upstream_error", "message": "<unchanged prose, for humans>", "agentproxy": { "reason": "upstream_invalid_request", "stage": "upstream", "retryable": false, "attempts": 3, "upstream_status": 400, "upstream_code": "invalid_request_error", "upstream_message": "Messages with role 'tool' must be a response to...", "backend": "litellm" }}} ``` Success, when the request was altered: ```json {"agentproxy": {"request_modified": { "prompt_trimmed": {"dropped_messages": 5, "original_tokens": 48226, "final_tokens": 45985, "budget_tokens": 47104}}}} ``` `reason` should be a closed enumeration, documented, and stable — that is the whole value. `retryable` should be **stated by the proxy**, which knows, rather than guessed by the caller from a string. This composes with #104 rather than competing: heartbeats carry in-flight state on streaming requests, this carries terminal state and applies to non-streaming too. ## What I am not claiming - **That this fixes the failures.** #113, #114 and #115 are real defects and stay real. This makes them legible to the caller instead of only to me in SigNoz. - **That the prose should go.** Keep it. Humans read it. Add structure beside it. - **A specific schema.** The shape above is illustrative. The load-bearing parts are: a stable enumerated `reason`, honest `retryable`, upstream status/code preserved, and request mutations disclosed. - **That agent-proxy is uniquely at fault.** litellm loses structure too, and sirens-echo does this to itself (`coilyco-gaming/sirens-echo#651`, where a harness-internal check rejected two correct answers and logged only `attempt: 1`). agent-proxy is the highest-leverage place to start because it sits in the middle and holds the most checks. ## Acceptance - Error responses carry a stable, enumerated, documented `reason` code. - Upstream status and error code are propagated, not collapsed into 502. - `retryable` is asserted by the proxy. - A request mutated before dispatch says so in the response, including on 200. - `AllBackendsFailed` appears only when more than one backend was attempted (also #114). - A caller can distinguish saturated backend / invalid request / validator rejection / timeout without parsing prose. ## Related - #108 — Kai's original statement of the problem; this is its response-path half. - #104 — SSE heartbeats; in-flight state, complementary. - #106 — upstream status not recorded on the span; the same loss, observability side. - #109 — backend identity missing from spans; one of the seven unanswerable questions. - #113, #114, #115 — the three defects from 2026-08-13 that this would have made self-diagnosing. - `coilyco-gaming/sirens-echo#137` — a validator rejection reaching the caller as 502. - `coilyco-gaming/sirens-echo#651` — the same disease inside the harness. - The cross-cutting tracker in `coilyco-gaming/sirens-echo`, filed alongside this. ## Next owner Engineer, after the schema call. > 🤖 Filed by Claude Code on Kai's behalf.

coilyco-ops added the

autonomy

async-consult

label

2026-08-13 18:31:07 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy#125

No description provided.

Rows
Columns