Every turn makes one failing HTTP 400 call to each MCP endpoint

coilyco-ops commented

2026-08-10 23:25:47 +00:00

Member

Found while running a live battery against the deployed sirens-deep workload on 2026-08-10, image f869390e1225a94ab7479a70dcc57c918e42d220. Predates the merge of #82.

Observation

Error spans from sirens-deep over one hour covering 75 generated turns, grouped by URL and response status:

https://eco-app.coilysiren.me/mcp                    400   75
http://sirens-deep-forgejo-mcp:8080/mcp              400   75
http://sirens-deep-forgejo-mcp:8080/api/list_issue   404   17
http://ser8:8080/v1/chat/completions                 502    7

The first two counts are exactly 75 against 75 turns. That is one failed call per turn per MCP endpoint, against both the Eco MCP and Deep's own Forgejo MCP, on every single turn including the ones that succeed.

The 404 line is tracked separately at #89. The 502 line is the model failure at #86.

Why it is easy to miss

These 400s do not fail the turn. Turns that produce them still return 200 with a good reply, and Deep successfully calls Eco tools in the same turns. So nothing surfaces to a caller and nothing appears in the worker's ERROR logs. The only signal is the span status.

The cost is that sirens-deep reports 174 error spans per hour when only 15 turns actually failed. Any error-rate alarm built on span status will be dominated by this, and the real failures at #86 are a 9% minority of the error spans.

What is not yet known

The 400 spans carry no status_message, so the request that is being rejected is not visible from traces alone. The perfect 1:1 with turn count points at a per-turn lifecycle step rather than a tool call, since tool calls vary per turn and these do not.

Worth checking whether the client issues an initialization, capability negotiation, or teardown request that the ward-mcp runtime rejects, and whether that request is even necessary.

Reproduce

Send any turn to POST /v1/turn, then query traces for service.name = 'sirens-deep' AND has_error = true. Two 400 spans appear per turn regardless of the turn's outcome.

Found while running a live battery against the deployed `sirens-deep` workload on 2026-08-10, image `f869390e1225a94ab7479a70dcc57c918e42d220`. Predates the merge of #82. ## Observation Error spans from `sirens-deep` over one hour covering 75 generated turns, grouped by URL and response status: ``` https://eco-app.coilysiren.me/mcp 400 75 http://sirens-deep-forgejo-mcp:8080/mcp 400 75 http://sirens-deep-forgejo-mcp:8080/api/list_issue 404 17 http://ser8:8080/v1/chat/completions 502 7 ``` The first two counts are **exactly 75 against 75 turns**. That is one failed call per turn per MCP endpoint, against both the Eco MCP and Deep's own Forgejo MCP, on every single turn including the ones that succeed. The 404 line is tracked separately at #89. The 502 line is the model failure at #86. ## Why it is easy to miss These 400s do not fail the turn. Turns that produce them still return 200 with a good reply, and Deep successfully calls Eco tools in the same turns. So nothing surfaces to a caller and nothing appears in the worker's ERROR logs. The only signal is the span status. The cost is that `sirens-deep` reports 174 error spans per hour when only 15 turns actually failed. Any error-rate alarm built on span status will be dominated by this, and the real failures at #86 are a 9% minority of the error spans. ## What is not yet known The 400 spans carry no `status_message`, so the request that is being rejected is not visible from traces alone. The perfect 1:1 with turn count points at a per-turn lifecycle step rather than a tool call, since tool calls vary per turn and these do not. Worth checking whether the client issues an initialization, capability negotiation, or teardown request that the `ward-mcp` runtime rejects, and whether that request is even necessary. ## Reproduce Send any turn to `POST /v1/turn`, then query traces for `service.name = 'sirens-deep' AND has_error = true`. Two 400 spans appear per turn regardless of the turn's outcome.

coilyco-ops referenced this issue from a commit

2026-08-10 23:29:47 +00:00

Add an error spans panel to the Sirens Deep console

coilyco-ops referenced this issue

2026-08-11 04:30:00 +00:00

Add operational resilience tooling #71

coilyco-ops referenced this issue

2026-08-11 06:48:38 +00:00

fix: attribute the harness to the real ingress and make Forgejo failures diagnosable #101

coilyco-ops referenced this issue

2026-08-11 08:07:48 +00:00

Drop the issue field from the response envelope and let the model use the Forgejo tool it already has #102

coilyco-ops commented

2026-08-11 09:41:06 +00:00

Author

Member

Investigated against current main (e0b3dc1). I cannot reproduce this, and the leading hypothesis in the issue is ruled out.

The DELETE theory is wrong

The obvious candidate was the MCP session-termination DELETE on Close(). It fits the symptoms well: once per session per turn, and streamableClientConn.Close never inspects the response status, so a rejected DELETE would leave closeErr nil and never reach the mcp.session.close.failed log while still recording a 400 span.

It is still wrong. ward-mcp builds its handler with Stateless: true (internal/mcpserver/server.go:213), so clients receive no Mcp-Session-Id. The client only sends the DELETE when SessionID() != "" (go-sdk v1.7.0 mcp/streamable.go:2726). No session id, no DELETE.

Direct reproduction

I stood up a stateless mcp.NewStreamableHTTPHandler with the same options ward-mcp uses, put a status-recording middleware in front, and drove it with the real MCPProvider.Open / Call / Close. Full request log for one turn:

POST accept="application/json, text/event-stream" -> 200   (initialize)
POST accept="application/json, text/event-stream" -> 200   (tools/list)
POST accept="application/json, text/event-stream" -> 200   (tools/call)

Three POSTs, no fourth request, no DELETE, no 400. Both sides are go-sdk v1.7.0.

For completeness, the 400 paths that do exist in serveStateless are the Accept check requiring both application/json and text/event-stream, and "no server available". The client sends both media types on every request, so neither fires here.

What I think is going on

The issue records the evidence as coming from image f869390e, and notes it predates #82. Two PRs have landed on the client since. The behavior is most likely already gone, or belongs to a server version that is no longer deployed.

Suggested next step

Re-measure against a current image before any code change. The query in the issue still works:

service.name = 'sirens-deep' AND has_error = true

If the 1:1 pattern is still there on a current deployment, the next thing to capture is the request method and path on those 400 spans, because the client-side lifecycle on main has no candidate left. I would rather leave this open and unfixed than ship a speculative change against a signature I could not reproduce.

Investigated against current `main` (`e0b3dc1`). **I cannot reproduce this, and the leading hypothesis in the issue is ruled out.** ## The DELETE theory is wrong The obvious candidate was the MCP session-termination `DELETE` on `Close()`. It fits the symptoms well: once per session per turn, and `streamableClientConn.Close` never inspects the response status, so a rejected DELETE would leave `closeErr` nil and never reach the `mcp.session.close.failed` log while still recording a 400 span. It is still wrong. `ward-mcp` builds its handler with `Stateless: true` (`internal/mcpserver/server.go:213`), so clients receive no `Mcp-Session-Id`. The client only sends the DELETE when `SessionID() != ""` (go-sdk v1.7.0 `mcp/streamable.go:2726`). No session id, no DELETE. ## Direct reproduction I stood up a stateless `mcp.NewStreamableHTTPHandler` with the same options ward-mcp uses, put a status-recording middleware in front, and drove it with the real `MCPProvider.Open` / `Call` / `Close`. Full request log for one turn: ``` POST accept="application/json, text/event-stream" -> 200 (initialize) POST accept="application/json, text/event-stream" -> 200 (tools/list) POST accept="application/json, text/event-stream" -> 200 (tools/call) ``` Three POSTs, no fourth request, no DELETE, no 400. Both sides are go-sdk v1.7.0. For completeness, the 400 paths that do exist in `serveStateless` are the Accept check requiring both `application/json` and `text/event-stream`, and "no server available". The client sends both media types on every request, so neither fires here. ## What I think is going on The issue records the evidence as coming from image `f869390e`, and notes it predates https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/82. Two PRs have landed on the client since. The behavior is most likely already gone, or belongs to a server version that is no longer deployed. ## Suggested next step Re-measure against a current image before any code change. The query in the issue still works: ``` service.name = 'sirens-deep' AND has_error = true ``` If the 1:1 pattern is still there on a current deployment, the next thing to capture is the request method and path on those 400 spans, because the client-side lifecycle on `main` has no candidate left. I would rather leave this open and unfixed than ship a speculative change against a signature I could not reproduce.