coilyco-flight-deck/mcp-beaver

Watch

sirens-deep-forgejo-mcp: request hangs exactly 180s, then connection closes #49

New issue

Closed

opened 2026-08-12 09:40:08 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-12 09:40:08 +00:00

Member

Symptom

A single MCP request to sirens-deep-forgejo-mcp hung for 180 seconds and then failed on a closed connection. The calling turn had already abandoned it.

Evidence

Trace 883ba8968cd370c32d678168dbad68b5, 2026-08-12T07:57:41Z (SigNoz: http://ser8:30808/trace/<trace_id>):

field	value
span	`HTTP POST`
url	`http://sirens-deep-forgejo-mcp:8080/mcp`
duration	`180002440885` ns = 180.002s
http status	`200`
span status	`Error`
status message	`read tcp 10.42.0.147:35628->10.43.160.192:8080: use of closed network connection`
parent_span_id	(empty — root span)

Two things stand out:

180.002s is a hard timeout constant, not load. The value is too exact to be contention.
The span is orphaned — no parent. The turn that issued it had already ended, so this request outlived its own caller and could never have produced a response.

10.42.0.147 is the sirens-deep pod; 10.43.160.192:8080 is the forgejo MCP service.

Impact

This is the main contributor to sirens-deep's p99 of 120.5s over 36 calls (11.1% error rate). It also produces user-visible silence in Discord, separately from the ungrounded_action_claim failures tracked in coilyco-gaming/sirens-echo.

Pod state rules out a crash

At the time of failure, sirens-deep-forgejo-mcp-78d5f7fcbf-xwlqw was Running with restart_count: 0, up 11h37m (ward-mcp:97a100274a72a281b11440af89a9f7638c2754cb). The sirens-deep pod was also Running, restart_count: 0, up since 07:35:59Z. Neither process died — the request hung inside healthy pods.

Open questions

Which side owns the 180s? Determine whether ward-mcp stopped responding or the client abandoned the read.
Which tool call was in flight? The span carries no MCP method attribute, so the hanging operation is unidentified. If ward-mcp does not attribute the MCP method onto its spans, that is worth fixing on its own — it is the difference between "an MCP call hung" and "this tool hung."
Does the caller need a shorter timeout regardless? 180s far exceeds any Discord interaction budget.

Acceptance

A hanging upstream produces a bounded error well under 180s.
The span identifies the MCP method in flight.
No orphaned root spans from abandoned MCP requests.

Next owner

Engineer.

## Symptom A single MCP request to `sirens-deep-forgejo-mcp` hung for 180 seconds and then failed on a closed connection. The calling turn had already abandoned it. ## Evidence Trace `883ba8968cd370c32d678168dbad68b5`, 2026-08-12T07:57:41Z (SigNoz: `http://ser8:30808/trace/<trace_id>`): | field | value | | --- | --- | | span | `HTTP POST` | | url | `http://sirens-deep-forgejo-mcp:8080/mcp` | | duration | `180002440885` ns = **180.002s** | | http status | `200` | | span status | `Error` | | status message | `read tcp 10.42.0.147:35628->10.43.160.192:8080: use of closed network connection` | | parent_span_id | *(empty — root span)* | Two things stand out: 1. **180.002s is a hard timeout constant, not load.** The value is too exact to be contention. 2. **The span is orphaned** — no parent. The turn that issued it had already ended, so this request outlived its own caller and could never have produced a response. `10.42.0.147` is the `sirens-deep` pod; `10.43.160.192:8080` is the forgejo MCP service. ## Impact This is the main contributor to `sirens-deep`'s p99 of 120.5s over 36 calls (11.1% error rate). It also produces user-visible silence in Discord, separately from the `ungrounded_action_claim` failures tracked in `coilyco-gaming/sirens-echo`. ## Pod state rules out a crash At the time of failure, `sirens-deep-forgejo-mcp-78d5f7fcbf-xwlqw` was `Running` with `restart_count: 0`, up 11h37m (`ward-mcp:97a100274a72a281b11440af89a9f7638c2754cb`). The `sirens-deep` pod was also `Running`, `restart_count: 0`, up since 07:35:59Z. Neither process died — the request hung inside healthy pods. ## Open questions 1. Which side owns the 180s? Determine whether ward-mcp stopped responding or the client abandoned the read. 2. Which tool call was in flight? The span carries no MCP method attribute, so the hanging operation is unidentified. If ward-mcp does not attribute the MCP method onto its spans, that is worth fixing on its own — it is the difference between "an MCP call hung" and "this tool hung." 3. Does the caller need a shorter timeout regardless? 180s far exceeds any Discord interaction budget. ## Acceptance - A hanging upstream produces a bounded error well under 180s. - The span identifies the MCP method in flight. - No orphaned root spans from abandoned MCP requests. ## Next owner Engineer.

coilyco-ops referenced this issue

2026-08-14 02:15:33 +00:00

First-class PDF processing: a large amount of authoritative reference material exists only as PDF #60

coilyco-ops referenced this issue

2026-08-14 04:33:39 +00:00

server-info should be on by default — the disclosure argument for opt-in does not survive reading the payload #61

coilyco-ops referenced this issue from a commit

2026-08-14 05:21:39 +00:00

fix: bound a request end to end and name the method in flight

coilyco-ops closed this issue

2026-08-14 05:21:39 +00:00

coilyco-ops referenced this issue from a commit

2026-08-14 05:24:08 +00:00

feat: add an outbound rate limit for public-good upstreams

coilyco-ops commented

2026-08-14 05:38:33 +00:00

Author

Member

Fixed in 719795b. Answering the three open questions from the runtime rather than from the trace, because reading the source turned up more than the span did.

Question 1 - who owned the 180s. Neither side owned it deliberately. Three things were unbounded at once:

http.Server was constructed with no timeouts at all - no read-header, no idle, nothing.
Proxy mode passed a nil HTTP client, which resolves to http.DefaultClient, which has no timeout. Spec mode was already bounded by opcore's 30s default client, which is why this shape is easy to miss.
The SDK's StreamableHTTPHandler does not tie handler work to the originating HTTP request unless PropagateRequestCancellation is set, and it was not.

That third one is the direct explanation for the orphaned root span. The caller had gone away and the work kept running, because nothing connected the two. It was not a stray span - it was an accurate picture of a request that genuinely outlived its caller.

Question 3 - yes, and it is now --request-timeout, default 60s. Applied at the tool handler rather than only at the edge, deliberately: the SDK propagates cancellation only for >= 2026-07-28 clients, so a bound written purely at the transport would look correct and do nothing for an older client - which is the case that hung.

Question 2 - the method is now on the transport span. mcp.method.name, and gen_ai.tool.name where there is one, are stamped on the enclosing span before dispatch, so a request that never returns still names what was in flight. Both values are already bounded to closed sets, so this adds no cardinality a caller can drive.

Two things that cost a probe to find, both now pinned by tests, because both would have shipped as false green:

The first version of the hang test passed in microseconds - the auth token was unset, so the call failed before any request was made. It asserted nothing. It now sets the token and asserts against a threshold far below opcore's 30s fallback, so an ineffective bound fails loudly.
Setting the transport and per-call deadlines to the same value made them expire on the same tick, and the request context died mid-serialization. The caller got an empty body - indistinguishable from a crashed pod, which is the confusion this issue started in. The transport deadline now carries five seconds of headroom so the tool is always what expires first and the runtime has room to say so.

/healthz is exempt from every deadline: a liveness probe a wedged upstream can fail turns one slow dependency into a restart loop.

Acceptance:

A hanging upstream produces a bounded error well under 180s
The span identifies the MCP method in flight
No orphaned root spans from abandoned MCP requests - for >= 2026-07-28 clients, via PropagateRequestCancellation. Older clients cannot be told the caller left, so for them the per-call deadline is the bound, and the span ends at 60s rather than never.

Worth noting for the deploy side: 60s is a runtime default, not a Discord-appropriate one. sirens-deep should probably set --request-timeout well below that.

Fixed in 719795b. Answering the three open questions from the runtime rather than from the trace, because reading the source turned up more than the span did. **Question 1 - who owned the 180s.** Neither side owned it deliberately. Three things were unbounded at once: * `http.Server` was constructed with no timeouts at all - no read-header, no idle, nothing. * Proxy mode passed a **nil** HTTP client, which resolves to `http.DefaultClient`, which has no timeout. Spec mode was already bounded by opcore's 30s default client, which is why this shape is easy to miss. * The SDK's `StreamableHTTPHandler` does **not** tie handler work to the originating HTTP request unless `PropagateRequestCancellation` is set, and it was not. That third one is the direct explanation for the orphaned root span. The caller had gone away and the work kept running, because nothing connected the two. It was not a stray span - it was an accurate picture of a request that genuinely outlived its caller. **Question 3 - yes, and it is now `--request-timeout`, default 60s.** Applied at the tool handler rather than only at the edge, deliberately: the SDK propagates cancellation only for >= 2026-07-28 clients, so a bound written purely at the transport would look correct and do nothing for an older client - which is the case that hung. **Question 2 - the method is now on the transport span.** `mcp.method.name`, and `gen_ai.tool.name` where there is one, are stamped on the enclosing span *before* dispatch, so a request that never returns still names what was in flight. Both values are already bounded to closed sets, so this adds no cardinality a caller can drive. **Two things that cost a probe to find**, both now pinned by tests, because both would have shipped as false green: 1. The first version of the hang test passed in microseconds - the auth token was unset, so the call failed before any request was made. It asserted nothing. It now sets the token and asserts against a threshold far below opcore's 30s fallback, so an ineffective bound fails loudly. 2. Setting the transport and per-call deadlines to the same value made them expire on the same tick, and the request context died mid-serialization. The caller got an **empty body** - indistinguishable from a crashed pod, which is the confusion this issue started in. The transport deadline now carries five seconds of headroom so the tool is always what expires first and the runtime has room to say so. `/healthz` is exempt from every deadline: a liveness probe a wedged upstream can fail turns one slow dependency into a restart loop. **Acceptance:** - [x] A hanging upstream produces a bounded error well under 180s - [x] The span identifies the MCP method in flight - [x] No orphaned root spans from abandoned MCP requests - for >= 2026-07-28 clients, via `PropagateRequestCancellation`. Older clients cannot be told the caller left, so for them the per-call deadline is the bound, and the span ends at 60s rather than never. Worth noting for the deploy side: 60s is a runtime default, not a Discord-appropriate one. `sirens-deep` should probably set `--request-timeout` well below that.

coilyco-ops referenced this issue

2026-08-14 05:38:48 +00:00

No client-side rate limiting, which is a stated shared requirement for five servers about to be built on this runtime #57

coilyco-ops referenced this issue

2026-08-14 05:40:42 +00:00

First-class PDF processing: a large amount of authoritative reference material exists only as PDF #60

coilyco-ops referenced this issue

2026-08-14 05:40:48 +00:00

code-fleet-watch: policy findings #37

coilyco-ops referenced this issue

2026-08-17 01:21:03 +00:00

First-class PDF processing: a large amount of authoritative reference material exists only as PDF #60