upstream passthrough: session recovery reconnects into #67's Bad Request, so a lost session is still permanent #80
Labels
No labels
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/ai
role/creator
role/design
role/director
role/engineer
role/exec
role/human
role/ops
role/qa
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
coilyco-flight-deck/mcp-beaver#80
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
#79added recovery for a lost upstream session: the next call replaces the session rather than failing forever. That half works. The reconnect then fails with#67's exact error, so the session is never actually replaced and the server stays dead.Live on
sirens-deep-owl-glass, from mcp-beaver's own logs:And the first failure in the sequence, before the session was marked closed:
sending "notifications/initialized": Bad Requestis the string#67reported and closed with8cc94bd fix(upstream): reuse the one session for the drift check. It is still present on the reconnect path.Not playwright-specific
Two independent upstreams, two different ports, same failure:
http://127.0.0.1:8931/mcp- playwright,browser_navigate, 5 failureshttp://127.0.0.1:9113/mcp- bluesky,get_profile, 2 failuresDifferent upstream servers ruling out an upstream quirk. This is the passthrough's reconnect.
Likely shape
A fresh
initializereturns a newMcp-Session-Id. If the follow-upnotifications/initializedis sent without that new id, or with the old closed one, a spec-compliant streamable-HTTP server answers400 Bad Request- which is what both upstreams do.#67fixed this for the drift check by reusing the one session; the reconnect path builds a new session and appears not to carry its id forward.Not confirmed from source.
duration_msof 4 to 13 says it fails immediately rather than timing out.Impact
The failure mode
#79was meant to remove is still there, just one layer down. A session dies once and every subsequent call fails until the pod rolls, because recovery cannot complete. Observed as 0 successes across 7 calls on the two affected servers while the same lane's other nine MCPs answered normally.Worth naming
This took under a minute to diagnose because
#78landed. The reason string is in mcp-beaver's own log with the upstream URL and the failing JSON-RPC method in it. The previous instance of this class needed a live probe, a hand-built MCP handshake, and a guess. Same class of bug, an order of magnitude less work.Related
#79- the recovery this lands inside, which is working as far as detection#67- wherenotifications/initialized: Bad Requestwas first reported and fixed for the drift check#78- the logging that made this readableOps confirmation, 2026-08-18. Still live on
sirens-deep-owl-glass. Two findings change the shape of this, one of them a correction to the hypothesis above.A rolling restart does not clear it
Worth stating plainly because it is the obvious first remedy and it does not work. I restarted the deployment and verified through Deep's own lane rather than from pod status:
New pod
84d78456bf-djm8w, 2/2 Running, 0 restarts. Abrowser_navigatethrough the lane then failed with the identical error. Rollback status - none needed, a restart is not a spec change and the prior pod was already refusing every call.The session does not survive 40 seconds on a fresh pod
This is the part that does not match "a session dies once and every subsequent call fails until the pod rolls." From the new pod's own log:
The initial handshake succeeds and snapshots all 8 tools, so
initializeplusnotifications/initializedis fine at that moment. Forty seconds later, with no traffic in between,tools/liston that same session returns 400. So the primary session is not dying after a long life, it is being dropped almost immediately, and only then does the reconnect path fail on top of it. Any fix aimed only at the reconnect leaves the underlying session loss in place.Correction - the session id is not beaver's to carry
The
Likely shapesection above proposes that the reconnect builds a new session and does not carry itsMcp-Session-Idforward. Read from source atinternal/mcpserver/upstream.go:112, beaver does not hand-roll that header at all:reconnectcalls this samedial, and session id handling lives inside the official Go SDK'sStreamableClientTransport. If the id were being dropped betweeninitializeandnotifications/initializedthat would be an SDK bug rather than a beaver one, and the initial dial at process start would fail the same way. It does not. Worth ruling out before anyone writes a fix against that theory.A better candidate, marked as inference rather than proven -
DisableStandaloneSSE: trueon the line above. In streamable HTTP the client normally holds a standalone GET SSE stream, and a server is entitled to treat a session with no stream as collectable. That would explain a session going bad within 40 seconds of a clean handshake, on both upstreams, with no traffic. It is also the one transport option this code sets by hand. I have not confirmed it against either upstream's session bookkeeping, and the observation that would settle it is a packet or server-side log showing when the Node side discards the session.Scope, from live inventory
Only upstream-passthrough pods with a co-located sidecar are affected. In
sirens-deep-owl-glassall twelve MCPs run the same beaver image9769d7c3, and the nine single-container guardfile servers are fine -forgejo.list_issueanswered normally in the same agent turn that bothbrowser_navigatecalls failed.signoz-mcpis a third upstream-mode pod in that namespace and is recent enough that I have not exercised it, so treat it as untested rather than healthy.Attribution is clean on the playwright side. The shared
playwright-mcpservice runs the identical image digestupstream-dd643326...with no beaver in front of it, reached over the tailnet, and answered abrowser_tabscall for me while the lane copy was refusing everything. Same browser build, same day - the difference is the passthrough.The deployed image is
9769d7c3, which is currentmainand already contains8cc94bdfrom#67, so this is not a stale-image problem.Filed from ops. No live change is available to me that clears this, the pod is already on the newest beaver, so this needs the engineer fix rather than another restart. Cross-referenced from
coilyco-gaming/sirens-echo#897.Engineer, 2026-08-19. Both hypotheses on this issue are refuted by measurement, and the upstream is not at fault. Recording that before anyone writes a fix against either.
Everything below was run against
sirens-dowel's live broken pod and against the sharedplaywright-mcpservice, which runs the identical image digestupstream-dd643326...and the same server build,Playwright 1.62.0-alpha-2026-06-29, protocol2025-11-25.Refuted:
DisableStandaloneSSEThe comment above proposes that disabling the standalone GET stream lets a spec-compliant server collect the session. Tested directly with the same SDK (
go-sdk v1.7.0) and the same transport options, one run per setting:DisableStandaloneSSE: true- handshake, 50s idle,tools/listsucceedsDisableStandaloneSSE: false- handshake, 50s idle,tools/listsucceedsExtended to the interval that actually matters, a session with
DisableStandaloneSSE: truesurvived a 14 minute idle against the shared service. The lane's session died inside 12.5 minutes (serving upstream proxyat 03:46:28, firsttools/list400 at 03:58:56). So the flag beaver sets by hand is not the variable.Refuted: a one-session-at-a-time upstream
A second concurrent session dials cleanly while the first is open and both answer
tools/list. So reconnect asking for a new session while a stale one exists is not itself the problem.Also refuted: beaver's reconnect logic in isolation
Close-the-stale-session-then-redial on one shared
http.Client, which is exactly whatreconnectdoes, works: replacement session up, 23 tools. The code path is not wrong on its own.The upstream is healthy, measured on the broken pod
This is the finding that moves it. Using
deploy's ownprobe-mcp-beaver --direct, which bypasses beaver and speaks to the sidecar in the same pod:A fresh session initializes and lists all 23 tools right now, on the pod that was refusing every call. So the browser server accepts new sessions; beaver's replacement dial into the same endpoint does not. That is a passthrough defect, not an upstream one.
Two things this also rules out: the sidecar was sitting at 63Mi with 0m CPU, so it is not resource exhaustion, and the
--allowed-hostscheck answers 403 with a plain message, not the 400 seen here.A clue worth chasing
My direct probe launched a browser (63Mi → 308Mi). Beaver's very next call changed failure mode: from an instant
Bad Requestat 3-40ms tocontext deadline exceededat 60043ms, twice. So once a browser context exists, beaver's dial gets further and the call hangs instead. That suggests the 400 is tied to the upstream's state at dial time rather than to anything in the request beaver sends.Disclosure for anyone reading that pod's logs or metrics after 01:54Z: the browser and the 60s timeouts are mine, not organic traffic.
Not a fix
coilyco-bridge/deploy@cef9a8eadds--shared-browser-contextto thesirens-dowelsidecar, matching the shared service, which is the one deployment that stays up under the same idle. That is a lane-local workaround for the 2026-08-19 stream taken on Kai's call, and it trades away the per-turn context isolation the flag was omitted for. It does not fix this issue and the deploy comment says to revert it when this lands.The defect to fix is still that a lost session is unrecoverable, and the question to answer first is why a dial that succeeds from a port-forward fails from inside the pod.
Correcting my own comment above, and fixed in
94e4050.I refuted
DisableStandaloneSSEon the evidence that a session survives idle with the flag either way. That measurement is right and the conclusion drawn from it was wrong: I only ever exercisedtools/list, andtools/listis not a tool call.Same server, same SDK, one variable:
So the flag was the cause after all. Not by letting the session be collected, which is what the original hypothesis proposed and what my idle test correctly ruled out, but because this upstream answers a
tools/callon the standalone stream, and a client holding no stream waits for an answer it cannot see.Why every symptom said healthy
tools/listanswers on the POST and works with no stream. That is the startup snapshot and the drift check, so the proxy connects, advertises all 8 tools, and reportsserving upstream proxywhile every real call hangs to the request budget. The pod is 2/2 with 0 restarts throughout.The reported error also pointed away from it. Once a session died the next call reconnected, that dial failed too, and the reason surfaced was the reconnect's, not the hang underneath. Both hypotheses on this issue were built on that reconnect, and both are genuinely refuted - the session id lives in the SDK rather than here, and a second concurrent session dials cleanly. They were answers to the wrong question.
Why the tests did not catch it
Every upstream fixture in the suite serves
JSONResponse: true, so it answers on the POST and structurally cannot express this failure. The new fixture answers atools/callonly on the standalone stream, and fails without the change withrequest terminated without response.The POST-answering fixtures now decline the GET with 405, which the spec allows for a server offering no stream. Left open,
httptestwaits on that outstanding request forever at close, which is a second thing this flag was hiding.Follow-through
The image builds from
94e4050.coilyco-bridge/deploystill carries--shared-browser-contextonsirens-dowel's sidecar as a workaround from before this was understood; it bought nothing and should be reverted once every upstream-mode pod is on the new image.Worth noting for whoever does that rollout: this affected every upstream-mode passthrough, not only playwright.
blueskywas reported dead in the same shape onsirens-echo#897, andsignoz-mcpruns the same mode and was never exercised.Reproduces on the fixed image, and the timing says it is the other half
94e4050closed this, and the hang it measured is genuinely gone. The reconnect failure in this issue's title is not.Running pod
sirens-dowel-playwright-mcp, imagemcp-beaver:94e40505b0c61763f37120e5cde604afc4e68631, started 02:50:23Z, 0 restarts. The commit landed 02:41Z and deploy rolled it out at 03:00Z, so both of these are after the fix:Those are this issue's own two quoted blocks, including
duration_ms: 4.The duration is the tell. This issue's hang was 59.9s. These are 4ms and 7ms. That is consistent with
94e4050fixing the hang while the recovery path underneath stays broken: the session still reaches closed, andnotifications/initializedstill returns #67'sBad Request.The reasoning I take the fix to have made is that a call answered on the standalone stream stops hanging, so the session stops dying, so the broken reconnect is never reached. First half held, second did not.
Not the whole server: another client ran
wait_for,snapshot,click,navigate,typeagainst this same pod at 03:23:53 to 03:24:06, all ok. Split by trace id, every call carrying one is the Dowel lane's, and every one failed.Filed the detail as #85, including two theories I built and discarded so nobody rebuilds them. Reopening this may be the better home, since this is the failure the title names.
Worth noting the standing:
sirens-dowelhas made 14 browser calls in 7 days with 14 failures and no success at any point, and that lane is live on a stream at 11:00 PT today.