TestRefreshIsIdempotentUntilTheNextTurn failed in CI on an unreachable fixture server, and Open discards the reason so nothing says why #809

New issue

Closed

opened 2026-08-15 16:07:02 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-15 16:07:02 +00:00

Member

Found while getting #803 green. Unrelated to that change, which touches only agent.go, replyfailure.go, and two docs.

The failure

Run 20603, job test:

--- FAIL: TestRefreshIsIdempotentUntilTheNextTurn (0.00s)
    mcp_refresh_test.go:129: Open: no configured MCP server is reachable

Line 129 is the first provider.Open, against the httptest server countingRosterServer had just started in the same test. It failed in 0.00s, so nothing timed out: mcpConnectTimeout is 10s and mcpListTimeout is 15s. The connect returned immediately.

Not reproduced

Locally, on the same commit:

the single test, -count=50
the package, six runs
the package, eight runs at GOMAXPROCS=2 -parallel 2
the package, twelve runs at GOMAXPROCS=2 -parallel 16 -race

All green, no race warnings. CI itself is green on the merge of main into that branch (run 20623). So this is a real intermittent, not a deterministic break, and rerunning hid it rather than fixing it.

Why it cannot be diagnosed from the failure

Open collects each unreachable server by name only and throws the error away:

if err != nil {
    opened.unavailable = append(opened.unavailable, entry.definition.Name)
    continue
}
...
if len(p.entries) > 0 && len(opened.unavailable) == len(p.entries) {
    return nil, fmt.Errorf("no configured MCP server is reachable")
}

readyLocked produces a specific error for every branch - backing off, connect failed, tools/list failed, resources/list failed, prompts/list failed - and none of them survives. The discovery span records mcp.discovery.stage but never the error, and under go test there is no exporter anyway. Downstream proxy.go logs unavailable_servers as a count.

So the observable evidence cannot distinguish connection-refused from a backoff from a failed listing. That is the same shape as #795, where the sentence a validator wrote was generated and discarded, and it is what makes this flake unfixable from CI output rather than merely unfixed.

Suggested order

Carry the reason. Join the per-entry errors into the message Open returns, or record each on its discovery span. Cheap, and it makes the next occurrence self-diagnosing instead of costing another investigation. Worth doing whether or not the flake is ever seen again.
Then reproduce. With the reason in hand the next CI failure names the branch, and the fix follows from which one it was.

Acceptance

A failure to reach every rostered server says why, per server, without opening the source.
The next occurrence of this test failure identifies which branch of readyLocked produced it.

Found while getting #803 green. Unrelated to that change, which touches only `agent.go`, `replyfailure.go`, and two docs. ## The failure Run 20603, job `test`: ``` --- FAIL: TestRefreshIsIdempotentUntilTheNextTurn (0.00s) mcp_refresh_test.go:129: Open: no configured MCP server is reachable ``` Line 129 is the **first** `provider.Open`, against the `httptest` server `countingRosterServer` had just started in the same test. It failed in 0.00s, so nothing timed out: `mcpConnectTimeout` is 10s and `mcpListTimeout` is 15s. The connect returned immediately. ## Not reproduced Locally, on the same commit: * the single test, `-count=50` * the package, six runs * the package, eight runs at `GOMAXPROCS=2 -parallel 2` * the package, twelve runs at `GOMAXPROCS=2 -parallel 16 -race` All green, no race warnings. CI itself is green on the merge of `main` into that branch (run 20623). So this is a real intermittent, not a deterministic break, and rerunning hid it rather than fixing it. ## Why it cannot be diagnosed from the failure `Open` collects each unreachable server by **name only** and throws the error away: ```go if err != nil { opened.unavailable = append(opened.unavailable, entry.definition.Name) continue } ... if len(p.entries) > 0 && len(opened.unavailable) == len(p.entries) { return nil, fmt.Errorf("no configured MCP server is reachable") } ``` `readyLocked` produces a specific error for every branch - backing off, connect failed, `tools/list` failed, `resources/list` failed, `prompts/list` failed - and none of them survives. The discovery span records `mcp.discovery.stage` but never the error, and under `go test` there is no exporter anyway. Downstream `proxy.go` logs `unavailable_servers` as a count. So the observable evidence cannot distinguish connection-refused from a backoff from a failed listing. That is the same shape as #795, where the sentence a validator wrote was generated and discarded, and it is what makes this flake unfixable from CI output rather than merely unfixed. ## Suggested order 1. **Carry the reason.** Join the per-entry errors into the message `Open` returns, or record each on its discovery span. Cheap, and it makes the next occurrence self-diagnosing instead of costing another investigation. Worth doing whether or not the flake is ever seen again. 2. **Then reproduce.** With the reason in hand the next CI failure names the branch, and the fix follows from which one it was. ## Acceptance * A failure to reach every rostered server says why, per server, without opening the source. * The next occurrence of this test failure identifies which branch of `readyLocked` produced it.

coilyco-ops referenced this issue

2026-08-15 16:07:43 +00:00

fix(telemetry): a turn that never sent is not a Discord verdict #803

coilyco-ops added the

autonomy

headless

priority

labels

2026-08-15 17:22:50 +00:00

coilyco-ops added the

role/engineer

label