Every pod shutdown ends in run.failed: HTTP shutdown deadline exceeded #986

New issue

Open

opened 2026-08-18 22:03:38 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-18 22:03:38 +00:00

Member

Found while verifying the deploy#682 rollout. Pre-existing and not caused by that change, but nothing appears to be tracking it.

What happens

Every sirens pod that is replaced logs, at ERROR, on its way out:

{"level":"ERROR","msg":"run.failed","error":"HTTP shutdown: context deadline exceeded"}

50 occurrences in SigNoz over the last three days, on all three lanes, oldest at 2026-08-17T18:13:47Z. That predates the coalescing lane entirely, so this is the shutdown path as it has been rather than something the lane introduced. One pod also logged shutdown.telemetry.failed on a metrics upload that could not reach the collector inside the same window.

Where it comes from

drainTurns in internal/community/shutdown.go runs httpServer.Shutdown(shutdownCtx) under a context bounded by SIRENS_ECHO_SHUTDOWN_GRACE, and returns fmt.Errorf("HTTP shutdown: %w", httpErr) when that deadline passes. The error reaches main as a failed run, so an ordinary rollout looks like a crash in the logs.

The likely cause is an idle keep-alive connection that Shutdown waits on rather than a turn still running: the drain logs shutdown.drained with turns_settled separately, and that is the field that actually says whether work was lost.

Why it is worth a look

A restart during the August 19 stream would read as a failure to anyone watching the logs, and would take the whole grace before the container exits.
An ERROR on every single restart is the shape that teaches a reader to skip the row that matters, the same argument sirens-echo#929 makes about a permanently red security row.

Not established

Whether any turn is actually lost, or whether this is purely the idle-connection wait. turns_settled in the neighbouring shutdown.drained line would settle it and I did not read it per-restart.
How far back it really goes. Three days is the window I looked at, not the age of the defect.

Found while verifying the deploy#682 rollout. **Pre-existing and not caused by that change**, but nothing appears to be tracking it. ## What happens Every sirens pod that is replaced logs, at ERROR, on its way out: ``` {"level":"ERROR","msg":"run.failed","error":"HTTP shutdown: context deadline exceeded"} ``` 50 occurrences in SigNoz over the last three days, on all three lanes, **oldest at 2026-08-17T18:13:47Z**. That predates the coalescing lane entirely, so this is the shutdown path as it has been rather than something the lane introduced. One pod also logged `shutdown.telemetry.failed` on a metrics upload that could not reach the collector inside the same window. ## Where it comes from `drainTurns` in `internal/community/shutdown.go` runs `httpServer.Shutdown(shutdownCtx)` under a context bounded by `SIRENS_ECHO_SHUTDOWN_GRACE`, and returns `fmt.Errorf("HTTP shutdown: %w", httpErr)` when that deadline passes. The error reaches `main` as a failed run, so **an ordinary rollout looks like a crash in the logs**. The likely cause is an idle keep-alive connection that `Shutdown` waits on rather than a turn still running: the drain logs `shutdown.drained` with `turns_settled` separately, and that is the field that actually says whether work was lost. ## Why it is worth a look * **A restart during the August 19 stream would read as a failure** to anyone watching the logs, and would take the whole grace before the container exits. * An ERROR on every single restart is the shape that teaches a reader to skip the row that matters, the same argument sirens-echo#929 makes about a permanently red security row. ## Not established * Whether any turn is actually lost, or whether this is purely the idle-connection wait. `turns_settled` in the neighbouring `shutdown.drained` line would settle it and I did not read it per-restart. * How far back it really goes. Three days is the window I looked at, not the age of the defect.

coilyco-ops referenced this issue

2026-08-18 22:07:43 +00:00

Testing the coalescing lane owes: a seam for the Discord adapter, one measured rate, and six live observations #988

coilyco-ops added the

labels

2026-08-19 03:38:46 +00:00