feat(shutdown): wait for the turns a restart would otherwise cut #612

Merged

coilyco-ops merged 1 commit from coilyco-ops/a-restart-is-not-a-failure into main

2026-08-13 17:11:09 +00:00

AGit

coilyco-ops commented

2026-08-13 17:09:28 +00:00

Member

closes #597 - a Discord turn was rooted at context.Background(), so shutdown could not see it, wait for it, or tell it to stop. It ended when the process did, which the member read as silence.

coilyco-ops added 1 commit

2026-08-13 17:09:28 +00:00

feat(shutdown): wait for the turns a restart would otherwise cut

ci / image-build (pull_request) Successful in 22s

Details

ci / test (pull_request) Successful in 34s

Details

ci / publish-echo-image (pull_request) Has been skipped

Details

ci / publish-observed (pull_request) Has been skipped

Details

9c25cdf160

SIGTERM cancelled the context handed to Run, which reached the HTTP listener and
nothing else. Every Discord turn was rooted at context.Background(), so shutdown
could not see one, wait for one, or tell one to stop. The turn ended when the
process did, and for the member that read as silence: the message kept its
accepted mark and never got an answer.

HTTP turns were never affected. They descend from their request, so
http.Server.Shutdown already waited for them. This is Discord only.

Discord turns now descend from a root the service owns and a counter tracks the
ones in flight. A restart stops admitting, waits out the grace for the turns
already running, cancels what is left, and only then returns to the deferred
closes of the session, the MCP connections, and the job runner. Those closes
were always last. What was missing is everything above them, so they ran while
turns were still executing.

A cancelled turn sees context.Canceled, which is what every other cancellation
also looks like, so the notice is chosen from context.Cause instead. A drained
turn reads "service restarting, retry shortly" rather than blaming the model
backend it was in the middle of calling, and the failure metric splits on
shutdown rather than stage_failed so a rollout does not read as an outage.

The grace is 15s by default, under SIRENS_ECHO_SHUTDOWN_GRACE. No manifest sets
terminationGracePeriodSeconds, so Kubernetes' 30s default is the ceiling. It is
deliberately shorter than the 3m RequestTimeout, because a turn allowed to run
that long cannot be waited out by any value that fits the window.

One behaviour change worth expecting: the HTTP drain was a hardcoded 5s and now
shares the grace, so a slow in-flight HTTP turn gets longer than it used to.

The env var is documented in the new shutdown doc rather than beside its two
sibling timeouts, because sirens-echo-admission.md is at the 80-line cap and
sirens-echo-config.md is at the 4000-char cap. Both rejected a two line
addition. Recorded on sirens-echo#537.

closes #597

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-13 17:09:28 +00:00

coilyco-ops referenced this pull request

2026-08-13 17:09:52 +00:00

gracefully end active turns on SIGTERM #597

coilyco-ops referenced this pull request

2026-08-13 17:10:06 +00:00

Thirteen docs remain at the 80-line cap with no clean subject seam #537

coilyco-ops referenced this pull request

2026-08-13 17:10:43 +00:00

@mentions in replies should surface the replied-to message as primary subject, not channel head #579

coilyco-ops merged commit 7a1849c6ed into main