coilyco-gaming/sirens-echo

Watch

A turn interrupted by a roll vanishes silently: mark it in-progress at start, sweep it at boot #989

New issue

Closed

opened 2026-08-18 22:18:40 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-18 22:18:40 +00:00

Member

Kai's idea, sharpened one step. Filed from the ops seat with the evidence rather than built, since this is harness behavior.

The gap

A lane roll kills whatever turn is running, and nobody learns. The Gateway event was already consumed so nothing redelivers it, and the person who summoned the agent simply never gets a reply.

This is the same shape as deploy#464, which fixed it for jobs:

Unset ran MemoryJobStore, so under the Recreate strategy above every job submitted before an image roll stopped existing: not failed, not resumed, not reported.

The Postgres store closed that for jobs. A turn is not a job. The README is explicit that a job "outlives the turn that created it", so the turn stays in-process and dies with the pod.

Why the loss is structural, not tunable

Measured against the deployed config, not assumed:

strategy: Recreate, and deploy's rationale file states it as a constraint rather than a preference: two pods both open an unsharded Gateway connection on one bot token, so every summon is answered twice. There is deliberately zero overlap.
terminationGracePeriodSeconds: 30, no preStop hook, no lifecycle block.
p95 community.turn is 182.2s, the number deploy#669 set the 5m request timeout against.

So a graceful drain could only rescue turns finishing inside 30 seconds, against a typical turn taking six times that. Raising the grace period is not the fix, and the useful goal is making the loss visible rather than preventing it.

The shape worth building

Not a SIGTERM-time write. That makes correctness depend on the least reliable moment in the process's life, and it catches only the graceful case.

Instead:

Mark in-progress when the turn starts, keyed by the Discord message id, with the channel, author, start time, and the instance identity of the writing process.
Clear it when the turn completes, success or handled failure.
Sweep at boot. Any record still in-progress that was not written by this process was interrupted.
The SIGTERM handler becomes an optimization, upgrading interrupted to interrupted-during-shutdown when it gets the chance. Nothing depends on it landing.

That ordering catches SIGKILL past the grace period, OOM kills, node failure, and ordinary crashes, none of which a shutdown hook sees.

Why the store can be trusted here

sirens-dowel-job-store is its own Deployment and was not rolled by any of today's lane rolls: 19h uptime while the lane pod cycled repeatedly. So the database is reachable both from a dying pod and from a booting one, which is the property an in-pod file store would not have. That was already Kai's reasoning on deploy#464 for Postgres over a volume.

Cost is one row write per turn on a path whose p95 is 182 seconds.

What to do with a swept record, in preference order

Report it. Log it, count it, and let it show up in telemetry. Cheapest, and it alone converts a silent drop into a known one, which is the whole complaint.
Notify the channel. On boot, post once per interrupted turn saying the summon was dropped by a restart. The room stops waiting on a reply that is never coming.
Do not auto-replay. Re-running the turn risks the double-answer the Recreate constraint exists to prevent, and since coilyco-bridge/deploy#683 the Moxn surface on the dowel lane carries documents, edit, branches, and comments, each with delete. A replayed turn can repeat a destructive tool call. If replay is ever wanted it needs idempotency at the tool layer first, which is a much larger piece of work.

Acceptance

A roll during an in-flight turn leaves exactly one record marked interrupted, naming the message id and channel.
The next boot finds it, reports it, and does not re-answer.
A turn that completes normally leaves no record behind.
A SIGKILL past the grace period produces the same interrupted record as a graceful SIGTERM, since nothing depends on the shutdown path.

Not urgent before the 2026-08-19 stream

The practical mitigation for the stream is simply not to merge under services/sirens-echo/** while it is live, since CD auto-rolls the lane on that path. scripts/ changes and Moxn content writes do not roll it.

Refs coilyco-bridge/deploy#464, coilyco-bridge/deploy#669, coilyco-bridge/deploy#683

Kai's idea, sharpened one step. Filed from the ops seat with the evidence rather than built, since this is harness behavior. ## The gap A lane roll kills whatever turn is running, and nobody learns. The Gateway event was already consumed so nothing redelivers it, and the person who summoned the agent simply never gets a reply. This is the same shape as deploy#464, which fixed it for **jobs**: > Unset ran MemoryJobStore, so under the Recreate strategy above every job submitted before an image roll stopped existing: not failed, not resumed, not reported. The Postgres store closed that for jobs. A **turn** is not a job. The README is explicit that a job "outlives the turn that created it", so the turn stays in-process and dies with the pod. ## Why the loss is structural, not tunable Measured against the deployed config, not assumed: * `strategy: Recreate`, and deploy's rationale file states it as a constraint rather than a preference: two pods both open an unsharded Gateway connection on one bot token, so every summon is answered twice. There is deliberately zero overlap. * `terminationGracePeriodSeconds: 30`, no `preStop` hook, no lifecycle block. * p95 `community.turn` is **182.2s**, the number deploy#669 set the 5m request timeout against. So a graceful drain could only rescue turns finishing inside 30 seconds, against a typical turn taking six times that. **Raising the grace period is not the fix**, and the useful goal is making the loss visible rather than preventing it. ## The shape worth building **Not** a SIGTERM-time write. That makes correctness depend on the least reliable moment in the process's life, and it catches only the graceful case. Instead: 1. **Mark in-progress when the turn starts**, keyed by the Discord message id, with the channel, author, start time, and the instance identity of the writing process. 2. **Clear it when the turn completes**, success or handled failure. 3. **Sweep at boot.** Any record still in-progress that was not written by this process was interrupted. 4. **The SIGTERM handler becomes an optimization**, upgrading `interrupted` to `interrupted-during-shutdown` when it gets the chance. Nothing depends on it landing. That ordering catches SIGKILL past the grace period, OOM kills, node failure, and ordinary crashes, none of which a shutdown hook sees. ## Why the store can be trusted here `sirens-dowel-job-store` is its own Deployment and was not rolled by any of today's lane rolls: 19h uptime while the lane pod cycled repeatedly. So the database is reachable both from a dying pod and from a booting one, which is the property an in-pod file store would not have. That was already Kai's reasoning on deploy#464 for Postgres over a volume. Cost is one row write per turn on a path whose p95 is 182 seconds. ## What to do with a swept record, in preference order 1. **Report it.** Log it, count it, and let it show up in telemetry. Cheapest, and it alone converts a silent drop into a known one, which is the whole complaint. 2. **Notify the channel.** On boot, post once per interrupted turn saying the summon was dropped by a restart. The room stops waiting on a reply that is never coming. 3. **Do not auto-replay.** Re-running the turn risks the double-answer the `Recreate` constraint exists to prevent, and since coilyco-bridge/deploy#683 the Moxn surface on the dowel lane carries `documents`, `edit`, `branches`, and `comments`, each with `delete`. A replayed turn can repeat a destructive tool call. If replay is ever wanted it needs idempotency at the tool layer first, which is a much larger piece of work. ## Acceptance * A roll during an in-flight turn leaves exactly one record marked interrupted, naming the message id and channel. * The next boot finds it, reports it, and does not re-answer. * A turn that completes normally leaves no record behind. * A SIGKILL past the grace period produces the same interrupted record as a graceful SIGTERM, since nothing depends on the shutdown path. ## Not urgent before the 2026-08-19 stream The practical mitigation for the stream is simply not to merge under `services/sirens-echo/**` while it is live, since CD auto-rolls the lane on that path. `scripts/` changes and Moxn content writes do not roll it. Refs coilyco-bridge/deploy#464, coilyco-bridge/deploy#669, coilyco-bridge/deploy#683

coilyco-ops referenced this issue

2026-08-18 22:43:52 +00:00

Give the harness an execution slot pool, target 8, so a lane stops answering one summon at a time #995

coilyco-ops added the

labels

2026-08-19 03:38:46 +00:00

coilyco-ops referenced this issue

2026-08-22 20:55:26 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops added this to the Headless engineer lane: the queue wait first milestone

2026-08-22 20:58:55 +00:00

coilyco-ops commented

2026-08-22 21:23:04 +00:00

Author

Member

Ordering call, because #995 and #1095 disagree about which of these goes first

Darren (director seat), 2026-08-22. #995 is being worked right now as milestone 17's first item, and its body says of this issue:

The interrupted-turn marker in #989 and this pool touch the same execution path and are worth doing in that order, or at least together.

That reads as this issue first. #1095 ranks #995 first instead, on the #1010 load-test measurement rather than on a label. Both are right about different things, and the conflict is only apparent, because they are about different moments.

#995's harness change is code. It sizes the execution slots from a knob instead of the literal 1, and merging it changes nothing that is running.
The risk this issue names arrives at rollout. A roll currently drops one in-flight turn silently. At a widened pool it drops up to eight, still silently, and that is #995's own observation about its own change.

The decision

#995 lands first, as #1095 ranks it. The measurement stands, the work is on the runSerialized path, and holding it behind this issue buys nothing while it sits unmerged.
This issue gates the deployed slot count, not the merge. The ops seat should not roll the lane to the widened pool until an interrupted turn leaves a record, because the blast radius of the silent drop scales with the pool and the whole point of the pool is to have eight turns in flight.
The engineer working #995 carries this into the ops handover, alongside SIRENS_ECHO_QUEUE_TIMEOUT, the pod limits, and the Agent Proxy capture-buffer measurement. A knob that ships at 8 and rolls at 8 before this exists is the one sequence to avoid.

Nothing here changes this issue's shape. The build is exactly as filed, and the preference order for a swept record stands, including the refusal to auto-replay.

## Ordering call, because #995 and #1095 disagree about which of these goes first **Darren (director seat), 2026-08-22.** #995 is being worked right now as milestone 17's first item, and its body says of this issue: > The interrupted-turn marker in #989 and this pool touch the same execution path and are worth doing in that order, or at least together. That reads as this issue first. #1095 ranks #995 first instead, on the #1010 load-test measurement rather than on a label. **Both are right about different things, and the conflict is only apparent, because they are about different moments.** * **#995's harness change is code.** It sizes the execution slots from a knob instead of the literal 1, and merging it changes nothing that is running. * **The risk this issue names arrives at rollout.** A roll currently drops one in-flight turn silently. At a widened pool it drops up to eight, still silently, and that is #995's own observation about its own change. ## The decision 1. **#995 lands first, as #1095 ranks it.** The measurement stands, the work is on the `runSerialized` path, and holding it behind this issue buys nothing while it sits unmerged. 2. **This issue gates the deployed slot count, not the merge.** The ops seat should not roll the lane to the widened pool until an interrupted turn leaves a record, because the blast radius of the silent drop scales with the pool and the whole point of the pool is to have eight turns in flight. 3. **The engineer working #995 carries this into the ops handover**, alongside `SIRENS_ECHO_QUEUE_TIMEOUT`, the pod limits, and the Agent Proxy capture-buffer measurement. A knob that ships at 8 and rolls at 8 before this exists is the one sequence to avoid. Nothing here changes this issue's shape. The build is exactly as filed, and the preference order for a swept record stands, including the refusal to auto-replay.

coilyco-ops referenced this issue

2026-08-22 21:23:12 +00:00

Give the harness an execution slot pool, target 8, so a lane stops answering one summon at a time #995

coilyco-ops referenced this issue from a commit

2026-08-22 21:36:06 +00:00

feat(admission): run eight turns at once instead of one

coilyco-ops referenced this issue from a commit

2026-08-22 21:53:36 +00:00

feat(turns): report the turns a restart took, instead of losing them

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-22 21:53:59 +00:00

feat(turns): report the turns a restart took, instead of losing them #1102