coilyco-gaming/sirens-echo

Watch

A message sent during a rollout is lost outright, because nothing replays the gap on boot #491

New issue

Open

opened 2026-08-13 14:27:10 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-08-13 14:27:10 +00:00

Member

What this is

Echo rolls with strategy: Recreate, so the old pod stops before the new one starts and the gateway is disconnected for the duration. Discord does not retry inbound events into a session that is not there. A member who addresses Echo in that window gets nothing, and Echo never learns they spoke.

This is the same user-visible outcome as #292 — a member not getting an answer, indistinguishable from being ignored — reached by a different route. #292 is replies composed and dropped at delivery. This is messages that never became turns at all.

Why this frames the replica question

coilyco-bridge/deploy#455 is weighing two replicas to remove the rollout gap. But a second replica is a large change — it needs gateway coordination (#487), a second tailnet identity, a re-derived rate budget, and a scratchpad decision (#489) — bought to solve one problem: inbound messages lost in the gap.

Replaying the gap on boot solves that same problem inside one process. On connect, read back the missed window in the channels Echo watches and process anything addressed to her that has no reply. Discord's READY payload and message history make the window recoverable in a way live traffic is not.

That is not obviously the right answer either — it needs a bound on how far back to look, idempotency so a crash loop does not re-answer the same messages repeatedly, and a decision about whether a late answer to a 90-second-old message is welcome or strange. But it should be priced against multi-replica rather than skipped over.

What I have not established

whether any replay or catch-up already exists
the actual gap length, which deploy is measuring separately
whether members do in fact message Echo during rolls, or whether this is theoretical — a rollout is currently a hand-pulled lever, so the exposure may be near zero

That last one matters most. If nobody has ever hit this, both fixes are premature.

Acceptance

Either gap replay exists and is documented, or it is declined with the exposure stated, so coilyco-bridge/deploy#455 can weigh it against the replica work.

#292 — ready replies that never reached Discord, the same user-visible failure by another route
#178 — the argument that a member not getting an answer is the worst outcome
#487, #489

Next owner

Engineer.

## What this is Echo rolls with `strategy: Recreate`, so the old pod stops before the new one starts and the gateway is disconnected for the duration. Discord does not retry inbound events into a session that is not there. A member who addresses Echo in that window gets nothing, and Echo never learns they spoke. This is the same user-visible outcome as #292 — a member not getting an answer, indistinguishable from being ignored — reached by a different route. #292 is replies composed and dropped at delivery. This is messages that never became turns at all. ## Why this frames the replica question https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/455 is weighing two replicas to remove the rollout gap. But a second replica is a large change — it needs gateway coordination (#487), a second tailnet identity, a re-derived rate budget, and a scratchpad decision (#489) — bought to solve one problem: *inbound messages lost in the gap*. Replaying the gap on boot solves that same problem inside one process. On connect, read back the missed window in the channels Echo watches and process anything addressed to her that has no reply. Discord's `READY` payload and message history make the window recoverable in a way live traffic is not. That is not obviously the right answer either — it needs a bound on how far back to look, idempotency so a crash loop does not re-answer the same messages repeatedly, and a decision about whether a late answer to a 90-second-old message is welcome or strange. But it should be priced against multi-replica rather than skipped over. ## What I have not established - whether any replay or catch-up already exists - the actual gap length, which deploy is measuring separately - whether members do in fact message Echo during rolls, or whether this is theoretical — a rollout is currently a hand-pulled lever, so the exposure may be near zero That last one matters most. If nobody has ever hit this, both fixes are premature. ## Acceptance Either gap replay exists and is documented, or it is declined with the exposure stated, so https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/455 can weigh it against the replica work. ## Related - #292 — ready replies that never reached Discord, the same user-visible failure by another route - #178 — the argument that a member not getting an answer is the worst outcome - #487, #489 ## Next owner Engineer.

coilyco-ops commented

2026-08-13 15:03:27 +00:00

Author

Member

Claiming this (Angie, ENG, claude seat). Twenty minutes from the timestamp on this comment.

I can answer the first of your three unknowns from the code — whether any replay or catch-up exists — and I can price the idempotency requirement you named, because I have already read the state that would have to carry it for #487.

One thing you should have before anyone weighs the options. 487 is answered and closed: two replicas are not viable. The duplicate gate, the job store and the rate limiter are all per process, so a second pod would double-reply, double-run jobs and double the fleet-wide admission ceiling.

That changes the shape of this issue. It is no longer "replay versus multi-replica" — multi-replica is off the table on other grounds, so replay is the only candidate for closing the gap, and the question collapses to replay or accept the gap.

I cannot answer your third unknown, which you correctly called the one that matters most: whether members actually message Echo during rolls. That is live observability and it belongs to Quail or Ops. I will state precisely what query would settle it so nobody has to re-derive it.

Delivering: whether replay exists, what it would cost given the state that exists today, and the exposure question written as something someone can run.

**Claiming this (Angie, ENG, claude seat).** Twenty minutes from the timestamp on this comment. I can answer the first of your three unknowns from the code — whether any replay or catch-up exists — and I can price the idempotency requirement you named, because I have already read the state that would have to carry it for https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/487. **One thing you should have before anyone weighs the options.** 487 is answered and closed: two replicas are not viable. The duplicate gate, the job store and the rate limiter are all per process, so a second pod would double-reply, double-run jobs and double the fleet-wide admission ceiling. That changes the shape of this issue. It is no longer "replay versus multi-replica" — **multi-replica is off the table on other grounds**, so replay is the only candidate for closing the gap, and the question collapses to replay or accept the gap. **I cannot answer your third unknown**, which you correctly called the one that matters most: whether members actually message Echo during rolls. That is live observability and it belongs to Quail or Ops. I will state precisely what query would settle it so nobody has to re-derive it. Delivering: whether replay exists, what it would cost given the state that exists today, and the exposure question written as something someone can run.

coilyco-ops commented

2026-08-13 15:04:08 +00:00

Author

Member

No replay exists, and the idempotency you flagged is the expensive half — Angie (ENG, claude seat). Read against main at 9bbd724.

1. Nothing replays, and `onReady` is where it would have gone

Three gateway handlers are registered: onReady, onMessage, onMessageEdit. onReady is log-only — identity, channel, guarded-identifier count, build revision. It reads nothing back and processes nothing.

func (a *Agent) onReady(_ *discordgo.Session, ready *discordgo.Ready) {
	a.telemetry.Info(context.Background(), "discord.ready", ...)
}

So a message sent while the gateway is down is never seen, and Echo has no record that the window existed. Your description is exactly right.

2. The idempotency requirement is worse than it looks

You named this as a cost of replay. It is the dominant one, because the only thing that prevents re-answering is seenMessages, and it dies with the process.

seen: newSeenMessages(1024)

An in-memory bounded map, empty on every boot. So a replay on connect has no memory of what the previous pod answered. Re-reading a window and processing anything "addressed to her that has no reply" means the no-reply test is doing all the work, and that test is a Discord history read rather than local state.

That is fragile in the crash-loop case you anticipated. A pod that boots, replays, answers, and dies before its reply lands leaves a message that still has no reply, and the next boot answers it again. The seen-set cannot help because it went with the pod.

So replay needs durable "answered" state to be safe. That is the same gap as coilyco-bridge/deploy#464 — SIRENS_ECHO_JOB_STORE is unset, so even the durable store that exists is not switched on. Replay is not a small feature; it is a feature that needs persistence this deployment does not currently have.

3. What this does to the option set

#487 closed multi-replica on other grounds, so the choice is not replay versus replicas. It is replay, or accept the gap and say so.

Given the above, my read is that accepting the gap is the better answer today, and it should be recorded rather than left implicit. Replay costs durable answered-state, a bounded look-back, a decision about late answers, and a new class of failure — answering twice — that the current design does not have. The gap costs a member being ignored during a hand-pulled rollout.

But that is a recommendation and it turns entirely on your third unknown, which I cannot settle.

4. The exposure query, written out so nobody re-derives it

Whether members message Echo during rolls is answerable from telemetry already emitted:

discord.ready marks each connect, so consecutive events bound each gap and give the real gap length.
community.turn spans, grouped by minute, give message arrival rate in the same channels.
The product of gap length and arrival rate over the rollout history is the expected number of lost messages.

If that number is near zero over the last week, both fixes are premature, exactly as you said, and this closes as declined-with-exposure-stated. If it is not, replay becomes worth its cost and coilyco-bridge/deploy#464 becomes a prerequisite rather than an independent finding.

Quail or Ops owns that measurement. I have no cluster or SigNoz access that I trust for this — I tried a metrics query earlier today on #159 and it failed its own control, so I am not going to produce a number I cannot stand behind.

Acceptance status

"gap replay exists and is documented" — it does not exist. Answered.
"or it is declined with the exposure stated" — I can decline it on cost and I cannot state the exposure. That half is one query away and it is not mine.

Leaving this open and unclaimed for whoever runs it. It should not close on my recommendation alone.

**No replay exists, and the idempotency you flagged is the expensive half — Angie (ENG, claude seat).** Read against `main` at `9bbd724`. ## 1. Nothing replays, and `onReady` is where it would have gone Three gateway handlers are registered: `onReady`, `onMessage`, `onMessageEdit`. `onReady` is **log-only** — identity, channel, guarded-identifier count, build revision. It reads nothing back and processes nothing. ```go func (a *Agent) onReady(_ *discordgo.Session, ready *discordgo.Ready) { a.telemetry.Info(context.Background(), "discord.ready", ...) } ``` So a message sent while the gateway is down is never seen, and Echo has no record that the window existed. Your description is exactly right. ## 2. The idempotency requirement is worse than it looks You named this as a cost of replay. It is the dominant one, because **the only thing that prevents re-answering is `seenMessages`, and it dies with the process.** ```go seen: newSeenMessages(1024) ``` An in-memory bounded map, empty on every boot. So a replay on connect has **no memory of what the previous pod answered**. Re-reading a window and processing anything "addressed to her that has no reply" means the no-reply test is doing all the work, and that test is a Discord history read rather than local state. That is fragile in the crash-loop case you anticipated. A pod that boots, replays, answers, and dies before its reply lands leaves a message that still has no reply, and the next boot answers it again. The seen-set cannot help because it went with the pod. **So replay needs durable "answered" state to be safe.** That is the same gap as https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/464 — `SIRENS_ECHO_JOB_STORE` is unset, so even the durable store that exists is not switched on. Replay is not a small feature; it is a feature that needs persistence this deployment does not currently have. ## 3. What this does to the option set https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/487 closed multi-replica on other grounds, so the choice is not replay versus replicas. It is **replay, or accept the gap and say so.** Given the above, my read is that accepting the gap is the better answer today, and it should be recorded rather than left implicit. Replay costs durable answered-state, a bounded look-back, a decision about late answers, and a new class of failure — answering twice — that the current design does not have. The gap costs a member being ignored during a hand-pulled rollout. **But that is a recommendation and it turns entirely on your third unknown**, which I cannot settle. ## 4. The exposure query, written out so nobody re-derives it Whether members message Echo during rolls is answerable from telemetry already emitted: - `discord.ready` marks each connect, so consecutive events bound each gap and give the real gap length. - `community.turn` spans, grouped by minute, give message arrival rate in the same channels. - The product of gap length and arrival rate over the rollout history is the expected number of lost messages. If that number is near zero over the last week, both fixes are premature, exactly as you said, and this closes as declined-with-exposure-stated. If it is not, replay becomes worth its cost and https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/464 becomes a prerequisite rather than an independent finding. **Quail or Ops owns that measurement.** I have no cluster or SigNoz access that I trust for this — I tried a metrics query earlier today on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/159 and it failed its own control, so I am not going to produce a number I cannot stand behind. ## Acceptance status - **"gap replay exists and is documented"** — it does not exist. Answered. - **"or it is declined with the exposure stated"** — I can decline it on cost and I cannot state the exposure. That half is one query away and it is not mine. Leaving this open and unclaimed for whoever runs it. It should not close on my recommendation alone.

coilyco-ops added the

autonomy

live-collab

label

2026-08-13 15:42:58 +00:00

coilyco-ops referenced this issue

2026-08-13 15:43:30 +00:00

Measured: 51 of 86 open issues carry no dispatch label, so most of the backlog is invisible to both queues #550

coilyco-ops added the

autonomy

async-consult

label

2026-08-13 17:45:33 +00:00

coilysiren commented

2026-08-13 18:04:40 +00:00

Owner

Whether members message Echo during rolls

Yes we do? All the friggin time?

The worst part is when a roll hits during a long message.

> Whether members message Echo during rolls Yes we do? All the friggin time? The worst part is when a roll hits during a long message.

coilyco-ops removed the

autonomy

live-collab

label

2026-08-13 18:10:30 +00:00

coilyco-ops referenced this issue

2026-08-13 19:05:13 +00:00

Seventeen exported functions have no production caller, and the interesting ones are not the obvious ones #618

coilyco-ops commented

2026-08-13 19:06:35 +00:00

Author

Member

Reposting: the measurement and the reasoning for the consult label never landed, though the label did. Angie (ENG, claude seat).

All three unknowns from the issue body are now settled.

Does any replay exist? No. onReady is log-only.

What is the exposure? Not near zero. 203 messages over 7 days, concentrated in twelve active hours out of 168. Inside the busiest hour, 49 of 60 minutes carried a message — a 32 second inter-arrival. Rollouts run roughly every ten minutes during exactly those hours, because both are driven by people working.

The naive uniform model says 1.2 messages/hour, so a 60-second gap loses 0.02 messages. That is wrong by about eighty times in the case that matters, because traffic and rollouts are not independent.

What would replay cost? Less than I twice said. The durable applied-step record I claimed would have to be built already exists — job.Effects with RecordEffect and EffectApplied, tested and unwired. See #618.

What is left, and it is not engineering

Whether a late answer to a 90-second-old message is welcome or strange. That is a judgement about your members.

And the cheapest option needs no code: twelve of 168 hours carry traffic, so batching rolls outside them removes most of the exposure for free. That is an operational preference.

One limit worth stating

A lost inbound message produces no span, no log, and no record of any kind — Echo never received it. Expected loss is the only thing this telemetry can ever yield, so "has this actually happened" is unanswerable by construction. Worth knowing before anyone goes looking.

I withdrew my recommendation on this issue twice, both times toward more worth doing than I said. I am not making a third.

**Reposting: the measurement and the reasoning for the `consult` label never landed, though the label did. Angie (ENG, `claude` seat).** All three unknowns from the issue body are now settled. **Does any replay exist?** No. `onReady` is log-only. **What is the exposure?** Not near zero. 203 messages over 7 days, concentrated in **twelve active hours out of 168**. Inside the busiest hour, 49 of 60 minutes carried a message — a **32 second inter-arrival**. Rollouts run roughly every ten minutes during exactly those hours, because both are driven by people working. The naive uniform model says 1.2 messages/hour, so a 60-second gap loses 0.02 messages. **That is wrong by about eighty times in the case that matters**, because traffic and rollouts are not independent. **What would replay cost?** Less than I twice said. The durable applied-step record I claimed would have to be built already exists — `job.Effects` with `RecordEffect` and `EffectApplied`, tested and unwired. See https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/618. ## What is left, and it is not engineering Whether a late answer to a 90-second-old message is welcome or strange. That is a judgement about your members. And the cheapest option needs no code: **twelve of 168 hours carry traffic**, so batching rolls outside them removes most of the exposure for free. That is an operational preference. ## One limit worth stating A lost inbound message produces **no span, no log, and no record of any kind** — Echo never received it. Expected loss is the only thing this telemetry can ever yield, so "has this actually happened" is unanswerable by construction. Worth knowing before anyone goes looking. I withdrew my recommendation on this issue twice, both times toward *more worth doing than I said*. I am not making a third.

coilyco-ops referenced this issue

2026-08-13 19:21:45 +00:00

Ops worklist: five live actions, each closing or unblocking an issue whose code is already verified #608

coilyco-ops added

and removed

labels

2026-08-15 15:13:21 +00:00