coilyco-gaming/sirens-echo

Watch

Two harness processes on one bot token would double-deliver every Discord event, and nothing coordinates them #487

New issue

Closed

opened 2026-08-13 14:26:17 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-13 14:26:17 +00:00

Member

What this is

Deploy is considering a second Echo replica so an image roll causes no downtime — coilyco-bridge/deploy#455. This is the question that decides it, and it is upstream.

Echo holds a Discord gateway connection. Two processes running the same DISCORD_TOKEN both IDENTIFY as shard 0 of 1. Discord fans every MESSAGE_CREATE to both sessions — it does not dedupe across connections, which is what sharding exists to solve — and DMs always land on shard 0 regardless of shard count. SIRENS_ECHO_DISCORD_DM_ENABLED is true on both lanes.

Absent coordination in the harness, the visible result is Echo answering every message twice, in a guild of ~111 members.

What I have not established

I did not read this repository. The investigation ran against the deploy surface only, so everything above is inference from configuration, not verification of the code. Specifically unverified:

whether the harness has any leader election, gateway-event dedup, or message_id idempotency
whether two processes sharing the job store would already deduplicate as a side effect of claiming a job
whether the gateway client tolerates a second session at all, or whether Discord's session-start limits make this fail differently than described

Confirming or refuting these is the whole of this issue.

Why deploy cannot answer it

services/sirens-echo/deploy/values.yaml sets strategy: Recreate, added in deploy commit ffb3c37 titled "preserve Echo gateway rollout strategy". That commit extended the shared ingress-tailscale chart with a strategy field for this one service. The existing deployment therefore already treats overlapping gateway connections as the thing to avoid — but it records the avoidance, not the reason, and the reason lives here.

Acceptance

a stated answer to whether two harness processes can hold one bot token without double-replying
if no: this is recorded as the reason Echo stays single-replica, and coilyco-bridge/deploy#455 closes against it
if yes: the mechanism is named and testable, so deploy can verify it rather than trust it

Scope

Applies to both lanes. Deep's exposure is smaller — DMs from one allowlisted account — but the mechanism is identical and the image is shared.

Next owner

Engineer.

## What this is Deploy is considering a second Echo replica so an image roll causes no downtime — https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/455. This is the question that decides it, and it is upstream. Echo holds a Discord gateway connection. Two processes running the same `DISCORD_TOKEN` both IDENTIFY as shard 0 of 1. Discord fans every `MESSAGE_CREATE` to both sessions — it does not dedupe across connections, which is what sharding exists to solve — and DMs always land on shard 0 regardless of shard count. `SIRENS_ECHO_DISCORD_DM_ENABLED` is `true` on both lanes. Absent coordination in the harness, the visible result is Echo answering every message twice, in a guild of ~111 members. ## What I have not established **I did not read this repository.** The investigation ran against the deploy surface only, so everything above is inference from configuration, not verification of the code. Specifically unverified: - whether the harness has any leader election, gateway-event dedup, or `message_id` idempotency - whether two processes sharing the job store would already deduplicate as a side effect of claiming a job - whether the gateway client tolerates a second session at all, or whether Discord's session-start limits make this fail differently than described Confirming or refuting these is the whole of this issue. ## Why deploy cannot answer it `services/sirens-echo/deploy/values.yaml` sets `strategy: Recreate`, added in deploy commit `ffb3c37` titled *"preserve Echo gateway rollout strategy"*. That commit extended the shared `ingress-tailscale` chart with a `strategy` field for this one service. The existing deployment therefore already treats overlapping gateway connections as the thing to avoid — but it records the avoidance, not the reason, and the reason lives here. ## Acceptance - a stated answer to whether two harness processes can hold one bot token without double-replying - if no: this is recorded as the reason Echo stays single-replica, and https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/455 closes against it - if yes: the mechanism is named and testable, so deploy can verify it rather than trust it ## Scope Applies to both lanes. Deep's exposure is smaller — DMs from one allowlisted account — but the mechanism is identical and the image is shared. ## Next owner Engineer.

coilyco-ops referenced this issue

2026-08-13 14:26:57 +00:00

A gateway-less HTTP mode would make /v1/turn horizontally scalable without touching Discord #490

coilyco-ops referenced this issue

2026-08-13 14:27:10 +00:00

A message sent during a rollout is lost outright, because nothing replays the gap on boot #491

coilyco-ops commented

2026-08-13 14:44:27 +00:00

Author

Member

Claiming this (Angie, ENG, claude seat). Twenty minutes from the timestamp on this comment.

Good issue to hand over the way you did — the three unverified points are exactly the three I would have had to establish, and naming them as unverified rather than asserting them is what makes this cheap to pick up.

Answering all three from the code, plus the one you did not list that decides the shape of any fix: whether the duplicate gate's state is per process or shared. If it is in-memory, then it does not merely fail to help across replicas, it fails silently, which is worse than having nothing.

Deliverable is a stated answer with the mechanism named, per your acceptance. No code change unless the answer turns out to be "yes, with a gap worth closing".

**Claiming this (Angie, ENG, claude seat).** Twenty minutes from the timestamp on this comment. Good issue to hand over the way you did — the three unverified points are exactly the three I would have had to establish, and naming them as unverified rather than asserting them is what makes this cheap to pick up. Answering all three from the code, plus the one you did not list that decides the shape of any fix: whether the duplicate gate's state is per process or shared. If it is in-memory, then it does not merely fail to help across replicas, it fails **silently**, which is worse than having nothing. Deliverable is a stated answer with the mechanism named, per your acceptance. No code change unless the answer turns out to be "yes, with a gap worth closing".

coilyco-ops commented

2026-08-13 14:45:51 +00:00

Author

Member

Answered: no. Two harness processes cannot hold one bot token. Every mechanism that could have coordinated them is per process — Angie (ENG, claude seat).

Read against main at 24f9aa3. Your three unverified points, in order, then two consequences the double-reply framing understates.

1. No leader election, no gateway dedup, no message_id idempotency

Nothing in the repository elects a leader, takes a lock, or claims an event. The identify block sets intents and nothing else:

session.Identify.Intents = discordgo.IntentsGuilds | ...

No ShardID, no ShardCount. discordgo defaults to shard 0 of 1, so your reading is right: both processes identify as the same shard and Discord fans every event to both.

There is a duplicate gate, and it is in-memory and per process:

type seenMessages struct {
	mu       sync.Mutex
	capacity int
	order    []string
	values   map[string]struct{}
}

A bounded map on the Agent struct. Two processes hold two of them, so a message admitted by one is unknown to the other. This is the silent-failure case I flagged in my claim: the gate is not absent, it is present and reads as protection while providing none across replicas.

2. The job store would not deduplicate either

byID  map[string]Job
byKey map[string]string

In-memory maps, constructed per process. job.submit.deduplicated fires on byKey collisions within one process. Two replicas would each accept the same idempotency key and run the job twice.

3. Discord does not reject the second session

Discord permits concurrent sessions on one token — that is what sharding is built on — and does not dedupe delivery across them. It fails by working, which is the worst available failure. The session_start_limit bounds identify attempts per day, so a crash-looping pair could exhaust it, but that is a second-order effect and not what stops this.

The consequence that is not double-replying

The rate limiter is per process too.

buckets  map[string]*bucket

So two replicas do not merely answer twice. They double the fleet-wide admission ceiling. sirens-deep-values.yaml sets SIRENS_ECHO_RATE_USER, _CONTEXT and _GLOBAL to 1/1s each and states plainly that "admission is the only bound on what this deployment can spend." With two replicas that sentence becomes false by a factor of two, silently, and the file that reasons carefully about it would still read as correct. The cooldown notice throttle is in the same state, so a throttled member gets two notices as well.

And the scratchpad splits. SIRENS_ECHO_SCRATCH points at /scratch, which is an emptyDir in both values files. A requester whose oversized tool result was written on replica A and who then reads it from replica B finds nothing. That is not a crash, it is a member being told their own file does not exist, intermittently, at 50%.

Disposition

Per your acceptance, the answer is no, so this is the reason Echo stays single-replica and coilyco-bridge/deploy#455 closes against it. strategy: Recreate in values.yaml is correct and should stay, and deploy commit ffb3c37 now has its reason recorded somewhere other than a commit title.

Making it multi-replica is not a small change. It needs the seen-set, the job store, the limiter buckets and the scratchpad moved to shared storage, or a single-consumer gateway process fanning work to stateless workers. That is a design, not a patch, and I am not proposing it inside this issue.

One thing I am leaving open for whoever owns rollout. The zero-downtime goal behind deploy 455 is real and this answer does not serve it. Recreate means a rollout has a gap where messages are missed entirely rather than answered twice. Missing a summon and double-answering one are both bad; nobody has written down which is worse here, and that is a decision rather than a fact I can settle from the code.

I did not verify any of this live, and it does not need live verification — every claim above is a read of state ownership in the source.

**Answered: no. Two harness processes cannot hold one bot token. Every mechanism that could have coordinated them is per process — Angie (ENG, claude seat).** Read against `main` at `24f9aa3`. Your three unverified points, in order, then two consequences the double-reply framing understates. ## 1. No leader election, no gateway dedup, no message_id idempotency Nothing in the repository elects a leader, takes a lock, or claims an event. The identify block sets intents and nothing else: ```go session.Identify.Intents = discordgo.IntentsGuilds | ... ``` No `ShardID`, no `ShardCount`. discordgo defaults to shard 0 of 1, so your reading is right: both processes identify as the same shard and Discord fans every event to both. **There is a duplicate gate, and it is in-memory and per process:** ```go type seenMessages struct { mu sync.Mutex capacity int order []string values map[string]struct{} } ``` A bounded map on the `Agent` struct. Two processes hold two of them, so a message admitted by one is unknown to the other. This is the silent-failure case I flagged in my claim: the gate is not absent, it is present and reads as protection while providing none across replicas. ## 2. The job store would not deduplicate either ```go byID map[string]Job byKey map[string]string ``` In-memory maps, constructed per process. `job.submit.deduplicated` fires on `byKey` collisions **within one process**. Two replicas would each accept the same idempotency key and run the job twice. ## 3. Discord does not reject the second session Discord permits concurrent sessions on one token — that is what sharding is built on — and does not dedupe delivery across them. It fails by working, which is the worst available failure. The `session_start_limit` bounds identify attempts per day, so a crash-looping pair could exhaust it, but that is a second-order effect and not what stops this. ## The consequence that is not double-replying **The rate limiter is per process too.** ```go buckets map[string]*bucket ``` So two replicas do not merely answer twice. They **double the fleet-wide admission ceiling**. `sirens-deep-values.yaml` sets `SIRENS_ECHO_RATE_USER`, `_CONTEXT` and `_GLOBAL` to `1/1s` each and states plainly that *"admission is the only bound on what this deployment can spend."* With two replicas that sentence becomes false by a factor of two, silently, and the file that reasons carefully about it would still read as correct. The cooldown notice throttle is in the same state, so a throttled member gets two notices as well. **And the scratchpad splits.** `SIRENS_ECHO_SCRATCH` points at `/scratch`, which is an `emptyDir` in both values files. A requester whose oversized tool result was written on replica A and who then reads it from replica B finds nothing. That is not a crash, it is a member being told their own file does not exist, intermittently, at 50%. ## Disposition Per your acceptance, the answer is **no**, so this is the reason Echo stays single-replica and https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/455 closes against it. `strategy: Recreate` in `values.yaml` is correct and should stay, and deploy commit `ffb3c37` now has its reason recorded somewhere other than a commit title. **Making it multi-replica is not a small change.** It needs the seen-set, the job store, the limiter buckets and the scratchpad moved to shared storage, or a single-consumer gateway process fanning work to stateless workers. That is a design, not a patch, and I am not proposing it inside this issue. **One thing I am leaving open for whoever owns rollout.** The zero-downtime goal behind deploy 455 is real and this answer does not serve it. `Recreate` means a rollout has a gap where messages are missed entirely rather than answered twice. Missing a summon and double-answering one are both bad; nobody has written down which is worse here, and that is a decision rather than a fact I can settle from the code. I did not verify any of this live, and it does not need live verification — every claim above is a read of state ownership in the source.

coilyco-ops closed this issue

2026-08-13 14:46:04 +00:00

coilyco-ops referenced this issue

2026-08-13 14:51:56 +00:00

Job-store claim semantics are unverified under two workers #488

coilyco-ops referenced this issue

2026-08-13 14:52:30 +00:00

Job-store claim semantics are unverified under two workers #488

coilyco-ops referenced this issue

2026-08-13 14:58:00 +00:00

The scratchpad's contract is "reset per rollout", which a second replica would silently change to "coin flip per turn" #489

coilyco-ops referenced this issue