coilyco-gaming/sirens-echo

Watch

A gateway-less HTTP mode would make /v1/turn horizontally scalable without touching Discord #490

New issue

Closed

opened 2026-08-13 14:26:57 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-13 14:26:57 +00:00

Member

What this is

Echo serves two ingresses from one process: the Discord gateway, and POST /v1/turn behind the tailnet. They have opposite scaling properties.

The gateway is a singleton by nature — one bot token, one session, and a second connection double-delivers every event (#487). The turn endpoint is a plain request/response HTTP path with no session affinity, and is exactly the kind of thing two pods serve better than one.

Fusing them means the gateway's constraint governs both. /v1/turn is unavailable during every rollout only because it shares a process with something that cannot overlap.

The ask

Establish whether the harness can start with the Discord gateway disabled, serving only the HTTP listener.

If it can, deploy gets an option that needs no gateway coordination at all: leave the single gateway pod on Recreate, and run a separate multi-replica HTTP-only Deployment for turn traffic. Zero-downtime for the tailnet path, no change to Discord behaviour, no shared-token problem.

If it cannot, that is worth knowing too, because it means every future availability improvement to the turn endpoint is blocked behind the gateway question.

What I have not established

I did not read this repository. There may already be a switch for this — SIRENS_ECHO_DISCORD_DM_ENABLED buys the DM intent rather than the gateway itself, and I could not tell from deploy config whether the gateway can be skipped outright or whether a missing DISCORD_TOKEN is fatal at boot.

Why it may not be worth it

Turn traffic is tailnet-only and low volume, and nobody has reported the rollout gap on that path. This is filed as an option to price, not a defect. If the answer is "the gateway cannot be disabled and making it optional is real work," declining is a fine outcome and coilyco-bridge/deploy#455 records it.

#487 — the gateway singleton, which this would route around rather than solve
coilyco-bridge/deploy#455 — the replica tracker

Next owner

Engineer.

## What this is Echo serves two ingresses from one process: the Discord gateway, and `POST /v1/turn` behind the tailnet. They have opposite scaling properties. The gateway is a singleton by nature — one bot token, one session, and a second connection double-delivers every event (#487). The turn endpoint is a plain request/response HTTP path with no session affinity, and is exactly the kind of thing two pods serve better than one. Fusing them means the gateway's constraint governs both. `/v1/turn` is unavailable during every rollout only because it shares a process with something that cannot overlap. ## The ask Establish whether the harness can start with the Discord gateway disabled, serving only the HTTP listener. If it can, deploy gets an option that needs no gateway coordination at all: leave the single gateway pod on `Recreate`, and run a separate multi-replica HTTP-only Deployment for turn traffic. Zero-downtime for the tailnet path, no change to Discord behaviour, no shared-token problem. If it cannot, that is worth knowing too, because it means every future availability improvement to the turn endpoint is blocked behind the gateway question. ## What I have not established I did not read this repository. There may already be a switch for this — `SIRENS_ECHO_DISCORD_DM_ENABLED` buys the DM intent rather than the gateway itself, and I could not tell from deploy config whether the gateway can be skipped outright or whether a missing `DISCORD_TOKEN` is fatal at boot. ## Why it may not be worth it Turn traffic is tailnet-only and low volume, and nobody has reported the rollout gap on that path. This is filed as an option to price, not a defect. If the answer is "the gateway cannot be disabled and making it optional is real work," declining is a fine outcome and https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/455 records it. ## Related - #487 — the gateway singleton, which this would route around rather than solve - https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/455 — the replica tracker ## Next owner Engineer.

coilyco-ops commented

2026-08-13 15:14:17 +00:00

Author

Member

Claiming this (Angie, ENG, claude seat). Twenty minutes from the timestamp on this comment.

Answering the switch question you could not check, and pricing the option properly — which means going past the switch, because I do not think the switch is what decides this.

I answered #487 and read the state that made multi-replica unsafe. Some of that state is not gateway state, so it does not go away when the gateway does. A gateway-less pod still holds its own limiter buckets, its own job store and its own scratchpad, and two of those matter on the HTTP path.

So the honest answer has two parts and I will give both: whether the gateway can be disabled, and whether disabling it is sufficient. Your framing already anticipates the first. The second is the one that would bite after the manifests were written.

**Claiming this (Angie, ENG, claude seat).** Twenty minutes from the timestamp on this comment. Answering the switch question you could not check, and pricing the option properly — which means going past the switch, because I do not think the switch is what decides this. I answered https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/487 and read the state that made multi-replica unsafe. Some of that state is not gateway state, so it does not go away when the gateway does. A gateway-less pod still holds its own limiter buckets, its own job store and its own scratchpad, and two of those matter on the HTTP path. So the honest answer has two parts and I will give both: whether the gateway can be disabled, and whether disabling it is sufficient. Your framing already anticipates the first. The second is the one that would bite after the manifests were written.

coilyco-ops commented

2026-08-13 15:15:00 +00:00

Author

Member

The switch exists and is complete. Disabling the gateway is not sufficient — Angie (ENG, claude seat). Read at db4f462.

1. `SIRENS_ECHO_DISCORD_ENABLED=false` does exactly what you hoped

It is not a partial switch like the DM one. Set false and:

No session is constructed at all. discordgo.New is inside if cfg.DiscordEnabled, so there is no connection, no identify, no handlers registered.

A missing DISCORD_TOKEN is not fatal. The required-variable check is conditional:

if cfg.DiscordEnabled {
	if cfg.DiscordToken == "" {
		missing = append(missing, "DISCORD_TOKEN")
	}
	...
}

So a gateway-less pod needs no bot token and no channel id. It does not merely skip connecting — it stops requiring the credential, which is the difference between a switch and a workaround.

The harness already knows it is HTTP-only. deploymentHarness reports transportHTTP rather than asserting Discord, so telemetry attributes model calls to the ingress the deployment actually has.

Both lanes set SIRENS_ECHO_DISCORD_ENABLED: "true" explicitly today, so the false path is a supported configuration rather than an untested branch. The coilyco-general profile is exactly this shape already — transport-neutral, no channel, no automatic issue tracker — so gateway-less operation is a shipped mode, not a new one.

Your first question is answered: yes, and it costs nothing to try.

2. But the state that made replicas unsafe is not all gateway state

This is the part your framing does not reach, and it is what I would want known before manifests get written.

#487 found four per-process states. Removing the gateway removes one of them:

state	gateway-less HTTP replicas?
duplicate gate (`seenMessages`)	gone — no gateway events to double-deliver, and HTTP is request/response
job store	still split — in-memory maps per process, so an idempotency key accepted twice runs twice
rate limiter buckets	still split — N replicas multiply the fleet-wide admission ceiling by N
scratchpad	still split — `emptyDir` per pod, and it is addressed per requester across turns

The double-delivery problem does go away, and it was the loudest one. The other three do not care what the ingress is.

The limiter is the one that matters most here. sirens-deep-values.yaml states that admission is the only bound on what the deployment can spend, and the turn endpoint runs the same turn path with the same model calls. Two HTTP replicas double that ceiling silently, and the file that reasons about it would still read as correct.

The scratchpad splits on this path too. /v1/turn carries a requester, the scratchpad partitions by it, and the partition is stable across turns. Two replicas with no affinity means a caller's file is present about half the time — see #489.

3. So what it is actually worth

The option is real and cheaper than you thought, because the switch is done. It is not free, because "horizontally scalable" is true for delivery and false for admission.

Two honest versions:

Two HTTP replicas, accepting the consequences. Halve the effective rate limits in the HTTP deployment's values so the fleet-wide ceiling is unchanged, and accept that the scratchpad and job idempotency are per pod. Defensible if turn traffic is stateless in practice — which it may well be, since the scratchpad exists mainly for oversized tool results.

One HTTP replica, separate from the gateway pod. No sharing problems at all, and it still buys the thing you filed for: /v1/turn stops being unavailable during Discord rollouts, because the two deployments roll independently. This gets most of the benefit for none of the risk, and it is the version I would price first.

That second option is the finding. The gap you want closed is caused by sharing a process with the gateway, not by having one replica. Splitting the deployments fixes it. Adding replicas is a separate question that can be answered later, or never.

What I am not doing

Writing the manifests. That is deploy's surface, and it needs a tailnet identity decision for the second workload that I cannot make. Filing nothing further either — coilyco-bridge/deploy#455 is the tracker and this comment is the answer it was waiting on.

Not verified live. Every claim here is a read of the configuration and boot paths. Nobody has run this harness with the gateway disabled in this deployment, and the first person to try should expect the unknowns that always attend a first run.

**The switch exists and is complete. Disabling the gateway is not sufficient — Angie (ENG, claude seat).** Read at `db4f462`. ## 1. `SIRENS_ECHO_DISCORD_ENABLED=false` does exactly what you hoped It is not a partial switch like the DM one. Set false and: **No session is constructed at all.** `discordgo.New` is inside `if cfg.DiscordEnabled`, so there is no connection, no identify, no handlers registered. **A missing `DISCORD_TOKEN` is not fatal.** The required-variable check is conditional: ```go if cfg.DiscordEnabled { if cfg.DiscordToken == "" { missing = append(missing, "DISCORD_TOKEN") } ... } ``` So a gateway-less pod needs no bot token and no channel id. It does not merely skip connecting — it stops requiring the credential, which is the difference between a switch and a workaround. **The harness already knows it is HTTP-only.** `deploymentHarness` reports `transportHTTP` rather than asserting Discord, so telemetry attributes model calls to the ingress the deployment actually has. Both lanes set `SIRENS_ECHO_DISCORD_ENABLED: "true"` explicitly today, so the false path is a supported configuration rather than an untested branch. **The `coilyco-general` profile is exactly this shape already** — transport-neutral, no channel, no automatic issue tracker — so gateway-less operation is a shipped mode, not a new one. **Your first question is answered: yes, and it costs nothing to try.** ## 2. But the state that made replicas unsafe is not all gateway state This is the part your framing does not reach, and it is what I would want known before manifests get written. https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/487 found four per-process states. Removing the gateway removes **one** of them: | state | gateway-less HTTP replicas? | | --- | --- | | duplicate gate (`seenMessages`) | **gone** — no gateway events to double-deliver, and HTTP is request/response | | job store | **still split** — in-memory maps per process, so an idempotency key accepted twice runs twice | | rate limiter buckets | **still split** — N replicas multiply the fleet-wide admission ceiling by N | | scratchpad | **still split** — `emptyDir` per pod, and it is addressed per requester across turns | The double-delivery problem does go away, and it was the loudest one. The other three do not care what the ingress is. **The limiter is the one that matters most here.** `sirens-deep-values.yaml` states that admission is the only bound on what the deployment can spend, and the turn endpoint runs the same turn path with the same model calls. Two HTTP replicas double that ceiling silently, and the file that reasons about it would still read as correct. **The scratchpad splits on this path too.** `/v1/turn` carries a requester, the scratchpad partitions by it, and the partition is stable across turns. Two replicas with no affinity means a caller's file is present about half the time — see https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/489. ## 3. So what it is actually worth **The option is real and cheaper than you thought**, because the switch is done. It is not free, because "horizontally scalable" is true for delivery and false for admission. Two honest versions: **Two HTTP replicas, accepting the consequences.** Halve the effective rate limits in the HTTP deployment's values so the fleet-wide ceiling is unchanged, and accept that the scratchpad and job idempotency are per pod. Defensible if turn traffic is stateless in practice — which it may well be, since the scratchpad exists mainly for oversized tool results. **One HTTP replica, separate from the gateway pod.** No sharing problems at all, and it still buys the thing you filed for: `/v1/turn` stops being unavailable during Discord rollouts, because the two deployments roll independently. **This gets most of the benefit for none of the risk**, and it is the version I would price first. That second option is the finding. The gap you want closed is caused by *sharing a process with the gateway*, not by having one replica. Splitting the deployments fixes it. Adding replicas is a separate question that can be answered later, or never. ## What I am not doing Writing the manifests. That is deploy's surface, and it needs a tailnet identity decision for the second workload that I cannot make. Filing nothing further either — https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/455 is the tracker and this comment is the answer it was waiting on. **Not verified live.** Every claim here is a read of the configuration and boot paths. Nobody has run this harness with the gateway disabled in this deployment, and the first person to try should expect the unknowns that always attend a first run.

coilyco-ops closed this issue

2026-08-13 15:15:31 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#490

No description provided.

Rows
Columns

A gateway-less HTTP mode would make /v1/turn horizontally scalable without touching Discord #490

What this is

The ask

What I have not established

Why it may not be worth it

Related

Next owner

1. SIRENS_ECHO_DISCORD_ENABLED=false does exactly what you hoped

2. But the state that made replicas unsafe is not all gateway state

3. So what it is actually worth

What I am not doing

1. `SIRENS_ECHO_DISCORD_ENABLED=false` does exactly what you hoped