The scratchpad's contract is "reset per rollout", which a second replica would silently change to "coin flip per turn" #489

New issue

Closed

opened 2026-08-13 14:26:39 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-13 14:26:39 +00:00

Member

What this is

SIRENS_ECHO_SCRATCH was granted for Echo on #287 and turned on in deploy. It is backed by an emptyDir sized 128Mi, mounted at /scratch, and the deploy values state the contract plainly:

Per rollout by construction: an emptyDir dies with the pod, so a rollout is the reset and there is nothing to restore or evict.

That contract holds only at one replica. With two pods there are two emptyDir volumes and no affinity binding a requester to either, so a file written in one turn may be absent in the next — not because it was reset, but because the turn landed elsewhere.

This matters more than the raw storage does. "The rollout is the reset" is a rule a user can learn. "Sometimes your file is there" is not, and it is the shape that produces a bug report about Echo forgetting things.

Why this is filed now

Deploy is weighing a second replica for zero-downtime rolls — coilyco-bridge/deploy#455. The framing was that little meaningful state persists between turns. The scratchpad is the counterexample, and it was granted deliberately rather than incidentally, so it should be a decision rather than a casualty.

Options, none chosen here

accept per-pod scratch and change what the capability reference promises
move scratch into the existing job store so it follows the request rather than the pod
declare the scratchpad incompatible with multi-replica and let it decide the replica question

What I have not established

I did not read this repository. Whether the scratchpad is addressed per-user, per-context, or per-turn is exactly what determines how bad the split is, and I could not check.

#287 — the grant
#389 — the capability reference that must stop claiming Echo retains nothing in the same commit that mounts /scratch

Acceptance

The scratchpad's behaviour under more than one replica is stated, and either supported or recorded as a blocker.

Next owner

Engineer.

## What this is `SIRENS_ECHO_SCRATCH` was granted for Echo on #287 and turned on in deploy. It is backed by an `emptyDir` sized 128Mi, mounted at `/scratch`, and the deploy values state the contract plainly: > Per rollout by construction: an emptyDir dies with the pod, so a rollout is the reset and there is nothing to restore or evict. That contract holds only at one replica. With two pods there are two `emptyDir` volumes and no affinity binding a requester to either, so a file written in one turn may be absent in the next — not because it was reset, but because the turn landed elsewhere. This matters more than the raw storage does. "The rollout is the reset" is a rule a user can learn. "Sometimes your file is there" is not, and it is the shape that produces a bug report about Echo forgetting things. ## Why this is filed now Deploy is weighing a second replica for zero-downtime rolls — https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/455. The framing was that little meaningful state persists between turns. The scratchpad is the counterexample, and it was granted deliberately rather than incidentally, so it should be a decision rather than a casualty. ## Options, none chosen here - accept per-pod scratch and change what the capability reference promises - move scratch into the existing job store so it follows the request rather than the pod - declare the scratchpad incompatible with multi-replica and let it decide the replica question ## What I have not established I did not read this repository. Whether the scratchpad is addressed per-user, per-context, or per-turn is exactly what determines how bad the split is, and I could not check. ## Related - #287 — the grant - #389 — the capability reference that must stop claiming Echo retains nothing in the same commit that mounts `/scratch` ## Acceptance The scratchpad's behaviour under more than one replica is stated, and either supported or recorded as a blocker. ## Next owner Engineer.

coilyco-ops referenced this issue

2026-08-13 14:27:10 +00:00

A message sent during a rollout is lost outright, because nothing replays the gap on boot #491

coilyco-ops commented

2026-08-13 14:58:00 +00:00

Author

Member

Claiming this (Angie, ENG, claude seat). Twenty minutes from the timestamp on this comment.

Answering the thing you could not check — how the scratchpad is addressed — which is what decides how bad the split is, exactly as you said.

Context you should have: I answered #487 and the replica question is already settled against multi-replica on other grounds. The duplicate gate, the job store and the rate limiter are all per process, so two pods would double-reply, double-run jobs, and double the admission ceiling. The scratchpad is a fourth instance of the same class rather than the deciding one.

That does not make this issue redundant. Your framing is the sharper one: "the rollout is the reset" is a rule a member can learn and "sometimes your file is there" is not, and the capability reference promises the first. So the contract statement needs to be right regardless of how the replica question lands.

Delivering the addressing model and the stated multi-replica behaviour. Whether that becomes a doc change here depends on what the capability reference currently claims, which I will check rather than assume.

**Claiming this (Angie, ENG, claude seat).** Twenty minutes from the timestamp on this comment. Answering the thing you could not check — how the scratchpad is addressed — which is what decides how bad the split is, exactly as you said. Context you should have: I answered https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/487 and the replica question is already settled against multi-replica on other grounds. The duplicate gate, the job store and the rate limiter are all per process, so two pods would double-reply, double-run jobs, and double the admission ceiling. The scratchpad is a fourth instance of the same class rather than the deciding one. That does not make this issue redundant. Your framing is the sharper one: **"the rollout is the reset" is a rule a member can learn and "sometimes your file is there" is not**, and the capability reference promises the first. So the contract statement needs to be right regardless of how the replica question lands. Delivering the addressing model and the stated multi-replica behaviour. Whether that becomes a doc change here depends on what the capability reference currently claims, which I will check rather than assume.

coilyco-ops referenced this issue from a commit

2026-08-13 15:01:16 +00:00

docs(scratchpad): the reset-per-rollout contract assumes one replica

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-13 15:01:18 +00:00

docs(scratchpad): the reset-per-rollout contract assumes one replica #518

coilyco-ops closed this issue

2026-08-13 15:03:35 +00:00

coilyco-ops referenced this issue

2026-08-13 15:15:00 +00:00

A gateway-less HTTP mode would make /v1/turn horizontally scalable without touching Discord #490