Add operational resilience tooling #71

New issue

Closed

opened 2026-08-05 21:59:37 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-08-05 21:59:37 +00:00

Owner

I have a bunch of o11y, so off the top of my head I'm just thinking about rate limiting. Quick google + more thoughts will probably surface more.

coilyco-ops commented

2026-08-11 04:30:00 +00:00

Member

Closed by #82.

The rate limiting this issue names is implemented as admission control at the shared turn boundary, so Discord and the private HTTP path are governed by one policy.

Per-user, per-context, and global token buckets, defaults 3/30s, 10/10s, 20/5s, all env-tunable or off.
A bounded pending queue, default 8, that sheds rather than queues, because a stale queued turn still costs a completion.
Every tier checked before any is charged, so a global refusal does not silently spend a member's own budget.
One cooldown notice per key per window rather than one per denial, since a reply per denial burns the bot's own Discord message budget and hands a flooder an amplifier.
Bucket state in a capacity-bounded LRU, because rotating identities is the cheapest way to attack a limiter that keeps unbounded per-key state.
sirens_echo.admissions with closed-set outcome and transport labels, so a flood cannot expand metric cardinality.

Discord lookups made during gate evaluation are bounded separately, so a member posting in an unscoped channel cannot force one API call per message.

Contract and tuning guidance in docs/sirens-echo-admission.md.

Beyond rate limiting, the same change fixed four adjacent resilience defects found while reading: the request timeout started before queue admission so queued turns burned their budget waiting, the typing indicator fired before admission and expired mid-turn, handler goroutines had no panic recovery so one bad turn could take down a process serving every guild, and scope resolution issued a Discord REST call for every unseen channel with no negative caching.

Two resilience gaps this issue's title covers remain open and are tracked separately: #86 for the 20% turn failure from tool-inflated context exhausting a fixed completion budget, and #90 for the per-turn failing MCP calls.

Closed by #82. The rate limiting this issue names is implemented as admission control at the shared turn boundary, so Discord and the private HTTP path are governed by one policy. * Per-user, per-context, and global token buckets, defaults `3/30s`, `10/10s`, `20/5s`, all env-tunable or `off`. * A bounded pending queue, default 8, that sheds rather than queues, because a stale queued turn still costs a completion. * Every tier checked before any is charged, so a global refusal does not silently spend a member's own budget. * One cooldown notice per key per window rather than one per denial, since a reply per denial burns the bot's own Discord message budget and hands a flooder an amplifier. * Bucket state in a capacity-bounded LRU, because rotating identities is the cheapest way to attack a limiter that keeps unbounded per-key state. * `sirens_echo.admissions` with closed-set `outcome` and `transport` labels, so a flood cannot expand metric cardinality. Discord lookups made during gate evaluation are bounded separately, so a member posting in an unscoped channel cannot force one API call per message. Contract and tuning guidance in `docs/sirens-echo-admission.md`. Beyond rate limiting, the same change fixed four adjacent resilience defects found while reading: the request timeout started before queue admission so queued turns burned their budget waiting, the typing indicator fired before admission and expired mid-turn, handler goroutines had no panic recovery so one bad turn could take down a process serving every guild, and scope resolution issued a Discord REST call for every unseen channel with no negative caching. Two resilience gaps this issue's title covers remain open and are tracked separately: #86 for the 20% turn failure from tool-inflated context exhausting a fixed completion budget, and #90 for the per-turn failing MCP calls.

coilyco-ops closed this issue

2026-08-11 04:30:11 +00:00

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#71

No description provided.

Rows
Columns