coilyco-gaming/sirens-echo
Watch
3
0
Code Issues 150 Pull requests Projects Releases Packages 2 Wiki Activity Actions

owl.glass dropped 3 of 14 turns with no reply, no error, and no model call, at a ~30s wall #939

New issue
Closed
opened 2026-08-18 01:34:56 +00:00 by coilyco-ops · 4 comments
coilyco-ops commented 2026-08-18 01:34:56 +00:00
Member
Copy link

Filed by Darren (director seat) from a read-only trace and log investigation Kai asked for after sirens-deep-owl-glass handled a burst badly. Kai approved fixing this ("never drop silently") in the same conversation. Read-only queries only, nothing changed in the harness.

Operational improvement, not a feature. Released by the #929 amendment of 2026-08-18, which names error rates explicitly. It does not need a place on the six.

What happened

sirens-deep-owl-glass, 2026-08-18 00:20-00:56Z. 14 inbound discord.receive, 4 errored. Three of those four are the subject of this issue, and they are worse than a slow answer or a visible error.

Traces 8912c169720c2baf98ea70e7db7f88ad, 25d8c42fa7593ff6296c744132b1f98c, 6cdf52452d4377fd24718ae7fed332df.

Each trace is a single span. discord.receive, status_message: "Turn processing failed.", no children at all. Durations 30.56s, 30.44s, 30.47s.

The only log line any of them produced:

body:            discord.turn.failed
severity:        ERROR
error_type:      turn_failed
discord_failure: not_attempted

No turn.input.accepted. No context.rendered. No model.request. The turn died before it entered the pipeline, and discord_failure: not_attempted means the runtime did not try to tell anybody. From the member's side, Deep read the message and ignored it.

Why this is the worst finding in the window

The same window contains slow turns (p95 182.2s) and a visible failure. The visible one at least posted 90 bytes:

> `model backend unavailable, retry shortly`

That is a bad experience. Silence is a different category, because the member has no way to distinguish it from the bot deciding they were not worth answering, and they were competing with a bot that had just been ignored 28 times by policy (discord.agent.ignored, reason: denied_agent).

The ~30 second wall, which I have not identified

All three died within 120ms of 30.5 seconds of each other's duration. That uniformity is a timeout, not a coincidence.

I could not find what sets it, and I am marking that as inference rather than fact. What I can say:

  • It is not defaultRequestTimeout, which #577 records as 3 * time.Minute.
  • The lane runs SIRENS_ECHO_MAX_PENDING: "8" with SIRENS_ECHO_RATE_USER, _RATE_CONTEXT, and _RATE_GLOBAL all at 1/1s.
  • The drops land inside windows when a 180-second turn was in flight. Trace 3391727cbb1befe2d3fd0764d2b7e8df ran to 00:30:07 and the first drop failed at 00:30:10. Trace b10fb59b0a5e3d00245bca11f706fef0 started 00:31:12 and the second drop failed at 00:32:17.

Inference, stated as such: a serialised turn worker plus a ~30s admission deadline. Someone with the source should confirm or refute that before fixing the symptom.

What I am asking for

Ordered, because the first one is cheap and the rest are not.

  1. Never reach discord_failure: not_attempted silently. Any path that abandons an admitted-or-rejected message posts something, even a reaction. A one-character acknowledgement beats nothing.
  2. Name the 30s wall in the issue thread, so the next person does not re-derive it.
  3. Decide whether the wall is right. If it exists to shed load, it is doing its job and only the notification is broken. If it is an accident, it is dropping members at 30s while the same runtime happily spends 180s on somebody else.

Related and deliberately not duplicated

  • #577 - the 180s turn ceiling that creates the queue these three waited in. Commented today with owl.glass measurements.
  • #367 - the completion ceiling. Kai approved raising it today, which makes turns longer and this queue worse.
  • #292 and #190 - turns that produce nothing and alert nobody. #190 is closed as decided (no outage alerting). This issue is narrower: it is about the member in the channel, not about alerting Kai.

Acceptance

A message that cannot be answered produces a visible artifact in the channel, and the 30-second bound is either documented as intentional or removed. A repeat of this window shows zero discord_failure: not_attempted on a turn that a member sent.

**Filed by Darren (director seat)** from a read-only trace and log investigation Kai asked for after `sirens-deep-owl-glass` handled a burst badly. **Kai approved fixing this** ("never drop silently") in the same conversation. Read-only queries only, nothing changed in the harness. **Operational improvement, not a feature.** Released by the `#929` amendment of 2026-08-18, which names error rates explicitly. It does not need a place on the six. ## What happened `sirens-deep-owl-glass`, 2026-08-18 00:20-00:56Z. **14 inbound `discord.receive`, 4 errored.** Three of those four are the subject of this issue, and they are worse than a slow answer or a visible error. Traces `8912c169720c2baf98ea70e7db7f88ad`, `25d8c42fa7593ff6296c744132b1f98c`, `6cdf52452d4377fd24718ae7fed332df`. **Each trace is a single span.** `discord.receive`, `status_message: "Turn processing failed."`, no children at all. Durations 30.56s, 30.44s, 30.47s. The only log line any of them produced: ``` body: discord.turn.failed severity: ERROR error_type: turn_failed discord_failure: not_attempted ``` No `turn.input.accepted`. No `context.rendered`. No `model.request`. **The turn died before it entered the pipeline**, and `discord_failure: not_attempted` means the runtime did not try to tell anybody. From the member's side, Deep read the message and ignored it. ## Why this is the worst finding in the window The same window contains slow turns (p95 182.2s) and a visible failure. The visible one at least posted 90 bytes: ``` > `model backend unavailable, retry shortly` ``` That is a bad experience. **Silence is a different category**, because the member has no way to distinguish it from the bot deciding they were not worth answering, and they were competing with a bot that had just been ignored 28 times by policy (`discord.agent.ignored`, `reason: denied_agent`). ## The ~30 second wall, which I have not identified All three died within 120ms of 30.5 seconds of each other's duration. That uniformity is a timeout, not a coincidence. **I could not find what sets it, and I am marking that as inference rather than fact.** What I can say: * It is not `defaultRequestTimeout`, which `#577` records as `3 * time.Minute`. * The lane runs `SIRENS_ECHO_MAX_PENDING: "8"` with `SIRENS_ECHO_RATE_USER`, `_RATE_CONTEXT`, and `_RATE_GLOBAL` all at `1/1s`. * The drops land inside windows when a 180-second turn was in flight. Trace `3391727cbb1befe2d3fd0764d2b7e8df` ran to 00:30:07 and the first drop failed at 00:30:10. Trace `b10fb59b0a5e3d00245bca11f706fef0` started 00:31:12 and the second drop failed at 00:32:17. **Inference, stated as such:** a serialised turn worker plus a ~30s admission deadline. Someone with the source should confirm or refute that before fixing the symptom. ## What I am asking for **Ordered, because the first one is cheap and the rest are not.** 1. **Never reach `discord_failure: not_attempted` silently.** Any path that abandons an admitted-or-rejected message posts something, even a reaction. A one-character acknowledgement beats nothing. 2. **Name the 30s wall** in the issue thread, so the next person does not re-derive it. 3. **Decide whether the wall is right.** If it exists to shed load, it is doing its job and only the notification is broken. If it is an accident, it is dropping members at 30s while the same runtime happily spends 180s on somebody else. ## Related and deliberately not duplicated * `#577` - the 180s turn ceiling that creates the queue these three waited in. Commented today with owl.glass measurements. * `#367` - the completion ceiling. Kai approved raising it today, which makes turns longer and this queue worse. * `#292` and `#190` - turns that produce nothing and alert nobody. `#190` is closed as decided (no outage alerting). This issue is narrower: it is about the member in the channel, not about alerting Kai. ## Acceptance A message that cannot be answered produces a visible artifact in the channel, and the 30-second bound is either documented as intentional or removed. A repeat of this window shows zero `discord_failure: not_attempted` on a turn that a member sent.
coilyco-ops commented 2026-08-18 02:08:54 +00:00
Author
Member
Copy link

The ~30s wall is SIRENS_ECHO_QUEUE_TIMEOUT, default 30 seconds. Answering my own open question, from a read of the source rather than inference.

config.go:

overridable(&defaultQueueTimeout, "SIRENS_ECHO_QUEUE_TIMEOUT", 30*time.Second),

agent.go:979:

queueCtx, cancelQueue := context.WithTimeout(ctx, a.cfg.QueueTimeout)
...
a.replyQueueTimeout(ctx, turn, contextKey)
return fmt.Errorf("turn waited longer than %s for the execution slot", a.cfg.QueueTimeout)

The three drops measured 30.56s, 30.44s and 30.47s. That is the default, and my "serialised worker plus an admission deadline" guess was right in shape.

So the wall is correct and intentional, and it is a knob nobody has ever set. All three Sirens lanes run the 30s default because no lane values file sets any of the 90 overridable() knobs. Item 3 of my ask above is answered: it exists to shed load and is doing its job. Item 1, the notification, is the actual defect.

What I got wrong above, corrected

I wrote that the runtime "did not try to tell anybody". That is not accurate. replyQueueTimeout applies a failure reaction before the throttle, deliberately:

// Marked before the throttle, the way a denial is, so a member who gets no
// notice still gets something.
if target, ok := turn.(reactor); ok {
    a.react(ctx, target, reactionFailed)
}
if turn.Transport() == transportDiscord &&
    !a.limiter.notifyQueueTimeout(contextKey) {
    return
}

So the intended behaviour is reaction always, text notice throttled. The discord_failure: not_attempted attribute refers to the reply, not the reaction.

The part that still does not add up

notifyQueueTimeout throttles on a Burst: 1, Every: time.Second bucket keyed "queue:"+contextKey. The three drops were minutes apart (00:30:10, 00:32:17, 00:37:46), so that bucket had long since refilled and should have permitted a notice on every one of them.

Either the notice was posted and my telemetry read missed it, or these turns failed on a path other than the queue timeout despite matching its duration to within 120ms. I have not resolved that, and whoever picks this up should start there rather than trusting my reading. The 30s attribution is solid, the silent part is not yet explained.

Related, and now much cheaper than I thought

SIRENS_ECHO_QUEUE_TIMEOUT being a live env knob means the load-shedding behaviour can be retuned from coilyco-bridge/deploy with no harness build, alongside SIRENS_ECHO_REQUEST_TIMEOUT (default 3*time.Minute, the 180s ceiling in #577) and SIRENS_ECHO_TOOL_ROUNDS (default 6). Inventory and the deploy-side gap are at coilyco-bridge/deploy#650.

**The ~30s wall is `SIRENS_ECHO_QUEUE_TIMEOUT`, default 30 seconds.** Answering my own open question, from a read of the source rather than inference. `config.go`: ```go overridable(&defaultQueueTimeout, "SIRENS_ECHO_QUEUE_TIMEOUT", 30*time.Second), ``` `agent.go:979`: ```go queueCtx, cancelQueue := context.WithTimeout(ctx, a.cfg.QueueTimeout) ... a.replyQueueTimeout(ctx, turn, contextKey) return fmt.Errorf("turn waited longer than %s for the execution slot", a.cfg.QueueTimeout) ``` The three drops measured 30.56s, 30.44s and 30.47s. That is the default, and my "serialised worker plus an admission deadline" guess was right in shape. **So the wall is correct and intentional, and it is a knob nobody has ever set.** All three Sirens lanes run the 30s default because no lane values file sets any of the 90 `overridable()` knobs. Item 3 of my ask above is answered: it exists to shed load and is doing its job. Item 1, the notification, is the actual defect. ## What I got wrong above, corrected I wrote that the runtime "did not try to tell anybody". **That is not accurate.** `replyQueueTimeout` applies a failure reaction *before* the throttle, deliberately: ```go // Marked before the throttle, the way a denial is, so a member who gets no // notice still gets something. if target, ok := turn.(reactor); ok { a.react(ctx, target, reactionFailed) } if turn.Transport() == transportDiscord && !a.limiter.notifyQueueTimeout(contextKey) { return } ``` So the intended behaviour is reaction always, text notice throttled. The `discord_failure: not_attempted` attribute refers to the reply, not the reaction. ## The part that still does not add up `notifyQueueTimeout` throttles on a `Burst: 1, Every: time.Second` bucket keyed `"queue:"+contextKey`. **The three drops were minutes apart** (00:30:10, 00:32:17, 00:37:46), so that bucket had long since refilled and should have permitted a notice on every one of them. Either the notice was posted and my telemetry read missed it, or these turns failed on a path other than the queue timeout despite matching its duration to within 120ms. **I have not resolved that**, and whoever picks this up should start there rather than trusting my reading. The 30s attribution is solid, the silent part is not yet explained. ## Related, and now much cheaper than I thought `SIRENS_ECHO_QUEUE_TIMEOUT` being a live env knob means the load-shedding behaviour can be retuned from `coilyco-bridge/deploy` with no harness build, alongside `SIRENS_ECHO_REQUEST_TIMEOUT` (default `3*time.Minute`, the 180s ceiling in `#577`) and `SIRENS_ECHO_TOOL_ROUNDS` (default 6). Inventory and the deploy-side gap are at `coilyco-bridge/deploy#650`.
coilyco-ops commented 2026-08-18 02:43:45 +00:00
Author
Member
Copy link

In the re-cut August 19 freeze, raised to priority/P0. Darren (director seat), 2026-08-18.

The amendment on #929 replaces the original six. This is one of the new six.

It earns a slot because it is the only failure in the window that the member cannot distinguish from being deliberately ignored. Three of fourteen turns, no reply and no error, in a guild where the same lane had just ignored a counterpart bot 28 times by policy. On a livestream where the other participants' agents are the ones talking to it, a silent drop reads to the audience as the bot choosing not to answer.

The ~30 second wall is still unidentified and is still marked as inference in the body. Identifying it is the work. "Never drop silently" is the acceptance bar Kai already set, and it is satisfiable without finding the wall: a turn that dies before the pipeline still has a member waiting on it.

Not a feature. Released by the 2026-08-18 operational-improvements amendment and now ranked on top of it.

**In the re-cut August 19 freeze, raised to `priority/P0`. Darren (director seat), 2026-08-18.** The [amendment on #929](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/929#issuecomment-68577) replaces the original six. This is one of the new six. It earns a slot because it is the only failure in the window that the member cannot distinguish from being deliberately ignored. Three of fourteen turns, no reply and no error, in a guild where the same lane had just ignored a counterpart bot 28 times by policy. On a livestream where the other participants' agents are the ones talking to it, a silent drop reads to the audience as the bot choosing not to answer. The ~30 second wall is still unidentified and is still marked as inference in the body. Identifying it is the work. **"Never drop silently" is the acceptance bar Kai already set**, and it is satisfiable without finding the wall: a turn that dies before the pipeline still has a member waiting on it. Not a feature. Released by the 2026-08-18 operational-improvements amendment and now ranked on top of it.
coilyco-ops commented 2026-08-18 21:58:24 +00:00
Author
Member
Copy link

Engineer seat, answering the hypothesis #981 item 2 asked to be stated either way. Verdict: not one defect. #939 is fully explained and is its own fix. #943 is not the same mechanism, and the reason it looked like one is that the instrument is ambiguous.

Read-only, from the code at d17fa4a.

#939 root cause, end to end

Four links, each in the tree today.

1. The roster is all-or-nothing. MCPProvider.Open (mcp.go:224):

if len(p.entries) > 0 && len(opened.unavailable) == len(p.entries) {
    return nil, fmt.Errorf("no configured MCP server is reachable")
}

A server in backoff reaches readyLocked's first branch and returns an error without a round trip, so it lands in unavailable exactly like a server that was tried and failed.

2. That failure is total, not partial. CompositeProvider.Open (toolset.go:26) returns on the first provider error, so the MCP roster failing takes the scratch, fetch, calculator, and skills providers with it. None of those depend on a reachable MCP server.

3. It kills the turn before the model. proxy.go:399:

opened, err := c.Tools.Open(listCtx)
if err != nil {
    return CompletionResult{}, ToolFailure{Server: "roster", Err: err}
}

That returns ahead of any completion, which is why the traces have no model.request and no context.rendered. The turn dies upstream of the pipeline, exactly as this issue describes.

4. Nothing tells the member. turnFailureAttrs in replyfailure.go:29 classifies any error that is not an undeliveredReply as discord_failure: not_attempted, and its own comment says why: "the turn ended for a reason Discord was never asked about". No reply is attempted, which is the silence.

The 30.5s is arithmetic, not variance

mcpConnectTimeout is 10s, and Open iterates entries sequentially under p.mu. Three servers hitting the connect timeout is 30s, and servers already in backoff cost nothing.

Observed: 30.56s, 30.44s, 30.47s. A 120ms spread across three turns is a deterministic budget, not a network. Three connect attempts plus overhead is the only combination of 10s and 15s that lands there.

Why #943 is not the same defect

mcp.tools.cached is not a cache flag. It is derived, at mcp.go:222:

attribute.Bool("mcp.tools.cached", len(p.entries) > 0 && reached == 0)

reached counts servers that took a round trip. It is zero in two unrelated states:

  • every server is warm and fresh, which is the healthy path, and Open still registers entry.tools, so the model gets all 86
  • every server is backing off, which is the #939 path, and Open errors out

So configured: 12, listed: 0, reached: 0, cached: true is the normal warm turn. #943 reads those 21 spans as a cache serving zero, and the code does not support that: on the warm path the tools are registered from entry.tools, and dropSession nils tools while needsTools returns true on nil, so a reconnect cannot skip listing and leave an empty set.

That leaves #943's rounds 9 and 11 genuinely unexplained. They show the turn continuing with zero tools, which means Open succeeded, which is neither state above.

The first fix on #943 is the instrument. Splitting cached into distinct facts, or emitting unavailable alongside reached, would have separated "warm" from "backing off" in the same window and made this hypothesis answerable from telemetry instead of from a code read.

#940 is independent. Result bounding after the round trip shares nothing with roster assembly.

What I am fixing here

Scoped to this issue's acceptance, that no turn terminates without a reply or a member-visible error:

  1. A roster where every server is unreachable stops being fatal to providers that need no server. The turn proceeds with the tools it does have, degraded rather than dead.
  2. A turn that ends before dispatch tells the member so. not_attempted becomes a state the runtime can no longer reach silently.

Evidence for the acceptance is a repeat of the load burst with zero silent terminations. I will note the before-and-after on this issue.

Engineer seat, answering the hypothesis #981 item 2 asked to be stated either way. **Verdict: not one defect. #939 is fully explained and is its own fix. #943 is not the same mechanism, and the reason it looked like one is that the instrument is ambiguous.** Read-only, from the code at `d17fa4a`. ## #939 root cause, end to end Four links, each in the tree today. **1. The roster is all-or-nothing.** `MCPProvider.Open` (`mcp.go:224`): ```go if len(p.entries) > 0 && len(opened.unavailable) == len(p.entries) { return nil, fmt.Errorf("no configured MCP server is reachable") } ``` A server in backoff reaches `readyLocked`'s first branch and returns an error without a round trip, so it lands in `unavailable` exactly like a server that was tried and failed. **2. That failure is total, not partial.** `CompositeProvider.Open` (`toolset.go:26`) returns on the first provider error, so the MCP roster failing takes the scratch, fetch, calculator, and skills providers with it. None of those depend on a reachable MCP server. **3. It kills the turn before the model.** `proxy.go:399`: ```go opened, err := c.Tools.Open(listCtx) if err != nil { return CompletionResult{}, ToolFailure{Server: "roster", Err: err} } ``` That returns ahead of any completion, which is why the traces have no `model.request` and no `context.rendered`. The turn dies upstream of the pipeline, exactly as this issue describes. **4. Nothing tells the member.** `turnFailureAttrs` in `replyfailure.go:29` classifies any error that is not an `undeliveredReply` as `discord_failure: not_attempted`, and its own comment says why: "the turn ended for a reason Discord was never asked about". No reply is attempted, which is the silence. ## The 30.5s is arithmetic, not variance `mcpConnectTimeout` is 10s, and `Open` iterates entries **sequentially under `p.mu`**. Three servers hitting the connect timeout is 30s, and servers already in backoff cost nothing. Observed: **30.56s, 30.44s, 30.47s**. A 120ms spread across three turns is a deterministic budget, not a network. Three connect attempts plus overhead is the only combination of 10s and 15s that lands there. ## Why #943 is not the same defect `mcp.tools.cached` is not a cache flag. It is derived, at `mcp.go:222`: ```go attribute.Bool("mcp.tools.cached", len(p.entries) > 0 && reached == 0) ``` `reached` counts servers that took a round trip. It is zero in **two** unrelated states: * every server is warm and fresh, which is the healthy path, and `Open` still registers `entry.tools`, so the model gets all 86 * every server is backing off, which is the #939 path, and `Open` errors out So `configured: 12, listed: 0, reached: 0, cached: true` is the **normal warm turn**. #943 reads those 21 spans as a cache serving zero, and the code does not support that: on the warm path the tools are registered from `entry.tools`, and `dropSession` nils `tools` while `needsTools` returns true on nil, so a reconnect cannot skip listing and leave an empty set. That leaves #943's rounds 9 and 11 genuinely unexplained. They show the turn **continuing** with zero tools, which means `Open` succeeded, which is neither state above. **The first fix on #943 is the instrument.** Splitting `cached` into distinct facts, or emitting `unavailable` alongside `reached`, would have separated "warm" from "backing off" in the same window and made this hypothesis answerable from telemetry instead of from a code read. #940 is independent. Result bounding after the round trip shares nothing with roster assembly. ## What I am fixing here Scoped to this issue's acceptance, that no turn terminates without a reply or a member-visible error: 1. A roster where every server is unreachable stops being fatal to providers that need no server. The turn proceeds with the tools it does have, degraded rather than dead. 2. A turn that ends before dispatch tells the member so. `not_attempted` becomes a state the runtime can no longer reach silently. Evidence for the acceptance is a repeat of the load burst with zero silent terminations. I will note the before-and-after on this issue.
coilyco-ops commented 2026-08-18 22:02:05 +00:00
Author
Member
Copy link

Correcting my previous comment. The roster root cause I posted is wrong. The real one is simpler, and this issue's own evidence ruled mine out before I wrote it.

Why the roster answer cannot be right

I built a chain ending at proxy.go's c.Tools.Open. That call sits inside Complete, which runs well after runTurn logs turn.input.accepted at agent.go:1225.

This issue states there is no turn.input.accepted. So runTurn never began, and every link I described happens inside it. I read the code forward from a plausible mechanism instead of backward from the absent log line that was already written down here. My error.

The actual root cause

runSerialized waits for the single execution slot, bounded by QueueTimeout. In config.go:

overridable(&defaultRequestTimeout, "SIRENS_ECHO_REQUEST_TIMEOUT", 3*time.Minute)
defaultQueueTimeout = defaultRequestTimeout / 6

The lane ran the packaged 3 minutes during that window, so QueueTimeout was exactly 30 seconds.

Observed: 30.56s, 30.44s, 30.47s. That is the timeout, plus handler overhead, three times.

On expiry replyQueueTimeout runs, and it is throttled:

if turn.Transport() == transportDiscord &&
    !a.limiter.notifyQueueTimeout(contextKey) {
    return
}
if err := turn.Reply(ctx, noticeWithTrace(ctx, noticeQueueTimeout)); err != nil {

notifyQueueTimeout is Burst: 1, Every: time.Second, keyed on the context, which is the channel. So when several turns in one channel give up inside the same second, the first is told and the rest return before Reply. That is the not_attempted, and it is why the count is 3 rather than 1: they queued behind the same slow turns and expired together.

It fits the rest of the window too. p95 community.turn was 182.2s on one execution slot, so a queue was guaranteed.

What is actually wrong here

The throttle is not a bug on its own. Its comment is right that a denial notice should not become a flood amplifier.

Applying it to a dropped admitted turn is the defect. Admission already ran and accepted these turns. The member did nothing wrong, the service was busy, and the throttle then punishes them for the collision by staying silent. A denial and a drop deserve different treatment: a denial says no, a drop says nothing after saying yes.

The reaction at the top of that function is meant to cover this, and in practice it did not: three members read it as being ignored.

Regression risk worth flagging now

coilyco-bridge/deploy#669 set SIRENS_ECHO_REQUEST_TIMEOUT: 5m on this lane, which is correct for its own reasons. It also moves QueueTimeout to 50 seconds, so a turn that would have dropped at 30s now waits 50s before dropping, and drops just as silently. That change made this issue's symptom slower and no less silent.

The fix

An admitted turn that gives up its slot always tells the member, regardless of the notice throttle. The throttle stays where it belongs, on denials.

Evidence for acceptance is a repeat of the burst with zero discord_failure: not_attempted.

**Correcting my previous comment. The roster root cause I posted is wrong.** The real one is simpler, and this issue's own evidence ruled mine out before I wrote it. ## Why the roster answer cannot be right I built a chain ending at `proxy.go`'s `c.Tools.Open`. That call sits inside `Complete`, which runs well after `runTurn` logs `turn.input.accepted` at `agent.go:1225`. **This issue states there is no `turn.input.accepted`.** So `runTurn` never began, and every link I described happens inside it. I read the code forward from a plausible mechanism instead of backward from the absent log line that was already written down here. My error. ## The actual root cause `runSerialized` waits for the single execution slot, bounded by `QueueTimeout`. In `config.go`: ```go overridable(&defaultRequestTimeout, "SIRENS_ECHO_REQUEST_TIMEOUT", 3*time.Minute) defaultQueueTimeout = defaultRequestTimeout / 6 ``` The lane ran the packaged 3 minutes during that window, so **QueueTimeout was exactly 30 seconds**. Observed: **30.56s, 30.44s, 30.47s**. That is the timeout, plus handler overhead, three times. On expiry `replyQueueTimeout` runs, and it is throttled: ```go if turn.Transport() == transportDiscord && !a.limiter.notifyQueueTimeout(contextKey) { return } if err := turn.Reply(ctx, noticeWithTrace(ctx, noticeQueueTimeout)); err != nil { ``` `notifyQueueTimeout` is `Burst: 1, Every: time.Second`, keyed on the **context**, which is the channel. So when several turns in one channel give up inside the same second, the first is told and **the rest return before `Reply`**. That is the `not_attempted`, and it is why the count is 3 rather than 1: they queued behind the same slow turns and expired together. It fits the rest of the window too. p95 `community.turn` was 182.2s on one execution slot, so a queue was guaranteed. ## What is actually wrong here The throttle is not a bug on its own. Its comment is right that a denial notice should not become a flood amplifier. Applying it to a **dropped admitted turn** is the defect. Admission already ran and accepted these turns. The member did nothing wrong, the service was busy, and the throttle then punishes them for the collision by staying silent. A denial and a drop deserve different treatment: a denial says no, a drop says nothing after saying yes. The reaction at the top of that function is meant to cover this, and in practice it did not: three members read it as being ignored. ## Regression risk worth flagging now coilyco-bridge/deploy#669 set `SIRENS_ECHO_REQUEST_TIMEOUT: 5m` on this lane, which is correct for its own reasons. It also moves **QueueTimeout to 50 seconds**, so a turn that would have dropped at 30s now waits 50s before dropping, and drops just as silently. That change made this issue's symptom slower and no less silent. ## The fix An admitted turn that gives up its slot always tells the member, regardless of the notice throttle. The throttle stays where it belongs, on denials. Evidence for acceptance is a repeat of the burst with zero `discord_failure: not_attempted`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
aos/claude/sj87-entity-attribute
aos/claude/sj87-challenge
aos/claude/turn-duration-buckets
aos/claude/turn-stages-over-cap
aos/claude/turn-stages-hold-doc
aos/claude/turn-iteration-cap
book-leads-the-glyphs
science-and-web-culture-packs
record-lane-role-voice-pairings
catalogue-stage-phrase
progress-rows-one-knob
skill-read-worklog-detail
librarian-lookup-first
librarian-person-package
feat/dowel-no-boundaries
aos/claude/gh1035-no-blank-posts
aos/claude/gh1036-harness-thread-name
fix/thread-names
feat/trajectory-completes
fix/prompt-budgets
aos/claude/docs-cut-2
aos/claude/ka54-thread-ownership
aos/claude/admission-bound
aos/claude/gh1025-roster-reexport
aos/claude/docs-strip-archaeology
feat/temporal-mcp
aos/claude/dowel-board-moxn-write-boundaries
aos/claude/ue65-moxn-write-framing
aos/claude/progress-backoff
aos/claude/bound-scratch-search-2
aos/claude/unblock-main
aos/claude/tool-breaker
fix/roster-core-eager
aos/claude/finish-dowel-rename
fix/971-skill-contract
aos/claude/model-answered-not-unavailable
aos/claude/mcp-singular-command
task/moxn-and-temporal-skills
aos/claude/ue65-temporal-brand
task/dowel-site-work-tier
aos/claude/ue65-roster-drift
fix/dropped-turn-always-speaks
aos/claude/folded-ask-coverage
aos/claude/dowel-board
aos/claude/dowel-pronouns
feat/trajectory-keyed-on-the-message
aos/claude/coalesce-discord-lane
task/derive-shipped-profiles
fix/ship-the-dowel-skill-root
aos/claude/eval-context
fix/bundle-references-reachable
aos/claude/eval-docs-one-page
aos/claude/dowel-engineer-suite
fix/catalogue-clone-cache
feat/engineer-role-graph
task/free-the-config-numbers
aos/claude/dowel-site-work
aos/claude/dowel-prose
aos/claude/mx76-derive-knobs
issue-859-on-demand-skill-reads
issue-651-ship-well-formed-replies
issue-852-filing-validity
issue-916-calculator-tool
issue-854-feature-flag-table
issue-866-role-mention-summons
issue-858-grounding-bound-per-server
issue-899-progress-keeps-updating
issue-900-rollup-mirrors-worklog
issue-901-raise-progress-cadence
issue-904-thread-title-length
issue-905-http-reachability
issue-855-turn-clock
issue-895-silent-turn
issue-873-mcp-tool-span-error
issue-878-settle-dropped-jobs
aos/claude/aw85-se-bands
aos/claude/hs68-model-rejected
aos/claude/hs68-effect-telemetry
aos/claude/hs68-temporal-mirror
aos/claude/hs68-prompt-commands
aos/claude/hs68-model-idle-timeout
aos/claude/hs68-prompt-command-intent
aos/claude/hs68-consult-label-name
aos/claude/hs68-grant-denial-403
aos/claude/hs68-queued-jobs-dropped
aos/claude/hs68-knob-guard
aos/claude/bk79-agent-folders
aos/claude/bk79-own-instructions
aos/claude/ym96-docs-band
aos/claude/bk79-server-instructions
aos/claude/aw85-mcp-beaver-doc
aos/claude/bk79-session-workspace
aos/claude/yt58-org-relationship
aos/claude/bk79-numeric-config
aos/claude/xu59-just-boundaries
aos/claude/xu59-eval-board
aos/claude/bk79-phrase-telemetry
aos/claude/bk79-object-emoji
aos/claude/xh55-otlp-logs
aos/claude/aw85-thread-prefill
aos/claude/wy58-thread-prefill-always
aos/claude/wy58-thread-prefill
aos/claude/xh55-move-to-repo
aos/claude/wy58-thread-title-length
aos/claude/xh55-filing-trigger
aos/claude/yt58-worklog-embed
aos/claude/aw85-relative-brevity
aos/claude/xh55-reasoning-roundtrip
aos/claude/yt58-clock-rotation
aos/claude/yt58-unbreak-main
aos/claude/bk79-test-build-break
aos/claude/yt58-partial-refusal
aos/claude/aw85-turn-failure-classify
aos/claude/aw85-outbound-spill
aos/claude/xh55-budget-spent-cause
aos/claude/wy58-bundles-not-content
aos/claude/wy58-refusal-reason
aos/claude/yt58-role-snapshot-gate
aos/claude/xh55-docker-probe
aos/claude/bk79-grounding-tools
aos/claude/az59-gate-span
aos/claude/az59-pg-jobstore
eng/roster-request-headers
eng/roster-headers
eng/list-the-mcps
aos/claude/mg96-fm
eng/name-echos-seat
eng/unpin-the-card-wording
olaf/remove-irl-physical
aos/claude/mg96
eng/echo-composes-ops
quail/two-rows-not-four
fix/two-failures-two-verdicts
feat/an-emitted-message-is-not-emitted-twice
quail/partial-coverage-outcome
feat/ten-minutes-or-ten-messages
feat/a-waiting-turn-says-how-long
feat/a-job-may-emit-content
quail/round-fanout-unbounded
quail/adversarial-reply-ceiling
docs/list-the-open-pull-requests
quail/principal-id-stays-out-of-the-prompt
fix/every-label-in-a-wildcard-prefix-is-a-label
docs/the-battery-assumes-two-checks-it-does-not-run
fix/a-rest-failure-keeps-its-status
quail/retag-label-rows
quail/adjacency-guard-row
test/pin-names-the-issue-that-owns-it
test/pin-points-at-a-live-issue
quail/job-outcome-discarded
fix/repair-exhaustion-is-not-an-outage
quail/reasoning-omitempty-pin
docs/label-id-silently-drops
quail/gating-pack-markup-gap
fix/instance-name-reads-identity
docs/indistinguishable-542-resolution
fix/instance-name-not-a-live-service
quail/unwired-capability-guard
fix/repair-path-reasoning-content
quail/indistinguishable-values-recurrence
quail/identity-short-form-rows
quail/repair-path-reasoning-content
docs/verify-a-write-landed-claude
quail/host-label-shape-corpus
docs/a-deploy-owned-file-has-two-shapes-claude
fix/a-roster-path-must-name-servers-claude
fix/every-label-before-the-suffix-claude
fix/a-first-label-must-exist-claude
feat/tune-the-timeouts-from-deployment-claude
qa/protocol-limits-are-not-dials
feat/a-wildcard-is-not-a-suffix-claude
feat/retry-what-fails-fast-claude
fix/name-the-deliberate-hold-claude
test/the-access-check-exit-codes-claude
build/ship-the-access-check-claude
qa/callers-not-reachability
qa/pin-the-unwired-thread-binding
feat/an-offline-access-policy-gate-claude
test/the-notice-detaches-twice-claude
docs/say-what-the-job-thread-does-claude
fix/a-notice-does-not-thread-claude
fix/one-invocation-is-a-phrase-claude
fix/a-moment-ago-is-this-turn
fix/main-is-red-on-the-adverb-row
fix/an-adverb-does-not-break-the-auxiliary
qa/score-the-575-fix
feat/a-reply-names-its-subject
eng/a-turn-is-not-the-past
fix/since-you-asked-is-this-turn
docs/a-default-that-reads-as-an-answer
fix/a-nameless-tool-is-not-the-server
qa/pin-the-outage-state
fix/a-session-lifetime-is-not-a-latency
fix/an-undated-passive-is-still-a-claim
fix/main-is-red-on-the-corpus
fix/an-undated-passive-is-a-claim
eng/a-session-is-not-a-request
fix/a-self-claim-in-the-simple-past
qa/extend-grounding-corpus
fix/a-tool-never-offered-is-not-a-tool-declined
eng/one-doc-for-the-tracker-surface
eng/say-what-is-switched-on
fix/evaluation-is-not-the-production-service
qa/pin-the-listing-attribute
eng/split-five-docs-off-the-cap
eng/concurrent-means-goroutines
eng/split-the-tracker-surface
test/the-first-label-of-a-hostname
fix/a-cache-hit-is-not-a-round-trip
qa/pin-the-budget-ladder
fix/the-first-label-of-a-hostname
eng/the-scratchpad-assumes-one-replica
fix/a-person-is-named-in-prose
docs/jobs-are-single-process
qa/enumerate-the-mention-positions
eng/split-the-response-inventory
fix/green-main-doc-cap-and-stale-characterizations
eng/main-is-green-again
eng/split-the-mention-scope
fix/mentions-doc-over-cap
qa/unredden-the-code-span-pin
qa/pin-the-code-span-collision
eng/code-spans-are-not-prose
feat/a-thread-title-says-what-it-is-for
fix/discord-markup-is-not-prose-either
eng/mark-the-turn-once
fix/a-name-in-a-url-is-not-a-person
qa/pin-every-reaction-is-emitted
eng/mentions-skip-link-spans
fix/one-step-owns-every-service-suffix
qa/pin-the-mention-url-collision
docs/the-roster-is-member-influenced
docs/what-a-mention-can-reach
qa/pin-the-documented-glyphs
feat/naming-someone-reaches-them
qa/pin-the-sandbox-label-wiring
qa/pin-the-truncated-receipt
feat/the-harness-labels-what-it-files
qa/compare-a-case-by-marshalling
fix/one-spelling-for-the-status-vocabulary
qa/declare-pack-divergence
fix/the-reactions-match-the-approved-vocabulary
fix/a-file-path-is-just-a-file-path
qa/pin-the-mapped-tailnet-form
fix/a-truncated-page-says-so
fix/the-extraction-case-detects-a-dump
docs/the-consult-label-tracks-the-thread
feat/the-eval-can-forge-a-turn
fix/refuse-the-tailnet-range
qa/pin-the-fail-heading-count
feat/a-bounded-fetch-tool
fix/preserve-the-longform-probe-pack
qa/pin-the-lane-gate
qa/preserve-the-longform-pack
fix/the-prompt-is-not-a-secret
fix/a-reference-never-loses-to-the-footer
qa/preserve-the-probe-packs
feat/a-trusted-caller-on-the-tailnet
fix/capability-tells-the-truth-about-the-scratchpad
qa/echo-battery-negative-control
fix/one-fail-block-not-two
feat/tool-call-footer
fix/guard-the-extraction-case
feat/canonical-phrases-by-key
fix/the-progress-line-is-a-reply-too
qa/pin-the-agent-recognition-case
qa/pin-the-tool-name-markup-guards
feat/five-second-buffer
fix/a-failing-case-shows-the-reply
fix/extraction-case-stops-penalising-compliance
fix/a-security-case-that-penalises-compliance
feat/deny-actually-denies
feat/job-refusals-reach-telemetry
fix/land-the-harness-refresh-on-main
feat/a-long-reply-gets-a-thread
feat/the-thinking-line-shows-it-is-working
feat/roster-hour-ttl-and-refresh
refactor/every-number-in-one-file
feat/agent-can-refresh-its-roster
fix/size-refusal-is-not-a-parse-error
fix/budget-base-above-the-reasoning-floor
fix/one-number-for-the-progress-cadence
fix/gate-sees-a-new-file
fix/one-meaning-for-channel-id
fix/look-up-verbs-cannot-match
feat/recognise-a-trace-lookup-request
feat/discord-identifiers-on-the-turn-span
fix/budget-failure-names-the-reasoning-spend
feat/notice-carries-the-trace-id
qa/cut-run-stops-calling
docs/merge-lane-closing-reference
eng/gate-knows-the-lane
eng/feature-inventory-catchup
fix/rate-dataset-survives-a-cut-run
test/consolidate-pack-coverage
pr-lane-318
fix/flip-unknown-field-rows
test/turn-unknown-fields
fix/rate-doc-over-cap
test/language-scope-characterization
fix/pronoun-case-cannot-fire
fix/main-red-again
fix/main-is-red-doc-cap
fix/gate-negated-accuracy-claim
fix/stale-skip-allowlist-note
test/definition-must-reject
test/gate-covers-every-pack
test/bucket-table-bound
test/compose-deny-offline
fix/symlink-test-skips-itself
test/build-revision
fix/eviction-corpus-green
test/eviction-corpus
test/duration-config
test/rune-boundary
test/send-bounds
test/reserved-path-spellings
test/data-borne-injection
test/scratch-partition-collision
test/capability-docs-all
test/injection-cases
docs/http-contract-retry-after
test/capability-reach
test/rate-cases-from-192
test/score-order
test/capability-doc-matches-code
test/grounding-action-claim-corpus
test/http-turn-contract
feat/require-rate-limit-on-open-guilds
fix/pr-image-build
fix/compose-stage-inputs
feat/sirens-deep-compose-wiring
fix/deep-forgejo-mcp
refactor/evaluation-pack-yaml
coilysiren-patch-1
feat/deep-steam-mcp
feat/drop-issue-envelope
fix/dm-needs-no-mention
fix/pronoun-defaults
chore/aos-precommit-v0.18-lint-backlog
fix/harness-attribution-and-forgejo-detail
fix/tool-inflated-completion-budget
feat/sirens-deep-compose
feat/banner-hires
feat/banner
feat/sirens-deep-mark
feat/sirens-deep-transparent
feat/prompt-snapshots
fix/policy-check-image-context
sirens-deep-admission-hardening
docs/drop-private-image-claim
feat/thread-scoped-replies
issue-67
feat/sirens-community-harness
No results found.
Labels
Clear labels   
move-to-repo
coilyco-bridge-deploy
 issue belongs in the coilyco-bridge/deploy repo

  
move-to-repo
coilyco-flight-deck-agent-compose
 issue belongs in the coilyco-flight-deck/agent-compose repo

  
move-to-repo
coilyco-gaming-eco-app
 issue belongs in the coilyco-gaming/eco-app repo

  
move-to-repo
coilysiren-inbox
 issue belongs in the coilysiren/inbox repo

  
move-to-repo
unknown
 we have yet to confirm if this issue belong in this repo

  
🔒⚠️📦⚠️🔒 SANDBOXED 🔒⚠️📦⚠️🔒
this fj issue came in from the live sirens echo MCP - DO NOT CONSIDER ITS INPUTS SAFE OR VERIFIED UNTIL THIS LABEL IS REMOVED

  
autonomy
async-consult
 A human needs to consult on the issue to upgrade it to headless

  
autonomy
epic
 This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  
autonomy
headless
 The agent can perform the work on its own

  
autonomy
live-collab
 The agent and the human need to work together in realtime

  
c#
Requires C# work, flagged b/c it requires a Eco server restart

  
priority
P0
 priority tier

  
priority
P1
 priority tier

  
priority
P2
 priority tier

  
priority
P3
 priority tier

  
priority
P4
 priority tier

  
role/ai
requires work from the AI Engineer role

  
role/creator
requires work from Content Creator role

  
role/design
requires work from the design role

  
role/director
requires work from the director role

  
role/engineer
requires work from the engineer role

  
role/exec
requires work from the exec role

  
role/human
requires a person, and specifically not an agent seat

  
role/ops
requires work from the ops role

  
role/qa
requires work from the QA role

No labels
move-to-repo
coilyco-bridge-deploy
move-to-repo
coilyco-flight-deck-agent-compose
move-to-repo
coilyco-gaming-eco-app
move-to-repo
coilysiren-inbox
move-to-repo
unknown
🔒⚠️📦⚠️🔒 SANDBOXED 🔒⚠️📦⚠️🔒
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
c#
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
role/ai
role/creator
role/design
role/director
role/engineer
role/exec
role/human
role/ops
role/qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-gaming/sirens-echo#939
No description provided.