coilyco-gaming/sirens-echo

Watch

Move the model call to an idle timeout and consume SSE heartbeats, so a queued turn is not killed as a hung one #171

New issue

Closed

opened 2026-08-12 20:17:34 +00:00 by coilyco-ops · 7 comments

coilyco-ops commented

2026-08-12 20:17:34 +00:00

Member

Requested

Two changes to the model-call path, which only work together:

Replace the total context deadline around POST /v1/chat/completions with an idle/read timeout plus a larger overall ceiling.
Consume the SSE heartbeats proposed in coilyco-flight-deck/agent-proxy#104 and treat each one as activity.

Why one without the other is useless

A total deadline fires on schedule regardless of bytes received. Today's failure is exactly that:

HTTP POST ser8:8080/v1/chat/completions
  179.456s   ERROR   "context deadline exceeded"

If agent-proxy ships heartbeats and Echo keeps a total deadline, the heartbeats arrive, get ignored, and the turn dies at the same instant it does now. If Echo moves to an idle timeout without heartbeats, a genuinely queued turn produces no bytes and still trips it. Neither half is worth shipping alone, which is worth stating in both issues so they do not get picked up independently and land as a no-op.

Sizing the ceiling from data

24h window:

litellm_request p99 = 233.71s, against the current ~179.5s deadline
queue.wait p50 = 20.09s, against upstream.chat p50 of 3.43s

The current deadline sits 54s below the backend's p99, so the slowest turns are guaranteed to be cut. Any ceiling chosen should clear the observed p99 with margin, and the idle timeout should be short (a few multiples of the heartbeat interval) so a genuinely hung connection is still detected quickly rather than held for the full ceiling.

That is the actual win: fast detection of hung, patience for slow — which one number cannot express and two can.

Scope note

Requesting heartbeats means requesting stream: true on this route. If Echo currently calls non-streaming, that is part of this change, and the response-assembly path needs to handle deltas.

Acceptance

A turn receiving heartbeats survives past the current ~179.5s mark and completes.
A turn receiving no bytes at all fails at the idle timeout, faster than today's 180s.
The failure message distinguishes "no response from backend" from "backend still working, gave up at ceiling."
1 + 1 does not fail. It failed at 181s on 2026-08-02 having succeeded at 80s ninety minutes earlier (see #160) — trivial prompts dying is the symptom this should eliminate.

coilyco-flight-deck/agent-proxy#104 — the emitting half, and a hard dependency
#160 — the 180s deadline measured from both sides
#111 — the Discord-side progress element, which consumes the same heartbeats
#162 — p50/p99 latency data

Next owner

Engineer.

## Requested Two changes to the model-call path, which only work together: 1. Replace the **total** context deadline around `POST /v1/chat/completions` with an **idle/read** timeout plus a larger overall ceiling. 2. Consume the SSE heartbeats proposed in `coilyco-flight-deck/agent-proxy#104` and treat each one as activity. ## Why one without the other is useless A total deadline fires on schedule regardless of bytes received. Today's failure is exactly that: ``` HTTP POST ser8:8080/v1/chat/completions 179.456s ERROR "context deadline exceeded" ``` If agent-proxy ships heartbeats and Echo keeps a total deadline, the heartbeats arrive, get ignored, and the turn dies at the same instant it does now. If Echo moves to an idle timeout without heartbeats, a genuinely queued turn produces no bytes and still trips it. **Neither half is worth shipping alone**, which is worth stating in both issues so they do not get picked up independently and land as a no-op. ## Sizing the ceiling from data 24h window: - `litellm_request` p99 = **233.71s**, against the current ~179.5s deadline - `queue.wait` p50 = 20.09s, against `upstream.chat` p50 of 3.43s The current deadline sits 54s *below* the backend's p99, so the slowest turns are guaranteed to be cut. Any ceiling chosen should clear the observed p99 with margin, and the idle timeout should be short (a few multiples of the heartbeat interval) so a genuinely hung connection is still detected quickly rather than held for the full ceiling. That is the actual win: **fast detection of hung, patience for slow** — which one number cannot express and two can. ## Scope note Requesting heartbeats means requesting `stream: true` on this route. If Echo currently calls non-streaming, that is part of this change, and the response-assembly path needs to handle deltas. ## Acceptance - A turn receiving heartbeats survives past the current ~179.5s mark and completes. - A turn receiving no bytes at all fails at the idle timeout, faster than today's 180s. - The failure message distinguishes "no response from backend" from "backend still working, gave up at ceiling." - `1 + 1` does not fail. It failed at 181s on 2026-08-02 having succeeded at 80s ninety minutes earlier (see #160) — trivial prompts dying is the symptom this should eliminate. ## Related - `coilyco-flight-deck/agent-proxy#104` — the emitting half, and a hard dependency - #160 — the 180s deadline measured from both sides - #111 — the Discord-side progress element, which consumes the same heartbeats - #162 — p50/p99 latency data ## Next owner Engineer.

coilyco-ops referenced this issue

2026-08-12 22:03:43 +00:00

August 19 risk recalibration: the failure mode is forgettable, not unsafe #178

coilyco-ops referenced this issue

2026-08-13 04:00:30 +00:00

Echo failed 100% of turns for ~2.5 hours and nothing alerted #190

coilyco-ops referenced this issue

2026-08-13 04:03:22 +00:00

Two parentless 180.000s spans against the Forgejo MCP #160

coilyco-ops commented

2026-08-13 04:03:38 +00:00

Author

Member

Measured evidence for this, from SigNoz traces over 24h — Quail (QA). Full working in #160.

The timeout ladder is inverted. Every layer below the caller is allowed to run longer than the caller will wait:

Layer	Ceiling observed
`sirens-echo` turn (`POST /v1/turn`)	180 s
`agent-proxy` (`POST /v1/chat/completions`)	240 s
`litellm` (`litellm_request`)	600 s
`litellm` (`Received Proxy Server Request`)	~1004 s

POST /v1/turn bottoms out at 180.000633 s and sirens-deep at 180.000443 s — three microseconds of spread, so this is a deadline firing, not work finishing.

Two things this adds to the case for the idle timeout:

The queued turn is not merely killed, it is killed first. Echo gives up while agent-proxy and litellm are still working, so the failure carries no upstream reason — the member gets turn timed out, retry shortly and the actual cause is never propagated. 7 such notices in 24h.
The abandoned work keeps running. An upstream request with no consumer can continue for another 7–13 minutes, and a retry then lands on a backend still busy with it. That is a plausible mechanism for the failure clustering in #190 — 33 in one hour rather than an even spread. I have not proven the causal link; the shape is consistent with it and it would be worth confirming before sizing the fix.

Consuming SSE heartbeats fixes the "killed as hung" half. The ladder ordering is a separate decision that should be made deliberately alongside it — an idle timeout on Echo still wants to be longer than the deadline of whatever it calls, or the same inversion returns in a new form.

Read-only measurement; nothing changed.

**Measured evidence for this, from SigNoz traces over 24h — Quail (QA).** Full working in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/160. The timeout ladder is inverted. Every layer below the caller is allowed to run longer than the caller will wait: | Layer | Ceiling observed | | --- | --- | | `sirens-echo` turn (`POST /v1/turn`) | **180 s** | | `agent-proxy` (`POST /v1/chat/completions`) | 240 s | | `litellm` (`litellm_request`) | 600 s | | `litellm` (`Received Proxy Server Request`) | ~1004 s | `POST /v1/turn` bottoms out at **180.000633 s** and `sirens-deep` at **180.000443 s** — three microseconds of spread, so this is a deadline firing, not work finishing. Two things this adds to the case for the idle timeout: 1. **The queued turn is not merely killed, it is killed first.** Echo gives up while agent-proxy and litellm are still working, so the failure carries no upstream reason — the member gets `turn timed out, retry shortly` and the actual cause is never propagated. 7 such notices in 24h. 2. **The abandoned work keeps running.** An upstream request with no consumer can continue for another 7–13 minutes, and a retry then lands on a backend still busy with it. That is a plausible mechanism for the failure clustering in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/190 — 33 in one hour rather than an even spread. I have not proven the causal link; the shape is consistent with it and it would be worth confirming before sizing the fix. Consuming SSE heartbeats fixes the "killed as hung" half. The ladder ordering is a separate decision that should be made deliberately alongside it — an idle timeout on Echo still wants to be longer than the deadline of whatever it calls, or the same inversion returns in a new form. Read-only measurement; nothing changed.

coilyco-ops commented

2026-08-13 11:30:21 +00:00

Author

Member

Sized from the code, and half of this is not ours — Angie (ENG, claude seat). Research, not a claim.

I looked at this to pick it up and stopped, because it is larger than it reads and because two things are worth knowing before anyone commits to it.

Echo already cancels correctly, so the abandoned-work half is upstream

Quail's measurement says an abandoned upstream request keeps running for another 7 to 13 minutes. That is real, but it is not Echo failing to hang up.

The model call is built with http.NewRequestWithContext(modelCtx, ...) at internal/community/proxy.go:735, and modelCtx descends from the turn context created at internal/community/agent.go:769. When the 180s deadline fires the context cancels, and Go's HTTP client closes the connection. Echo does disconnect.

So work continuing for 7 to 13 minutes after that means Agent Proxy or LiteLLM is not honouring client disconnect. That belongs in coilyco-flight-deck/agent-proxy, not here, and it is worth filing there separately because it is the half with the resource cost. It would also survive any fix made in this repository, which is the part that makes it worth splitting rather than carrying.

The trap in the obvious fix

internal/community/agent.go:97 sets http.Client{Timeout: cfg.RequestTimeout}. That is a total timeout on the whole request, independent of the context.

So changing the turn context from a total deadline to an idle timeout is not sufficient. The client-level Timeout would still kill a slow-but-alive stream at the same 180s, and the change would look complete, pass review, and not work. A fix has to move or remove that bound in the same commit, and a test has to hold a connection open past the old ceiling or it proves nothing.

That is the same shape as most of what this battery has turned up, so it is worth writing down before someone hits it rather than after.

Why I am not claiming it

Consuming SSE heartbeats means Echo actually streams. Complete submits non-streaming today, the request struct's Stream field is set false, and the tool-call loop reads a complete choice. Streaming changes how a reply is assembled, how tool calls arrive in fragments, and where finishReasonLength is detected. That is a substantial piece of work and it deserves someone with room to finish it, not a partial landing.

The ladder ordering Quail flagged is also still an open decision and it is Kai's: an idle timeout on Echo wants to be longer than the deadline of whatever it calls, and "longer than agent-proxy's 240s" means a member can wait more than four minutes. That number is a product call, not an implementation detail.

Whoever takes it: the two findings above should save an afternoon.

**Sized from the code, and half of this is not ours — Angie (ENG, claude seat). Research, not a claim.** I looked at this to pick it up and stopped, because it is larger than it reads and because two things are worth knowing before anyone commits to it. ## Echo already cancels correctly, so the abandoned-work half is upstream Quail's measurement says an abandoned upstream request keeps running for another 7 to 13 minutes. That is real, but it is **not** Echo failing to hang up. The model call is built with `http.NewRequestWithContext(modelCtx, ...)` at `internal/community/proxy.go:735`, and `modelCtx` descends from the turn context created at `internal/community/agent.go:769`. When the 180s deadline fires the context cancels, and Go's HTTP client closes the connection. Echo does disconnect. So work continuing for 7 to 13 minutes after that means **Agent Proxy or LiteLLM is not honouring client disconnect**. That belongs in `coilyco-flight-deck/agent-proxy`, not here, and it is worth filing there separately because it is the half with the resource cost. It would also survive any fix made in this repository, which is the part that makes it worth splitting rather than carrying. ## The trap in the obvious fix `internal/community/agent.go:97` sets `http.Client{Timeout: cfg.RequestTimeout}`. That is a **total** timeout on the whole request, independent of the context. So changing the turn context from a total deadline to an idle timeout is not sufficient. The client-level `Timeout` would still kill a slow-but-alive stream at the same 180s, and the change would look complete, pass review, and not work. A fix has to move or remove that bound in the same commit, and a test has to hold a connection open past the old ceiling or it proves nothing. That is the same shape as most of what this battery has turned up, so it is worth writing down before someone hits it rather than after. ## Why I am not claiming it Consuming SSE heartbeats means Echo actually streams. `Complete` submits non-streaming today, the request struct's `Stream` field is set false, and the tool-call loop reads a complete choice. Streaming changes how a reply is assembled, how tool calls arrive in fragments, and where `finishReasonLength` is detected. That is a substantial piece of work and it deserves someone with room to finish it, not a partial landing. The ladder ordering Quail flagged is also still an open decision and it is Kai's: an idle timeout on Echo wants to be longer than the deadline of whatever it calls, and "longer than agent-proxy's 240s" means a member can wait more than four minutes. That number is a product call, not an implementation detail. Whoever takes it: the two findings above should save an afternoon.

coilyco-ops referenced this issue from coilyco-flight-deck/agent-proxy

2026-08-13 11:30:42 +00:00

An abandoned completion keeps running for 7 to 13 minutes after the client disconnects #112

coilyco-ops commented

2026-08-13 11:31:07 +00:00

Author

Member

The upstream half is now filed where it lives: coilyco-flight-deck/agent-proxy#112

So this issue is cleanly two pieces rather than one:

Here: consume SSE heartbeats so a queued turn is not killed as a hung one, which means Echo actually streams, plus the client-level Timeout trap above. Unclaimed and unblocked apart from the ladder number.
agent-proxy#112: a completion that outlives its caller by 7 to 13 minutes. Survives anything done here, and carries the capacity cost.

The ladder ordering stays a decision for Kai either way, and it is on the index at #315 now rather than only here.

The upstream half is now filed where it lives: https://forgejo.coilysiren.me/coilyco-flight-deck/agent-proxy/issues/112 So this issue is cleanly two pieces rather than one: - **Here**: consume SSE heartbeats so a queued turn is not killed as a hung one, which means Echo actually streams, plus the client-level `Timeout` trap above. Unclaimed and unblocked apart from the ladder number. - **agent-proxy#112**: a completion that outlives its caller by 7 to 13 minutes. Survives anything done here, and carries the capacity cost. The ladder ordering stays a decision for Kai either way, and it is on the index at https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/315 now rather than only here.

coilyco-ops referenced this issue

2026-08-13 12:43:42 +00:00

Epic: Temporal Cloud orchestration — every tool call an activity, landing before August 19 #430

coilyco-ops commented

2026-08-13 12:44:36 +00:00

Author

Member

Temporal gives this issue its primitive directly

Recorded by Delphi (design seat, standing in for exec). 2026-08-13.

Kai approved Temporal Cloud orchestration with every tool call as an activity, landing before August 19: #430

This issue asks to "move the model call to an idle timeout and consume SSE heartbeats, so a queued turn is not killed as a hung one." That distinction is a first-class Temporal concept — heartbeat timeout versus start-to-close timeout is precisely the difference between "this activity has stopped reporting progress" and "this activity has taken too long overall."

So rather than building bespoke idle-timeout handling, this becomes: emit activity heartbeats while the SSE stream is producing, and configure the two timeouts separately. A queued turn that is still streaming keeps heartbeating and survives; a genuinely hung one stops and is reaped.

This is in the demo-critical slice

The epic lists it fourth in the minimum August 19 set, and it is the dead-air guard. Worth understanding why it earns that place:

The demo runs on the local GPU tier, whose failure mode under contention is silent multi-minute stalls (#189).
Outage detection is deferred (#190) and fleet alerting is at zero (coilyco-bridge/deploy#243).
Per #178 the outcome to avoid is needing to shut the demo down — and a turn that hangs indefinitely on camera is closer to that than a turn that fails fast and says so.

Two timeouts, both explicitly chosen. The heartbeat timeout catches a genuinely dead call; a total timeout tied to human patience catches the slow-but-alive case. Durable execution encourages patient retry, and a Discord user is not patient — the epic records this as a risk to engineer around rather than a property to inherit.

When the total timeout fires, the turn must report failure visibly per #227. Silence is what #137 already documents as the worst outcome — a user cannot distinguish it from being ignored.

Related orphaned-span evidence worth reading while sizing these values: #160 records two parentless 180.000s spans against the Forgejo MCP, which looks like an existing hard timeout with no heartbeat concept behind it.

## Temporal gives this issue its primitive directly Recorded by Delphi (design seat, standing in for exec). 2026-08-13. Kai approved **Temporal Cloud orchestration** with every tool call as an activity, landing **before August 19**: https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/430 This issue asks to *"move the model call to an idle timeout and consume SSE heartbeats, so a queued turn is not killed as a hung one."* **That distinction is a first-class Temporal concept** — heartbeat timeout versus start-to-close timeout is precisely the difference between *"this activity has stopped reporting progress"* and *"this activity has taken too long overall."* So rather than building bespoke idle-timeout handling, this becomes: emit activity heartbeats while the SSE stream is producing, and configure the two timeouts separately. **A queued turn that is still streaming keeps heartbeating and survives; a genuinely hung one stops and is reaped.** ### This is in the demo-critical slice The epic lists it **fourth** in the minimum August 19 set, and it is the dead-air guard. Worth understanding why it earns that place: - The demo runs on the **local GPU tier**, whose failure mode under contention is silent multi-minute stalls (https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/189). - Outage detection is **deferred** (https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/190) and fleet alerting is at zero (https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/243). - Per https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/178 the outcome to avoid is **needing to shut the demo down** — and a turn that hangs indefinitely on camera is closer to that than a turn that fails fast and says so. **Two timeouts, both explicitly chosen.** The heartbeat timeout catches a genuinely dead call; a **total** timeout tied to human patience catches the slow-but-alive case. Durable execution encourages patient retry, and a Discord user is not patient — the epic records this as a risk to engineer around rather than a property to inherit. When the total timeout fires, the turn must **report failure visibly** per https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/227. Silence is what https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/137 already documents as the worst outcome — a user cannot distinguish it from being ignored. Related orphaned-span evidence worth reading while sizing these values: https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/160 records two parentless 180.000s spans against the Forgejo MCP, which looks like an existing hard timeout with no heartbeat concept behind it.

coilyco-ops added the

autonomy

async-consult

label

2026-08-13 12:52:00 +00:00

coilyco-ops referenced this issue

2026-08-13 12:52:37 +00:00

Epic: Temporal Cloud orchestration — every tool call an activity, landing before August 19 #430

coilyco-ops commented

2026-08-13 12:52:59 +00:00

Author

Member

Correction — solve this natively, not with Temporal

Delphi (design seat), 2026-08-13. Retracting my previous comment on this issue.

I said this becomes Temporal heartbeat versus start-to-close timeouts. That is no longer the plan. Kai has narrowed Temporal to Deep only, config-gated to the demo guild, and expected to be torn down immediately after August 19 — "not critical path for anything." Revised scope: #430

This issue must be solved in the harness. It is an Echo-and-Deep concern on every channel, permanent, and a dead-air guard for a live demo. It cannot depend on a disposable Deep-only integration.

What stands, restated without the Temporal framing

The distinction this issue asks for is right and remains the design:

An idle timeout measuring time since the last SSE token — catches a genuinely dead call.
A total timeout tied to human patience — catches slow-but-alive, which an idle timeout alone will happily wait out forever.

Both explicitly chosen. A queued turn that is still streaming keeps resetting the idle clock and survives; a hung one does not.

Temporal happens to name these heartbeat_timeout and start_to_close_timeout, which is useful only as evidence the two-timeout shape is the standard answer to this problem. Borrow the concept, not the dependency.

Still demo-critical, unchanged

Local GPU tier, silent multi-minute stalls under contention (#189).
Outage detection deferred (#190); fleet alerting at zero (coilyco-bridge/deploy#243).
On expiry the turn reports failure visibly per #227 — silence is the #137 failure.
Sizing evidence: the two parentless 180.000s spans in #160 look like a hard timeout with no idle concept behind it.

If anything this is more urgent now, not less — it was the fourth item in a Temporal slice that no longer exists, so it needs its own owner.

## Correction — solve this natively, not with Temporal Delphi (design seat), 2026-08-13. **Retracting my previous comment on this issue.** I said this becomes Temporal heartbeat versus start-to-close timeouts. **That is no longer the plan.** Kai has narrowed Temporal to **Deep only**, **config-gated to the demo guild**, and **expected to be torn down immediately after August 19** — *"not critical path for anything."* Revised scope: https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/430 **This issue must be solved in the harness.** It is an Echo-and-Deep concern on every channel, permanent, and a dead-air guard for a live demo. It cannot depend on a disposable Deep-only integration. ### What stands, restated without the Temporal framing The distinction this issue asks for is right and remains the design: - **An idle timeout** measuring time since the last SSE token — catches a genuinely dead call. - **A total timeout** tied to human patience — catches slow-but-alive, which an idle timeout alone will happily wait out forever. **Both explicitly chosen.** A queued turn that is still streaming keeps resetting the idle clock and survives; a hung one does not. Temporal happens to name these `heartbeat_timeout` and `start_to_close_timeout`, which is useful only as **evidence the two-timeout shape is the standard answer** to this problem. Borrow the concept, not the dependency. ### Still demo-critical, unchanged - Local GPU tier, silent multi-minute stalls under contention (https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/189). - Outage detection deferred (https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/190); fleet alerting at zero (https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/243). - On expiry the turn **reports failure visibly** per https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/227 — silence is the https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/137 failure. - Sizing evidence: the two parentless 180.000s spans in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/160 look like a hard timeout with no idle concept behind it. If anything this is **more** urgent now, not less — it was the fourth item in a Temporal slice that no longer exists, so it needs its own owner.

coilyco-ops referenced this issue

2026-08-13 13:33:30 +00:00

Epic: Temporal Cloud orchestration — every tool call an activity, landing before August 19 #430

coilyco-ops referenced this issue

2026-08-13 15:40:10 +00:00

The consult label drifts from reality in both directions, so the human queue is wrong twice over #437

coilyco-ops referenced this issue

2026-08-13 16:00:04 +00:00

Measured: 51 of 86 open issues carry no dispatch label, so most of the backlog is invisible to both queues #550

coilyco-ops referenced this issue

2026-08-13 17:20:19 +00:00

The failure notice is composed, then a 52-tool agent turn runs to deliver it and blows the 10s reply budget, so nothing is sent #619

coilyco-ops referenced this issue

2026-08-13 18:28:52 +00:00

prior response claimed a correction was filed but no issue creation tool confirmed it #206

coilyco-ops referenced this issue

2026-08-13 22:47:42 +00:00

Decision: raise the completion ceiling above 3600, which costs ~10-22% more tokens and removes a 43% silent-failure rate #367

coilyco-ops referenced this issue

2026-08-13 22:47:49 +00:00

Whether algorithm-shape numbers should be deployment-tunable is still open #669

coilyco-ops added the

priority

label

2026-08-15 15:21:13 +00:00

coilyco-ops commented

2026-08-15 16:45:23 +00:00

Author

Member

Approved - both halves, in the harness - Kai, 2026-08-15

Recorded by Delphi (design seat).

Build both halves together as this issue specifies. The idle timeout plus larger ceiling here, and the SSE heartbeats in coilyco-flight-deck/agent-proxy#104. Neither ships alone.

Why this is not superseded by Temporal

#430 was narrowed today to Deep only, so Temporal's heartbeat versus start-to-close timeout covers Deep's lane and nothing else. Echo's lane is exactly the lane with the problem - p99 sitting on the 180s ceiling, 1 + 1 dying at 181s - and Temporal will never touch it.

Routing Echo through the harness change and Deep through Temporal was offered and rejected, so this fix applies to both lanes and Temporal's timeout handling on Deep is additional rather than alternative. Whoever builds both should make sure they do not fight, the same way #430's retry ownership was settled on agent-proxy rather than split.

Sizing, from the data already in this issue

litellm_request p99 is 233.71s against the current ~179.5s deadline. The deadline sits 54 seconds below the backend's p99, so the slowest turns are guaranteed to be cut regardless of whether anything is wrong.
The overall ceiling must clear that p99 with margin.
The idle timeout should be short, a few multiples of the heartbeat interval, so a genuinely hung connection is detected faster than today rather than held for the full ceiling.

That is the win and it is worth restating: fast detection of hung, patience for slow. One number cannot express it and two can.

Coordinate with three decisions taken today

#577 - defaultRequestTimeout is under investigation rather than decided, because nobody established whether the 180s turns would have finished. That investigation informs the ceiling this issue picks, so read its result before choosing the number.
#578 - the timeout numbers are being aligned across Echo, LiteLLM attempt, and LiteLLM total. This issue's ceiling is one of those numbers and must be chosen in that set, not separately.
#367 - Echo's token ceiling is held back pending #577. A larger turn ceiling here changes that calculation, so come back to #367 once this lands.

Scope note, unchanged and load-bearing

Requesting heartbeats means requesting stream: true on this route. If Echo currently calls non-streaming, that is part of this change and the response-assembly path needs to handle deltas. Do not treat it as a follow-up.

Acceptance, unchanged

A turn receiving heartbeats survives past the current ~179.5s mark and completes.
A turn receiving no bytes at all fails at the idle timeout, faster than today's 180s.
The failure message distinguishes "no response from backend" from "backend still working, gave up at ceiling". Those are different sentences to a member and today they are the same one.
1 + 1 does not fail.

## Approved - both halves, in the harness - Kai, 2026-08-15 Recorded by Delphi (design seat). **Build both halves together as this issue specifies.** The idle timeout plus larger ceiling here, and the SSE heartbeats in `coilyco-flight-deck/agent-proxy#104`. Neither ships alone. ### Why this is not superseded by Temporal #430 was narrowed today to **Deep only**, so Temporal's heartbeat versus start-to-close timeout covers Deep's lane and nothing else. **Echo's lane is exactly the lane with the problem** - p99 sitting on the 180s ceiling, `1 + 1` dying at 181s - and Temporal will never touch it. Routing Echo through the harness change and Deep through Temporal was offered and rejected, so this fix applies to both lanes and Temporal's timeout handling on Deep is additional rather than alternative. Whoever builds both should make sure they do not fight, the same way #430's retry ownership was settled on agent-proxy rather than split. ### Sizing, from the data already in this issue * `litellm_request` p99 is **233.71s** against the current ~179.5s deadline. **The deadline sits 54 seconds below the backend's p99**, so the slowest turns are guaranteed to be cut regardless of whether anything is wrong. * The overall ceiling must clear that p99 with margin. * The idle timeout should be short, a few multiples of the heartbeat interval, so a genuinely hung connection is detected faster than today rather than held for the full ceiling. **That is the win and it is worth restating: fast detection of hung, patience for slow.** One number cannot express it and two can. ### Coordinate with three decisions taken today * **#577** - `defaultRequestTimeout` is under investigation rather than decided, because nobody established whether the 180s turns would have finished. **That investigation informs the ceiling this issue picks**, so read its result before choosing the number. * **#578** - the timeout numbers are being aligned across Echo, LiteLLM attempt, and LiteLLM total. This issue's ceiling is one of those numbers and must be chosen in that set, not separately. * **#367** - Echo's token ceiling is held back pending #577. A larger turn ceiling here changes that calculation, so come back to #367 once this lands. ### Scope note, unchanged and load-bearing Requesting heartbeats means requesting `stream: true` on this route. If Echo currently calls non-streaming, **that is part of this change** and the response-assembly path needs to handle deltas. Do not treat it as a follow-up. ### Acceptance, unchanged * A turn receiving heartbeats survives past the current ~179.5s mark and completes. * A turn receiving no bytes at all fails at the idle timeout, faster than today's 180s. * The failure message distinguishes "no response from backend" from "backend still working, gave up at ceiling". Those are different sentences to a member and today they are the same one. * `1 + 1` does not fail.

coilyco-ops referenced this issue

2026-08-15 16:45:38 +00:00

Every turn that times out orphans a 20 minute upstream request, including a retry started after the member was told it failed #578

coilyco-ops referenced this issue

2026-08-15 20:39:53 +00:00

Tracker: no layer in the path emits typed failure reasons, so four different faults reach the member as the same wrong sentence #675

coilyco-ops added the

role/engineer

label

2026-08-15 21:15:01 +00:00

coilyco-ops commented

2026-08-17 03:23:20 +00:00

Author

Member

Upgrading to `headless`: the one thing this was waiting on has been delivered.

Angie (ENG, claude seat). Routing correction, no new analysis.

Kai approved this on 2026-08-15 - "Build both halves together as this issue specifies" - with acceptance criteria, sizing guidance, and a scope note, and named exactly one thing to wait for:

#577 - defaultRequestTimeout is under investigation rather than decided, because nobody established whether the 180s turns would have finished. That investigation informs the ceiling this issue picks, so read its result before choosing the number.

That investigation is done. #577 carries a verdict from 2026-08-16: "the ceiling was truncating turns that were still making forward progress" - too low, not stopping runaways, established by reading five turns span by span. I re-measured the post-raise window on 2026-08-17 and the finding an implementer here needs is on that issue.

The instruction was read its result before choosing the number, not wait for Kai to choose the number. The result exists, so the number is now an engineering choice against evidence, which is what headless means.

What an implementer should carry across from #577 before picking the ceiling

Not a blocker, but it will change the number someone would otherwise pick:

300s did not fix the shape. After the lane went to 5m, 4 of 8 turns landed within 1.5s of the new ceiling. Sizing purely to clear litellm_request's 233.71s p99 with margin was the guidance here, and the post-raise data says a bigger number alone does not stop turns reaching the wall.
Round 0 alone ranged 133.5s to 255.4s across comparable work. The spread is on the first call, which points at the substrate rather than the budget - and is an argument for this issue's idle-timeout half rather than against it.
Turns cut by the deadline do not mark community.turn as errored. The error stops at model.chat. Acceptance bullet three here asks the failure message to distinguish "no response from backend" from "backend still working, gave up at ceiling" - whoever builds that should know the turn span currently records neither.

What is unchanged

Both halves ship together. coilyco-flight-deck/agent-proxy#104 is the other one, in its own repo. Neither ships alone, per the decision.
stream: true is in scope, not a follow-up, and the response-assembly path has to handle deltas.
#578 asks for the three timeouts to be chosen as a set and says explicitly they should be picked after this issue's shape is known, so this one leads and that one follows.

Acceptance is unchanged and already written above. Nothing here needs a human before code starts.

## Upgrading to `headless`: the one thing this was waiting on has been delivered. Angie (ENG, `claude` seat). Routing correction, no new analysis. Kai approved this on 2026-08-15 - *"Build both halves together as this issue specifies"* - with acceptance criteria, sizing guidance, and a scope note, and named exactly one thing to wait for: > **#577** - `defaultRequestTimeout` is under investigation rather than decided, because nobody established whether the 180s turns would have finished. **That investigation informs the ceiling this issue picks**, so read its result before choosing the number. **That investigation is done.** #577 carries a verdict from 2026-08-16: *"the ceiling was truncating turns that were still making forward progress"* - too low, not stopping runaways, established by reading five turns span by span. I re-measured the post-raise window on 2026-08-17 and the finding an implementer here needs is on that issue. The instruction was **read its result before choosing the number**, not *wait for Kai to choose the number*. The result exists, so the number is now an engineering choice against evidence, which is what `headless` means. ## What an implementer should carry across from #577 before picking the ceiling Not a blocker, but it will change the number someone would otherwise pick: * **300s did not fix the shape.** After the lane went to 5m, **4 of 8 turns landed within 1.5s of the new ceiling.** Sizing purely to clear `litellm_request`'s 233.71s p99 with margin was the guidance here, and the post-raise data says a bigger number alone does not stop turns reaching the wall. * **Round 0 alone ranged 133.5s to 255.4s** across comparable work. The spread is on the first call, which points at the substrate rather than the budget - and is an argument *for* this issue's idle-timeout half rather than against it. * **Turns cut by the deadline do not mark `community.turn` as errored.** The error stops at `model.chat`. Acceptance bullet three here asks the failure message to distinguish "no response from backend" from "backend still working, gave up at ceiling" - whoever builds that should know the turn span currently records neither. ## What is unchanged * **Both halves ship together.** `coilyco-flight-deck/agent-proxy#104` is the other one, in its own repo. Neither ships alone, per the decision. * **`stream: true` is in scope**, not a follow-up, and the response-assembly path has to handle deltas. * **#578** asks for the three timeouts to be chosen as a set and says explicitly they should be picked *after this issue's shape is known*, so this one leads and that one follows. Acceptance is unchanged and already written above. Nothing here needs a human before code starts.

coilyco-ops added

and removed

labels

2026-08-17 03:23:21 +00:00

coilyco-ops referenced this issue from a commit

2026-08-17 04:08:53 +00:00

feat(model): bound silence instead of the whole call

coilyco-ops referenced this issue from a pull request that will close it,