coilyco-gaming/sirens-echo

Watch

The Temporal namespace shows 0 Completed and 60 TimedOut, which is by design and reads as broken on a Temporal stream #1041

New issue

Closed

opened 2026-08-19 03:51:47 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-19 03:51:47 +00:00

Member

Found while debugging the Temporal MCP at Kai's direction. Read-only, measured against the live namespace. The integration is working as written. The problem is what it looks like.

Measured

coilyco.gcdqf, counted through ListWorkflow with the lane's own credential:

ExecutionStatus='Running'        12
ExecutionStatus='Completed'       0
ExecutionStatus='Terminated'      1
ExecutionStatus='TimedOut'       60

Every trajectory that has reached a terminal state reached TimedOut. Not one has ever completed.

Why, and why it is not a defect

temporalmirror.go calls SignalWithStartWorkflow onto task queue sirens-dowel-tool-mirror, and there is no worker anywhere in this repository - no worker.New, no RegisterWorkflow, nothing polling that queue. So the workflow is created, accumulates signals, and is never executed.

That is deliberate, and the code says so:

// The turn is over long before this. A trajectory that never sees a
// second call must not sit open forever.
WorkflowExecutionTimeout: trajectoryLifetime,

Temporal is being used as a durable append-only event log. The signals are the payload, the history is the record, and the timeout is garbage collection. The data is genuinely there: get_workflow_history reads it back correctly.

Why it matters on Wednesday

The audience is a Temporal audience, on a stream whose whole subject is Temporal. If anyone opens that namespace, the Web UI shows a wall of red TimedOut executions and a zero completion rate. To a viewer who knows Temporal, that reads as an integration that is failing, and the true explanation - "those are deliberately unexecuted workflows used as a log" - is a harder thing to say live than it is to write here.

It also forecloses the more interesting demo. "Every tool the agent called is a durable, queryable execution" is a much better story with green terminal states than with timeouts.

Cheapest fix, for after the freeze

A worker that registers ToolTrajectory as a workflow which waits on signals and returns after a quiet period. That is a small amount of code and it turns 60 red terminal states into 60 green ones without changing what is recorded. The trajectory lifetime then becomes a bound rather than the normal exit.

I am not proposing this before the stream. It adds a worker process to the demo lane hours before air, and the current shape is at least honest and working. Filing so the choice is made deliberately rather than discovered on camera.

If it stays as-is

Worth having one sentence ready, since the question is likely: the trajectories are a durable log rather than orchestrated workflows, so they are signalled and never executed, and the timeout is how they retire. Better said in advance than improvised.

Found while debugging the Temporal MCP at Kai's direction. **Read-only, measured against the live namespace.** The integration is working as written. The problem is what it looks like. ## Measured `coilyco.gcdqf`, counted through `ListWorkflow` with the lane's own credential: ``` ExecutionStatus='Running' 12 ExecutionStatus='Completed' 0 ExecutionStatus='Terminated' 1 ExecutionStatus='TimedOut' 60 ``` Every trajectory that has reached a terminal state reached `TimedOut`. **Not one has ever completed.** ## Why, and why it is not a defect `temporalmirror.go` calls `SignalWithStartWorkflow` onto task queue `sirens-dowel-tool-mirror`, and **there is no worker anywhere in this repository** - no `worker.New`, no `RegisterWorkflow`, nothing polling that queue. So the workflow is created, accumulates signals, and is never executed. That is deliberate, and the code says so: ```go // The turn is over long before this. A trajectory that never sees a // second call must not sit open forever. WorkflowExecutionTimeout: trajectoryLifetime, ``` Temporal is being used as a durable append-only event log. The signals are the payload, the history is the record, and the timeout is garbage collection. The data is genuinely there: `get_workflow_history` reads it back correctly. ## Why it matters on Wednesday The audience is a Temporal audience, on a stream whose whole subject is Temporal. If anyone opens that namespace, the Web UI shows a wall of **red** `TimedOut` executions and a **zero** completion rate. To a viewer who knows Temporal, that reads as an integration that is failing, and the true explanation - "those are deliberately unexecuted workflows used as a log" - is a harder thing to say live than it is to write here. It also forecloses the more interesting demo. "Every tool the agent called is a durable, queryable execution" is a much better story with green terminal states than with timeouts. ## Cheapest fix, for after the freeze A worker that registers `ToolTrajectory` as a workflow which waits on signals and returns after a quiet period. That is a small amount of code and it turns 60 red terminal states into 60 green ones without changing what is recorded. The trajectory lifetime then becomes a bound rather than the normal exit. **I am not proposing this before the stream.** It adds a worker process to the demo lane hours before air, and the current shape is at least honest and working. Filing so the choice is made deliberately rather than discovered on camera. ## If it stays as-is Worth having one sentence ready, since the question is likely: *the trajectories are a durable log rather than orchestrated workflows, so they are signalled and never executed, and the timeout is how they retire.* Better said in advance than improvised.

coilyco-ops added the

labels

2026-08-19 04:07:01 +00:00

coilyco-ops commented

2026-08-19 04:08:28 +00:00

Author

Member

Correcting my own filing. I proposed a worker as the "cheapest fix" without having read #887, and #887 rules that out deliberately rather than by omission.

Design settled with Kai 2026-08-16. This is mirroring, not orchestration. Temporal observes the turn, it does not run it.

It rejects #430's turn-as-workflow shape for two specific hazards, both of which a worker would reintroduce:

Retry multiplication - a Temporal retry policy stacked on the agent-proxy fallback for sirens-echo/deepseek compounds, turning a 502 into a slow expensive 502.
Latency in the user-visible turn - an activity round trip per tool call sits in front of a Discord member, and dead air is the demo failure mode.

So "add a worker so the states go green" is not a small cosmetic change. It is a reversal of a settled architectural decision, and it would trade a presentation problem for the two failure modes that decision was made to avoid. Withdrawing that suggestion.

What survives from this issue

Only the presentation half, and it is smaller than I made it sound:

Running      12
Completed     0
TimedOut     60

A Temporal audience opening the namespace sees red terminal states. That is worth having an answer ready for, and #887 already wrote the answer better than I did:

"We mirror our agent's tool-call trajectory into Temporal so there is a durable audit record" is true and specific. "Our harness runs on Temporal" would not have been.

That is the sentence. The workflows are records rather than executions, they are signalled and never run on purpose, and the timeout is how a record retires.

Confirmed against the server, not just the source

DescribeTaskQueue on both lanes, workflow and activity types:

sirens-dowel-tool-mirror   Workflow pollers=0   Activity pollers=0
sirens-deep-tool-mirror    Workflow pollers=0   Activity pollers=0

So the absence is real and total rather than a worker living somewhere I did not grep. Which is exactly what #887 specifies.

What might genuinely be worth doing later

If the red states are worth removing at all, the option that does not touch the control path is a retention or archival setting on the namespace, or accepting TimedOut as the normal terminal state and saying so in the one sentence above. Neither adds a poller to the turn. Retitling this issue toward that, since "there is no worker" reads as a defect report and it is not one.

**Correcting my own filing.** I proposed a worker as the "cheapest fix" without having read #887, and #887 rules that out deliberately rather than by omission. > Design settled with Kai 2026-08-16. This is **mirroring, not orchestration.** Temporal observes the turn, it does not run it. It rejects #430's turn-as-workflow shape for two specific hazards, both of which a worker would reintroduce: * **Retry multiplication** - a Temporal retry policy stacked on the agent-proxy fallback for `sirens-echo/deepseek` compounds, turning a 502 into a slow expensive 502. * **Latency in the user-visible turn** - an activity round trip per tool call sits in front of a Discord member, and dead air is the demo failure mode. So "add a worker so the states go green" is not a small cosmetic change. It is a reversal of a settled architectural decision, and it would trade a presentation problem for the two failure modes that decision was made to avoid. **Withdrawing that suggestion.** ## What survives from this issue Only the presentation half, and it is smaller than I made it sound: ``` Running 12 Completed 0 TimedOut 60 ``` A Temporal audience opening the namespace sees red terminal states. That is worth having an answer ready for, and #887 already wrote the answer better than I did: > "We mirror our agent's tool-call trajectory into Temporal so there is a durable audit record" is true and specific. "Our harness runs on Temporal" would not have been. That is the sentence. The workflows are records rather than executions, they are signalled and never run on purpose, and the timeout is how a record retires. ## Confirmed against the server, not just the source `DescribeTaskQueue` on both lanes, workflow and activity types: ``` sirens-dowel-tool-mirror Workflow pollers=0 Activity pollers=0 sirens-deep-tool-mirror Workflow pollers=0 Activity pollers=0 ``` So the absence is real and total rather than a worker living somewhere I did not grep. Which is exactly what #887 specifies. ## What might genuinely be worth doing later If the red states are worth removing at all, the option that does **not** touch the control path is a retention or archival setting on the namespace, or accepting `TimedOut` as the normal terminal state and saying so in the one sentence above. Neither adds a poller to the turn. Retitling this issue toward that, since "there is no worker" reads as a defect report and it is not one.

coilyco-ops added this to the August 19 Temporal Vibe Check milestone

2026-08-19 04:12:49 +00:00

coilyco-ops referenced this issue from a commit

2026-08-19 05:02:16 +00:00

feat(temporal): run the trajectory workflow, so a turn's record closes

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-19 05:02:42 +00:00

feat(temporal): run the trajectory workflow, so a turn's record closes #1057

coilyco-ops commented

2026-08-19 05:03:03 +00:00

Author

Member

Second correction, and this one reverses the first. Fixed in #1057.

I said adding a worker would reverse #887's "mirroring, not orchestration" decision, and withdrew the suggestion. That was wrong. I reasoned from the issue rather than from the file, and the file already contains the answer:

// ToolTrajectoryWorkflow accumulates one turn's records and ends when they stop.
// It performs no activity.
func ToolTrajectoryWorkflow(ctx workflow.Context) ([]ToolCallRecord, error) {

The workflow has been in internal/community/temporalmirror.go since the mirror shipped. Nothing ever registered it, so it was signalled onto a queue with no poller. It runs no activity, the turn never waits on it, and no retry policy touches a tool call - #887's two named hazards, neither of which registering it reintroduces. This was an unfinished wiring, not a design position, and my "by design" reading of pollers=0 was the wrong conclusion from a correct measurement.

#1057 registers it and drops trajectoryIdle from 10 minutes to 2, which is the part that actually decides how long a finished turn lingers.

**Second correction, and this one reverses the first.** Fixed in #1057. I said adding a worker would reverse #887's "mirroring, not orchestration" decision, and withdrew the suggestion. That was wrong. I reasoned from the issue rather than from the file, and the file already contains the answer: ```go // ToolTrajectoryWorkflow accumulates one turn's records and ends when they stop. // It performs no activity. func ToolTrajectoryWorkflow(ctx workflow.Context) ([]ToolCallRecord, error) { ``` The workflow has been in `internal/community/temporalmirror.go` since the mirror shipped. **Nothing ever registered it**, so it was signalled onto a queue with no poller. It runs no activity, the turn never waits on it, and no retry policy touches a tool call - #887's two named hazards, neither of which registering it reintroduces. This was an unfinished wiring, not a design position, and my "by design" reading of `pollers=0` was the wrong conclusion from a correct measurement. #1057 registers it and drops `trajectoryIdle` from 10 minutes to 2, which is the part that actually decides how long a finished turn lingers.

coilysiren closed this issue

2026-08-19 05:09:11 +00:00

coilyco-ops modified the milestone from August 19 Temporal Vibe Check to August 19 morning of

2026-08-19 05:35:02 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#1041

No description provided.

Rows
Columns