Every mirrored trajectory closes TimedOut, so the audit record's normal state is indistinguishable from a broken one #930

New issue

Closed

opened 2026-08-17 23:52:08 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-17 23:52:08 +00:00

Member

What is happening

The tool-call mirror from #887 works. Verified live against Temporal Cloud namespace coilyco.gcdqf on 2026-08-17, on the owl-glass lane's queue sirens-deep-owl-glass-tool-mirror.

Every trajectory it creates ends TimedOut. Counted across the namespace: 12 TimedOut, 3 Running, 0 Completed. There has never been a completed one, and by the current design there cannot be.

Why, precisely

SignalWithStartWorkflow starts a workflow and schedules a workflow task. Nothing polls that task queue, which #888 and the values-file comment both state on purpose - this is mirroring, not orchestration. So the workflow task is scheduled and never dispatched, and the run ages out at its timeout.

History of sirens-deep-trajectory-3e6e2ef838aa5b1374380069a0faf020, a representative one:

WORKFLOW_EXECUTION_STARTED - execution timeout 3600s, run timeout 3600s, task timeout 10s
WORKFLOW_TASK_SCHEDULED - one, never started
WORKFLOW_EXECUTION_SIGNALED - four, signal name tool-call
WORKFLOW_EXECUTION_TIMED_OUT - exactly one hour after start

The record itself is intact and correct. Payload keys on a live signal are server, tool, outcome, elapsed_millis, trace_id - the five #888 promised, confirmed in production rather than only in TestTheMirroredPayloadCarriesNothingButMetadata.

Why it is worth a decision rather than a shrug

The behaviour is defensible: the signals live in history, history is what the audit record is, and a worker would cost more than it buys today. Three reasons not to leave it undecided:

It burns the one signal a reader has. #137 and #190 are both incidents where nothing alerted because silence and health looked identical. This is the same shape from the other side: a wall of red terminal states is what a real outage would also look like, so the display can no longer carry a warning.
It makes the record perishable in a way nobody chose. A TimedOut workflow is closed, and closed workflows age out on the namespace retention clock. If the mirror is an audit record, its lifetime should be a decision, not a side effect of a timeout default.
It costs actions. A timeout is an Action. Small against #889's finding that the $100/month floor dominates at this volume, but it is a recurring charge for an outcome nobody wanted.

Options, roughly in cost order

Accept it and say so. One line in the values-file comment and in whatever reads the mirror: TimedOut is the expected terminal state, and history is the record. Costs nothing, removes the ambiguity for the next reader.
Shorten the timeout. A turn's calls arrive in seconds, so an hour is inherited rather than chosen. A short run timeout closes the record promptly and stops each trajectory sitting Running long after its turn ended. Still TimedOut, just sooner.
Run a worker that completes the workflow. A trivial workflow that drains signals and returns leaves Completed records and a clean list. Costs a deployment, a worker process, and the thing #888 deliberately avoided by not putting Temporal in the control path - though a worker for a mirror is still outside the turn's path.

Preference, stated as a preference: shorten the timeout and document TimedOut as expected, unless #889 lands on keeping the mirror with a real consumer, at which point the worker earns itself.

Relationship to the open cost decision

#889 asks whether the mirror is worth roughly $100/month once the trial lapses, and answers "if no consumer emerges that SigNoz cannot serve, drop it." This issue does not pre-empt that. If #889 says drop, this closes unfixed. If #889 says keep, the terminal state is part of what "keep" has to mean.

Design: #887
Implementation: #888
Cost decision: #889
Lane placement: coilyco-bridge/deploy#625

## What is happening The tool-call mirror from #887 works. Verified live against Temporal Cloud namespace `coilyco.gcdqf` on 2026-08-17, on the owl-glass lane's queue `sirens-deep-owl-glass-tool-mirror`. Every trajectory it creates ends `TimedOut`. Counted across the namespace: **12 TimedOut, 3 Running, 0 Completed.** There has never been a completed one, and by the current design there cannot be. ## Why, precisely `SignalWithStartWorkflow` starts a workflow and schedules a workflow task. Nothing polls that task queue, which #888 and the values-file comment both state on purpose - this is mirroring, not orchestration. So the workflow task is scheduled and never dispatched, and the run ages out at its timeout. History of `sirens-deep-trajectory-3e6e2ef838aa5b1374380069a0faf020`, a representative one: * `WORKFLOW_EXECUTION_STARTED` - execution timeout 3600s, run timeout 3600s, task timeout 10s * `WORKFLOW_TASK_SCHEDULED` - one, never started * `WORKFLOW_EXECUTION_SIGNALED` - four, signal name `tool-call` * `WORKFLOW_EXECUTION_TIMED_OUT` - exactly one hour after start The record itself is intact and correct. Payload keys on a live signal are `server`, `tool`, `outcome`, `elapsed_millis`, `trace_id` - the five #888 promised, confirmed in production rather than only in `TestTheMirroredPayloadCarriesNothingButMetadata`. ## Why it is worth a decision rather than a shrug The behaviour is defensible: the signals live in history, history is what the audit record is, and a worker would cost more than it buys today. Three reasons not to leave it undecided: 1. **It burns the one signal a reader has.** #137 and #190 are both incidents where nothing alerted because silence and health looked identical. This is the same shape from the other side: a wall of red terminal states is what a real outage would also look like, so the display can no longer carry a warning. 2. **It makes the record perishable in a way nobody chose.** A `TimedOut` workflow is closed, and closed workflows age out on the namespace retention clock. If the mirror is an audit record, its lifetime should be a decision, not a side effect of a timeout default. 3. **It costs actions.** A timeout is an Action. Small against #889's finding that the $100/month floor dominates at this volume, but it is a recurring charge for an outcome nobody wanted. ## Options, roughly in cost order * **Accept it and say so.** One line in the values-file comment and in whatever reads the mirror: TimedOut is the expected terminal state, and history is the record. Costs nothing, removes the ambiguity for the next reader. * **Shorten the timeout.** A turn's calls arrive in seconds, so an hour is inherited rather than chosen. A short run timeout closes the record promptly and stops each trajectory sitting Running long after its turn ended. Still TimedOut, just sooner. * **Run a worker that completes the workflow.** A trivial workflow that drains signals and returns leaves Completed records and a clean list. Costs a deployment, a worker process, and the thing #888 deliberately avoided by not putting Temporal in the control path - though a worker for a mirror is still outside the turn's path. Preference, stated as a preference: shorten the timeout and document TimedOut as expected, unless #889 lands on keeping the mirror with a real consumer, at which point the worker earns itself. ## Relationship to the open cost decision #889 asks whether the mirror is worth roughly $100/month once the trial lapses, and answers "if no consumer emerges that SigNoz cannot serve, drop it." This issue does not pre-empt that. If #889 says drop, this closes unfixed. If #889 says keep, the terminal state is part of what "keep" has to mean. ## Related * Design: #887 * Implementation: #888 * Cost decision: #889 * Lane placement: coilyco-bridge/deploy#625

coilyco-ops added the

labels

2026-08-19 03:38:47 +00:00

coilyco-ops referenced this issue

2026-08-22 20:55:26 +00:00

Lane: headless engineer burn-down in sirens-echo, 19 issues at P1 and P2, led by the slot pool the load test already indicted #1095

coilyco-ops added this to the Headless engineer lane: the queue wait first milestone

2026-08-22 20:58:57 +00:00