coilyco-flight-deck/infrastructure

Watch

ComfyUI died on kai-tower-3026 and the scheduled task did not restart it, leaving Tailscale Serve forwarding to a dead listener #883

New issue

Open

opened 2026-08-20 05:02:55 +00:00 by coilyco-ops · 5 comments

coilyco-ops commented

2026-08-20 05:02:55 +00:00

Owner

Symptom

Observed on kai-tower-3026 at 2026-08-19 21:56 PDT:

task: Ready          (not Running)
  last: 3221225786
comfyui: DOWN (Unable to connect to the remote server)

3221225786 is 0xC000013A, STATUS_CONTROL_C_EXIT. The process was terminated rather than crashing on its own fault.

comfyui.log ends on a clean, successful startup about 75 minutes earlier:

[INFO] ComfyUI version: 0.29.0
[INFO] Using RAM pressure cache.
[INFO] Starting server
[INFO] To see the GUI go to: http://127.0.0.1:8188

Nothing follows. Log mtime is 2026-08-19 20:42:40 PDT.

Two separate problems

1. The scheduled task did not restart it. docs/comfyui-runtime.md:12 states "a user-level scheduled task restarts ComfyUI after failure and at logon". The task exists and registers fine, but its state is Ready and the runtime has stayed down. Either the restart trigger is not configured the way the doc claims, or STATUS_CONTROL_C_EXIT is not a condition it treats as failure. Worth checking what Register-ScheduledTask in scripts/converge-comfyui-windows.ps1 around line 434 actually sets for restart-on-failure, against what the doc promises.

2. Tailscale Serve keeps forwarding to the corpse. Serve is converged and healthy:

|-- tcp://kai-tower-3026.tail09a41b.ts.net:8188 (tailnet only)
|-- tcp://100.118.232.24:8188
|-- tcp://[fd7a:115c:a1e0::e832:e819]:8188
|--> tcp://127.0.0.1:8188

With nothing bound to loopback 8188, the tailnet endpoint still accepts at the TCP layer and then fails. That is the worst shape for a caller. coilyco-bridge/agentic-os-xxx points its ComfyUIBackend at this endpoint, so an AOSX run against the tower gets a connection-level failure rather than a clear "runtime is down".

docs/comfyui-runtime.md:79 says "when the tower is off, the ser8 gateway stays up and returns a clear proxy failure instead of wedging the rollout". That covers the tower being off. It does not cover the tower being on with the process dead, which is the state observed here.

What is not wrong

Ruling out the usual suspects, since all of these check out:

All six models plus the Juggernaut XL checkpoint verify against their pinned sizes and SHA-256 digests.
Portable release 0.29.0 on disk matches the pin.
Serve topology matches the doc exactly.
The startup itself was clean, including CUDA. The log shows comfy-aimdo inited for GPU: NVIDIA GeForce RTX 3090 Ti (VRAM: 24563 MB) and DynamicVRAM support detected and enabled.

This is purely a supervision and liveness gap, not a convergence gap.

Do

Find out what terminated the process at roughly 20:42 PDT on 2026-08-19. STATUS_CONTROL_C_EXIT on a scheduled task usually means an explicit stop, a logoff, or a console-attached parent going away, so start there.
Make the scheduled task actually restart the runtime after a termination of this kind, and make the doc match whatever it really does.
Decide what the tailnet endpoint should do when the process is down. Accepting TCP and then failing is worse than refusing, and a health-gated Serve or a readiness surface would let AOSX distinguish "tower off" from "runtime dead".

Live access required

This needs someone on kai-tower-3026 to inspect Task Scheduler history and the Windows event log. Labeled autonomy/live-collab for that reason. A sealed engineer clone cannot observe any of it.

Related: infrastructure#882 (the -WhatIf preview bug found on the same pass), infrastructure#763 (AOSX Qwen Edit live verification, which needs this runtime up to proceed).

## Symptom Observed on `kai-tower-3026` at 2026-08-19 21:56 PDT: ``` task: Ready (not Running) last: 3221225786 comfyui: DOWN (Unable to connect to the remote server) ``` `3221225786` is `0xC000013A`, `STATUS_CONTROL_C_EXIT`. The process was terminated rather than crashing on its own fault. `comfyui.log` ends on a clean, successful startup about 75 minutes earlier: ``` [INFO] ComfyUI version: 0.29.0 [INFO] Using RAM pressure cache. [INFO] Starting server [INFO] To see the GUI go to: http://127.0.0.1:8188 ``` Nothing follows. Log mtime is 2026-08-19 20:42:40 PDT. ## Two separate problems **1. The scheduled task did not restart it.** [`docs/comfyui-runtime.md:12`](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/src/branch/main/docs/comfyui-runtime.md#L12) states "a user-level scheduled task restarts ComfyUI after failure and at logon". The task exists and registers fine, but its state is `Ready` and the runtime has stayed down. Either the restart trigger is not configured the way the doc claims, or `STATUS_CONTROL_C_EXIT` is not a condition it treats as failure. Worth checking what `Register-ScheduledTask` in `scripts/converge-comfyui-windows.ps1` around line 434 actually sets for restart-on-failure, against what the doc promises. **2. Tailscale Serve keeps forwarding to the corpse.** Serve is converged and healthy: ``` |-- tcp://kai-tower-3026.tail09a41b.ts.net:8188 (tailnet only) |-- tcp://100.118.232.24:8188 |-- tcp://[fd7a:115c:a1e0::e832:e819]:8188 |--> tcp://127.0.0.1:8188 ``` With nothing bound to loopback 8188, the tailnet endpoint still accepts at the TCP layer and then fails. That is the worst shape for a caller. `coilyco-bridge/agentic-os-xxx` points its `ComfyUIBackend` at this endpoint, so an AOSX run against the tower gets a connection-level failure rather than a clear "runtime is down". `docs/comfyui-runtime.md:79` says "when the tower is off, the ser8 gateway stays up and returns a clear proxy failure instead of wedging the rollout". That covers the tower being off. It does not cover the tower being on with the process dead, which is the state observed here. ## What is not wrong Ruling out the usual suspects, since all of these check out: * All six models plus the Juggernaut XL checkpoint verify against their pinned sizes and SHA-256 digests. * Portable release `0.29.0` on disk matches the pin. * Serve topology matches the doc exactly. * The startup itself was clean, including CUDA. The log shows `comfy-aimdo inited for GPU: NVIDIA GeForce RTX 3090 Ti (VRAM: 24563 MB)` and `DynamicVRAM support detected and enabled`. This is purely a supervision and liveness gap, not a convergence gap. ## Do 1. Find out what terminated the process at roughly 20:42 PDT on 2026-08-19. `STATUS_CONTROL_C_EXIT` on a scheduled task usually means an explicit stop, a logoff, or a console-attached parent going away, so start there. 2. Make the scheduled task actually restart the runtime after a termination of this kind, and make the doc match whatever it really does. 3. Decide what the tailnet endpoint should do when the process is down. Accepting TCP and then failing is worse than refusing, and a health-gated Serve or a readiness surface would let AOSX distinguish "tower off" from "runtime dead". ## Live access required This needs someone on `kai-tower-3026` to inspect Task Scheduler history and the Windows event log. Labeled `autonomy/live-collab` for that reason. A sealed engineer clone cannot observe any of it. Related: infrastructure#882 (the `-WhatIf` preview bug found on the same pass), infrastructure#763 (AOSX Qwen Edit live verification, which needs this runtime up to proceed).

coilyco-ops added the

autonomy

live-collab

priority

labels

2026-08-20 05:02:55 +00:00

coilyco-ops referenced this issue

2026-08-20 05:17:27 +00:00

Install and live-verify Qwen Image Edit 2511 for AOSX #763

coilyco-ops commented

2026-08-20 05:29:21 +00:00

Author

Owner

Warning before anyone runs the converge again

Someone ran windows-comfyui-converge on the tower at 22:18 on 2026-08-19, apparently to fix this issue. It took the endpoint from working to dead. Please read the timeline below before running it again. The runtime is down as of 22:26:21 PDT.

Timeline from the Task Scheduler operational log

22:11:43  id=100/200  instance 8d4dd07e started, cmd.exe launched   -> ComfyUI came up
22:12:25              comfyui.log last written                       -> clean startup, 6th "Starting server"
22:18:18  id=140      task definition UPDATED by firem               -> a converge re-registered the task
22:18:19  id=330      instance stopped "as request by user"
22:18:19  id=201      instance completed, return code 2147943691
22:18:19  id=111      instance terminated
22:18:23  id=200      new instance 8ecc79e2 launched cmd.exe
22:18:23  id=201      new instance completed, return code 0          <- same second
22:22:46              live check: UP, version 0.29.0, cuda:0 RTX 3090 Ti
~22:23                orphaned python exits
22:24:04 - 22:25:05   sampled every 10s: no loopback listener, HTTP down, every sample
22:26:21              still down

The converge stopped the healthy 22:11 instance and registered a replacement that exited successfully in under one second and never started the runtime. comfyui.log has had no new content since 22:12:25 and still shows six lifetime Starting server entries, so the 22:18:23 launch produced no startup at all.

What answered my 22:22:46 check was the orphan: the python child of the 22:11 instance outlived its stopped parent and kept serving for about five more minutes. That is why this looked briefly fixed. It was not.

Root cause is now concrete

The task is configured like this:

action:  cmd.exe /d /c ""C:\Users\firem\.local\share\comfyui\serve.cmd""
trigger: MSFT_TaskLogonTrigger        (logon only, no periodic trigger)
restartCount=5  restartInterval=PT1M

Restart-on-failure is configured, five retries a minute apart. It never fires because the instance reports success. The action returns immediately with code 0 rather than staying attached to the runtime, so from Task Scheduler's point of view a dead ComfyUI is indistinguishable from a completed job. There is nothing for it to restart.

That upgrades the original finding here. It is not that the restart policy is missing. It is that the supervision contract is wrong: serve.cmd is invoked in a way that detaches, so the task's liveness signal is meaningless. docs/comfyui-runtime.md:12 promising "restarts ComfyUI after failure" cannot hold under this shape.

It also explains the 20:42 death recorded in the issue body. Same signature, no log output on exit, Ready state afterward, no restart.

Suggested direction

Whatever the fix, it needs the task action to remain attached for the runtime's lifetime so a death is an instance failure, or it needs a separate liveness check that can trigger recovery. A logon-only trigger with no periodic re-check means an unattended tower stays down until the next logon regardless.

Also worth deciding what the tailnet edge should do meanwhile. Right now tailscaled holds 8188 on both tailnet addresses with nothing on 127.0.0.1:8188 behind it, so callers get a connect-time failure rather than a clear signal.

Note on attribution

Events 140 and 330 name user kai-tower-3026\firem, which is the account every local process runs as, so the log cannot distinguish a human from an agent. Six Claude sessions were running on the box at the time. No branch or commit touches the ComfyUI files, so this was an operational run rather than a code change.

## Warning before anyone runs the converge again Someone ran `windows-comfyui-converge` on the tower at 22:18 on 2026-08-19, apparently to fix this issue. **It took the endpoint from working to dead.** Please read the timeline below before running it again. The runtime is down as of 22:26:21 PDT. ## Timeline from the Task Scheduler operational log ``` 22:11:43 id=100/200 instance 8d4dd07e started, cmd.exe launched -> ComfyUI came up 22:12:25 comfyui.log last written -> clean startup, 6th "Starting server" 22:18:18 id=140 task definition UPDATED by firem -> a converge re-registered the task 22:18:19 id=330 instance stopped "as request by user" 22:18:19 id=201 instance completed, return code 2147943691 22:18:19 id=111 instance terminated 22:18:23 id=200 new instance 8ecc79e2 launched cmd.exe 22:18:23 id=201 new instance completed, return code 0 <- same second 22:22:46 live check: UP, version 0.29.0, cuda:0 RTX 3090 Ti ~22:23 orphaned python exits 22:24:04 - 22:25:05 sampled every 10s: no loopback listener, HTTP down, every sample 22:26:21 still down ``` The converge stopped the healthy 22:11 instance and registered a replacement that **exited successfully in under one second** and never started the runtime. `comfyui.log` has had no new content since 22:12:25 and still shows six lifetime `Starting server` entries, so the 22:18:23 launch produced no startup at all. What answered my 22:22:46 check was the orphan: the python child of the 22:11 instance outlived its stopped parent and kept serving for about five more minutes. That is why this looked briefly fixed. It was not. ## Root cause is now concrete The task is configured like this: ``` action: cmd.exe /d /c ""C:\Users\firem\.local\share\comfyui\serve.cmd"" trigger: MSFT_TaskLogonTrigger (logon only, no periodic trigger) restartCount=5 restartInterval=PT1M ``` Restart-on-failure **is** configured, five retries a minute apart. It never fires because the instance reports **success**. The action returns immediately with code 0 rather than staying attached to the runtime, so from Task Scheduler's point of view a dead ComfyUI is indistinguishable from a completed job. There is nothing for it to restart. That upgrades the original finding here. It is not that the restart policy is missing. It is that the supervision contract is wrong: `serve.cmd` is invoked in a way that detaches, so the task's liveness signal is meaningless. `docs/comfyui-runtime.md:12` promising "restarts ComfyUI after failure" cannot hold under this shape. It also explains the 20:42 death recorded in the issue body. Same signature, no log output on exit, `Ready` state afterward, no restart. ## Suggested direction Whatever the fix, it needs the task action to remain attached for the runtime's lifetime so a death is an instance failure, or it needs a separate liveness check that can trigger recovery. A logon-only trigger with no periodic re-check means an unattended tower stays down until the next logon regardless. Also worth deciding what the tailnet edge should do meanwhile. Right now `tailscaled` holds 8188 on both tailnet addresses with nothing on `127.0.0.1:8188` behind it, so callers get a connect-time failure rather than a clear signal. ## Note on attribution Events 140 and 330 name user `kai-tower-3026\firem`, which is the account every local process runs as, so the log cannot distinguish a human from an agent. Six Claude sessions were running on the box at the time. No branch or commit touches the ComfyUI files, so this was an operational run rather than a code change.

coilyco-ops commented

2026-08-20 05:31:01 +00:00

Author

Owner

Correction to my previous comment, and the runtime is back up

I called the root cause wrong above. New evidence contradicts it, and it changes what the fix should be.

Current status, 22:30:35 PDT: ComfyUI is UP. Loopback listener present, /system_stats returns 200, task state Running, LastTaskResult=267009 which is SCHED_S_TASK_RUNNING. It started at 22:27:06 and comfyui.log recorded its 7th Starting server at 22:27:21.

What I got wrong

I wrote that "the action returns immediately with code 0 rather than staying attached to the runtime" and concluded the supervision contract was fundamentally broken. That is not right. The 22:27:06 instance has stayed attached and Running, and the 22:11:43 instance stayed attached for about seven minutes before it was stopped. serve.cmd is untouched since 2026-07-29, so the converge never rewrote it. The action supervises correctly under normal conditions.

What actually happened at 22:18

A restart race, not a supervision defect:

The converge stopped the task instance, which terminated cmd.exe but left the child python alive still holding 127.0.0.1:8188.
Four seconds later it relaunched. The new python could not bind the still-occupied port and exited immediately, which is the return code 0 in one second, with nothing written to the log.
The orphan kept serving until about 22:23, then exited, leaving nothing behind.
The next launch at 22:27:06, with the port finally free, succeeded normally.

So the defect is in the converge's "restart under scheduled-task ownership" step: it does not wait for the previous process to release the port before relaunching. The window between stop and start is about four seconds, and the orphan outlived it by roughly five minutes.

What still stands from the original report

The 20:42 death was real and the runtime stayed down for about ninety minutes. That instance exited 3221225786 (0xC000013A), which is non-zero, so the five-retry restart policy should have fired and did not bring it back. That part of this issue is unexplained and still open.
Tailscale Serve holds 8188 on both tailnet addresses regardless of whether anything is behind it, so a dead runtime presents as a connect-time failure. Unchanged.
The only trigger is logon, with no periodic re-check, so an unattended tower that loses the runtime stays down until the next logon.

Revised suggestion

Two separable fixes:

Make the converge's restart wait for 127.0.0.1:8188 to be free, or stop the process tree rather than just the task instance, before relaunching. Right now running the converge against a live runtime reliably leaves it down for several minutes.
Work out why restart-on-failure did not recover the 20:42 exit. That is the original bug and the timeline above does not explain it.

Apologies for the noise in the prior comment. The warning at the top of it still holds for reason 1, just not for the reason I gave.

## Correction to my previous comment, and the runtime is back up I called the root cause wrong above. New evidence contradicts it, and it changes what the fix should be. **Current status, 22:30:35 PDT: ComfyUI is UP.** Loopback listener present, `/system_stats` returns 200, task state `Running`, `LastTaskResult=267009` which is `SCHED_S_TASK_RUNNING`. It started at 22:27:06 and `comfyui.log` recorded its 7th `Starting server` at 22:27:21. ### What I got wrong I wrote that "the action returns immediately with code 0 rather than staying attached to the runtime" and concluded the supervision contract was fundamentally broken. That is not right. The 22:27:06 instance has stayed attached and `Running`, and the 22:11:43 instance stayed attached for about seven minutes before it was stopped. `serve.cmd` is untouched since 2026-07-29, so the converge never rewrote it. The action supervises correctly under normal conditions. ### What actually happened at 22:18 A restart race, not a supervision defect: 1. The converge stopped the task instance, which terminated `cmd.exe` but left the child python alive still holding `127.0.0.1:8188`. 2. Four seconds later it relaunched. The new python could not bind the still-occupied port and exited immediately, which is the `return code 0` in one second, with nothing written to the log. 3. The orphan kept serving until about 22:23, then exited, leaving nothing behind. 4. The next launch at 22:27:06, with the port finally free, succeeded normally. So the defect is in the converge's "restart under scheduled-task ownership" step: it does not wait for the previous process to release the port before relaunching. The window between stop and start is about four seconds, and the orphan outlived it by roughly five minutes. ### What still stands from the original report * The 20:42 death was real and the runtime stayed down for about ninety minutes. That instance exited `3221225786` (`0xC000013A`), which is non-zero, so the five-retry restart policy should have fired and did not bring it back. That part of this issue is unexplained and still open. * Tailscale Serve holds 8188 on both tailnet addresses regardless of whether anything is behind it, so a dead runtime presents as a connect-time failure. Unchanged. * The only trigger is logon, with no periodic re-check, so an unattended tower that loses the runtime stays down until the next logon. ### Revised suggestion Two separable fixes: 1. Make the converge's restart wait for `127.0.0.1:8188` to be free, or stop the process tree rather than just the task instance, before relaunching. Right now running the converge against a live runtime reliably leaves it down for several minutes. 2. Work out why restart-on-failure did not recover the 20:42 exit. That is the original bug and the timeline above does not explain it. Apologies for the noise in the prior comment. The warning at the top of it still holds for reason 1, just not for the reason I gave.

coilysiren referenced this issue from a commit

2026-08-20 05:48:31 +00:00

fix(comfyui): stop the converger relaunching into a port it cannot free

coilyco-ops referenced this issue

2026-08-20 05:49:45 +00:00

fix(comfyui): stop the converger relaunching into a port it cannot free #890

coilyco-ops commented

2026-08-20 05:50:08 +00:00

Author

Owner

Both open items resolved, and the second one traces to #888

Third update, and it supersedes my root-cause claim in both earlier comments. Runtime is UP as of 22:45.

Item 2, why restart-on-failure did not recover the 20:42 death: it never could

The restart policy is not broken. There was no task instance to restart. Every ComfyUI start and stop event in the Task Scheduler operational log for the last seven days:

total ComfyUI start/stop events in 7 days: 9
08-19 22:11:43 id=100  started instance 8d4dd07e
08-19 22:18:19 id=330  stopped  instance 8d4dd07e  (as request by user)
08-19 22:18:19 id=201  completed, return code 2147943691
08-19 22:18:19 id=111  terminated
08-19 22:18:23 id=100  started instance 8ecc79e2
08-19 22:18:23 id=201  completed, return code 0
08-19 22:18:23 id=102  finished
08-19 22:27:06 id=100  started instance a4d64f2b

The task started exactly three times in seven days, all tonight, the first at 22:11:43. Nothing at 20:42. So the runtime that died at 20:42 was never task-owned, there was no instance failure, and restart-on-failure had nothing to act on. The LastTaskResult=3221225786 I quoted in the issue body was a stale value from a run older than seven days, not evidence of a 20:42 failure.

The real pattern is orphaned runtimes that no task owns. Nothing recovers those, by design.

Item 1, the restart race: the kill was never landing

The converger's orphan-kill filtered on Win32_Process.ExecutablePath and CommandLine. Both read back empty for the live runtime, so the filter could not match and the orphan always survived the restart. Measured:

listener pid   : 39132
Name           : python.exe
ExecutablePath :
CommandLine    :

Fixed in #890 by matching on loopback port ownership and waiting for the port to clear.

Why those fields are empty, and this is the part that needs a decision

pid=39132 name=python sessionId=0
Stop-Process FAILED: ... Access is denied
user=kai-tower-3026\firem  elevated=False  session=1

The runtime is in session 0. The converger runs unelevated in session 1, so it cannot read that process's details or signal it.

This traces to #888, merged 22:41:38 PDT today, which set the ComfyUI task principal to -LogonType S4U. S4U runs in session 0. Before it, the task was Interactive, the runtime shared the operator's session, and the original filter would have had the visibility it needed. The 22:18 converge that took the endpoint down was running that branch, which is why event 140 re-registered the task and why the 22:27 launch landed in session 0.

So #888 removed the converger's ability to manage its own runtime. #890 makes that a loud, self-diagnosing failure that leaves a working runtime alone instead of a dead endpoint, but it deliberately does not paper over the placement itself.

What is left to decide

Three options, and it belongs to whoever owns #888 since the tradeoff that motivated S4U is theirs:

Return the task to a logon type the converger can manage.
Give the converger an elevated path to stop an S4U task.
Stop restarting through Stop-Process and drive it entirely through the task surface, so session never matters.

Until one lands, a converge against a live ComfyUI cannot restart it, and will now say so rather than breaking it.

## Both open items resolved, and the second one traces to #888 Third update, and it supersedes my root-cause claim in both earlier comments. Runtime is UP as of 22:45. ### Item 2, why restart-on-failure did not recover the 20:42 death: it never could The restart policy is not broken. There was no task instance to restart. Every ComfyUI start and stop event in the Task Scheduler operational log for the last seven days: ``` total ComfyUI start/stop events in 7 days: 9 08-19 22:11:43 id=100 started instance 8d4dd07e 08-19 22:18:19 id=330 stopped instance 8d4dd07e (as request by user) 08-19 22:18:19 id=201 completed, return code 2147943691 08-19 22:18:19 id=111 terminated 08-19 22:18:23 id=100 started instance 8ecc79e2 08-19 22:18:23 id=201 completed, return code 0 08-19 22:18:23 id=102 finished 08-19 22:27:06 id=100 started instance a4d64f2b ``` The task started exactly three times in seven days, all tonight, the first at 22:11:43. Nothing at 20:42. So the runtime that died at 20:42 was never task-owned, there was no instance failure, and restart-on-failure had nothing to act on. The `LastTaskResult=3221225786` I quoted in the issue body was a stale value from a run older than seven days, not evidence of a 20:42 failure. The real pattern is orphaned runtimes that no task owns. Nothing recovers those, by design. ### Item 1, the restart race: the kill was never landing The converger's orphan-kill filtered on `Win32_Process.ExecutablePath` and `CommandLine`. Both read back **empty** for the live runtime, so the filter could not match and the orphan always survived the restart. Measured: ``` listener pid : 39132 Name : python.exe ExecutablePath : CommandLine : ``` Fixed in #890 by matching on loopback port ownership and waiting for the port to clear. ### Why those fields are empty, and this is the part that needs a decision ``` pid=39132 name=python sessionId=0 Stop-Process FAILED: ... Access is denied user=kai-tower-3026\firem elevated=False session=1 ``` The runtime is in **session 0**. The converger runs unelevated in **session 1**, so it cannot read that process's details or signal it. This traces to **#888**, merged 22:41:38 PDT today, which set the ComfyUI task principal to `-LogonType S4U`. S4U runs in session 0. Before it, the task was `Interactive`, the runtime shared the operator's session, and the original filter would have had the visibility it needed. The 22:18 converge that took the endpoint down was running that branch, which is why event 140 re-registered the task and why the 22:27 launch landed in session 0. So #888 removed the converger's ability to manage its own runtime. #890 makes that a loud, self-diagnosing failure that leaves a working runtime alone instead of a dead endpoint, but it deliberately does not paper over the placement itself. ### What is left to decide Three options, and it belongs to whoever owns #888 since the tradeoff that motivated S4U is theirs: 1. Return the task to a logon type the converger can manage. 2. Give the converger an elevated path to stop an S4U task. 3. Stop restarting through `Stop-Process` and drive it entirely through the task surface, so session never matters. Until one lands, a converge against a live ComfyUI cannot restart it, and will now say so rather than breaking it.

coilyco-ops commented

2026-08-20 06:25:05 +00:00

Author

Owner

Retracting my item 2 conclusion: the evidence window was 72 minutes, not 7 days

Fourth update. My previous comment claimed the restart policy "never could" have fired because the task started only three times in seven days. That claim is unsupported and I am withdrawing it.

The Task Scheduler Operational log does not cover the period I queried:

enabled      : True
mode         : Circular
maxSizeBytes : 16777216
recordCount  : 498
oldest event : 08/19/2026 22:03:02
newest event : 08/19/2026 23:15:01

Oldest retained event is 22:03:02. My seven-day query silently returned only what existed, which was everything from 22:03 onward. "Three starts in seven days" was really "three starts in the last 72 minutes." The 20:42 window is simply not covered, so the log says nothing either way about whether a task instance was running then.

Note the record count is 498 against a 16 MB cap, so this did not roll from volume. The channel looks to have been cleared around 22:03.

What the session log does show

20:24:58  id=23  Session logoff succeeded
20:41:09  id=21  Session logon succeeded

This reorders the night. The logoff at 20:24:58 is what reaped the Interactive-bound task. The 20:41:09 logon then re-fired the logon trigger and started ComfyUI, which is what wrote the clean startup ending at 20:42:40. So 20:42 was a birth, not a death, and the LastTaskResult=3221225786 I recorded in the issue body came from the instance killed at 20:24:58.

That also means the outage I opened this issue about ran from roughly 20:24:58, briefly interrupted by a runtime that came up at 20:41 and died again before my 21:56 check.

What is actually still unknown

ComfyUI died between 20:42:40 and 21:56 with no further log output and no intervening session event in the TerminalServices log. Under the Interactive principal in force at the time, a session change is the expected killer, and there wasn't one. That is the residual mystery here and I have no evidence for it. The task-side record that might have explained it is gone.

Summary of what survives from this issue

Confirmed and fixed or in flight:

The converger could never match the running process, because ExecutablePath and CommandLine read empty. Fixed in #890.
Restarting into a still-held port leaves the runtime down. Fixed in #890.
Under S4U the runtime is in session 0 and an unelevated session 1 converger cannot stop it at all. Surfaced loudly by #890, not solved, needs the decision below.

Withdrawn:

"Restart-on-failure is broken" and "the action does not stay attached." Both wrong.
"The task did not run for seven days." Unsupported, evidence window too short.

Standing question for #888: whether the converger regains a way to manage an S4U task, or the task moves back to a principal it can manage. That tradeoff is yours.

Apologies for the churn on this one. Three of my four comments here have corrected the previous one, which is a sign I was reasoning ahead of the evidence rather than checking retention before drawing conclusions from a log.

## Retracting my item 2 conclusion: the evidence window was 72 minutes, not 7 days Fourth update. My previous comment claimed the restart policy "never could" have fired because the task started only three times in seven days. **That claim is unsupported and I am withdrawing it.** The Task Scheduler Operational log does not cover the period I queried: ``` enabled : True mode : Circular maxSizeBytes : 16777216 recordCount : 498 oldest event : 08/19/2026 22:03:02 newest event : 08/19/2026 23:15:01 ``` Oldest retained event is **22:03:02**. My seven-day query silently returned only what existed, which was everything from 22:03 onward. "Three starts in seven days" was really "three starts in the last 72 minutes." The 20:42 window is simply not covered, so the log says nothing either way about whether a task instance was running then. Note the record count is 498 against a 16 MB cap, so this did not roll from volume. The channel looks to have been cleared around 22:03. ## What the session log does show ``` 20:24:58 id=23 Session logoff succeeded 20:41:09 id=21 Session logon succeeded ``` This reorders the night. The logoff at **20:24:58** is what reaped the Interactive-bound task. The **20:41:09 logon** then re-fired the logon trigger and *started* ComfyUI, which is what wrote the clean startup ending at 20:42:40. So 20:42 was a birth, not a death, and the `LastTaskResult=3221225786` I recorded in the issue body came from the instance killed at 20:24:58. That also means the outage I opened this issue about ran from roughly **20:24:58**, briefly interrupted by a runtime that came up at 20:41 and died again before my 21:56 check. ## What is actually still unknown ComfyUI died between 20:42:40 and 21:56 with no further log output and **no intervening session event** in the TerminalServices log. Under the Interactive principal in force at the time, a session change is the expected killer, and there wasn't one. That is the residual mystery here and I have no evidence for it. The task-side record that might have explained it is gone. ## Summary of what survives from this issue Confirmed and fixed or in flight: * The converger could never match the running process, because `ExecutablePath` and `CommandLine` read empty. Fixed in #890. * Restarting into a still-held port leaves the runtime down. Fixed in #890. * Under S4U the runtime is in session 0 and an unelevated session 1 converger cannot stop it at all. Surfaced loudly by #890, not solved, needs the decision below. Withdrawn: * "Restart-on-failure is broken" and "the action does not stay attached." Both wrong. * "The task did not run for seven days." Unsupported, evidence window too short. Standing question for #888: whether the converger regains a way to manage an S4U task, or the task moves back to a principal it can manage. That tradeoff is yours. Apologies for the churn on this one. Three of my four comments here have corrected the previous one, which is a sign I was reasoning ahead of the evidence rather than checking retention before drawing conclusions from a log.

coilyco-ops referenced this issue

2026-08-20 06:25:32 +00:00

Run the Windows GPU service tasks S4U instead of Interactive #888

coilyco-ops commented

2026-08-20 06:34:20 +00:00

Author

Owner

The runtime that is up right now is an orphan, and the converger made it one

Fifth update, from a live session on the tower at 23:30 PDT. This is not another correction of the timeline. It is a currently-reproducing state that none of the comments above record, and it has a concrete cause in the code #890 landed.

Measured now

now            : 2026-08-19 23:30:41 -07:00
task \ComfyUI  : State=Ready  LastTaskResult=267014 (0x41306 SCHED_S_TASK_TERMINATED)
                 LastRunTime=22:27:06  NextRunTime=(none)
                 LogonType=S4U  RunLevel=Limited  Trigger=MSFT_TaskLogonTrigger only
                 RestartCount=5  RestartInterval=PT1M

127.0.0.1:8188 : LISTEN  pid 39132  python  sessionId=0
  wmi CreationDate  : 08/19/2026 22:27:06
  wmi ParentPid     : 42056
  parent alive      : NO
  ExecutablePath    : ''      (empty, as #890 found)
  CommandLine       : ''

/system_stats  : HTTP 200 on loopback and on the tailnet endpoint

ComfyUI answers. The task that owns it does not exist. Python 39132 was created at 22:27:06 by task instance a4d64f2b, its parent cmd.exe 42056 is gone, and the task has been Ready since.

What killed the parent

The operational log now carries events past the window quoted above:

22:27:06  id=100/200  instance a4d64f2b started, cmd.exe pid 42056
22:41:28  id=330      instance a4d64f2b stopped "as request by user"
22:41:28  id=201      completed, return code 2147943691
22:41:28  id=111      terminated
(nothing since)

No id=140, so no task definition was rewritten at 22:41:28. This was a bare Stop-ScheduledTask, not a re-register. #888 merged at 22:41:40, twelve seconds later.

The cause, in the code merged by #890

scripts/converge-comfyui-windows.ps1, the $restartNeeded block:

Stop-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue   # <- detaches the runtime
$refused = Stop-LoopbackListener -Port $Port                           # <- Access denied, session 0
if (-not (Wait-LoopbackPortFree -Port $Port -TimeoutSeconds 30)) {
  throw (
    "Loopback port $Port is still held, so no restart was attempted and the " +
    "running ComfyUI was left alone.$detail"
  )
}
Start-ScheduledTask -TaskName $TaskName

The stop runs before the manageability check. So on an S4U host, where Stop-LoopbackListener is guaranteed to be refused, the sequence is: detach the runtime from its supervisor, discover the port cannot be freed, then throw a message saying the runtime was left alone.

It was not left alone. It was orphaned. The message is the opposite of what happened, which is why this passed as a safe failure.

#890 correctly stopped the converger from killing the endpoint. The remaining defect is quieter and worse: the endpoint stays up, the operator reads "left alone" and believes nothing changed, and the runtime is now unsupervised. Task state is Ready, so there is no instance for RestartCount=5 to act on, and the only trigger is logon. The next time this python exits, nothing brings it back until someone logs in.

Fix

Prove the port can be freed before detaching anything:

$refused = Stop-LoopbackListener -Port $Port
if (-not (Wait-LoopbackPortFree -Port $Port -TimeoutSeconds 30)) {
  throw ...   # now actually true: the task instance is untouched
}
Stop-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue
Start-ScheduledTask -TaskName $TaskName

Stop-ScheduledTask still has to run before Start-ScheduledTask, because MultipleInstances IgnoreNew would drop the new start while the old instance lives. Moving it after the guard keeps that ordering and makes the throw honest.

This fix is correct under all three of the #888 options, so it does not need that decision to land first.

On the orphan pattern generally

This is where the "orphaned runtimes that no task owns" in the third comment come from. They are not mysterious. A converge run against a live S4U runtime manufactures one every time, and reports success at leaving it alone.

Whether the same mechanism produced the original 20:42 to 21:56 outage is not something this evidence settles. That was pre-#890 code and the operational log still starts at 22:03:02. The shape is consistent. I am not claiming it as proven.

Recovering the current state needs a human

The live runtime cannot be re-adopted from an unelevated session. I am kai-tower-3026\firem, session 1, elevated=False, and pid 39132 is session 0, so Stop-Process is refused. Start-ScheduledTask on its own would make it worse: the new instance's python would fail to bind the port the orphan still holds and exit in under a second, which is exactly the 22:18:23 signature.

Recovery is: stop pid 39132 from an elevated context, confirm 8188 is free, then Start-ScheduledTask -TaskName ComfyUI. That drops the endpoint for the length of a ComfyUI startup, so it wants a person who knows nothing is mid-render.

Also still standing

Tailscale Serve holds 8188 on both tailnet addresses regardless of what is behind it. Right now that is harmless because the orphan answers. It is the same connect-time-failure shape the moment the orphan exits, and nothing about this changes that.

## The runtime that is up right now is an orphan, and the converger made it one Fifth update, from a live session on the tower at 23:30 PDT. This is not another correction of the timeline. It is a currently-reproducing state that none of the comments above record, and it has a concrete cause in the code #890 landed. ### Measured now ``` now : 2026-08-19 23:30:41 -07:00 task \ComfyUI : State=Ready LastTaskResult=267014 (0x41306 SCHED_S_TASK_TERMINATED) LastRunTime=22:27:06 NextRunTime=(none) LogonType=S4U RunLevel=Limited Trigger=MSFT_TaskLogonTrigger only RestartCount=5 RestartInterval=PT1M 127.0.0.1:8188 : LISTEN pid 39132 python sessionId=0 wmi CreationDate : 08/19/2026 22:27:06 wmi ParentPid : 42056 parent alive : NO ExecutablePath : '' (empty, as #890 found) CommandLine : '' /system_stats : HTTP 200 on loopback and on the tailnet endpoint ``` ComfyUI answers. The task that owns it does not exist. Python 39132 was created at 22:27:06 by task instance `a4d64f2b`, its parent `cmd.exe` 42056 is gone, and the task has been `Ready` since. ### What killed the parent The operational log now carries events past the window quoted above: ``` 22:27:06 id=100/200 instance a4d64f2b started, cmd.exe pid 42056 22:41:28 id=330 instance a4d64f2b stopped "as request by user" 22:41:28 id=201 completed, return code 2147943691 22:41:28 id=111 terminated (nothing since) ``` No `id=140`, so no task definition was rewritten at 22:41:28. This was a bare `Stop-ScheduledTask`, not a re-register. #888 merged at 22:41:40, twelve seconds later. ### The cause, in the code merged by #890 `scripts/converge-comfyui-windows.ps1`, the `$restartNeeded` block: ```powershell Stop-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue # <- detaches the runtime $refused = Stop-LoopbackListener -Port $Port # <- Access denied, session 0 if (-not (Wait-LoopbackPortFree -Port $Port -TimeoutSeconds 30)) { throw ( "Loopback port $Port is still held, so no restart was attempted and the " + "running ComfyUI was left alone.$detail" ) } Start-ScheduledTask -TaskName $TaskName ``` The stop runs **before** the manageability check. So on an S4U host, where `Stop-LoopbackListener` is guaranteed to be refused, the sequence is: detach the runtime from its supervisor, discover the port cannot be freed, then throw a message saying the runtime was left alone. It was not left alone. It was orphaned. The message is the opposite of what happened, which is why this passed as a safe failure. #890 correctly stopped the converger from killing the endpoint. The remaining defect is quieter and worse: the endpoint stays up, the operator reads "left alone" and believes nothing changed, and the runtime is now unsupervised. Task state is `Ready`, so there is no instance for `RestartCount=5` to act on, and the only trigger is logon. The next time this python exits, nothing brings it back until someone logs in. ### Fix Prove the port can be freed before detaching anything: ```powershell $refused = Stop-LoopbackListener -Port $Port if (-not (Wait-LoopbackPortFree -Port $Port -TimeoutSeconds 30)) { throw ... # now actually true: the task instance is untouched } Stop-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue Start-ScheduledTask -TaskName $TaskName ``` `Stop-ScheduledTask` still has to run before `Start-ScheduledTask`, because `MultipleInstances IgnoreNew` would drop the new start while the old instance lives. Moving it after the guard keeps that ordering and makes the throw honest. This fix is correct under all three of the #888 options, so it does not need that decision to land first. ### On the orphan pattern generally This is where the "orphaned runtimes that no task owns" in the third comment come from. They are not mysterious. A converge run against a live S4U runtime manufactures one every time, and reports success at leaving it alone. Whether the same mechanism produced the original 20:42 to 21:56 outage is **not** something this evidence settles. That was pre-#890 code and the operational log still starts at 22:03:02. The shape is consistent. I am not claiming it as proven. ### Recovering the current state needs a human The live runtime cannot be re-adopted from an unelevated session. I am `kai-tower-3026\firem`, session 1, `elevated=False`, and pid 39132 is session 0, so `Stop-Process` is refused. `Start-ScheduledTask` on its own would make it worse: the new instance's python would fail to bind the port the orphan still holds and exit in under a second, which is exactly the 22:18:23 signature. Recovery is: stop pid 39132 from an elevated context, confirm 8188 is free, then `Start-ScheduledTask -TaskName ComfyUI`. That drops the endpoint for the length of a ComfyUI startup, so it wants a person who knows nothing is mid-render. ### Also still standing Tailscale Serve holds 8188 on both tailnet addresses regardless of what is behind it. Right now that is harmless because the orphan answers. It is the same connect-time-failure shape the moment the orphan exits, and nothing about this changes that.

coilysiren referenced this issue from a commit

2026-08-20 06:38:41 +00:00

fix(comfyui): stop the converger orphaning the runtime it cannot kill

coilyco-ops referenced this issue

2026-08-20 06:39:10 +00:00

fix(comfyui): stop the converger orphaning the runtime it cannot kill #892

coilysiren referenced this issue from a commit

2026-08-20 07:02:59 +00:00

fix(comfyui): make the task instance own its runtime with a job object

coilyco-ops referenced this issue

2026-08-20 07:03:27 +00:00

fix(comfyui): make the task instance own its runtime with a job object #894