coilyco-flight-deck/infrastructure

Watch

Run the Windows GPU service tasks S4U instead of Interactive #888

Merged

coilysiren merged 1 commit from windows-tasks-s4u into main

2026-08-20 05:41:40 +00:00

coilysiren commented

2026-08-20 05:41:34 +00:00

Owner

The ComfyUI, OllamaServe, LlamaServerGptOss120b and GpuHeartbeat tasks were
all registered with -LogonType Interactive, which binds them to the console
session. Two consequences on kai-tower-3026:

A session disconnect kills them with 0xC000013A (STATUS_CONTROL_C_EXIT).
Confirmed by a TerminalServices event 21 at 20:41:09 lining up with a
LastRunTime of 20:41:10 on ComfyUI, OllamaServe and GpuHeartbeat alike.
GpuHeartbeat had been up 5.5 days before that disconnect took it down,
and RestartCount/RestartInterval did not bring any of them back.
The next logon re-fires the logon trigger in the interactive session, so
the console-subsystem tasks flash (or, for ComfyUI's cmd.exe, park) a
black window on screen.

S4U runs them in session 0: no console is ever created and a disconnect no
longer reaps them. CUDA is unaffected there - verified with ComfyUI reporting
"Device: cuda:0 NVIDIA GeForce RTX 3090 Ti" and Ollama serving at 100% GPU.

OllamaServe additionally moves off "ollama app.exe". That is the GUI tray app,
which cannot run in session 0 and silently took the API down when switched;
"ollama.exe serve" is the headless server and is what the task should have
been running. Its tuning already lives in User env vars (OLLAMA_HOST,
OLLAMA_CONTEXT_LENGTH, OLLAMA_KEEP_ALIVE, ...), which S4U still loads.

Each idempotence guard now also compares the principal, so a host still
carrying an Interactive task actually converges instead of being skipped as
already-correct.

Also fixes the model list in the heartbeat log line always rendering as
[none]. Get-OllamaModels returns through "return $models", which unwraps a
single-element array into a scalar PSCustomObject, and .Count on that is
empty rather than 1 - so the truthiness guard never took the populated
branch. Since this box serves one model at a time it was every line, even
ones reporting a 20 GiB resident footprint. @() normalises it. The Prometheus
and OTLP paths iterate with foreach and were never affected.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

The ComfyUI, OllamaServe, LlamaServerGptOss120b and GpuHeartbeat tasks were all registered with -LogonType Interactive, which binds them to the console session. Two consequences on kai-tower-3026: * A session disconnect kills them with 0xC000013A (STATUS_CONTROL_C_EXIT). Confirmed by a TerminalServices event 21 at 20:41:09 lining up with a LastRunTime of 20:41:10 on ComfyUI, OllamaServe and GpuHeartbeat alike. GpuHeartbeat had been up 5.5 days before that disconnect took it down, and RestartCount/RestartInterval did not bring any of them back. * The next logon re-fires the logon trigger in the interactive session, so the console-subsystem tasks flash (or, for ComfyUI's cmd.exe, park) a black window on screen. S4U runs them in session 0: no console is ever created and a disconnect no longer reaps them. CUDA is unaffected there - verified with ComfyUI reporting "Device: cuda:0 NVIDIA GeForce RTX 3090 Ti" and Ollama serving at 100% GPU. OllamaServe additionally moves off "ollama app.exe". That is the GUI tray app, which cannot run in session 0 and silently took the API down when switched; "ollama.exe serve" is the headless server and is what the task should have been running. Its tuning already lives in User env vars (OLLAMA_HOST, OLLAMA_CONTEXT_LENGTH, OLLAMA_KEEP_ALIVE, ...), which S4U still loads. Each idempotence guard now also compares the principal, so a host still carrying an Interactive task actually converges instead of being skipped as already-correct. Also fixes the model list in the heartbeat log line always rendering as [none]. Get-OllamaModels returns through "return $models", which unwraps a single-element array into a scalar PSCustomObject, and .Count on that is empty rather than 1 - so the truthiness guard never took the populated branch. Since this box serves one model at a time it was every line, even ones reporting a 20 GiB resident footprint. @() normalises it. The Prometheus and OTLP paths iterate with foreach and were never affected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

coilysiren added 1 commit

2026-08-20 05:41:34 +00:00

Run the Windows GPU service tasks S4U instead of Interactive

TruffleHog / Scan for secrets (pull_request) Successful in 5s

Details

CI / lint (pull_request) Successful in 46s

Details

9bf5d21ed9

The ComfyUI, OllamaServe, LlamaServerGptOss120b and GpuHeartbeat tasks were
all registered with -LogonType Interactive, which binds them to the console
session. Two consequences on kai-tower-3026:

  * A session disconnect kills them with 0xC000013A (STATUS_CONTROL_C_EXIT).
    Confirmed by a TerminalServices event 21 at 20:41:09 lining up with a
    LastRunTime of 20:41:10 on ComfyUI, OllamaServe and GpuHeartbeat alike.
    GpuHeartbeat had been up 5.5 days before that disconnect took it down,
    and RestartCount/RestartInterval did not bring any of them back.
  * The next logon re-fires the logon trigger in the interactive session, so
    the console-subsystem tasks flash (or, for ComfyUI's cmd.exe, park) a
    black window on screen.

S4U runs them in session 0: no console is ever created and a disconnect no
longer reaps them. CUDA is unaffected there - verified with ComfyUI reporting
"Device: cuda:0 NVIDIA GeForce RTX 3090 Ti" and Ollama serving at 100% GPU.

OllamaServe additionally moves off "ollama app.exe". That is the GUI tray app,
which cannot run in session 0 and silently took the API down when switched;
"ollama.exe serve" is the headless server and is what the task should have
been running. Its tuning already lives in User env vars (OLLAMA_HOST,
OLLAMA_CONTEXT_LENGTH, OLLAMA_KEEP_ALIVE, ...), which S4U still loads.

Each idempotence guard now also compares the principal, so a host still
carrying an Interactive task actually converges instead of being skipped as
already-correct.

Also fixes the model list in the heartbeat log line always rendering as
[none]. Get-OllamaModels returns through "return $models", which unwraps a
single-element array into a scalar PSCustomObject, and .Count on that is
empty rather than 1 - so the truthiness guard never took the populated
branch. Since this box serves one model at a time it was every line, even
ones reporting a 20 GiB resident footprint. @() normalises it. The Prometheus
and OTLP paths iterate with foreach and were never affected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

coilysiren merged commit e5e8802a46 into main

2026-08-20 05:41:40 +00:00

coilysiren referenced this pull request from a commit

2026-08-20 05:41:41 +00:00

Merge pull request 'Run the Windows GPU service tasks S4U instead of Interactive' (#888) from windows-tasks-s4u into main

coilyco-ops referenced this pull request

2026-08-20 05:49:45 +00:00

fix(comfyui): stop the converger relaunching into a port it cannot free #890

coilyco-ops referenced this pull request

2026-08-20 05:50:08 +00:00

ComfyUI died on kai-tower-3026 and the scheduled task did not restart it, leaving Tailscale Serve forwarding to a dead listener #883

coilyco-ops referenced this pull request

2026-08-20 06:25:05 +00:00

ComfyUI died on kai-tower-3026 and the scheduled task did not restart it, leaving Tailscale Serve forwarding to a dead listener #883

coilyco-ops commented

2026-08-20 06:25:32 +00:00

Owner

Small correction to the evidence, the conclusion is unaffected

Post-merge note from chasing the same tower in #883. The change is right, but one cited event is inverted and I would rather that not get reused as precedent.

The description says:

A session disconnect kills them with 0xC000013A (STATUS_CONTROL_C_EXIT). Confirmed by a TerminalServices event 21 at 20:41:09 lining up with a LastRunTime of 20:41:10 on ComfyUI, OllamaServe and GpuHeartbeat alike.

In Microsoft-Windows-TerminalServices-LocalSessionManager/Operational, event 21 is "Session logon succeeded" and event 23 is "Session logoff succeeded." The full set for that day on kai-tower-3026:

20:24:58  id=23  Session logoff succeeded
20:41:09  id=21  Session logon succeeded

So 20:41:09 is a logon, not a disconnect. Reading it that way, LastRunTime of 20:41:10 is not the kill lining up with the disconnect. It is the logon trigger firing one second after logon and starting the tasks, which is exactly what an -AtLogOn trigger does. The corresponding ComfyUI startup is in comfyui.log, a clean boot ending 20:42:40.

The actual reaping event is the logoff at 20:24:58, sixteen minutes earlier. That is what produced the 0xC000013A you saw.

None of this weakens the change. Interactive really is session-bound, a logoff really does reap those tasks, and S4U really does fix it. If anything the corrected reading is a cleaner argument: one session change killed all three tasks, and the only thing that revived them was a human logging back in, which is precisely the dependency S4U removes. The 5.5-day GpuHeartbeat uptime ending at a session boundary still stands as evidence.

One consequence worth knowing about

S4U puts these tasks in session 0, which is a security boundary from your interactive session. Measured against the live ComfyUI afterwards:

listener pid   : 39132   sessionId=0
ExecutablePath : (empty)
CommandLine    : (empty)
Stop-Process   : Access is denied      (caller: session 1, elevated=False)

The ComfyUI converger's restart step identified its own runtime by ExecutablePath and CommandLine, both of which now read empty, and then tried to Stop-Process it, which is now refused. So after this merge the converger can neither see nor stop the runtime it manages, and a converge against a live ComfyUI would stop the task, fail to kill the process, relaunch into a still-held port, and leave the endpoint down. That is what happened at 22:18 and it cost about nine minutes of downtime.

#890 makes that a loud, self-diagnosing failure that leaves a working runtime alone instead of killing it. It does not resolve the underlying question, which is yours to call: give the converger an elevated path to stop an S4U task, drive restart entirely through the task surface so session never matters, or accept that restart is a manual step on this host. Tracked in #883.

Also worth noting S4U tokens carry no network credentials. It does not affect ComfyUI, but it would affect any of these tasks that later needs SMB or Windows-authenticated network access.

## Small correction to the evidence, the conclusion is unaffected Post-merge note from chasing the same tower in #883. The change is right, but one cited event is inverted and I would rather that not get reused as precedent. The description says: > A session disconnect kills them with 0xC000013A (STATUS_CONTROL_C_EXIT). Confirmed by a TerminalServices event 21 at 20:41:09 lining up with a LastRunTime of 20:41:10 on ComfyUI, OllamaServe and GpuHeartbeat alike. In `Microsoft-Windows-TerminalServices-LocalSessionManager/Operational`, **event 21 is "Session logon succeeded"** and event 23 is "Session logoff succeeded." The full set for that day on `kai-tower-3026`: ``` 20:24:58 id=23 Session logoff succeeded 20:41:09 id=21 Session logon succeeded ``` So 20:41:09 is a logon, not a disconnect. Reading it that way, `LastRunTime` of 20:41:10 is not the kill lining up with the disconnect. It is the **logon trigger firing one second after logon and starting the tasks**, which is exactly what an `-AtLogOn` trigger does. The corresponding ComfyUI startup is in `comfyui.log`, a clean boot ending 20:42:40. The actual reaping event is the **logoff at 20:24:58**, sixteen minutes earlier. That is what produced the `0xC000013A` you saw. **None of this weakens the change.** Interactive really is session-bound, a logoff really does reap those tasks, and S4U really does fix it. If anything the corrected reading is a cleaner argument: one session change killed all three tasks, and the only thing that revived them was a human logging back in, which is precisely the dependency S4U removes. The 5.5-day GpuHeartbeat uptime ending at a session boundary still stands as evidence. ## One consequence worth knowing about S4U puts these tasks in session 0, which is a security boundary from your interactive session. Measured against the live ComfyUI afterwards: ``` listener pid : 39132 sessionId=0 ExecutablePath : (empty) CommandLine : (empty) Stop-Process : Access is denied (caller: session 1, elevated=False) ``` The ComfyUI converger's restart step identified its own runtime by `ExecutablePath` and `CommandLine`, both of which now read empty, and then tried to `Stop-Process` it, which is now refused. So after this merge the converger can neither see nor stop the runtime it manages, and a converge against a live ComfyUI would stop the task, fail to kill the process, relaunch into a still-held port, and leave the endpoint down. That is what happened at 22:18 and it cost about nine minutes of downtime. #890 makes that a loud, self-diagnosing failure that leaves a working runtime alone instead of killing it. It does not resolve the underlying question, which is yours to call: give the converger an elevated path to stop an S4U task, drive restart entirely through the task surface so session never matters, or accept that restart is a manual step on this host. Tracked in #883. Also worth noting S4U tokens carry no network credentials. It does not affect ComfyUI, but it would affect any of these tasks that later needs SMB or Windows-authenticated network access.

coilyco-ops referenced this pull request

2026-08-20 06:34:20 +00:00

ComfyUI died on kai-tower-3026 and the scheduled task did not restart it, leaving Tailscale Serve forwarding to a dead listener #883

coilysiren referenced this pull request from a commit

2026-08-20 06:38:41 +00:00

fix(comfyui): stop the converger orphaning the runtime it cannot kill

coilyco-ops referenced this pull request

2026-08-20 06:39:10 +00:00

fix(comfyui): stop the converger orphaning the runtime it cannot kill #892

coilyco-ops referenced this pull request

2026-08-20 07:03:27 +00:00

fix(comfyui): make the task instance own its runtime with a job object #894