Tailscale Serve publishes ComfyUI on 8188 with no liveness signal behind it #896

New issue

Open

opened 2026-08-20 07:22:05 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-20 07:22:05 +00:00

Owner

What is wrong

Tailscale Serve publishes raw TCP 8188 to the tailnet unconditionally. The mapping is static converger-managed config with no relationship to whether ComfyUI is behind it.

Measured on kai-tower-3026:

|-- tcp://kai-tower-3026.<tailnet>.ts.net:8188 (tailnet only)
|-- tcp://100.x.x.x:8188
|-- tcp://[fd7a:...]:8188
|--> tcp://127.0.0.1:8188

tailscale serve status renders exactly this whether the loopback listener exists or not. #883 records it rendering identically at 21:56 on 2026-08-19 while the runtime was down and a caller got Unable to connect to the remote server.

So the tailnet endpoint carries no liveness signal. A caller cannot distinguish these three states without attempting a real request and interpreting a transport error:

the tower is off
the tower is up, tailscaled is up, the runtime is dead
everything is healthy

Those want different responses. The first is wait or route elsewhere. The second is recoverable on the tower. The third is a real fault in the request.

What is not established

The #883 body asserts the endpoint "still accepts at the TCP layer and then fails", and calls that the worst shape for a caller. I could not verify the accept-then-fail shape, because doing so means taking the live runtime down. The recorded symptom is a connect-level failure, which is consistent with either a refusal or an accept-then-close. Whoever picks up this issue should measure it rather than inherit the claim.

The verifiable part is narrower and enough on its own: the mapping is unconditional, so nothing upstream of a request tells a caller which of the three states it is in.

The #883 body says coilyco-bridge/agentic-os-xxx points its ComfyUIBackend at the tailnet endpoint. On disk it does not. aosx/config.py and config.toml both carry endpoint = "http://127.0.0.1:8188", so on-tower AOSX talks to loopback and never crosses Serve. The tailnet endpoint serves remote callers, the ser8 gateway among them. That changes who is actually affected here and is worth confirming before anyone designs around the original framing.

Options

Health-gate Serve. Withdraw the mapping when the runtime is down and restore it when it comes back. Makes a dead runtime present as an unreachable host, which is honest but still coarse, and it needs something to own the gating loop.
Serve HTTP instead of raw TCP. tailscaled proxies at the HTTP layer and can answer 502 with nothing behind it, which is a far clearer signal than a transport error. It changes the client URL shape, so every caller including the ser8 gateway has to move.
Readiness surface beside the runtime. A small always-up responder on the tower that reports whether ComfyUI is live, so a caller can ask before committing to a generation. ComfyUI's own /system_stats is the natural probe behind it. Costs another thing to run and supervise.

Why now

#894 and #895 close the supervision gap, so the runtime should stop dying unattended. This is the remaining item from #883: what the tailnet edge should say when it does.

Related: #883 (the outage this came from), #888 (the S4U placement that made the supervision gap visible).

## What is wrong Tailscale Serve publishes raw TCP 8188 to the tailnet unconditionally. The mapping is static converger-managed config with no relationship to whether ComfyUI is behind it. Measured on kai-tower-3026: ``` |-- tcp://kai-tower-3026.<tailnet>.ts.net:8188 (tailnet only) |-- tcp://100.x.x.x:8188 |-- tcp://[fd7a:...]:8188 |--> tcp://127.0.0.1:8188 ``` `tailscale serve status` renders exactly this whether the loopback listener exists or not. [#883](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/883) records it rendering identically at 21:56 on 2026-08-19 while the runtime was down and a caller got `Unable to connect to the remote server`. So the tailnet endpoint carries no liveness signal. A caller cannot distinguish these three states without attempting a real request and interpreting a transport error: * the tower is off * the tower is up, tailscaled is up, the runtime is dead * everything is healthy Those want different responses. The first is wait or route elsewhere. The second is recoverable on the tower. The third is a real fault in the request. ## What is not established The #883 body asserts the endpoint "still accepts at the TCP layer and then fails", and calls that the worst shape for a caller. I could not verify the accept-then-fail shape, because doing so means taking the live runtime down. The recorded symptom is a connect-level failure, which is consistent with either a refusal or an accept-then-close. Whoever picks up this issue should measure it rather than inherit the claim. The verifiable part is narrower and enough on its own: the mapping is unconditional, so nothing upstream of a request tells a caller which of the three states it is in. ## Correction to a related assumption in #883 The #883 body says `coilyco-bridge/agentic-os-xxx` points its `ComfyUIBackend` at the tailnet endpoint. On disk it does not. `aosx/config.py` and `config.toml` both carry `endpoint = "http://127.0.0.1:8188"`, so on-tower AOSX talks to loopback and never crosses Serve. The tailnet endpoint serves remote callers, the ser8 gateway among them. That changes who is actually affected here and is worth confirming before anyone designs around the original framing. ## Options 1. **Health-gate Serve.** Withdraw the mapping when the runtime is down and restore it when it comes back. Makes a dead runtime present as an unreachable host, which is honest but still coarse, and it needs something to own the gating loop. 2. **Serve HTTP instead of raw TCP.** tailscaled proxies at the HTTP layer and can answer 502 with nothing behind it, which is a far clearer signal than a transport error. It changes the client URL shape, so every caller including the ser8 gateway has to move. 3. **Readiness surface beside the runtime.** A small always-up responder on the tower that reports whether ComfyUI is live, so a caller can ask before committing to a generation. ComfyUI's own `/system_stats` is the natural probe behind it. Costs another thing to run and supervise. ## Why now [#894](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/894) and [#895](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/pulls/895) close the supervision gap, so the runtime should stop dying unattended. This is the remaining item from #883: what the tailnet edge should say when it does. Related: #883 (the outage this came from), #888 (the S4U placement that made the supervision gap visible).

coilyco-ops added the

role/sysadmin

priority

labels

2026-08-25 08:02:06 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:30:57 +00:00

coilyco-ops added this to the Networking and DNS milestone

2026-08-29 06:09:30 +00:00