Run gpt-oss:120b as a durable llama-server service for local agent harnesses #517

New issue

Open

opened 2026-07-09 23:20:57 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-09 23:20:57 +00:00

Member

Goal

Make gpt-oss:120b available as a durable local llama-server service instead of a manual :8080 launch, so local harnesses can reliably target it through the gateway.

Context

The stale agentic-os-kai per-harness setup tickets for aider and opencode were blocked because gpt-oss:120b is only available as a manual llama-server :8080 process today. The durable work belongs in coilyco-flight-deck/infrastructure, not in agentic-os-kai, because this is host/service convergence.

Old tickets being closed as misplaced symptoms:

coilyco-bridge/agentic-os-kai#694 - aider against gpt-oss:120b
coilyco-bridge/agentic-os-kai#695 - opencode against gpt-oss:120b

Related gateway blocker named by those tickets: coilyco-flight-deck/agent-proxy#3.

Scope

Define how gpt-oss:120b is launched durably on the local model host, likely as a systemd-managed llama-server process.
Ensure restart behavior, logs, health checks, and port ownership are explicit.
Route or expose the service through the local model gateway so harnesses do not point at an ad hoc manual process.
Document the endpoint shape that aider and opencode should consume.
After the durable endpoint exists, the harness setup can be verified from current harness docs rather than tracked as stale per-seat aosk issues.

Acceptance

gpt-oss:120b comes back after host reboot without a manual llama-server command.
The gateway has a stable target for it, or the issue documents why direct service access is the chosen shape.
aider and opencode can be pointed at the stable endpoint and run the reliability harness honestly.
The manual llama-server :8080 dependency is gone from the local-agent setup story.

## Goal Make `gpt-oss:120b` available as a durable local `llama-server` service instead of a manual `:8080` launch, so local harnesses can reliably target it through the gateway. ## Context The stale `agentic-os-kai` per-harness setup tickets for `aider` and `opencode` were blocked because `gpt-oss:120b` is only available as a manual `llama-server :8080` process today. The durable work belongs in `coilyco-flight-deck/infrastructure`, not in `agentic-os-kai`, because this is host/service convergence. Old tickets being closed as misplaced symptoms: * https://forgejo.coilysiren.me/coilyco-bridge/agentic-os-kai/issues/694 - aider against gpt-oss:120b * https://forgejo.coilysiren.me/coilyco-bridge/agentic-os-kai/issues/695 - opencode against gpt-oss:120b Related gateway blocker named by those tickets: `coilyco-flight-deck/agent-proxy#3`. ## Scope * Define how `gpt-oss:120b` is launched durably on the local model host, likely as a systemd-managed `llama-server` process. * Ensure restart behavior, logs, health checks, and port ownership are explicit. * Route or expose the service through the local model gateway so harnesses do not point at an ad hoc manual process. * Document the endpoint shape that `aider` and `opencode` should consume. * After the durable endpoint exists, the harness setup can be verified from current harness docs rather than tracked as stale per-seat `aosk` issues. ## Acceptance * `gpt-oss:120b` comes back after host reboot without a manual `llama-server` command. * The gateway has a stable target for it, or the issue documents why direct service access is the chosen shape. * `aider` and `opencode` can be pointed at the stable endpoint and run the reliability harness honestly. * The manual `llama-server :8080` dependency is gone from the local-agent setup story.

coilyco-ops added the

consult

labels

2026-07-10 08:59:49 +00:00

coilyco-ops commented

2026-07-10 12:55:20 +00:00

Author

Member

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Findings

This belongs in coilyco-flight-deck/infrastructure. The stale aosk harness tickets were symptoms. The durable boundary is host convergence, service ownership, and gateway wiring.
The proxy-side blocker named in the issue is mostly resolved. coilyco-flight-deck/agent-proxy#3 is closed, and the current proxy code has an OpenAI-dialect adapter for llama-server-style /v1/chat/completions backends.
Infrastructure does not yet converge this runtime. The existing agent-proxy role still writes only PROXY_TOWER_BASE_URL=http://<tower>:11434, which targets Ollama. It does not configure the llama-server :8080 target.
Do not treat llama-server as Ollama with a different port. It needs /health, /v1/chat/completions, launch-time context via --ctx-size / -c, no options.num_ctx injection, and explicit gpt-oss chat / reasoning formatting flags.

Recommendation
The engineer should implement this in two layers.

Durable tower runtime - Add a tracked infra convergence path for llama-server-gpt-oss-120b on the actual model host.

If the manual process runs inside kai-tower-3026-wsl, use a systemd unit there, plus a Windows boot/logon trigger that starts the WSL distro after host reboot. A WSL systemd unit alone does not satisfy the reboot acceptance if Windows never starts the distro.
If the manual process runs native Windows, mirror the existing converge-ollama-windows.ps1 pattern with a Scheduled Task, because native-Windows Ansible is still staged and not converged.
Either way, the service should own the port, logs, restart behavior, and health check. Recommended service name: llama-server-gpt-oss-120b.

Gateway target - Keep harnesses pointed at the local gateway, not the raw tower process, once routing is confirmed.

Preferred harness shape: OpenAI-compatible base_url at the local proxy, for example http://127.0.0.1:8082/v1, model gpt-oss:120b, and a dummy API key only if the client requires one.
Direct tower access, for example http://<tower-tailnet>:8080/v1, should remain a debugging fallback only. It bypasses proxy tracing, queueing, retries, validation, and future routing.

Important gateway caveat
The cloned proxy code supports OpenAI backends, but its current catalog and fallback model are still centered on the primary backend’s /api/tags. Simply appending a gpt-oss backend to PROXY_BACKENDS_JSON is risky: gpt-oss:120b can 404 when Ollama is primary, and unrelated Ollama model requests can fall through to the gpt backend if it is configured as a generic fallback. Before closing this issue, the engineer should verify that curl http://127.0.0.1:8082/v1/models exposes gpt-oss:120b and that a chat request for that exact model routes to llama-server. If not, the clean fix is an agent-proxy follow-up for static per-model backend registration, or an interim second proxy instance dedicated to gpt-oss on a separate loopback port.

Launch contract to lock down

Start with --ctx-size 32768 and make the proxy’s guard value match it exactly. For llama-server, context is launch-time state, not a per-request option.
Preserve the benchmarked --n-cpu-moe 24 split unless a fresh benchmark changes it.
Include gpt-oss chat-template / Jinja and reasoning-format flags so tool calls and reasoning content land in the OpenAI-shaped fields harnesses expect.
Bind only to loopback or tailnet-scoped access. If binding 0.0.0.0, pair it with a firewall rule limited to Tailscale CGNAT, not LAN-wide exposure.
Make the port explicit. Reusing 8080 is acceptable only if the unit is the documented owner and the old manual process is stopped. A less ambiguous dedicated port is safer if any same-host proxy or dev server also uses 8080.

Acceptance checks

Reboot the model host, then verify the service returns without a manual llama-server command.
Verify systemctl status / Scheduled Task state, journal or task logs, and curl /health.
Verify the gateway sees the model and can complete POST /v1/chat/completions with model: "gpt-oss:120b".
Run the agent-proxy reliability harness against the stable endpoint, then point aider and opencode at the gateway base URL.

Open questions

Where exactly does the current manual llama-server :8080 process run: native Windows, WSL, or another Linux host?
What is the tracked model path or model download source for gpt-oss:120b?
Should infrastructure keep :8080 for compatibility, or move llama-server to a named dedicated port and update gateway config?
Does agent-proxy need a small routing follow-up for static per-model OpenAI backends, or is the closed #3 implementation sufficient once infra supplies the env?

Main risks

WSL systemd without a Windows boot trigger will look durable during an active session but fail the reboot requirement.
A --ctx-size mismatch between service launch and proxy guard silently reintroduces truncation.
Wrong gpt-oss template or reasoning flags will make tool calls look like raw text and break harness reliability.
A generic proxy fallback chain can accidentally route non-gpt models to gpt-oss unless model-specific routing is verified.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- **Findings** * This belongs in `coilyco-flight-deck/infrastructure`. The stale `aosk` harness tickets were symptoms. The durable boundary is host convergence, service ownership, and gateway wiring. * The proxy-side blocker named in the issue is mostly resolved. `coilyco-flight-deck/agent-proxy#3` is closed, and the current proxy code has an OpenAI-dialect adapter for llama-server-style `/v1/chat/completions` backends. * Infrastructure does not yet converge this runtime. The existing `agent-proxy` role still writes only `PROXY_TOWER_BASE_URL=http://<tower>:11434`, which targets Ollama. It does not configure the llama-server `:8080` target. * Do not treat llama-server as Ollama with a different port. It needs `/health`, `/v1/chat/completions`, launch-time context via `--ctx-size` / `-c`, no `options.num_ctx` injection, and explicit gpt-oss chat / reasoning formatting flags. **Recommendation** The engineer should implement this in two layers. 1. **Durable tower runtime** - Add a tracked infra convergence path for `llama-server-gpt-oss-120b` on the actual model host. * If the manual process runs inside `kai-tower-3026-wsl`, use a systemd unit there, plus a Windows boot/logon trigger that starts the WSL distro after host reboot. A WSL systemd unit alone does not satisfy the reboot acceptance if Windows never starts the distro. * If the manual process runs native Windows, mirror the existing `converge-ollama-windows.ps1` pattern with a Scheduled Task, because native-Windows Ansible is still staged and not converged. * Either way, the service should own the port, logs, restart behavior, and health check. Recommended service name: `llama-server-gpt-oss-120b`. 2. **Gateway target** - Keep harnesses pointed at the local gateway, not the raw tower process, once routing is confirmed. * Preferred harness shape: OpenAI-compatible `base_url` at the local proxy, for example `http://127.0.0.1:8082/v1`, model `gpt-oss:120b`, and a dummy API key only if the client requires one. * Direct tower access, for example `http://<tower-tailnet>:8080/v1`, should remain a debugging fallback only. It bypasses proxy tracing, queueing, retries, validation, and future routing. **Important gateway caveat** The cloned proxy code supports OpenAI backends, but its current catalog and fallback model are still centered on the primary backend’s `/api/tags`. Simply appending a `gpt-oss` backend to `PROXY_BACKENDS_JSON` is risky: `gpt-oss:120b` can 404 when Ollama is primary, and unrelated Ollama model requests can fall through to the gpt backend if it is configured as a generic fallback. Before closing this issue, the engineer should verify that `curl http://127.0.0.1:8082/v1/models` exposes `gpt-oss:120b` and that a chat request for that exact model routes to llama-server. If not, the clean fix is an agent-proxy follow-up for static per-model backend registration, or an interim second proxy instance dedicated to gpt-oss on a separate loopback port. **Launch contract to lock down** * Start with `--ctx-size 32768` and make the proxy’s guard value match it exactly. For llama-server, context is launch-time state, not a per-request option. * Preserve the benchmarked `--n-cpu-moe 24` split unless a fresh benchmark changes it. * Include gpt-oss chat-template / Jinja and reasoning-format flags so tool calls and reasoning content land in the OpenAI-shaped fields harnesses expect. * Bind only to loopback or tailnet-scoped access. If binding `0.0.0.0`, pair it with a firewall rule limited to Tailscale CGNAT, not LAN-wide exposure. * Make the port explicit. Reusing `8080` is acceptable only if the unit is the documented owner and the old manual process is stopped. A less ambiguous dedicated port is safer if any same-host proxy or dev server also uses `8080`. **Acceptance checks** * Reboot the model host, then verify the service returns without a manual `llama-server` command. * Verify `systemctl status` / Scheduled Task state, journal or task logs, and `curl /health`. * Verify the gateway sees the model and can complete `POST /v1/chat/completions` with `model: "gpt-oss:120b"`. * Run the agent-proxy reliability harness against the stable endpoint, then point `aider` and `opencode` at the gateway base URL. **Open questions** * Where exactly does the current manual `llama-server :8080` process run: native Windows, WSL, or another Linux host? * What is the tracked model path or model download source for `gpt-oss:120b`? * Should infrastructure keep `:8080` for compatibility, or move llama-server to a named dedicated port and update gateway config? * Does agent-proxy need a small routing follow-up for static per-model OpenAI backends, or is the closed #3 implementation sufficient once infra supplies the env? **Main risks** * WSL systemd without a Windows boot trigger will look durable during an active session but fail the reboot requirement. * A `--ctx-size` mismatch between service launch and proxy guard silently reintroduces truncation. * Wrong gpt-oss template or reasoning flags will make tool calls look like raw text and break harness reliability. * A generic proxy fallback chain can accidentally route non-gpt models to gpt-oss unless model-specific routing is verified. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Codex, via `ward agent`