Expose gpt-oss:120b (llama-server) as a routed gateway target #3

New issue

Closed

opened 2026-06-25 11:04:14 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented 2026-06-25 11:04:14 +00:00

Member

Copy link

Part of the roster-setup epic coilyco-bridge/agentic-os-kai#693 and the local-model-stack mission (coilysiren/inbox#118), milestone M3.

Today gpt-oss:120b runs as a manual llama-server launch on the tower at :8080 (OpenAI-compatible), at the benchmarked --n-cpu-moe 24 split (34 tok/s - see agentic-os-hardware docs/llama-moe-offload.md). It is a second runtime next to Ollama on :11434 and is not auto-discovered or routed. The roster assigns aider and opencode to gpt-oss:120b, so both are blocked until the gateway can route to this endpoint.

Tasks:

• Add the llama-server :8080 endpoint as a backend the gateway can route to (router.json in agentic-os-hardware names servers, not this runtime yet).
• Decide lifecycle - keep the manual llama-serve launch, or supervise it so the gateway can rely on it being up.
• Expose it under a stable model name the harnesses point at.

Blocks - the aider and opencode setup tickets in coilyco-bridge/agentic-os-kai.

Part of the roster-setup epic coilyco-bridge/agentic-os-kai#693 and the local-model-stack mission (coilysiren/inbox#118), milestone M3. Today gpt-oss:120b runs as a manual llama-server launch on the tower at :8080 (OpenAI-compatible), at the benchmarked --n-cpu-moe 24 split (34 tok/s - see agentic-os-hardware docs/llama-moe-offload.md). It is a second runtime next to Ollama on :11434 and is not auto-discovered or routed. The roster assigns aider and opencode to gpt-oss:120b, so both are blocked until the gateway can route to this endpoint. Tasks: - Add the llama-server :8080 endpoint as a backend the gateway can route to (router.json in agentic-os-hardware names servers, not this runtime yet). - Decide lifecycle - keep the manual llama-serve launch, or supervise it so the gateway can rely on it being up. - Expose it under a stable model name the harnesses point at. Blocks - the aider and opencode setup tickets in coilyco-bridge/agentic-os-kai.

coilyco-ops commented 2026-07-02 02:43:03 +00:00

Author

Member

Copy link

🔎 ward agent advisor

ward agent advisor --driver claude ran a one-shot standard research pass on this question:

The phase-1 build (#2) is in flight and its models.py registry maps logical names to backend_url/ollama_tag/num_ctx. What exactly does exposing gpt-oss:120b via llama-server as a routed gateway target require: registry entry shape for a non-ollama backend, API-dialect differences llama-server introduces on /api/chat vs OpenAI surface, safe num_ctx, and what if anything in resilience.py's validation assumes ollama response shape? Post a concrete config + code-touchpoint proposal the phase-2 engineer can carry directly.

---

Exposing gpt-oss:120b (llama-server) as a routed backend

I cloned the repo. main is still the empty scaffold, so this is written against the phase-1 spec in docs/FEATURES.md and README.md (the {backend_url, ollama_tag, num_ctx} registry, ollama /api/chat + options.num_ctx injection, validate/retry/fallback/breaker). The short version: llama-server is not an ollama backend wearing a different URL. It speaks a different dialect on a different endpoint and returns a different response shape, so phase-1's ollama assumptions break in three concrete places. Here is what phase-2 has to touch.

The core mismatch

Phase-1's num_ctx fix works by POSTing to the backend's native ollama /api/chat with options.num_ctx injected per request (docs/FEATURES.md, num_ctx injection). llama-server does not implement /api/chat, /api/generate, or /api/tags. It exposes:

• OpenAI surface - /v1/chat/completions, /v1/completions, /v1/models (server README).
• Native surface - /completion, /props, /health, /slots, /tokenize.

And critically: context size is a launch flag (-c / --ctx-size), not a per-request option. There is no num_ctx equivalent in the request body. So for this backend num_ctx stops being an injected value and becomes a guard-only value that must equal whatever -c the server was launched with. Nothing to inject, everything to respect.

1. Registry entry shape (models.py)

The {backend_url, ollama_tag, num_ctx} triple can't describe a non-ollama backend. Add a dialect discriminator and relax the ollama-specific fields:

# models.py - extend the registry entry
@dataclass
class Backend:
    backend_url: str          # resolved from SSM, not hardcoded
    dialect: str = "ollama"   # "ollama" | "openai"
    upstream_model: str = ""  # was ollama_tag; the name sent in the request body
    num_ctx: int = 32768      # ollama: injected. openai/llama-server: guard-only, must equal launch -c
    chat_path: str = "/api/chat"      # llama-server -> "/v1/chat/completions"
    health_path: str = "/api/tags"    # llama-server -> "/health"
    injects_num_ctx: bool = True      # False for llama-server

Concrete gpt-oss entry (slot it under whatever logical name the aider/opencode tickets expect. Given it is a large reasoning model with big context, ctx-think fits the existing fast-think/fast/ctx-think/ctx/tune taxonomy. If those slots are already assigned, add a dedicated gpt-oss-120b logical name):

"ctx-think": Backend(
    backend_url=ssm("/coilysiren/kai-tower-3026/tailnet-fqdn") -> "http://<fqdn>:8080",
    dialect="openai",
    upstream_model="gpt-oss-120b",   # llama-server is loose on this; any stable alias works
    num_ctx=32768,                   # guard value == launch -c (see below)
    chat_path="/v1/chat/completions",
    health_path="/health",
    injects_num_ctx=False,
),

backend_url resolves from SSM at boot per AGENTS.md (tower FQDN is /coilysiren/kai-tower-3026/tailnet-fqdn). The :8080 port collides with the proxy's own default proxy_port=8080 in config.py, so on kai-server deployment either the proxy or the tower launch needs a distinct port. Flag that now.

2. API-dialect handling (upstream.py)

This is the real work. The upstream client must branch on dialect:

• Endpoint - ollama /api/chat vs llama-server /v1/chat/completions.
• Request body - ollama {model, messages, options:{num_ctx}, stream} vs OpenAI {model, messages, max_tokens, temperature, stream}. Drop options.num_ctx for llama-server, there is nowhere to put it.
• gpt-oss specifics - launch llama-server with --jinja and the built-in gpt-oss chat template so Harmony tool calls parse into OpenAI tool_calls, and set --reasoning-format so chain-of-thought lands in a reasoning_content field instead of polluting content (gpt-oss + llama.cpp guide). Without --jinja, tool calls come back as raw Harmony text and every downstream tool-call validator fails.

The clean design: normalize at the upstream boundary. Each dialect gets a request encoder and a response decoder that returns one canonical internal shape (normalized to OpenAI, since that is already the proxy's public surface). Then resilience.py and validation only ever see one shape. Do not let ollama-specific keys leak past upstream.py.

3. What breaks in resilience.py validation

Phase-1 validation (non-empty completion, well-formed tool-call JSON, no degenerate repetition) will be written against the ollama /api/chat response, which is a flat top-level object:

• ollama - resp["message"]["content"], resp["message"]["tool_calls"] (args as an object), token counts at resp["prompt_eval_count"] / resp["eval_count"], terminal resp["done"].
• llama-server / OpenAI - resp["choices"][0]["message"]["content"], resp["choices"][0]["message"]["tool_calls"] (args as a JSON string), token counts at resp["usage"]["prompt_tokens"] / ["completion_tokens"], terminal via choices[0]["finish_reason"] (which is "tool_calls" when a tool fires).

Any validator that reaches for resp["message"] or resp["prompt_eval_count"] will KeyError on every llama-server response. Two things to check specifically:

• Tool-call JSON validation - ollama hands you a parsed object, OpenAI hands you a string that still needs json.loads. If the validator assumes one, it rejects the other.
• Streaming - ollama streams newline-delimited JSON each carrying message.content with a final done:true. OpenAI streams SSE data: {...} frames with choices[0].delta.content and a terminal data: [DONE]. Completely different framing. If the proxy normalizes ollama to OpenAI for its own output, a llama-server backend is already OpenAI-shaped and must skip that adaptation, not double-apply it.

The fix follows from #2: if upstream.py normalizes both dialects to the canonical shape before anything else runs, resilience.py needs zero changes. That is the argument for putting the adapter at the upstream boundary rather than sprinkling dialect checks through the resilience layer.

4. Safe num_ctx

gpt-oss:120b is natively 128k-capable, but this instance runs CPU-MoE offloaded (--n-cpu-moe 24, 34 tok/s per the hardware doc), so KV cache and prefill on large contexts are the constraint, not the model. Recommendation:

• Start at num_ctx=32768, and launch llama-server with a matching -c 32768. This clears the aider/opencode working-context needs and keeps KV modest on an offloaded model.
• The registry value and the launch flag must be identical. Unlike ollama, a mismatch is silent: the proxy's context-budget guard would trust a number the server does not honor, and the server truncates at its own -c. Treat them as one config with two homes.
• Raise to 49152 or 65536 only after a benchmark at the offload split shows acceptable prefill latency. Note it in the hardware doc when you do.

5. Lifecycle / supervision (the issue's open question)

Route it around the breaker's failure model. ollama is always-up and lazy-loads on demand, so a dead backend self-heals on next request. llama-server holds one model resident and does not lazy-load - if the process is down, the model is fully gone, no recovery without a restart. So:

• Supervise it. A systemd user unit on the tower wrapping the benchmarked launch (llama-server -m gpt-oss-120b ... --n-cpu-moe 24 -c 32768 --jinja --host 0.0.0.0 --port 8080) with Restart=on-failure. This is what lets the gateway "rely on it being up." The manual launch cannot, so replace it.
• Wire /health into the breaker. llama-server returns 200 when the model is loaded and 503 while loading (which can be minutes for a 120B on cold start). The breaker's health probe must read /health for openai-dialect backends (not /api/tags) and treat 503 as "loading, hold" distinct from "dead."

Touchpoint summary for phase-2

• models.py - add dialect, chat_path, health_path, injects_num_ctx, relax ollama_tag to upstream_model. Add the gpt-oss entry with SSM-resolved URL.
• upstream.py - dialect-branched request encoder + response decoder normalizing both to canonical OpenAI shape. Skip num_ctx injection when injects_num_ctx=False.
• resilience.py - no change if normalization lands in upstream.py. Audit it to confirm it never touches resp["message"] / prompt_eval_count directly.
• breaker health probe - dialect-aware path, 503-means-loading handling.
• config / deploy - resolve the :8080 port collision with the proxy, add the systemd unit for the tower launch, keep num_ctx and -c locked together.

The one-line design principle: make llama-server a second upstream adapter that normalizes to the proxy's canonical shape, so the dialect difference dies at the upstream.py boundary and never reaches the resilience core.

Sources: llama.cpp server README, gpt-oss with llama.cpp guide.

---

Researched and posted automatically by ward agent advisor --driver claude (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Claude (she/her), via ward agent

### 🔎 ward agent advisor `ward agent advisor --driver claude` ran a one-shot **standard** research pass on this question: > The phase-1 build (#2) is in flight and its models.py registry maps logical names to backend_url/ollama_tag/num_ctx. What exactly does exposing gpt-oss:120b via llama-server as a routed gateway target require: registry entry shape for a non-ollama backend, API-dialect differences llama-server introduces on /api/chat vs OpenAI surface, safe num_ctx, and what if anything in resilience.py's validation assumes ollama response shape? Post a concrete config + code-touchpoint proposal the phase-2 engineer can carry directly. --- ## Exposing gpt-oss:120b (llama-server) as a routed backend I cloned the repo. `main` is still the empty scaffold, so this is written against the phase-1 spec in `docs/FEATURES.md` and `README.md` (the `{backend_url, ollama_tag, num_ctx}` registry, ollama `/api/chat` + `options.num_ctx` injection, validate/retry/fallback/breaker). The short version: **llama-server is not an ollama backend wearing a different URL. It speaks a different dialect on a different endpoint and returns a different response shape, so phase-1's ollama assumptions break in three concrete places.** Here is what phase-2 has to touch. ### The core mismatch Phase-1's `num_ctx` fix works by POSTing to the backend's native ollama `/api/chat` with `options.num_ctx` injected per request (`docs/FEATURES.md`, num_ctx injection). llama-server does **not** implement `/api/chat`, `/api/generate`, or `/api/tags`. It exposes: * **OpenAI surface** - `/v1/chat/completions`, `/v1/completions`, `/v1/models` ([server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)). * **Native surface** - `/completion`, `/props`, `/health`, `/slots`, `/tokenize`. And critically: **context size is a launch flag (`-c` / `--ctx-size`), not a per-request option.** There is no `num_ctx` equivalent in the request body. So for this backend `num_ctx` stops being an injected value and becomes a **guard-only** value that must equal whatever `-c` the server was launched with. Nothing to inject, everything to respect. ### 1. Registry entry shape (`models.py`) The `{backend_url, ollama_tag, num_ctx}` triple can't describe a non-ollama backend. Add a `dialect` discriminator and relax the ollama-specific fields: ```python # models.py - extend the registry entry @dataclass class Backend: backend_url: str # resolved from SSM, not hardcoded dialect: str = "ollama" # "ollama" | "openai" upstream_model: str = "" # was ollama_tag; the name sent in the request body num_ctx: int = 32768 # ollama: injected. openai/llama-server: guard-only, must equal launch -c chat_path: str = "/api/chat" # llama-server -> "/v1/chat/completions" health_path: str = "/api/tags" # llama-server -> "/health" injects_num_ctx: bool = True # False for llama-server ``` Concrete gpt-oss entry (slot it under whatever logical name the aider/opencode tickets expect. Given it is a large reasoning model with big context, `ctx-think` fits the existing `fast-think/fast/ctx-think/ctx/tune` taxonomy. If those slots are already assigned, add a dedicated `gpt-oss-120b` logical name): ```python "ctx-think": Backend( backend_url=ssm("/coilysiren/kai-tower-3026/tailnet-fqdn") -> "http://<fqdn>:8080", dialect="openai", upstream_model="gpt-oss-120b", # llama-server is loose on this; any stable alias works num_ctx=32768, # guard value == launch -c (see below) chat_path="/v1/chat/completions", health_path="/health", injects_num_ctx=False, ), ``` `backend_url` resolves from SSM at boot per AGENTS.md (tower FQDN is `/coilysiren/kai-tower-3026/tailnet-fqdn`). The `:8080` port collides with the proxy's own default `proxy_port=8080` in `config.py`, so on kai-server deployment either the proxy or the tower launch needs a distinct port. Flag that now. ### 2. API-dialect handling (`upstream.py`) This is the real work. The upstream client must branch on `dialect`: * **Endpoint** - ollama `/api/chat` vs llama-server `/v1/chat/completions`. * **Request body** - ollama `{model, messages, options:{num_ctx}, stream}` vs OpenAI `{model, messages, max_tokens, temperature, stream}`. Drop `options.num_ctx` for llama-server, there is nowhere to put it. * **gpt-oss specifics** - launch llama-server with `--jinja` and the built-in gpt-oss chat template so Harmony tool calls parse into OpenAI `tool_calls`, and set `--reasoning-format` so chain-of-thought lands in a `reasoning_content` field instead of polluting `content` ([gpt-oss + llama.cpp guide](https://github.com/ggml-org/llama.cpp/discussions/15396)). Without `--jinja`, tool calls come back as raw Harmony text and every downstream tool-call validator fails. The clean design: **normalize at the upstream boundary.** Each dialect gets a request encoder and a response decoder that returns one canonical internal shape (normalized to OpenAI, since that is already the proxy's public surface). Then resilience.py and validation only ever see one shape. Do not let ollama-specific keys leak past `upstream.py`. ### 3. What breaks in `resilience.py` validation Phase-1 validation (non-empty completion, well-formed tool-call JSON, no degenerate repetition) will be written against the ollama `/api/chat` response, which is a **flat top-level object**: * ollama - `resp["message"]["content"]`, `resp["message"]["tool_calls"]` (args as an **object**), token counts at `resp["prompt_eval_count"]` / `resp["eval_count"]`, terminal `resp["done"]`. * llama-server / OpenAI - `resp["choices"][0]["message"]["content"]`, `resp["choices"][0]["message"]["tool_calls"]` (args as a **JSON string**), token counts at `resp["usage"]["prompt_tokens"]` / `["completion_tokens"]`, terminal via `choices[0]["finish_reason"]` (which is `"tool_calls"` when a tool fires). Any validator that reaches for `resp["message"]` or `resp["prompt_eval_count"]` will `KeyError` on every llama-server response. Two things to check specifically: * **Tool-call JSON validation** - ollama hands you a parsed object, OpenAI hands you a string that still needs `json.loads`. If the validator assumes one, it rejects the other. * **Streaming** - ollama streams newline-delimited JSON each carrying `message.content` with a final `done:true`. OpenAI streams SSE `data: {...}` frames with `choices[0].delta.content` and a terminal `data: [DONE]`. Completely different framing. If the proxy normalizes ollama to OpenAI for its own output, a llama-server backend is **already** OpenAI-shaped and must skip that adaptation, not double-apply it. The fix follows from #2: if `upstream.py` normalizes both dialects to the canonical shape before anything else runs, resilience.py needs **zero** changes. That is the argument for putting the adapter at the upstream boundary rather than sprinkling `dialect` checks through the resilience layer. ### 4. Safe num_ctx gpt-oss:120b is natively 128k-capable, but this instance runs CPU-MoE offloaded (`--n-cpu-moe 24`, 34 tok/s per the hardware doc), so KV cache and prefill on large contexts are the constraint, not the model. Recommendation: * **Start at `num_ctx=32768`**, and launch llama-server with a matching `-c 32768`. This clears the aider/opencode working-context needs and keeps KV modest on an offloaded model. * **The registry value and the launch flag must be identical.** Unlike ollama, a mismatch is silent: the proxy's context-budget guard would trust a number the server does not honor, and the server truncates at its own `-c`. Treat them as one config with two homes. * Raise to `49152` or `65536` only after a benchmark at the offload split shows acceptable prefill latency. Note it in the hardware doc when you do. ### 5. Lifecycle / supervision (the issue's open question) Route it around the breaker's failure model. ollama is always-up and lazy-loads on demand, so a dead backend self-heals on next request. **llama-server holds one model resident and does not lazy-load** - if the process is down, the model is fully gone, no recovery without a restart. So: * **Supervise it.** A systemd user unit on the tower wrapping the benchmarked launch (`llama-server -m gpt-oss-120b ... --n-cpu-moe 24 -c 32768 --jinja --host 0.0.0.0 --port 8080`) with `Restart=on-failure`. This is what lets the gateway "rely on it being up." The manual launch cannot, so replace it. * **Wire `/health` into the breaker.** llama-server returns 200 when the model is loaded and **503 while loading** (which can be minutes for a 120B on cold start). The breaker's health probe must read `/health` for openai-dialect backends (not `/api/tags`) and treat 503 as "loading, hold" distinct from "dead." ### Touchpoint summary for phase-2 * **`models.py`** - add `dialect`, `chat_path`, `health_path`, `injects_num_ctx`, relax `ollama_tag` to `upstream_model`. Add the gpt-oss entry with SSM-resolved URL. * **`upstream.py`** - dialect-branched request encoder + response decoder normalizing both to canonical OpenAI shape. Skip `num_ctx` injection when `injects_num_ctx=False`. * **`resilience.py`** - no change **if** normalization lands in `upstream.py`. Audit it to confirm it never touches `resp["message"]` / `prompt_eval_count` directly. * **breaker health probe** - dialect-aware path, 503-means-loading handling. * **config / deploy** - resolve the `:8080` port collision with the proxy, add the systemd unit for the tower launch, keep `num_ctx` and `-c` locked together. The one-line design principle: **make llama-server a second upstream adapter that normalizes to the proxy's canonical shape, so the dialect difference dies at the `upstream.py` boundary and never reaches the resilience core.** Sources: [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), [gpt-oss with llama.cpp guide](https://github.com/ggml-org/llama.cpp/discussions/15396). --- Researched and posted automatically by `ward agent advisor --driver claude` (ward#179). This is one-shot research, not a carried change - verify before acting on it. <!-- ward-agent-reply --> <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`

coilyco-ops commented 2026-07-02 09:19:10 +00:00

Author

Member

Copy link

🔒 Reserved by ward agent --driver codex — container engineer-codex-agent-proxy-3 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T09:19:08Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Codex, via ward agent

<!-- ward-agent-reservation --> 🔒 Reserved by `ward agent --driver codex` — container `engineer-codex-agent-proxy-3` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T09:19:08Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. <!-- ward-agent-signature --> — Codex, via `ward agent`

coilysiren referenced this issue from a commit 2026-07-02 09:23:25 +00:00

Expose gpt-oss llama-server backend

coilysiren closed this issue 2026-07-02 09:23:25 +00:00

coilysiren commented 2026-07-02 09:23:56 +00:00

Owner

Copy link

WARD-OUTCOME: done - routed gpt-oss:120b through the gateway via a llama-server/OpenAI adapter, added the stable gpt-oss-120b alias, and verified the full suite.\n\nThis felt straightforward once the dialect split was isolated in app/upstream.py. The only friction was the repo/tooling mismatch around ward exec and the missing pre-commit config, but the code path itself held together cleanly. Confidence is high. The main rough edge left is lifecycle outside this repo, since the tower process is still managed externally.

WARD-OUTCOME: done - routed gpt-oss:120b through the gateway via a llama-server/OpenAI adapter, added the stable gpt-oss-120b alias, and verified the full suite.\n\nThis felt straightforward once the dialect split was isolated in app/upstream.py. The only friction was the repo/tooling mismatch around ward exec and the missing pre-commit config, but the code path itself held together cleanly. Confidence is high. The main rough edge left is lifecycle outside this repo, since the tower process is still managed externally.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

issue-34

ward-salvage/agent-proxy-09877ee0

No results found.

Labels

Clear labels

  

coherence-core

Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  

consult

A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  

headless

Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  

interactive

Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  

P0

now / blocking / urgent

  

P1

next / important

  

P2

backlog / will do

  

P3

someday / low

  

P4

icebox / frozen / reopen if it becomes real

No labels

coherence-core

consult

headless

interactive

P0

P1

P2

P3

P4

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

coilyco-flight-deck/agent-proxy#3

No description provided.