external design session notes #18

New issue

Closed

opened 2026-07-02 03:06:48 +00:00 by coilysiren · 0 comments

coilysiren commented

2026-07-02 03:06:48 +00:00

Owner

Title: agent-proxy - agent-to-model gateway with heavy o11y, doubling as harness-fit test-matrix instrument

Origin
Started from local models dying on long context (long wait then 0 to 1 bytes = prefill done, then EOS on token one). Root cause is Ollama truncating silently to fit num_ctx instead of erroring. Generalized into a fail-loud gateway, then into the agent-proxy concept below.

Concept
Third guard in the coily / o2r family. cli-guard gates tool execution, o2r is the agent-to-agent relay, agent-proxy is the agent-to-model relay. Sits at the OpenAI-compatible API boundary (harness-agnostic), agents identify via request headers (x-agent-id, x-repo, x-trust-tier), and it is the single authorization and audit surface for inference the way coily is for tool exec.

Stack decisions reached

Language - settled - FastAPI plus the litellm SDK imported as a library, not the standalone litellm-proxy. Python is forced by litellm being the commodity core, not by "it is a web server" or "ML eventually". Keeps the repo Python-clean.
Telemetry - settled - emit the OTel GenAI gen_ai.* semconv (request.model, usage input and output tokens, response.finish_reasons, plus metrics gen_ai.client.token.usage and operation.duration). Opt into latest with OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. Composes with VictoriaTraces and OpenLLMetry, no private schema.
Hot path - settled - thin fast passthrough. Structural detectors (preflight token count, done_reason, empty-completion) stay inline and cheap. Heavy ML never runs in the hot path.
ML isolation - settled - separate process, not just a thread. On k3s that is two containers in one pod, proxy at Guaranteed QoS (cpu request equals limit), ML sidecar capped. Container boundary does not isolate CPU, cgroups do. Pin torch and OMP thread counts or one inference call pegs every core.
Load shedding - settled - ML is sheddable by design. Bounded queue, sample, drop on full, never backpressure the hot path. Tail-sample around 5 percent, 100 percent on errors.
Fleet routing - settled - the proxy's own ML analysis is itself inference, so route it through the same Mode 1 / Mode 2 / hosted tiering. Do not run heavy CPU ML on kai-server, which is CPU-only and already running Mode 2 Qwen3-30B-A3B.

The wedge (why build, not buy Portkey)

Agent-scoped policy - model allowlists per agent at the proxy (qwen-opencode local-only, Claude Code hosted-allowed), mirroring coily.yaml capability allowlists one layer down.
Context enrichment - agentproxy.* attributes (agent, session, repo, task, trust) on every gen_ai span, joining the model call to o2r activity spans and session-lattice on one trace_id.
Silent-failure detection - the crown jewel and the bug-finder thesis at the inference boundary. Catch truncation-imminent (counted prompt over num_ctx), done_reason=length or empty completion (the EOS-on-token-one signature), and the prefill-vs-generation latency split (CPU spill). Each becomes a span event plus a GlitchTip issue, and the Seer replicant files the Forgejo root-cause issue. Output is a labeled corpus of agent-distinctive inference failures, which is benchmark raw material.

Roadmap after core stability and o11y
Order by mutation authority and blast radius (observe then mutate). Shadow-mode every mutation before enforcing.

Credential injection - easy - virtual keys per agent, with budgets, rate limits, and spend attribution falling out. Caveat: the proxy becomes a crown-jewel secret store, so fetch keys from SSM at use-time and let the virtual-key map be the containment. Fails loud.
Request shaping - medium - param clamping (max_tokens, temperature, context budget) and system-prompt augmentation per tier. Fails observably.
Capability clamping - harder - filter the tools array the model is offered by tier (least-privilege by omission) and observe tool_use. Not injection, not execution. Execution-time enforcement stays at coily and cli-guard, the MCP-gateway plane already owned. Fails silent, so shadow-mode is mandatory here.

Harness-fit test matrix (the real application)

Goal - test a large array of harnesses (29-plus harness-fit yamls: opencode, openwebui, crush, cline, plandex, gptme, openhands, smolagents, k8sgpt, holmesgpt, kagent, botkube, and others) for fit against models, especially weak local ones.
Proxy as test instrument - strong - the one normalized telemetry plane across heterogeneous harnesses. Any harness that speaks OpenAI-compat emits uniform gen_ai.* spans, making the matrix comparable.
Proxy as upskill instrument - contaminating - injecting tools or scaffolding to rescue weak combos measures the harness-plus-proxy composite, not native fit.
Resolution - two conditions - Condition A observes only (baseline native fit), Condition B upskills. The A-to-B delta per cell is the signal: harness limitation (B fixes it), model limitation (B does not), or plumbing (B fixes it with tool advertisement alone). Gauntlet-shaped, holdout split.
Gating column - base_url override - the proxy only reaches harnesses that let you redirect model traffic to it (OPENAI_BASE_URL). First column of every harness-fit yaml is "can I point it at the proxy at all". The k8s-native ones may not, and need bespoke adapters.

Reference facts pinned (so future-me does not re-derive)

Opus 4.8 context - 1M delivered on API and Claude Code, 500K in the claude.ai app, 200K on MS Foundry, 128k max output.
qwen3:30b context - 256K on the current tag (original April-2025 weights are 32K native, 128K with YaRN, the 2507 refresh raised native to 256K). Ollama serves roughly 2K to 4K by default via num_ctx and truncates silently. Practical on kai-server CPU is about 32K to 64K, bounded by RAM for the KV cache. The 3090 buys real headroom.
Failure asymmetry - the real point - hosted API returns a clean error on overrun, Ollama truncates from the front silently. Fail-loud counting before the send is the proxy's job.
open-webui MCP - native HTTP/SSE MCP exists but is admin-only and new, most servers need the mcpo bridge (tools-only, no resources), and the weak-model wall is native-vs-default function calling, not MCP transport.
Fail-loud tokenizers - HF AutoTokenizer, Rust tokenizers, Go daulet/tokenizers. vLLM and llama.cpp expose /tokenize and 400 on overflow. Ollama has no preflight tokenize endpoint.

Open questions

Is upskilling worth productizing, or does it stay a test-only treatment condition? Decide from the A-to-B delta data, not up front.
Which harnesses can be pointed at the proxy vs need adapters?
Build-vs-buy checkpoint: litellm-under-hood plus a thin shell now, extract a Go-native o2r-sibling only once the detectors earn it.
Free-threaded Python (3.14, PEP 779) plus sub-interpreters (PEP 734) revisit in 12 to 18 months for in-process ML. Run the test suite against python3.14t meanwhile.

Candidate sub-issues (Kai files when this graduates)

Proxy core: FastAPI plus litellm-SDK skeleton emitting gen_ai.* OTel spans.
Structural detector set plus the agentproxy.* attribute schema.
Harness-fit yaml schema: base_url-override gating column plus a tool-calling-mode axis.
ML sidecar plus fleet-routing for analysis inference.

Title: agent-proxy - agent-to-model gateway with heavy o11y, doubling as harness-fit test-matrix instrument **Origin** Started from local models dying on long context (long wait then 0 to 1 bytes = prefill done, then EOS on token one). Root cause is Ollama truncating silently to fit num_ctx instead of erroring. Generalized into a fail-loud gateway, then into the agent-proxy concept below. **Concept** Third guard in the coily / o2r family. cli-guard gates tool execution, o2r is the agent-to-agent relay, agent-proxy is the agent-to-model relay. Sits at the OpenAI-compatible API boundary (harness-agnostic), agents identify via request headers (x-agent-id, x-repo, x-trust-tier), and it is the single authorization and audit surface for inference the way coily is for tool exec. **Stack decisions reached** * Language - settled - FastAPI plus the litellm SDK imported as a library, not the standalone litellm-proxy. Python is forced by litellm being the commodity core, not by "it is a web server" or "ML eventually". Keeps the repo Python-clean. * Telemetry - settled - emit the OTel GenAI gen_ai.* semconv (request.model, usage input and output tokens, response.finish_reasons, plus metrics gen_ai.client.token.usage and operation.duration). Opt into latest with OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. Composes with VictoriaTraces and OpenLLMetry, no private schema. * Hot path - settled - thin fast passthrough. Structural detectors (preflight token count, done_reason, empty-completion) stay inline and cheap. Heavy ML never runs in the hot path. * ML isolation - settled - separate process, not just a thread. On k3s that is two containers in one pod, proxy at Guaranteed QoS (cpu request equals limit), ML sidecar capped. Container boundary does not isolate CPU, cgroups do. Pin torch and OMP thread counts or one inference call pegs every core. * Load shedding - settled - ML is sheddable by design. Bounded queue, sample, drop on full, never backpressure the hot path. Tail-sample around 5 percent, 100 percent on errors. * Fleet routing - settled - the proxy's own ML analysis is itself inference, so route it through the same Mode 1 / Mode 2 / hosted tiering. Do not run heavy CPU ML on kai-server, which is CPU-only and already running Mode 2 Qwen3-30B-A3B. **The wedge (why build, not buy Portkey)** * Agent-scoped policy - model allowlists per agent at the proxy (qwen-opencode local-only, Claude Code hosted-allowed), mirroring coily.yaml capability allowlists one layer down. * Context enrichment - agentproxy.* attributes (agent, session, repo, task, trust) on every gen_ai span, joining the model call to o2r activity spans and session-lattice on one trace_id. * Silent-failure detection - the crown jewel and the bug-finder thesis at the inference boundary. Catch truncation-imminent (counted prompt over num_ctx), done_reason=length or empty completion (the EOS-on-token-one signature), and the prefill-vs-generation latency split (CPU spill). Each becomes a span event plus a GlitchTip issue, and the Seer replicant files the Forgejo root-cause issue. Output is a labeled corpus of agent-distinctive inference failures, which is benchmark raw material. **Roadmap after core stability and o11y** Order by mutation authority and blast radius (observe then mutate). Shadow-mode every mutation before enforcing. * Credential injection - easy - virtual keys per agent, with budgets, rate limits, and spend attribution falling out. Caveat: the proxy becomes a crown-jewel secret store, so fetch keys from SSM at use-time and let the virtual-key map be the containment. Fails loud. * Request shaping - medium - param clamping (max_tokens, temperature, context budget) and system-prompt augmentation per tier. Fails observably. * Capability clamping - harder - filter the tools array the model is offered by tier (least-privilege by omission) and observe tool_use. Not injection, not execution. Execution-time enforcement stays at coily and cli-guard, the MCP-gateway plane already owned. Fails silent, so shadow-mode is mandatory here. **Harness-fit test matrix (the real application)** * Goal - test a large array of harnesses (29-plus harness-fit yamls: opencode, openwebui, crush, cline, plandex, gptme, openhands, smolagents, k8sgpt, holmesgpt, kagent, botkube, and others) for fit against models, especially weak local ones. * Proxy as test instrument - strong - the one normalized telemetry plane across heterogeneous harnesses. Any harness that speaks OpenAI-compat emits uniform gen_ai.* spans, making the matrix comparable. * Proxy as upskill instrument - contaminating - injecting tools or scaffolding to rescue weak combos measures the harness-plus-proxy composite, not native fit. * Resolution - two conditions - Condition A observes only (baseline native fit), Condition B upskills. The A-to-B delta per cell is the signal: harness limitation (B fixes it), model limitation (B does not), or plumbing (B fixes it with tool advertisement alone). Gauntlet-shaped, holdout split. * Gating column - base_url override - the proxy only reaches harnesses that let you redirect model traffic to it (OPENAI_BASE_URL). First column of every harness-fit yaml is "can I point it at the proxy at all". The k8s-native ones may not, and need bespoke adapters. **Reference facts pinned (so future-me does not re-derive)** * Opus 4.8 context - 1M delivered on API and Claude Code, 500K in the claude.ai app, 200K on MS Foundry, 128k max output. * qwen3:30b context - 256K on the current tag (original April-2025 weights are 32K native, 128K with YaRN, the 2507 refresh raised native to 256K). Ollama serves roughly 2K to 4K by default via num_ctx and truncates silently. Practical on kai-server CPU is about 32K to 64K, bounded by RAM for the KV cache. The 3090 buys real headroom. * Failure asymmetry - the real point - hosted API returns a clean error on overrun, Ollama truncates from the front silently. Fail-loud counting before the send is the proxy's job. * open-webui MCP - native HTTP/SSE MCP exists but is admin-only and new, most servers need the mcpo bridge (tools-only, no resources), and the weak-model wall is native-vs-default function calling, not MCP transport. * Fail-loud tokenizers - HF AutoTokenizer, Rust tokenizers, Go daulet/tokenizers. vLLM and llama.cpp expose /tokenize and 400 on overflow. Ollama has no preflight tokenize endpoint. **Open questions** * Is upskilling worth productizing, or does it stay a test-only treatment condition? Decide from the A-to-B delta data, not up front. * Which harnesses can be pointed at the proxy vs need adapters? * Build-vs-buy checkpoint: litellm-under-hood plus a thin shell now, extract a Go-native o2r-sibling only once the detectors earn it. * Free-threaded Python (3.14, PEP 779) plus sub-interpreters (PEP 734) revisit in 12 to 18 months for in-process ML. Run the test suite against python3.14t meanwhile. **Candidate sub-issues (Kai files when this graduates)** * Proxy core: FastAPI plus litellm-SDK skeleton emitting gen_ai.* OTel spans. * Structural detector set plus the agentproxy.* attribute schema. * Harness-fit yaml schema: base_url-override gating column plus a tool-calling-mode axis. * ML sidecar plus fleet-routing for analysis inference.