Build the resilience proxy core (phase 1, aosh leg 04): num_ctx injection, queue, resilience, OpenAI surface #2
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Goal
Build the resilience proxy core in
agent-proxyper the locked aosh design, and prove the per-modelnum_ctxfix end to end against the live tower. This is aosh headless leg 04 - the design is locked, this is engineering to spec. Do not redesign. Read the legs.Read first (source of truth - do not invent)
coilyco-bridge/agentic-os-hardwaredocs/plan/04-headless-proxy-build.md- the build leg, the authority for this issue.docs/plan/02-reference-architecture.md- the locked architecture and theapp/layout.docs/plan/01-reference-diagnosis.md- the measured 32k-truncation root cause this kills.docs/plan/03-headless-tower-benchmark.md- the per-model safenum_ctxvalues (use them when they exist).coilysiren/inbox#118.Dependency / sequencing
kai-tower-3026ollama over the tailnet, FQDN at SSM/coilysiren/kai-tower-3026/tailnet-fqdn). That route iscoilyco-flight-deck/ward#330. Carry this on a tailnet-capable host with the network flag +--awssocurl http://$TOWER:11434/api/versionworks from inside the container. If the route is not yet available, build steps 1 and 3-6 still proceed against a stub ollama, but step 2's live proof is gated on ward#330.agent-proxy#1(doc reorg) merges, or rebase on latestmain, to avoid README/FEATURES churn collisions. Keepdocs/FEATURES.mdcurrent as features land (the trifecta).Stack
Python 3.12, FastAPI, uvicorn, httpx, pydantic-settings, structlog, prometheus-client, opentelemetry-sdk plus the FastAPI and httpx instrumentations, sentry-sdk, tiktoken (or the model tokenizer) for token counting. Manage with
uv, declared inpyproject.toml. Add.ward/ward.yamlso dev commands route through ward.Build order (each step ends green and committed, per leg 04)
app/tree from leg 02 (main.pyqueue.pyupstream.pyresilience.pymodels.pyanalysis.pyobs.pyconfig.py). Wireobs.pybefore any logic - structlog JSON, prometheus/metrics, OTel tracer, Sentry init fromconfig.py.main.pyexposes/healthz-> 200. Confirm/healthzand/metricsunder uvicorn.upstream.py,models.py). A logical-model table maps name ->{backend_url, ollama_tag, num_ctx}. The httpx client forwards to the backend's native ollama/api/chatwithoptions.num_ctxinjected. Prove end to end against the tower: a 55k-token request to logicalfast-thinkreturnsprompt_eval_countnear the injectednum_ctx, not 32767. This is the core win - verify before building further. (Live-test gated on the tailnet route above.)queue.py). Boundedasyncio.Queue+ worker pool. Route enqueues a job and awaits its future, a worker pops and dispatches viaupstream.py. Backpressure -> 429 when full. Exportllm_queue_depth.resilience.py). Wrap dispatch with response validation (non-empty, tool-call JSON parses, no degenerate repetition), retries with backoff, the fallback chain (next backend in the logical model's list), per-backend circuit breaker with cooldown. Each path increments its metric (llm_retries_total,llm_fallbacks_total,llm_circuit_state).analysis.py). Count prompt tokens. If a request exceeds the model's safenum_ctx, trim oldest non-system turns (or summarize) before forward and incrementllm_truncation_avoided_total. Never pass an over-budget prompt to the model./v1/chat/completions(streaming + non-streaming),/v1/completions,/v1/modelslisting the logical names. Shape responses to the OpenAI schema so harnesses need no special handling.Per-model num_ctx source
Use the leg-03 safe values when they exist. Until then default
fast-think(Qwen3-A3B) to 49152 (proven safe, leg 01) and others conservatively to 32768. Values live in config the deploy can override (a ConfigMap later).Local validation
baseURL->http://127.0.0.1:8080/v1) and run a short agentic loop.Secrets and config
config.pyreads from env and SSM, never hardcodes. Sentry DSN and any API-fallback keys come from SSM at boot. No secret in a tracked file. The tower FQDN resolves at runtime from/coilysiren/kai-tower-3026/tailnet-fqdn.Guardrails
docs/FEATURES.mdcurrent.agent-proxy#1). Stay tightly the reliability proxy - the capability phases (tool injection, MCP credential passthrough, RAG, upskilling) are later, not here.Done-condition
The proxy runs locally, injects per-model
num_ctx, validates/retries/falls back, exposes metrics/traces/logs/errors, serves the OpenAI-compatible surface, and demonstrably removes the 32k truncation for a repointed harness against the live tower. Committed and pushed tomain.🔒 Reserved by
ward agent --driver claude— containerengineer-claude-agent-proxy-2on hostkais-macbook-pro-2.localis carrying this issue (reserved 2026-07-02T02:23:31Z). Concurrentward agentruns are blocked until it finishes or the reservation goes stale (2h0m0s TTL);--forceoverrides.— Claude (she/her), via
ward agentWARD-OUTCOME: done - resilience proxy core landed on main (num_ctx injection, queue, resilience, context guard, OpenAI surface); 32k cliff proven gone live (32767 -> 49151).
Honest retro: this one went smoothly because the design was genuinely locked - legs 01/02/04 left almost nothing to invent, so it was engineering to spec, and the tower happened to be reachable directly so the live proof wasn't gated after all. The core win landed on the first real try: direct tower
/v1caps at 32767, through the proxyfast-thinkkeeps 49151.What fought back was all downstream of one fact I under-weighted:
fast-think(qwen3:30b-a3b) is a reasoning model. It puts its output inmessage.thinkingand returns emptycontentuntil it finishes, so my first two proof runs 502'd - not a bug in num_ctx, but my own validator. That forced two real refinements, both surfaced by live testing rather than guessed: a legitimately short answer ("OK") must not read as truncation garbage (only non-word junk does), and a thinking-only response that ran out of budget did real work and must not be rerolled into a 502. I'm glad the leg says "verify before building further" - those only showed up against the real model. A third fix came from the reliability harness: the queue bound itsasyncio.Queueat construction, which broke across event loops - now built instart().Confidence: high on the core (proven live, 26 tests green, tool-calls round-trip as proper OpenAI JSON). Lower-confidence rough edges worth follow-ups: (1) the leg-05 reliability harness runs and scores but my context-growth config wasn't aggressive enough to reproduce the ~75% degradation in a few turns - tuning it into a real before/after is genuinely leg-05 work with its own SLO consult; (2) streaming skips full-content validation by design (a stream can't be revalidated) - documented, but worth revisiting; (3) the fallback chain ships only the tower primary by default since the sibling/CPU/API backends aren't addressable from here - the structure is there, config adds them.
Single-repo run: the agentic-os-hardware grant was read-only source-of-truth for leg 04, so I touched nothing there and there's no cross-repo push to verify. Did not deploy to kai-server (that's leg 09, out of scope).