Integrate Agent Proxy with the proven LiteLLM inner gateway #58

New issue

Closed

opened 2026-07-25 01:59:08 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-25 01:59:08 +00:00

Member

Parent: #56

Outcome

Route Agent Proxy's provider-facing inner hop through the proven standalone LiteLLM service while Agent Proxy remains the sole public model endpoint, policy and correlation boundary, safe-context owner, and durable trajectory owner.

This is an integration and first-routing slice. It does not authorize deletion of Agent Proxy queueing, retries, fallback, circuit behavior, or provider transport until the joined live evidence proves which behavior can retire.

Scope

Add secret-safe bearer authentication for OpenAI-shaped upstreams using a mounted key file, never a tracked or logged value.
Discover the virtual key's curated LiteLLM model catalog while retaining tower-derived context metadata and Agent Proxy safe num_ctx policy.
Send the derived safe num_ctx through LiteLLM and preserve Agent Proxy delivered-context detection.
Forward body-safe Ward and Agent Proxy correlation metadata and preserve W3C trace context across the inner hop.
Keep direct tower routing as an explicit observed rollback mode.
Cover non-streaming, streaming, tools, reasoning, usage, errors, authentication, catalog filtering, safe context, and metadata propagation with unit tests.
Update the product boundary and shipped feature inventory.

Live gates

The Agent Proxy endpoint remains healthy and exposes only models authorized by its LiteLLM service key.
Synthetic chat and streaming traffic traverses Agent Proxy to LiteLLM to the tower.
The tower observes the Agent Proxy-derived context window through LiteLLM.
SigNoz joins Agent Proxy and LiteLLM spans under one trace without prompt or response content.
The durable Agent Proxy request lifecycle still records bounded metadata.
A direct-tower rollback and restore are both observed without changing the durable trajectory claim.

Done when

The integration is landed, deployed, verified against all live gates, rolled back once, restored once, and #56 records the resulting evidence. Any code-removal decision becomes a separate issue after that proof.

Parent: #56 ## Outcome Route Agent Proxy's provider-facing inner hop through the proven standalone LiteLLM service while Agent Proxy remains the sole public model endpoint, policy and correlation boundary, safe-context owner, and durable trajectory owner. This is an integration and first-routing slice. It does not authorize deletion of Agent Proxy queueing, retries, fallback, circuit behavior, or provider transport until the joined live evidence proves which behavior can retire. ## Scope * Add secret-safe bearer authentication for OpenAI-shaped upstreams using a mounted key file, never a tracked or logged value. * Discover the virtual key's curated LiteLLM model catalog while retaining tower-derived context metadata and Agent Proxy safe `num_ctx` policy. * Send the derived safe `num_ctx` through LiteLLM and preserve Agent Proxy delivered-context detection. * Forward body-safe Ward and Agent Proxy correlation metadata and preserve W3C trace context across the inner hop. * Keep direct tower routing as an explicit observed rollback mode. * Cover non-streaming, streaming, tools, reasoning, usage, errors, authentication, catalog filtering, safe context, and metadata propagation with unit tests. * Update the product boundary and shipped feature inventory. ## Live gates * The Agent Proxy endpoint remains healthy and exposes only models authorized by its LiteLLM service key. * Synthetic chat and streaming traffic traverses Agent Proxy to LiteLLM to the tower. * The tower observes the Agent Proxy-derived context window through LiteLLM. * SigNoz joins Agent Proxy and LiteLLM spans under one trace without prompt or response content. * The durable Agent Proxy request lifecycle still records bounded metadata. * A direct-tower rollback and restore are both observed without changing the durable trajectory claim. ## Done when The integration is landed, deployed, verified against all live gates, rolled back once, restored once, and #56 records the resulting evidence. Any code-removal decision becomes a separate issue after that proof.

coilyco-ops referenced this issue

2026-07-25 01:59:26 +00:00

[interactive] Deploy trajectory ledger, evaluate LiteLLM, and verify first activation #56

coilyco-ops commented

2026-07-25 03:11:30 +00:00

Author

Member

WARD-OUTCOME: done

The product integration and live gates are complete.

Agent Proxy canonical main now supports secret-file authentication for OpenAI-shaped upstreams, LiteLLM-backed catalog discovery intersected with tower context metadata, safe context forwarding, streaming and nonstream translation, and body-safe correlation metadata.
The product suite passed 168 tests plus lint, type checking, and formatting.
The deploy integration is landed on canonical main. coilyco-bridge/deploy#207 records the live rollout evidence.
The live service-key catalog intersection, nonstream completion, streaming completion, 48,128-token delivered context, cross-service body-safe trace, and trajectory receipts all passed.
The verifier observed direct-tower rollback and LiteLLM restoration without changing either durable claim.
Agent Proxy is restored and Ready in LiteLLM mode with zero restarts.

This issue authorizes no deletion of Agent Proxy queueing, retries, fallback, or circuit behavior. The parent #56 receives the post-integration ownership packet and remains the decision surface for any later cleanup.

WARD-OUTCOME: done The product integration and live gates are complete. * Agent Proxy canonical `main` now supports secret-file authentication for OpenAI-shaped upstreams, LiteLLM-backed catalog discovery intersected with tower context metadata, safe context forwarding, streaming and nonstream translation, and body-safe correlation metadata. * The product suite passed 168 tests plus lint, type checking, and formatting. * The deploy integration is landed on canonical `main`. coilyco-bridge/deploy#207 records the live rollout evidence. * The live service-key catalog intersection, nonstream completion, streaming completion, 48,128-token delivered context, cross-service body-safe trace, and trajectory receipts all passed. * The verifier observed direct-tower rollback and LiteLLM restoration without changing either durable claim. * Agent Proxy is restored and Ready in LiteLLM mode with zero restarts. This issue authorizes no deletion of Agent Proxy queueing, retries, fallback, or circuit behavior. The parent #56 receives the post-integration ownership packet and remains the decision surface for any later cleanup.

coilyco-ops closed this issue

2026-07-25 03:11:32 +00:00

coilyco-ops referenced this issue

2026-07-25 03:12:05 +00:00

[interactive] Deploy trajectory ledger, evaluate LiteLLM, and verify first activation #56