coilyco-flight-deck/agent-proxy

Watch

[interactive] Deploy trajectory ledger, evaluate LiteLLM, and verify first activation #56

New issue

Closed

opened 2026-07-24 19:07:01 +00:00 by coilyco-ops · 7 comments

coilyco-ops commented

2026-07-24 19:07:01 +00:00

Member

Outcome

Complete the remaining durable trajectory deployment and LiteLLM evaluation headlessly, then present one bounded feature-ownership decision to Kai. Agent Proxy retains a separate append-only raw trajectory ledger for deterministic replay, provenance, and reproducible training-data construction. Signoz remains the durable operational observability surface for traces, logs, metrics, and correlation joins. No existing production traffic is being migrated.

workflow: interactive

Landed inputs

#41 selects a standalone LiteLLM boundary and supplies the executable parity runner.
#42 through #47 implement the trajectory contract, durable retention, materialization, evaluation, datasets, and governed views.
#51 maps agent-compose bundle evidence.
#52 maps cli-guard audit and specgen policy evidence.
#54 durably retains Ward skill-use observations.
#55 adds opt-in bounded request lifecycle emission.

Architecture boundary

Agent Proxy owns identity and policy boundaries, context safety, request correlation, body-safe event emission, durable raw retention, replay, provenance, and reproducible trajectory materialization.
LiteLLM is the candidate owner for commodity gateway behavior such as provider routing, retries, fallback, virtual keys, budgets, and spend attribution.
Signoz owns operational traces, logs, metrics, dashboards, and live correlation joins. Signoz is not the sole replay source or training-data provenance store.
The SQLite trajectory ledger is authoritative for canonical raw envelopes and reproducible replay. It remains off the latency-sensitive request path.
Request and response body capture remains disabled.

Deferred until a concrete consumer asks

The deployment does not activate or require:

Restricted-body access policy.
Legal hold and bespoke evidence-retention policy.
Destructive historical compaction governance.
A public or privileged trajectory-query product.
Restricted body references in dataset exports.

The first deployment retains body-safe internal metadata only and exposes no new historical-data interface. Storage capacity remains observable. Kai revisits lifecycle and access governance before any restricted data, external consumer, or destructive cleanup is introduced.

Headless execution

Deploy and prove the durable raw ledger
- Select the production volume and ownership pattern for PROXY_TRAJECTORY_DB_PATH using existing deployment precedent.
- Mount the database path durably before enabling PROXY_TRAJECTORY_REQUEST_EMISSION_ENABLED.
- Configure the real agent-compose, cli-guard/specgen, Ward skill-use, and bounded request-lifecycle sources without copying credentials or private bodies.
- Prove restart persistence, SQLite online backup or quiesced-volume recovery, replay into a fresh store, and zero unexplained quarantines.
- Verify Signoz receives correlated operational telemetry while the raw ledger independently preserves canonical replay and dataset provenance.
Deploy and evaluate the LiteLLM candidate
- Provision standalone LiteLLM and its required Postgres state through the owning deployment repositories and secret surfaces.
- Configure virtual-key isolation, budgets, spend attribution, provider routing, retry, and fallback without moving Agent Proxy identity, context safety, correlation, or trajectory responsibilities.
- Run ward exec litellm-parity against the current gateway and candidate using representative synthetic traffic and the resident tower models.
- Demonstrate model discovery, chat and legacy completion shape, streaming order and termination, tools, reasoning fields, finish reasons, token usage, unknown-model errors, safe num_ctx, and delivered-context checks.
- Inject deterministic provider failures and prove fallback evidence. Prove budget rejection, spend attribution, and Ward to Agent Proxy to LiteLLM trace joins without body attributes.
Verify first activation behavior
- Publish and roll the proposed composition once, then seed representative synthetic chat, streaming, failure, Ward, compose, guard, and skill-use traffic.
- Verify raw trajectory receipts, replay, body-safe Signoz joins, and visible emission or persistence failures.
- Exercise deployment disable or rollback and prove the unused system returns cleanly to its previous state.
- Do not describe this as a production traffic cutover. There is no incumbent traffic to migrate.

One human decision checkpoint

After the headless evaluation, post one disposition packet covering each overlapping or dormant feature:

Feature and current owner.
Candidate owner.
Observed parity or evidence gap.
Recommendation: retain, replace, or drop.
Consequence of that recommendation.

The packet must explicitly cover provider routing, retries, fallback, model discovery, virtual keys, budgets, spend attribution, Agent Proxy queueing and resilience, correlation and context accounting, and the dormant governed query and restricted-data capabilities. Kai decides only the unresolved ownership choices and whether an intended first consumer justifies activation.

After that decision, the agent implements the approved cleanup. The agent retires code in models.py, upstream.py, queue.py, or commodity portions of resilience.py only when the corresponding responsibility is proven elsewhere and Kai approves its disposition.

Done when

The durable raw ledger survives restart, backup or recovery, and deterministic replay into a fresh store.
Signoz receives body-safe correlated operational evidence without becoming the sole replay or dataset-provenance store.
Every applicable LiteLLM parity gate in docs/litellm-parity.md passes against resident models using synthetic traffic.
The deployment and disable or rollback path have observed evidence.
The issue thread contains one retain, replace, or drop packet and Kai's resulting decision.
Restricted-data governance and a historical-data product remain inactive and are not completion gates.
No production cutover is claimed unless a real first consumer exists.

## Outcome Complete the remaining durable trajectory deployment and LiteLLM evaluation headlessly, then present one bounded feature-ownership decision to Kai. Agent Proxy retains a separate append-only raw trajectory ledger for deterministic replay, provenance, and reproducible training-data construction. Signoz remains the durable operational observability surface for traces, logs, metrics, and correlation joins. No existing production traffic is being migrated. workflow: interactive ## Landed inputs * #41 selects a standalone LiteLLM boundary and supplies the executable parity runner. * #42 through #47 implement the trajectory contract, durable retention, materialization, evaluation, datasets, and governed views. * #51 maps agent-compose bundle evidence. * #52 maps cli-guard audit and specgen policy evidence. * #54 durably retains Ward skill-use observations. * #55 adds opt-in bounded request lifecycle emission. ## Architecture boundary * Agent Proxy owns identity and policy boundaries, context safety, request correlation, body-safe event emission, durable raw retention, replay, provenance, and reproducible trajectory materialization. * LiteLLM is the candidate owner for commodity gateway behavior such as provider routing, retries, fallback, virtual keys, budgets, and spend attribution. * Signoz owns operational traces, logs, metrics, dashboards, and live correlation joins. Signoz is not the sole replay source or training-data provenance store. * The SQLite trajectory ledger is authoritative for canonical raw envelopes and reproducible replay. It remains off the latency-sensitive request path. * Request and response body capture remains disabled. ## Deferred until a concrete consumer asks The deployment does not activate or require: * Restricted-body access policy. * Legal hold and bespoke evidence-retention policy. * Destructive historical compaction governance. * A public or privileged trajectory-query product. * Restricted body references in dataset exports. The first deployment retains body-safe internal metadata only and exposes no new historical-data interface. Storage capacity remains observable. Kai revisits lifecycle and access governance before any restricted data, external consumer, or destructive cleanup is introduced. ## Headless execution 1. **Deploy and prove the durable raw ledger** * Select the production volume and ownership pattern for `PROXY_TRAJECTORY_DB_PATH` using existing deployment precedent. * Mount the database path durably before enabling `PROXY_TRAJECTORY_REQUEST_EMISSION_ENABLED`. * Configure the real agent-compose, cli-guard/specgen, Ward skill-use, and bounded request-lifecycle sources without copying credentials or private bodies. * Prove restart persistence, SQLite online backup or quiesced-volume recovery, replay into a fresh store, and zero unexplained quarantines. * Verify Signoz receives correlated operational telemetry while the raw ledger independently preserves canonical replay and dataset provenance. 2. **Deploy and evaluate the LiteLLM candidate** * Provision standalone LiteLLM and its required Postgres state through the owning deployment repositories and secret surfaces. * Configure virtual-key isolation, budgets, spend attribution, provider routing, retry, and fallback without moving Agent Proxy identity, context safety, correlation, or trajectory responsibilities. * Run `ward exec litellm-parity` against the current gateway and candidate using representative synthetic traffic and the resident tower models. * Demonstrate model discovery, chat and legacy completion shape, streaming order and termination, tools, reasoning fields, finish reasons, token usage, unknown-model errors, safe `num_ctx`, and delivered-context checks. * Inject deterministic provider failures and prove fallback evidence. Prove budget rejection, spend attribution, and Ward to Agent Proxy to LiteLLM trace joins without body attributes. 3. **Verify first activation behavior** * Publish and roll the proposed composition once, then seed representative synthetic chat, streaming, failure, Ward, compose, guard, and skill-use traffic. * Verify raw trajectory receipts, replay, body-safe Signoz joins, and visible emission or persistence failures. * Exercise deployment disable or rollback and prove the unused system returns cleanly to its previous state. * Do not describe this as a production traffic cutover. There is no incumbent traffic to migrate. ## One human decision checkpoint After the headless evaluation, post one disposition packet covering each overlapping or dormant feature: * Feature and current owner. * Candidate owner. * Observed parity or evidence gap. * Recommendation: retain, replace, or drop. * Consequence of that recommendation. The packet must explicitly cover provider routing, retries, fallback, model discovery, virtual keys, budgets, spend attribution, Agent Proxy queueing and resilience, correlation and context accounting, and the dormant governed query and restricted-data capabilities. Kai decides only the unresolved ownership choices and whether an intended first consumer justifies activation. After that decision, the agent implements the approved cleanup. The agent retires code in `models.py`, `upstream.py`, `queue.py`, or commodity portions of `resilience.py` only when the corresponding responsibility is proven elsewhere and Kai approves its disposition. ## Done when * The durable raw ledger survives restart, backup or recovery, and deterministic replay into a fresh store. * Signoz receives body-safe correlated operational evidence without becoming the sole replay or dataset-provenance store. * Every applicable LiteLLM parity gate in `docs/litellm-parity.md` passes against resident models using synthetic traffic. * The deployment and disable or rollback path have observed evidence. * The issue thread contains one retain, replace, or drop packet and Kai's resulting decision. * Restricted-data governance and a historical-data product remain inactive and are not completion gates. * No production cutover is claimed unless a real first consumer exists.

coilyco-ops changed title from ~~[interactive] Coordinate live rollout, LiteLLM cutover, and data governance~~ to [interactive] Evaluate LiteLLM ownership and verify first activation

2026-07-24 20:13:53 +00:00

coilyco-ops commented

2026-07-24 20:14:40 +00:00

Author

Member

Scope corrected after Kai's architecture review:

Signoz is the durable observability store.
Agent Proxy local state is a bounded delivery spool or cache, not the system of record.
Speculative data governance is deferred until a concrete consumer asks for it.
LiteLLM evaluation runs headlessly and ends with one retain, replace, or drop decision packet.
There is no incumbent production traffic, so the live step is first-activation verification rather than cutover.

The rewritten issue body is the execution contract.

Scope corrected after Kai's architecture review: * Signoz is the durable observability store. * Agent Proxy local state is a bounded delivery spool or cache, not the system of record. * Speculative data governance is deferred until a concrete consumer asks for it. * LiteLLM evaluation runs headlessly and ends with one retain, replace, or drop decision packet. * There is no incumbent production traffic, so the live step is first-activation verification rather than cutover. The rewritten issue body is the execution contract.

coilyco-ops commented

2026-07-24 20:17:57 +00:00

Author

Member

Execution checkpoint: no headless engineer or live rollout was launched.

Dispatcher preflight exposed a canonical architecture conflict with the revised scope. The repository currently assigns Agent Proxy durable raw retention, replay, trajectory assembly, evaluation joins, and dataset materialization. It explicitly states that Signoz and OTLP are operational evidence surfaces, not the sole durable trajectory store. docs/trajectory-retention.md also requires durable SQLite storage and backup or recovery.

The revised issue instead makes Signoz authoritative and SQLite a disposable delivery spool. That would be an architecture reversal and feature-removal program, not a deployment-only interpretation.

Execution is paused at the human decision wall: either preserve the separate durable trajectory ledger while deferring speculative access governance, or explicitly replace the landed trajectory architecture with Signoz-only durability and scope the corresponding code and documentation removal.

Execution checkpoint: no headless engineer or live rollout was launched. Dispatcher preflight exposed a canonical architecture conflict with the revised scope. The repository currently assigns Agent Proxy durable raw retention, replay, trajectory assembly, evaluation joins, and dataset materialization. It explicitly states that Signoz and OTLP are operational evidence surfaces, not the sole durable trajectory store. `docs/trajectory-retention.md` also requires durable SQLite storage and backup or recovery. The revised issue instead makes Signoz authoritative and SQLite a disposable delivery spool. That would be an architecture reversal and feature-removal program, not a deployment-only interpretation. Execution is paused at the human decision wall: either preserve the separate durable trajectory ledger while deferring speculative access governance, or explicitly replace the landed trajectory architecture with Signoz-only durability and scope the corresponding code and documentation removal.

coilyco-ops changed title from ~~[interactive] Evaluate LiteLLM ownership and verify first activation~~ to [interactive] Deploy trajectory ledger, evaluate LiteLLM, and verify first activation

2026-07-24 20:26:07 +00:00

coilyco-ops commented

2026-07-24 20:26:13 +00:00

Author

Member

Architecture decision resolved:

Agent Proxy keeps a separate durable, append-only raw ledger because deterministic replay, stable provenance, and reproducible training-data construction do not fit cleanly in an observability-only store.
Signoz remains the durable operational surface for traces, logs, metrics, dashboards, and correlation joins.
Restricted-data governance and a historical-data product remain deferred until a concrete consumer asks.
LiteLLM evaluation stays headless and the live step remains first-activation verification, not traffic cutover.

Execution may proceed under the updated issue body.

Architecture decision resolved: * Agent Proxy keeps a separate durable, append-only raw ledger because deterministic replay, stable provenance, and reproducible training-data construction do not fit cleanly in an observability-only store. * Signoz remains the durable operational surface for traces, logs, metrics, dashboards, and correlation joins. * Restricted-data governance and a historical-data product remain deferred until a concrete consumer asks. * LiteLLM evaluation stays headless and the live step remains first-activation verification, not traffic cutover. Execution may proceed under the updated issue body.

coilyco-ops referenced this issue

2026-07-24 20:40:24 +00:00

Add headless ledger recovery and LiteLLM evaluation evidence #57

coilyco-ops referenced this issue from coilyco-flight-deck/infrastructure

2026-07-24 20:40:24 +00:00

Deprovision host-local Agent Proxy and remove misplaced LiteLLM role #629

coilyco-ops commented

2026-07-24 21:05:24 +00:00

Author

Member

Repository implementation is landed.

Agent Proxy commit 659695c adds counts-only SQLite online-backup and replay evidence, secret-safe LiteLLM credentials, expanded surface probes, and explicit live-only gates. All 165 tests, lint, type checking, and formatting pass.
Infrastructure commits 4306df2 and 47bf9c7 add durable Agent Proxy trajectory storage, body-safe lifecycle emission, read-only Ward skill-use ingestion, a disabled-by-default standalone LiteLLM 1.92.0 plus PostgreSQL 16.14 candidate, deterministic fallback, body-safe OTLP, independent disable behavior, and idempotent SSM secret provisioning. The full pre-commit suite and repository tests pass.
Agent Proxy #57 and infrastructure #633/#634 close the landed authoring slices.

Current live wall: the target Mac is intentionally local-run and absent from the remote Ansible inventory. This Windows surface cannot resolve or reach that Mac, so the agent cannot provision its SSM values, run the Mac-local converge, or observe Docker and tower evidence from here. LiteLLM remains disabled, so the landed change cannot break the current converge. Agent Proxy can activate its durable ledger independently on the next Mac converge.

No deployment, parity, rollback, or retain/replace/drop claim is being made without that observation.

Repository implementation is landed. * Agent Proxy commit `659695c` adds counts-only SQLite online-backup and replay evidence, secret-safe LiteLLM credentials, expanded surface probes, and explicit live-only gates. All 165 tests, lint, type checking, and formatting pass. * Infrastructure commits `4306df2` and `47bf9c7` add durable Agent Proxy trajectory storage, body-safe lifecycle emission, read-only Ward skill-use ingestion, a disabled-by-default standalone LiteLLM 1.92.0 plus PostgreSQL 16.14 candidate, deterministic fallback, body-safe OTLP, independent disable behavior, and idempotent SSM secret provisioning. The full pre-commit suite and repository tests pass. * Agent Proxy #57 and infrastructure #633/#634 close the landed authoring slices. Current live wall: the target Mac is intentionally local-run and absent from the remote Ansible inventory. This Windows surface cannot resolve or reach that Mac, so the agent cannot provision its SSM values, run the Mac-local converge, or observe Docker and tower evidence from here. LiteLLM remains disabled, so the landed change cannot break the current converge. Agent Proxy can activate its durable ledger independently on the next Mac converge. No deployment, parity, rollback, or retain/replace/drop claim is being made without that observation.

coilyco-ops referenced this issue from coilyco-flight-deck/infrastructure

2026-07-24 22:37:20 +00:00

[interactive] Verify retired Agent Proxy is absent on Kai's Mac #640

coilyco-ops commented

2026-07-24 22:38:27 +00:00

Author

Member

Retain, replace, or drop packet

The headless deployment and evaluation are complete. Agent Proxy is the sole public model endpoint on ser8. LiteLLM remains a separate ser8 candidate with no Agent Proxy traffic.

Durable evidence plane

Trajectory ledger - Current owner: Agent Proxy. Candidate owner: none. Evidence: 8 events survived pod replacement, online backup, and fresh replay with zero quarantines. Recommendation: retain. Consequence: SigNoz stays operational storage while SQLite remains the replay and training-provenance source.
Correlation and context accounting - Current owner: Agent Proxy. Candidate owner: Agent Proxy above LiteLLM. Evidence: LiteLLM accepts explicit num_ctx, but the tower-delivered context and Ward to Agent Proxy to LiteLLM trace join cannot exist before inner-hop integration. Recommendation: retain. Consequence: no context or Ward evidence moves into LiteLLM.
Governed query and restricted-data product - Current owner: dormant Agent Proxy code and documentation. Candidate owner: none. Evidence: the deployment retains metadata only and exposes no historical query surface. Recommendation: drop from active scope until a named consumer asks. Consequence: the agent performs no policy, legal-hold, restricted-body, or query-product work now.

Commodity gateway plane

Provider routing - Current owner: Agent Proxy upstream dispatch. Candidate owner: LiteLLM. Evidence: both endpoints pass representative surface parity. Recommendation: replace later, after the inner-hop trace and delivered-context gates. Consequence: Agent Proxy keeps the current path today.
Retries - Current owner: Agent Proxy resilience. Candidate owner: LiteLLM. Evidence: the evaluation intentionally set LiteLLM retries to zero so fallback was deterministic. Recommendation: retain for now. Consequence: the agent removes no retry code until configured LiteLLM retry evidence exists.
Fallback - Current owner: Agent Proxy resilience. Candidate owner: LiteLLM. Evidence: the failed-provider route fell back successfully with HTTP 200. Recommendation: replace after integration. Consequence: LiteLLM can own commodity fallback once the inner hop is approved.
Model discovery - Current owner: Agent Proxy direct tower discovery with 10 live models. Candidate owner: LiteLLM's curated catalog with 3 configured models. Evidence: both catalogs are valid, but they serve different boundaries. Recommendation: retain Agent Proxy filtering and replace raw provider discovery later. Consequence: Agent Proxy continues exposing the live catalog until Kai chooses a curated public catalog.
Virtual keys - Current owner: no active Agent Proxy equivalent. Candidate owner: LiteLLM. Evidence: an allowed model returned 200 and a denied model returned 403. Recommendation: retain in LiteLLM only. Consequence: Agent Proxy does not grow a duplicate key system.
Budgets and spend attribution - Current owner: no active Agent Proxy equivalent. Candidate owner: LiteLLM plus PostgreSQL. Evidence: spend persisted at 0.00002 and the next request returned 429. Recommendation: retain in LiteLLM only. Consequence: Agent Proxy consumes accounting facts instead of implementing billing state.
Queueing and circuit resilience - Current owner: Agent Proxy. Candidate owner: overlapping LiteLLM routing and retry machinery. Evidence: surface and fallback parity passed, but no real consumer supplied admission-pressure or circuit-breaker evidence. Recommendation: retain until a first consumer exists. Consequence: the agent performs no speculative deletion from queue.py or resilience.py.

Deployment evidence

Agent Proxy is Ready on ser8 with a durable 5 Gi trajectory claim.
LiteLLM and PostgreSQL are Ready on ser8 with a durable 5 Gi PostgreSQL claim.
The LiteLLM disable and restore cycle preserved state and did not disturb Agent Proxy.
SigNoz received 90 body-safe LiteLLM spans with zero synthetic prompt-marker matches.
Infrastructure commit 0231c83 retires all host-local Agent Proxy configuration.
Infrastructure #640 holds the native Mac absence proof because this session cannot reach the Mac over SSH.

Recommended decision

The agent recommends:

Retain Agent Proxy as the sole ser8 endpoint, policy boundary, context accountant, correlation tap, and durable trajectory owner.
Retain LiteLLM on ser8 as the proven inner-gateway candidate, without traffic, until a first consumer or explicit cutover request justifies the remaining integration gates.
Drop restricted-data governance and historical-query product work from active scope until a concrete consumer asks.
Do not delete Agent Proxy routing, retry, queue, or resilience code in this issue.

Kai's remaining choice is whether the idle LiteLLM candidate stays running, stays disabled with its PostgreSQL claim preserved, or proceeds into a separate Agent Proxy to LiteLLM integration issue.

## Retain, replace, or drop packet The headless deployment and evaluation are complete. Agent Proxy is the sole public model endpoint on ser8. LiteLLM remains a separate ser8 candidate with no Agent Proxy traffic. ### Durable evidence plane * **Trajectory ledger** - Current owner: Agent Proxy. Candidate owner: none. Evidence: 8 events survived pod replacement, online backup, and fresh replay with zero quarantines. Recommendation: **retain**. Consequence: SigNoz stays operational storage while SQLite remains the replay and training-provenance source. * **Correlation and context accounting** - Current owner: Agent Proxy. Candidate owner: Agent Proxy above LiteLLM. Evidence: LiteLLM accepts explicit `num_ctx`, but the tower-delivered context and Ward to Agent Proxy to LiteLLM trace join cannot exist before inner-hop integration. Recommendation: **retain**. Consequence: no context or Ward evidence moves into LiteLLM. * **Governed query and restricted-data product** - Current owner: dormant Agent Proxy code and documentation. Candidate owner: none. Evidence: the deployment retains metadata only and exposes no historical query surface. Recommendation: **drop from active scope** until a named consumer asks. Consequence: the agent performs no policy, legal-hold, restricted-body, or query-product work now. ### Commodity gateway plane * **Provider routing** - Current owner: Agent Proxy upstream dispatch. Candidate owner: LiteLLM. Evidence: both endpoints pass representative surface parity. Recommendation: **replace later**, after the inner-hop trace and delivered-context gates. Consequence: Agent Proxy keeps the current path today. * **Retries** - Current owner: Agent Proxy resilience. Candidate owner: LiteLLM. Evidence: the evaluation intentionally set LiteLLM retries to zero so fallback was deterministic. Recommendation: **retain for now**. Consequence: the agent removes no retry code until configured LiteLLM retry evidence exists. * **Fallback** - Current owner: Agent Proxy resilience. Candidate owner: LiteLLM. Evidence: the failed-provider route fell back successfully with HTTP 200. Recommendation: **replace after integration**. Consequence: LiteLLM can own commodity fallback once the inner hop is approved. * **Model discovery** - Current owner: Agent Proxy direct tower discovery with 10 live models. Candidate owner: LiteLLM's curated catalog with 3 configured models. Evidence: both catalogs are valid, but they serve different boundaries. Recommendation: **retain Agent Proxy filtering and replace raw provider discovery later**. Consequence: Agent Proxy continues exposing the live catalog until Kai chooses a curated public catalog. * **Virtual keys** - Current owner: no active Agent Proxy equivalent. Candidate owner: LiteLLM. Evidence: an allowed model returned 200 and a denied model returned 403. Recommendation: **retain in LiteLLM only**. Consequence: Agent Proxy does not grow a duplicate key system. * **Budgets and spend attribution** - Current owner: no active Agent Proxy equivalent. Candidate owner: LiteLLM plus PostgreSQL. Evidence: spend persisted at 0.00002 and the next request returned 429. Recommendation: **retain in LiteLLM only**. Consequence: Agent Proxy consumes accounting facts instead of implementing billing state. * **Queueing and circuit resilience** - Current owner: Agent Proxy. Candidate owner: overlapping LiteLLM routing and retry machinery. Evidence: surface and fallback parity passed, but no real consumer supplied admission-pressure or circuit-breaker evidence. Recommendation: **retain until a first consumer exists**. Consequence: the agent performs no speculative deletion from `queue.py` or `resilience.py`. ### Deployment evidence * Agent Proxy is Ready on ser8 with a durable 5 Gi trajectory claim. * LiteLLM and PostgreSQL are Ready on ser8 with a durable 5 Gi PostgreSQL claim. * The LiteLLM disable and restore cycle preserved state and did not disturb Agent Proxy. * SigNoz received 90 body-safe LiteLLM spans with zero synthetic prompt-marker matches. * Infrastructure commit `0231c83` retires all host-local Agent Proxy configuration. * Infrastructure #640 holds the native Mac absence proof because this session cannot reach the Mac over SSH. ### Recommended decision The agent recommends: * **Retain** Agent Proxy as the sole ser8 endpoint, policy boundary, context accountant, correlation tap, and durable trajectory owner. * **Retain** LiteLLM on ser8 as the proven inner-gateway candidate, without traffic, until a first consumer or explicit cutover request justifies the remaining integration gates. * **Drop** restricted-data governance and historical-query product work from active scope until a concrete consumer asks. * **Do not delete** Agent Proxy routing, retry, queue, or resilience code in this issue. Kai's remaining choice is whether the idle LiteLLM candidate stays running, stays disabled with its PostgreSQL claim preserved, or proceeds into a separate Agent Proxy to LiteLLM integration issue.

coilyco-ops referenced this issue

2026-07-25 01:59:08 +00:00

Integrate Agent Proxy with the proven LiteLLM inner gateway #58

coilyco-ops commented

2026-07-25 01:59:26 +00:00

Author

Member

Continuation decision: proceed into the separate Agent Proxy to LiteLLM inner-hop integration while keeping Agent Proxy as the public policy, context, correlation, and trajectory boundary. Agent Proxy #58 and deploy #207 hold the bounded implementation and live rollback proof. The current direct-tower path remains the rollback until the joined gates pass.

coilyco-ops referenced this issue

2026-07-25 03:11:30 +00:00

Integrate Agent Proxy with the proven LiteLLM inner gateway #58

coilyco-ops commented

2026-07-25 03:12:05 +00:00

Author

Member

Post-integration ownership packet

The inner-hop integration is complete. Agent Proxy is Ready on ser8 as the sole public endpoint and routes its provider-facing traffic through LiteLLM. Direct routing remains the observed rollback mode. Agent Proxy #58 and deploy #207 are closed with live evidence.

Provider routing - Agent Proxy currently owns public dispatch and rollback. LiteLLM owns the active provider-facing route. The integrated nonstream and streaming gates passed. Recommendation: replace the normal direct-provider path with LiteLLM, which is now deployed, while retaining direct mode as rollback. Consequence: Agent Proxy remains the public policy boundary and no deletion follows from this issue.
Retries - Agent Proxy currently owns retries. LiteLLM is the candidate owner. The prior candidate evaluation intentionally used zero retries and the integrated proof did not add retry-failure injection. Recommendation: retain Agent Proxy retries. Consequence: the agent removes no retry code.
Fallback - Agent Proxy currently owns resilience fallback. LiteLLM proved deterministic provider fallback in the candidate evaluation, while the integrated proof exercised deployment-level direct rollback. Recommendation: retain Agent Proxy fallback until failure injection is repeated through the integrated path in a separate removal issue. Consequence: the agent removes no fallback code now.
Model discovery - Agent Proxy owns public filtering and tower context metadata. LiteLLM owns the provider catalog. The live endpoint exposed only the service-key catalog intersection. Recommendation: retain Agent Proxy filtering and context metadata, and keep LiteLLM as raw provider discovery. Consequence: the public catalog remains curated and context-safe.
Virtual keys - LiteLLM owns the capability and Agent Proxy owns no duplicate. The dedicated service key authenticated and constrained discovery. Recommendation: retain in LiteLLM only. Consequence: Agent Proxy does not grow another key system.
Budgets and spend attribution - LiteLLM plus PostgreSQL own the capability and Agent Proxy owns no duplicate. The earlier evaluation proved persisted spend and budget rejection. Recommendation: retain in LiteLLM only. Consequence: Agent Proxy consumes accounting outcomes rather than storing billing state.
Queueing and circuit resilience - Agent Proxy owns both. LiteLLM has overlapping gateway machinery, but no first consumer supplied admission-pressure or circuit-breaker evidence. Recommendation: retain Agent Proxy queueing and circuit behavior. Consequence: the agent removes nothing from queue.py or the circuit portions of resilience.py.
Correlation and context accounting - Agent Proxy owns both above LiteLLM. The tower observed the 48,128-token safe override and SigNoz joined both services on one body-safe trace. Recommendation: retain in Agent Proxy. Consequence: LiteLLM does not become the public context or correlation boundary.
Durable trajectory - Agent Proxy owns the append-only ledger. The live proof advanced the ledger from 18 to 22 events and preserved it through rollback and restoration. Recommendation: retain in Agent Proxy. Consequence: SigNoz remains operational evidence while Agent Proxy remains the replay and provenance source.
Governed query and restricted data - Dormant Agent Proxy code and documentation hold the current shape. No candidate or consumer exists, and the deployed path retains metadata only. Recommendation: drop from active scope. Consequence: the agent performs no restricted-body, legal-hold, or historical-query work.

Decision required

The agent recommends that Kai approve this disposition, authorize no code deletion, and close #56. If Kai wants commodity retry or fallback consolidation, the agent should open a separate removal issue with integrated failure-injection gates before changing upstream.py, queue.py, or resilience.py.

## Post-integration ownership packet The inner-hop integration is complete. Agent Proxy is Ready on ser8 as the sole public endpoint and routes its provider-facing traffic through LiteLLM. Direct routing remains the observed rollback mode. Agent Proxy #58 and deploy #207 are closed with live evidence. * **Provider routing** - Agent Proxy currently owns public dispatch and rollback. LiteLLM owns the active provider-facing route. The integrated nonstream and streaming gates passed. Recommendation: **replace the normal direct-provider path with LiteLLM, which is now deployed, while retaining direct mode as rollback**. Consequence: Agent Proxy remains the public policy boundary and no deletion follows from this issue. * **Retries** - Agent Proxy currently owns retries. LiteLLM is the candidate owner. The prior candidate evaluation intentionally used zero retries and the integrated proof did not add retry-failure injection. Recommendation: **retain Agent Proxy retries**. Consequence: the agent removes no retry code. * **Fallback** - Agent Proxy currently owns resilience fallback. LiteLLM proved deterministic provider fallback in the candidate evaluation, while the integrated proof exercised deployment-level direct rollback. Recommendation: **retain Agent Proxy fallback until failure injection is repeated through the integrated path in a separate removal issue**. Consequence: the agent removes no fallback code now. * **Model discovery** - Agent Proxy owns public filtering and tower context metadata. LiteLLM owns the provider catalog. The live endpoint exposed only the service-key catalog intersection. Recommendation: **retain Agent Proxy filtering and context metadata, and keep LiteLLM as raw provider discovery**. Consequence: the public catalog remains curated and context-safe. * **Virtual keys** - LiteLLM owns the capability and Agent Proxy owns no duplicate. The dedicated service key authenticated and constrained discovery. Recommendation: **retain in LiteLLM only**. Consequence: Agent Proxy does not grow another key system. * **Budgets and spend attribution** - LiteLLM plus PostgreSQL own the capability and Agent Proxy owns no duplicate. The earlier evaluation proved persisted spend and budget rejection. Recommendation: **retain in LiteLLM only**. Consequence: Agent Proxy consumes accounting outcomes rather than storing billing state. * **Queueing and circuit resilience** - Agent Proxy owns both. LiteLLM has overlapping gateway machinery, but no first consumer supplied admission-pressure or circuit-breaker evidence. Recommendation: **retain Agent Proxy queueing and circuit behavior**. Consequence: the agent removes nothing from `queue.py` or the circuit portions of `resilience.py`. * **Correlation and context accounting** - Agent Proxy owns both above LiteLLM. The tower observed the 48,128-token safe override and SigNoz joined both services on one body-safe trace. Recommendation: **retain in Agent Proxy**. Consequence: LiteLLM does not become the public context or correlation boundary. * **Durable trajectory** - Agent Proxy owns the append-only ledger. The live proof advanced the ledger from 18 to 22 events and preserved it through rollback and restoration. Recommendation: **retain in Agent Proxy**. Consequence: SigNoz remains operational evidence while Agent Proxy remains the replay and provenance source. * **Governed query and restricted data** - Dormant Agent Proxy code and documentation hold the current shape. No candidate or consumer exists, and the deployed path retains metadata only. Recommendation: **drop from active scope**. Consequence: the agent performs no restricted-body, legal-hold, or historical-query work. ## Decision required The agent recommends that Kai approve this disposition, authorize no code deletion, and close #56. If Kai wants commodity retry or fallback consolidation, the agent should open a separate removal issue with integrated failure-injection gates before changing `upstream.py`, `queue.py`, or `resilience.py`.

coilysiren closed this issue

2026-07-25 03:39:54 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy#56

No description provided.

Rows
Columns