Add headless ledger recovery and LiteLLM evaluation evidence #57

New issue

Closed

opened 2026-07-24 20:40:24 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-07-24 20:40:24 +00:00

Member

Parent: #56

Outcome

Provide secret-safe, machine-readable evidence for the durable trajectory ledger and the standalone LiteLLM candidate. The tooling must support a headless evaluation without authorizing a traffic cutover.

Scope

Add a ledger verification command that creates a consistent SQLite backup, restores or opens the backup, replays canonical events into a fresh store, and reports counts plus quarantine outcomes without emitting stored bodies.
Extend the LiteLLM parity command to accept credentials from environment variables so keys never appear in argv or artifacts.
Probe the explicit parity gates required by #56 where an OpenAI-compatible request can prove them, including tool-call shape, reasoning fields when present, caller-supplied safe num_ctx, streaming termination, usage, and unknown-model behavior.
Keep live-only key, budget, spend, fallback, trace-join, and delivered-context checks explicit in the report instead of claiming them from repository evidence.
Keep cutover_authorized false.

Acceptance

Unit tests cover backup/replay evidence and every new parity result.
JSON output contains no API keys, prompts, responses, or stored raw envelopes.
Ward exposes the verification commands.
Repository validation passes.

Closes as a child implementation slice of #56.

Parent: #56 ## Outcome Provide secret-safe, machine-readable evidence for the durable trajectory ledger and the standalone LiteLLM candidate. The tooling must support a headless evaluation without authorizing a traffic cutover. ## Scope * Add a ledger verification command that creates a consistent SQLite backup, restores or opens the backup, replays canonical events into a fresh store, and reports counts plus quarantine outcomes without emitting stored bodies. * Extend the LiteLLM parity command to accept credentials from environment variables so keys never appear in argv or artifacts. * Probe the explicit parity gates required by #56 where an OpenAI-compatible request can prove them, including tool-call shape, reasoning fields when present, caller-supplied safe `num_ctx`, streaming termination, usage, and unknown-model behavior. * Keep live-only key, budget, spend, fallback, trace-join, and delivered-context checks explicit in the report instead of claiming them from repository evidence. * Keep `cutover_authorized` false. ## Acceptance * Unit tests cover backup/replay evidence and every new parity result. * JSON output contains no API keys, prompts, responses, or stored raw envelopes. * Ward exposes the verification commands. * Repository validation passes. Closes as a child implementation slice of #56.