[proxy 10/13] resilience.py: retry with backoff + fallback chain #14

New issue

Closed

opened 2026-06-26 09:48:20 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-06-26 09:48:20 +00:00

Owner

Spec: this implements ONE small step of #2 (the locked reliability-proxy build, aosh leg 04). Do not redesign. Stay tightly scoped to this single step.

Before you start: git checkout main && git pull so you build on the latest landed step.
Extend app/resilience.py with retry and fallback. Build on step 09's validation.

dispatch_resilient(logical_name, messages): call upstream.dispatch_chat + validate_response; on failure RETRY up to N times (default 3) with exponential backoff. Increment app.obs.llm_retries_total per retry.
FALLBACK: extend the registry so a logical model can list an ordered backend chain; on exhausting retries for one backend, advance to the next. Increment app.obs.llm_fallbacks_total per fallback. (For now a single-entry chain is fine - just wire the mechanism.)
Route the worker through dispatch_resilient instead of calling upstream directly.

Acceptance: pytests with a monkeypatched upstream that fails twice then succeeds -> result returned, llm_retries_total incremented; a backend that always fails -> falls back / surfaces a clean error.

When done:

pre-commit run --all-files (or the repo's gate) must pass, commit your work, and push to canonical main (this is an ephemeral feature container - finish the merge, do not leave it for a human).
Update the matching line in docs/FEATURES.md from planned to landed for this feature, in the same push.
Dispatch the next step: ward agent engineer coilyco-flight-deck/agent-proxy#15 --driver goose

**Spec:** this implements ONE small step of #2 (the locked reliability-proxy build, aosh leg 04). **Do not redesign. Stay tightly scoped to this single step.** **Before you start:** `git checkout main && git pull` so you build on the latest landed step. Extend `app/resilience.py` with retry and fallback. **Build on step 09's validation.** - `dispatch_resilient(logical_name, messages)`: call `upstream.dispatch_chat` + `validate_response`; on failure RETRY up to N times (default 3) with exponential backoff. Increment `app.obs.llm_retries_total` per retry. - FALLBACK: extend the registry so a logical model can list an ordered backend chain; on exhausting retries for one backend, advance to the next. Increment `app.obs.llm_fallbacks_total` per fallback. (For now a single-entry chain is fine - just wire the mechanism.) - Route the worker through `dispatch_resilient` instead of calling upstream directly. **Acceptance:** pytests with a monkeypatched upstream that fails twice then succeeds -> result returned, `llm_retries_total` incremented; a backend that always fails -> falls back / surfaces a clean error. **When done:** 1. `pre-commit run --all-files` (or the repo's gate) must pass, commit your work, and push to canonical `main` (this is an ephemeral feature container - finish the merge, do not leave it for a human). 2. Update the matching line in `docs/FEATURES.md` from `planned` to `landed` for this feature, in the same push. 3. Dispatch the next step: `ward agent engineer coilyco-flight-deck/agent-proxy#15 --driver goose`

coilyco-ops added the

headless

labels

2026-07-02 16:42:42 +00:00

coilyco-ops commented

2026-07-02 16:45:52 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-agent-proxy-14 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T16:45:50Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agent-proxy-14` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T16:45:50Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-02 16:58:46 +00:00

feat(resilience): name-based dispatch_resilient entry + retry/fallback acceptance

coilysiren closed this issue

2026-07-02 16:58:46 +00:00

coilysiren referenced this issue from a commit

2026-07-02 16:58:46 +00:00

Merge issue-14: name-based dispatch_resilient + retry/fallback acceptance (closes #14)

coilysiren commented

2026-07-02 16:59:55 +00:00

Author

Owner

WARD-OUTCOME: done - dispatch_resilient(logical_name, messages) landed on main with retry/fallback acceptance tests; also fixed a latent OTel attempt-span leak the tests surfaced. Gate green (58 passed).

This one turned out to be more "finish and verify" than "build": the resilience engine (dispatch, with retry-backoff, fallback chain, circuit breaker, and step-09 validation) had already landed as part of the resilience-core chunk, and the worker already routed through it. So the honest gap for this step was the spec'd name-based entry point, which didn't exist, plus the acceptance tests the issue actually asks for, which were genuinely missing. I added dispatch_resilient as a thin adapter over the existing engine rather than redesigning anything.

What fought back: writing the acceptance tests immediately surfaced a real bug I didn't expect. The per-attempt OTel span was only closed on the success and validation-terminal paths, so every retry or fallback leaked its context token, and the next attempt's detach crashed with "token created in a different Context" whenever a tracer was active - i.e. any deploy with opentelemetry installed. It had been latent since the observability instrumentation landed because no prior test exercised retries under a live tracer. Moved the span close into a finally, which fixed it cleanly.

The other thing that fought back was the box itself: the shared host disk was pegged at 100% for the first several minutes (couldn't even build a venv), then a sibling container freed ~5G and the full suite ran fine. Confidence in the code is high - 58 tests pass on the merged-with-latest-main tree, and the change is additive and low-blast-radius.

Rough edges / follow-ups: (1) the automatic dispatch of the next step (#15) failed - this nested container has no Docker daemon, so ward agent can't spawn a sibling here; #15 will need to be kicked off from a host that can. (2) Worth considering whether the worker should route through dispatch_resilient by name for consistency, but it already holds a resolved LogicalModel from the route's 404 guard, so re-resolving would be a small regression - I left it calling the engine directly on purpose.

— Claude (she/her), via ward agent

WARD-OUTCOME: done - `dispatch_resilient(logical_name, messages)` landed on main with retry/fallback acceptance tests; also fixed a latent OTel attempt-span leak the tests surfaced. Gate green (58 passed). This one turned out to be more "finish and verify" than "build": the resilience engine (`dispatch`, with retry-backoff, fallback chain, circuit breaker, and step-09 validation) had already landed as part of the resilience-core chunk, and the worker already routed through it. So the honest gap for this step was the spec'd **name-based** entry point, which didn't exist, plus the acceptance tests the issue actually asks for, which were genuinely missing. I added `dispatch_resilient` as a thin adapter over the existing engine rather than redesigning anything. What fought back: writing the acceptance tests immediately surfaced a real bug I didn't expect. The per-attempt OTel span was only closed on the success and validation-terminal paths, so every retry or fallback leaked its context token, and the next attempt's detach crashed with "token created in a different Context" whenever a tracer was active - i.e. any deploy with opentelemetry installed. It had been latent since the observability instrumentation landed because no prior test exercised retries under a live tracer. Moved the span close into a `finally`, which fixed it cleanly. The other thing that fought back was the box itself: the shared host disk was pegged at 100% for the first several minutes (couldn't even build a venv), then a sibling container freed ~5G and the full suite ran fine. Confidence in the code is high - 58 tests pass on the merged-with-latest-main tree, and the change is additive and low-blast-radius. Rough edges / follow-ups: (1) the automatic dispatch of the next step (#15) failed - this nested container has no Docker daemon, so `ward agent` can't spawn a sibling here; #15 will need to be kicked off from a host that can. (2) Worth considering whether the worker should route through `dispatch_resilient` by name for consistency, but it already holds a resolved `LogicalModel` from the route's 404 guard, so re-resolving would be a small regression - I left it calling the engine directly on purpose. — Claude (she/her), via ward agent