[proxy 11/13] resilience.py: per-backend circuit breaker with cooldown

coilysiren commented

2026-06-26 09:48:20 +00:00

Owner

Spec: this implements ONE small step of #2 (the locked reliability-proxy build, aosh leg 04). Do not redesign. Stay tightly scoped to this single step.

Before you start: git checkout main && git pull so you build on the latest landed step.
Extend app/resilience.py with a per-backend circuit breaker. Build on steps 09-10.

Track consecutive failures per backend_url. After a threshold (default 5) OPEN the breaker for a cooldown (default 30s) - while OPEN, dispatch_resilient skips that backend and goes straight to fallback. HALF-OPEN on cooldown expiry: one trial request decides re-close or re-open.
Reflect state in app.obs.llm_circuit_state (0 closed / 1 open / 2 half-open), labelled by backend.

Acceptance: a pytest trips the breaker with repeated failures (asserts gauge == open and the backend is skipped), then after a patched/short cooldown asserts it allows a trial.

When done:

pre-commit run --all-files (or the repo's gate) must pass, commit your work, and push to canonical main (this is an ephemeral feature container - finish the merge, do not leave it for a human).
Update the matching line in docs/FEATURES.md from planned to landed for this feature, in the same push.
Dispatch the next step: ward agent engineer coilyco-flight-deck/agent-proxy#16 --driver goose

**Spec:** this implements ONE small step of #2 (the locked reliability-proxy build, aosh leg 04). **Do not redesign. Stay tightly scoped to this single step.** **Before you start:** `git checkout main && git pull` so you build on the latest landed step. Extend `app/resilience.py` with a per-backend circuit breaker. **Build on steps 09-10.** - Track consecutive failures per backend_url. After a threshold (default 5) OPEN the breaker for a cooldown (default 30s) - while OPEN, `dispatch_resilient` skips that backend and goes straight to fallback. HALF-OPEN on cooldown expiry: one trial request decides re-close or re-open. - Reflect state in `app.obs.llm_circuit_state` (0 closed / 1 open / 2 half-open), labelled by backend. **Acceptance:** a pytest trips the breaker with repeated failures (asserts gauge == open and the backend is skipped), then after a patched/short cooldown asserts it allows a trial. **When done:** 1. `pre-commit run --all-files` (or the repo's gate) must pass, commit your work, and push to canonical `main` (this is an ephemeral feature container - finish the merge, do not leave it for a human). 2. Update the matching line in `docs/FEATURES.md` from `planned` to `landed` for this feature, in the same push. 3. Dispatch the next step: `ward agent engineer coilyco-flight-deck/agent-proxy#16 --driver goose`

coilyco-ops added the

headless

P3

labels

2026-07-02 16:42:43 +00:00

coilysiren commented

2026-07-02 16:59:14 +00:00

Author

Owner

🔒 Reserved by ward agent --driver goose — container engineer-goose-agent-proxy-15 on host 366b54d82a69 is carrying this issue (reserved 2026-07-02T16:59:14Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Goose, via ward agent

🔒 Reserved by `ward agent --driver goose` — container `engineer-goose-agent-proxy-15` on host `366b54d82a69` is carrying this issue (reserved 2026-07-02T16:59:14Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Goose, via `ward agent`

coilysiren referenced this issue

2026-07-02 16:59:55 +00:00

[proxy 10/13] resilience.py: retry with backoff + fallback chain #14

coilyco-ops commented

2026-07-02 17:48:26 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-agent-proxy-15 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T17:48:24Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agent-proxy-15` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T17:48:24Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren commented

2026-07-02 17:48:33 +00:00

Author

Owner

🔓 Reservation released by ward container reap — container agent-proxy (--driver claude) exited without launching the agent (smoke-test death, ward#222/#264), so the hold it took is retracted. A plain ward agent retry no longer needs --force.

— Claude (she/her), via ward agent

🔓 Reservation released by `ward container reap` — container `agent-proxy` (`--driver claude`) exited without launching the agent (smoke-test death, ward#222/#264), so the hold it took is retracted. A plain `ward agent` retry no longer needs `--force`.  — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-02 17:49:13 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-agent-proxy-15 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T17:49:12Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

The pre-flight judged this issue GO for an unattended run. Its justification:

pre-flight read (GO)

Context to front-load: Before my first edit I will read, in the fresh clone: (1) app/resilience.py in full — the existing dispatch_resilient dispatch/fallback flow and whatever failure-tracking or per-backend structures steps 09-10 landed, since I must extend not rewrite it; (2) the app.obs module to learn how gauges/metrics are declared and labelled, so llm_circuit_state matches the existing observability wiring convention (Prometheus client style, label naming); (3) the existing test layout under tests/ for the resilience suite, to match fixture/patch conventions (how cooldown/time is patched, how backends are stubbed); (4) docs/FEATURES.md to find the exact planned line to flip to landed; (5) the repo gate (pre-commit/ward) config. These are all locatable from the issue's own references and I will actually read each before editing.

Main risk: This step is explicitly built on steps 09-10, so the real unknown is whether those landed and left the expected seams (a per-backend failure counter, a dispatch_resilient with a clean fallback branch, and an obs gauge-registration pattern). If 09-10 are on main the work is mechanical and the acceptance test is straightforward (trip breaker → assert gauge==open + backend skipped → patch cooldown → assert half-open trial). The scope is small, reversible, self-contained, and fully verifiable by the specified pytest, with no destructive or external-facing action beyond the standard push-to-main this build cadence already uses.

GO

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agent-proxy-15` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T17:49:12Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. The pre-flight judged this issue **GO** for an unattended run. Its justification: <details><summary>pre-flight read (GO)</summary> **Context to front-load:** Before my first edit I will read, in the fresh clone: (1) `app/resilience.py` in full — the existing `dispatch_resilient` dispatch/fallback flow and whatever failure-tracking or per-backend structures steps 09-10 landed, since I must extend not rewrite it; (2) the `app.obs` module to learn how gauges/metrics are declared and labelled, so `llm_circuit_state` matches the existing observability wiring convention (Prometheus client style, label naming); (3) the existing test layout under `tests/` for the resilience suite, to match fixture/patch conventions (how cooldown/time is patched, how backends are stubbed); (4) `docs/FEATURES.md` to find the exact `planned` line to flip to `landed`; (5) the repo gate (`pre-commit`/`ward`) config. These are all locatable from the issue's own references and I will actually read each before editing. **Main risk:** This step is explicitly built on steps 09-10, so the real unknown is whether those landed and left the expected seams (a per-backend failure counter, a `dispatch_resilient` with a clean fallback branch, and an `obs` gauge-registration pattern). If 09-10 are on `main` the work is mechanical and the acceptance test is straightforward (trip breaker → assert gauge==open + backend skipped → patch cooldown → assert half-open trial). The scope is small, reversible, self-contained, and fully verifiable by the specified pytest, with no destructive or external-facing action beyond the standard push-to-main this build cadence already uses. GO </details>  — Claude (she/her), via `ward agent`

coilysiren commented

2026-07-03 19:33:31 +00:00

Author

Owner

Landed. app/resilience.py carries the per-backend circuit breaker (closed/open/half-open with cooldown), the llm_circuit_state gauge is exported, and tests/test_circuit.py covers the trip-then-recover path. Closing as done.

coilysiren closed this issue

2026-07-03 19:33:31 +00:00

coilysiren referenced this issue from coilyco-flight-deck/ward

2026-07-14 15:27:09 +00:00

director should emit colorized, space buffered, visually distinct messages about the categories of actions its performing on startup #1283

Rows
Columns

[proxy 11/13] resilience.py: per-backend circuit breaker with cooldown #15