[proxy 12/13] analysis.py: context-budget guard (token count + trim) #16

New issue

Closed

opened 2026-06-26 09:48:20 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-06-26 09:48:20 +00:00

Owner

Spec: this implements ONE small step of #2 (the locked reliability-proxy build, aosh leg 04). Do not redesign. Stay tightly scoped to this single step.

Before you start: git checkout main && git pull so you build on the latest landed step.
Fill in app/analysis.py - never let an over-budget prompt reach the model.

count_tokens(messages) -> int using tiktoken (a cl100k/o200k base encoder is fine as an estimate).
fit_to_budget(messages, num_ctx) -> messages: if the prompt exceeds a safe fraction (e.g. 90%) of num_ctx, trim the OLDEST non-system turns (preserve all system messages and the latest user turn) until it fits. Increment app.obs.llm_truncation_avoided_total when it trims.
Call fit_to_budget in the request path (before dispatch) using the logical model's num_ctx.

Acceptance: a pytest builds an over-budget message list and asserts the result fits the budget, the system message survives, and the counter incremented.

When done:

pre-commit run --all-files (or the repo's gate) must pass, commit your work, and push to canonical main (this is an ephemeral feature container - finish the merge, do not leave it for a human).
Update the matching line in docs/FEATURES.md from planned to landed for this feature, in the same push.
Dispatch the next step: ward agent engineer coilyco-flight-deck/agent-proxy#17 --driver goose

**Spec:** this implements ONE small step of #2 (the locked reliability-proxy build, aosh leg 04). **Do not redesign. Stay tightly scoped to this single step.** **Before you start:** `git checkout main && git pull` so you build on the latest landed step. Fill in `app/analysis.py` - never let an over-budget prompt reach the model. - `count_tokens(messages) -> int` using tiktoken (a cl100k/o200k base encoder is fine as an estimate). - `fit_to_budget(messages, num_ctx) -> messages`: if the prompt exceeds a safe fraction (e.g. 90%) of `num_ctx`, trim the OLDEST non-system turns (preserve all `system` messages and the latest user turn) until it fits. Increment `app.obs.llm_truncation_avoided_total` when it trims. - Call `fit_to_budget` in the request path (before dispatch) using the logical model's num_ctx. **Acceptance:** a pytest builds an over-budget message list and asserts the result fits the budget, the system message survives, and the counter incremented. **When done:** 1. `pre-commit run --all-files` (or the repo's gate) must pass, commit your work, and push to canonical `main` (this is an ephemeral feature container - finish the merge, do not leave it for a human). 2. Update the matching line in `docs/FEATURES.md` from `planned` to `landed` for this feature, in the same push. 3. Dispatch the next step: `ward agent engineer coilyco-flight-deck/agent-proxy#17 --driver goose`

coilyco-ops added the

headless

labels

2026-07-02 16:42:44 +00:00

coilyco-ops commented

2026-07-02 16:44:59 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-agent-proxy-16 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T16:44:57Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agent-proxy-16` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T16:44:57Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-02 16:48:18 +00:00

feat: add fit_to_budget + message-aware count_tokens public surface

coilysiren closed this issue

2026-07-02 16:48:18 +00:00

coilysiren commented

2026-07-02 16:49:06 +00:00

Author

Owner

WARD-OUTCOME: done - fit_to_budget + message-aware count_tokens landed on main (f35607d), full pytest green, issue closed via trailer.

Honest retro: the substantive context-budget guard was already sitting on main from the resilience-proxy core commit, wired into the request path, tested, and marked landed in FEATURES.md. So the real work here was reconciling the spec's named public surface (count_tokens(messages) -> int, fit_to_budget(messages, num_ctx) -> messages) with the richer internals that already existed under different names (count_message_tokens, apply_context_budget). I chose thin additive wrappers over a rename so I honored the spec's API without a redesign or touching the live request-path call, which felt like the right read of "do not redesign, stay tightly scoped."

What fought back: the container disk was pinned at 100%, so the full dev extra (mypy + ruff) could not extract. I installed just pytest into the venv to run the gate. All 39 tests pass. The other rough edge: the ward agent engineer #17 --driver goose dispatch could not fire because there is no Docker daemon in this feature container, so #17 still needs a launch from an environment that can spin up siblings. Confidence in the code is high (small, well-tested wrappers over already-proven internals). Worth a follow-up: someone should re-trigger the #17 dispatch.

WARD-OUTCOME: done - fit_to_budget + message-aware count_tokens landed on main (f35607d), full pytest green, issue closed via trailer. Honest retro: the substantive context-budget guard was already sitting on main from the resilience-proxy core commit, wired into the request path, tested, and marked landed in FEATURES.md. So the real work here was reconciling the spec's named public surface (`count_tokens(messages) -> int`, `fit_to_budget(messages, num_ctx) -> messages`) with the richer internals that already existed under different names (`count_message_tokens`, `apply_context_budget`). I chose thin additive wrappers over a rename so I honored the spec's API without a redesign or touching the live request-path call, which felt like the right read of "do not redesign, stay tightly scoped." What fought back: the container disk was pinned at 100%, so the full `dev` extra (mypy + ruff) could not extract. I installed just pytest into the venv to run the gate. All 39 tests pass. The other rough edge: the `ward agent engineer #17 --driver goose` dispatch could not fire because there is no Docker daemon in this feature container, so #17 still needs a launch from an environment that can spin up siblings. Confidence in the code is high (small, well-tested wrappers over already-proven internals). Worth a follow-up: someone should re-trigger the #17 dispatch.