coilyco-flight-deck/agent-proxy

Watch

Decide how operational views stay cheap as the ledger grows: staleness, incremental, or cache #145

New issue

Closed

opened 2026-08-19 15:53:16 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-19 15:53:16 +00:00

Owner

The decision

_operational_builder() in app/trajectory/api.py rebuilds every retained trajectory on every request to /v1/trajectory/views/{view_name} and /v1/trajectory/dossiers/{trajectory_id}. That rebuild is what makes a read reflect events ingested a moment ago. It is also why read cost tracks total retained history rather than the size of the answer.

Making it cheap means giving something up. Which trade to take is the open question, and no code should land until it is picked.

Where it stands now

Measured on main at 4fa0bbe, synthetic ledger of request-lifecycle event pairs, steady state where nothing changed and every write is a no-op. One _operational_builder() call:

500 turns - 65.2 ms
1000 turns - 128.4 ms
2000 turns - 296.9 ms
4000 turns - 648.0 ms
8000 turns - 1500.0 ms

Two constant-factor fixes already landed and are not what remains: #143 removed a SQLite connect per record, #144 removed a second full read of the events table. Together they took a 2000-turn rebuild from 617 ms to 297 ms. Neither touched the shape of the curve.

Correcting #142: it called the growth linear at about 0.2 ms per turn. Measuring further out shows it is mildly superlinear. Per-turn cost climbs from 0.130 ms at 500 turns to 0.188 ms at 8000, and each doubling of the ledger costs about 2.2 to 2.3 times as much rather than 2.0. Roughly O(n^1.15).

The ledger cannot shrink

This is not a case where retention eventually caps the problem. events carries a SQL trigger that aborts any delete (app/trajectory/store.py:170, "events are append-only"), and there is no pruning, expiry, or compaction anywhere in the tree. retention_class is a tag carried on the envelope and read by nothing that removes rows. So the curve above has no ceiling short of a schema change.

At #140's observed rate of roughly 375 completions a day, 8000 turns is about three weeks. Extrapolating the fitted curve past what was measured, a year of retention puts a view request in the tens of seconds.

The dossier case is the sharpest version

OperationalViewBuilder.dossier needs exactly one materialized trajectory plus that trajectory's evaluations. Once the rebuild is done, the lookup itself measures at under 0.01 ms. So essentially 100 percent of a dossier request's cost is rebuilding the rest of the ledger it then discards.

MaterializationStore.latest(trajectory_id) and EvaluationStore.for_trajectory(trajectory_id) already exist and are both index-backed. The only thing standing between a dossier and an O(1) read is the freshness promise.

Options

Accept a staleness window. Serve dossiers from latest plus for_trajectory, and move the rebuild to a timer or to ingest. Smallest change, and it is the only one that makes a dossier genuinely O(1). A read can miss an event ingested seconds ago, so the window has to be stated and probably surfaced on the response.

Materialize incrementally. Advance from a stored watermark and touch only trajectories with new events. Keeps read-your-writes exactly as it is today. Harder than it sounds: trajectory_id is derived by unioning strong correlations across events (_STRONG_CORRELATIONS, materialize.py:22), not stored on a row, so there is no event-to-trajectory mapping to index. A new event can also merge two previously separate trajectories. This needs a real design pass, not a patch.

Cache the built views with invalidation on ingest. Keeps freshness for the view endpoints and is easy to reason about. Does nothing for the first request after any write, which on a busy lane may be most of them, and nothing for dossiers of distinct ids.

What would settle it

How stale may an operational view be? If seconds are acceptable, the staleness option is clearly right and cheap. If a view must never miss a just-ingested event, incremental materialization is the only real answer and should be scoped as its own piece of work.

Reproducing

Ingest N request-lifecycle event pairs into a TrajectoryStore, call _operational_builder() once to warm it, then time a second call. The second call does no useful work and costs the full amount.

#142 - predecessor, closed
#143, #144 - the landed constant-factor fixes
#140 - the completion-rate figure the extrapolation uses

Successor to #142, which is closed. That issue accumulated two landed fixes and its numbers no longer describe the code. This one carries only the part still open, which is a decision rather than a defect. ## The decision `_operational_builder()` in [`app/trajectory/api.py`](app/trajectory/api.py) rebuilds every retained trajectory on every request to `/v1/trajectory/views/{view_name}` and `/v1/trajectory/dossiers/{trajectory_id}`. That rebuild is what makes a read reflect events ingested a moment ago. It is also why read cost tracks total retained history rather than the size of the answer. Making it cheap means giving something up. **Which trade to take is the open question**, and no code should land until it is picked. ## Where it stands now Measured on `main` at 4fa0bbe, synthetic ledger of request-lifecycle event pairs, steady state where nothing changed and every write is a no-op. One `_operational_builder()` call: * 500 turns - 65.2 ms * 1000 turns - 128.4 ms * 2000 turns - 296.9 ms * 4000 turns - 648.0 ms * 8000 turns - 1500.0 ms Two constant-factor fixes already landed and are **not** what remains: #143 removed a SQLite connect per record, #144 removed a second full read of the events table. Together they took a 2000-turn rebuild from 617 ms to 297 ms. Neither touched the shape of the curve. **Correcting #142:** it called the growth linear at about 0.2 ms per turn. Measuring further out shows it is mildly superlinear. Per-turn cost climbs from 0.130 ms at 500 turns to 0.188 ms at 8000, and each doubling of the ledger costs about 2.2 to 2.3 times as much rather than 2.0. Roughly O(n^1.15). ## The ledger cannot shrink This is not a case where retention eventually caps the problem. `events` carries a SQL trigger that aborts any delete (`app/trajectory/store.py:170`, "events are append-only"), and there is no pruning, expiry, or compaction anywhere in the tree. `retention_class` is a tag carried on the envelope and read by nothing that removes rows. So the curve above has no ceiling short of a schema change. At #140's observed rate of roughly 375 completions a day, 8000 turns is about three weeks. Extrapolating the fitted curve past what was measured, a year of retention puts a view request in the tens of seconds. ## The dossier case is the sharpest version `OperationalViewBuilder.dossier` needs exactly one materialized trajectory plus that trajectory's evaluations. Once the rebuild is done, the lookup itself measures at under 0.01 ms. So essentially **100 percent** of a dossier request's cost is rebuilding the rest of the ledger it then discards. `MaterializationStore.latest(trajectory_id)` and `EvaluationStore.for_trajectory(trajectory_id)` already exist and are both index-backed. The only thing standing between a dossier and an O(1) read is the freshness promise. ## Options **Accept a staleness window.** Serve dossiers from `latest` plus `for_trajectory`, and move the rebuild to a timer or to ingest. Smallest change, and it is the only one that makes a dossier genuinely O(1). A read can miss an event ingested seconds ago, so the window has to be stated and probably surfaced on the response. **Materialize incrementally.** Advance from a stored watermark and touch only trajectories with new events. Keeps read-your-writes exactly as it is today. Harder than it sounds: `trajectory_id` is derived by unioning strong correlations across events (`_STRONG_CORRELATIONS`, `materialize.py:22`), not stored on a row, so there is no event-to-trajectory mapping to index. A new event can also merge two previously separate trajectories. This needs a real design pass, not a patch. **Cache the built views** with invalidation on ingest. Keeps freshness for the view endpoints and is easy to reason about. Does nothing for the first request after any write, which on a busy lane may be most of them, and nothing for dossiers of distinct ids. ## What would settle it How stale may an operational view be? If seconds are acceptable, the staleness option is clearly right and cheap. If a view must never miss a just-ingested event, incremental materialization is the only real answer and should be scoped as its own piece of work. ## Reproducing Ingest N request-lifecycle event pairs into a `TrajectoryStore`, call `_operational_builder()` once to warm it, then time a second call. The second call does no useful work and costs the full amount. ## Related * #142 - predecessor, closed * #143, #144 - the landed constant-factor fixes * #140 - the completion-rate figure the extrapolation uses

coilyco-ops referenced this issue

2026-08-19 15:53:30 +00:00

Operational views and dossiers rebuild the whole ledger on every request, so read cost grows without bound #142

coilyco-ops commented

2026-08-19 15:58:11 +00:00

Author

Owner

Nothing automated consumes them

The only caller anywhere is app/trajectory/query.py, the just trajectory-query helper, driven by a human or an agent doing an investigation. There is no poller, no dashboard scrape, no service dependency.

In particular Ward does not call them, despite WardDossierInput carrying the name. I cloned ward and grepped: it references agent-proxy only as a model transport endpoint (internal/agents/opencode/config.go:67 names it as a provider, cmd/ward/agent.go prints the endpoint). No reference to /v1/trajectory/dossiers, /v1/trajectory/views, or dossier-input. coilyco-bridge/deploy has none either.

The docs already call them cold path

operational-views.md opens with "internal cold-path views", and the Ward boundary section pins may_authorize to always false, with Ward alone deciding authorization. So a dossier is evidence someone reads, never a gate anything waits on.

The contract already has a staleness field, and it is currently always zero

ViewFreshness (views.py:44) ships generated_at, source_materialized_at, complete_through, and age_seconds on every view response. complete_through is the watermark: this view is complete through time T.

That field set exists precisely to describe a view that was built earlier. Rebuilding on read makes age_seconds always about zero, so the service is paying full price for a freshness guarantee its own published contract never promised. Moving to a built-ahead view needs no contract change at all, only a field that currently reads ~0 starting to carry a real number.

The per-request figures understate it

investigate fetches four views and then one dossier per matched trajectory, each a separate HTTP request and so each a separate full rebuild. One command costs 4 + N rebuilds.

Extrapolating from the measured 1500 ms at 8000 turns, an investigation matching 10 trajectories is 14 rebuilds, so about 21 seconds for one just trajectory-query investigate. That, not the single-request number, is what an operator actually feels.

Recommendation

Take the staleness option. On this evidence it is not a trade so much as aligning the implementation with the contract already published: cold path by documentation, no automated consumer, no authorization role, and a freshness envelope built to report an age.

Concretely: build views ahead (on ingest or on a timer), serve dossiers from MaterializationStore.latest plus EvaluationStore.for_trajectory, and let age_seconds and complete_through report the real window. Dossiers become O(1) and investigate stops multiplying.

What would still change the answer: a plan to have Ward, or anything else automated, read a dossier at a decision point. That would make read-your-writes load-bearing and push this back toward incremental materialization. Worth confirming that is not on the roadmap before anyone builds.

Went looking for who actually reads these endpoints, because the staleness question is unanswerable without that. Four findings, and together they make the question much narrower than the body above implies. ## Nothing automated consumes them The only caller anywhere is `app/trajectory/query.py`, the `just trajectory-query` helper, driven by a human or an agent doing an investigation. There is no poller, no dashboard scrape, no service dependency. In particular **Ward does not call them**, despite `WardDossierInput` carrying the name. I cloned ward and grepped: it references agent-proxy only as a model transport endpoint (`internal/agents/opencode/config.go:67` names it as a provider, `cmd/ward/agent.go` prints the endpoint). No reference to `/v1/trajectory/dossiers`, `/v1/trajectory/views`, or `dossier-input`. coilyco-bridge/deploy has none either. ## The docs already call them cold path [operational-views.md](docs/operational-views.md) opens with "internal **cold-path** views", and the Ward boundary section pins `may_authorize` to always false, with Ward alone deciding authorization. So a dossier is evidence someone reads, never a gate anything waits on. ## The contract already has a staleness field, and it is currently always zero `ViewFreshness` (`views.py:44`) ships `generated_at`, `source_materialized_at`, `complete_through`, and `age_seconds` on every view response. `complete_through` is the watermark: this view is complete through time T. That field set exists precisely to describe a view that was built earlier. Rebuilding on read makes `age_seconds` always about zero, so **the service is paying full price for a freshness guarantee its own published contract never promised**. Moving to a built-ahead view needs no contract change at all, only a field that currently reads ~0 starting to carry a real number. ## The per-request figures understate it `investigate` fetches four views and then one dossier per matched trajectory, each a separate HTTP request and so each a separate full rebuild. One command costs **4 + N rebuilds**. Extrapolating from the measured 1500 ms at 8000 turns, an investigation matching 10 trajectories is 14 rebuilds, so about 21 seconds for one `just trajectory-query investigate`. That, not the single-request number, is what an operator actually feels. ## Recommendation Take the staleness option. On this evidence it is not a trade so much as aligning the implementation with the contract already published: cold path by documentation, no automated consumer, no authorization role, and a freshness envelope built to report an age. Concretely: build views ahead (on ingest or on a timer), serve dossiers from `MaterializationStore.latest` plus `EvaluationStore.for_trajectory`, and let `age_seconds` and `complete_through` report the real window. Dossiers become O(1) and `investigate` stops multiplying. **What would still change the answer:** a plan to have Ward, or anything else automated, read a dossier at a decision point. That would make read-your-writes load-bearing and push this back toward incremental materialization. Worth confirming that is not on the roadmap before anyone builds.

coilyco-ops added the

role/science

priority

labels

2026-08-25 08:11:54 +00:00

coilyco-ops added the

autonomy

async-consult

label

2026-08-25 09:31:03 +00:00

coilyco-ops commented

2026-08-27 03:19:40 +00:00

Author

Owner

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901).

Closing. The only caller is a human-driven query helper, the docs already describe these as cold-path views, and ViewFreshness already ships the staleness contract. A 1.5 second rebuild at 8000 turns on a path nobody polls does not earn a design decision.

This was priority/P3, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless.

If this is live work, reopen it. The whole set is recoverable with state:closed label:burndown-2026-08.

Closed in the 2026-08-26 backlog burn-down (coilyco-bridge/agentic-os-kai#901). Closing. The only caller is a human-driven query helper, the docs already describe these as cold-path views, and ViewFreshness already ships the staleness contract. A 1.5 second rebuild at 8000 turns on a path nobody polls does not earn a design decision. This was `priority/P3`, the default tier that unsure and unscored issues land in, so it had never been positively judged worth keeping. The 2026-08-26 burn-down ranked the whole P3 pool and cut the bottom. Closed here means not on the list, not wrong or worthless. If this is live work, reopen it. The whole set is recoverable with `state:closed label:burndown-2026-08`.

coilyco-ops

2026-08-27 03:19:43 +00:00

closed this issue
added the
burndown-2026-08
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy#145

No description provided.

Rows
Columns