coilyco-flight-deck/agent-compose

Watch

Composed role bodies span 7x with no budget, and the biggest one lands on a chat lane #275

New issue

Open

opened 2026-08-12 20:35:58 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-12 20:35:58 +00:00

Member

Agent Compose has size budgets on its sources (the agent-compose-size and agent-compose-dedup hooks) but none on its output. The composed body is what a consumer pays for on every single turn, and nothing measures or bounds it.

The numbers

From coilyco-bridge/deploy services/sirens-echo/rendered/sirens-deep-bundle.txt, the eight roles available to the Sirens Deep lane:

role	composed body bytes
director	6,630
qa	6,636
strats	6,641
engineer	6,708
ops	7,674
design	8,590
ai	15,297
creator	46,942

Six roles sit under 9 KB. creator is 7x the median and 3x the next largest, carried mostly by a block of aos-public/personal-preference-* skills.

Why this surfaced

agent-proxy#101 measured Sirens Deep shipping a 53,133 byte system prompt against a four byte user turn, on all 46 turns of the window.

53,133 is very close to creator's 46,942 composed body plus its 2,487 identity card plus harness framing. No other role is within 35 KB of that figure. I have not confirmed which role Deep runs at runtime and the bundle enumerates all eight rather than naming one, so please treat the attribution as an inference from arithmetic and confirm it before acting.

Why prompt caching does not close this

The obvious reading of #101 was "enable prompt caching". It was already on. DeepSeek caches repeated prefixes automatically and a synthetic back to back probe through Agent Proxy measured a 94.8% hit rate.

Caching makes the prefix cheaper. It does not make it smaller, and it leaves the following untouched:

Context occupancy. The tokens are resident every turn regardless of what they cost to produce. On sirens-echo/default against ornith:35b the derived num_ctx is roughly 39k, so a 15k token prefix consumes better than a third of the window before any conversation exists.
Decode cost. A cache hit skips prefill, not attention. Every generated token still reads across the whole cached prefix.
Cold starts. Deep's traffic is bursty, with 20 hour gaps between clusters. Provider caches expire, so the first turn after every gap pays the full uncached rate.
Attention quality. More resident context is not free accuracy.

The question

Should a composed body carry a budget that varies by consumer class? An unattended coding harness with a large window and a long task can absorb 47 KB. An interactive chat lane answering a four byte ping cannot, and pays it on every turn forever.

Concretely, worth deciding:

Is creator at 47 KB intentional, or is it accumulated drift? The personal-preference-* block is the obvious place to look first.
Should composition emit its output size as a first class number, the way the source hooks already bound inputs?
Should a role destined for a chat lane compose a reduced body, or should chat lanes select a smaller role?

Measurement is already in place

Agent Proxy now records gen_ai.request.system_bytes, gen_ai.request.tool_count, and gen_ai.request.tool_bytes on every request.chat span, plus provider prompt cache accounting. So the before and after of any change here is measurable at the proxy without new instrumentation. See agent-proxy#103.

agent-proxy#101 - the original measurement, now closed, with the caching premise corrected in its comments.
sirens-echo#162 - the consumer side, including the separate tool roster question.
deploy#404 - the caching and roster contracts, now recorded in Deploy.

Agent Compose has size budgets on its *sources* (the `agent-compose-size` and `agent-compose-dedup` hooks) but none on its *output*. The composed body is what a consumer pays for on every single turn, and nothing measures or bounds it. ## The numbers From `coilyco-bridge/deploy` `services/sirens-echo/rendered/sirens-deep-bundle.txt`, the eight roles available to the Sirens Deep lane: | role | composed body bytes | |---|---| | director | 6,630 | | qa | 6,636 | | strats | 6,641 | | engineer | 6,708 | | ops | 7,674 | | design | 8,590 | | ai | 15,297 | | **creator** | **46,942** | Six roles sit under 9 KB. `creator` is **7x the median** and 3x the next largest, carried mostly by a block of `aos-public/personal-preference-*` skills. ## Why this surfaced [agent-proxy#101](https://forgejo.coilysiren.me/coilyco-flight-deck/agent-proxy/issues/101) measured Sirens Deep shipping a **53,133 byte** system prompt against a four byte user turn, on all 46 turns of the window. 53,133 is very close to `creator`'s 46,942 composed body plus its 2,487 identity card plus harness framing. No other role is within 35 KB of that figure. **I have not confirmed which role Deep runs at runtime** and the bundle enumerates all eight rather than naming one, so please treat the attribution as an inference from arithmetic and confirm it before acting. ## Why prompt caching does not close this The obvious reading of #101 was "enable prompt caching". It was already on. DeepSeek caches repeated prefixes automatically and a synthetic back to back probe through Agent Proxy measured a 94.8% hit rate. Caching makes the prefix cheaper. It does not make it smaller, and it leaves the following untouched: - **Context occupancy.** The tokens are resident every turn regardless of what they cost to produce. On `sirens-echo/default` against `ornith:35b` the derived `num_ctx` is roughly 39k, so a 15k token prefix consumes better than a third of the window before any conversation exists. - **Decode cost.** A cache hit skips prefill, not attention. Every generated token still reads across the whole cached prefix. - **Cold starts.** Deep's traffic is bursty, with 20 hour gaps between clusters. Provider caches expire, so the first turn after every gap pays the full uncached rate. - **Attention quality.** More resident context is not free accuracy. ## The question Should a composed body carry a budget that varies by consumer class? An unattended coding harness with a large window and a long task can absorb 47 KB. An interactive chat lane answering a four byte ping cannot, and pays it on every turn forever. Concretely, worth deciding: 1. Is `creator` at 47 KB intentional, or is it accumulated drift? The `personal-preference-*` block is the obvious place to look first. 2. Should composition emit its output size as a first class number, the way the source hooks already bound inputs? 3. Should a role destined for a chat lane compose a reduced body, or should chat lanes select a smaller role? ## Measurement is already in place Agent Proxy now records `gen_ai.request.system_bytes`, `gen_ai.request.tool_count`, and `gen_ai.request.tool_bytes` on every `request.chat` span, plus provider prompt cache accounting. So the before and after of any change here is measurable at the proxy without new instrumentation. See [agent-proxy#103](https://forgejo.coilysiren.me/coilyco-flight-deck/agent-proxy/pulls/103). ## Related - [agent-proxy#101](https://forgejo.coilysiren.me/coilyco-flight-deck/agent-proxy/issues/101) - the original measurement, now closed, with the caching premise corrected in its comments. - [sirens-echo#162](https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/162) - the consumer side, including the separate tool roster question. - [deploy#404](https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/404) - the caching and roster contracts, now recorded in Deploy.

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-12 20:36:13 +00:00

Every turn ships a 53 KB system prompt, byte-identical, uncached #162

coilyco-ops commented

2026-08-12 20:40:55 +00:00

Author

Member

Measured this on the live sirens-echo/deepseek route through Agent Proxy, using a synthetic 53,133 byte system prompt matching the size #101 reported. Five calls, varying only where the prompt changes:

call	prompt tokens	cached	hit rate
A, cold	10,714	0	0.0%
B, identical prefix, different user turn	10,718	10,624	99.1%
C, 9 bytes prepended to the front	10,716	0	0.0%
D, 9 bytes appended to the tail	10,716	10,624	99.1%
E, identical prefix again	10,716	10,624	99.1%

Two results worth carrying into the design.

Caching is prefix-based, not request-based. B and E change the user turn entirely and still hit 99.1%. So a stable composed body is cached across every turn that shares it, which is the good news.

Position dominates size. C and D are the same nine bytes. At the front they destroyed 10,624 cached tokens. At the tail they cost nothing. Causal attention means a token's K/V depends on everything before it, so the first byte that differs invalidates the entire remainder.

That gives composition a second requirement beyond any byte budget: order the composed body by stability, most stable first. Anything that varies per session, per date, or per invocation has to land at the end, or it silently converts a 99% hit into a 0% hit for every consumer on every turn.

Worth auditing whether anything variable is currently emitted near the top of a composed body, since the failure mode is invisible: no error, no warning, just a bill and a latency floor that never improve.

Method note: these are synthetic probes I generated on Deep's production route, tagged x-request-id: synthetic-cache-probe-*. They will appear in the Sirens Deep Console alongside organic turns.

Measured this on the live `sirens-echo/deepseek` route through Agent Proxy, using a synthetic 53,133 byte system prompt matching the size #101 reported. Five calls, varying only where the prompt changes: | call | prompt tokens | cached | hit rate | |---|---|---|---| | A, cold | 10,714 | 0 | 0.0% | | B, identical prefix, different user turn | 10,718 | 10,624 | **99.1%** | | C, **9 bytes prepended to the front** | 10,716 | 0 | **0.0%** | | D, **9 bytes appended to the tail** | 10,716 | 10,624 | **99.1%** | | E, identical prefix again | 10,716 | 10,624 | **99.1%** | Two results worth carrying into the design. **Caching is prefix-based, not request-based.** B and E change the user turn entirely and still hit 99.1%. So a stable composed body is cached across every turn that shares it, which is the good news. **Position dominates size.** C and D are the same nine bytes. At the front they destroyed 10,624 cached tokens. At the tail they cost nothing. Causal attention means a token's K/V depends on everything before it, so the first byte that differs invalidates the entire remainder. That gives composition a second requirement beyond any byte budget: **order the composed body by stability, most stable first.** Anything that varies per session, per date, or per invocation has to land at the end, or it silently converts a 99% hit into a 0% hit for every consumer on every turn. Worth auditing whether anything variable is currently emitted near the top of a composed body, since the failure mode is invisible: no error, no warning, just a bill and a latency floor that never improve. Method note: these are synthetic probes I generated on Deep's production route, tagged `x-request-id: synthetic-cache-probe-*`. They will appear in the Sirens Deep Console alongside organic turns.

coilyco-ops commented

2026-08-12 21:12:39 +00:00

Author

Member

Correction. Organic traffic has now been measured and one of the arguments in this issue does not survive it.

Sirens Deep ran ~17 real turns through the instrumented build. Measured:

	value
prompt cache hit rate	99.17%
cached tokens	226,304
missed tokens	1,897, about 112 per turn
system prompt bytes	54,304 avg, matching the 53,133 in #101
tool schemas	17
tool schema bytes	7,029 avg

The cold-start argument was wrong. I claimed the first turn after each gap would pay a full uncached prefix, and that the recurring cost would concentrate there. A full cold start would appear as a single miss of 10,000 to 15,000 tokens. Total misses across the entire window were 1,897. It did not happen. Either the cache survived from earlier traffic or DeepSeek's retention is longer than I assumed, but either way I asserted it more confidently than the evidence supported, and it should be struck from the reasoning here.

Consequence for this issue. At 99.17%, the 53 KB composed body is close to free in spend. The cost case for reducing it is largely gone. What remains is real but narrower:

Context occupancy. The tokens are resident regardless of what they cost. Unchanged by caching, and it still binds on the tower routes where num_ctx is roughly 39k.
Decode bandwidth. Every generated token attends across the full cached prefix. A cache hit skips prefill, not attention.
Attention quality. More resident context is not free accuracy.

Those are context budget and model quality arguments, not spend arguments. This issue should be prioritized accordingly, which is lower than I originally pitched it.

What is unaffected. The two structural findings stand on their own:

The 7x spread across composed role bodies, with creator at 46,942 against a median under 9 KB, and 23 skills including two role melds and a duplicated personality-editorial. That is a design and drift question independent of what caching costs.
The stability-ordering requirement from the front-versus-tail experiment above. Nine bytes at the front cost 10,624 cached tokens. That failure mode is silent and would apply to any consumer, and at a 99% baseline there is now a great deal to lose.

**Correction. Organic traffic has now been measured and one of the arguments in this issue does not survive it.** Sirens Deep ran ~17 real turns through the instrumented build. Measured: | | value | |---|---| | prompt cache hit rate | **99.17%** | | cached tokens | 226,304 | | missed tokens | 1,897, about 112 per turn | | system prompt bytes | 54,304 avg, matching the 53,133 in #101 | | tool schemas | 17 | | tool schema bytes | 7,029 avg | **The cold-start argument was wrong.** I claimed the first turn after each gap would pay a full uncached prefix, and that the recurring cost would concentrate there. A full cold start would appear as a single miss of 10,000 to 15,000 tokens. Total misses across the entire window were 1,897. It did not happen. Either the cache survived from earlier traffic or DeepSeek's retention is longer than I assumed, but either way I asserted it more confidently than the evidence supported, and it should be struck from the reasoning here. **Consequence for this issue.** At 99.17%, the 53 KB composed body is close to free in spend. The cost case for reducing it is largely gone. What remains is real but narrower: - **Context occupancy.** The tokens are resident regardless of what they cost. Unchanged by caching, and it still binds on the tower routes where `num_ctx` is roughly 39k. - **Decode bandwidth.** Every generated token attends across the full cached prefix. A cache hit skips prefill, not attention. - **Attention quality.** More resident context is not free accuracy. Those are context budget and model quality arguments, not spend arguments. This issue should be prioritized accordingly, which is lower than I originally pitched it. **What is unaffected.** The two structural findings stand on their own: 1. The 7x spread across composed role bodies, with `creator` at 46,942 against a median under 9 KB, and 23 skills including two role melds and a duplicated `personality-editorial`. That is a design and drift question independent of what caching costs. 2. The stability-ordering requirement from the front-versus-tail experiment above. Nine bytes at the front cost 10,624 cached tokens. That failure mode is silent and would apply to any consumer, and at a 99% baseline there is now a great deal to lose.

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-12 21:12:54 +00:00

Every turn ships a 53 KB system prompt, byte-identical, uncached #162

coilyco-ops referenced this issue from coilyco-flight-deck/agent-proxy

2026-08-12 21:13:07 +00:00

Every turn ships a 53 KB system prompt, byte-identical, uncached #101

coilyco-ops referenced this issue

2026-08-17 23:33:56 +00:00

The identity card summarizes the doctrine it tells the agent to load, so the load step gets skipped and nothing notices #303