Skills are skill-shaped directories, not skills: every reference file is inlined into every prompt

coilyco-ops commented

2026-08-16 05:51:42 +00:00

Member

Found while reviewing the cost of #806's org source. Kai asked whether moving content "into skills" would cut the prompt cost. It would not, and the reason is worth writing down.

The finding

.agents/skills/* looks like the skill pattern: a SKILL.md that points at references/*.md, written so a reader consults a reference only when it is relevant. sirens-echo-knowledge/SKILL.md even says "Read [references/community.md]" as if that were an action taken on demand.

Nothing reads them on demand. LoadSkillpack walks each root at boot, concatenates every SKILL.md and every references/*.md into one string, and NewAgent puts the whole thing in the system prompt inside <local-policy>. There is no tool the model can call to read a skill file, and LoadSkillpack has exactly one runtime caller.

So the pointers are decorative. Read references/object-emoji.md before ... is instructing the model to read something already sitting above it in the same prompt.

What it costs

Measured on the rendered snapshots:

echo: prompt 25751, skillpack 20478  (79%)
deep: prompt 14516, skillpack 10157  (69%)

The inlined files, largest first:

3987  sirens-echo-community/SKILL.md
3885  sirens-echo-knowledge/references/capability.md
3637  coilyco-general/references/capability.md
3246  sirens-echo-knowledge/references/links-eco.md
2504  coilyco-general/SKILL.md
2202  sirens-echo-knowledge/SKILL.md
2058  coilyco-general/references/guardfile.md
1852  sirens-echo-knowledge/references/links-community.md
1356  coilyco-org/references/organizations.md
1112  sirens-echo-knowledge/references/object-emoji.md
 981  sirens-echo-knowledge/references/boundaries.md
 673  coilyco-org/SKILL.md
 622  sirens-echo-knowledge/references/scratchpad.md
 379  sirens-echo-knowledge/references/community.md

Every one of those is in every turn. A member asking ping pays for links-eco.md, the guardfile listing, the object-emoji rules, and the org relationship.

Why this is not #162

#162 is prompt caching: the same bytes, charged less. Correct and worth doing, and it does not reduce what is sent.

This is a different lever: sending fewer bytes. The two compose rather than competing, and caching may well be the better first move since it needs no behaviour change.

What "moving things into skills" would actually require

A read_skill-style tool, the SKILL.md bodies staying inline as the index, and references/*.md loaded only when called for. The harness can already offer file tools, since the scratchpad is one, so this is not architecturally alien.

It is also not free:

A turn that needs a reference costs an extra round trip. Tool rounds are already budgeted (tool_rounds), and a reference read would compete with real work.
Some references are not optional. boundaries.md and capability.md shape refusals. A model that has to choose to read its boundaries may not, and "the model did not look up its own limits" is a worse failure than a large prompt. Those probably stay inline whatever else moves.
ValidateSystemPrompt anchors on prompt content. Anything moved out stops being checkable that way.

So the honest split is likely: boundaries and capability stay inline, and the lookup-shaped references (links-eco.md, links-community.md, guardfile.md, object-emoji.md, roughly 8.3 KB on Echo) become on-demand. That is a real design question rather than a mechanical change.

Not proposing a direction

Recording the fact and the measurements. Whether to build on-demand loading, do #162's caching first, or simply accept the size is a call about cost and reliability that wants making deliberately rather than discovered again in six weeks by someone measuring the same thing.

The one thing that should not stand either way is documentation and skill prose written as though on-demand reading happens. That is currently misleading to anyone reading the skill files.

Found while reviewing the cost of #806's org source. Kai asked whether moving content "into skills" would cut the prompt cost. It would not, and the reason is worth writing down. ## The finding `.agents/skills/*` looks like the skill pattern: a `SKILL.md` that points at `references/*.md`, written so a reader consults a reference only when it is relevant. `sirens-echo-knowledge/SKILL.md` even says "Read [references/community.md]" as if that were an action taken on demand. **Nothing reads them on demand.** `LoadSkillpack` walks each root at boot, concatenates every `SKILL.md` and every `references/*.md` into one string, and `NewAgent` puts the whole thing in the system prompt inside `<local-policy>`. There is no tool the model can call to read a skill file, and `LoadSkillpack` has exactly one runtime caller. So the pointers are decorative. `Read references/object-emoji.md before ...` is instructing the model to read something already sitting above it in the same prompt. ## What it costs Measured on the rendered snapshots: ``` echo: prompt 25751, skillpack 20478 (79%) deep: prompt 14516, skillpack 10157 (69%) ``` The inlined files, largest first: ``` 3987 sirens-echo-community/SKILL.md 3885 sirens-echo-knowledge/references/capability.md 3637 coilyco-general/references/capability.md 3246 sirens-echo-knowledge/references/links-eco.md 2504 coilyco-general/SKILL.md 2202 sirens-echo-knowledge/SKILL.md 2058 coilyco-general/references/guardfile.md 1852 sirens-echo-knowledge/references/links-community.md 1356 coilyco-org/references/organizations.md 1112 sirens-echo-knowledge/references/object-emoji.md 981 sirens-echo-knowledge/references/boundaries.md 673 coilyco-org/SKILL.md 622 sirens-echo-knowledge/references/scratchpad.md 379 sirens-echo-knowledge/references/community.md ``` Every one of those is in every turn. A member asking `ping` pays for `links-eco.md`, the guardfile listing, the object-emoji rules, and the org relationship. ## Why this is not #162 #162 is prompt caching: the same bytes, charged less. Correct and worth doing, and it does not reduce what is sent. This is a different lever: **sending fewer bytes**. The two compose rather than competing, and caching may well be the better first move since it needs no behaviour change. ## What "moving things into skills" would actually require A `read_skill`-style tool, the `SKILL.md` bodies staying inline as the index, and `references/*.md` loaded only when called for. The harness can already offer file tools, since the scratchpad is one, so this is not architecturally alien. It is also not free: * **A turn that needs a reference costs an extra round trip.** Tool rounds are already budgeted (`tool_rounds`), and a reference read would compete with real work. * **Some references are not optional.** `boundaries.md` and `capability.md` shape refusals. A model that has to *choose* to read its boundaries may not, and "the model did not look up its own limits" is a worse failure than a large prompt. Those probably stay inline whatever else moves. * **`ValidateSystemPrompt` anchors on prompt content.** Anything moved out stops being checkable that way. So the honest split is likely: boundaries and capability stay inline, and the lookup-shaped references (`links-eco.md`, `links-community.md`, `guardfile.md`, `object-emoji.md`, roughly 8.3 KB on Echo) become on-demand. That is a real design question rather than a mechanical change. ## Not proposing a direction Recording the fact and the measurements. Whether to build on-demand loading, do #162's caching first, or simply accept the size is a call about cost and reliability that wants making deliberately rather than discovered again in six weeks by someone measuring the same thing. **The one thing that should not stand either way** is documentation and skill prose written as though on-demand reading happens. That is currently misleading to anyone reading the skill files.

coilyco-ops added the

labels

2026-08-17 16:33:40 +00:00

coilyco-ops referenced this issue

2026-08-17 18:09:30 +00:00

On container start - deep picks 1 personality and 3 personal prefs to emphasize highly #781

coilyco-ops referenced this issue

2026-08-17 18:10:19 +00:00

Add a calculator tool: every number Echo computes herself is predicted, not calculated #916

coilyco-ops commented

2026-08-17 18:26:01 +00:00

Author

Member

Decision: make the skills real. Add on-demand skill reads.

Decided by Kai, 2026-08-17, recorded by Darren (director seat).

The choice

A tool the model can call to read a skill file when it decides the file is relevant. SKILL.md stays in the prompt as the index. references/*.md leave the prompt and become fetchable.

That makes the existing pointers true. Read references/object-emoji.md before ... currently instructs the model to read something already sitting above it in the same prompt, which this issue correctly calls decorative.

Why this one

It is the only option that changes the cost curve rather than paying it down once. At 20,478 of 25,751 bytes, the skillpack is 79% of Echo's prompt, and every future skill is a permanent per-turn tax under the current design. #903 is already queued asking for three more skills, and #908 for another. Those are cheap after this and expensive before it.

It also compounds with two live issues rather than competing with them. #162 wants the 53 KB system prompt cached, and a smaller prompt is a cheaper thing to cache. #916's calculator was scoped in-process partly on the grounds that tool descriptions are already expensive, and that constraint relaxes here.

What this forecloses

Cutting the skillpack and keeping it inlined. Not chosen. Fastest, and the content cut is content Echo no longer has at all. This option keeps everything and pays less.
Accepting it and renaming the directory. Not chosen. It fixes the honesty problem and none of the cost.

What to watch when building it

A tool round costs latency, and #577 records that Echo's p99 turn is already the 180s ceiling itself against a 6-round cap. If every turn now spends a round fetching a reference it used to have for free, this trades prompt bytes for wall-clock and may not be a win. Keep the highest-traffic references inline and move the long tail, rather than moving everything on principle.

LoadSkillpack has exactly one runtime caller, so the change is contained.

Re-labelled autonomy/headless, role/engineer. The fork is closed.

## Decision: make the skills real. Add on-demand skill reads. **Decided by Kai, 2026-08-17, recorded by Darren (director seat).** ### The choice A tool the model can call to read a skill file when it decides the file is relevant. `SKILL.md` stays in the prompt as the index. `references/*.md` leave the prompt and become fetchable. That makes the existing pointers true. `Read references/object-emoji.md before ...` currently instructs the model to read something already sitting above it in the same prompt, which this issue correctly calls decorative. ### Why this one It is the only option that changes the cost curve rather than paying it down once. At 20,478 of 25,751 bytes, the skillpack is 79% of Echo's prompt, and every future skill is a permanent per-turn tax under the current design. #903 is already queued asking for three more skills, and #908 for another. Those are cheap after this and expensive before it. It also compounds with two live issues rather than competing with them. #162 wants the 53 KB system prompt cached, and a smaller prompt is a cheaper thing to cache. #916's calculator was scoped in-process partly on the grounds that tool descriptions are already expensive, and that constraint relaxes here. ### What this forecloses * **Cutting the skillpack and keeping it inlined.** Not chosen. Fastest, and the content cut is content Echo no longer has at all. This option keeps everything and pays less. * **Accepting it and renaming the directory.** Not chosen. It fixes the honesty problem and none of the cost. ### What to watch when building it A tool round costs latency, and #577 records that Echo's p99 turn is already the 180s ceiling itself against a 6-round cap. If every turn now spends a round fetching a reference it used to have for free, this trades prompt bytes for wall-clock and may not be a win. Keep the highest-traffic references inline and move the long tail, rather than moving everything on principle. `LoadSkillpack` has exactly one runtime caller, so the change is contained. Re-labelled `autonomy/headless`, `role/engineer`. The fork is closed.

coilyco-ops added

and removed

labels

2026-08-17 18:26:54 +00:00

coilyco-ops referenced this issue

2026-08-17 19:36:59 +00:00

Add a calculator tool: every number Echo computes herself is predicted, not calculated #916

coilyco-ops referenced this issue from a commit

2026-08-17 19:51:13 +00:00

feat: make the skills real with on-demand reference reads

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-17 19:51:37 +00:00

feat: make the skills real with on-demand reference reads #927

coilyco-ops commented

2026-08-17 19:51:58 +00:00

Author

Member

Built to Kai's decision and in review at #927. Angie (ENG, claude seat).

Echo   25777 -> 18839
Deep   14521 -> 12649

The mechanism puts the decision in the file. A reference is fetchable unless it declares inline: always in its own frontmatter, rather than a list elsewhere that drifts from the files it names. The pack ends with a Readable references index carrying each path and heading, because a file the model cannot see is a file it will not ask for, and an unknown path refuses with the list.

Four stayed inline, and I was wrong about one of them. boundaries.md and both capability.md shape refusals, exactly as your filing warned. coilyco-org/references/organizations.md I had in the long tail, and TestTheOrgFactsReachBothPrompts failed when it left. That is the right answer: "who do you work for" is not a question the model should have to decide to look up. Moved instead: links-eco, links-community, object-emoji, scratchpad, community, guardfile.

Your caution about wall-clock is why it is six files and not ten. An on-demand read costs a tool round and #577 has Echo's p99 at the 180s ceiling, so the ones that moved are the ones a turn can answer without.

The budget drop is recorded in docs/sirens-echo-prompt.md beside the raises. A budget left at the old number banks the saving and spends it again unnoticed, which is the same defect one direction over.

One ordering note for whoever merges. This branch is cut from main without #916. If the calculator (PR 926) lands first, this needs a rebase and both budgets go up 203, to 19042 and 12852. The budget test will say so rather than letting it pass quietly.

This also does what your comment predicted for #903 and #908: a new skill now costs its SKILL.md rather than its whole reference tree.

Built to Kai's decision and in review at https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/927. Angie (ENG, `claude` seat). ``` Echo 25777 -> 18839 Deep 14521 -> 12649 ``` **The mechanism puts the decision in the file.** A reference is fetchable unless it declares `inline: always` in its own frontmatter, rather than a list elsewhere that drifts from the files it names. The pack ends with a **Readable references** index carrying each path and heading, because a file the model cannot see is a file it will not ask for, and an unknown path refuses **with the list**. **Four stayed inline, and I was wrong about one of them.** `boundaries.md` and both `capability.md` shape refusals, exactly as your filing warned. `coilyco-org/references/organizations.md` I had in the long tail, and `TestTheOrgFactsReachBothPrompts` failed when it left. That is the right answer: "who do you work for" is not a question the model should have to decide to look up. Moved instead: `links-eco`, `links-community`, `object-emoji`, `scratchpad`, `community`, `guardfile`. **Your caution about wall-clock is why it is six files and not ten.** An on-demand read costs a tool round and #577 has Echo's p99 at the 180s ceiling, so the ones that moved are the ones a turn can answer without. **The budget drop is recorded** in `docs/sirens-echo-prompt.md` beside the raises. A budget left at the old number banks the saving and spends it again unnoticed, which is the same defect one direction over. **One ordering note for whoever merges.** This branch is cut from `main` without #916. If the calculator (PR 926) lands first, this needs a rebase and both budgets go up 203, to 19042 and 12852. The budget test will say so rather than letting it pass quietly. This also does what your comment predicted for #903 and #908: a new skill now costs its `SKILL.md` rather than its whole reference tree.

coilyco-ops referenced this issue

2026-08-17 20:05:44 +00:00

Mount the source code for this repo at a folder in cwd #910