Sparse-mount substrate reference repos to their declared reference surface #523

New issue

Closed

opened 2026-07-02 08:39:20 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-07-02 08:39:20 +00:00

Owner

Idea

Make substrate reference repos hydrate as a sparse checkout of their declared reference surface instead of the current full working clone (warmSubstrateRepo, cmd/ward/container_bootstrap.go:693 does git clone --quiet <mirror> /substrate/<name> — a full tree today). The goal is stronger information hygiene: force each repo's reference surface to actually carry what a cross-repo reader needs, instead of letting agents spelunk source to route around thin docs. Low-context agents benefit most from a small curated surface — and are hurt most when docs are thin, so the change is only net-good if the doc-quality forcing function actually fires.

Scope boundary (keep this tight)

This is substrate only — the read-only reference clones under /substrate/<name>. The /workspace target clone and any --repo granted clones keep full trees (you cannot implement against a docs-only tree). The target-overlap case is unaffected: when the target is also on the manifest, /workspace/<name> stays full and authoritative and /substrate/<name> was always the read-only shadow.

Design questions for the advisor to resolve

Derive the sparse pattern from the trifecta, not hand-picked paths. Every repo already must maintain README / AGENTS / docs/FEATURES.md plus docs/ and .agents/skills/ — that is the public reference surface. Mounting exactly that fuses "keep your reference surface current" (already mandatory) with "what substrate exposes" into one obligation, so the forcing function becomes teeth on an existing rule rather than a new axis of config. Confirm the exact pattern (top-level files + docs/ + .agents/skills/ + trifecta) and whether it's derived or declared per-repo.
The sparse cut keys off the RUN ROLE, not the repo. This is the settled axis. advisor runs get a full substrate clone — their entire purpose is to gather info, so ground truth beats a curated surface for them, and they are well-placed to detect doc gaps and file them. engineer and default runs get the sparse reference surface — they are implementing, which is where the small surface and the forcing function pay off, and engineers own the docs so they are the right ones to feel a gap. So the profile is not per-repo (full/reference by repo identity — dropped) and not a --substrate-full break-glass (dropped). Hydration branches on the run's role, known at container launch. warmSubstrateRepo reads the role and picks full-tree vs trifecta-sparse. The manifest line stays owner/name [tier] — no new per-repo field.

Corollary that keeps --repo as the only escape for engineers. An engineer run that genuinely needs B's full source uses the existing --repo owner/name (full feature working copy in /workspace at launch) — not a substrate knob. Advisors rarely need it because they already get full substrate. This split also strengthens the forcing function rather than weakening it: the doc gap bites the engineer (who writes docs), while the advisor (who gathers info) is the one positioned to notice and file it.
Why the exception is role-keyed and not repo-keyed (incl. the dev CLIs). The tempting carve-out was "dev CLIs need source." It proves too much — all behavior lives in code, so that would exempt everything. The real split is why you read a substrate repo: (a) honor a contract the repo publishes → the reference surface is the contract, a gap is a doc bug, not a reason to read source; (b) operate the repo as a tool → runtime introspection (--help / describe / --json) is the right reference, and the checkout may not even match the installed binary, so the tree is the wrong reference; (c) reason about B's implementation as the task subject → that's a dispatch-into-B run, or a --repo B pull if you need B alongside your current repo in one session. All three are served for engineers without a repo-keyed substrate full-mount — via runtime introspection, dispatch-into-B, or --repo. The property that distinguishes repos here — "has a runtime introspection surface in-container" (ward does via --help/describe; cli-guard is not on PATH and does not) — is a real distinction, but the remedy it once pointed at (a per-repo standing full mount) is redundant. cli-guard-needs-reading is a --repo case for an engineer, and a non-issue for an advisor (who already has full substrate). So the only standing full is the role one in question 2, never a repo one.
Filable gaps keep the forcing function honest. Teach that a substrate reference gap an agent hits is itself a filable work item against that repo's docs/FEATURES, so the pain accretes into doc debt (paid via capture-and-dispatch) instead of dissipating into quiet spelunking or reflexive --repo pulls. This is what makes the sparse cut improve docs over time rather than just relocating the workaround.
The --repo-as-read nuance (secondary, likely a follow-up). --repo today is a work grant: the reaper verifies each granted repo landed a push and treats a non-landed grant as a hard failure (salvage branch + reopened issue). So using --repo purely to read B collides with that landing check. The pure-read case is rare, so this is probably accept-the-friction, but the advisor should note whether a lighter read-only grant variant is worth a separate follow-up issue. Do not fold it into this design.
Honest cost note. Sparse-checkout on a full mirror clone still fetches all objects — the win is attention/scan surface (smaller tree to grep and wander), not network or gitcache bandwidth. Say so plainly in the design so it isn't oversold as an infra optimization.

Deliverable

A concrete design proposal: the trifecta-derived sparse-pattern, confirmation that substrate is uniformly reference with --repo (not a substrate knob) as the full-source escape, the doctrine edit that makes substrate reference gaps filable, and a note on the --repo-as-read reaper nuance as a possible follow-up. Implementation touches warmSubstrateRepo in cmd/ward/container_bootstrap.go.

## Idea Make substrate reference repos hydrate as a **sparse checkout of their declared reference surface** instead of the current full working clone (`warmSubstrateRepo`, `cmd/ward/container_bootstrap.go:693` does `git clone --quiet <mirror> /substrate/<name>` — a full tree today). The goal is stronger information hygiene: force each repo's reference surface to actually carry what a cross-repo reader needs, instead of letting agents spelunk source to route around thin docs. Low-context agents benefit most from a small curated surface — and are hurt most when docs are thin, so the change is only net-good if the doc-quality forcing function actually fires. ## Scope boundary (keep this tight) This is **substrate only** — the read-only reference clones under `/substrate/<name>`. The `/workspace` target clone and any `--repo` granted clones keep full trees (you cannot implement against a docs-only tree). The target-overlap case is unaffected: when the target is also on the manifest, `/workspace/<name>` stays full and authoritative and `/substrate/<name>` was always the read-only shadow. ## Design questions for the advisor to resolve 1. **Derive the sparse pattern from the trifecta, not hand-picked paths.** Every repo already must maintain README / AGENTS / `docs/FEATURES.md` plus `docs/` and `.agents/skills/` — that *is* the public reference surface. Mounting exactly that fuses "keep your reference surface current" (already mandatory) with "what substrate exposes" into one obligation, so the forcing function becomes teeth on an existing rule rather than a new axis of config. Confirm the exact pattern (top-level files + `docs/` + `.agents/skills/` + trifecta) and whether it's derived or declared per-repo. 2. **The sparse cut keys off the RUN ROLE, not the repo.** This is the settled axis. `advisor` runs get a **full** substrate clone — their entire purpose is to gather info, so ground truth beats a curated surface for them, and they are well-placed to *detect* doc gaps and file them. `engineer` and default runs get the **sparse reference** surface — they are implementing, which is where the small surface and the forcing function pay off, and engineers own the docs so they are the right ones to *feel* a gap. So the profile is not per-repo (`full`/`reference` by repo identity — dropped) and not a `--substrate-full` break-glass (dropped). Hydration branches on the run's role, known at container launch. `warmSubstrateRepo` reads the role and picks full-tree vs trifecta-sparse. The manifest line stays `owner/name [tier]` — no new per-repo field. **Corollary that keeps `--repo` as the only escape for engineers.** An engineer run that genuinely needs B's full source uses the existing `--repo owner/name` (full feature working copy in `/workspace` at launch) — not a substrate knob. Advisors rarely need it because they already get full substrate. This split also strengthens the forcing function rather than weakening it: the doc gap bites the engineer (who writes docs), while the advisor (who gathers info) is the one positioned to notice and file it. 3. **Why the exception is role-keyed and not repo-keyed (incl. the dev CLIs).** The tempting carve-out was "dev CLIs need source." It proves too much — all behavior lives in code, so that would exempt everything. The real split is *why* you read a substrate repo: (a) honor a contract the repo *publishes* → the reference surface is the contract, a gap is a doc bug, not a reason to read source; (b) *operate* the repo as a tool → runtime introspection (`--help` / `describe` / `--json`) is the right reference, and the checkout may not even match the installed binary, so the tree is the *wrong* reference; (c) *reason about B's implementation as the task subject* → that's a **dispatch-into-B** run, or a `--repo B` pull if you need B alongside your current repo in one session. All three are served for engineers without a *repo-keyed* substrate full-mount — via runtime introspection, dispatch-into-B, or `--repo`. The property that *distinguishes* repos here — "has a runtime introspection surface in-container" (ward does via `--help`/`describe`; cli-guard is not on PATH and does not) — is a real distinction, but the remedy it once pointed at (a per-repo standing full mount) is redundant. cli-guard-needs-reading is a `--repo` case for an engineer, and a non-issue for an advisor (who already has full substrate). So the only standing full is the **role** one in question 2, never a repo one. 4. **Filable gaps keep the forcing function honest.** Teach that a substrate reference gap an agent hits is *itself a filable work item* against that repo's docs/FEATURES, so the pain accretes into doc debt (paid via capture-and-dispatch) instead of dissipating into quiet spelunking or reflexive `--repo` pulls. This is what makes the sparse cut improve docs over time rather than just relocating the workaround. 5. **The `--repo`-as-read nuance (secondary, likely a follow-up).** `--repo` today is a *work grant*: the reaper verifies each granted repo *landed a push* and treats a non-landed grant as a hard failure (salvage branch + reopened issue). So using `--repo` purely to *read* B collides with that landing check. The pure-read case is rare, so this is probably accept-the-friction, but the advisor should note whether a lighter read-only grant variant is worth a separate follow-up issue. Do not fold it into this design. 6. **Honest cost note.** Sparse-checkout on a full mirror clone still fetches all objects — the win is *attention/scan surface* (smaller tree to grep and wander), not network or gitcache bandwidth. Say so plainly in the design so it isn't oversold as an infra optimization. ## Deliverable A concrete design proposal: the trifecta-derived sparse-pattern, confirmation that substrate is uniformly `reference` with `--repo` (not a substrate knob) as the full-source escape, the doctrine edit that makes substrate reference gaps filable, and a note on the `--repo`-as-read reaper nuance as a possible follow-up. Implementation touches `warmSubstrateRepo` in `cmd/ward/container_bootstrap.go`.

coilyco-ops commented

2026-07-02 08:41:52 +00:00

Member

🔎 ward agent advisor

ward agent advisor --driver claude ran a one-shot standard research pass on this question:

Design the substrate sparse-mount proposal per the issue. Land your design as a comment on ward#523. Key framing to honor: (1) derive the sparse pattern from the trifecta reference surface, not hand-picked paths; (2) the standing 'full' exception criterion is 'no runtime introspection surface in-container' (ward has one via --help/describe, cli-guard does not), NOT the category 'dev CLIs' - full must be earned with written justification and empty at launch; (3) residual source-read need is cross-repo debugging, served by a per-run --substrate-full break-glass, not a standing profile; (4) a substrate reference gap must be a filable work item so the forcing function accretes into doc debt; (5) be honest that sparse-checkout saves attention/scan surface, not gitcache bandwidth. Implementation touches warmSubstrateRepo in cmd/ward/container_bootstrap.go.

I have enough grounding. This is single-repo work (all in ward), so the full design goes in summary with an empty issues array.

{
  "summary": "## Design: sparse-mount substrate reference repos\n\nConfirmed against the code: `warmSubstrateRepo` (`cmd/ward/container_bootstrap.go:693`) does a full `git clone --quiet <mirror> <work>` out of the shared gitcache bare mirror. The manifest is `cmd/ward/containerassets/preclone-repos.txt`, parsed as `owner/name [tier]` in `warmSubstrate` (line ~731), and the same list is documented in `docs/container-substrate.md`. Nothing else changes in the warm path. Here is the concrete design answering all five questions.\n\n---\n\n### Q1 — Derive the sparse pattern from the trifecta, do not declare it per-repo\n\n**The pattern is derived, identical for every `reference` repo, and expressed as git cone-mode sparse patterns.** No per-repo path lists.\n\nThe reference surface every repo already must maintain is: top-level files (README, AGENTS.md), plus `docs/` (which carries `docs/FEATURES.md`), plus `.agents/skills/`. Git's **cone mode** produces exactly this set from two directory patterns:\n\n```\ngit -C <work> sparse-checkout set --cone docs .agents/skills\n```\n\nCone mode always materializes every file in the repo root and in each parent directory along an included path, then the full subtree of each named directory. So `docs .agents/skills` yields: all root files (README + AGENTS + any top-level md) + `docs/**` (FEATURES lives here) + `.agents/skills/**`. That **is** the published reference surface, and it is the same two-token pattern for coilyco-flight-deck/agentic-os as for coilyco-bridge/lore. Derived, not declared.\n\nThe forcing function is a property of cone mode, not extra code: **a repo with a thin `docs/` gets a thin substrate mount.** If a cross-repo contract lives only in source, it simply is not present under `/substrate/<name>`, and the fix is to write it into the reference surface (see Q4), not to widen the mount. No trifecta *validation* is added here (that already lives in the `documentation-layout` / `catalog-trifecta` pre-commit hooks upstream in agentic-os) — this change just makes substrate *expose* only what those hooks already require.\n\nEdge cases the pattern handles for free:\n- A repo with no `docs/` or no `.agents/skills/`: cone `set` on a nonexistent dir is a no-op, root files (README/AGENTS) still land. Thin repo, thin mount, correct.\n- No hand-maintained allowlist to drift as repos add/remove doc dirs.\n\n### Q2 — Manifest schema\n\nExtend the line to `owner/name [tier] [profile]`:\n\n```\ncoilyco-flight-deck/agentic-os        image   reference\ncoilyco-flight-deck/cli-guard         image   full        # no runtime introspection surface — see Q3\ncoilyco-flight-deck/ward              image   reference\n```\n\n- **Third field, optional, default `reference`.** Backward compatible: existing two-field lines parse to `reference`, which is the intended default for all.\n- `tier` (field 2) is unchanged and orthogonal — it controls the *seed source* (image vs cache), never the checkout shape. Keep the two axes independent in the parser.\n- Parse in `warmSubstrate`: `profile := \"reference\"; if len(fields) > 2 { profile = fields[2] }`, threaded into `warmSubstrateRepo(ctx, e, owner, name, tier, profile)`.\n- Reject unknown profile tokens with a `blog` warning and fall back to `reference` (fail toward the smaller surface, best-effort like the rest of warming).\n- Update the header comment in `preclone-repos.txt` and `docs/container-substrate.md` to document the third field and the default.\n\n### Q3 — The standing `full` exception: criterion is \"no runtime introspection surface in-container\"\n\n**`full` is not a category (\"dev CLIs\") — it is earned by a property of the repo.** The three reasons an agent reads a substrate repo:\n\n- **(a) honor a published contract** → docs are the right reference. Forcing function is correct. → `reference`.\n- **(b) operate the repo as a tool** → the *runtime* surface (`--help`, `describe`, `--json`) is the right reference, and the in-container checkout may not even match the installed binary, so the tree is the **wrong** reference. → `reference` (read the running tool, not stale source).\n- **(c) reason about the repo's implementation as the task subject** → this is a **dispatch-into-B** task per doctrine, not a substrate read. Not served here at all.\n\nSo the residual standing need for source is narrow: **a repo that has no runtime introspection surface reachable inside the container.** That is the whole test.\n\n- **ward** is installed and self-describing (`ward --help`, `ward ops <area> describe`, `--json`). It **fails** the test → `reference`. Its own docs + running binary are the reference.\n- **cli-guard** is the engine under ward, not on PATH in-container, so the checkout is the *only* window into its behavior. It **passes** the test → the one honest standing-`full` candidate.\n\n**`full` ships empty at launch** except for the cli-guard candidate, and every `full` line carries an inline written justification naming *why the repo has no runtime surface* (as in the schema example above). This keeps `full` from becoming the quiet default escape from writing docs. A repo does not get `full` because reading its source is convenient — only because there is no running surface to read instead. If someone later puts cli-guard behavior behind a `cli-guard describe` or exposes it via `ward`, cli-guard drops back to `reference`.\n\n### Q4 — Break-glass + filable gaps\n\n**Per-run break-glass, not a standing profile**, for the genuine residual: cross-repo debugging where the cause is in B's source, the failure manifests in A's state, and B has no runtime surface exposing it.\n\n- **Flag:** `--substrate-full owner/name`, repeatable, on the `ward container` / `ward agent` surface. It resolves into `WARD_SUBSTRATE_FULL` (comma-separated slug list) passed through to the bootstrap env, alongside the existing `WARD_SUBSTRATE_*` vars in `bootstrapEnv`.\n- **Env:** `WARD_SUBSTRATE_FULL=owner/a,owner/b` for direct/headless use. Accept a bare `WARD_SUBSTRATE_FULL=1` (or `all`) as the blanket \"give me every substrate repo full for this debugging session\" hatch, since a deep cross-repo debug may not know in advance which B holds the cause.\n- **Semantics:** in `warmSubstrateRepo`, a repo whose slug is in the break-glass set (or the blanket flag) checks out full regardless of its manifest profile. It is per-run and leaves no residue: nothing is written back to the manifest, so the next run reverts to `reference`. This is the key property — the escape hatch cannot silently become the standing config.\n- **Log it:** `blog(\"substrate: %s/%s full via break-glass (--substrate-full)\", owner, name)` so a run's transcript shows the surface was widened deliberately.\n\n**Filable gaps — the accretion mechanism.** Teach (in `AGENTS.container.md` and `docs/container-substrate.md`) that **a substrate reference gap is itself a filable work item against the offending repo's `docs/`/`FEATURES.md`.** When an agent hits a wall because the substrate mount is missing something a cross-repo reader legitimately needs, the correct move is capture-and-dispatch: file an issue against repo B's tracker (\"reference surface missing X that cross-repo reader needed\"), optionally break-glass to unblock the current run, and keep going. Pain accretes into visible doc debt paid down deliberately, instead of dissipating into quiet source-spelunking that leaves the thin docs thin forever. This closes the loop that makes the forcing function real: the smaller surface only improves docs if the friction it creates is *routed* to a doc fix rather than absorbed silently.\n\n### Q5 — Honest cost note\n\n**Sparse-checkout on a full bare mirror still fetches every object.** The gitcache mirror is `git clone --mirror` and the working copy is a local clone out of it — sparse only limits which blobs get *written into the working tree*, not which objects the mirror holds or transfers. So there is **no network win and no gitcache-bandwidth win.** The win is **attention/scan surface**: a smaller tree for the agent to `grep`, `ls`, and wander, so a low-context agent lands on the curated reference surface instead of spelunking source to route around thin docs. This must be stated plainly in `docs/container-substrate.md` so the change is not oversold as an infra optimization. (A future partial-clone / blobless-mirror change could add a real bandwidth win, but that is a separate axis and out of scope here.)\n\n---\n\n### Implementation sketch (`warmSubstrateRepo`, `cmd/ward/container_bootstrap.go`)\n\nOnly the working-clone tail changes; the mirror hydrate/refresh under flock is untouched.\n\n```go\nfunc (r *Runner) warmSubstrateRepo(ctx context.Context, e bootstrapEnv, owner, name, tier, profile string) {\n    // ... unchanged: mirror hydrate + TTL refresh under flock ...\n    if isDir(mirror) {\n        work := filepath.Join(e.SubstrateDest, name)\n        _ = os.RemoveAll(work)\n        full := profile == \"full\" || e.substrateFull(owner, name) // manifest OR break-glass\n        if full {\n            if cerr := r.Runner.Exec(ctx, \"git\", \"clone\", \"--quiet\", mirror, work); cerr != nil {\n                blog(\"substrate: working clone failed %s/%s\", owner, name)\n                return\n            }\n        } else {\n            if cerr := r.Runner.Exec(ctx, \"git\", \"clone\", \"--quiet\", \"--no-checkout\", mirror, work); cerr != nil {\n                blog(\"substrate: sparse clone failed %s/%s\", owner, name)\n                return\n            }\n            // Cone-mode reference surface: root files + docs/** + .agents/skills/**.\n            if serr := r.Runner.Exec(ctx, \"git\", \"-C\", work, \"sparse-checkout\", \"set\", \"--cone\", \"docs\", \".agents/skills\"); serr != nil {\n                blog(\"substrate: sparse-checkout set failed %s/%s (falling back to full)\", owner, name)\n            }\n            if cerr := r.Runner.Exec(ctx, \"git\", \"-C\", work, \"checkout\"); cerr != nil {\n                blog(\"substrate: sparse checkout failed %s/%s\", owner, name)\n            }\n        }\n        _ = r.Runner.Exec(ctx, \"git\", \"-C\", work, \"remote\", \"set-url\", \"origin\", url)\n    }\n}\n```\n\nNotes:\n- `e.substrateFull(owner, name)` is a helper over the parsed `WARD_SUBSTRATE_FULL` set (blanket `1`/`all` → always true). Parsed once alongside `ExtraRepos`.\n- Cone mode needs git ≥ 2.36 for `sparse-checkout set --cone`. The dev-base image is well past that; note the floor in the doc.\n- Keep it **best-effort**: any sparse step failing logs and continues, consistent with the rest of warming. A sane fallback is: on `sparse-checkout` failure, the tree stays as the `--no-checkout` clone left it (empty working tree but valid git dir) — acceptable, or optionally re-run a plain checkout to degrade to full. Pick the degrade-to-full fallback so a git-version surprise never yields an empty mount.\n- **Target overlap unchanged:** the target is skipped in the `warmSubstrate` loop, so `/workspace/<name>` stays full and authoritative and only the read-only `/substrate/<name>` shadow is sparse. `--repo` grants clone through their own path, also unaffected.\n\n### Doctrine / doc edits (same change)\n\n- `cmd/ward/containerassets/preclone-repos.txt` — header comment documents the third `profile` field, default `reference`, the \"no runtime introspection surface\" bar for `full`, and the inline-justification requirement. Set cli-guard to `full` with its justification.\n- `docs/container-substrate.md` — new section on profiles + the honest cost note (attention surface, not bandwidth) + the cone-mode pattern derivation.\n- `cmd/ward/containerassets/AGENTS.container.md` — the agent-facing rule: substrate is the reference *surface*, operate tools via their runtime surface not the checkout, and a reference gap is a **filable work item** (capture-and-dispatch against repo B), with `--substrate-full` as the per-run unblock.\n- `docs/FEATURES.md` — update the substrate entry to mention reference/full profiles and break-glass (trifecta upkeep per house rules).\n- `docs/agent-flags.md` (and wherever container flags are enumerated) — document `--substrate-full`.\n\n### Open question for the implementer\n\nThe `.agents/skills` cone pattern also pulls any loose files sitting directly in `.agents/` (cone includes parent-dir files). That is negligible and arguably desirable (an `.agents/README` is reference-y), but if a repo parks something large and non-reference at `.agents/` root it would ride along. Not worth special-casing now — flag only if it bites."
}

Researched and posted automatically by ward agent advisor --driver claude (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Claude (she/her), via ward agent

### 🔎 ward agent advisor `ward agent advisor --driver claude` ran a one-shot **standard** research pass on this question: > Design the substrate sparse-mount proposal per the issue. Land your design as a comment on ward#523. Key framing to honor: (1) derive the sparse pattern from the trifecta reference surface, not hand-picked paths; (2) the standing 'full' exception criterion is 'no runtime introspection surface in-container' (ward has one via --help/describe, cli-guard does not), NOT the category 'dev CLIs' - full must be earned with written justification and empty at launch; (3) residual source-read need is cross-repo debugging, served by a per-run --substrate-full break-glass, not a standing profile; (4) a substrate reference gap must be a filable work item so the forcing function accretes into doc debt; (5) be honest that sparse-checkout saves attention/scan surface, not gitcache bandwidth. Implementation touches warmSubstrateRepo in cmd/ward/container_bootstrap.go. --- I have enough grounding. This is single-repo work (all in ward), so the full design goes in `summary` with an empty `issues` array. ```json { "summary": "## Design: sparse-mount substrate reference repos\n\nConfirmed against the code: `warmSubstrateRepo` (`cmd/ward/container_bootstrap.go:693`) does a full `git clone --quiet <mirror> <work>` out of the shared gitcache bare mirror. The manifest is `cmd/ward/containerassets/preclone-repos.txt`, parsed as `owner/name [tier]` in `warmSubstrate` (line ~731), and the same list is documented in `docs/container-substrate.md`. Nothing else changes in the warm path. Here is the concrete design answering all five questions.\n\n---\n\n### Q1 — Derive the sparse pattern from the trifecta, do not declare it per-repo\n\n**The pattern is derived, identical for every `reference` repo, and expressed as git cone-mode sparse patterns.** No per-repo path lists.\n\nThe reference surface every repo already must maintain is: top-level files (README, AGENTS.md), plus `docs/` (which carries `docs/FEATURES.md`), plus `.agents/skills/`. Git's **cone mode** produces exactly this set from two directory patterns:\n\n```\ngit -C <work> sparse-checkout set --cone docs .agents/skills\n```\n\nCone mode always materializes every file in the repo root and in each parent directory along an included path, then the full subtree of each named directory. So `docs .agents/skills` yields: all root files (README + AGENTS + any top-level md) + `docs/**` (FEATURES lives here) + `.agents/skills/**`. That **is** the published reference surface, and it is the same two-token pattern for coilyco-flight-deck/agentic-os as for coilyco-bridge/lore. Derived, not declared.\n\nThe forcing function is a property of cone mode, not extra code: **a repo with a thin `docs/` gets a thin substrate mount.** If a cross-repo contract lives only in source, it simply is not present under `/substrate/<name>`, and the fix is to write it into the reference surface (see Q4), not to widen the mount. No trifecta *validation* is added here (that already lives in the `documentation-layout` / `catalog-trifecta` pre-commit hooks upstream in agentic-os) — this change just makes substrate *expose* only what those hooks already require.\n\nEdge cases the pattern handles for free:\n- A repo with no `docs/` or no `.agents/skills/`: cone `set` on a nonexistent dir is a no-op, root files (README/AGENTS) still land. Thin repo, thin mount, correct.\n- No hand-maintained allowlist to drift as repos add/remove doc dirs.\n\n### Q2 — Manifest schema\n\nExtend the line to `owner/name [tier] [profile]`:\n\n```\ncoilyco-flight-deck/agentic-os image reference\ncoilyco-flight-deck/cli-guard image full # no runtime introspection surface — see Q3\ncoilyco-flight-deck/ward image reference\n```\n\n- **Third field, optional, default `reference`.** Backward compatible: existing two-field lines parse to `reference`, which is the intended default for all.\n- `tier` (field 2) is unchanged and orthogonal — it controls the *seed source* (image vs cache), never the checkout shape. Keep the two axes independent in the parser.\n- Parse in `warmSubstrate`: `profile := \"reference\"; if len(fields) > 2 { profile = fields[2] }`, threaded into `warmSubstrateRepo(ctx, e, owner, name, tier, profile)`.\n- Reject unknown profile tokens with a `blog` warning and fall back to `reference` (fail toward the smaller surface, best-effort like the rest of warming).\n- Update the header comment in `preclone-repos.txt` and `docs/container-substrate.md` to document the third field and the default.\n\n### Q3 — The standing `full` exception: criterion is \"no runtime introspection surface in-container\"\n\n**`full` is not a category (\"dev CLIs\") — it is earned by a property of the repo.** The three reasons an agent reads a substrate repo:\n\n- **(a) honor a published contract** → docs are the right reference. Forcing function is correct. → `reference`.\n- **(b) operate the repo as a tool** → the *runtime* surface (`--help`, `describe`, `--json`) is the right reference, and the in-container checkout may not even match the installed binary, so the tree is the **wrong** reference. → `reference` (read the running tool, not stale source).\n- **(c) reason about the repo's implementation as the task subject** → this is a **dispatch-into-B** task per doctrine, not a substrate read. Not served here at all.\n\nSo the residual standing need for source is narrow: **a repo that has no runtime introspection surface reachable inside the container.** That is the whole test.\n\n- **ward** is installed and self-describing (`ward --help`, `ward ops <area> describe`, `--json`). It **fails** the test → `reference`. Its own docs + running binary are the reference.\n- **cli-guard** is the engine under ward, not on PATH in-container, so the checkout is the *only* window into its behavior. It **passes** the test → the one honest standing-`full` candidate.\n\n**`full` ships empty at launch** except for the cli-guard candidate, and every `full` line carries an inline written justification naming *why the repo has no runtime surface* (as in the schema example above). This keeps `full` from becoming the quiet default escape from writing docs. A repo does not get `full` because reading its source is convenient — only because there is no running surface to read instead. If someone later puts cli-guard behavior behind a `cli-guard describe` or exposes it via `ward`, cli-guard drops back to `reference`.\n\n### Q4 — Break-glass + filable gaps\n\n**Per-run break-glass, not a standing profile**, for the genuine residual: cross-repo debugging where the cause is in B's source, the failure manifests in A's state, and B has no runtime surface exposing it.\n\n- **Flag:** `--substrate-full owner/name`, repeatable, on the `ward container` / `ward agent` surface. It resolves into `WARD_SUBSTRATE_FULL` (comma-separated slug list) passed through to the bootstrap env, alongside the existing `WARD_SUBSTRATE_*` vars in `bootstrapEnv`.\n- **Env:** `WARD_SUBSTRATE_FULL=owner/a,owner/b` for direct/headless use. Accept a bare `WARD_SUBSTRATE_FULL=1` (or `all`) as the blanket \"give me every substrate repo full for this debugging session\" hatch, since a deep cross-repo debug may not know in advance which B holds the cause.\n- **Semantics:** in `warmSubstrateRepo`, a repo whose slug is in the break-glass set (or the blanket flag) checks out full regardless of its manifest profile. It is per-run and leaves no residue: nothing is written back to the manifest, so the next run reverts to `reference`. This is the key property — the escape hatch cannot silently become the standing config.\n- **Log it:** `blog(\"substrate: %s/%s full via break-glass (--substrate-full)\", owner, name)` so a run's transcript shows the surface was widened deliberately.\n\n**Filable gaps — the accretion mechanism.** Teach (in `AGENTS.container.md` and `docs/container-substrate.md`) that **a substrate reference gap is itself a filable work item against the offending repo's `docs/`/`FEATURES.md`.** When an agent hits a wall because the substrate mount is missing something a cross-repo reader legitimately needs, the correct move is capture-and-dispatch: file an issue against repo B's tracker (\"reference surface missing X that cross-repo reader needed\"), optionally break-glass to unblock the current run, and keep going. Pain accretes into visible doc debt paid down deliberately, instead of dissipating into quiet source-spelunking that leaves the thin docs thin forever. This closes the loop that makes the forcing function real: the smaller surface only improves docs if the friction it creates is *routed* to a doc fix rather than absorbed silently.\n\n### Q5 — Honest cost note\n\n**Sparse-checkout on a full bare mirror still fetches every object.** The gitcache mirror is `git clone --mirror` and the working copy is a local clone out of it — sparse only limits which blobs get *written into the working tree*, not which objects the mirror holds or transfers. So there is **no network win and no gitcache-bandwidth win.** The win is **attention/scan surface**: a smaller tree for the agent to `grep`, `ls`, and wander, so a low-context agent lands on the curated reference surface instead of spelunking source to route around thin docs. This must be stated plainly in `docs/container-substrate.md` so the change is not oversold as an infra optimization. (A future partial-clone / blobless-mirror change could add a real bandwidth win, but that is a separate axis and out of scope here.)\n\n---\n\n### Implementation sketch (`warmSubstrateRepo`, `cmd/ward/container_bootstrap.go`)\n\nOnly the working-clone tail changes; the mirror hydrate/refresh under flock is untouched.\n\n```go\nfunc (r *Runner) warmSubstrateRepo(ctx context.Context, e bootstrapEnv, owner, name, tier, profile string) {\n // ... unchanged: mirror hydrate + TTL refresh under flock ...\n if isDir(mirror) {\n work := filepath.Join(e.SubstrateDest, name)\n _ = os.RemoveAll(work)\n full := profile == \"full\" || e.substrateFull(owner, name) // manifest OR break-glass\n if full {\n if cerr := r.Runner.Exec(ctx, \"git\", \"clone\", \"--quiet\", mirror, work); cerr != nil {\n blog(\"substrate: working clone failed %s/%s\", owner, name)\n return\n }\n } else {\n if cerr := r.Runner.Exec(ctx, \"git\", \"clone\", \"--quiet\", \"--no-checkout\", mirror, work); cerr != nil {\n blog(\"substrate: sparse clone failed %s/%s\", owner, name)\n return\n }\n // Cone-mode reference surface: root files + docs/** + .agents/skills/**.\n if serr := r.Runner.Exec(ctx, \"git\", \"-C\", work, \"sparse-checkout\", \"set\", \"--cone\", \"docs\", \".agents/skills\"); serr != nil {\n blog(\"substrate: sparse-checkout set failed %s/%s (falling back to full)\", owner, name)\n }\n if cerr := r.Runner.Exec(ctx, \"git\", \"-C\", work, \"checkout\"); cerr != nil {\n blog(\"substrate: sparse checkout failed %s/%s\", owner, name)\n }\n }\n _ = r.Runner.Exec(ctx, \"git\", \"-C\", work, \"remote\", \"set-url\", \"origin\", url)\n }\n}\n```\n\nNotes:\n- `e.substrateFull(owner, name)` is a helper over the parsed `WARD_SUBSTRATE_FULL` set (blanket `1`/`all` → always true). Parsed once alongside `ExtraRepos`.\n- Cone mode needs git ≥ 2.36 for `sparse-checkout set --cone`. The dev-base image is well past that; note the floor in the doc.\n- Keep it **best-effort**: any sparse step failing logs and continues, consistent with the rest of warming. A sane fallback is: on `sparse-checkout` failure, the tree stays as the `--no-checkout` clone left it (empty working tree but valid git dir) — acceptable, or optionally re-run a plain checkout to degrade to full. Pick the degrade-to-full fallback so a git-version surprise never yields an empty mount.\n- **Target overlap unchanged:** the target is skipped in the `warmSubstrate` loop, so `/workspace/<name>` stays full and authoritative and only the read-only `/substrate/<name>` shadow is sparse. `--repo` grants clone through their own path, also unaffected.\n\n### Doctrine / doc edits (same change)\n\n- `cmd/ward/containerassets/preclone-repos.txt` — header comment documents the third `profile` field, default `reference`, the \"no runtime introspection surface\" bar for `full`, and the inline-justification requirement. Set cli-guard to `full` with its justification.\n- `docs/container-substrate.md` — new section on profiles + the honest cost note (attention surface, not bandwidth) + the cone-mode pattern derivation.\n- `cmd/ward/containerassets/AGENTS.container.md` — the agent-facing rule: substrate is the reference *surface*, operate tools via their runtime surface not the checkout, and a reference gap is a **filable work item** (capture-and-dispatch against repo B), with `--substrate-full` as the per-run unblock.\n- `docs/FEATURES.md` — update the substrate entry to mention reference/full profiles and break-glass (trifecta upkeep per house rules).\n- `docs/agent-flags.md` (and wherever container flags are enumerated) — document `--substrate-full`.\n\n### Open question for the implementer\n\nThe `.agents/skills` cone pattern also pulls any loose files sitting directly in `.agents/` (cone includes parent-dir files). That is negligible and arguably desirable (an `.agents/README` is reference-y), but if a repo parks something large and non-reference at `.agents/` root it would ride along. Not worth special-casing now — flag only if it bites." } ``` --- Researched and posted automatically by `ward agent advisor --driver claude` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-02 08:43:59 +00:00

Member

🔎 ward agent advisor

ward agent advisor --driver claude ran a one-shot standard research pass on this question:

Design the substrate sparse-mount proposal per the (updated) issue body and land your design as a comment on ward#523. Non-negotiable framing already settled with the operator: (1) derive the sparse pattern from the trifecta reference surface (top-level + docs/ + .agents/skills/ + README/AGENTS/FEATURES), not hand-picked paths; (2) substrate is UNIFORMLY 'reference' - there is NO full/reference profile split and NO --substrate-full break-glass; the escape for 'need B's full source this session' is the EXISTING --repo owner/name flag, decided at launch; (3) do not reintroduce a standing full mount, including for cli-guard - cli-guard-needs-reading is just a --repo case; (4) a substrate reference gap must be a filable work item against that repo's docs so the forcing function accretes into doc debt; (5) flag but do NOT solve the '--repo is a work grant, reaper expects it to land a push, so using it purely to read collides' nuance - note it as a possible separate follow-up; (6) be honest that sparse-checkout saves attention/scan surface, not gitcache bandwidth. Implementation touches warmSubstrateRepo in cmd/ward/container_bootstrap.go.

I have all the grounding I need. The prior advisor comment proposed exactly what the updated framing rejects (a full/reference profile split + --substrate-full break-glass + a standing full mount for cli-guard). My job is the corrected design. This is single-repo (all in ward), so it goes in summary with empty issues.

{
  "summary": "## Design: sparse-mount substrate to its trifecta reference surface\n\n> **Supersedes the earlier advisor comment on this issue.** That draft proposed a `full`/`reference` profile field in the manifest, a `--substrate-full` break-glass, and a standing `full` mount for cli-guard. The operator has since settled the framing the other way: **substrate is uniformly `reference`, there is no profile field, no `--substrate-full`, and no standing full mount for anything.** The full-source escape is the *existing* `--repo owner/name` flag. This comment is the corrected design; where it differs from the prior one, this one wins.\n\n### Confirmed against the code (shallow clone of `main`)\n\n- `warmSubstrateRepo` — `cmd/ward/container_bootstrap.go:693`, signature `(ctx, e, owner, name, tier string)`. The mirror hydrate/refresh runs under an flock (lines 699-719); the **working-clone tail is lines 720-728**, currently `git clone --quiet <mirror> <work>` then `git -C <work> remote set-url origin <url>`. That tail is the *only* thing this change touches.\n- Manifest — `cmd/ward/containerassets/preclone-repos.txt`, parsed in `warmSubstrate` (`:749-767`) as `owner/name [tier]`. **`tier` controls the seed source only** (`image` = public, baked into dev-base + gitcache; `cache` = private, gitcache-only). It has nothing to do with checkout shape, and this design leaves it entirely alone.\n- `--repo` path — grants clone as **full** feature working copies into `/workspace/<name>` via `cloneExtraRepos` (`:554`), tracked as `ExtraRepos`, and the reaper's `verifyExtraReposLanded` (`:964`, consumed again at `:1054`) treats each grant as work that must land a push. This is the escape hatch *and* the source of the Q5 nuance below.\n- Target-overlap — `warmSubstrate` skips the target owner/name in its loop (`:764`), so `/workspace/<name>` stays full and authoritative; only the read-only `/substrate/<name>` shadow is affected.\n- git 2.50 locally; dev-base is well past the 2.36 floor cone mode needs.\n\n---\n\n### Q1 — The sparse pattern is derived from the trifecta, uniform for every repo\n\nThe reference surface every repo already must maintain (README, AGENTS.md, `docs/FEATURES.md`, plus `docs/` and `.agents/skills/`) maps exactly onto git **cone-mode** sparse-checkout from two directory patterns:\n\n```\ndocs\n.agents/skills\n```\n\nCone mode always materializes (a) every file in the repo root and (b) the full subtree of each named directory. So this yields: **all top-level files (README + AGENTS + any root `*.md`) + `docs/**` (FEATURES lives here) + `.agents/skills/**`** — the published reference surface, and nothing else. It is the **same two-token pattern for every repo**, derived from the trifecta contract, never a hand-picked or per-repo path list. There is no new per-repo config to drift as repos add or drop doc dirs.\n\nThe forcing function is a free property of cone mode: **a repo with thin docs gets a thin mount.** If a cross-repo contract lives only in source, it is simply absent under `/substrate/<name>`, and the remedy is to write it into the reference surface (Q4), not widen the cut. Edge cases fall out correctly with no special-casing: a repo missing `docs/` or `.agents/skills/` still gets its root files (README/AGENTS) — cone `set` on a nonexistent dir is a no-op. Thin repo, thin mount.\n\nNote this change adds **no trifecta *validation*** — that already lives in the `documentation-layout` / `catalog-trifecta` pre-commit hooks upstream in agentic-os. This change only makes substrate *expose* the surface those hooks already require.\n\n### Q2 — Substrate is uniformly `reference`; the manifest schema does not change\n\n**No profile field. The manifest line stays `owner/name [tier]`.** Every substrate repo is mounted as its reference surface, period. There is no `full` value to write, so `warmSubstrate`'s parser and `warmSubstrateRepo`'s signature are unchanged — the sparse cut is unconditional inside the working-clone tail.\n\nRationale (settled): a per-repo `full`/`reference` profile would be a **second mechanism for a job `--repo` already does**. The escape for \"this session needs B's full source\" is decided at launch with `--repo owner/name`, which pulls a full feature working copy into `/workspace`. Adding a substrate knob duplicates that with a worse property (standing config that quietly defeats the forcing function). So: one surface shape for substrate, one escape mechanism (`--repo`), no overlap.\n\n### Q3 — No standing full mount, including for cli-guard\n\nThe three reasons an agent reads a substrate repo, and why none of them needs a standing full mount:\n\n- **(a) honor a contract the repo publishes** → the reference surface *is* the contract. A gap is a doc bug (Q4), not a reason to read source. → served by the sparse mount.\n- **(b) operate the repo as a tool** → the right reference is the **runtime introspection surface** (`--help`, `describe`, `--json`), and the in-container checkout may not even match the installed binary — so the tree is the *wrong* reference. → served by running the tool, not by source.\n- **(c) reason about B's implementation as the task subject** → that is a **dispatch-into-B** run, or a `--repo B` pull to have B alongside the current repo in one session. → served by `--repo`, at launch.\n\nThe property that once seemed to justify a carve-out — \"has a runtime introspection surface in-container\" (ward does, via `--help`/`describe`; cli-guard is not on PATH and does not) — **is a real distinction, but the remedy it pointed at is redundant.** A repo with no runtime surface is read exactly the way any full-source read is served: `--repo owner/name`. So **cli-guard-needs-reading is just another `--repo` case**, not a standing `full` line. The tempting \"dev CLIs need source\" carve-out proves too much (all behavior lives in code, so it would exempt everything); the honest split is *why* you read, and every why above resolves without a standing full mount. `full` ships **nonexistent**, not merely empty.\n\n### Q4 — A substrate reference gap is a filable work item (the accretion mechanism)\n\nThis is what makes the sparse cut *improve* docs rather than just relocate the workaround. Teach, in `AGENTS.container.md` and `docs/container-substrate.md`, that **when an agent hits a wall because the substrate mount is missing something a cross-repo reader legitimately needs, the correct move is capture-and-dispatch:** file an issue against repo B's tracker (\"reference surface missing X that a cross-repo reader needed\"), then either dispatch-into-B or relaunch with `--repo B` to unblock the current run.\n\nThe anti-pattern to name explicitly: silently `grep`-ing around source to route past thin docs, or reflexively reaching for `--repo` as a habitual escape without filing anything. Both dissipate the pain instead of accreting it. The smaller surface only pays off if the friction it creates is **routed to a doc fix**. Pain becomes visible doc debt, paid down deliberately, and the reference surface gets thicker over time — which is the entire point of the forcing function.\n\n### Q5 — `--repo`-as-read collides with the reaper (flag only, likely a follow-up)\n\n`--repo` today is a **work grant**: `verifyExtraReposLanded` (`:964`) verifies each granted repo *landed a push* and treats a non-landed grant as a hard failure (salvage branch + reopened issue). So using `--repo B` purely to **read** B — the exact case Q3 routes cli-guard-needs-reading and cross-repo debugging into — collides with that landing check: a read-only grant lands no push and trips the reaper.\n\nThis is real but rare, so the settled call is **accept the friction for now and do not solve it here.** Worth a **separate follow-up issue** in ward: a lighter *read-only grant* variant (e.g. `--repo-ro owner/name`, or a read flag on the existing grant) that mounts B full into `/workspace` but is exempt from the landing verification. That variant is what would make \"cli-guard-needs-reading is just a `--repo` case\" fully honest — until then, a read-only `--repo` user must expect the reaper to complain (or push a trivial no-op, which is worse). **Do not fold this into the sparse-mount change.**\n\n### Q6 — Honest cost note: attention surface, not bandwidth\n\n**Sparse-checkout on a full bare mirror still fetches every object.** The gitcache entry is a `git clone --mirror` and the container working copy is a local clone out of it; sparse only limits which blobs get *written into the working tree*, not which objects the mirror holds or transfers. So there is **no network win and no gitcache-bandwidth win.** The win is purely **attention / scan surface**: a smaller tree for a low-context agent to `ls`, `grep`, and wander, so it lands on the curated reference surface instead of spelunking source. State this plainly in `docs/container-substrate.md` so the change is not oversold as an infra optimization. (A future partial/blobless-mirror change could add a real bandwidth win — separate axis, out of scope.)\n\n---\n\n### Implementation sketch (`warmSubstrateRepo`, working-clone tail only)\n\nSignature and the flocked mirror block are unchanged. Only lines 720-728 change:\n\n```go\nif isDir(mirror) {\n    work := filepath.Join(e.SubstrateDest, name)\n    _ = os.RemoveAll(work)\n    // Sparse: clone without a checkout, set the trifecta reference surface in\n    // cone mode, then check out. Best-effort like the rest of warming.\n    if cerr := r.Runner.Exec(ctx, \"git\", \"clone\", \"--quiet\", \"--no-checkout\", mirror, work); cerr != nil {\n        blog(\"substrate: working clone failed %s/%s\", owner, name)\n        return\n    }\n    // Root files + docs/** + .agents/skills/** — the published reference surface.\n    if serr := r.Runner.Exec(ctx, \"git\", \"-C\", work, \"sparse-checkout\", \"set\", \"--cone\", \"docs\", \".agents/skills\"); serr != nil {\n        blog(\"substrate: sparse-checkout set failed %s/%s (degrading to full)\", owner, name)\n        _ = r.Runner.Exec(ctx, \"git\", \"-C\", work, \"sparse-checkout\", \"disable\")\n    }\n    if cerr := r.Runner.Exec(ctx, \"git\", \"-C\", work, \"checkout\"); cerr != nil {\n        blog(\"substrate: checkout failed %s/%s\", owner, name)\n    }\n    _ = r.Runner.Exec(ctx, \"git\", \"-C\", work, \"remote\", \"set-url\", \"origin\", url)\n}\n```\n\nNotes for the implementer:\n- **Degrade-to-full, never empty.** If `sparse-checkout set` fails (a git-version surprise, cone-mode quirk), disable sparse so the subsequent `checkout` produces a *full* tree rather than an empty one. A too-big mount is a strictly better failure than a missing one. Keep every step best-effort with a `blog` line, consistent with the surrounding warming code.\n- `git clone --sparse` (cone-init with only root files) is an equivalent alternative to `--no-checkout` + `set` + `checkout`; either is fine. The `--no-checkout` form is used above because the degrade path is more explicit.\n- Cone mode needs git ≥ 2.36; note this floor in the doc. dev-base is well past it.\n- No parser or signature change — the cut is unconditional. `tier` stays untouched.\n\n### Doc / doctrine edits (same commit, per house trifecta rules)\n\n- `docs/container-substrate.md` — new section: substrate is mounted as its **trifecta reference surface** (cone pattern `docs` + `.agents/skills` + root files), *why* (forcing function on an already-mandatory rule), the **honest cost note** (attention surface, not bandwidth), and the git-version floor.\n- `cmd/ward/containerassets/AGENTS.container.md` — the agent-facing rules: (1) substrate is the reference *surface*, not full source; operate tools via their runtime surface (`--help`/`describe`), not the checkout; (2) full source for a session comes from `--repo owner/name` at launch (dispatch-into-B for implementation work); (3) **a substrate reference gap is a filable work item** — capture-and-dispatch against repo B's docs/FEATURES, do not silently spelunk.\n- `cmd/ward/containerassets/preclone-repos.txt` — header comment: state that substrate mounts the reference surface (sparse cone checkout), that there is **no profile field** (line stays `owner/name [tier]`, `tier` = seed source only), and that full source is `--repo`, not a manifest knob.\n- `docs/FEATURES.md` — update the substrate entry: reference-surface sparse mount + the `--repo` escape (trifecta upkeep).\n\n### Follow-up to file (do not fold in)\n\n- **ward** — *\"Read-only `--repo` grant variant exempt from the reaper landing check.\"* `--repo` is a work grant that `verifyExtraReposLanded` requires to land a push, so using it purely to read B (cli-guard, cross-repo debugging) trips the reaper. A lighter read-only grant would make \"just use `--repo`\" honest for the pure-read case. Downstream of this issue; not a blocker for the sparse mount."
}

Researched and posted automatically by ward agent advisor --driver claude (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Claude (she/her), via ward agent

### 🔎 ward agent advisor `ward agent advisor --driver claude` ran a one-shot **standard** research pass on this question: > Design the substrate sparse-mount proposal per the (updated) issue body and land your design as a comment on ward#523. Non-negotiable framing already settled with the operator: (1) derive the sparse pattern from the trifecta reference surface (top-level + docs/ + .agents/skills/ + README/AGENTS/FEATURES), not hand-picked paths; (2) substrate is UNIFORMLY 'reference' - there is NO full/reference profile split and NO --substrate-full break-glass; the escape for 'need B's full source this session' is the EXISTING --repo owner/name flag, decided at launch; (3) do not reintroduce a standing full mount, including for cli-guard - cli-guard-needs-reading is just a --repo case; (4) a substrate reference gap must be a filable work item against that repo's docs so the forcing function accretes into doc debt; (5) flag but do NOT solve the '--repo is a work grant, reaper expects it to land a push, so using it purely to read collides' nuance - note it as a possible separate follow-up; (6) be honest that sparse-checkout saves attention/scan surface, not gitcache bandwidth. Implementation touches warmSubstrateRepo in cmd/ward/container_bootstrap.go. --- I have all the grounding I need. The prior advisor comment proposed exactly what the updated framing rejects (a `full`/`reference` profile split + `--substrate-full` break-glass + a standing `full` mount for cli-guard). My job is the corrected design. This is single-repo (all in ward), so it goes in `summary` with empty `issues`. ```json { "summary": "## Design: sparse-mount substrate to its trifecta reference surface\n\n> **Supersedes the earlier advisor comment on this issue.** That draft proposed a `full`/`reference` profile field in the manifest, a `--substrate-full` break-glass, and a standing `full` mount for cli-guard. The operator has since settled the framing the other way: **substrate is uniformly `reference`, there is no profile field, no `--substrate-full`, and no standing full mount for anything.** The full-source escape is the *existing* `--repo owner/name` flag. This comment is the corrected design; where it differs from the prior one, this one wins.\n\n### Confirmed against the code (shallow clone of `main`)\n\n- `warmSubstrateRepo` — `cmd/ward/container_bootstrap.go:693`, signature `(ctx, e, owner, name, tier string)`. The mirror hydrate/refresh runs under an flock (lines 699-719); the **working-clone tail is lines 720-728**, currently `git clone --quiet <mirror> <work>` then `git -C <work> remote set-url origin <url>`. That tail is the *only* thing this change touches.\n- Manifest — `cmd/ward/containerassets/preclone-repos.txt`, parsed in `warmSubstrate` (`:749-767`) as `owner/name [tier]`. **`tier` controls the seed source only** (`image` = public, baked into dev-base + gitcache; `cache` = private, gitcache-only). It has nothing to do with checkout shape, and this design leaves it entirely alone.\n- `--repo` path — grants clone as **full** feature working copies into `/workspace/<name>` via `cloneExtraRepos` (`:554`), tracked as `ExtraRepos`, and the reaper's `verifyExtraReposLanded` (`:964`, consumed again at `:1054`) treats each grant as work that must land a push. This is the escape hatch *and* the source of the Q5 nuance below.\n- Target-overlap — `warmSubstrate` skips the target owner/name in its loop (`:764`), so `/workspace/<name>` stays full and authoritative; only the read-only `/substrate/<name>` shadow is affected.\n- git 2.50 locally; dev-base is well past the 2.36 floor cone mode needs.\n\n---\n\n### Q1 — The sparse pattern is derived from the trifecta, uniform for every repo\n\nThe reference surface every repo already must maintain (README, AGENTS.md, `docs/FEATURES.md`, plus `docs/` and `.agents/skills/`) maps exactly onto git **cone-mode** sparse-checkout from two directory patterns:\n\n```\ndocs\n.agents/skills\n```\n\nCone mode always materializes (a) every file in the repo root and (b) the full subtree of each named directory. So this yields: **all top-level files (README + AGENTS + any root `*.md`) + `docs/**` (FEATURES lives here) + `.agents/skills/**`** — the published reference surface, and nothing else. It is the **same two-token pattern for every repo**, derived from the trifecta contract, never a hand-picked or per-repo path list. There is no new per-repo config to drift as repos add or drop doc dirs.\n\nThe forcing function is a free property of cone mode: **a repo with thin docs gets a thin mount.** If a cross-repo contract lives only in source, it is simply absent under `/substrate/<name>`, and the remedy is to write it into the reference surface (Q4), not widen the cut. Edge cases fall out correctly with no special-casing: a repo missing `docs/` or `.agents/skills/` still gets its root files (README/AGENTS) — cone `set` on a nonexistent dir is a no-op. Thin repo, thin mount.\n\nNote this change adds **no trifecta *validation*** — that already lives in the `documentation-layout` / `catalog-trifecta` pre-commit hooks upstream in agentic-os. This change only makes substrate *expose* the surface those hooks already require.\n\n### Q2 — Substrate is uniformly `reference`; the manifest schema does not change\n\n**No profile field. The manifest line stays `owner/name [tier]`.** Every substrate repo is mounted as its reference surface, period. There is no `full` value to write, so `warmSubstrate`'s parser and `warmSubstrateRepo`'s signature are unchanged — the sparse cut is unconditional inside the working-clone tail.\n\nRationale (settled): a per-repo `full`/`reference` profile would be a **second mechanism for a job `--repo` already does**. The escape for \"this session needs B's full source\" is decided at launch with `--repo owner/name`, which pulls a full feature working copy into `/workspace`. Adding a substrate knob duplicates that with a worse property (standing config that quietly defeats the forcing function). So: one surface shape for substrate, one escape mechanism (`--repo`), no overlap.\n\n### Q3 — No standing full mount, including for cli-guard\n\nThe three reasons an agent reads a substrate repo, and why none of them needs a standing full mount:\n\n- **(a) honor a contract the repo publishes** → the reference surface *is* the contract. A gap is a doc bug (Q4), not a reason to read source. → served by the sparse mount.\n- **(b) operate the repo as a tool** → the right reference is the **runtime introspection surface** (`--help`, `describe`, `--json`), and the in-container checkout may not even match the installed binary — so the tree is the *wrong* reference. → served by running the tool, not by source.\n- **(c) reason about B's implementation as the task subject** → that is a **dispatch-into-B** run, or a `--repo B` pull to have B alongside the current repo in one session. → served by `--repo`, at launch.\n\nThe property that once seemed to justify a carve-out — \"has a runtime introspection surface in-container\" (ward does, via `--help`/`describe`; cli-guard is not on PATH and does not) — **is a real distinction, but the remedy it pointed at is redundant.** A repo with no runtime surface is read exactly the way any full-source read is served: `--repo owner/name`. So **cli-guard-needs-reading is just another `--repo` case**, not a standing `full` line. The tempting \"dev CLIs need source\" carve-out proves too much (all behavior lives in code, so it would exempt everything); the honest split is *why* you read, and every why above resolves without a standing full mount. `full` ships **nonexistent**, not merely empty.\n\n### Q4 — A substrate reference gap is a filable work item (the accretion mechanism)\n\nThis is what makes the sparse cut *improve* docs rather than just relocate the workaround. Teach, in `AGENTS.container.md` and `docs/container-substrate.md`, that **when an agent hits a wall because the substrate mount is missing something a cross-repo reader legitimately needs, the correct move is capture-and-dispatch:** file an issue against repo B's tracker (\"reference surface missing X that a cross-repo reader needed\"), then either dispatch-into-B or relaunch with `--repo B` to unblock the current run.\n\nThe anti-pattern to name explicitly: silently `grep`-ing around source to route past thin docs, or reflexively reaching for `--repo` as a habitual escape without filing anything. Both dissipate the pain instead of accreting it. The smaller surface only pays off if the friction it creates is **routed to a doc fix**. Pain becomes visible doc debt, paid down deliberately, and the reference surface gets thicker over time — which is the entire point of the forcing function.\n\n### Q5 — `--repo`-as-read collides with the reaper (flag only, likely a follow-up)\n\n`--repo` today is a **work grant**: `verifyExtraReposLanded` (`:964`) verifies each granted repo *landed a push* and treats a non-landed grant as a hard failure (salvage branch + reopened issue). So using `--repo B` purely to **read** B — the exact case Q3 routes cli-guard-needs-reading and cross-repo debugging into — collides with that landing check: a read-only grant lands no push and trips the reaper.\n\nThis is real but rare, so the settled call is **accept the friction for now and do not solve it here.** Worth a **separate follow-up issue** in ward: a lighter *read-only grant* variant (e.g. `--repo-ro owner/name`, or a read flag on the existing grant) that mounts B full into `/workspace` but is exempt from the landing verification. That variant is what would make \"cli-guard-needs-reading is just a `--repo` case\" fully honest — until then, a read-only `--repo` user must expect the reaper to complain (or push a trivial no-op, which is worse). **Do not fold this into the sparse-mount change.**\n\n### Q6 — Honest cost note: attention surface, not bandwidth\n\n**Sparse-checkout on a full bare mirror still fetches every object.** The gitcache entry is a `git clone --mirror` and the container working copy is a local clone out of it; sparse only limits which blobs get *written into the working tree*, not which objects the mirror holds or transfers. So there is **no network win and no gitcache-bandwidth win.** The win is purely **attention / scan surface**: a smaller tree for a low-context agent to `ls`, `grep`, and wander, so it lands on the curated reference surface instead of spelunking source. State this plainly in `docs/container-substrate.md` so the change is not oversold as an infra optimization. (A future partial/blobless-mirror change could add a real bandwidth win — separate axis, out of scope.)\n\n---\n\n### Implementation sketch (`warmSubstrateRepo`, working-clone tail only)\n\nSignature and the flocked mirror block are unchanged. Only lines 720-728 change:\n\n```go\nif isDir(mirror) {\n work := filepath.Join(e.SubstrateDest, name)\n _ = os.RemoveAll(work)\n // Sparse: clone without a checkout, set the trifecta reference surface in\n // cone mode, then check out. Best-effort like the rest of warming.\n if cerr := r.Runner.Exec(ctx, \"git\", \"clone\", \"--quiet\", \"--no-checkout\", mirror, work); cerr != nil {\n blog(\"substrate: working clone failed %s/%s\", owner, name)\n return\n }\n // Root files + docs/** + .agents/skills/** — the published reference surface.\n if serr := r.Runner.Exec(ctx, \"git\", \"-C\", work, \"sparse-checkout\", \"set\", \"--cone\", \"docs\", \".agents/skills\"); serr != nil {\n blog(\"substrate: sparse-checkout set failed %s/%s (degrading to full)\", owner, name)\n _ = r.Runner.Exec(ctx, \"git\", \"-C\", work, \"sparse-checkout\", \"disable\")\n }\n if cerr := r.Runner.Exec(ctx, \"git\", \"-C\", work, \"checkout\"); cerr != nil {\n blog(\"substrate: checkout failed %s/%s\", owner, name)\n }\n _ = r.Runner.Exec(ctx, \"git\", \"-C\", work, \"remote\", \"set-url\", \"origin\", url)\n}\n```\n\nNotes for the implementer:\n- **Degrade-to-full, never empty.** If `sparse-checkout set` fails (a git-version surprise, cone-mode quirk), disable sparse so the subsequent `checkout` produces a *full* tree rather than an empty one. A too-big mount is a strictly better failure than a missing one. Keep every step best-effort with a `blog` line, consistent with the surrounding warming code.\n- `git clone --sparse` (cone-init with only root files) is an equivalent alternative to `--no-checkout` + `set` + `checkout`; either is fine. The `--no-checkout` form is used above because the degrade path is more explicit.\n- Cone mode needs git ≥ 2.36; note this floor in the doc. dev-base is well past it.\n- No parser or signature change — the cut is unconditional. `tier` stays untouched.\n\n### Doc / doctrine edits (same commit, per house trifecta rules)\n\n- `docs/container-substrate.md` — new section: substrate is mounted as its **trifecta reference surface** (cone pattern `docs` + `.agents/skills` + root files), *why* (forcing function on an already-mandatory rule), the **honest cost note** (attention surface, not bandwidth), and the git-version floor.\n- `cmd/ward/containerassets/AGENTS.container.md` — the agent-facing rules: (1) substrate is the reference *surface*, not full source; operate tools via their runtime surface (`--help`/`describe`), not the checkout; (2) full source for a session comes from `--repo owner/name` at launch (dispatch-into-B for implementation work); (3) **a substrate reference gap is a filable work item** — capture-and-dispatch against repo B's docs/FEATURES, do not silently spelunk.\n- `cmd/ward/containerassets/preclone-repos.txt` — header comment: state that substrate mounts the reference surface (sparse cone checkout), that there is **no profile field** (line stays `owner/name [tier]`, `tier` = seed source only), and that full source is `--repo`, not a manifest knob.\n- `docs/FEATURES.md` — update the substrate entry: reference-surface sparse mount + the `--repo` escape (trifecta upkeep).\n\n### Follow-up to file (do not fold in)\n\n- **ward** — *\"Read-only `--repo` grant variant exempt from the reaper landing check.\"* `--repo` is a work grant that `verifyExtraReposLanded` requires to land a push, so using it purely to read B (cli-guard, cross-repo debugging) trips the reaper. A lighter read-only grant would make \"just use `--repo`\" honest for the pure-read case. Downstream of this issue; not a blocker for the sparse mount." } ``` --- Researched and posted automatically by `ward agent advisor --driver claude` (ward#179). This is one-shot research, not a carried change - verify before acting on it.   — Claude (she/her), via `ward agent`

coilysiren commented

2026-07-02 08:45:50 +00:00

Author

Owner

Framing correction landed AFTER the advisor run was dispatched (the advisor launched on the pre-correction prompt, and there is no agent-to-agent channel to steer it mid-flight). The settled axis is now in question 2 of the body: the sparse cut keys off the RUN ROLE, not the repo. Advisor runs get a FULL substrate clone (their whole purpose is info-gathering, so ground truth beats a curated surface, and they are positioned to detect and file doc gaps). Engineer/default runs get the sparse trifecta reference surface (they implement, which is where the small surface and forcing function pay off, and they own the docs so they should feel a gap). If the in-flight advisor's design omits this role-keyed exception, graft it in on review: hydration branches on the run role in warmSubstrateRepo, engineers use the existing --repo for full source, and there is no per-repo full profile.

coilyco-ops added the

consult

labels

2026-07-02 09:02:24 +00:00

coilysiren referenced this issue

2026-07-03 17:50:30 +00:00

Mount the issue-corpus into /substrate via coilysiren/inbox double-duty (consumer half of agentic-os#297) #575

coilysiren referenced this issue

2026-07-03 17:59:00 +00:00

Mount the issue-corpus into /substrate via coilysiren/inbox double-duty (consumer half of agentic-os#297) #575

coilysiren referenced this issue

2026-07-04 04:24:25 +00:00