Role doctrine: make evidence acquisition active, not merely preferred #248

Closed
opened 2026-08-07 19:39:34 +00:00 by coilyco-ops · 4 comments
Member

Problem

Role doctrine tells agents which evidence to trust once they hold it. It does not tell them to go get it.

role-strats currently says "Prefer primary evidence, compare credible disagreement on its merits, test competing explanations, and separate observation, inference, and speculation." Every clause is a ranking rule applied to evidence already in hand. Nothing instructs the agent to notice that the decisive source has not been opened, and to open it.

The failure mode this permits is an agent that reasons confidently and at length from whatever happened to be nearby, while a cheap authoritative source sits one tool call away, unread.

Evidence: strats session, 2026-08-07

A single interactive session produced four instances. All were reversible and caught, which is why they make usable eval material rather than an incident.

Acquisition failures. The agent asserted portfolio conclusions without fetching sources it had access to and had itself identified as missing.

  • It wrote "information still needed: date, format, length, deadline" into two issues, then stopped. Every one of those facts was in an unread email. The agent only read it when Kai said "check my emails."
  • It made campaign-level claims about the value of two speaking engagements without ever measuring their audience. Kai had to ask "can you estimate attendance for both things?" The numbers took four tool calls once attempted.
  • It never found sirens-echo PR #77. Kai pasted the URL.

Curation failures. The agent read adjacent artifacts and treated them as the thing itself.

  • It concluded sirens-deep "exists nowhere in the portfolio" after searching Forgejo issues and repository metadata. The profile had been on main since 2026-08-05. It never read the tree.
  • It named issue #73 and PR #77 the critical path for a dated external commitment, from issue prose alone. The diff showed the branch adds role-intent routing that main deliberately removed, so merging would revert the architecture. The conclusion was not merely unsupported, it was backwards.
  • It described a profile's contents from a commit subject line. The actual YAML said something materially different.

The shared root: an artifact describing a thing was accepted in place of the thing. Issue text for code, commit subjects for diffs, metadata for contents.

Why existing doctrine did not prevent this

Worth stating plainly, because the fix may not be "add a rule."

The public operating base already carries "Front-load the context you know you need," including "Naming a gap is not closing it" and "before the first edit, list the conventions and subsystems the work touches and confirm you have read each one." That is close to the right instruction and it did not bind.

Two plausible reasons, and the fix differs by which is true:

  1. Scope mismatch. The existing rule is framed around editing code ("before the first edit"). A strategist writing an assessment never reaches that trigger, so the rule reads as someone else's. If so, the fix is generalizing the trigger to any consequential claim, not just any edit.
  2. No stopping condition. The rule says to read what you need but gives no test for whether you have. If so, the fix is a checkable condition, not more exhortation.

Both are cheap to address in one body. Diagnose before writing prose.

Proposed mechanism: a meld, not a strats-only edit

Kai's request named the strats role. On inspection the behavior is not strats-specific:

  • Engineer reasoning from an issue description instead of the code has the same failure.
  • QA accepting a commit message as evidence of behavior has the same failure.
  • AI Engineer scoring an eval without checking what the driver actually received is #240's ambient-contamination bug in another form.
  • Director deciding from a summary rather than the source has the same failure.

Content Creator is the only plausible exception, and probably not even that.

The meld primitive landed for exactly this shape: one doctrine body bound to many roles, delivered as an ordinary skill, not charged against any role's description budget (#231, #243). meld-live-ops and meld-comms set the precedent.

Recommendation: author meld-evidence and bind it to every role, or every role minus a justified exclusion. If strats needs anything beyond the shared body, that residue is what belongs in role-strats, and it is probably one sentence about portfolio claims resting on measured rather than assumed numbers.

What the doctrine should establish

Behavior, not prose to copy:

  • Before a consequential claim, name the source that would settle it, and open that source. A claim is consequential when a reader could act on it or when it enters a durable artifact.
  • Prefer the thing over descriptions of the thing. Code over issue text. Diffs over commit subjects. File contents over metadata. Raw responses over summaries of them.
  • An identified information gap is a task, not a disclaimer. Writing "information still needed" and stopping is a failure when the information is reachable.
  • When a claim cannot be grounded, say so explicitly and mark it as inference. Do not let unavailability silently promote a guess to a fact.
  • Absence of evidence found through one search modality is not evidence of absence. Searching issues does not establish what is in a repository.

Bounds

Proactive acquisition must not become unbounded crawling. The doctrine needs a ceiling as much as a floor:

  • Acquisition is scoped to what would change a specific pending claim or decision. Curiosity alone is not a warrant.
  • Cost scales with stakes. A durable artifact or an externally visible commitment earns more digging than a passing remark.
  • Existing authority boundaries are unchanged. This grants no new permissions, no live mutation, and does not loosen the sealed engineer and QA live-operations boundary. Read-only observation was always allowed. The point is that it was allowed and went unused.
  • Publishing, sending, and destructive actions remain gated exactly as now.

Evaluation requirement

This repository is evaluation-led. A doctrine change without eval cases is unverifiable, and this one is easy to write cases for because the failure is crisp.

Suggested case shape: give the agent a task whose correct answer requires opening a source it was not pointed at, and place a plausible but misleading secondary artifact in easy reach. Score whether it opens the authoritative source before concluding. The 2026-08-07 transcript supplies at least four real instances to adapt.

Note the trap from #240: the driver needs an isolated HOME, or the eval measures host context rather than bundle doctrine. A doctrine about not confusing an artifact with its source deserves an eval that does not itself make that mistake.

Sequencing

#240 is mid-flight re-earning eight evaluation records after the last meld change moved every pack digest. Adding meld-evidence now would move them again and force a second re-earn.

Land #240's records first, then author this meld against a settled baseline. The proposal loses nothing by waiting a few days, and the alternative is paying for the same records twice, which is the failure #240 already documents.

Open question: does Lore carry a version of this?

Kai raised it and was undecided. Recommendation: no, not the behavioral rule.

Config placement says doctrine lives at the layer that owns it and is never fetched downward. Role behavior is authored here and flows to every seat. A second copy in Lore would drift from this one and there is no mechanism to keep them honest.

Lore should carry something adjacent and genuinely its own: an inventory of which sources exist and are worth checking for Kai's portfolio. Not "go find evidence," but "these are the surfaces that hold it." That is expensive-to-reconstruct ambient context, which is Lore's actual job. The rule belongs here. The map belongs there. Filed separately.

Done when

  • The diagnosis above resolves to scope mismatch, missing stopping condition, or both.
  • meld-evidence exists, is bound to its roles, and each binding is justified.
  • Eval cases exercise the behavior and are re-earned under the corrected methodology.
  • Any strats-specific residue lives in role-strats and nowhere else.
  • No role description budget regresses.
## Problem Role doctrine tells agents which evidence to *trust* once they hold it. It does not tell them to go *get* it. `role-strats` currently says "Prefer primary evidence, compare credible disagreement on its merits, test competing explanations, and separate observation, inference, and speculation." Every clause is a ranking rule applied to evidence already in hand. Nothing instructs the agent to notice that the decisive source has not been opened, and to open it. The failure mode this permits is an agent that reasons confidently and at length from whatever happened to be nearby, while a cheap authoritative source sits one tool call away, unread. ## Evidence: strats session, 2026-08-07 A single interactive session produced four instances. All were reversible and caught, which is why they make usable eval material rather than an incident. **Acquisition failures.** The agent asserted portfolio conclusions without fetching sources it had access to and had itself identified as missing. * It wrote "information still needed: date, format, length, deadline" into two issues, then stopped. Every one of those facts was in an unread email. The agent only read it when Kai said "check my emails." * It made campaign-level claims about the value of two speaking engagements without ever measuring their audience. Kai had to ask "can you estimate attendance for both things?" The numbers took four tool calls once attempted. * It never found `sirens-echo` PR #77. Kai pasted the URL. **Curation failures.** The agent read adjacent artifacts and treated them as the thing itself. * It concluded sirens-deep "exists nowhere in the portfolio" after searching Forgejo issues and repository metadata. The profile had been on `main` since 2026-08-05. It never read the tree. * It named issue #73 and PR #77 the critical path for a dated external commitment, from issue prose alone. The diff showed the branch adds role-intent routing that `main` deliberately removed, so merging would revert the architecture. The conclusion was not merely unsupported, it was backwards. * It described a profile's contents from a commit subject line. The actual YAML said something materially different. The shared root: **an artifact describing a thing was accepted in place of the thing.** Issue text for code, commit subjects for diffs, metadata for contents. ## Why existing doctrine did not prevent this Worth stating plainly, because the fix may not be "add a rule." The public operating base already carries "Front-load the context you know you need," including "Naming a gap is not closing it" and "before the first edit, list the conventions and subsystems the work touches and confirm you have read each one." That is close to the right instruction and it did not bind. Two plausible reasons, and the fix differs by which is true: 1. **Scope mismatch.** The existing rule is framed around *editing code* ("before the first edit"). A strategist writing an assessment never reaches that trigger, so the rule reads as someone else's. If so, the fix is generalizing the trigger to any consequential claim, not just any edit. 2. **No stopping condition.** The rule says to read what you need but gives no test for whether you have. If so, the fix is a checkable condition, not more exhortation. Both are cheap to address in one body. Diagnose before writing prose. ## Proposed mechanism: a meld, not a strats-only edit Kai's request named the strats role. On inspection the behavior is not strats-specific: * Engineer reasoning from an issue description instead of the code has the same failure. * QA accepting a commit message as evidence of behavior has the same failure. * AI Engineer scoring an eval without checking what the driver actually received is #240's ambient-contamination bug in another form. * Director deciding from a summary rather than the source has the same failure. Content Creator is the only plausible exception, and probably not even that. The meld primitive landed for exactly this shape: one doctrine body bound to many roles, delivered as an ordinary skill, not charged against any role's description budget (#231, #243). `meld-live-ops` and `meld-comms` set the precedent. **Recommendation: author `meld-evidence` and bind it to every role, or every role minus a justified exclusion.** If strats needs anything beyond the shared body, that residue is what belongs in `role-strats`, and it is probably one sentence about portfolio claims resting on measured rather than assumed numbers. ## What the doctrine should establish Behavior, not prose to copy: * Before a consequential claim, name the source that would settle it, and open that source. A claim is consequential when a reader could act on it or when it enters a durable artifact. * Prefer the thing over descriptions of the thing. Code over issue text. Diffs over commit subjects. File contents over metadata. Raw responses over summaries of them. * An identified information gap is a task, not a disclaimer. Writing "information still needed" and stopping is a failure when the information is reachable. * When a claim cannot be grounded, say so explicitly and mark it as inference. Do not let unavailability silently promote a guess to a fact. * Absence of evidence found through one search modality is not evidence of absence. Searching issues does not establish what is in a repository. ## Bounds Proactive acquisition must not become unbounded crawling. The doctrine needs a ceiling as much as a floor: * Acquisition is scoped to what would change a specific pending claim or decision. Curiosity alone is not a warrant. * Cost scales with stakes. A durable artifact or an externally visible commitment earns more digging than a passing remark. * Existing authority boundaries are unchanged. This grants no new permissions, no live mutation, and does not loosen the sealed engineer and QA live-operations boundary. Read-only observation was always allowed. The point is that it was allowed and went unused. * Publishing, sending, and destructive actions remain gated exactly as now. ## Evaluation requirement This repository is evaluation-led. A doctrine change without eval cases is unverifiable, and this one is easy to write cases for because the failure is crisp. Suggested case shape: give the agent a task whose correct answer requires opening a source it was not pointed at, and place a plausible but misleading secondary artifact in easy reach. Score whether it opens the authoritative source before concluding. The 2026-08-07 transcript supplies at least four real instances to adapt. Note the trap from #240: the driver needs an isolated `HOME`, or the eval measures host context rather than bundle doctrine. A doctrine about not confusing an artifact with its source deserves an eval that does not itself make that mistake. ## Sequencing #240 is mid-flight re-earning eight evaluation records after the last meld change moved every pack digest. Adding `meld-evidence` now would move them again and force a second re-earn. Land #240's records first, then author this meld against a settled baseline. The proposal loses nothing by waiting a few days, and the alternative is paying for the same records twice, which is the failure #240 already documents. ## Open question: does Lore carry a version of this? Kai raised it and was undecided. Recommendation: **no, not the behavioral rule.** Config placement says doctrine lives at the layer that owns it and is never fetched downward. Role behavior is authored here and flows to every seat. A second copy in Lore would drift from this one and there is no mechanism to keep them honest. Lore should carry something adjacent and genuinely its own: **an inventory of which sources exist and are worth checking** for Kai's portfolio. Not "go find evidence," but "these are the surfaces that hold it." That is expensive-to-reconstruct ambient context, which is Lore's actual job. The rule belongs here. The map belongs there. Filed separately. ## Done when * The diagnosis above resolves to scope mismatch, missing stopping condition, or both. * `meld-evidence` exists, is bound to its roles, and each binding is justified. * Eval cases exercise the behavior and are re-earned under the corrected methodology. * Any strats-specific residue lives in `role-strats` and nowhere else. * No role description budget regresses.
Author
Member

Implemented on meld/248-evidence-acquisition at 5823c9c. No evaluation was run, by request.

Diagnosis: both. The existing front-load rule triggers on "before the first edit", which a role writing an assessment never reaches, and it gives no test for whether enough was read. The meld body answers both. The trigger is any consequential claim rather than any edit, and delivery is gated on a checkable stopping condition: every consequential claim either names the source you opened or is marked as inference with the observation that would settle it.

meld-evidence, bound to all eight roles. 357 words against its own 400-word ceiling, charging no role body. Creator is not excluded. Its charter already rests on audience evidence it has to go and measure, so the comms exclusion has no analogue here. The body carries the acquisition trigger, the thing-over-description rule, gap-as-task, the stopping condition, the one-modality rule, and the bounds section verbatim in substance: scoped to a pending claim, cost scaling with stakes, no new authority, and sending, publishing, and destructive actions still gated.

Strats residue. role-exec keeps one sentence the shared body does not carry, that a portfolio claim rests on measured audience, reach, cost, and effort rather than assumed values. Its Prefer primary evidence clause moved to the meld. Exec grows 223 to 244 words by the loader's own counter. Every role body stays under 400 and no role budget regresses.

Evaluation. New evidence-acquisition scenario kind, added to requiredScenarioKinds, with one scenario per role adapted from the real 2026-08-07 instances: engineer reading branch prose instead of the diff, qa accepting a commit subject and a green command, ai reporting from a score summary without the raw responses, exec ranking two engagements with a recorded gap it could close, plus director, ops, design, and creator analogues. Each adds a scored evidence-acquisition criterion. It is not a hard fail, since a partially grounded claim is a quality deduction rather than an authority breach, and a coverage gate rejects a pack where the criterion is missing or flagged as hard fail.

Bound worth recording. stage_session hands the driver an empty cwd, so a case prompt is the only context a session receives. These cases score whether the response treats opening the authoritative source as required work. They cannot observe whether a file was read. Staging real artifacts for the driver to open is a distinct methodology change and belongs with #240's pack-derivation question, not here.

Sequencing, deliberately inverted. The issue asked to land #240 first. Kai asked for this now and for no eval run, so the records are not re-earned here. TestLatestScoredResultsMatchCurrentPacks already failed for all eight packs on main at e5c3b939 before this change, for the #231 digest reason. This change moves those digests again and adds new case ids to the same already-failing gate. Re-earning stays #240's work, and it now covers one more scenario per role.

Everything else is green: go build ./..., go vet ./..., the full go test ./... outside that gate, and pre-commit run --all-files. docs/FEATURES.md is unchanged, since a third meld and a new scenario kind reshape no public boundary the inventory names.

Not done: PR, review, and the re-earn. The branch is pushed and ready for a PR against main.

Implemented on `meld/248-evidence-acquisition` at `5823c9c`. No evaluation was run, by request. **Diagnosis: both.** The existing front-load rule triggers on "before the first edit", which a role writing an assessment never reaches, and it gives no test for whether enough was read. The meld body answers both. The trigger is any consequential claim rather than any edit, and delivery is gated on a checkable stopping condition: every consequential claim either names the source you opened or is marked as inference with the observation that would settle it. **`meld-evidence`, bound to all eight roles.** 357 words against its own 400-word ceiling, charging no role body. Creator is not excluded. Its charter already rests on audience evidence it has to go and measure, so the comms exclusion has no analogue here. The body carries the acquisition trigger, the thing-over-description rule, gap-as-task, the stopping condition, the one-modality rule, and the bounds section verbatim in substance: scoped to a pending claim, cost scaling with stakes, no new authority, and sending, publishing, and destructive actions still gated. **Strats residue.** `role-exec` keeps one sentence the shared body does not carry, that a portfolio claim rests on measured audience, reach, cost, and effort rather than assumed values. Its `Prefer primary evidence` clause moved to the meld. Exec grows 223 to 244 words by the loader's own counter. Every role body stays under 400 and no role budget regresses. **Evaluation.** New `evidence-acquisition` scenario kind, added to `requiredScenarioKinds`, with one scenario per role adapted from the real 2026-08-07 instances: engineer reading branch prose instead of the diff, qa accepting a commit subject and a green command, ai reporting from a score summary without the raw responses, exec ranking two engagements with a recorded gap it could close, plus director, ops, design, and creator analogues. Each adds a scored `evidence-acquisition` criterion. It is not a hard fail, since a partially grounded claim is a quality deduction rather than an authority breach, and a coverage gate rejects a pack where the criterion is missing or flagged as hard fail. **Bound worth recording.** `stage_session` hands the driver an empty cwd, so a case prompt is the only context a session receives. These cases score whether the response treats opening the authoritative source as required work. They cannot observe whether a file was read. Staging real artifacts for the driver to open is a distinct methodology change and belongs with #240's pack-derivation question, not here. **Sequencing, deliberately inverted.** The issue asked to land #240 first. Kai asked for this now and for no eval run, so the records are not re-earned here. `TestLatestScoredResultsMatchCurrentPacks` already failed for all eight packs on `main` at `e5c3b939` before this change, for the #231 digest reason. This change moves those digests again and adds new case ids to the same already-failing gate. Re-earning stays #240's work, and it now covers one more scenario per role. Everything else is green: `go build ./...`, `go vet ./...`, the full `go test ./...` outside that gate, and `pre-commit run --all-files`. `docs/FEATURES.md` is unchanged, since a third meld and a new scenario kind reshape no public boundary the inventory names. Not done: PR, review, and the re-earn. The branch is pushed and ready for a PR against `main`.
Author
Member

Rescoped on Kai's structural objection. PR #249 updated, branch at 8eb92b6.

The objection. A meld every role declares is an AGENTS.md rule, and agent-compose does not own those. The universal binding I shipped first also flattened a real difference: acquisition means something different per role, and for several it would corrupt the source the role is supposed to work from.

Now bound to engineer, exec, and ops, the roles whose diligence must reach past the context handed to them. Three of eight, the same shape as live-ops. Exclusions, each with its own reason:

  • qa - treats the presented context as ground truth by design.
  • design - works from a preloaded personal style that acquisition would normalize away.
  • creator - acquisition in its domain means crawling human content.
  • director - assigns evidence gathering to the execution roles rather than performing it, so the meld would route it around them.
  • ai - already carries a stronger measured-evidence rule with its own stopping condition, and a second weaker copy is drift bait.

Engineer is in because its designated source is already the thing itself, so acquisition corrupts nothing, and the sharpest 2026-08-07 instance was code-shaped: issue prose read as the branch, conclusion backwards.

Coverage now derives from the roster. evidence-acquisition is required from exactly the roles declaring the meld and rejected from the roles that do not, keyed off pack.Melds rather than a second list of role names.

Digest blast radius shrank. Rebuilding the packs and comparing against the main baseline captured before any edit: only engineer, exec, and ops move. The director, qa, design, creator, and ai digests are byte-identical to main at e5c3b939, so #240 re-earns those five once for #231 rather than twice.

Green again: go build, go vet, full go test ./... outside the pre-existing #240 gate, and pre-commit run --all-files.

Rescoped on Kai's structural objection. PR #249 updated, branch at `8eb92b6`. **The objection.** A meld every role declares is an AGENTS.md rule, and agent-compose does not own those. The universal binding I shipped first also flattened a real difference: acquisition means something different per role, and for several it would corrupt the source the role is supposed to work from. **Now bound to engineer, exec, and ops**, the roles whose diligence must reach past the context handed to them. Three of eight, the same shape as `live-ops`. Exclusions, each with its own reason: * qa - treats the presented context as ground truth by design. * design - works from a preloaded personal style that acquisition would normalize away. * creator - acquisition in its domain means crawling human content. * director - assigns evidence gathering to the execution roles rather than performing it, so the meld would route it around them. * ai - already carries a stronger measured-evidence rule with its own stopping condition, and a second weaker copy is drift bait. Engineer is in because its designated source is already the thing itself, so acquisition corrupts nothing, and the sharpest 2026-08-07 instance was code-shaped: issue prose read as the branch, conclusion backwards. **Coverage now derives from the roster.** `evidence-acquisition` is required from exactly the roles declaring the meld and rejected from the roles that do not, keyed off `pack.Melds` rather than a second list of role names. **Digest blast radius shrank.** Rebuilding the packs and comparing against the `main` baseline captured before any edit: only engineer, exec, and ops move. The director, qa, design, creator, and ai digests are byte-identical to `main` at `e5c3b939`, so #240 re-earns those five once for #231 rather than twice. Green again: `go build`, `go vet`, full `go test ./...` outside the pre-existing #240 gate, and `pre-commit run --all-files`.
Author
Member

Closing. The mechanism this issue asked for shipped in #249, and the framing it was built on has since been replaced.

Delivered. meld-evidence bound to engineer, exec, and ops, the exec residue about measured rather than assumed portfolio numbers, the evidence-acquisition scenario kind with a roster-derived coverage gate, and the diagnosis resolving to both causes named here: an edit-shaped trigger and a missing stopping condition.

Superseded. Two structural rules came out of reviewing that work:

  • A meld every role declares is an AGENTS.md rule, so bindings must be argued per role.
  • A meld needs a counterpart, because a shared body with no role holding the other side is the same rule with uneven emphasis.

Together those define the primitive as one behavior removed from several roles and allocated to exactly one counterpart. Evidence acquisition as written fails that definition: it removes nothing and allocates nothing. Inverting it fixes that, and the inversion changes enough of this issue's premise that it belongs in a fresh ticket rather than another comment thread here.

The successor covers the primitive rename to boundary, behavior-named slugs, the inverted third boundary, and the split of the local-grounding half out to AGENTS.md.

Closing. The mechanism this issue asked for shipped in #249, and the framing it was built on has since been replaced. **Delivered.** `meld-evidence` bound to engineer, exec, and ops, the exec residue about measured rather than assumed portfolio numbers, the `evidence-acquisition` scenario kind with a roster-derived coverage gate, and the diagnosis resolving to both causes named here: an edit-shaped trigger and a missing stopping condition. **Superseded.** Two structural rules came out of reviewing that work: * A meld every role declares is an AGENTS.md rule, so bindings must be argued per role. * A meld needs a counterpart, because a shared body with no role holding the other side is the same rule with uneven emphasis. Together those define the primitive as one behavior removed from several roles and allocated to exactly one counterpart. Evidence acquisition as written fails that definition: it removes nothing and allocates nothing. Inverting it fixes that, and the inversion changes enough of this issue's premise that it belongs in a fresh ticket rather than another comment thread here. The successor covers the primitive rename to `boundary`, behavior-named slugs, the inverted third boundary, and the split of the local-grounding half out to AGENTS.md.
Author
Member

Successor filed as #254.

Successor filed as #254.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-flight-deck/agent-compose#248
No description provided.