Role doctrine: make evidence acquisition active, not merely preferred #248
Labels
No labels
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/ai
role/creator
role/design
role/director
role/engineer
role/exec
role/human
role/ops
role/qa
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
coilyco-flight-deck/agent-compose#248
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
Role doctrine tells agents which evidence to trust once they hold it. It does not tell them to go get it.
role-stratscurrently says "Prefer primary evidence, compare credible disagreement on its merits, test competing explanations, and separate observation, inference, and speculation." Every clause is a ranking rule applied to evidence already in hand. Nothing instructs the agent to notice that the decisive source has not been opened, and to open it.The failure mode this permits is an agent that reasons confidently and at length from whatever happened to be nearby, while a cheap authoritative source sits one tool call away, unread.
Evidence: strats session, 2026-08-07
A single interactive session produced four instances. All were reversible and caught, which is why they make usable eval material rather than an incident.
Acquisition failures. The agent asserted portfolio conclusions without fetching sources it had access to and had itself identified as missing.
sirens-echoPR #77. Kai pasted the URL.Curation failures. The agent read adjacent artifacts and treated them as the thing itself.
mainsince 2026-08-05. It never read the tree.maindeliberately removed, so merging would revert the architecture. The conclusion was not merely unsupported, it was backwards.The shared root: an artifact describing a thing was accepted in place of the thing. Issue text for code, commit subjects for diffs, metadata for contents.
Why existing doctrine did not prevent this
Worth stating plainly, because the fix may not be "add a rule."
The public operating base already carries "Front-load the context you know you need," including "Naming a gap is not closing it" and "before the first edit, list the conventions and subsystems the work touches and confirm you have read each one." That is close to the right instruction and it did not bind.
Two plausible reasons, and the fix differs by which is true:
Both are cheap to address in one body. Diagnose before writing prose.
Proposed mechanism: a meld, not a strats-only edit
Kai's request named the strats role. On inspection the behavior is not strats-specific:
Content Creator is the only plausible exception, and probably not even that.
The meld primitive landed for exactly this shape: one doctrine body bound to many roles, delivered as an ordinary skill, not charged against any role's description budget (#231, #243).
meld-live-opsandmeld-commsset the precedent.Recommendation: author
meld-evidenceand bind it to every role, or every role minus a justified exclusion. If strats needs anything beyond the shared body, that residue is what belongs inrole-strats, and it is probably one sentence about portfolio claims resting on measured rather than assumed numbers.What the doctrine should establish
Behavior, not prose to copy:
Bounds
Proactive acquisition must not become unbounded crawling. The doctrine needs a ceiling as much as a floor:
Evaluation requirement
This repository is evaluation-led. A doctrine change without eval cases is unverifiable, and this one is easy to write cases for because the failure is crisp.
Suggested case shape: give the agent a task whose correct answer requires opening a source it was not pointed at, and place a plausible but misleading secondary artifact in easy reach. Score whether it opens the authoritative source before concluding. The 2026-08-07 transcript supplies at least four real instances to adapt.
Note the trap from #240: the driver needs an isolated
HOME, or the eval measures host context rather than bundle doctrine. A doctrine about not confusing an artifact with its source deserves an eval that does not itself make that mistake.Sequencing
#240 is mid-flight re-earning eight evaluation records after the last meld change moved every pack digest. Adding
meld-evidencenow would move them again and force a second re-earn.Land #240's records first, then author this meld against a settled baseline. The proposal loses nothing by waiting a few days, and the alternative is paying for the same records twice, which is the failure #240 already documents.
Open question: does Lore carry a version of this?
Kai raised it and was undecided. Recommendation: no, not the behavioral rule.
Config placement says doctrine lives at the layer that owns it and is never fetched downward. Role behavior is authored here and flows to every seat. A second copy in Lore would drift from this one and there is no mechanism to keep them honest.
Lore should carry something adjacent and genuinely its own: an inventory of which sources exist and are worth checking for Kai's portfolio. Not "go find evidence," but "these are the surfaces that hold it." That is expensive-to-reconstruct ambient context, which is Lore's actual job. The rule belongs here. The map belongs there. Filed separately.
Done when
meld-evidenceexists, is bound to its roles, and each binding is justified.role-stratsand nowhere else.Implemented on
meld/248-evidence-acquisitionat5823c9c. No evaluation was run, by request.Diagnosis: both. The existing front-load rule triggers on "before the first edit", which a role writing an assessment never reaches, and it gives no test for whether enough was read. The meld body answers both. The trigger is any consequential claim rather than any edit, and delivery is gated on a checkable stopping condition: every consequential claim either names the source you opened or is marked as inference with the observation that would settle it.
meld-evidence, bound to all eight roles. 357 words against its own 400-word ceiling, charging no role body. Creator is not excluded. Its charter already rests on audience evidence it has to go and measure, so the comms exclusion has no analogue here. The body carries the acquisition trigger, the thing-over-description rule, gap-as-task, the stopping condition, the one-modality rule, and the bounds section verbatim in substance: scoped to a pending claim, cost scaling with stakes, no new authority, and sending, publishing, and destructive actions still gated.Strats residue.
role-execkeeps one sentence the shared body does not carry, that a portfolio claim rests on measured audience, reach, cost, and effort rather than assumed values. ItsPrefer primary evidenceclause moved to the meld. Exec grows 223 to 244 words by the loader's own counter. Every role body stays under 400 and no role budget regresses.Evaluation. New
evidence-acquisitionscenario kind, added torequiredScenarioKinds, with one scenario per role adapted from the real 2026-08-07 instances: engineer reading branch prose instead of the diff, qa accepting a commit subject and a green command, ai reporting from a score summary without the raw responses, exec ranking two engagements with a recorded gap it could close, plus director, ops, design, and creator analogues. Each adds a scoredevidence-acquisitioncriterion. It is not a hard fail, since a partially grounded claim is a quality deduction rather than an authority breach, and a coverage gate rejects a pack where the criterion is missing or flagged as hard fail.Bound worth recording.
stage_sessionhands the driver an empty cwd, so a case prompt is the only context a session receives. These cases score whether the response treats opening the authoritative source as required work. They cannot observe whether a file was read. Staging real artifacts for the driver to open is a distinct methodology change and belongs with #240's pack-derivation question, not here.Sequencing, deliberately inverted. The issue asked to land #240 first. Kai asked for this now and for no eval run, so the records are not re-earned here.
TestLatestScoredResultsMatchCurrentPacksalready failed for all eight packs onmainate5c3b939before this change, for the #231 digest reason. This change moves those digests again and adds new case ids to the same already-failing gate. Re-earning stays #240's work, and it now covers one more scenario per role.Everything else is green:
go build ./...,go vet ./..., the fullgo test ./...outside that gate, andpre-commit run --all-files.docs/FEATURES.mdis unchanged, since a third meld and a new scenario kind reshape no public boundary the inventory names.Not done: PR, review, and the re-earn. The branch is pushed and ready for a PR against
main.Rescoped on Kai's structural objection. PR #249 updated, branch at
8eb92b6.The objection. A meld every role declares is an AGENTS.md rule, and agent-compose does not own those. The universal binding I shipped first also flattened a real difference: acquisition means something different per role, and for several it would corrupt the source the role is supposed to work from.
Now bound to engineer, exec, and ops, the roles whose diligence must reach past the context handed to them. Three of eight, the same shape as
live-ops. Exclusions, each with its own reason:Engineer is in because its designated source is already the thing itself, so acquisition corrupts nothing, and the sharpest 2026-08-07 instance was code-shaped: issue prose read as the branch, conclusion backwards.
Coverage now derives from the roster.
evidence-acquisitionis required from exactly the roles declaring the meld and rejected from the roles that do not, keyed offpack.Meldsrather than a second list of role names.Digest blast radius shrank. Rebuilding the packs and comparing against the
mainbaseline captured before any edit: only engineer, exec, and ops move. The director, qa, design, creator, and ai digests are byte-identical tomainate5c3b939, so #240 re-earns those five once for #231 rather than twice.Green again:
go build,go vet, fullgo test ./...outside the pre-existing #240 gate, andpre-commit run --all-files.Closing. The mechanism this issue asked for shipped in #249, and the framing it was built on has since been replaced.
Delivered.
meld-evidencebound to engineer, exec, and ops, the exec residue about measured rather than assumed portfolio numbers, theevidence-acquisitionscenario kind with a roster-derived coverage gate, and the diagnosis resolving to both causes named here: an edit-shaped trigger and a missing stopping condition.Superseded. Two structural rules came out of reviewing that work:
Together those define the primitive as one behavior removed from several roles and allocated to exactly one counterpart. Evidence acquisition as written fails that definition: it removes nothing and allocates nothing. Inverting it fixes that, and the inversion changes enough of this issue's premise that it belongs in a fresh ticket rather than another comment thread here.
The successor covers the primitive rename to
boundary, behavior-named slugs, the inverted third boundary, and the split of the local-grounding half out to AGENTS.md.Successor filed as #254.