Add custom evaluation matrices and pack-bound result v2 #125

New issue

Closed

opened 2026-07-29 04:39:44 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-29 04:39:44 +00:00

Member

Outcome

Support complete profile-owned evaluation matrices and bind every newly written
evaluation result to the exact canonical pack it scored.

Parent: #114
Run contract: #115

This issue is the sole implementation owner of
agent-compose.evaluation-result.v2. Issue #109 consumes the format when it
refreshes stored baselines.

Evaluation matrices

The engine ships one complete generic fallback matrix.
A profile may declare one complete custom matrix for a role.
A custom matrix replaces the generic matrix as one unit. The loader does not
merge individual fields.
A loaded role cannot silently opt out of evaluation.
Custom matrices may declare arbitrary named lanes, stable seat keys, cases,
prompts, bundle model classes, rubric references, and review metadata.

Result format v2

New results record a deterministic SHA-256 digest of the exact canonical
evaluation pack.
The digest covers selected profile, role, seat, role skill, invariant, active
personality definitions, copy contract, protocol, review rule, prompts, and
rubrics after canonical encoding.
Validation rebuilds the pack and rejects a mismatched digest before checking
scores.
The decoder continues reading v1 records throughout v1.x. New writes use v2.

Dependencies

#110 supplies engine and profile-owned evaluation assets.
#111 supplies profile and library provenance.
The generalized-seat slice supplies arbitrary evaluation seat keys.
The copy-contract slice supplies projected vocabulary criteria.

Acceptance

Generic fallback and custom replacement matrices both validate.
Tests cover arbitrary lanes, custom roles, non-harness seats, missing
references, malformed matrices, digest drift, and v1 compatibility.
Prose-only role or personality changes invalidate the pack digest.
Generated packs and v2 records are deterministic.
ward exec test and ward exec smoke pass.
The implementation lands on canonical main and closes this issue.

## Outcome Support complete profile-owned evaluation matrices and bind every newly written evaluation result to the exact canonical pack it scored. Parent: #114 Run contract: #115 This issue is the sole implementation owner of `agent-compose.evaluation-result.v2`. Issue #109 consumes the format when it refreshes stored baselines. ## Evaluation matrices * The engine ships one complete generic fallback matrix. * A profile may declare one complete custom matrix for a role. * A custom matrix replaces the generic matrix as one unit. The loader does not merge individual fields. * A loaded role cannot silently opt out of evaluation. * Custom matrices may declare arbitrary named lanes, stable seat keys, cases, prompts, bundle model classes, rubric references, and review metadata. ## Result format v2 * New results record a deterministic SHA-256 digest of the exact canonical evaluation pack. * The digest covers selected profile, role, seat, role skill, invariant, active personality definitions, copy contract, protocol, review rule, prompts, and rubrics after canonical encoding. * Validation rebuilds the pack and rejects a mismatched digest before checking scores. * The decoder continues reading v1 records throughout v1.x. New writes use v2. ## Dependencies * #110 supplies engine and profile-owned evaluation assets. * #111 supplies profile and library provenance. * The generalized-seat slice supplies arbitrary evaluation seat keys. * The copy-contract slice supplies projected vocabulary criteria. ## Acceptance * Generic fallback and custom replacement matrices both validate. * Tests cover arbitrary lanes, custom roles, non-harness seats, missing references, malformed matrices, digest drift, and v1 compatibility. * Prose-only role or personality changes invalidate the pack digest. * Generated packs and v2 records are deterministic. * `ward exec test` and `ward exec smoke` pass. * The implementation lands on canonical `main` and closes this issue.

coilyco-ops referenced this issue

2026-07-29 04:41:06 +00:00

Refresh portfolio-native evaluation baselines for every previously scored role #109

coilyco-ops referenced this issue

2026-07-29 04:41:07 +00:00

Adapt persona system to true use case #114

coilyco-ops referenced this issue

2026-07-29 04:41:08 +00:00

Run contract: complete the Agent Compose v1.x issue suite #115

coilyco-ops added the

priority

autonomy

headless

labels

2026-07-29 04:42:06 +00:00

coilyco-ops commented

2026-07-29 05:24:47 +00:00

Author

Member

Landed on canonical main through daa9996, completing the evaluation slice begun by earlier commits. Complete custom matrices now replace generic run protocol, review rule, and arbitrary cases as one unit. Roles without one retain the generic fallback, while legacy prompt-only assets remain readable. New result writes select evaluation-result.v2 with a deterministic full-pack digest. Tests prove v1 compatibility and rejection after role or personality prose drift. ward exec test and ward exec smoke passed.

Landed on canonical main through daa9996, completing the evaluation slice begun by earlier commits. Complete custom matrices now replace generic run protocol, review rule, and arbitrary cases as one unit. Roles without one retain the generic fallback, while legacy prompt-only assets remain readable. New result writes select evaluation-result.v2 with a deterministic full-pack digest. Tests prove v1 compatibility and rejection after role or personality prose drift. ward exec test and ward exec smoke passed.

coilyco-ops closed this issue

2026-07-29 05:24:52 +00:00

coilyco-ops referenced this issue

2026-07-29 06:30:33 +00:00

Run contract: complete the Agent Compose v1.x issue suite #115