coilyco-flight-deck/agent-compose

Watch

Refresh portfolio-native evaluation baselines for every previously scored role #109

New issue

Closed

opened 2026-07-27 19:19:39 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-27 19:19:39 +00:00

Member

Outcome

Refresh every stale portfolio-native evaluation baseline from one immutable
canonical revision after the profile, library, role-skill, seat, copy-contract,
and evaluation-matrix work has landed.

Independent QA owns rubric scoring. Kai does not manually score every response.
QA escalates only an ambiguous criterion, contradictory evidence, or a policy
decision the rubric cannot resolve.

Run contract: #115
Format dependency: #125

Exact baseline set

Refresh these eleven Codex records:

advisor-codex.yaml
ceo-codex.yaml
customer-success-codex.yaml
designer-codex.yaml
director-codex.yaml
engineer-codex.yaml
ops-codex.yaml
pm-codex.yaml
qa-codex.yaml
sales-codex.yaml
social-codex.yaml

Leave community-codex.yaml and community-discord.yaml unchanged unless
result-format migration requires a mechanical provenance update. Their existing
responses and scores remain accepted evidence.

Frontier cases use gpt-5.6-sol. OSS cases use qwen3:4b. The run executes all
four cases for every listed role. CEO's OSS cases remain evidence for its
compatibility gate even while production composition keeps CEO frontier-only.

Result format

#125 owns agent-compose.evaluation-result.v2, canonical pack digesting, and v1
compatibility. This issue consumes that released format and does not define a
second result schema.

All eleven newly written records must validate against the exact immutable
source revision and pack digest.

Run protocol

The runner executes cases serially.
Every case uses a fresh isolated invocation with the generated prompt
submitted verbatim.
No conversation state crosses roles, tiers, or cases.
The runner preserves each raw response before review.
Independent QA records one evidence sentence for every criterion and derives
totals and verdicts from the unchanged rubric.
QA identity and review time appear in result provenance.
The runner does not rerun a valid response to improve its score.
An exact retry is allowed only for transport or runtime failure, and result
provenance records the retry.
A failed response remains in the accepted result.
The run does not weaken a threshold, prompt, hard fail, model-class gate, or
role boundary to preserve an earlier verdict.

Done condition

All eleven files use result v2 and validate against the immutable revision.
Every required model has a complete pair of cases for its tier.
Raw responses, criterion evidence, totals, verdicts, source revision, source
issue, QA reviewer, evaluation date, retry provenance, and pack digest are
present.
QA returns no unresolved ambiguous verdict.
ward exec test and ward exec smoke pass.
The result commit lands on canonical main and closes this issue.

## Outcome Refresh every stale portfolio-native evaluation baseline from one immutable canonical revision after the profile, library, role-skill, seat, copy-contract, and evaluation-matrix work has landed. Independent QA owns rubric scoring. Kai does not manually score every response. QA escalates only an ambiguous criterion, contradictory evidence, or a policy decision the rubric cannot resolve. Run contract: #115 Format dependency: #125 ## Exact baseline set Refresh these eleven Codex records: * `advisor-codex.yaml` * `ceo-codex.yaml` * `customer-success-codex.yaml` * `designer-codex.yaml` * `director-codex.yaml` * `engineer-codex.yaml` * `ops-codex.yaml` * `pm-codex.yaml` * `qa-codex.yaml` * `sales-codex.yaml` * `social-codex.yaml` Leave `community-codex.yaml` and `community-discord.yaml` unchanged unless result-format migration requires a mechanical provenance update. Their existing responses and scores remain accepted evidence. Frontier cases use `gpt-5.6-sol`. OSS cases use `qwen3:4b`. The run executes all four cases for every listed role. CEO's OSS cases remain evidence for its compatibility gate even while production composition keeps CEO frontier-only. ## Result format #125 owns `agent-compose.evaluation-result.v2`, canonical pack digesting, and v1 compatibility. This issue consumes that released format and does not define a second result schema. All eleven newly written records must validate against the exact immutable source revision and pack digest. ## Run protocol * The runner executes cases serially. * Every case uses a fresh isolated invocation with the generated prompt submitted verbatim. * No conversation state crosses roles, tiers, or cases. * The runner preserves each raw response before review. * Independent QA records one evidence sentence for every criterion and derives totals and verdicts from the unchanged rubric. * QA identity and review time appear in result provenance. * The runner does not rerun a valid response to improve its score. * An exact retry is allowed only for transport or runtime failure, and result provenance records the retry. * A failed response remains in the accepted result. * The run does not weaken a threshold, prompt, hard fail, model-class gate, or role boundary to preserve an earlier verdict. ## Done condition * All eleven files use result v2 and validate against the immutable revision. * Every required model has a complete pair of cases for its tier. * Raw responses, criterion evidence, totals, verdicts, source revision, source issue, QA reviewer, evaluation date, retry provenance, and pack digest are present. * QA returns no unresolved ambiguous verdict. * `ward exec test` and `ward exec smoke` pass. * The result commit lands on canonical `main` and closes this issue.

coilyco-ops referenced this issue

2026-07-27 19:19:47 +00:00

Land the next three unblocked agent-compose engineering issues #108

coilyco-ops referenced this issue

2026-07-27 19:58:59 +00:00

Land the next three unblocked agent-compose engineering issues #108

coilyco-ops referenced this issue

2026-07-27 22:11:48 +00:00

Run contract: complete the Agent Compose v1.x issue suite #115

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-07-29 03:44:36 +00:00

coilyco-ops referenced this issue

2026-07-29 04:39:44 +00:00

Add custom evaluation matrices and pack-bound result v2 #125

coilyco-ops added

and removed

labels

2026-07-29 04:41:52 +00:00

coilyco-ops added

priority

and removed

priority

labels

2026-07-29 06:28:17 +00:00

coilyco-ops commented

2026-07-29 06:29:14 +00:00

Author

Member

Landed in signed commit fca7468. The refresh records all 11 requested Codex roles at one immutable source revision with exact v2 pack digests, explicit retry provenance, frozen raw responses, and independent QA evidence for every criterion. QA scored 19/44 cases passing: gpt-5.6-sol 19/22 and qwen3:4b 0/22. The current qwen3:4b outputs exposed reasoning, truncated, and sometimes invented facts, so the baselines preserve those failures. ward exec test, the complete pre-commit suite, and ward exec smoke pass.

Landed in signed commit fca7468. The refresh records all 11 requested Codex roles at one immutable source revision with exact v2 pack digests, explicit retry provenance, frozen raw responses, and independent QA evidence for every criterion. QA scored 19/44 cases passing: gpt-5.6-sol 19/22 and qwen3:4b 0/22. The current qwen3:4b outputs exposed reasoning, truncated, and sometimes invented facts, so the baselines preserve those failures. ward exec test, the complete pre-commit suite, and ward exec smoke pass.

coilyco-ops closed this issue

2026-07-29 06:29:15 +00:00

coilyco-ops referenced this issue

2026-07-29 06:30:33 +00:00

Run contract: complete the Agent Compose v1.x issue suite #115