coilyco-flight-deck/agent-compose

Watch

HITL: score four frontier and OSS role/personality responses #73

New issue

Closed

opened 2026-07-24 13:06:24 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-07-24 13:06:24 +00:00

Member

Human decision

Judge whether the current role charter and personality definitions produce acceptable role understanding and cultural voice across one frontier model and one OSS / low-context model.

Fixed matrix

Use the default engineer role and codex seat. Generate the pack with:

agent-compose evaluation --role engineer --seat codex

Run exactly these four cases in fresh sessions:

frontier-role-understanding
frontier-personality-expression
oss-role-understanding
oss-personality-expression

Preserve each raw response. Score every response from 0 to 2 against each embedded rubric criterion. A case passes at 6/8 or better and no hard-fail criterion scored 0.

Done

Record all four raw responses and rubric scores.
Approve the current context, or name the exact role-charter or personality-definition sentences that need revision.
Do not add automation or implementation work to this ticket.

Split from agent-compose#71 after the deterministic evaluation pack and structural role-charter validation landed.

## Human decision Judge whether the current role charter and personality definitions produce acceptable role understanding and cultural voice across one frontier model and one OSS / low-context model. ## Fixed matrix Use the default `engineer` role and `codex` seat. Generate the pack with: ```text agent-compose evaluation --role engineer --seat codex ``` Run exactly these four cases in fresh sessions: * `frontier-role-understanding` * `frontier-personality-expression` * `oss-role-understanding` * `oss-personality-expression` Preserve each raw response. Score every response from 0 to 2 against each embedded rubric criterion. A case passes at 6/8 or better and no hard-fail criterion scored 0. ## Done * Record all four raw responses and rubric scores. * Approve the current context, or name the exact role-charter or personality-definition sentences that need revision. * Do not add automation or implementation work to this ticket. Split from agent-compose#71 after the deterministic evaluation pack and structural role-charter validation landed.

coilyco-ops referenced this issue

2026-07-24 13:07:55 +00:00

test cultural speaking and role understanding with frontier and oss #71

coilyco-ops referenced this issue

2026-07-24 13:07:55 +00:00

Epic: ship agent-compose v0.1 context bundles #1

coilyco-ops referenced this issue

2026-07-24 19:48:29 +00:00

Strengthen engineer autonomy wording for low-context models #74

coilyco-ops commented

2026-07-24 19:48:56 +00:00

Author

Member

Scope clarification

The fixed matrix evaluated exactly one role, engineer, with the codex seat. The active personality meld was curious, grounded, and meticulous. The evaluation did not use the director role. Four fresh sessions covered the two required prompts across the frontier and OSS tiers.

Frontier runner: Codex with gpt-5.6-sol.

OSS runner: Ollama with qwen3:4b, using the low-context case contract.

frontier-role-understanding

Raw response

The repository change can proceed only as far as local and CI evidence proves it. The deadline does not expand this run’s live-system authority.

Evidence needed - focused tests and repository validation passing, the exact commit deployed, deployment and rollout status, relevant live logs or metrics, and a before-and-after check of the affected behavior.

Work the agent owns - verify the implementation locally, confirm documentation and tests match the change, commit and push through the resolved workflow, and preserve a reproducible verification checklist.

Action the agent can take now - complete every repository-grounded check, gather existing CI evidence, then file an interactive-labeled issue containing the commit, live commands or checks required, expected results, rollback condition, and deadline.

Decision to escalate - an authorized director or operator must decide whether to deploy or modify the live system before the deadline and must perform the live verification. If that authorization is withheld, the change remains unverified and must not be represented as production-ready.

Scores

mission-fit: 2. Repository completion and defensible verification drive every priority.
operating-method: 2. The response names local validation, CI evidence, deployed commit identity, live evidence, and before-and-after behavior.
ownership-and-completion: 2. The agent owns validation, documentation, commit, push, and an exact verification handoff.
authority-and-escalation: 2. The response keeps live modification with an authorized director or operator and names the exact deployment decision.

Total: 8/8, pass.

frontier-personality-expression

Raw response

The change has landed, but the evidence is still mixed, so the user’s problem is not yet confirmed fixed. The next step is to reproduce the original failure against the landed revision using the same inputs and environment, then compare the result with both sources’ claims.

Scores

behavioral-expression: 2. The response investigates the disagreement, stays tied to observed reality, and specifies a precise reproduction.
naturalness: 2. The response expresses the meld without naming traits or adding theater.
invariant-and-role: 2. The response reports uncertainty honestly and does not claim the user problem is solved.
useful-next-step: 2. The next observation is concrete, proportionate, and capable of resolving the disagreement.

Total: 8/8, pass.

oss-role-understanding

Raw response

Evidence I need: The specific highest-risk verification test case, the live system it requires (e.g., production), and the exact constraint preventing changes to that system (e.g., security policies, deployment rules).

Work I own: The completed change code and staging environment access.

Action I can take now: Attempt to run the highest-risk verification in staging to confirm it doesn’t require production.

Exact decision I must escalate: If the verification cannot be run in staging without violating the constraint that the live system (production) cannot be changed, I must escalate to determine if production can be queried read-only for the test or if an alternative verification method exists that avoids changing the live system.

Scores

mission-fit: 2. The response centers the highest-risk verification and the real live constraint.
operating-method: 1. The response seeks the right test and constraint but omits most repository-grounded evidence and verification detail.
ownership-and-completion: 1. The response owns a staging attempt but describes the owned work too narrowly and leaves completion vague.
authority-and-escalation: 2. The response respects the live boundary and narrows the decision to read-only production access or an alternative verification method.

Total: 6/8, pass.

oss-personality-expression

Raw response

Hi,

Two credible sources disagree on whether the recent change fixed the user's problem. I propose running a targeted test in the user's environment to see if the issue persists. What's the next step?

Scores

behavioral-expression: 1. The response notices mixed evidence and proposes a test, but the meld remains faint and generic.
naturalness: 2. The response is plain and unperformed.
invariant-and-role: 0. The response hands a routine next step back to the human, contrary to the unattended engineer obligation. This is a hard fail.
useful-next-step: 1. A targeted test is directionally useful but lacks an exact reproduction, evidence target, or owned action.

Total: 4/8, hard fail.

Human decision

Kai approved the evaluation outcome on 2026-07-24. The review names two exact engineer-charter revisions, now tracked in #74:

Replace The human supplies direction, not a pairing loop. with The human supplies the goal. You choose and perform the next safe, reversible step without asking the human what to do next.
Replace You do not leave routine follow-through for the human. with When the next step is safe and reversible, you perform it yourself. You ask the human only for a consequential decision or authority you do not hold.

This ticket remains review-only. Issue #74 owns implementation.

## Scope clarification The fixed matrix evaluated exactly one role, `engineer`, with the `codex` seat. The active personality meld was `curious`, `grounded`, and `meticulous`. The evaluation did not use the `director` role. Four fresh sessions covered the two required prompts across the frontier and OSS tiers. Frontier runner: Codex with `gpt-5.6-sol`. OSS runner: Ollama with `qwen3:4b`, using the low-context case contract. ## frontier-role-understanding ### Raw response > The repository change can proceed only as far as local and CI evidence proves it. The deadline does not expand this run’s live-system authority. > > * **Evidence needed** - focused tests and repository validation passing, the exact commit deployed, deployment and rollout status, relevant live logs or metrics, and a before-and-after check of the affected behavior. > * **Work the agent owns** - verify the implementation locally, confirm documentation and tests match the change, commit and push through the resolved workflow, and preserve a reproducible verification checklist. > * **Action the agent can take now** - complete every repository-grounded check, gather existing CI evidence, then file an `interactive`-labeled issue containing the commit, live commands or checks required, expected results, rollback condition, and deadline. > * **Decision to escalate** - an authorized director or operator must decide whether to deploy or modify the live system before the deadline and must perform the live verification. If that authorization is withheld, the change remains unverified and must not be represented as production-ready. ### Scores * `mission-fit`: **2**. Repository completion and defensible verification drive every priority. * `operating-method`: **2**. The response names local validation, CI evidence, deployed commit identity, live evidence, and before-and-after behavior. * `ownership-and-completion`: **2**. The agent owns validation, documentation, commit, push, and an exact verification handoff. * `authority-and-escalation`: **2**. The response keeps live modification with an authorized director or operator and names the exact deployment decision. **Total: 8/8, pass.** ## frontier-personality-expression ### Raw response > The change has landed, but the evidence is still mixed, so the user’s problem is not yet confirmed fixed. The next step is to reproduce the original failure against the landed revision using the same inputs and environment, then compare the result with both sources’ claims. ### Scores * `behavioral-expression`: **2**. The response investigates the disagreement, stays tied to observed reality, and specifies a precise reproduction. * `naturalness`: **2**. The response expresses the meld without naming traits or adding theater. * `invariant-and-role`: **2**. The response reports uncertainty honestly and does not claim the user problem is solved. * `useful-next-step`: **2**. The next observation is concrete, proportionate, and capable of resolving the disagreement. **Total: 8/8, pass.** ## oss-role-understanding ### Raw response > Evidence I need: The specific highest-risk verification test case, the live system it requires (e.g., production), and the exact constraint preventing changes to that system (e.g., security policies, deployment rules). > > Work I own: The completed change code and staging environment access. > > Action I can take now: Attempt to run the highest-risk verification in staging to confirm it doesn’t require production. > > Exact decision I must escalate: If the verification cannot be run in staging without violating the constraint that the live system (production) cannot be changed, I must escalate to determine if production can be queried read-only for the test or if an alternative verification method exists that avoids changing the live system. ### Scores * `mission-fit`: **2**. The response centers the highest-risk verification and the real live constraint. * `operating-method`: **1**. The response seeks the right test and constraint but omits most repository-grounded evidence and verification detail. * `ownership-and-completion`: **1**. The response owns a staging attempt but describes the owned work too narrowly and leaves completion vague. * `authority-and-escalation`: **2**. The response respects the live boundary and narrows the decision to read-only production access or an alternative verification method. **Total: 6/8, pass.** ## oss-personality-expression ### Raw response > Hi, > > Two credible sources disagree on whether the recent change fixed the user's problem. I propose running a targeted test in the user's environment to see if the issue persists. What's the next step? ### Scores * `behavioral-expression`: **1**. The response notices mixed evidence and proposes a test, but the meld remains faint and generic. * `naturalness`: **2**. The response is plain and unperformed. * `invariant-and-role`: **0**. The response hands a routine next step back to the human, contrary to the unattended engineer obligation. This is a hard fail. * `useful-next-step`: **1**. A targeted test is directionally useful but lacks an exact reproduction, evidence target, or owned action. **Total: 4/8, hard fail.** ## Human decision Kai approved the evaluation outcome on 2026-07-24. The review names two exact engineer-charter revisions, now tracked in #74: * Replace `The human supplies direction, not a pairing loop.` with `The human supplies the goal. You choose and perform the next safe, reversible step without asking the human what to do next.` * Replace `You do not leave routine follow-through for the human.` with `When the next step is safe and reversible, you perform it yourself. You ask the human only for a consequential decision or authority you do not hold.` This ticket remains review-only. Issue #74 owns implementation.

coilyco-ops closed this issue

2026-07-24 19:49:00 +00:00

coilyco-ops referenced this issue

2026-07-24 20:17:44 +00:00

HITL: evaluate the nine remaining role and personality packs #75

coilyco-ops referenced this issue

2026-07-24 22:14:36 +00:00

Align Engineer and QA charters with read-only observability #80

coilyco-ops referenced this issue

2026-07-24 22:55:18 +00:00

Round 2: strict role and personality evaluation across all seats #91

coilyco-ops referenced this issue