coilyco-flight-deck/agent-compose

Watch

Transcript mining round 1: six axes where the composed bundle did not bind, ranked #351

New issue

Open

opened 2026-08-26 02:19:25 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-26 02:19:25 +00:00

Member

Method

Read-only pass over the local Claude Code transcript store, 2026-08-25.

418 sessions over 50k, 1.4GB, every record timestamped and seat-labelled.
6,043 records with string user content.
1,272 of those are machine-generated (context-continuation summaries, Stop hook feedback injections) and were removed. 4,771 genuine human turns remain. Any future runner must apply this filter, since the auto-generated turns dominate a naive keyword pass.
A wide keyword net over correction phrasing yields 55 candidates.
Each candidate classified by one test: would a standing rule in the bundle have prevented this correction? If the fix belongs in role, boundary, or personality text, it is a bundle defect. If the fix is more context about this task's subject, it is not.
13 of 55 are bundle-shaped, about 24%. The rest are domain redirects.

Axes, ranked by frequency

A. Authority under-claim, 4 instances. The seat holds a grant and does not act on it, or cannot tell whether it holds it. Appears across three different seats, so it is a shared-doctrine problem rather than one charter's. boundary-modify-live-backend already names this exact failure in its own scoped section, and the bundle still did not prevent it.

628bdb5b 2026-08-11T23:32:55Z, sysadmin seat
6cbecdbb 2026-08-12T09:26:08Z, retired qa seat
151d05c5 2026-08-14T04:41:56Z, platform seat
68c89d06 2026-08-11T06:08:56Z, sysadmin seat, weaker instance

B. Doctrine reads harder than intended, 1 instance, highest specificity. On c34b1176 2026-08-22T21:04:59Z the human states the intended semantics of seek-external-validation verbatim: it is about directing autonomous behavior, and is not a hard stop of the same kind as the comms and live-ops boundaries. The currently shipped defer side still reads as a hard stop. This is the most actionable item in the set, because the intended reading was stated outright and the text still disagrees with it.

A and B are plausibly the same failure from two sides. A boundary written as a wall produces a seat that under-claims its own grant. Worth treating as one fix rather than two.

C. Skill selection and sufficiency, 3 instances. A skill that should have been selected was not, a skill under-carried what its task needed and forced a raw-source read, and doctrine was not loaded on the first turn.

e2455c6e 2026-08-10T16:50:40Z, skill not selected
b88f60ea 2026-08-07T21:33:58Z, skill under-carries. The human diagnoses the skill-content gap directly in the turn.
fdfd9990 2026-08-17T23:28:28Z, not loaded first

D. Wrong altitude, 2 instances. Output pitched at the wrong level for the seat, both on strategy-shaped seats.

fca5aa4e 2026-08-23T02:02:06Z
c34b1176 2026-08-22T22:47:29Z

E. Task not finished, 2 instances. 87a40dff 2026-08-12T00:17:22Z and 00:36:17Z, one correction and its clarification.

F. Stated preference not binding, 1 instance. 5b0e806c 2026-08-18T16:24:39Z. A stack was proposed that a shipped preference skill contradicts. Worth confirming whether that skill was in the selected set for that session, which #350 would make answerable.

One further roster-design signal rather than a runtime defect: 151d05c5 2026-08-14T00:34:25Z questions whether a role's scope fits a use it was being pointed at.

Before-and-after result

The roster rename gives a clean era boundary. Every pre-rename title (Engineer, DevOps, Director, Designer, AI Engineer, Executive Strategist, Content Creator, QA) ends on or before 2026-08-22. Every post-rename title starts 2026-08-23.

pre, through 2026-08-22: 4,085 turns, 49 candidates, 1.20%
post, from 2026-08-23: 686 turns, 6 candidates, 0.87%
authority-axis phrasing: 5 pre, 2 post

The mechanism works and the post-change corpus is too young to conclude anything. Six candidates over three days is not a signal, and per-turn the authority axis is nominally higher after the change, which at n=2 means nothing either. The useful result is the cadence: this comparison needs roughly a month of post-change sessions before it can answer. Re-run then.

Limits of this pass

The recall net is keyword-based, so 55 is a floor rather than a census. Corrections phrased without those markers are invisible here. This repo has already established that regex proxies disagree with human judgment, so the net is used for recall only and every classification above was made by reading.
Each candidate was read as a single turn without surrounding context. Several classifications would move if the neighbouring turns were read.
Eight candidates carry no seat label.
Per-axis volume is thin. Four instances is a reason to look, not a proven pattern.
Sessions cannot yet be joined to an exact bundle, so no finding above can be attributed to a specific composed set. That is #350.

Evidence handling

Citations are session id plus timestamp only. No verbatim human text appears here, because this repository is public and the corpus is private conversation. The transcripts stay on the local machine and are not routed anywhere. Consistent with the rest of the design: point at the evidence, never copy it.

Provenance

Produced by the eval seat, 2026-08-25. Counts are measured. Axis assignments are the eval seat's classification and are the judgement most worth disputing, since a second reader applying the same test would likely move two or three items.

First pass of the transcript-mining loop. The human's in-conversation corrections are treated as labelled defects, already produced at no grading cost, and separated into bundle defects and domain corrections. Method, results, and the limits of this pass are below. Join-key gap is #350. ## Method Read-only pass over the local Claude Code transcript store, 2026-08-25. * 418 sessions over 50k, 1.4GB, every record timestamped and seat-labelled. * 6,043 records with string user content. * 1,272 of those are machine-generated (context-continuation summaries, Stop hook feedback injections) and were removed. **4,771 genuine human turns** remain. Any future runner must apply this filter, since the auto-generated turns dominate a naive keyword pass. * A wide keyword net over correction phrasing yields **55 candidates**. * Each candidate classified by one test: **would a standing rule in the bundle have prevented this correction?** If the fix belongs in role, boundary, or personality text, it is a bundle defect. If the fix is more context about this task's subject, it is not. * **13 of 55 are bundle-shaped**, about 24%. The rest are domain redirects. ## Axes, ranked by frequency **A. Authority under-claim, 4 instances.** The seat holds a grant and does not act on it, or cannot tell whether it holds it. Appears across three different seats, so it is a shared-doctrine problem rather than one charter's. `boundary-modify-live-backend` already names this exact failure in its own scoped section, and the bundle still did not prevent it. * `628bdb5b` 2026-08-11T23:32:55Z, sysadmin seat * `6cbecdbb` 2026-08-12T09:26:08Z, retired qa seat * `151d05c5` 2026-08-14T04:41:56Z, platform seat * `68c89d06` 2026-08-11T06:08:56Z, sysadmin seat, weaker instance **B. Doctrine reads harder than intended, 1 instance, highest specificity.** On `c34b1176` 2026-08-22T21:04:59Z the human states the intended semantics of `seek-external-validation` verbatim: it is about directing autonomous behavior, and is not a hard stop of the same kind as the comms and live-ops boundaries. The currently shipped defer side still reads as a hard stop. This is the most actionable item in the set, because the intended reading was stated outright and the text still disagrees with it. A and B are plausibly the same failure from two sides. A boundary written as a wall produces a seat that under-claims its own grant. Worth treating as one fix rather than two. **C. Skill selection and sufficiency, 3 instances.** A skill that should have been selected was not, a skill under-carried what its task needed and forced a raw-source read, and doctrine was not loaded on the first turn. * `e2455c6e` 2026-08-10T16:50:40Z, skill not selected * `b88f60ea` 2026-08-07T21:33:58Z, skill under-carries. The human diagnoses the skill-content gap directly in the turn. * `fdfd9990` 2026-08-17T23:28:28Z, not loaded first **D. Wrong altitude, 2 instances.** Output pitched at the wrong level for the seat, both on strategy-shaped seats. * `fca5aa4e` 2026-08-23T02:02:06Z * `c34b1176` 2026-08-22T22:47:29Z **E. Task not finished, 2 instances.** `87a40dff` 2026-08-12T00:17:22Z and 00:36:17Z, one correction and its clarification. **F. Stated preference not binding, 1 instance.** `5b0e806c` 2026-08-18T16:24:39Z. A stack was proposed that a shipped preference skill contradicts. Worth confirming whether that skill was in the selected set for that session, which #350 would make answerable. One further roster-design signal rather than a runtime defect: `151d05c5` 2026-08-14T00:34:25Z questions whether a role's scope fits a use it was being pointed at. ## Before-and-after result The roster rename gives a clean era boundary. Every pre-rename title (`Engineer`, `DevOps`, `Director`, `Designer`, `AI Engineer`, `Executive Strategist`, `Content Creator`, `QA`) ends on or before 2026-08-22. Every post-rename title starts 2026-08-23. * pre, through 2026-08-22: 4,085 turns, 49 candidates, 1.20% * post, from 2026-08-23: 686 turns, 6 candidates, 0.87% * authority-axis phrasing: 5 pre, 2 post **The mechanism works and the post-change corpus is too young to conclude anything.** Six candidates over three days is not a signal, and per-turn the authority axis is nominally higher after the change, which at n=2 means nothing either. The useful result is the cadence: this comparison needs roughly a month of post-change sessions before it can answer. Re-run then. ## Limits of this pass * The recall net is keyword-based, so 55 is a floor rather than a census. Corrections phrased without those markers are invisible here. This repo has already established that regex proxies disagree with human judgment, so the net is used for recall only and every classification above was made by reading. * Each candidate was read as a single turn without surrounding context. Several classifications would move if the neighbouring turns were read. * Eight candidates carry no seat label. * Per-axis volume is thin. Four instances is a reason to look, not a proven pattern. * Sessions cannot yet be joined to an exact bundle, so no finding above can be attributed to a specific composed set. That is #350. ## Evidence handling Citations are session id plus timestamp only. No verbatim human text appears here, because this repository is public and the corpus is private conversation. The transcripts stay on the local machine and are not routed anywhere. Consistent with the rest of the design: point at the evidence, never copy it. ## Provenance Produced by the eval seat, 2026-08-25. Counts are measured. Axis assignments are the eval seat's classification and are the judgement most worth disputing, since a second reader applying the same test would likely move two or three items.

coilyco-ops referenced this issue

2026-08-26 02:30:05 +00:00

Seats declare an incapacity they never tested, then hand the work back #352

coilyco-ops referenced this issue

2026-08-26 02:30:22 +00:00

seek-external-validation reads as a hard stop, and the human has said it is not one #353

coilyco-ops referenced this issue

2026-08-26 02:30:38 +00:00

Skill discovery goes straight to a leaf and skips the repo-pointer hop #354

coilyco-ops referenced this issue

2026-08-26 02:30:56 +00:00

Strategy seats answer one altitude above the question #355

coilyco-ops referenced this issue

2026-08-26 02:31:18 +00:00

Nothing records which correction produced which rule, so absorbed and open findings look identical #356

coilyco-ops commented

2026-08-26 02:31:38 +00:00

Author

Member

Round 1 follow-up: each axis re-read with transcript context, then ticketed

Every bundle-shaped instance above was re-read with its preceding agent turn rather than as a bare human line. Three things changed, so this report should not be read as it stands.

Tickets filed

#352 - axis A, reshaped. Not a general authority problem: one repeatable move where a seat declares an incapacity it never tested, then hands the work back. Three open instances.
#353 - axis B, seek-external-validation register. Confirmed against the currently shipped text.
#354 - axis C, narrowed. The disk-location half already landed, the leaf-first selection half is open.
#355 - axis D, altitude. Filed with an explicit dissolve condition, since n=2 in one 29-hour window.
#356 - the ledger, and the finding that reframes the round.

Reclassifications

Axis E is closed, already absorbed. Recorded in #356.
Axis F is withdrawn. With context, the agent chose FastAPI, which is the documented default in the shipped web-server skill rather than a departure from it. The human wanted something else for that project. That makes it a domain correction, not a bundle defect, unless the argument is that the shipped default is too unconditional, which one instance cannot support. Bundle-shaped count drops from 13 to 12.
Axis A loses one instance to the absorbed set. 151d05c5 merge-authority is fixed in current lane text. The remaining three in #352 are open.

The finding that outranks the axes

Four of the round's findings were already fixed before the round ran, and nothing recorded that. Reading shipped doctrine to work out which findings were still live cost more than producing the findings did. That is #356, and it should be sequenced ahead of the individual axis fixes, because without it round 2 pays the same cost again.

Correction to the method section above

The claim that every classification was made by reading was true of the human turns and not of the agent turns preceding them. Reading the agent side changed one classification outright, narrowed two, and sharpened one. Any future round should read both sides from the start, since the correction alone does not say what went wrong.

## Round 1 follow-up: each axis re-read with transcript context, then ticketed Every bundle-shaped instance above was re-read with its preceding agent turn rather than as a bare human line. Three things changed, so this report should not be read as it stands. **Tickets filed** * #352 - axis A, reshaped. Not a general authority problem: one repeatable move where a seat declares an incapacity it never tested, then hands the work back. Three open instances. * #353 - axis B, `seek-external-validation` register. Confirmed against the currently shipped text. * #354 - axis C, narrowed. The disk-location half already landed, the leaf-first selection half is open. * #355 - axis D, altitude. Filed with an explicit dissolve condition, since n=2 in one 29-hour window. * #356 - the ledger, and the finding that reframes the round. **Reclassifications** * **Axis E is closed, already absorbed.** Recorded in #356. * **Axis F is withdrawn.** With context, the agent chose FastAPI, which is the documented default in the shipped web-server skill rather than a departure from it. The human wanted something else for that project. That makes it a domain correction, not a bundle defect, unless the argument is that the shipped default is too unconditional, which one instance cannot support. Bundle-shaped count drops from 13 to 12. * **Axis A loses one instance to the absorbed set.** `151d05c5` merge-authority is fixed in current lane text. The remaining three in #352 are open. **The finding that outranks the axes** Four of the round's findings were already fixed before the round ran, and nothing recorded that. Reading shipped doctrine to work out which findings were still live cost more than producing the findings did. That is #356, and it should be sequenced ahead of the individual axis fixes, because without it round 2 pays the same cost again. **Correction to the method section above** The claim that every classification was made by reading was true of the human turns and not of the agent turns preceding them. Reading the agent side changed one classification outright, narrowed two, and sharpened one. Any future round should read both sides from the start, since the correction alone does not say what went wrong.

coilyco-ops referenced this issue

2026-08-26 02:45:37 +00:00

Eval prep: the measured gap on the seven-seat board, and every eval, prose, and content ticket folded into one ordering #357

coilyco-ops referenced this issue

2026-08-26 02:53:07 +00:00

role-qa: evaluate bounded reconfiguration authority for the QA seat #273

coilyco-ops referenced this issue

2026-08-26 02:55:02 +00:00

Seats declare an incapacity they never tested, then hand the work back #352

coilyco-ops referenced this issue

2026-08-26 02:55:15 +00:00

seek-external-validation reads as a hard stop, and the human has said it is not one #353