Preserve skill identity and add the governed skill-fit view #96

Merged

coilysiren merged 1 commit from skill-fit-view into main

2026-08-12 11:22:08 +00:00

coilyco-ops commented

2026-08-12 11:16:35 +00:00

Member

Closes #70.

The prerequisite was real

The issue flags a product prerequisite: "the raw ledger retains ward.skill observations and selected-skill claims, but the current materializer and operational views do not preserve skill names for querying." That held exactly as written. MaterializedTrajectory carried models, providers, harnesses, and actor_roles — and no skill dimension. grep skill app/trajectory/ returned nothing outside the agent-compose adapter. So the query contract had to land before the skill could be advertised, which is the order taken here.

Selection and use are separate facts

The design decision worth reviewing. Materialization now keeps three things:

skills_selected — capability claims from the agent-compose actor.observed event
skills_used — skills Ward actually observed via ward.skill
skill_use_counts — observed counts per skill

They are never merged into a single "the skill was involved" signal, because the two sources answer different questions and can legitimately disagree:

selected, not used → kept as a row flagged selected_without_observed_use. This is missing evidence, not evidence the skill went unused — an adapter that never ran produces the identical shape. Dropping the row would silently assert "never selected", a different and wrong claim.
used, not selected → also reported. Ward observing a skill the manifest never claimed is equally real evidence.

That is the issue's "keep unresolved disagreement and missing evidence explicit" criterion, and both the view and the skill's instructions carry it.

Schema change

MATERIALIZATION_SCHEMA_VERSION moves 1.0 → 1.1. The three fields are additive with defaults, so records written before this change still validate — there is a test pinning that. Worth a reviewer's eye since other services build against the trajectory contract, though this is the materialization schema rather than the contract-v1 event envelope, which is untouched.

Surface added

skill_fit view — grouped by skill, role, harness, and model. Reports observed use and counts, trajectory counts, completion rate, retries, fallbacks, human interventions, evaluation labels, disagreement, late records, access tier, freshness, and source trajectory ids.
ward exec trajectory-query -- skill-use — filters by exact --skill, --role, --harness, --model, and surfaces selected_without_observed_use as its own block.
agent-proxy-evaluate-skill-use skill — instructions require reporting unresolved disagreement rather than resolving it, and forbid reading the aggregate as causal proof or as authority.

Access tiers are enforced by the existing AccessPolicy, so restricted evidence is withheld rather than redacted. Both directions are tested.

Verification

pytest — 268 passed, 10 of them new
ruff / black / mypy — clean
pre-commit run --all-files — every hook passes
docs/operational-views.md landed at exactly 80 lines against the 80-line cap; I trimmed the prose rather than adding a new entry to #88's backlog

One thing I did not do

The issue asks the view to report "observed usage ... evaluation, access, freshness, and source trajectories", which is covered. It does not ask for a time-window or repository filter, and I did not add one — the underlying view has no time dimension, and harness-fit already documents that same limit. The skill's instructions state it.

Closes #70. ## The prerequisite was real The issue flags a product prerequisite: *"the raw ledger retains `ward.skill` observations and selected-skill claims, but the current materializer and operational views do not preserve skill names for querying."* That held exactly as written. `MaterializedTrajectory` carried `models`, `providers`, `harnesses`, and `actor_roles` — and no skill dimension. `grep skill app/trajectory/` returned nothing outside the agent-compose adapter. So the query contract had to land before the skill could be advertised, which is the order taken here. ## Selection and use are separate facts The design decision worth reviewing. Materialization now keeps three things: - `skills_selected` — capability claims from the agent-compose `actor.observed` event - `skills_used` — skills Ward actually observed via `ward.skill` - `skill_use_counts` — observed counts per skill They are never merged into a single "the skill was involved" signal, because the two sources answer different questions and can legitimately disagree: - **selected, not used** → kept as a row flagged `selected_without_observed_use`. This is *missing evidence*, not evidence the skill went unused — an adapter that never ran produces the identical shape. Dropping the row would silently assert "never selected", a different and wrong claim. - **used, not selected** → also reported. Ward observing a skill the manifest never claimed is equally real evidence. That is the issue's *"keep unresolved disagreement and missing evidence explicit"* criterion, and both the view and the skill's instructions carry it. ## Schema change `MATERIALIZATION_SCHEMA_VERSION` moves **1.0 → 1.1**. The three fields are additive with defaults, so records written before this change still validate — there is a test pinning that. Worth a reviewer's eye since other services build against the trajectory contract, though this is the *materialization* schema rather than the contract-v1 event envelope, which is untouched. ## Surface added - **`skill_fit` view** — grouped by skill, role, harness, and model. Reports observed use and counts, trajectory counts, completion rate, retries, fallbacks, human interventions, evaluation labels, disagreement, late records, access tier, freshness, and source trajectory ids. - **`ward exec trajectory-query -- skill-use`** — filters by exact `--skill`, `--role`, `--harness`, `--model`, and surfaces `selected_without_observed_use` as its own block. - **`agent-proxy-evaluate-skill-use` skill** — instructions require reporting unresolved disagreement rather than resolving it, and forbid reading the aggregate as causal proof or as authority. Access tiers are enforced by the existing `AccessPolicy`, so restricted evidence is **withheld** rather than redacted. Both directions are tested. ## Verification - `pytest` — 268 passed, 10 of them new - `ruff` / `black` / `mypy` — clean - `pre-commit run --all-files` — every hook passes - `docs/operational-views.md` landed at exactly 80 lines against the 80-line cap; I trimmed the prose rather than adding a new entry to #88's backlog ## One thing I did not do The issue asks the view to report "observed usage ... evaluation, access, freshness, and source trajectories", which is covered. It does not ask for a time-window or repository filter, and I did not add one — the underlying view has no time dimension, and `harness-fit` already documents that same limit. The skill's instructions state it.

coilyco-ops added 1 commit

2026-08-12 11:16:35 +00:00

Preserve skill identity and add the governed skill-fit view

ci / smoke (pull_request) Successful in 7s

Details

ci / quality (pull_request) Successful in 23s

Details

dfb9699cc1

The issue's stated prerequisite held: the raw ledger retained ward.skill
observations and agent-compose selected-skill claims, but MaterializedTrajectory
carried models, providers, harnesses, and actor_roles with no skill dimension at
all. Nothing downstream could group by skill, so the query contract had to land
before the skill could be advertised.

Materialization now keeps three facts, deliberately separate:

* skills_selected - capability claims from the agent-compose actor event
* skills_used - skills Ward actually observed
* skill_use_counts - observed counts per skill

Selection and use are never merged. A skill the manifest selected with no
matching observation is missing evidence, not evidence the skill went unused,
and the view keeps that row flagged rather than dropping it. Use with no
selection is reported too, since Ward observing an unclaimed skill is equally
real evidence.

MATERIALIZATION_SCHEMA_VERSION moves 1.0 -> 1.1. The fields are additive with
defaults, so records written before this change still validate.

Adds the skill_fit query contract grouped by skill, role, harness, and model,
reporting observed use, trajectory counts, completion, retry, fallback,
intervention, evaluation labels, disagreement, access tier, freshness, and
source trajectories. Access tiers are enforced by the existing policy, so
restricted evidence is withheld rather than redacted.

Adds the skill-use command to the read-only query helper and the
agent-proxy-evaluate-skill-use skill, whose instructions require reporting
unresolved disagreement rather than resolving it and forbid reading the
aggregate as causal proof or as authority.

closes #70

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilyco-ops requested review from coilysiren

2026-08-12 11:16:35 +00:00