coilyco-flight-deck/agentic-os

Watch

Add behavioral evaluation to skill authoring #671

New issue

Closed

opened 2026-07-24 00:10:08 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-07-24 00:10:08 +00:00

Member

workflow: pull-request-and-merge

Source: https://github.com/anthropics/skills/tree/main/skills/skill-creator
License: Apache-2.0

Goal

AOS extends tooling-skill-authoring with a provider-neutral evaluation loop for behavior and trigger quality. The existing admission and structural validation remain the base.

Acceptance

The authoring workflow generates realistic should-trigger and should-not-trigger prompts for the skill description.
The agent compares representative outputs with and without the skill when the claimed capability delta is behavioral.
A fresh-context grader checks objective assertions. A blind comparison is available for subjective output quality when it will change the decision.
The workflow reports false triggers, missed triggers, task regressions, and the before-and-after description.
The workflow keeps model calls optional and budget-bounded. Deterministic validators remain the required structural gate.
The tooling-skill-authoring frontmatter carries this source URL and the Apache-2.0 license once adapted content lands.
AOS adds no centralized provenance registry or automatic upstream sync.

Related: #646 defined admission and placement. This issue adds measured behavior after admission.

workflow: pull-request-and-merge Source: https://github.com/anthropics/skills/tree/main/skills/skill-creator License: Apache-2.0 ## Goal AOS extends tooling-skill-authoring with a provider-neutral evaluation loop for behavior and trigger quality. The existing admission and structural validation remain the base. ## Acceptance * The authoring workflow generates realistic should-trigger and should-not-trigger prompts for the skill description. * The agent compares representative outputs with and without the skill when the claimed capability delta is behavioral. * A fresh-context grader checks objective assertions. A blind comparison is available for subjective output quality when it will change the decision. * The workflow reports false triggers, missed triggers, task regressions, and the before-and-after description. * The workflow keeps model calls optional and budget-bounded. Deterministic validators remain the required structural gate. * The tooling-skill-authoring frontmatter carries this source URL and the Apache-2.0 license once adapted content lands. * AOS adds no centralized provenance registry or automatic upstream sync. Related: #646 defined admission and placement. This issue adds measured behavior after admission.

coilyco-ops added the

autonomy

headless

priority

labels

2026-07-29 03:44:51 +00:00

coilyco-ops added

priority

and removed

priority

labels

2026-07-29 06:19:33 +00:00

coilyco-ops referenced this issue from a commit

2026-07-29 17:11:53 +00:00

feat(skills): add behavioral authoring evaluation

coilyco-ops closed this issue

2026-07-29 17:12:46 +00:00

coilyco-ops commented

2026-07-29 17:14:36 +00:00

Author

Member

WARD-WORKFLOW: done ✅

details

review: in-container gate intentionally skipped. Implementation and QA are role-separated. QA is a separate, opt-in, exact-commit verification role bound to 8d09e179.

workflow: merge-remote-main; review summary: skipped, role-separated exact-commit QA

felt: clean, compact adaptation. A fresh-context probe caught one grader-budget ambiguity.
confidence: high.
surprises: main advanced once and merged cleanly.
follow-ups: optional QA at 8d09e179.
landed: 8d09e179, merged on main as d369f7e7.

WARD-WORKFLOW: done ✅ <details><summary>details</summary> review: in-container gate intentionally skipped. Implementation and QA are role-separated. QA is a separate, opt-in, exact-commit verification role bound to `8d09e179`. workflow: merge-remote-main; review summary: skipped, role-separated exact-commit QA felt: clean, compact adaptation. A fresh-context probe caught one grader-budget ambiguity. confidence: high. surprises: `main` advanced once and merged cleanly. follow-ups: optional QA at `8d09e179`. landed: `8d09e179`, merged on `main` as `d369f7e7`. </details>

coilyco-ops commented

2026-07-29 17:14:50 +00:00

Author

Member

WARD-WORKFLOW: reservation-released

release details

Run finished with WARD-WORKFLOW: done ✅.

ward container reap released container engineer-codex-agentic-os-671 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARD-WORKFLOW: done ✅`. `ward container reap` released container `engineer-codex-agentic-os-671` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#671

No description provided.

Rows
Columns