feat: add agent tool evaluation skill #787

Merged

coilyco-ops merged 2 commits from issue-670 into main 2026-07-29 17:27:40 +00:00

Conversation 2 Commits 2 Files changed 5 +266 -14

coilyco-ops commented 2026-07-29 17:01:33 +00:00

Owner

Copy link

ward.workflow: pull-request-and-merge

Adds a provider-neutral cross-harness method for evaluating agent tool usability. Tool definitions, evaluation tasks, expected outcomes, and observed runs remain separated, with focused useful, ambiguous, and avoidable-retry fixtures.

Validation:

• ward pre-commit-all
• ward test -- --ignore=tests/test_aosguard.py (593 passed, 1 skipped on current main)
• Full pre-merge ward test reached 594 passed and 1 skipped. Eight aosguard cases could not start because this container lacks specgen.

Review summary: skipped. The in-container review gate was intentionally skipped because implementation and QA are role-separated. QA remains a separate opt-in exact-commit verification role.

closes coilyco-flight-deck/agentic-os#670

ward.workflow: pull-request-and-merge Adds a provider-neutral cross-harness method for evaluating agent tool usability. Tool definitions, evaluation tasks, expected outcomes, and observed runs remain separated, with focused useful, ambiguous, and avoidable-retry fixtures. Validation: * ward pre-commit-all * ward test -- --ignore=tests/test_aosguard.py (593 passed, 1 skipped on current main) * Full pre-merge ward test reached 594 passed and 1 skipped. Eight aosguard cases could not start because this container lacks specgen. Review summary: skipped. The in-container review gate was intentionally skipped because implementation and QA are role-separated. QA remains a separate opt-in exact-commit verification role. closes coilyco-flight-deck/agentic-os#670

coilyco-ops added 1 commit 2026-07-29 17:01:33 +00:00

feat: add agent tool evaluation skill ... 5eb8b420f4

Ship a provider-neutral cross-harness method with separated definitions, tasks, and observed-run fixtures for usability decisions.

closes coilyco-flight-deck/agentic-os#670

Co-authored-by: Codex <noreply@openai.com>

coilyco-ops requested review from coilysiren 2026-07-29 17:01:33 +00:00

coilyco-ops commented 2026-07-29 17:02:35 +00:00

Author

Owner

Copy link

WARD-WORKFLOW: blocked 🛑

details

manual close/update snapshot at 2026-07-29T17:01:33Z is newer than the latest ward acknowledgement at 2026-07-29T16:50:12Z

This action is blocked until the feedback is visibly acknowledged.

— Claude (she/her), via ward agent

WARD-WORKFLOW: blocked 🛑 <details><summary>details</summary> manual close/update snapshot at 2026-07-29T17:01:33Z is newer than the latest ward acknowledgement at 2026-07-29T16:50:12Z This action is blocked until the feedback is visibly acknowledged. </details> <!-- ward-agent-signature --> — Claude (she/her), via `ward agent`

coilyco-ops added 1 commit 2026-07-29 17:23:34 +00:00

Merge remote-tracking branch 'origin/main' into issue-670 ... b52f496352

# Conflicts:
#	docs/FEATURES.md

coilyco-ops commented 2026-07-29 17:27:03 +00:00

Author

Owner

Copy link

WARD-WORKFLOW: blocked 🛑

details

manual close/update snapshot at 2026-07-29T17:26:58Z is newer than the latest ward acknowledgement at 2026-07-29T17:02:35Z

This action is blocked until the feedback is visibly acknowledged.

— Codex, via ward agent

WARD-WORKFLOW: blocked 🛑 <details><summary>details</summary> manual close/update snapshot at 2026-07-29T17:26:58Z is newer than the latest ward acknowledgement at 2026-07-29T17:02:35Z This action is blocked until the feedback is visibly acknowledged. </details> <!-- ward-agent-signature --> — Codex, via `ward agent`

coilyco-ops merged commit 3d828a89a7 into main 2026-07-29 17:27:40 +00:00

coilyco-ops deleted branch issue-670 2026-07-29 17:27:40 +00:00

coilyco-ops referenced this pull request from a commit 2026-07-29 17:27:40 +00:00

Merge pull request 'feat: add agent tool evaluation skill' (#787) from issue-670 into main

coilyco-ops referenced this pull request 2026-07-29 17:28:30 +00:00

Add cross-harness agent tool evaluation skill #670

Sign in to join this conversation.

Reviewers

No reviewers

coilysiren

Labels

Clear labels   

burndown-2026-06

Backlog burndown June 2026

  

burndown-2026-08

Closed in the 2026-08-26 backlog burn-down. Reopen freely: state:closed label:burndown-2026-08 recovers the whole set.

  

autonomy

async-consult

A human needs to consult on the issue to upgrade it to headless

  

autonomy

epic

This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  

autonomy

headless

The agent can perform the work on its own

  

autonomy

live-collab

The agent and the human need to work together in realtime

  

coherence-core

Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  

priority

P0

priority tier

  

priority

P1

priority tier

  

priority

P2

priority tier

  

priority

P3

priority tier

  

priority

P4

priority tier

  

qa-fixture

Disposable issue admitted to the bounded Ward QA verification lane.

  

role/advocate

requires work from the Developer Advocate seat

  

role/director

requires work from the Portfolio Director seat

  

role/exec

requires work from the exec role

  

role/frontend

requires work from the Frontend Engineer seat

  

role/gamedev

requires work from the Game Developer seat

  

role/human

requires a person, and specifically not an agent seat

  

role/platform

requires work from the Platform Engineer seat

  

role/qa

requires work from the QA role

  

role/science

requires work from the Applied Scientist seat

  

role/sysadmin

requires work from the Systems Administrator seat

  

state

ambient

ambient and ephemeral work, held as a maintained document rather than a queue

No labels

burndown-2026-06

burndown-2026-08

autonomy

async-consult

autonomy

epic

autonomy

headless

autonomy

live-collab

coherence-core

priority

P0

priority

P1

priority

P2

priority

P3

priority

P4

qa-fixture

role/advocate

role/director

role/exec

role/frontend

role/gamedev

role/human

role/platform

role/qa

role/science

role/sysadmin

state

ambient

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os!787

No description provided.