coilyco-flight-deck/agentic-os

Watch

Add cross-harness agent tool evaluation skill #670

New issue

Closed

opened 2026-07-24 00:09:53 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-24 00:09:53 +00:00

Owner

workflow: pull-request-and-merge

Source: https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/use-cases/tools-evaluation.ipynb
License: MIT
Corroborating first-party pattern: https://github.com/anthropics/claude-cookbooks/blob/main/tool_evaluation/tool_evaluation.ipynb

Goal

AOS ships a provider-neutral ordinary skill for evaluating whether an agent tool is understandable and useful in practice. The skill snapshots the small reusable method, not either provider API implementation.

Acceptance

The skill keeps evaluation tasks and expected outcomes separate from tool definitions.
The agent runs representative tasks and records correctness, duration, tool-call count, failures, and actionable feedback.
The evaluation checks tool names, descriptions, parameter clarity, required versus optional inputs, error guidance, and oversized results.
The agent compares the tool against a baseline or prior definition when the comparison can change a decision.
The skill frontmatter carries this source URL and the MIT license.
AOS adds no centralized provenance registry or automatic upstream sync.
Focused fixtures demonstrate a useful tool, an ambiguous tool, and a tool whose result is correct only after avoidable retries.

workflow: pull-request-and-merge Source: https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/use-cases/tools-evaluation.ipynb License: MIT Corroborating first-party pattern: https://github.com/anthropics/claude-cookbooks/blob/main/tool_evaluation/tool_evaluation.ipynb ## Goal AOS ships a provider-neutral ordinary skill for evaluating whether an agent tool is understandable and useful in practice. The skill snapshots the small reusable method, not either provider API implementation. ## Acceptance * The skill keeps evaluation tasks and expected outcomes separate from tool definitions. * The agent runs representative tasks and records correctness, duration, tool-call count, failures, and actionable feedback. * The evaluation checks tool names, descriptions, parameter clarity, required versus optional inputs, error guidance, and oversized results. * The agent compares the tool against a baseline or prior definition when the comparison can change a decision. * The skill frontmatter carries this source URL and the MIT license. * AOS adds no centralized provenance registry or automatic upstream sync. * Focused fixtures demonstrate a useful tool, an ambiguous tool, and a tool whose result is correct only after avoidable retries.

coilyco-ops added the

autonomy

headless

priority

labels

2026-07-29 03:44:52 +00:00

coilyco-ops referenced this issue from a commit

2026-07-29 16:59:19 +00:00

feat: add agent tool evaluation skill

coilyco-ops referenced this issue from a pull request that will close it,

2026-07-29 17:01:33 +00:00

feat: add agent tool evaluation skill #787

coilyco-ops commented

2026-07-29 17:02:35 +00:00

Author

Owner

WARD-WORKFLOW: blocked 🛑

details

manual close/update snapshot at 2026-07-29T17:01:33Z is newer than the latest ward acknowledgement at 2026-07-29T16:50:12Z

This action is blocked until the feedback is visibly acknowledged.

— Claude (she/her), via ward agent

WARD-WORKFLOW: blocked 🛑 <details><summary>details</summary> manual close/update snapshot at 2026-07-29T17:01:33Z is newer than the latest ward acknowledgement at 2026-07-29T16:50:12Z This action is blocked until the feedback is visibly acknowledged. </details>  — Claude (she/her), via `ward agent`

coilyco-ops commented

2026-07-29 17:27:03 +00:00

Author

Owner

WARD-WORKFLOW: blocked 🛑

details

manual close/update snapshot at 2026-07-29T17:26:58Z is newer than the latest ward acknowledgement at 2026-07-29T17:02:35Z

This action is blocked until the feedback is visibly acknowledged.

— Codex, via ward agent

WARD-WORKFLOW: blocked 🛑 <details><summary>details</summary> manual close/update snapshot at 2026-07-29T17:26:58Z is newer than the latest ward acknowledgement at 2026-07-29T17:02:35Z This action is blocked until the feedback is visibly acknowledged. </details>  — Codex, via `ward agent`

coilyco-ops closed this issue

2026-07-29 17:27:40 +00:00

coilyco-ops commented

2026-07-29 17:28:30 +00:00

Author

Owner

WARD-WORKFLOW: done ✅

details

workflow: pull-request-and-merge; review summary: skipped because implementation and QA are role-separated. QA is a separate, opt-in exact-commit verification role.

Landed PR #787 at 3d828a89 from checked head b52f4963. Pre-commit passed, local runnable tests passed 593 with one skip, and all three CI contexts passed.

Felt: crisp method extraction, noisy landing after main moved and the PR needed Ward's exact marker spelling.

Confidence: high.

Surprises: generated aosguard required unavailable SSM despite the injected Forgejo token. Ward's native merge gate recovered after its visible acknowledgement.

Follow-ups: none for this issue. Missing local specgen and the aosguard credential-source mismatch remain environment observations.

WARD-WORKFLOW: done ✅ <details><summary>details</summary> workflow: pull-request-and-merge; review summary: skipped because implementation and QA are role-separated. QA is a separate, opt-in exact-commit verification role. Landed PR #787 at `3d828a89` from checked head `b52f4963`. Pre-commit passed, local runnable tests passed 593 with one skip, and all three CI contexts passed. Felt: crisp method extraction, noisy landing after main moved and the PR needed Ward's exact marker spelling. Confidence: high. Surprises: generated aosguard required unavailable SSM despite the injected Forgejo token. Ward's native merge gate recovered after its visible acknowledgement. Follow-ups: none for this issue. Missing local `specgen` and the aosguard credential-source mismatch remain environment observations. </details>

coilyco-ops commented

2026-07-29 17:28:43 +00:00

Author

Owner

WARD-WORKFLOW: reservation-released

release details

Run finished with WARD-WORKFLOW: done ✅.

ward container reap released container engineer-codex-agentic-os-670 (--harness codex): the terminal outcome supersedes the reservation, so a later redispatch no longer needs --override-reservation.

— Codex, via ward agent

WARD-WORKFLOW: reservation-released <details><summary>release details</summary> Run finished with `WARD-WORKFLOW: done ✅`. `ward container reap` released container `engineer-codex-agentic-os-670` (`--harness codex`): the terminal outcome supersedes the reservation, so a later redispatch no longer needs `--override-reservation`. </details>  — Codex, via `ward agent`

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#670

No description provided.

Rows
Columns