finding (audit): 2026-05-13 - audit finding walkthrough prints a agentic-os-kai path for findings that actually live under coily #40

Open
opened 2026-05-23 20:54:01 +00:00 by coilysiren · 0 comments
Owner

Originally filed by @coilysiren on 2026-05-18T03:42:45Z - https://github.com/coilysiren/coily/issues/217

Migrated from coily-audit-meta/findings/2026-05-13-audit-finding-walkthrough-emits-wrong-skills-path.md on 2026-05-17 as part of coilysiren/coily#215. Original file preserved in git history; see deletion commit on coilysiren/coily#215.

2026-05-13 - audit finding walkthrough prints a agentic-os-kai path for findings that actually live under coily

What was observed

Running coily --commit-scope=/home/kai/projects/coilysiren/agentic-os-kai audit finding --id 019e23b4-73d9-7829-be9f-dd2529090b90 --slug ops-aws-passthrough-mangles-query-output-flags (audit row 019e23bb-9a70-7cc4-bdfe-fa2b60f5dc84, ts 1778715957, verb audit.finding) emitted a Step 3 instruction with this path:

/home/kai/projects/coilysiren/agentic-os-kai/.claude/skills/coily-<area>-meta/findings/YYYY-MM-DD-<slug>.md

The coily-*-meta skill directories do not exist under agentic-os-kai/.claude/skills/. They live under coily/.claude/skills/. The "References" block at the end of the walkthrough reinforces the same wrong location for coily-meta-improvement/SKILL.md and coily-skill-authoring/SKILL.md.

The agent following the walkthrough caught this on contact (the target directory was missing, ran a find over /home/kai/projects/coilysiren to locate the real coily-ops-aws-meta, then wrote the finding to the correct path under coily/). The walkthrough did not catch itself.

Why it slipped

The walkthrough copy was probably authored when the meta skills lived (or were planned to live) under agentic-os-kai, and the path string was hardcoded into the walkthrough emitter. When the meta skills landed under coily/.claude/skills/ instead (where they sit alongside the source they document), the walkthrough emitter wasn't updated. Nothing in the build verifies the printed path resolves on disk before shipping a coily release.

This is a class of drift the audit-finding loop is uniquely vulnerable to. The walkthrough is the onboarding doc for new agents writing findings. If it points to a wrong path and an agent trusts it verbatim, the finding lands in an orphan tree under agentic-os-kai/.claude/skills/coily-<area>-meta/findings/ that the meta-improvement loop never reads from. The bad outcome is a finding that quietly disappears, not a loud failure.

Rule it produced

Anti-signal candidate (data only, not promoted): "the path printed by coily audit finding's walkthrough is the canonical location to write the finding." Today it is not. Verify the directory exists under coily/.claude/skills/coily-<area>-meta/findings/ before writing. Forward action filed at coily#148.

_Originally filed by @coilysiren on 2026-05-18T03:42:45Z - [https://github.com/coilysiren/coily/issues/217](https://github.com/coilysiren/coily/issues/217)_ _Migrated from `coily-audit-meta/findings/2026-05-13-audit-finding-walkthrough-emits-wrong-skills-path.md` on 2026-05-17 as part of coilysiren/coily#215. Original file preserved in git history; see deletion commit on coilysiren/coily#215._ # 2026-05-13 - audit finding walkthrough prints a agentic-os-kai path for findings that actually live under coily ## What was observed Running `coily --commit-scope=/home/kai/projects/coilysiren/agentic-os-kai audit finding --id 019e23b4-73d9-7829-be9f-dd2529090b90 --slug ops-aws-passthrough-mangles-query-output-flags` (audit row `019e23bb-9a70-7cc4-bdfe-fa2b60f5dc84`, ts 1778715957, verb `audit.finding`) emitted a Step 3 instruction with this path: ``` /home/kai/projects/coilysiren/agentic-os-kai/.claude/skills/coily-<area>-meta/findings/YYYY-MM-DD-<slug>.md ``` The `coily-*-meta` skill directories do not exist under `agentic-os-kai/.claude/skills/`. They live under `coily/.claude/skills/`. The "References" block at the end of the walkthrough reinforces the same wrong location for `coily-meta-improvement/SKILL.md` and `coily-skill-authoring/SKILL.md`. The agent following the walkthrough caught this on contact (the target directory was missing, ran a `find` over `/home/kai/projects/coilysiren` to locate the real `coily-ops-aws-meta`, then wrote the finding to the correct path under `coily/`). The walkthrough did not catch itself. ## Why it slipped The walkthrough copy was probably authored when the meta skills lived (or were planned to live) under `agentic-os-kai`, and the path string was hardcoded into the walkthrough emitter. When the meta skills landed under `coily/.claude/skills/` instead (where they sit alongside the source they document), the walkthrough emitter wasn't updated. Nothing in the build verifies the printed path resolves on disk before shipping a coily release. This is a class of drift the audit-finding loop is uniquely vulnerable to. The walkthrough is the onboarding doc for new agents writing findings. If it points to a wrong path and an agent trusts it verbatim, the finding lands in an orphan tree under `agentic-os-kai/.claude/skills/coily-<area>-meta/findings/` that the meta-improvement loop never reads from. The bad outcome is a finding that quietly disappears, not a loud failure. ## Rule it produced Anti-signal candidate (data only, not promoted): "the path printed by `coily audit finding`'s walkthrough is the canonical location to write the finding." Today it is not. Verify the directory exists under `coily/.claude/skills/coily-<area>-meta/findings/` before writing. Forward action filed at [coily#148](https://github.com/coilysiren/coily/issues/148).
coilysiren added
P4
and removed
P3
labels 2026-05-31 06:59:50 +00:00
Sign in to join this conversation.
No labels
P0
P1
P2
P3
P4
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-bridge/coily#40
No description provided.