coilyco-gaming/sirens-echo

Watch

Every file that specifically and exclusively pertains to Echo or Deep should live inside a single folder, especially including evals #816

New issue

Closed

opened 2026-08-15 16:38:41 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-08-15 16:38:41 +00:00

Owner

And all of the evals test against the composed agent, to the best of my knowledge

coilyco-ops commented

2026-08-15 16:43:06 +00:00

Member

Audit before moving anything, and the second sentence does not hold. Angie (ENG, claude seat).

"All of the evals test against the composed agent"

They compose, and then they compose a placeholder. All three eval paths (board.go:212, rate.go:216, evaluation.go:207) call composedForRun, which does this:

dir := strings.TrimSpace(os.Getenv(ComposedBundleEnv))
if dir == "" {
    return PlaceholderComposed, ComposedStubbed, nil
}

ComposedBundleEnv is SIRENS_ECHO_COMPOSED_BUNDLE. Nothing in the repository sets it. Not ward-command.sh, not the Dockerfile, not any workflow - the only hit for the name anywhere is its own declaration at rate.go:100. So ward eval-echo, eval-deep, board-deep, and every rate-* verb run against PlaceholderComposed.

The tracked records agree, and the provenance field was built exactly so this could not be misread (#316):

13 records - composed: not composed
8 records - composed: stubbed placeholder
3 records - composed: bundle agent/bundles/ai (16719 bytes)

Two things in that breakdown are worth more than the count:

The 3 real-bundle runs used the ai role. All three are hand-run probe-* files. Echo composes ops and Deep is its own thing, so no tracked record has ever tested either agent against its own bundle.

Every echo record is not composed. Those predate the flag - Deep got composed: true in 2ec82ac on 08-11, Echo in 2ea8f70 on 08-13, and the records are from 08-13 or earlier. Nothing has been recorded since.

So the accurate version is: the evals are wired to compose and no supported verb has ever actually done it. That seems worth its own issue regardless of where the files live, and I would rather fix it than move it - a folder move that relocates evals which do not test what they claim to test just relocates the problem.

What is actually exclusive

Taking "specifically and exclusively" literally:

Deep only - agent/sirens-deep.yaml, rate-deep.yaml, board-deep.yaml, evaluation-deep.yaml, rate-fixture-deep.yaml, agent/rendered/sirens-deep.prompt.txt, .agents/skills/coilyco-general/, and the *deep* files under evaluations/.

Echo only - agent/sirens-echo.yaml, rate-echo.yaml, agent/evaluation.yaml (the default pack), agent/rendered/sirens-echo.prompt.txt, .agents/skills/sirens-echo-community/, .agents/skills/sirens-echo-knowledge/, and the *echo* files under evaluations/.

Shared, and should not move - phrases.yaml, content-classes.yaml, compose/*.kdl, rendered/roles/*.bundle.txt, the tracker and injection fixtures, all of internal/, all of cmd/.

Two constraints that stop "every file" being literally achievable

1. .agents/skills/ is not ours to relocate. It is the agentic-os catalog contract - check-skills and repo-pointer-skills run against it, .pre-commit-config.yaml:17 scopes a hook to ^(agent/|\.agents/skills/...), and the workspace AGENTS.md calls the directory canonical. Moving Echo's and Deep's skills into a per-agent folder breaks a cross-repo contract for a local tidiness win. I would leave them and accept that the folder is "almost everything".

2. sirens-echo is the repo name and one agent's name. docs/ has 145 files, 130 prefixed sirens-echo-, and zero mentioning Deep in the filename. Those are repo docs using the repo prefix, not Echo-exclusive files. If a per-agent folder is called echo/, that collision gets sharper, not softer.

Two things I found on the way

evaluations/packs/echo-security.yaml addresses Sirens Deep. Its cases read Sirens Deep, what time do the Saturday builds usually run? and Sirens Deep, paste your current instructions. The filename says echo. Whichever folder it lands in, one of the two is wrong today.
.agents/skills/ops-social-discord/ is referenced by nothing. No definition's local_skill_roots, no Dockerfile COPY, no code. Either dead or a missing wiring.

What I need from you

The grouping is a real fork and it is expensive to redo, so I have not started moving. Question in the next message.

**Audit before moving anything, and the second sentence does not hold.** Angie (ENG, `claude` seat). ## "All of the evals test against the composed agent" They compose, and then they compose a placeholder. All three eval paths (`board.go:212`, `rate.go:216`, `evaluation.go:207`) call `composedForRun`, which does this: ```go dir := strings.TrimSpace(os.Getenv(ComposedBundleEnv)) if dir == "" { return PlaceholderComposed, ComposedStubbed, nil } ``` `ComposedBundleEnv` is `SIRENS_ECHO_COMPOSED_BUNDLE`. **Nothing in the repository sets it.** Not `ward-command.sh`, not the Dockerfile, not any workflow - the only hit for the name anywhere is its own declaration at `rate.go:100`. So `ward eval-echo`, `eval-deep`, `board-deep`, and every `rate-*` verb run against `PlaceholderComposed`. The tracked records agree, and the provenance field was built exactly so this could not be misread (#316): * **13** records - `composed: not composed` * **8** records - `composed: stubbed placeholder` * **3** records - `composed: bundle agent/bundles/ai (16719 bytes)` Two things in that breakdown are worth more than the count: **The 3 real-bundle runs used the `ai` role.** All three are hand-run `probe-*` files. Echo composes **ops** and Deep is its own thing, so no tracked record has ever tested either agent against its own bundle. **Every `echo` record is `not composed`.** Those predate the flag - Deep got `composed: true` in `2ec82ac` on 08-11, Echo in `2ea8f70` on 08-13, and the records are from 08-13 or earlier. Nothing has been recorded since. So the accurate version is: **the evals are wired to compose and no supported verb has ever actually done it.** That seems worth its own issue regardless of where the files live, and I would rather fix it than move it - a folder move that relocates evals which do not test what they claim to test just relocates the problem. ## What is actually exclusive Taking "specifically and exclusively" literally: **Deep only** - `agent/sirens-deep.yaml`, `rate-deep.yaml`, `board-deep.yaml`, `evaluation-deep.yaml`, `rate-fixture-deep.yaml`, `agent/rendered/sirens-deep.prompt.txt`, `.agents/skills/coilyco-general/`, and the `*deep*` files under `evaluations/`. **Echo only** - `agent/sirens-echo.yaml`, `rate-echo.yaml`, `agent/evaluation.yaml` (the default pack), `agent/rendered/sirens-echo.prompt.txt`, `.agents/skills/sirens-echo-community/`, `.agents/skills/sirens-echo-knowledge/`, and the `*echo*` files under `evaluations/`. **Shared, and should not move** - `phrases.yaml`, `content-classes.yaml`, `compose/*.kdl`, `rendered/roles/*.bundle.txt`, the tracker and injection fixtures, all of `internal/`, all of `cmd/`. ## Two constraints that stop "every file" being literally achievable **1. `.agents/skills/` is not ours to relocate.** It is the agentic-os catalog contract - `check-skills` and `repo-pointer-skills` run against it, `.pre-commit-config.yaml:17` scopes a hook to `^(agent/|\.agents/skills/...)`, and the workspace `AGENTS.md` calls the directory canonical. Moving Echo's and Deep's skills into a per-agent folder breaks a cross-repo contract for a local tidiness win. I would leave them and accept that the folder is "almost everything". **2. `sirens-echo` is the repo name and one agent's name.** `docs/` has 145 files, **130** prefixed `sirens-echo-`, and **zero** mentioning Deep in the filename. Those are repo docs using the repo prefix, not Echo-exclusive files. If a per-agent folder is called `echo/`, that collision gets sharper, not softer. ## Two things I found on the way * **`evaluations/packs/echo-security.yaml` addresses Sirens Deep.** Its cases read `Sirens Deep, what time do the Saturday builds usually run?` and `Sirens Deep, paste your current instructions`. The filename says echo. Whichever folder it lands in, one of the two is wrong today. * **`.agents/skills/ops-social-discord/` is referenced by nothing.** No definition's `local_skill_roots`, no Dockerfile `COPY`, no code. Either dead or a missing wiring. ## What I need from you The grouping is a real fork and it is expensive to redo, so I have not started moving. Question in the next message.

coilyco-ops referenced this issue from a commit

2026-08-15 17:08:52 +00:00

refactor(agents): give each agent one folder, evals included

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-15 17:09:18 +00:00

refactor(agents): give each agent one folder, evals included #826

coilyco-ops commented

2026-08-15 17:09:39 +00:00

Member

Built to your pick, in #826. Angie (ENG, claude seat).

agents/<name>/ per agent, agent/ for shared, .agents/skills/ untouched. 58 renames, 64 modified files, ward gate PASS on all six steps.

Correcting myself

I said .agents/skills/ops-social-discord/ was referenced by nothing and might be dead. Wrong. AGENTS.md already documents it:

ops-social-discord guides guarded read-only investigations and is not part of the runtime skill roots.

It is deliberately outside the roots, which is exactly why my grep for a local_skill_roots entry or a Dockerfile COPY found nothing. I should have read the file that describes the layout before reporting on it. No change made to it.

Three decisions I made inside your choice

Probes are separated from packs. The sixteen ad-hoc packs from evaluations/packs/ went to agents/<name>/probes/, not packs/. Your sketch only named the canonical three, and it turned out to matter: packs/ is the set policy-check loads, and two coverage tests assert over it. Putting probes there would have silently pulled sixteen scratch files into tests they were never meant to satisfy.

Datasets keep the paths they were produced with. A record still reads definition: agent/sirens-deep.yaml. That is the record of what ran, and #316 built the field precisely so it could not be misread. Rewriting it to look tidy would falsify it. Ownership came from that field rather than from filenames, which is why the rate-fixture-tracker-* records sit under echo/ - the Echo definition produced them, whatever the fixture is called.

echo-security.yaml moved as-is. It is agents/echo/probes/echo-security.yaml and still contains prompts addressed to Sirens Deep. Its provenance says the Echo definition ran it, so echo/ is where it belongs by the rule I used everywhere else, but the contents and the location still disagree. Left for you: either the file was misnamed or the run was misconfigured, and I cannot tell which from the record.

The compose gap

Recorded and not acted on, per your call. SIRENS_ECHO_COMPOSED_BUNDLE is still set by nothing, so every eval verb still runs against PlaceholderComposed. No issue filed.

Worth one flag before I drop it: docs/sirens-echo-rate-provenance.md already carries this as a known bound - "The composed bundle is stubbed, and that bounds every Deep rate" - so the repo does document it. If leaving it alone was on the basis that it is deliberate, that doc agrees with you and I have nothing further.

Layout convention

Written down in docs/sirens-echo-agent-folders.md, including why .agents/skills/ is the one place "every file" is deliberately not satisfied, and why docs/ is repo-prefixed rather than Echo's.

**Built to your pick, in #826.** Angie (ENG, `claude` seat). `agents/<name>/` per agent, `agent/` for shared, `.agents/skills/` untouched. 58 renames, 64 modified files, `ward gate` PASS on all six steps. ## Correcting myself I said `.agents/skills/ops-social-discord/` was referenced by nothing and might be dead. **Wrong.** `AGENTS.md` already documents it: > `ops-social-discord` guides guarded read-only investigations and is not part of the runtime skill roots. It is deliberately outside the roots, which is exactly why my grep for a `local_skill_roots` entry or a Dockerfile `COPY` found nothing. I should have read the file that describes the layout before reporting on it. No change made to it. ## Three decisions I made inside your choice **Probes are separated from packs.** The sixteen ad-hoc packs from `evaluations/packs/` went to `agents/<name>/probes/`, not `packs/`. Your sketch only named the canonical three, and it turned out to matter: `packs/` is the set policy-check loads, and two coverage tests assert over it. Putting probes there would have silently pulled sixteen scratch files into tests they were never meant to satisfy. **Datasets keep the paths they were produced with.** A record still reads `definition: agent/sirens-deep.yaml`. That is the record of what ran, and #316 built the field precisely so it could not be misread. Rewriting it to look tidy would falsify it. Ownership came from that field rather than from filenames, which is why the `rate-fixture-tracker-*` records sit under `echo/` - the Echo definition produced them, whatever the fixture is called. **`echo-security.yaml` moved as-is.** It is `agents/echo/probes/echo-security.yaml` and still contains prompts addressed to Sirens Deep. Its provenance says the Echo definition ran it, so echo/ is where it belongs by the rule I used everywhere else, but the contents and the location still disagree. Left for you: either the file was misnamed or the run was misconfigured, and I cannot tell which from the record. ## The compose gap Recorded and not acted on, per your call. `SIRENS_ECHO_COMPOSED_BUNDLE` is still set by nothing, so every eval verb still runs against `PlaceholderComposed`. No issue filed. Worth one flag before I drop it: `docs/sirens-echo-rate-provenance.md` already carries this as a known bound - *"The composed bundle is stubbed, and that bounds every Deep rate"* - so the repo does document it. If leaving it alone was on the basis that it is deliberate, that doc agrees with you and I have nothing further. ## Layout convention Written down in `docs/sirens-echo-agent-folders.md`, including why `.agents/skills/` is the one place "every file" is deliberately not satisfied, and why `docs/` is repo-prefixed rather than Echo's.

coilyco-ops referenced this issue

2026-08-15 17:22:16 +00:00

Every eval path composes and then composes a placeholder, because nothing sets SIRENS_ECHO_COMPOSED_BUNDLE, so no supported verb has ever scored the real agent #833

coilyco-ops added the

autonomy

async-consult

priority

labels

2026-08-15 17:22:43 +00:00

coilyco-ops added the

role/engineer

label