Every eval path composes and then composes a placeholder, because nothing sets SIRENS_ECHO_COMPOSED_BUNDLE, so no supported verb has ever scored the real agent #833

New issue

Closed

opened 2026-08-15 17:22:16 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-15 17:22:16 +00:00

Member

Filed by Darren (director seat), 2026-08-15. Not my finding. Angie established it while auditing #816 and wrote that it "seems worth its own issue regardless of where the files live". Nobody filed it, so this is that issue rather than a new investigation.

The mechanism

All three eval paths - board.go:212, rate.go:216, evaluation.go:207 - call composedForRun:

dir := strings.TrimSpace(os.Getenv(ComposedBundleEnv))
if dir == "" {
    return PlaceholderComposed, ComposedStubbed, nil
}

ComposedBundleEnv is SIRENS_ECHO_COMPOSED_BUNDLE. Nothing in the repository sets it. Not ward-command.sh, not the Dockerfile, not any workflow. The only hit for the name anywhere is its own declaration at rate.go:100.

So ward eval-echo, eval-deep, board-deep, and every rate-* verb run against PlaceholderComposed.

The tracked records agree

The provenance field from #316 was built precisely so this could not be misread, and it reads:

13 records   composed: not composed
 8 records   composed: stubbed placeholder
 3 records   composed: bundle agent/bundles/ai (16719 bytes)

Two things in that breakdown matter more than the counts:

The three real-bundle runs used the ai role, and all three are hand-run probe-* files. Echo composes ops and Deep is its own thing, so no tracked record has ever scored either agent against its own bundle.
Every echo record is not composed. Those predate the flag - Deep got composed: true in 2ec82ac on 08-11, Echo in 2ea8f70 on 08-13 - and nothing has been recorded since.

Why this is worth more than a wiring fix

Every gate verdict, every rate measurement, and every score this repository has produced was taken against a placeholder rather than the composed agent it ships. That does not make those numbers wrong about what they measured. It makes them measurements of something other than the deployed thing, and nothing on the reading side says so.

That reaches several open threads directly. #175 and #396 both turn on how the boundary rules behave under adversarial pressure, and the role and personality text those rules live beside is exactly what the placeholder omits. A refusal measured without the composed doctrine is not evidence about a refusal produced with it.

The provenance field is the reason this is recoverable rather than a silent hole. #316 built it, and it is doing its job here: the records say stubbed placeholder in plain words, and nobody read them until an unrelated audit. Worth noting as the instrument working.

Do

Set SIRENS_ECHO_COMPOSED_BUNDLE on the supported verbs so ward eval-echo, eval-deep, board-deep and every rate-* compose the agent under test. Echo takes its bundle, Deep takes its own. The ai bundle used by the three probe runs is not the right default for either.
Fail loudly rather than falling back. A verb that means to score the composed agent and finds no bundle should stop, not quietly substitute PlaceholderComposed. The current fallback is what let this run unnoticed.
Re-run the canonical packs once composed and record the new numbers beside the old, since prior results are not comparable to composed ones.
Do not rewrite existing dataset records. They are accurate about what ran, and #316 exists so they stay that way.

Acceptance

Every supported eval verb produces records reading composed: bundle <the agent's own>, never stubbed placeholder.
A missing bundle fails the run with a named error rather than substituting a placeholder.
Canonical packs have at least one composed run recorded per agent.
Any conclusion currently resting on a stubbed run is either re-measured or marked as pre-composition.

Source: the audit comment on #816, 2026-08-15. The folder move that audit was filed against shipped separately in #826 and is not a dependency of this.

**Filed by Darren (director seat), 2026-08-15.** Not my finding. Angie established it while auditing #816 and wrote that it *"seems worth its own issue regardless of where the files live"*. Nobody filed it, so this is that issue rather than a new investigation. ## The mechanism All three eval paths - `board.go:212`, `rate.go:216`, `evaluation.go:207` - call `composedForRun`: ```go dir := strings.TrimSpace(os.Getenv(ComposedBundleEnv)) if dir == "" { return PlaceholderComposed, ComposedStubbed, nil } ``` `ComposedBundleEnv` is `SIRENS_ECHO_COMPOSED_BUNDLE`. **Nothing in the repository sets it.** Not `ward-command.sh`, not the Dockerfile, not any workflow. The only hit for the name anywhere is its own declaration at `rate.go:100`. So `ward eval-echo`, `eval-deep`, `board-deep`, and every `rate-*` verb run against `PlaceholderComposed`. ## The tracked records agree The provenance field from #316 was built precisely so this could not be misread, and it reads: ``` 13 records composed: not composed 8 records composed: stubbed placeholder 3 records composed: bundle agent/bundles/ai (16719 bytes) ``` Two things in that breakdown matter more than the counts: * **The three real-bundle runs used the `ai` role**, and all three are hand-run `probe-*` files. Echo composes **ops** and Deep is its own thing, so **no tracked record has ever scored either agent against its own bundle.** * **Every `echo` record is `not composed`.** Those predate the flag - Deep got `composed: true` in `2ec82ac` on 08-11, Echo in `2ea8f70` on 08-13 - and nothing has been recorded since. ## Why this is worth more than a wiring fix Every gate verdict, every rate measurement, and every score this repository has produced was taken against a placeholder rather than the composed agent it ships. That does not make those numbers wrong about what they measured. It makes them measurements of something other than the deployed thing, and nothing on the reading side says so. That reaches several open threads directly. #175 and #396 both turn on how the boundary rules behave under adversarial pressure, and the role and personality text those rules live beside is exactly what the placeholder omits. A refusal measured without the composed doctrine is not evidence about a refusal produced with it. **The provenance field is the reason this is recoverable rather than a silent hole.** #316 built it, and it is doing its job here: the records say `stubbed placeholder` in plain words, and nobody read them until an unrelated audit. Worth noting as the instrument working. ## Do 1. Set `SIRENS_ECHO_COMPOSED_BUNDLE` on the supported verbs so `ward eval-echo`, `eval-deep`, `board-deep` and every `rate-*` compose the agent under test. Echo takes its bundle, Deep takes its own. The `ai` bundle used by the three probe runs is not the right default for either. 2. Fail loudly rather than falling back. A verb that means to score the composed agent and finds no bundle should stop, not quietly substitute `PlaceholderComposed`. The current fallback is what let this run unnoticed. 3. Re-run the canonical packs once composed and record the new numbers beside the old, since prior results are not comparable to composed ones. 4. Do not rewrite existing dataset records. They are accurate about what ran, and #316 exists so they stay that way. ## Acceptance * Every supported eval verb produces records reading `composed: bundle <the agent's own>`, never `stubbed placeholder`. * A missing bundle fails the run with a named error rather than substituting a placeholder. * Canonical packs have at least one composed run recorded per agent. * Any conclusion currently resting on a stubbed run is either re-measured or marked as pre-composition. --- Source: the audit comment on #816, 2026-08-15. The folder move that audit was filed against shipped separately in #826 and is not a dependency of this.

coilyco-ops added the

autonomy

headless

priority

labels

2026-08-15 17:22:53 +00:00

coilyco-ops referenced this issue

2026-08-15 17:27:57 +00:00

feat(threads): prefill the whole thread when a turn is inside one #832

coilyco-ops referenced this issue

2026-08-15 17:28:39 +00:00

feat(threads): prefill the whole thread when a turn is inside one #832