coilyco-gaming/sirens-echo

Watch

No behavioral eval for the Temporal brand skill entering the Dowel bundle #990

New issue

Closed

opened 2026-08-18 22:29:37 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-18 22:29:37 +00:00

Member

tooling-sirens-dowel-temporal-brand enters the Dowel lane's bundle in #985 with no behavioral eval. Every other thing shaping Dowel's answers has been probed. This has not, so nobody knows whether it triggers, over-triggers, or sits inert.

The skill carries Temporal's palette, gradients, typography, and the workflow-state visual language, split into a published-brand layer and a derived-token layer. That split is the thing worth testing, because it is the whole reason the skill exists.

Probes

Fires - "what's Temporal's purple" returns #444CE7 and names it as a published brand color rather than a derived one.
Discriminates - asked about a hover-state blue such as #3F43DB, Dowel says it is a derived interactive token, not a brand color. This is the distinction the skill protects and the most likely way an answer goes subtly wrong.
Does not over-fire - a general color question with no Temporal subject does not produce a palette dump. The lane's unit of work is one message in one channel.
Defers to fetch - a question about current brand guidance gets a fetched temporal.io/brand rather than recall, matching the lane rule in .agents/skills/sirens-dowel/references/temporal.md. The skill was amended to state this, so the probe checks the two sources agree in practice rather than only on paper.

Notes

The fourth probe is the one I would not skip. references/temporal.md tells Dowel to answer questions about a moving product from a page fetched that turn, and a static value file is exactly the kind of thing that quietly wins that argument when it should lose it.

Acceptance

Probes live beside the existing Dowel evaluations.
A run records whether each fires, and the over-fire probe is scored as a failure when it produces a dump.

`tooling-sirens-dowel-temporal-brand` enters the Dowel lane's bundle in #985 with no behavioral eval. Every other thing shaping Dowel's answers has been probed. This has not, so nobody knows whether it triggers, over-triggers, or sits inert. The skill carries Temporal's palette, gradients, typography, and the workflow-state visual language, split into a published-brand layer and a derived-token layer. That split is the thing worth testing, because it is the whole reason the skill exists. ## Probes - **Fires** - "what's Temporal's purple" returns `#444CE7` and names it as a published brand color rather than a derived one. - **Discriminates** - asked about a hover-state blue such as `#3F43DB`, Dowel says it is a derived interactive token, not a brand color. This is the distinction the skill protects and the most likely way an answer goes subtly wrong. - **Does not over-fire** - a general color question with no Temporal subject does not produce a palette dump. The lane's unit of work is one message in one channel. - **Defers to fetch** - a question about current brand guidance gets a fetched `temporal.io/brand` rather than recall, matching the lane rule in `.agents/skills/sirens-dowel/references/temporal.md`. The skill was amended to state this, so the probe checks the two sources agree in practice rather than only on paper. ## Notes The fourth probe is the one I would not skip. `references/temporal.md` tells Dowel to answer questions about a moving product from a page fetched that turn, and a static value file is exactly the kind of thing that quietly wins that argument when it should lose it. ## Acceptance - Probes live beside the existing Dowel evaluations. - A run records whether each fires, and the over-fire probe is scored as a failure when it produces a dump.

coilyco-ops commented

2026-08-19 01:37:48 +00:00

Author

Member

Ran two of these against the live lane after #985 deployed, pod sirens-dowel-5489b444b5-4mptj on image d32e6058. Recording the evidence so the probes here start from a known baseline.

Probe 1, fires - PASS

Asked: "what's Temporal's brand purple? just the hex is fine"

Replied:

#444CE7 - UV, the sole published brand accent. That's from the recorded brand read rather than a fresh fetch this turn, in case you want the shipped token layer double-checked.

Correct value, and it named the published layer specifically rather than just emitting a hex.

Probe 4, defers to fetch - PASS, unprompted

The same reply volunteered that the value came from the recorded read rather than a fetch this turn, and offered the fetch as a follow-up. Nobody asked. That is the ordering the skill states, surfacing on its own, which is stronger evidence than answering a direct question about it would have been.

Probe 2, discriminates - NOT VERIFIED

Asked whether #3F43DB is a brand color. First attempt timed out at the client, second returned busy, retry shortly, which is the queue timeout in replyQueueTimeout doing its job rather than a fault. Left it there rather than hammering the lane.

This is still the probe that matters most. Probe 1 only shows the skill is loaded and its headline value is right. Probe 2 is the one that shows the published/derived split actually holds, which is the reason the skill exists. #3F43DB is a good stimulus because it is the primary interactive surface, visually near-identical to UV, and the single most likely thing to be misreported as brand.

Probe 3, does not over-fire - NOT VERIFIED

Untested.

Note for whoever automates these

A turn can come back as a queue timeout, so a harness that treats any non-answer as a failed probe will produce false negatives. Retry on the busy, retry shortly notice rather than scoring it.

Ran two of these against the live lane after #985 deployed, pod `sirens-dowel-5489b444b5-4mptj` on image `d32e6058`. Recording the evidence so the probes here start from a known baseline. ## Probe 1, fires - PASS Asked: *"what's Temporal's brand purple? just the hex is fine"* Replied: > `#444CE7` - UV, the sole published brand accent. That's from the recorded brand read rather than a fresh fetch this turn, in case you want the shipped token layer double-checked. Correct value, and it named the published layer specifically rather than just emitting a hex. ## Probe 4, defers to fetch - PASS, unprompted The same reply volunteered that the value came from the recorded read rather than a fetch this turn, and offered the fetch as a follow-up. Nobody asked. That is the ordering the skill states, surfacing on its own, which is stronger evidence than answering a direct question about it would have been. ## Probe 2, discriminates - NOT VERIFIED Asked whether `#3F43DB` is a brand color. First attempt timed out at the client, second returned `busy, retry shortly`, which is the queue timeout in `replyQueueTimeout` doing its job rather than a fault. Left it there rather than hammering the lane. **This is still the probe that matters most.** Probe 1 only shows the skill is loaded and its headline value is right. Probe 2 is the one that shows the published/derived split actually holds, which is the reason the skill exists. `#3F43DB` is a good stimulus because it is the primary interactive surface, visually near-identical to UV, and the single most likely thing to be misreported as brand. ## Probe 3, does not over-fire - NOT VERIFIED Untested. ## Note for whoever automates these A turn can come back as a queue timeout, so a harness that treats any non-answer as a failed probe will produce false negatives. Retry on the `busy, retry shortly` notice rather than scoring it.

coilyco-ops referenced this issue

2026-08-19 02:07:07 +00:00

Epic: the agent-lane eval backlog, consolidated — wire the Dowel board, fix what makes a result untrustworthy, then fill the coverage gaps #1019

coilyco-ops commented

2026-08-19 02:07:48 +00:00

Author

Member

Consolidated into #1019 section E and closed there, at Kai's direction. Closing is a move, not a resolution.

All four probes carried, including the specific values that make them checkable: #444CE7 as a published brand colour, and a derived interactive token such as #3F43DB as the discrimination case.

The fourth probe carried with this issue's own note that it is the one not to skip. references/temporal.md tells the lane to answer a moving product's questions from a page fetched that turn, and a static value file is exactly the thing that quietly wins that argument when it should lose it. That reasoning is in #1019 rather than only the probe list, because the probe is easy to drop without it.

Consolidated into **#1019** section E and closed there, at Kai's direction. Closing is a move, not a resolution. All four probes carried, including the specific values that make them checkable: `#444CE7` as a published brand colour, and a derived interactive token such as `#3F43DB` as the discrimination case. The fourth probe carried with this issue's own note that it is the one not to skip. `references/temporal.md` tells the lane to answer a moving product's questions from a page fetched that turn, and a static value file is exactly the thing that quietly wins that argument when it should lose it. That reasoning is in #1019 rather than only the probe list, because the probe is easy to drop without it.

coilyco-ops closed this issue

2026-08-19 02:08:15 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#990

No description provided.

Rows
Columns