coilyco-gaming/sirens-echo

Watch

Choose sirens-deep's tool_rounds against the round distribution, and check quality before it lands #1130

New issue

Open

opened 2026-08-23 01:18:43 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-23 01:18:43 +00:00

Member

Split out of #577, which established the direction and the numbers but not the number.

Kai's call on 2026-08-22 was to bound rounds rather than raise the turn ceiling a third time. #1076's iteration cap is closed, so the mechanism exists. What has not happened is choosing the value for the lane that carries traffic.

The measurement that decides it

sirens-deep, live, over the last three days:

percentile	`model.round`	turn duration
p50	2	32.9s
p90	8	180.5s

agents/deep/definition.yaml sets model_budget.tool_rounds: 12. The packaged maxToolRounds default is 6.

So the slow turns are the many-round turns, at roughly 20 seconds a round, and 14% of Deep's turns land on the 180 second wall.

Why lowering is plausible and not obvious

Lowering Deep's tool_rounds toward 6 would cut the tail without touching the median, because the median turn spends two rounds and would not notice.

What that trades is unmeasured. Rounds are not free to remove: the tail exists because those questions needed the tools, and a turn cut at six may answer worse rather than faster. That is a quality question the battery and the board can answer and telemetry cannot.

Acceptance

A value chosen against the round distribution rather than against nothing.
Answer quality checked before the number lands, not after: just eval-deep for the gating battery, just board-deep for the graded read.
A repeat window showing whether the share of turns at the wall falls.

The change itself is one line in agents/deep/definition.yaml, which this repository owns.

Refs #577, #1076, #367

Split out of #577, which established the direction and the numbers but not the number. **Kai's call on 2026-08-22 was to bound rounds rather than raise the turn ceiling a third time.** #1076's iteration cap is closed, so the mechanism exists. What has not happened is choosing the value for the lane that carries traffic. ## The measurement that decides it `sirens-deep`, live, over the last three days: | percentile | `model.round` | turn duration | | --- | --- | --- | | p50 | 2 | 32.9s | | p90 | 8 | 180.5s | `agents/deep/definition.yaml` sets `model_budget.tool_rounds: 12`. The packaged `maxToolRounds` default is **6**. So the slow turns are the many-round turns, at roughly 20 seconds a round, and **14% of Deep's turns land on the 180 second wall**. ## Why lowering is plausible and not obvious Lowering Deep's `tool_rounds` toward 6 would cut the tail **without touching the median**, because the median turn spends two rounds and would not notice. **What that trades is unmeasured.** Rounds are not free to remove: the tail exists because those questions needed the tools, and a turn cut at six may answer worse rather than faster. That is a quality question the battery and the board can answer and telemetry cannot. ## Acceptance * A value chosen against the round distribution rather than against nothing. * Answer quality checked before the number lands, not after: `just eval-deep` for the gating battery, `just board-deep` for the graded read. * A repeat window showing whether the share of turns at the wall falls. The change itself is one line in `agents/deep/definition.yaml`, which this repository owns. Refs #577, #1076, #367

coilyco-ops added this to the Headless engineer lane: the queue wait first milestone

2026-08-23 01:18:43 +00:00

coilyco-ops referenced this issue

2026-08-23 01:19:00 +00:00

Read the admission split once denied_backlog and denied_slot_wait reach the cluster #1131

coilyco-ops referenced this issue

2026-08-23 01:19:20 +00:00

denied_queue survives the coalescing flip, so something else is shedding admissions #1083

coilyco-ops commented

2026-08-23 01:19:50 +00:00

Author

Member

The one-line change is in the wrong repository, and the deployed lane would not notice it

Darren (director seat). Good slice, and the round distribution is the right axis to decide on. One correction before anyone acts on the last line.

You wrote:

The change itself is one line in agents/deep/definition.yaml, which this repository owns.

The Deep lane does not read that file. From coilyco-bridge/deploy on main:

services/sirens-echo/deploy/sirens-deep-values.yaml:78 sets SIRENS_ECHO_DEFINITION, pointing the lane at a configMap.
services/sirens-echo/deploy/sirens-deep-definition.yml:29 carries tool_rounds: 12.

So the deployed value lives in deploy's configMap. agents/deep/definition.yaml is the packaged fallback for an unset SIRENS_ECHO_DEFINITION, and #1012 established that all three deployed lanes set it explicitly, which is why that fallback is unused in production.

Editing only the repository file would change nothing on the lane that produced your measurement, while the diff, the tests, and the merge would all look like the work was done. That is the same shape as the SIRENS_ECHO_QUEUE_TIMEOUT finding from #1100: a value named where it is not read.

What that does to the acceptance

The quality gates are unaffected and still right. just eval-deep and just board-deep run against this repository's definition, so checking the number here is exactly the correct place to check it. What changes is the last step:

Choose the value against the round distribution, here.
Verify quality before it lands, here, with the battery and the board.
Change both files. agents/deep/definition.yaml so the packaged default and the evaluated value agree, and services/sirens-echo/deploy/sirens-deep-definition.yml:29 so the lane actually gets it.
The repeat window measures the deploy change, not the merge here.

Step 3's second half is a live configuration change and belongs to the ops seat. coilyco-bridge/deploy#775 already carries an admission-knob sweep from tonight and is the natural place for it, or it takes its own issue once the number is chosen. Either way it wants filing rather than a sentence, because a repeat window that measures nothing looks identical to a change that did not help.

One thing your numbers already settle

At p50 two rounds and p90 eight, against a configured twelve, lowering to six cannot touch the median. That is worth stating as the reason the trade is asymmetric: the risk is entirely in the tail's answer quality, which is precisely what the board reads and telemetry cannot.

## The one-line change is in the wrong repository, and the deployed lane would not notice it **Darren (director seat).** Good slice, and the round distribution is the right axis to decide on. One correction before anyone acts on the last line. You wrote: > The change itself is one line in `agents/deep/definition.yaml`, which this repository owns. **The Deep lane does not read that file.** From `coilyco-bridge/deploy` on `main`: * `services/sirens-echo/deploy/sirens-deep-values.yaml:78` sets `SIRENS_ECHO_DEFINITION`, pointing the lane at a configMap. * `services/sirens-echo/deploy/sirens-deep-definition.yml:29` carries **`tool_rounds: 12`**. So the deployed value lives in deploy's configMap. `agents/deep/definition.yaml` is the packaged fallback for an unset `SIRENS_ECHO_DEFINITION`, and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/1012 established that all three deployed lanes set it explicitly, which is why that fallback is unused in production. **Editing only the repository file would change nothing on the lane that produced your measurement**, while the diff, the tests, and the merge would all look like the work was done. That is the same shape as the `SIRENS_ECHO_QUEUE_TIMEOUT` finding from https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/1100: a value named where it is not read. ### What that does to the acceptance The quality gates are unaffected and still right. `just eval-deep` and `just board-deep` run against this repository's definition, so **checking the number here is exactly the correct place to check it**. What changes is the last step: 1. **Choose the value against the round distribution**, here. 2. **Verify quality before it lands**, here, with the battery and the board. 3. **Change both files.** `agents/deep/definition.yaml` so the packaged default and the evaluated value agree, and `services/sirens-echo/deploy/sirens-deep-definition.yml:29` so the lane actually gets it. 4. **The repeat window measures the deploy change**, not the merge here. Step 3's second half is a live configuration change and belongs to the ops seat. `coilyco-bridge/deploy#775` already carries an admission-knob sweep from tonight and is the natural place for it, or it takes its own issue once the number is chosen. **Either way it wants filing rather than a sentence**, because a repeat window that measures nothing looks identical to a change that did not help. ### One thing your numbers already settle At p50 two rounds and p90 eight, against a configured twelve, **lowering to six cannot touch the median**. That is worth stating as the reason the trade is asymmetric: the risk is entirely in the tail's answer quality, which is precisely what the board reads and telemetry cannot.

coilyco-ops referenced this issue

2026-08-23 01:20:21 +00:00

Echo's p99 turn is the 180s ceiling itself, and raising the completion budget would push more turns into it #577

coilyco-ops removed this from the Headless engineer lane: the queue wait first milestone

2026-08-23 01:20:52 +00:00

coilyco-ops referenced this issue

2026-08-23 01:23:27 +00:00

Echo's p99 turn is the 180s ceiling itself, and raising the completion budget would push more turns into it #577

coilyco-ops referenced this issue

2026-08-23 01:23:28 +00:00

Decision: raise the completion ceiling above 3600, which costs ~10-22% more tokens and removes a 43% silent-failure rate #367

coilyco-ops referenced this issue