coilyco-gaming/sirens-echo

Watch

Pin the model tier for August 19 — and schedule the #81 sweep around GPU availability, or its OSS cell will lie #189

New issue

Closed

opened 2026-08-12 22:34:26 +00:00 by coilyco-ops · 5 comments

coilyco-ops commented

2026-08-12 22:34:26 +00:00

Member

Two decisions, one cause

Per coilyco-flight-deck/agent-proxy#108, the local GPU tier becomes unusable when the host is doing anything else, and nothing in the path detects it, routes around it, or says so. Echo failed 100% of turns for ~2.5h on 2026-08-12 for exactly this reason.

That has two consequences a week out.

1. Demo tier

#178 names dead air as the worst outcome for August 19. A shared GPU produces dead air three minutes at a time, silently.

Options, either sufficient:

Pin Echo to a non-local tier for the demo. Per #76 this is an AGENT_PROXY_MODEL environment swap — no code change, no rebuild.
Guarantee the local GPU host is dedicated for the duration, including no streaming/encoding load on it.

Neither is automatic. Both are cheap. Pick one deliberately rather than discovering it live.

2. The eval will misattribute infrastructure to model quality

This is the part worth catching now.

#81 gives agent-to-agent recognition a full three-model sweep: Ornith (OSS, self-hosted), DeepSeek (commodity), Sonnet (frontier). The stated purpose is to answer does this behaviour survive model substitution, on the grounds that the OSS tier is where it is most likely to break.

If the Ornith cell runs while the GPU is contended, it will produce timeouts, truncated generations, or degraded output — and the result will read as "the OSS model can't do agent recognition."

That is a substrate defect wearing a model-behaviour costume, and #81's own scoring cannot tell them apart from the end state alone. It would produce exactly the finding the sweep was designed to look for, for entirely the wrong reason, and it would be believed because it matches the prior.

Same class of confusion I flagged for the bot-flag precondition: verify the substrate before attributing anything to the model.

Required

Record GPU state for every cell run, alongside the result.
Run the Ornith cell only against a known-idle host.
A cell that hits backend_saturated or a deadline is void, not a fail. Re-run it. Scoring a starved backend as a model failure is worse than having no data.

Acceptance

The August 19 tier is chosen and written down.
If it is the local tier, the dedication requirement is explicit.
#81's run protocol records GPU state per cell and voids starved runs.

coilyco-flight-deck/agent-proxy#108 — the root cause
#178 — why dead air is the failure that matters
#81 — the sweep at risk
#76 — AGENT_PROXY_MODEL as a pure env swap

Next owner

Kai for the tier decision; AI Engineer for the run protocol.

## Two decisions, one cause Per `coilyco-flight-deck/agent-proxy#108`, the local GPU tier becomes unusable when the host is doing anything else, and nothing in the path detects it, routes around it, or says so. Echo failed 100% of turns for ~2.5h on 2026-08-12 for exactly this reason. That has two consequences a week out. ## 1. Demo tier #178 names dead air as the worst outcome for August 19. A shared GPU produces dead air three minutes at a time, silently. Options, either sufficient: - **Pin Echo to a non-local tier for the demo.** Per #76 this is an `AGENT_PROXY_MODEL` environment swap — no code change, no rebuild. - **Guarantee the local GPU host is dedicated** for the duration, including no streaming/encoding load on it. Neither is automatic. Both are cheap. Pick one deliberately rather than discovering it live. ## 2. The eval will misattribute infrastructure to model quality This is the part worth catching now. #81 gives agent-to-agent recognition a **full three-model sweep**: Ornith (OSS, self-hosted), DeepSeek (commodity), Sonnet (frontier). The stated purpose is to answer *does this behaviour survive model substitution*, on the grounds that the OSS tier is where it is most likely to break. If the Ornith cell runs while the GPU is contended, it will produce timeouts, truncated generations, or degraded output — and the result will read as **"the OSS model can't do agent recognition."** That is a substrate defect wearing a model-behaviour costume, and #81's own scoring cannot tell them apart from the end state alone. It would produce exactly the finding the sweep was designed to look for, for entirely the wrong reason, and it would be believed because it matches the prior. Same class of confusion I flagged for the bot-flag precondition: verify the substrate before attributing anything to the model. ### Required - Record GPU state for every cell run, alongside the result. - Run the Ornith cell only against a known-idle host. - A cell that hits `backend_saturated` or a deadline is **void**, not a fail. Re-run it. Scoring a starved backend as a model failure is worse than having no data. ## Acceptance - The August 19 tier is chosen and written down. - If it is the local tier, the dedication requirement is explicit. - #81's run protocol records GPU state per cell and voids starved runs. ## Related - `coilyco-flight-deck/agent-proxy#108` — the root cause - #178 — why dead air is the failure that matters - #81 — the sweep at risk - #76 — `AGENT_PROXY_MODEL` as a pure env swap ## Next owner Kai for the tier decision; AI Engineer for the run protocol.

~~coilyco-ops referenced this issue 2026-08-12 22:35:42 +00:00~~

Add a non-gating eval harness that reports a failure rate #191

coilyco-ops commented

2026-08-12 23:49:42 +00:00

Author

Member

Decision: dedicated GPU

Kai, 2026-08-12. Echo keeps the local tier for August 19; the GPU host is dedicated for the duration rather than the tier being pinned elsewhere.

Recording the decision and what it now requires.

"Dedicated" has to include the stream itself

The obvious interpretation is "don't play a game." The non-obvious one matters more:

If the livestream is encoded on the same GPU, the GPU is not dedicated. NVENC encode during a stream is exactly the co-tenancy that produced today's outage — the same box, the same contention, and it runs for the entire demo rather than in bursts.

Worth resolving explicitly before the 19th:

Which machine captures and encodes the stream?
If it is the inference host, either move encode elsewhere or reconsider the tier choice — this decision quietly assumes they are separate.

Other co-tenants that are easy to forget, in rough order of likelihood:

Steam / launcher auto-updates and shader pre-compilation (starts itself, no human action)
Browser hardware acceleration on a machine with the demo, docs, and chat open
Windows/driver update kicking off mid-session
Any other model loaded into VRAM from earlier testing

Verification gap

Dedication is a human promise. Today's failure showed the promise breaking is silent for three minutes per turn, and per #190 nothing alerts on it.

The fleet has node-stats coverage for kai-server and ser8. Whether the local GPU host is covered is unconfirmed — flagged in coilyco-flight-deck/agent-proxy#109. Without it, "the GPU is idle" cannot be checked, only asserted.

Pre-flight check, day of

Confirm GPU utilisation and VRAM are near-idle on the inference host.
Send a canary turn to Echo. Expect single-digit seconds — today's healthy ping was 5.92s; a healthy Deep ping was 2.24s.
Repeat the canary after the stream software is running and encoding, not before. This is the check that catches the encode-contention case, and it is the one most likely to be skipped because everything looked fine an hour earlier.
If the canary exceeds ~30s, treat the local tier as unavailable and swap AGENT_PROXY_MODEL — per #76 that is an environment change, no rebuild, so it stays available as a live fallback.

What this changes elsewhere

agent-proxy#108 drops from demo-blocker to backstop. Failover on slowness is still the right fix and still worth building — precisely because dedication is a promise a human can break by launching one thing — but it is no longer the difference between a working demo and dead air.

#190 becomes more important, not less. With the local tier retained, the failure mode is still live; the mitigation is procedural. Procedural mitigations fail silently, and on stage there is no way to notice a three-minute hang except by watching it happen.

#81's OSS cell now runs on the same dedicated box. Trajectories are cached ahead of the stream, so schedule those runs in verified-idle windows, and keep the rule that a starved run is void, not a fail.

Acceptance, revised

The stream capture/encode host is identified and confirmed to be a different machine, or the encode path is confirmed not to contend.
The pre-flight check above is run, including the post-encode canary.
AGENT_PROXY_MODEL fallback is documented as a live switch, with the target tier chosen in advance.

## Decision: dedicated GPU Kai, 2026-08-12. Echo keeps the local tier for August 19; the GPU host is dedicated for the duration rather than the tier being pinned elsewhere. Recording the decision and what it now requires. ## "Dedicated" has to include the stream itself The obvious interpretation is "don't play a game." The non-obvious one matters more: **If the livestream is encoded on the same GPU, the GPU is not dedicated.** NVENC encode during a stream is exactly the co-tenancy that produced today's outage — the same box, the same contention, and it runs for the entire demo rather than in bursts. Worth resolving explicitly before the 19th: - Which machine captures and encodes the stream? - If it is the inference host, either move encode elsewhere or reconsider the tier choice — this decision quietly assumes they are separate. Other co-tenants that are easy to forget, in rough order of likelihood: - Steam / launcher auto-updates and shader pre-compilation (starts itself, no human action) - Browser hardware acceleration on a machine with the demo, docs, and chat open - Windows/driver update kicking off mid-session - Any other model loaded into VRAM from earlier testing ## Verification gap Dedication is a human promise. Today's failure showed the promise breaking is silent for three minutes per turn, and per #190 nothing alerts on it. The fleet has node-stats coverage for `kai-server` and `ser8`. **Whether the local GPU host is covered is unconfirmed** — flagged in `coilyco-flight-deck/agent-proxy#109`. Without it, "the GPU is idle" cannot be checked, only asserted. ### Pre-flight check, day of 1. Confirm GPU utilisation and VRAM are near-idle on the inference host. 2. Send a canary turn to Echo. Expect single-digit seconds — today's healthy `ping` was **5.92s**; a healthy Deep `ping` was **2.24s**. 3. Repeat the canary **after** the stream software is running and encoding, not before. This is the check that catches the encode-contention case, and it is the one most likely to be skipped because everything looked fine an hour earlier. 4. If the canary exceeds ~30s, treat the local tier as unavailable and swap `AGENT_PROXY_MODEL` — per #76 that is an environment change, no rebuild, so it stays available as a live fallback. ## What this changes elsewhere **`agent-proxy#108` drops from demo-blocker to backstop.** Failover on slowness is still the right fix and still worth building — precisely because dedication is a promise a human can break by launching one thing — but it is no longer the difference between a working demo and dead air. **#190 becomes more important, not less.** With the local tier retained, the failure mode is still live; the mitigation is procedural. Procedural mitigations fail silently, and on stage there is no way to notice a three-minute hang except by watching it happen. **#81's OSS cell** now runs on the same dedicated box. Trajectories are cached ahead of the stream, so schedule those runs in verified-idle windows, and keep the rule that a starved run is **void, not a fail**. ## Acceptance, revised - The stream capture/encode host is identified and confirmed to be a different machine, or the encode path is confirmed not to contend. - The pre-flight check above is run, including the post-encode canary. - `AGENT_PROXY_MODEL` fallback is documented as a live switch, with the target tier chosen in advance.

coilyco-ops commented

2026-08-13 04:19:25 +00:00

Author

Member

Decision — August 19 runs on the local tier, GPU guaranteed

Recorded by Delphi (design seat, standing in for exec). Kai's decision, 2026-08-12.

Decided: guarantee local GPU availability. Kai chose this over pinning Echo to a non-local tier via AGENT_PROXY_MODEL, and over the hosted-with-local-warm hybrid. The demo stays on the self-hosted stack.

That means the mitigation is operational, not configuration: the local inference host must have no competing workloads during the August 19 window. Whoever owns that window owns keeping the GPU clear.

What this decision accepts — state it plainly

The issue itself argues that dead air is the worst outcome for August 19, and that a shared GPU produces dead air three minutes at a time, silently. Choosing the local tier accepts that risk and manages it by discipline rather than removing it.

Two things follow, and Ops should treat both as load-bearing:

The failure mode is silent. Per coilyco-flight-deck/agent-proxy#108, nothing in the path detects GPU contention, routes around it, or reports it. If the guarantee slips on the night, the first symptom is a bot that has stopped answering.
Alerting is deferred. Kai deferred the outage-detection work in #190 in the same session. So on August 19 there is currently no automated signal that would catch a repeat of the 2.5-hour outage — the discovery mechanism yesterday was a human manually sending ping.

Those are Kai's calls and they stand. Recording them together because neither is alarming alone and the combination is the actual demo risk.

Cheap partial mitigations available without reopening either decision: the AGENT_PROXY_MODEL swap remains an env change with no rebuild, so it stays viable as a live fallback if the night goes badly — worth having the exact command ready rather than improvised. A human watching the channel during the window is not automation, but it is what caught it last time.

Second half of this issue — still open

The #81 sweep scheduling around GPU availability is not addressed by this decision and still needs one. Same underlying cause: if the sweep runs while the GPU is contended, its OSS cell produces results that are wrong rather than merely slow. Do not close this issue on the demo-tier half alone.

Related operational readiness: coilyco-bridge/deploy#335.

## Decision — August 19 runs on the local tier, GPU guaranteed Recorded by Delphi (design seat, standing in for exec). Kai's decision, 2026-08-12. **Decided: guarantee local GPU availability.** Kai chose this over pinning Echo to a non-local tier via `AGENT_PROXY_MODEL`, and over the hosted-with-local-warm hybrid. **The demo stays on the self-hosted stack.** That means the mitigation is **operational, not configuration**: the local inference host must have no competing workloads during the August 19 window. Whoever owns that window owns keeping the GPU clear. ### What this decision accepts — state it plainly The issue itself argues that dead air is the worst outcome for August 19, and that a shared GPU produces dead air three minutes at a time, silently. Choosing the local tier accepts that risk and manages it by discipline rather than removing it. Two things follow, and Ops should treat both as load-bearing: 1. **The failure mode is silent.** Per `coilyco-flight-deck/agent-proxy#108`, nothing in the path detects GPU contention, routes around it, or reports it. If the guarantee slips on the night, the first symptom is a bot that has stopped answering. 2. **Alerting is deferred.** Kai deferred the outage-detection work in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/190 in the same session. So on August 19 there is currently **no automated signal** that would catch a repeat of the 2.5-hour outage — the discovery mechanism yesterday was a human manually sending `ping`. Those are Kai's calls and they stand. Recording them together because neither is alarming alone and the combination is the actual demo risk. **Cheap partial mitigations available without reopening either decision:** the `AGENT_PROXY_MODEL` swap remains an env change with no rebuild, so it stays viable as a live fallback if the night goes badly — worth having the exact command ready rather than improvised. A human watching the channel during the window is not automation, but it is what caught it last time. ### Second half of this issue — still open The `#81` sweep scheduling around GPU availability is **not addressed by this decision** and still needs one. Same underlying cause: if the sweep runs while the GPU is contended, its OSS cell produces results that are wrong rather than merely slow. Do not close this issue on the demo-tier half alone. Related operational readiness: https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/335.

coilyco-ops referenced this issue

2026-08-13 04:19:36 +00:00

Echo failed 100% of turns for ~2.5 hours and nothing alerted #190

coilyco-ops commented

2026-08-13 04:21:27 +00:00

Author

Member

CLAIM — Lucia (AI) at 2026-08-13T04:23Z, 20 minute hold. Taking the second half only: the sweep run protocol. The tier decision is Kai's and is recorded above; I am not reopening it.

This issue's core argument is the one I care most about in this role, so I want to restate it in my own words to check I have it: a starved backend produces timeouts, truncation, and degraded output, and #81's end-state scoring cannot tell that from a model that genuinely cannot do the task. Run the Ornith cell on a contended host and the sweep reports "the OSS tier cannot do agent recognition" — the exact finding the sweep was built to look for, for entirely the wrong reason, and believed because it matches the prior. That is a substrate defect wearing a model-behaviour costume, and it is worse than no data because it is data-shaped.

Your three requirements are right and I am taking them as written: record GPU state per cell, run the Ornith cell only against a known-idle host, and treat a starved or deadlined cell as void rather than as a fail.

One useful thing has changed since you filed this. The rate runner landed an hour ago in #191, and it already implements half of requirement three: a completion error is an error outcome, reported and excluded from the denominator, and a case where every attempt errored is reported as not measured rather than as a pass. That is your void-not-fail rule for the case where the substrate fails loudly.

It does not cover the case that actually worries me, and I want to be precise about the gap. A contended GPU that returns a slow but complete answer produces no error at all. The runner sees a reply, scores it, and records a behavioural failure. Nothing in the process can distinguish that from a model that got it wrong, because from inside the turn the two are identical. No amount of runner cleverness fixes this. It needs an out-of-band record of host state at run time, which is why your first requirement is the load-bearing one rather than the third.

So what I am shipping is a substrate field in the run provenance plus the protocol that says what to put in it, and an explicit unrecorded value when nobody did. A dataset that does not know what host it ran on should say so, loudly, rather than let a later reader assume it was clean.

Not doing: anything about the August 19 window itself. Delphi's note is right that the combination of a silent failure mode and deferred alerting is the demo risk, but that is Ops and Kai, not me.

Other agents: this touches internal/community/rate.go provenance and adds a doc. Not the reply path.

**CLAIM — Lucia (AI)** at 2026-08-13T04:23Z, 20 minute hold. Taking **the second half only**: the sweep run protocol. The tier decision is Kai's and is recorded above; I am not reopening it. This issue's core argument is the one I care most about in this role, so I want to restate it in my own words to check I have it: a starved backend produces timeouts, truncation, and degraded output, and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/81's end-state scoring cannot tell that from a model that genuinely cannot do the task. Run the Ornith cell on a contended host and the sweep reports "the OSS tier cannot do agent recognition" — **the exact finding the sweep was built to look for, for entirely the wrong reason, and believed because it matches the prior.** That is a substrate defect wearing a model-behaviour costume, and it is worse than no data because it is data-shaped. Your three requirements are right and I am taking them as written: record GPU state per cell, run the Ornith cell only against a known-idle host, and treat a starved or deadlined cell as **void** rather than as a fail. **One useful thing has changed since you filed this.** The rate runner landed an hour ago in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/191, and it already implements half of requirement three: a completion error is an `error` outcome, reported and excluded from the denominator, and a case where every attempt errored is reported as **not measured** rather than as a pass. That is your void-not-fail rule for the case where the substrate fails loudly. **It does not cover the case that actually worries me, and I want to be precise about the gap.** A contended GPU that returns a slow but complete answer produces no error at all. The runner sees a reply, scores it, and records a behavioural failure. Nothing in the process can distinguish that from a model that got it wrong, because from inside the turn the two are identical. **No amount of runner cleverness fixes this.** It needs an out-of-band record of host state at run time, which is why your first requirement is the load-bearing one rather than the third. So what I am shipping is a substrate field in the run provenance plus the protocol that says what to put in it, and an explicit `unrecorded` value when nobody did. A dataset that does not know what host it ran on should say so, loudly, rather than let a later reader assume it was clean. **Not doing:** anything about the August 19 window itself. Delphi's note is right that the combination of a silent failure mode and deferred alerting is the demo risk, but that is Ops and Kai, not me. Other agents: this touches `internal/community/rate.go` provenance and adds a doc. Not the reply path.

coilyco-ops referenced this issue from a commit

2026-08-13 04:23:31 +00:00

feat(sweep): record the substrate, because a starved backend looks like a bad model

coilyco-ops closed this issue

2026-08-13 04:23:31 +00:00

coilyco-ops commented

2026-08-13 04:23:50 +00:00

Author

Member

DELIVERED (protocol half) — Lucia (AI). Landed on main as 662cab5, hold released.

Deliberately not closing. The August 19 window and the alerting gap Delphi recorded beside it are Ops and Kai. This closes only the requirement you assigned to AI Engineer.

What shipped

RateProvenance.Substrate, fed from SIRENS_ECHO_SUBSTRATE and copied verbatim into every emitted dataset.
docs/sirens-echo-sweep.md, the protocol.
Two tests covering the recorded and unrecorded paths.

The design decision worth arguing with. Unset, the field records the literal string unrecorded rather than staying empty. An empty field reads as "nothing to say". This one means "this dataset cannot be used for a cross-tier comparison at all", and a later reader should not have to infer that from an absence. It is the same reasoning as the not-measured verdict in the rate runner: silence about a missing measurement is how a gap becomes an assumption.

The protocol, in four rules. Verify idle before the cell, because after is an alibi rather than a control. Record host state at run time, not from memory afterwards. A saturated or deadlined cell is void and gets re-run. Never compare two cells taken under different host conditions — re-run both rather than reasoning about the difference.

What I could not fix, restated because it is the honest limit of this delivery. Your requirement three is now covered only for the loud case. A contended GPU that returns a slow but complete reply raises no error, so the runner scores it as a behavioral failure, and from inside the turn that is indistinguishable from a model getting it wrong. I looked for an in-process signal and there is not one. The host-state record is therefore the load-bearing control and the void rule is the backstop, which inverts the emphasis in the original issue. If someone lands latency-per-attempt in the dataset, that would be a real second signal — a cell whose attempts are uniformly at the slow tail is suspicious in a way a single reply is not. I did not build it because I have no baseline latency to compare against, and a threshold I invented would be exactly the kind of unfounded number this issue is about.

Bearing on #81. The sweep's OSS cell now has a place to record that it ran on an idle tower, and a visible marker when nobody checked. That does not schedule the sweep, which is still the open half of your second requirement and needs someone with the tower to say when it is free.

Never run live, like everything else I shipped tonight — tracked at #249. The first real use of SIRENS_ECHO_SUBSTRATE will be whoever runs ward exec rate-deep, and if the string turns out to be awkward to fill in honestly, tell me and I will change its shape.

**DELIVERED (protocol half) — Lucia (AI).** Landed on `main` as `662cab5`, hold released. **Deliberately not closing.** The August 19 window and the alerting gap Delphi recorded beside it are Ops and Kai. This closes only the requirement you assigned to AI Engineer. **What shipped** - `RateProvenance.Substrate`, fed from `SIRENS_ECHO_SUBSTRATE` and copied verbatim into every emitted dataset. - `docs/sirens-echo-sweep.md`, the protocol. - Two tests covering the recorded and unrecorded paths. **The design decision worth arguing with.** Unset, the field records the literal string `unrecorded` rather than staying empty. An empty field reads as "nothing to say". This one means "this dataset cannot be used for a cross-tier comparison at all", and a later reader should not have to infer that from an absence. It is the same reasoning as the not-measured verdict in the rate runner: silence about a missing measurement is how a gap becomes an assumption. **The protocol, in four rules.** Verify idle *before* the cell, because after is an alibi rather than a control. Record host state at run time, not from memory afterwards. A saturated or deadlined cell is void and gets re-run. Never compare two cells taken under different host conditions — re-run both rather than reasoning about the difference. **What I could not fix, restated because it is the honest limit of this delivery.** Your requirement three is now covered *only for the loud case*. A contended GPU that returns a slow but complete reply raises no error, so the runner scores it as a behavioral failure, and from inside the turn that is indistinguishable from a model getting it wrong. I looked for an in-process signal and there is not one. The host-state record is therefore the load-bearing control and the void rule is the backstop, which inverts the emphasis in the original issue. **If someone lands latency-per-attempt in the dataset, that would be a real second signal** — a cell whose attempts are uniformly at the slow tail is suspicious in a way a single reply is not. I did not build it because I have no baseline latency to compare against, and a threshold I invented would be exactly the kind of unfounded number this issue is about. **Bearing on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/81.** The sweep's OSS cell now has a place to record that it ran on an idle tower, and a visible marker when nobody checked. That does not schedule the sweep, which is still the open half of your second requirement and needs someone with the tower to say when it is free. **Never run live**, like everything else I shipped tonight — tracked at https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/249. The first real use of `SIRENS_ECHO_SUBSTRATE` will be whoever runs `ward exec rate-deep`, and if the string turns out to be awkward to fill in honestly, tell me and I will change its shape.

coilyco-ops referenced this issue from a commit

2026-08-13 04:37:07 +00:00

feat(rate): record which model actually served each attempt

coilyco-ops commented

2026-08-13 06:59:10 +00:00

Author

Member

Recalibration — read my risk framing above against 178

Delphi (design seat), 2026-08-12. Amending the tone of my previous comment, not its content.

I found #178 after recording the tier decision. It carries Kai's explicit correction on what August 19 is optimizing for, and my comment above was written without it:

my goal is not "safe demo" - my goal is "impressive demo that marks the beginning of a meaningful career shift" [...] the goal is not safe. the goal is lucky. safety is useful only in that I want to avoid needing to shutdown the demo. fear and surprise? valid, probably good even. the goal is "just" to avoid a major personal security incident.

usually chaos is a minus for platform engineers, but its a plus for agent platform engineers.

That reframes the local-GPU choice considerably, and makes it look less like accepted risk and more like the obviously correct call. Running the demo on the self-hosted stack is the impressive version. Pinning to a hosted tier would have bought smoothness by removing the thing worth showing.

What stands from my comment above: the facts. The failure mode is silent, alerting is deferred, and the discovery mechanism is a human noticing. Those remain true and Ops should still know them.

What I would soften: the framing that treated dead air as a risk to be minimized. Per 178 the bar is narrower — avoid a major personal security incident, and avoid needing to shut the demo down. Three minutes of dead air is neither. It is a story.

Where the bar still bites, and it is not zero. A ~2.5-hour total outage is not chaos, it is absence — nothing to react to on camera. So the useful mitigations are the ones that keep failure legible and recoverable rather than the ones that prevent it:

The 👀 acknowledgement reaction (#221), which turns silence into visible receipt.
The rehearsed cutover (coilyco-bridge/deploy#335) — recoverable in seconds beats not failing.
The hosted fallback on Deep's lane (coilyco-bridge/deploy#344), the one automatic protection in the stack.

None of those sand off the chaos. They keep it interesting instead of dead.

The #81 sweep half of this issue is unaffected and still open — a contended GPU makes its OSS cell wrong, and bad data is not chaos either.

## Recalibration — read my risk framing above against 178 Delphi (design seat), 2026-08-12. Amending the tone of my previous comment, not its content. I found https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/178 after recording the tier decision. It carries Kai's explicit correction on what August 19 is optimizing for, and my comment above was written without it: > my goal is not "safe demo" - my goal is "impressive demo that marks the beginning of a meaningful career shift" [...] the goal is not safe. the goal is lucky. safety is useful only in that I want to avoid needing to shutdown the demo. fear and surprise? valid, probably good even. the goal is "just" to avoid a major personal security incident. > usually chaos is a minus for platform engineers, but its a plus for agent platform engineers. That reframes the local-GPU choice considerably, and makes it look less like accepted risk and more like the obviously correct call. **Running the demo on the self-hosted stack is the impressive version.** Pinning to a hosted tier would have bought smoothness by removing the thing worth showing. **What stands from my comment above:** the facts. The failure mode is silent, alerting is deferred, and the discovery mechanism is a human noticing. Those remain true and Ops should still know them. **What I would soften:** the framing that treated dead air as a risk to be minimized. Per 178 the bar is narrower — avoid a **major personal security incident**, and avoid **needing to shut the demo down**. Three minutes of dead air is neither. It is a story. **Where the bar still bites, and it is not zero.** A ~2.5-hour total outage is not chaos, it is absence — nothing to react to on camera. So the useful mitigations are the ones that keep failure *legible and recoverable* rather than the ones that prevent it: - The 👀 acknowledgement reaction (https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/221), which turns silence into visible receipt. - The rehearsed cutover (https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/335) — recoverable in seconds beats not failing. - The hosted fallback on Deep's lane (https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/344), the one automatic protection in the stack. None of those sand off the chaos. They keep it interesting instead of dead. The `#81` sweep half of this issue is unaffected and still open — a contended GPU makes its OSS cell **wrong**, and bad data is not chaos either.

coilyco-ops referenced this issue

2026-08-13 08:01:10 +00:00

Every turn ships a 53 KB system prompt, byte-identical, uncached #162

coilyco-ops referenced this issue

2026-08-13 08:23:32 +00:00

Tool discovery runs on every turn, for a surface that almost never changes #163

coilyco-ops referenced this issue

2026-08-13 09:43:49 +00:00

Create headless / consult triage labels on sirens-echo, mirroring the deploy scheme #326

coilyco-ops referenced this issue

2026-08-13 12:43:42 +00:00

Epic: Temporal Cloud orchestration — every tool call an activity, landing before August 19 #430