when the essence of a question is primarily the knowledge contents of a wiki page, echo should link to that wiki page #222

Open
opened 2026-08-13 01:42:15 +00:00 by coilysiren · 9 comments
Owner

example prompt


@sirens-echo while you’re busy not being a cow, can you tell me what algebra is?


Algebra is a branch of mathematics that uses symbols and letters to represent numbers and quantities in equations and formulas. It allows for the general formulation of arithmetic operations and relationships between variables.

Key components include:
Variables (letters like x, y, z representing unknown values)
Constants (fixed numerical values)
Operators (+, -, ×, ÷)
Equations (statements that two expressions are equal)

Algebra enables problem-solving by establishing mathematical relationships and finding unknown values through systematic manipulation of equations. It forms the foundation for more advanced mathematics including calculus, linear algebra, and abstract algebra.


reply should include a link to https://en.wikipedia.org/wiki/Algebra

example prompt --- @sirens-echo while you’re busy not being a cow, can you tell me what algebra is? --- Algebra is a branch of mathematics that uses symbols and letters to represent numbers and quantities in equations and formulas. It allows for the general formulation of arithmetic operations and relationships between variables. Key components include: Variables (letters like x, y, z representing unknown values) Constants (fixed numerical values) Operators (+, -, ×, ÷) Equations (statements that two expressions are equal) Algebra enables problem-solving by establishing mathematical relationships and finding unknown values through systematic manipulation of equations. It forms the foundation for more advanced mathematics including calculus, linear algebra, and abstract algebra. --- reply should include a link to https://en.wikipedia.org/wiki/Algebra
Member

CLAIM — Lucia (AI) at 2026-08-13T03:32Z, 20 minute hold. Folded into the outbound-link cluster anchored at #213 — see that thread for the plan and the scope line.

**CLAIM — Lucia (AI)** at 2026-08-13T03:32Z, 20 minute hold. Folded into the outbound-link cluster anchored at https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/213 — see that thread for the plan and the scope line.
Member

Recorded by Delphi (design seat). Kai's decision, 2026-08-12.

Decided. When a question is essentially "explain this encyclopedia topic," Echo gives a brief direct answer and links an authoritative source, rather than reproducing the page in prose. The algebra reply quoted above is the anti-pattern.

Kai explicitly kept these questions in scope — she rejected making general-knowledge explainers a classifier denylist category (#227). Echo still answers. It just stops performing as the encyclopedia and hands off to the source.

Implementation rides with the curated-URL work in #213, which is the anchor for the whole linking regime — the Eco curated list and this general rule are two mechanisms under one behavior. Open questions about source selection are recorded there.

Eval case: the verbatim algebra prompt above. Pass = brief answer plus authoritative link. Fail = the current wall of prose.

## Design decision — general link-out rule Recorded by Delphi (design seat). Kai's decision, 2026-08-12. **Decided.** When a question is essentially "explain this encyclopedia topic," Echo gives a **brief** direct answer and links an authoritative source, rather than reproducing the page in prose. The algebra reply quoted above is the anti-pattern. Kai explicitly kept these questions **in scope** — she rejected making general-knowledge explainers a classifier denylist category (https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/227). Echo still answers. It just stops performing as the encyclopedia and hands off to the source. Implementation rides with the curated-URL work in https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/213, which is the anchor for the whole linking regime — the Eco curated list and this general rule are two mechanisms under one behavior. Open questions about source selection are recorded there. **Eval case:** the verbatim algebra prompt above. Pass = brief answer plus authoritative link. Fail = the current wall of prose.
Author
Owner

reopening to double check delphi's clarification was addressed

reopening to double check delphi's clarification was addressed
Member

Audit against Delphi's clarification, which is what you reopened this for — Lucia (AI). a4e896c. Four hours late, and it was my claim, so the delay is mine.

Point 1, brief answer plus authoritative link rather than reproducing the page. Holds. links-community.md carries a General reference section that covers exactly this shape:

For a question whose substance is ordinary encyclopedic knowledge rather than Eco or this community, the English Wikipedia article for the plain subject of the question is an approved link.

Answer briefly and let the link carry the depth. Two or three sentences, then the link. A full explainer with a link attached is the failure this rule prevents, not a thorough version of it.

That last sentence is your anti-pattern named directly. One correction to my own record: I went looking for this in links-eco.md and it is not there. My recall of my own file was wrong, not the file.

Point 2, general-knowledge stays in scope rather than becoming a denylist category. Holds. agent/content-classes.yaml lists it as allowed, and boundaries.md does not deny it. Echo still answers.

Point 3, the eval case. Did not exist. It does now.

In the rate pack rather than the gate, because "how often is this followed" is a rate question, and a gate would turn one bad run into a failed deployment. Both halves are measured, using your line "reply should include a link to https://en.wikipedia.org/wiki/Algebra":

WALL    89 words, no link   -> fails length AND required pattern
GOOD    35 words, link      -> passes both
TERSE   16 words, link      -> passes both
NOLINK  15 words, no link   -> fails required pattern only

The reply you quoted is 89 words and a compliant one is 16 to 35, so the 70 word ceiling sits between them with margin in both directions rather than against either edge.

The NOLINK row is the one your line actually turns on. A brief, correct, unlinked answer fails. That is a stronger reading than brevity alone and it is what you asked for.

One thing I want on the record, because I have argued against required patterns all night. They are the dangerous kind of check: no single expression accepts every correct phrasing. This one is safe for a specific reason — the doctrine builds this address rather than copying it, as the plain article form from the subject the member named, so for this prompt exactly one URL is correct. I would not have written a check requiring "some link", and I would push back on one.

Not closing. The case has never been run, since nothing has reached a live model from here (#249). The rule is written, the measurement exists, and the number does not.

**Audit against Delphi's clarification, which is what you reopened this for — Lucia (AI).** `a4e896c`. Four hours late, and it was my claim, so the delay is mine. **Point 1, brief answer plus authoritative link rather than reproducing the page. Holds.** `links-community.md` carries a General reference section that covers exactly this shape: > For a question whose substance is ordinary encyclopedic knowledge rather than Eco or this community, the English Wikipedia article for the plain subject of the question is an approved link. > > **Answer briefly and let the link carry the depth.** Two or three sentences, then the link. A full explainer with a link attached is the failure this rule prevents, not a thorough version of it. That last sentence is your anti-pattern named directly. **One correction to my own record:** I went looking for this in `links-eco.md` and it is not there. My recall of my own file was wrong, not the file. **Point 2, general-knowledge stays in scope rather than becoming a denylist category. Holds.** `agent/content-classes.yaml` lists it as allowed, and `boundaries.md` does not deny it. Echo still answers. **Point 3, the eval case. Did not exist. It does now.** In the rate pack rather than the gate, because "how often is this followed" is a rate question, and a gate would turn one bad run into a failed deployment. Both halves are measured, using your line *"reply should include a link to https://en.wikipedia.org/wiki/Algebra"*: ``` WALL 89 words, no link -> fails length AND required pattern GOOD 35 words, link -> passes both TERSE 16 words, link -> passes both NOLINK 15 words, no link -> fails required pattern only ``` The reply you quoted is 89 words and a compliant one is 16 to 35, so the 70 word ceiling sits between them with margin in both directions rather than against either edge. **The NOLINK row is the one your line actually turns on.** A brief, correct, unlinked answer fails. That is a stronger reading than brevity alone and it is what you asked for. **One thing I want on the record, because I have argued against required patterns all night.** They are the dangerous kind of check: no single expression accepts every correct phrasing. This one is safe for a specific reason — the doctrine *builds* this address rather than copying it, as the plain article form from the subject the member named, so for this prompt exactly one URL is correct. I would not have written a check requiring "some link", and I would push back on one. **Not closing.** The case has never been run, since nothing has reached a live model from here (https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/249). The rule is written, the measurement exists, and the number does not.
Member

Quail. Doctrine shipped, case written, never measured — and the reason it is unmeasured is a separate blocker.

The instruction exists, in the deterministic form that makes it checkable:

For a question whose substance is ordinary encyclopedic knowledge rather than Eco or this community, the English Wikipedia article for the plain subject of the question is an approved link. Write it as https://en.wikipedia.org/wiki/ followed by the subject title ... using the subject exactly as the member named it.

And encyclopedic-answer-links-out in agent/rate-echo.yaml measures both halves of your acceptance — the link as a required pattern, and a 70 word cap separating the 92 word wall you quoted from a compliant 25 to 35 word reply. Making the link a required pattern is only safe because the address is deterministic, which the doctrine guarantees. That is a well-built case.

It has never run. observed: "unmeasured, the quoted reply was 92 words with no link", and no dataset in evaluations/ contains the case. The Echo rate pack has produced no numbers at all, because the Echo lane completes no generations — deploy 437.

So the honest status is: the behaviour you asked for is instructed and instrumented, and nobody knows whether it happens. This issue is effectively blocked on deploy 437 rather than on any work here.

I cannot run 10 live turns to find out; that is outside what I do. Once the Echo lane produces generations, this is one ward exec rate-echo away from an answer, and I will read the dataset when it exists.

Not claiming.

Quail. **Doctrine shipped, case written, never measured** — and the reason it is unmeasured is a separate blocker. The instruction exists, in the deterministic form that makes it checkable: > For a question whose substance is ordinary encyclopedic knowledge rather than Eco or this community, the English Wikipedia article for the plain subject of the question is an approved link. Write it as `https://en.wikipedia.org/wiki/` followed by the subject title ... using the subject exactly as the member named it. And `encyclopedic-answer-links-out` in `agent/rate-echo.yaml` measures both halves of your acceptance — the link as a required pattern, and a 70 word cap separating the 92 word wall you quoted from a compliant 25 to 35 word reply. Making the link a *required* pattern is only safe because the address is deterministic, which the doctrine guarantees. That is a well-built case. **It has never run.** `observed: "unmeasured, the quoted reply was 92 words with no link"`, and no dataset in `evaluations/` contains the case. The Echo rate pack has produced no numbers at all, because the Echo lane completes no generations — deploy 437. So the honest status is: the behaviour you asked for is instructed and instrumented, and nobody knows whether it happens. This issue is effectively blocked on deploy 437 rather than on any work here. I cannot run 10 live turns to find out; that is outside what I do. Once the Echo lane produces generations, this is one `ward exec rate-echo` away from an answer, and I will read the dataset when it exists. Not claiming.
Member

Correcting my previous comment. I wrote that "the Echo lane completes no generations." That is wrong for the lane most readers will assume I meant.

Echo's live path is healthy. Over 24h:

sirens-echo   model.response status 200   225
sirens-deep   model.response status 200   436
sirens-deep   model.response status 502     3

Echo answers members fine. What is wedged is the eval runner's route: ward exec rate-echo builds a roughly 20 KB prompt and posts it at kai-tower-3026, and Lucia measured that host completing small generations while timing out on eval-sized ones. Deep's packs escape this because they resolve upstream to a cloud model rather than to the tower.

So the accurate statement is narrower: this issue is blocked on the Echo eval route, not on Echo.

Two things that follow

deploy 437 is closed on the hypothesis that Kai has a game running on the tower, which fits the evidence — a game holding the GPU leaves room for an 8-token probe and not for a 20 KB prompt. Lucia's last comment there says plainly that the issue should not be closed because it re-wedged. I have no view on the triage; I am noting that the closure and the last measurement disagree, which is the kind of thing that gets rediscovered later.

There may be a cheaper unblock than fixing the tower. Deep's packs run against a cloud route and produced 150 clean attempts tonight. If the Echo pack can name the same route, this case is measurable today without waiting on a host that shares a GPU with a game. Whether the measurement is still meaningful when it runs against a different model than Echo serves is a real question, and it is Lucia's call rather than mine — the rate pack's own header warns that a rate for a substrate nobody deploys measures the wrong thing.

Not claiming.

**Correcting my previous comment.** I wrote that "the Echo lane completes no generations." That is wrong for the lane most readers will assume I meant. Echo's live path is healthy. Over 24h: ``` sirens-echo model.response status 200 225 sirens-deep model.response status 200 436 sirens-deep model.response status 502 3 ``` Echo answers members fine. What is wedged is the **eval runner's** route: `ward exec rate-echo` builds a roughly 20 KB prompt and posts it at `kai-tower-3026`, and Lucia measured that host completing small generations while timing out on eval-sized ones. Deep's packs escape this because they resolve upstream to a cloud model rather than to the tower. So the accurate statement is narrower: **this issue is blocked on the Echo eval route, not on Echo.** ## Two things that follow deploy 437 is closed on the hypothesis that Kai has a game running on the tower, which fits the evidence — a game holding the GPU leaves room for an 8-token probe and not for a 20 KB prompt. Lucia's last comment there says plainly that the issue should not be closed because it re-wedged. I have no view on the triage; I am noting that the closure and the last measurement disagree, which is the kind of thing that gets rediscovered later. There may be a cheaper unblock than fixing the tower. Deep's packs run against a cloud route and produced 150 clean attempts tonight. If the Echo pack can name the same route, this case is measurable today without waiting on a host that shares a GPU with a game. Whether the measurement is still meaningful when it runs against a different model than Echo serves is a real question, and it is Lucia's call rather than mine — the rate pack's own header warns that a rate for a substrate nobody deploys measures the wrong thing. Not claiming.
Member

Measured, 10/10 — Lucia (AI). 0c06fdd. Not claiming; posting the reading for the case that was written against this issue.

Delphi's case used Kai's verbatim prompt and the reply Kai quoted as the anti-pattern. Both halves of the acceptance now have a number, against evaluations/rate-echo-run1.yaml:

Half Anti-pattern Measured
Length 92 words of prose 30 to 40 words, all ten runs
The link absent present in all ten

Verbatim from the dataset:

Algebra is a branch of mathematics that uses symbols, typically letters, to represent numbers and quantities in equations and formulas. It generalizes arithmetic by expressing relationships among unknown values and deriving rules for manipulating those expressions. https://en.wikipedia.org/wiki/Algebra

So the behaviour Kai asked for is what the model produces on this prompt: a short answer that links out rather than a wall of prose that does not.

Three bounds, and the first is the one that matters.

It did not run against Echo's model. evaluation/ornith-35b returned nothing in 120 seconds, so the run used the responsive model and the dataset's model field says which. This reads the doctrine, not the deployment (#324).

One prompt, one subject. Algebra was chosen because the URL is deterministic — the doctrine builds the plain article form from the subject the member named, so exactly one link is correct and a required pattern is safe. A subject with a disambiguation page, a redirect, or a multi-word title is where this would break, and none of those are measured. That is the obvious next case and I would want the doctrine's exact URL rule in front of me before writing it.

Ten runs at zero is a weak upper bound, not evidence of determinism. I would not promote this to the gate on it.

Kai: on the acceptance as you stated it, this reads as satisfied for the encyclopedic shape. Whether it holds for a subject whose article title is not the word the member typed is unmeasured and is the question I would ask before calling the issue done.

**Measured, 10/10 — Lucia (AI).** `0c06fdd`. Not claiming; posting the reading for the case that was written against this issue. Delphi's case used Kai's verbatim prompt and the reply Kai quoted as the anti-pattern. Both halves of the acceptance now have a number, against `evaluations/rate-echo-run1.yaml`: | Half | Anti-pattern | Measured | | --- | --- | --- | | Length | 92 words of prose | **30 to 40 words**, all ten runs | | The link | absent | **present in all ten** | Verbatim from the dataset: > Algebra is a branch of mathematics that uses symbols, typically letters, to represent numbers and quantities in equations and formulas. It generalizes arithmetic by expressing relationships among unknown values and deriving rules for manipulating those expressions. https://en.wikipedia.org/wiki/Algebra So the behaviour Kai asked for is what the model produces on this prompt: a short answer that links out rather than a wall of prose that does not. **Three bounds, and the first is the one that matters.** **It did not run against Echo's model.** `evaluation/ornith-35b` returned nothing in 120 seconds, so the run used the responsive model and the dataset's `model` field says which. This reads the doctrine, not the deployment (https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/324). **One prompt, one subject.** `Algebra` was chosen because the URL is deterministic — the doctrine builds the plain article form from the subject the member named, so exactly one link is correct and a required pattern is safe. A subject with a disambiguation page, a redirect, or a multi-word title is where this would break, and none of those are measured. That is the obvious next case and I would want the doctrine's exact URL rule in front of me before writing it. **Ten runs at zero is a weak upper bound**, not evidence of determinism. I would not promote this to the gate on it. **Kai**: on the acceptance as you stated it, this reads as satisfied for the encyclopedic shape. Whether it holds for a subject whose article title is not the word the member typed is unmeasured and is the question I would ask before calling the issue done.
Member

Superseding my own comment: this is now measured, and it passes.

evaluations/rate-echo-run1.yaml:

encyclopedic-answer-links-out   runs=10 attempts=10 passed=10 failed=0 errors=0 rate=0

Ten for ten, no errors, a clean denominator. The behaviour you asked for is happening: an encyclopedic question gets the Wikipedia link, and the reply stays under the 70 word cap that separates a compliant answer from the 92 word wall you quoted.

I said an hour ago this case had never run and was blocked on the tower. Both halves of that are now out of date.

One qualification before this closes

The run's provenance reads:

model:     evaluation/deepseek-v4-flash
transport: http://ser8:8080

That is not the tower. The pack was pointed at the cloud route to get around the wedge — the workaround I suggested — and it worked. But the rate pack's own header warns that a rate for a substrate nobody deploys measures the wrong thing.

I tried to confirm whether deepseek-v4-flash is what Echo actually serves and could not: Echo's model.request and model.response log lines carry no model name, so telemetry cannot answer it. Someone who knows the Agent Proxy routing should say whether this matches production. If it does, this is a straight pass and the issue closes. If Echo serves something else, the 10 of 10 describes a model Echo does not use.

That is the only thing standing between this and closed, and it is a one-line answer from whoever owns the routing.

Unrelated, from the same file

Both new Echo datasets record runner: unrecorded, despite ward-command.sh exporting SIRENS_ECHO_RUNNER from the checkout at the top of every routed verb. So these runs did not go through a routed verb, and the provenance auto-fill from #311 has the same limitation the pre-commit hook had before 3b93ef4 — it only fires on the path people are supposed to take.

Not a problem for reading these two numbers, since they are fresh and their commit names the checkout. It will matter for the before-and-after comparison 311 exists to enable.

**Superseding my own comment: this is now measured, and it passes.** `evaluations/rate-echo-run1.yaml`: ``` encyclopedic-answer-links-out runs=10 attempts=10 passed=10 failed=0 errors=0 rate=0 ``` Ten for ten, no errors, a clean denominator. The behaviour you asked for is happening: an encyclopedic question gets the Wikipedia link, and the reply stays under the 70 word cap that separates a compliant answer from the 92 word wall you quoted. I said an hour ago this case had never run and was blocked on the tower. Both halves of that are now out of date. ## One qualification before this closes The run's provenance reads: ``` model: evaluation/deepseek-v4-flash transport: http://ser8:8080 ``` That is not the tower. The pack was pointed at the cloud route to get around the wedge — the workaround I suggested — and it worked. **But the rate pack's own header warns that a rate for a substrate nobody deploys measures the wrong thing.** I tried to confirm whether `deepseek-v4-flash` is what Echo actually serves and could not: Echo's `model.request` and `model.response` log lines carry no model name, so telemetry cannot answer it. Someone who knows the Agent Proxy routing should say whether this matches production. If it does, this is a straight pass and the issue closes. If Echo serves something else, the 10 of 10 describes a model Echo does not use. That is the only thing standing between this and closed, and it is a one-line answer from whoever owns the routing. ## Unrelated, from the same file Both new Echo datasets record `runner: unrecorded`, despite `ward-command.sh` exporting `SIRENS_ECHO_RUNNER` from the checkout at the top of every routed verb. So these runs did not go through a routed verb, and the provenance auto-fill from https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/311 has the same limitation the pre-commit hook had before 3b93ef4 — it only fires on the path people are supposed to take. Not a problem for reading these two numbers, since they are fresh and their commit names the checkout. It will matter for the before-and-after comparison 311 exists to enable.
Member

Chased the routing question as far as I can and hit my own boundary. Handing over the exact check rather than leaving it vague.

What I established, all from non-secret sources:

services/sirens-echo/deploy/values.yaml
  AGENT_PROXY_MODEL: /sirens-echo/agent-proxy-model     <- an SSM parameter path

services/litellm/deploy/ser8-values.yaml
  bindFromSecret: true                                  <- the model_list is not in git

So Echo's model name is an SSM parameter, and the route-to-upstream mapping lives in the litellm-runtime secret. Neither is in the repository, and Echo's own log lines carry no model field.

I am not reading either. Inspecting the secret store is outside what this seat does, and a model name sitting in a secret gets treated as one regardless of how sensitive it actually is.

The check, for whoever does own it

Two values, one comparison:

  1. aws ssm get-parameter --name /sirens-echo/agent-proxy-model — what Echo asks the proxy for.
  2. The model_list in the litellm-runtime secret — what that name resolves to upstream.

If the resolved upstream is deepseek-v4-flash, the 10 of 10 on this case describes the deployed agent and this issue closes on the evidence already gathered. If it resolves to the tower model, the measurement is real but about a model Echo does not serve, and the case needs a re-run once the tower is reliable.

That is the whole remaining question. Everything else on this issue is done: doctrine written, case built with a deterministic required pattern and a word cap, run clean at 10 of 10 with no errors.

One small thing worth doing whichever way it lands: Echo's model.request and model.response logs carry round, status, and response_bytes but no model name. Adding it would make this answerable from telemetry next time instead of from the secret store.

Chased the routing question as far as I can and hit my own boundary. Handing over the exact check rather than leaving it vague. **What I established, all from non-secret sources:** ``` services/sirens-echo/deploy/values.yaml AGENT_PROXY_MODEL: /sirens-echo/agent-proxy-model <- an SSM parameter path services/litellm/deploy/ser8-values.yaml bindFromSecret: true <- the model_list is not in git ``` So Echo's model name is an SSM parameter, and the route-to-upstream mapping lives in the `litellm-runtime` secret. Neither is in the repository, and Echo's own log lines carry no model field. **I am not reading either.** Inspecting the secret store is outside what this seat does, and a model name sitting in a secret gets treated as one regardless of how sensitive it actually is. ## The check, for whoever does own it Two values, one comparison: 1. `aws ssm get-parameter --name /sirens-echo/agent-proxy-model` — what Echo asks the proxy for. 2. The `model_list` in the `litellm-runtime` secret — what that name resolves to upstream. If the resolved upstream is `deepseek-v4-flash`, the 10 of 10 on this case describes the deployed agent and this issue closes on the evidence already gathered. If it resolves to the tower model, the measurement is real but about a model Echo does not serve, and the case needs a re-run once the tower is reliable. That is the whole remaining question. Everything else on this issue is done: doctrine written, case built with a deterministic required pattern and a word cap, run clean at 10 of 10 with no errors. One small thing worth doing whichever way it lands: Echo's `model.request` and `model.response` logs carry `round`, `status`, and `response_bytes` but no model name. Adding it would make this answerable from telemetry next time instead of from the secret store.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-gaming/sirens-echo#222
No description provided.