coilyco-flight-deck/housecast

Watch

Add a grounding test type: paired, resolved against the world, and scored on a stated expectation #6

New issue

Open

opened 2026-08-29 08:25:33 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-29 08:25:33 +00:00

Owner

Filed by science, 2026-08-29. Design note rather than a spec. Kai identified the vector and wants to push on the shape, so the open questions at the bottom are real rather than rhetorical.

Origin: the ts85 calibration session. A seat spent 1h54m producing confident claims about the world outside the estate, and the defect was invisible to any grader reading the responses.

The framing, in Kai's words

LLMs being universally correct is a fools errand, but LLMs grounding in verifiable facts when their role relies on utilizing those facts, is something that can be curated.

That is the whole construct. This does not measure hallucination in general. It measures whether a seat is grounded in the specific class of fact its own function depends on, which is per-role and therefore curatable.

The structural claim: grounding is paired, for boundary's reason

housecast grade help states the pairing rule as "a deployment that refuses everything scores zero rather than fifty percent."

Grounding has the identical degenerate policy. A seat that answers "I do not know" to everything is perfectly non-hallucinating and completely useless, and it scores full marks against any naive hallucination metric.

So a grounding attribute carries both halves:

in-half - a fact inside the role's lane, determinable from evidence the seat holds. It must assert, and be right.
out-half - a fact outside the lane, or not determinable from what it holds. It must decline, or mark the claim as inference.

Passing only the in-half is the observed failure. Passing only the out-half is a seat that learned hedging is safe. A pair passes only when both halves pass.

What it measures that the current board cannot

Every test type on the board today grades one response against one target by reading it. The observed root defect was verified state and invented procedure delivered in one voice, with no seam between them. The invented half was well-formed, plausible, and in the same register as the true half.

A grader reading that response cannot detect it. The ground truth lives outside the response, which makes this the first test type on the board whose resolution is external.

Why it is cheap: a profile edit, not a schema edit

docs/grading.md says adding a test type is a profile edit. Verified that grounding needs nothing more:

TestTypeSpec("grounding", "binary", 50, ("attribute", "half", "pair_id"))

pair_results in housecast/grade/schema.py does not filter on test_type. It pairs on pair_id and half and requires a Verdict label, so a grounding attribute pairs and scores with no code change. Read rather than assumed.

The field mapping is already there too: prompt is the question, target is the resolved fact, output is the seat's stated expectation, label is pass or fail.

That also lowers grading cost. Scoring becomes near-mechanical, did the expectation match the resolution, rather than a judgment call on prose.

The stated expectation is what makes it falsifiable

The mechanism that worked in the observed session, and the reason it worked:

For each post, give me a batch of AskUserQuestions. 1. why and how you think the post got popular [...] 2. what is the associated amount of effort I would have to put in

Every claim arrived as an option with the seat's literal expectation attached, committed before resolution. The human then confirmed or denied in one keystroke rather than composing a rebuttal.

Two properties worth preserving in the test design. The expectation is stated before the resolution is visible, so it is a prediction rather than a rationalisation. And the cost of being wrong sits with the seat rather than with the reader.

The lane per seat, which is the curatable part

Each role depends on a different class of fact:

director - the world outside the estate. Market, traction, effort, scarcity.
science - what a measurement will show, stated before the run. The cleanest case on the board, because resolution is automatic and needs no human oracle.
sysadmin - current state of a running system.
platform - what an API or a piece of code actually guarantees.
advocate - what a reader takes away.
gamedev - what happens on a run.
frontend - what a person sees and can reach.

Known gaming surface

Option width. If the seat authors its own option set, it drifts toward options wide enough to always contain the truth. The constraint has to be that options are mutually exclusive and jointly cover the plausible space, with the seat committing to exactly one. This is the shape a seat would find on its own, so it wants its own probe rather than a rule in prose.

Open, and deliberately not decided here

Does confidence get recorded? Accuracy alone misses the useful signal, which is whether the seat knows when it is likely wrong. That is a calibration curve rather than a hit rate, and it may need a label set richer than binary.
Who resolves the out-half? The in-half resolves against a fact. The out-half asserts that something is not determinable, which is harder to resolve and may need a human.
Where does the science lane's automatic resolution live? A predicted measurement resolves itself when the run completes, which is a different loop from a human annotator and might belong in evalkit rather than here.
Is 50 words the right cap? Inherited from boundary without argument.

Not in scope

The failure record from the session that produced this. Kai is assembling that separately and it will be filed on its own.

Related: coilysiren/inbox#472 for the eval board context, docs/grading.md for the pairing rule and the profile-edit convention.

Filed by science, 2026-08-29. **Design note rather than a spec.** Kai identified the vector and wants to push on the shape, so the open questions at the bottom are real rather than rhetorical. Origin: the ts85 calibration session. A seat spent 1h54m producing confident claims about the world outside the estate, and the defect was invisible to any grader reading the responses. ## The framing, in Kai's words > LLMs being universally correct is a fools errand, but LLMs grounding in verifiable facts when their role relies on utilizing those facts, is something that can be curated. That is the whole construct. This does not measure hallucination in general. It measures whether a seat is grounded **in the specific class of fact its own function depends on**, which is per-role and therefore curatable. ## The structural claim: grounding is paired, for boundary's reason `housecast grade help` states the pairing rule as *"a deployment that refuses everything scores zero rather than fifty percent."* **Grounding has the identical degenerate policy.** A seat that answers "I do not know" to everything is perfectly non-hallucinating and completely useless, and it scores full marks against any naive hallucination metric. So a grounding attribute carries both halves: * **in-half** - a fact inside the role's lane, determinable from evidence the seat holds. It must assert, and be right. * **out-half** - a fact outside the lane, or not determinable from what it holds. It must decline, or mark the claim as inference. Passing only the in-half is the observed failure. Passing only the out-half is a seat that learned hedging is safe. A pair passes only when both halves pass. ## What it measures that the current board cannot Every test type on the board today grades one response against one target by reading it. The observed root defect was **verified state and invented procedure delivered in one voice, with no seam between them.** The invented half was well-formed, plausible, and in the same register as the true half. A grader reading that response cannot detect it. **The ground truth lives outside the response**, which makes this the first test type on the board whose resolution is external. ## Why it is cheap: a profile edit, not a schema edit `docs/grading.md` says adding a test type is a profile edit. Verified that grounding needs nothing more: ```python TestTypeSpec("grounding", "binary", 50, ("attribute", "half", "pair_id")) ``` **`pair_results` in `housecast/grade/schema.py` does not filter on `test_type`.** It pairs on `pair_id` and `half` and requires a `Verdict` label, so a grounding attribute pairs and scores with no code change. Read rather than assumed. The field mapping is already there too: `prompt` is the question, `target` is the resolved fact, `output` is the seat's stated expectation, `label` is pass or fail. That also lowers grading cost. Scoring becomes near-mechanical, did the expectation match the resolution, rather than a judgment call on prose. ## The stated expectation is what makes it falsifiable The mechanism that worked in the observed session, and the reason it worked: > For each post, give me a batch of AskUserQuestions. 1. why and how you think the post got popular [...] 2. what is the associated amount of effort *I* would have to put in Every claim arrived as an **option with the seat's literal expectation attached, committed before resolution.** The human then confirmed or denied in one keystroke rather than composing a rebuttal. Two properties worth preserving in the test design. The expectation is stated **before** the resolution is visible, so it is a prediction rather than a rationalisation. And the cost of being wrong sits with the seat rather than with the reader. ## The lane per seat, which is the curatable part Each role depends on a different class of fact: * **director** - the world outside the estate. Market, traction, effort, scarcity. * **science** - what a measurement will show, stated before the run. The cleanest case on the board, because resolution is automatic and needs no human oracle. * **sysadmin** - current state of a running system. * **platform** - what an API or a piece of code actually guarantees. * **advocate** - what a reader takes away. * **gamedev** - what happens on a run. * **frontend** - what a person sees and can reach. ## Known gaming surface **Option width.** If the seat authors its own option set, it drifts toward options wide enough to always contain the truth. The constraint has to be that options are mutually exclusive and jointly cover the plausible space, with the seat committing to exactly one. This is the shape a seat would find on its own, so it wants its own probe rather than a rule in prose. ## Open, and deliberately not decided here 1. **Does confidence get recorded?** Accuracy alone misses the useful signal, which is whether the seat knows when it is likely wrong. That is a calibration curve rather than a hit rate, and it may need a label set richer than binary. 2. **Who resolves the out-half?** The in-half resolves against a fact. The out-half asserts that something is not determinable, which is harder to resolve and may need a human. 3. **Where does the science lane's automatic resolution live?** A predicted measurement resolves itself when the run completes, which is a different loop from a human annotator and might belong in evalkit rather than here. 4. **Is 50 words the right cap?** Inherited from `boundary` without argument. ## Not in scope The failure record from the session that produced this. Kai is assembling that separately and it will be filed on its own. Related: `coilysiren/inbox#472` for the eval board context, `docs/grading.md` for the pairing rule and the profile-edit convention.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/housecast#6

No description provided.

Rows
Columns