Scientific citation classifier for agent replies #874

New issue

Open

opened 2026-08-17 02:07:51 +00:00 by coilyco-ops-gaming · 0 comments

coilyco-ops-gaming commented

2026-08-17 02:07:51 +00:00

Member

Classification contract

Needs a citation: any empirical claim about the natural world, species, fossils, papers, dates, or mechanisms.
Passes: claim plus a live link where the source actually supports the claim.
Fails: a scientific claim with no citation; citation markers with no URLs; a link that does not resolve; a link that does not support the claim; a claim that overstates its source.
Boundary: fictional canon (movie raptors, time travel) needs no paper. Claims about real dinosaurs do.
On fail: do not answer. Kick a retry that verifies via web search and labels epistemic state: cited, hypothesis, or unknown.

Labeled eval cases (from the source thread, paraphrased)

fail: unsourced claim about dinosaur breath.
fail: "blue eggs in 2017" where the year was right but the paper was misattributed.
fail: citation markers like [1,2,3] with no URLs.
pass: claim plus a live Nature or Science DOI that supports it.
pass: fictional canon claims with no citation.
boundary: a claim that overstates its cited source.

Acceptance criteria

The eval set above classifies with the expected labels.
Failed claims never reach the user as an answer.
Every citation in an accepted answer resolves and supports the claim.
Retry path verifies via web search before re-answering.

Content-side spec for a classifier that gates agent replies containing scientific claims. Triggered by a thread where an agent answered paleontology questions, one factual slip slipped through, and the agreed standard became: scientific claims need tool-verified citations. This issue carries the rubric and eval set; the model build is engineer work. ## Classification contract - Needs a citation: any empirical claim about the natural world, species, fossils, papers, dates, or mechanisms. - Passes: claim plus a live link where the source actually supports the claim. - Fails: a scientific claim with no citation; citation markers with no URLs; a link that does not resolve; a link that does not support the claim; a claim that overstates its source. - Boundary: fictional canon (movie raptors, time travel) needs no paper. Claims about real dinosaurs do. - On fail: do not answer. Kick a retry that verifies via web search and labels epistemic state: cited, hypothesis, or unknown. ## Labeled eval cases (from the source thread, paraphrased) - fail: unsourced claim about dinosaur breath. - fail: "blue eggs in 2017" where the year was right but the paper was misattributed. - fail: citation markers like [1,2,3] with no URLs. - pass: claim plus a live Nature or Science DOI that supports it. - pass: fictional canon claims with no citation. - boundary: a claim that overstates its cited source. ## Acceptance criteria - The eval set above classifies with the expected labels. - Failed claims never reach the user as an answer. - Every citation in an accepted answer resolves and supports the claim. - Retry path verifies via web search before re-answering.