Gauntlet-Bench: third mode that benchmarks agent platform bug detection #1

New issue

Closed

opened 2026-05-23 20:54:21 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-23 20:54:21 +00:00

Owner

Originally filed by @coilysiren on 2026-05-13T05:39:03Z - https://github.com/coilysiren/gauntlet/issues/22

Scope issue for Gauntlet-Bench, the third adversarial mode. Decided in coilysiren/coilyco-ai#365 (bug-finder absorbed into gauntlet as mode 3).

Three modes recap

Gauntlet-Code (today). SUT is running code. Spec is truth. Attacker attacks code against spec.
Gauntlet-Spec (#12). SUT is a spec. User goal is truth. Attacker attacks spec against goal.
Gauntlet-Bench (this issue). SUT is an agent platform. Attacker generates bug-injected scenarios. Inspector scores whether the agent platform caught them. Output is a benchmark scoreboard.

Design questions to answer

SUT contract

What does an "agent platform under test" look like as a Gauntlet SUT? Candidates:

A Claude Code plugin or skill bundle, invoked through a harness.
A coding-agent CLI (claude, aider, cline, etc.) pointed at a prepared workspace.
A more abstract "agent + tools" interface with a task description as input and a fix attempt as output.

Decision: pick the narrowest contract that covers the seven scored failure-mode candidates without locking out future targets.

Attacker scenario generation

Mode 3's attacker doesn't attack the SUT directly. It generates bug-injected scenarios (a codebase plus a planted bug) that the agent platform is then asked to detect or fix. Open questions:

Is scenario generation procedural (mutation operators on a real codebase) or curated (hand-picked bug seeds with metadata)?
How does the attacker iterate? In modes 1 and 2 the attacker mutates plans based on inspector findings. In mode 3 the inspector is judging the agent, not the attacker's plan. What does mutation pressure look like?
Reuse the existing mutate_plans MCP tool, or define a mode-3 specific variant?

Inspector scoring rubric

Mode 3's inspector judges whether the agent platform detected the planted bug, not whether the SUT violated a spec. Rubric needs:

Detection signal definition (the agent wrote a fix, named the bug in output, flagged it for review, etc.).
Partial credit (caught the symptom but not the cause, fixed it but introduced a regression, etc.).
Per-failure-mode scoring buckets (the seven candidates from the original bug-finder scope).

Holdout-evaluator role in benchmark mode

In modes 1 and 2 the holdout evaluator derives acceptance plans from blockers and runs them against the SUT. In mode 3, what is the analog? Candidates:

A held-out scenario set the agent has not seen during the attacker loop, scored by the same inspector rubric.
A second pass with a different agent platform, for cross-platform calibration.
Skip the role for mode 3 if it doesn't carry.

Frozen-corpus reproducibility

The decision in coilyco-ai#365 named "replay this run" as a mode-3 feature. Concrete shape:

Scenario set, scoring rubric, and inspector decisions are all reproducible from the run buffer plus the original scenario IDs.
/luca-export integration so a substrate-instrumented Gauntlet-Bench run produces a portable corpus.
Scoreboard publication format (CSV plus markdown summary, separate repo, gist, etc.).

Done when

This issue is decomposed into per-question implementation issues with concrete scope.
The SUT contract, attacker mutation loop, inspector rubric, and holdout-evaluator decision are written down (probably as a docs/gauntlet-bench.md design doc).
A first scenario is runnable end-to-end through the existing MCP role discipline.

Out of scope

Implementing all seven failure-mode candidates. First scenario plus the design doc is enough to validate the shape.
Publishing the scoreboard. That follows once the rubric is stable.
Substrate-side wiring (/luca-export for benchmark corpora). Tracked separately under coilysiren/luca.

_Originally filed by @coilysiren on 2026-05-13T05:39:03Z - [https://github.com/coilysiren/gauntlet/issues/22](https://github.com/coilysiren/gauntlet/issues/22)_ Scope issue for Gauntlet-Bench, the third adversarial mode. Decided in coilysiren/coilyco-ai#365 (bug-finder absorbed into gauntlet as mode 3). ## Three modes recap - **Gauntlet-Code** (today). SUT is running code. Spec is truth. Attacker attacks code against spec. - **Gauntlet-Spec** (#12). SUT is a spec. User goal is truth. Attacker attacks spec against goal. - **Gauntlet-Bench** (this issue). SUT is an agent platform. Attacker generates bug-injected scenarios. Inspector scores whether the agent platform caught them. Output is a benchmark scoreboard. ## Design questions to answer ### SUT contract What does an "agent platform under test" look like as a Gauntlet SUT? Candidates: - A Claude Code plugin or skill bundle, invoked through a harness. - A coding-agent CLI (claude, aider, cline, etc.) pointed at a prepared workspace. - A more abstract "agent + tools" interface with a task description as input and a fix attempt as output. Decision: pick the narrowest contract that covers the seven scored failure-mode candidates without locking out future targets. ### Attacker scenario generation Mode 3's attacker doesn't attack the SUT directly. It generates bug-injected scenarios (a codebase plus a planted bug) that the agent platform is then asked to detect or fix. Open questions: - Is scenario generation procedural (mutation operators on a real codebase) or curated (hand-picked bug seeds with metadata)? - How does the attacker iterate? In modes 1 and 2 the attacker mutates plans based on inspector findings. In mode 3 the inspector is judging the agent, not the attacker's plan. What does mutation pressure look like? - Reuse the existing `mutate_plans` MCP tool, or define a mode-3 specific variant? ### Inspector scoring rubric Mode 3's inspector judges whether the agent platform detected the planted bug, not whether the SUT violated a spec. Rubric needs: - Detection signal definition (the agent wrote a fix, named the bug in output, flagged it for review, etc.). - Partial credit (caught the symptom but not the cause, fixed it but introduced a regression, etc.). - Per-failure-mode scoring buckets (the seven candidates from the original bug-finder scope). ### Holdout-evaluator role in benchmark mode In modes 1 and 2 the holdout evaluator derives acceptance plans from blockers and runs them against the SUT. In mode 3, what is the analog? Candidates: - A held-out scenario set the agent has not seen during the attacker loop, scored by the same inspector rubric. - A second pass with a different agent platform, for cross-platform calibration. - Skip the role for mode 3 if it doesn't carry. ### Frozen-corpus reproducibility The decision in coilyco-ai#365 named "replay this run" as a mode-3 feature. Concrete shape: - Scenario set, scoring rubric, and inspector decisions are all reproducible from the run buffer plus the original scenario IDs. - `/luca-export` integration so a substrate-instrumented Gauntlet-Bench run produces a portable corpus. - Scoreboard publication format (CSV plus markdown summary, separate repo, gist, etc.). ## Done when - This issue is decomposed into per-question implementation issues with concrete scope. - The SUT contract, attacker mutation loop, inspector rubric, and holdout-evaluator decision are written down (probably as a `docs/gauntlet-bench.md` design doc). - A first scenario is runnable end-to-end through the existing MCP role discipline. ## Out of scope - Implementing all seven failure-mode candidates. First scenario plus the design doc is enough to validate the shape. - Publishing the scoreboard. That follows once the rubric is stable. - Substrate-side wiring (`/luca-export` for benchmark corpora). Tracked separately under coilysiren/luca. ## See also - coilysiren/coilyco-ai#365 - decision to absorb bug-finder into gauntlet as mode 3. - coilysiren/coilyco-ai#368 - substrate.md doc update reflecting this decision. - #12 - Gauntlet-Spec, the other open mode.

coilysiren commented

2026-05-30 05:44:13 +00:00

Author

Owner

Iceboxed in the 2026-05-29 backlog burn-down: Far-future Gauntlet-Bench third mode, speculative. Reopen anytime if it becomes real.

coilysiren

2026-05-30 05:44:13 +00:00