Gauntlet-Spec: adversarial mode that targets the spec, not the code #4

New issue

Closed

opened 2026-05-23 20:54:22 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-23 20:54:22 +00:00

Owner

Originally filed by @coilysiren on 2026-05-08T12:46:54Z - https://github.com/coilysiren/gauntlet/issues/12

Problem

Gauntlet today is one-sided. The attacker/inspector loop targets code-against-spec: spec is treated as truth, attacker tries to make code violate spec invariants, inspector judges. This produces pure downward pressure on the code. Useful, but a system with only downward pressure stalls. Hardening without a counter-force converges on "everything fails closed and nothing ships."

The missing half is the dual. Same two-role adversarial shape, different target.

Proposal

Treat gauntlet as having two modes, not one.

Gauntlet-Code (today). Spec is truth. Attacker attacks code. Output: bugs and invariant violations. Pressure goes down onto code.
Gauntlet-Spec (new). User goal or business outcome is truth. Attacker attacks the spec. Output: spec gaps, Goodhart paths, underspecification, internal contradictions, proposed acceptance criteria, proposed deletions. Pressure goes up onto the spec.

The "SUT" in Gauntlet-Spec is the spec plus a world model, not a running HTTP service. Findings become next round's input. Same MCP shape, same role discipline (attacker/inspector/holdout-evaluator), different blocker schema and different finding shape.

Why this matters for the larger stack

This closes the loop with the LUCA evolutionary feedback story. With both modes wired in, the meta-controller's job each turn is to decide whether the cheaper move is to harden the code (Gauntlet-Code found something), re-aim the spec (Gauntlet-Spec found something), or ship (both quiet). Gauntlet stops being only a hardening tool and becomes the bidirectional truth-finder under LUCA.

Scope

Umbrella issue. Tracer-bullet slices to follow:

Blocker schema for goal/outcome blockers.
Spec-attacker role generating Goodhart and underspecification scenarios.
Spec-inspector finding shape and proposed-edit output format.
Meta-controller integration point for LUCA (out of scope for gauntlet itself, captured in luca).

Prior art to steal from

Property-based testing (Hypothesis, QuickCheck) for "what does the spec not cover."
Constitutional AI / RLAIF for the dual-model critique structure.
Eval-driven development as the cultural framing.

_Originally filed by @coilysiren on 2026-05-08T12:46:54Z - [https://github.com/coilysiren/gauntlet/issues/12](https://github.com/coilysiren/gauntlet/issues/12)_ ## Problem Gauntlet today is one-sided. The attacker/inspector loop targets code-against-spec: spec is treated as truth, attacker tries to make code violate spec invariants, inspector judges. This produces pure downward pressure on the code. Useful, but a system with only downward pressure stalls. Hardening without a counter-force converges on "everything fails closed and nothing ships." The missing half is the dual. Same two-role adversarial shape, different target. ## Proposal Treat gauntlet as having two modes, not one. - **Gauntlet-Code** (today). Spec is truth. Attacker attacks code. Output: bugs and invariant violations. Pressure goes down onto code. - **Gauntlet-Spec** (new). User goal or business outcome is truth. Attacker attacks the spec. Output: spec gaps, Goodhart paths, underspecification, internal contradictions, proposed acceptance criteria, proposed deletions. Pressure goes up onto the spec. The "SUT" in Gauntlet-Spec is the spec plus a world model, not a running HTTP service. Findings become next round's input. Same MCP shape, same role discipline (attacker/inspector/holdout-evaluator), different blocker schema and different finding shape. ## Why this matters for the larger stack This closes the loop with the LUCA evolutionary feedback story. With both modes wired in, the meta-controller's job each turn is to decide whether the cheaper move is to harden the code (Gauntlet-Code found something), re-aim the spec (Gauntlet-Spec found something), or ship (both quiet). Gauntlet stops being only a hardening tool and becomes the bidirectional truth-finder under LUCA. ## Scope Umbrella issue. Tracer-bullet slices to follow: - Blocker schema for goal/outcome blockers. - Spec-attacker role generating Goodhart and underspecification scenarios. - Spec-inspector finding shape and proposed-edit output format. - Meta-controller integration point for LUCA (out of scope for gauntlet itself, captured in luca). ## Prior art to steal from - Property-based testing (Hypothesis, QuickCheck) for "what does the spec not cover." - Constitutional AI / RLAIF for the dual-model critique structure. - Eval-driven development as the cultural framing.

coilysiren commented

2026-05-30 05:44:12 +00:00

Author

Owner

Iceboxed in the 2026-05-29 backlog burn-down: Far-future Gauntlet-Spec mode, no current dependency. Reopen anytime if it becomes real.

coilysiren closed this issue

2026-05-30 05:44:12 +00:00

coilysiren referenced this issue

2026-05-30 05:44:12 +00:00

Gauntlet-Spec: blocker schema for goal/outcome blockers #3

coilysiren referenced this issue

2026-05-30 05:44:13 +00:00