Gauntlet-Spec: spec-attacker role generating Goodhart and underspecification scenarios #2

New issue

Closed

opened 2026-05-23 20:54:21 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-05-23 20:54:21 +00:00

Owner

Originally filed by @coilysiren on 2026-05-08T12:47:28Z - https://github.com/coilysiren/gauntlet/issues/14

Slice of #12. Depends on #13 (goal blocker schema).

Problem

The spec-attacker is the upward-pressure analogue to today's code-attacker. Instead of generating requests that make code violate spec, it generates scenarios that make spec violate goal.

Three classes of attack worth distinguishing in the role prompt:

Goodhart attacks. Construct a behavior that satisfies every spec invariant while violating the stated user goal. The closer this is to "technically allowed but obviously wrong," the more valuable the finding.
Underspecification attacks. Construct a scenario the spec literally cannot answer (state the spec doesn't cover, input the spec doesn't classify, contradiction between two sections). Output: "what should happen here?"
Internal-contradiction attacks. Walk the spec for clauses that conflict, including dead clauses, redundant clauses, and clauses whose preconditions never co-occur.

Scope

New role: gauntlet-spec-attacker, mirroring the existing attacker role's shape (read attacker-safe brief, compose plan, execute against the spec-as-SUT, append iteration).
Plan execution against a spec is different from execution against an HTTP service. Likely shape: feed spec excerpts plus world-model context to a model and ask it to instantiate the attack, then record the scenario as the iteration artifact.
Finding shape: scenario, which class (Goodhart / underspec / contradiction), proposed spec edit OR proposed acceptance criterion. Inspector role judges.

Out of scope

Inspector role implementation (separate slice if it diverges enough from today's inspector; may be reusable as-is).
LUCA meta-controller integration.

Done when

A spec-attacker can run one round against a trial that has goal blockers (per #13) and produce at least one finding in each of the three classes against an intentionally weak spec.

_Originally filed by @coilysiren on 2026-05-08T12:47:28Z - [https://github.com/coilysiren/gauntlet/issues/14](https://github.com/coilysiren/gauntlet/issues/14)_ Slice of #12. Depends on #13 (goal blocker schema). ## Problem The spec-attacker is the upward-pressure analogue to today's code-attacker. Instead of generating requests that make code violate spec, it generates scenarios that make spec violate goal. Three classes of attack worth distinguishing in the role prompt: - **Goodhart attacks.** Construct a behavior that satisfies every spec invariant while violating the stated user goal. The closer this is to "technically allowed but obviously wrong," the more valuable the finding. - **Underspecification attacks.** Construct a scenario the spec literally cannot answer (state the spec doesn't cover, input the spec doesn't classify, contradiction between two sections). Output: "what should happen here?" - **Internal-contradiction attacks.** Walk the spec for clauses that conflict, including dead clauses, redundant clauses, and clauses whose preconditions never co-occur. ## Scope - New role: `gauntlet-spec-attacker`, mirroring the existing attacker role's shape (read attacker-safe brief, compose plan, execute against the spec-as-SUT, append iteration). - Plan execution against a spec is different from execution against an HTTP service. Likely shape: feed spec excerpts plus world-model context to a model and ask it to instantiate the attack, then record the scenario as the iteration artifact. - Finding shape: scenario, which class (Goodhart / underspec / contradiction), proposed spec edit OR proposed acceptance criterion. Inspector role judges. ## Out of scope - Inspector role implementation (separate slice if it diverges enough from today's inspector; may be reusable as-is). - LUCA meta-controller integration. ## Done when A spec-attacker can run one round against a trial that has goal blockers (per #13) and produce at least one finding in each of the three classes against an intentionally weak spec.

coilysiren commented

2026-05-30 05:44:13 +00:00

Author

Owner

Iceboxed in the 2026-05-29 backlog burn-down: Explicit slice of #4 Gauntlet-Spec cluster (cluster fully iceboxed). Reopen anytime if it becomes real.

coilysiren

2026-05-30 05:44:13 +00:00