coilyco-flight-deck/mcp-beaver

Watch

Declare mcp-beaver grant boundaries in aos-eval shape: deny-by-absence is tested, "everything you declared works" is not #84

New issue

Open

opened 2026-08-18 22:03:08 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-18 22:03:08 +00:00

Member

Outcome

Declare mcp-beaver's grant boundaries in aos-eval.boundaries.v1 shape, derived from the .mcp.kdl guardfile, so aos-eval boundaries check can name every deny rule with no reachable-case counterpart. deny-by-absence is the product claim, and nothing currently measures the other half of it.

The specific risk here

The README states the claim plainly: "Everything you declared works. Nothing else is reachable." Those are two assertions and the test suite is weighted toward the second one.

Read on main at depth 30: 192 test functions across 32 _test.go files, roughly 46 naming a refusal (Reject*, Refus*, Den*, Block*) against roughly 16 naming a permit (Allow*, Permit*, Accept*, Serves*).

Those numbers are a signal rather than a measurement. They read test-function names, so a reachable case asserted inside a table test without Allow in its name is invisible to them. The reason a better number does not exist is the thing to fix: no file declares the complete boundary set, so nothing can distinguish a real missing half from a naming artifact.

Why the pairing rule fits a dam better than an agent

From aos_eval/boundaries.py:

A boundary is only measured by a pair. The in-half proves the rule fires, the out-half proves it does not fire on the neighbouring case that must still be served.

"A dam is not a wall. It decides what gets through." A wall passes a refusal-only suite perfectly. The half that separates this runtime from a wall is the one with less coverage, and TestArgPinAllowsTheMatchingValue beside TestArgPinRefusesAnOverride shows the shape is already natural here. It is just not declared, so no tool can report the pairs nobody wrote.

Two modes, two different claims

The pairing needs to distinguish them, because docs/conformance.md and the README describe genuinely different guarantees:

spec mode - an undeclared operation has no handler at all. The out-half is absence.
upstream-proxy mode - undeclared tools exist behind an endpoint the container holds credentials for, and the runtime re-checks allowlist membership on every call. The out-half is a live refusal, "unreachable rather than absent."

Those two out-halves fail differently and a single declaration that flattens them would measure the weaker one. The proxy-mode half is the one with real blast radius, since the credential is present and only the check stands between a call and the upstream.

It costs nothing to run

aos-eval holds no runner and no model client, so boundaries derive and boundaries check run offline with no inference and no human grader. This is the deterministic-battery half of the eval stack, not the graded-board half.

Expect the first output to be a gap

Against sirens-echo's pilot board the same tool reported 56 derived slots, none authored. docs/aos-eval.md: "The first honest output in a new repo is usually a gap" and "A coverage report that cannot come back negative is decoration." A clean first run here should be read as a bug in the declaration.

Complete when

A boundary declaration exists in aos-eval.boundaries.v1 shape, derived from the guardfile grant tree rather than hand-restated, with origin naming the source.
spec mode and upstream-proxy mode carry distinct out-halves.
aos-eval boundaries check runs as a just verb or in CI, and its current gap is recorded rather than suppressed.

Not in scope

No human-graded board and no model subject. The agent-compose and sirens-echo triple does not transfer, because mcp-beaver is a runtime rather than an agent. Also not MCP protocol conformance, which docs/conformance.md already covers and which asks a different question.

Source

Opened after a portfolio-wide eval coverage read on 2026-08-18. docs/aos-eval.md in agentic-os names exactly two consumers, agent-compose and sirens-echo. mcp-beaver has none. Sibling issue filed against umbra, whose KDL policy this runtime renders.

## Outcome Declare mcp-beaver's grant boundaries in `aos-eval.boundaries.v1` shape, derived from the `.mcp.kdl` guardfile, so `aos-eval boundaries check` can name every deny rule with no reachable-case counterpart. **deny-by-absence is the product claim, and nothing currently measures the other half of it.** ## The specific risk here The README states the claim plainly: "Everything you declared works. Nothing else is reachable." Those are two assertions and the test suite is weighted toward the second one. Read on `main` at depth 30: 192 test functions across 32 `_test.go` files, roughly 46 naming a refusal (`Reject*`, `Refus*`, `Den*`, `Block*`) against roughly 16 naming a permit (`Allow*`, `Permit*`, `Accept*`, `Serves*`). **Those numbers are a signal rather than a measurement.** They read test-function names, so a reachable case asserted inside a table test without `Allow` in its name is invisible to them. The reason a better number does not exist is the thing to fix: no file declares the complete boundary set, so nothing can distinguish a real missing half from a naming artifact. ## Why the pairing rule fits a dam better than an agent From `aos_eval/boundaries.py`: > A boundary is only measured by a pair. The in-half proves the rule fires, the out-half proves it does not fire on the neighbouring case that must still be served. **"A dam is not a wall. It decides what gets through."** A wall passes a refusal-only suite perfectly. The half that separates this runtime from a wall is the one with less coverage, and `TestArgPinAllowsTheMatchingValue` beside `TestArgPinRefusesAnOverride` shows the shape is already natural here. It is just not declared, so no tool can report the pairs nobody wrote. ## Two modes, two different claims The pairing needs to distinguish them, because `docs/conformance.md` and the README describe genuinely different guarantees: * **spec mode** - an undeclared operation has no handler at all. The out-half is absence. * **upstream-proxy mode** - undeclared tools exist behind an endpoint the container holds credentials for, and the runtime re-checks allowlist membership on every call. The out-half is a live refusal, "unreachable rather than absent." **Those two out-halves fail differently and a single declaration that flattens them would measure the weaker one.** The proxy-mode half is the one with real blast radius, since the credential is present and only the check stands between a call and the upstream. ## It costs nothing to run `aos-eval` holds no runner and no model client, so `boundaries derive` and `boundaries check` run offline with no inference and no human grader. This is the deterministic-battery half of the eval stack, not the graded-board half. ## Expect the first output to be a gap Against sirens-echo's pilot board the same tool reported 56 derived slots, none authored. `docs/aos-eval.md`: "The first honest output in a new repo is usually a gap" and "A coverage report that cannot come back negative is decoration." A clean first run here should be read as a bug in the declaration. ## Complete when * A boundary declaration exists in `aos-eval.boundaries.v1` shape, derived from the guardfile grant tree rather than hand-restated, with `origin` naming the source. * spec mode and upstream-proxy mode carry distinct out-halves. * `aos-eval boundaries check` runs as a `just` verb or in CI, and its current gap is recorded rather than suppressed. ## Not in scope No human-graded board and no model subject. The `agent-compose` and `sirens-echo` triple does not transfer, because mcp-beaver is a runtime rather than an agent. Also not MCP protocol conformance, which `docs/conformance.md` already covers and which asks a different question. ## Source Opened after a portfolio-wide eval coverage read on 2026-08-18. `docs/aos-eval.md` in agentic-os names exactly two consumers, agent-compose and sirens-echo. mcp-beaver has none. Sibling issue filed against `umbra`, whose KDL policy this runtime renders.

coilyco-ops commented

2026-08-18 22:31:05 +00:00

Author

Member

Third sibling filed: ward#1676.

Worth reading alongside this one, because it walks back the opening argument both of these use. The refusal-versus-permit test-name ratio pointed at a gap in Ward's director merge gate, and checking the actual call sites found it properly paired with three unnamed positive-path assertions. The name-based count is unreliable in both directions, which is the argument for declaring the boundary set rather than counting test names, and it applies to the numbers quoted here too.

Third sibling filed: [ward#1676](https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1676). Worth reading alongside this one, because it walks back the opening argument both of these use. The refusal-versus-permit test-name ratio pointed at a gap in Ward's director merge gate, and checking the actual call sites found it properly paired with three unnamed positive-path assertions. **The name-based count is unreliable in both directions**, which is the argument for declaring the boundary set rather than counting test names, and it applies to the numbers quoted here too.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/mcp-beaver#84

No description provided.

Rows
Columns