coilyco-flight-deck/umbra

Watch

Declare umbra boundaries in aos-eval shape: the guard is tested for what it refuses, not for what it must still serve #305

New issue

Open

opened 2026-08-18 22:02:54 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-18 22:02:54 +00:00

Member

Outcome

Declare umbra's authority boundaries in aos-eval.boundaries.v1 shape, so aos-eval boundaries derive and boundaries check can name every guard rule with no permitted-case counterpart. umbra is the third-largest agent-facing authority surface in the portfolio and the only eval question it cannot currently answer is the one that matters: does the guard still serve what it is supposed to serve.

Why this is not "add evals"

umbra already tests refusal heavily. Read on main at depth 30: 619 test functions across 80 _test.go files, of which roughly 120 name a refusal (Deny, Refus*, Reject*, Unauthoriz*, Block*, FailsClosed) against roughly 23 naming a permit (Allow*, Permit*, Accept*, Serves*).

Treat those two numbers as a signal rather than a measurement. They come from test-function names, and a permitted case asserted inside a table test without Allow in its name does not show up. That imprecision is the point: nothing in this repo can currently tell a real pairing gap from a naming artifact, because no file declares what the complete boundary set is.

The rule this borrows

From aos_eval/boundaries.py:

A boundary is only measured by a pair. The in-half proves the rule fires, the out-half proves it does not fire on the neighbouring case that must still be served. Grading one half alone rewards a deployment that refuses everything.

A guard is the sharpest possible case for that rule. A specgen policy that denied every verb would pass a refusal-only suite perfectly and be useless. The existing pairs prove the shape is already natural here, TestAllowParseFailsClosed beside TestAllowedSingle and TestAllowedMulti, but they are ad hoc rather than declared, so no tool can report the ones nobody wrote.

What makes umbra easier than the two existing consumers

agent-compose derives its slots from a role roster and sirens-echo authors them against prompt clauses cited by line. umbra already holds a machine-readable authority declaration: the KDL policy and its committed locks. Boundary carries origin and derived for exactly this, with the docstring warning that "a boundary restated here rather than derived from its source drifts the moment the source changes."

So the target is derivation from the guardfile rather than a hand-maintained list, which is the same authoring-vs-rollout discipline the repo already follows elsewhere.

And it needs no model

aos-eval holds no runner and no model client. boundaries derive and boundaries check run offline. This costs no inference, needs no human grader, and produces a coverage report rather than a score. It is closer to sirens-echo's deterministic battery than to its human-graded board.

Expect the first output to be a gap

Against sirens-echo's pilot board the same tool reported 56 derived slots, none authored, and 10 authored cases no declaration derived. docs/aos-eval.md calls that the point rather than an embarrassment: "A coverage report that cannot come back negative is decoration." A first run here that comes back clean should be treated as suspicious.

Complete when

A boundary declaration exists in aos-eval.boundaries.v1 shape, derived from the KDL policy rather than hand-restated, with origin naming the source.
aos-eval boundaries check runs in CI or as a just verb and its current gap is recorded rather than suppressed.
docs/FEATURES.md gains an entry only if this lands as a real capability, not for the declaration file alone.

Not in scope

No human-graded board and no model subject. umbra is not an agent, so the triple in agent-compose and sirens-echo does not transfer. This issue is the offline coverage half only.

Source

Opened after a portfolio-wide eval coverage read on 2026-08-18. docs/aos-eval.md in agentic-os names exactly two consumers, agent-compose and sirens-echo. umbra has none. Sibling issue filed against mcp-beaver.

## Outcome Declare umbra's authority boundaries in `aos-eval.boundaries.v1` shape, so `aos-eval boundaries derive` and `boundaries check` can name every guard rule with no permitted-case counterpart. umbra is the third-largest agent-facing authority surface in the portfolio and the only eval question it cannot currently answer is the one that matters: does the guard still serve what it is supposed to serve. ## Why this is not "add evals" umbra already tests refusal heavily. Read on `main` at depth 30: 619 test functions across 80 `_test.go` files, of which roughly 120 name a refusal (`Deny`, `Refus*`, `Reject*`, `Unauthoriz*`, `Block*`, `FailsClosed`) against roughly 23 naming a permit (`Allow*`, `Permit*`, `Accept*`, `Serves*`). **Treat those two numbers as a signal rather than a measurement.** They come from test-function names, and a permitted case asserted inside a table test without `Allow` in its name does not show up. That imprecision is the point: **nothing in this repo can currently tell a real pairing gap from a naming artifact**, because no file declares what the complete boundary set is. ## The rule this borrows From `aos_eval/boundaries.py`: > A boundary is only measured by a pair. The in-half proves the rule fires, the out-half proves it does not fire on the neighbouring case that must still be served. Grading one half alone rewards a deployment that refuses everything. A guard is the sharpest possible case for that rule. A `specgen` policy that denied every verb would pass a refusal-only suite perfectly and be useless. The existing pairs prove the shape is already natural here, `TestAllowParseFailsClosed` beside `TestAllowedSingle` and `TestAllowedMulti`, but they are ad hoc rather than declared, so no tool can report the ones nobody wrote. ## What makes umbra easier than the two existing consumers agent-compose derives its slots from a role roster and sirens-echo authors them against prompt clauses cited by line. **umbra already holds a machine-readable authority declaration**: the KDL policy and its committed locks. `Boundary` carries `origin` and `derived` for exactly this, with the docstring warning that "a boundary restated here rather than derived from its source drifts the moment the source changes." So the target is derivation from the guardfile rather than a hand-maintained list, which is the same authoring-vs-rollout discipline the repo already follows elsewhere. ## And it needs no model `aos-eval` holds no runner and no model client. `boundaries derive` and `boundaries check` run offline. This costs no inference, needs no human grader, and produces a coverage report rather than a score. It is closer to sirens-echo's deterministic battery than to its human-graded board. ## Expect the first output to be a gap Against sirens-echo's pilot board the same tool reported 56 derived slots, none authored, and 10 authored cases no declaration derived. `docs/aos-eval.md` calls that the point rather than an embarrassment: "A coverage report that cannot come back negative is decoration." A first run here that comes back clean should be treated as suspicious. ## Complete when * A boundary declaration exists in `aos-eval.boundaries.v1` shape, derived from the KDL policy rather than hand-restated, with `origin` naming the source. * `aos-eval boundaries check` runs in CI or as a `just` verb and its current gap is recorded rather than suppressed. * `docs/FEATURES.md` gains an entry only if this lands as a real capability, not for the declaration file alone. ## Not in scope No human-graded board and no model subject. umbra is not an agent, so the triple in `agent-compose` and `sirens-echo` does not transfer. This issue is the offline coverage half only. ## Source Opened after a portfolio-wide eval coverage read on 2026-08-18. `docs/aos-eval.md` in agentic-os names exactly two consumers, agent-compose and sirens-echo. umbra has none. Sibling issue filed against `mcp-beaver`.

coilyco-ops commented

2026-08-18 22:31:02 +00:00

Author

Member

Third sibling filed: ward#1676.

Worth reading alongside this one, because it walks back the opening argument both of these use. The refusal-versus-permit test-name ratio pointed at a gap in Ward's director merge gate, and checking the actual call sites found it properly paired with three unnamed positive-path assertions. The name-based count is unreliable in both directions, which is the argument for declaring the boundary set rather than counting test names, and it applies to the numbers quoted here too.

Third sibling filed: [ward#1676](https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1676). Worth reading alongside this one, because it walks back the opening argument both of these use. The refusal-versus-permit test-name ratio pointed at a gap in Ward's director merge gate, and checking the actual call sites found it properly paired with three unnamed positive-path assertions. **The name-based count is unreliable in both directions**, which is the argument for declaring the boundary set rather than counting test names, and it applies to the numbers quoted here too.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/umbra#305

No description provided.

Rows
Columns