Declare Ward boundaries in aos-eval shape: the mechanical half is offline-checkable, the lane clauses are not #1676

New issue

Open

opened 2026-08-18 22:30:34 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-18 22:30:34 +00:00

Member

Outcome

Declare Ward's boundaries in aos-eval.boundaries.v1 shape, and split them by kind, because Ward is the first repo in the portfolio that needs both halves of the eval stack rather than one. The mechanical boundaries are declarable and offline-checkable today. The lane boundaries are claims about agent behavior and need a subject and a grader.

Sibling issues: umbra#305 and mcp-beaver#84. Ward differs from both and the difference is the substance of this issue.

First, a correction to the premise those siblings ran on

Read on main at depth 30: 997 test functions across 126 _test.go files, roughly 81 naming a refusal against roughly 16 naming a permit. On the sibling issues that ratio was the opening argument.

Then I checked the highest-stakes gate in the repo and it is properly paired. directorMergeEligibility in cmd/ward/agent_director_merge_test.go carries four refusal tests by name and three unnamed positive-path assertions, at lines 521, 664, and 682, each wanting true/""/729. directorMergeDecision is paired the same way at line 38. Nobody would learn that from the function names, which is exactly the failure mode the siblings warn about, now demonstrated in the other direction.

That does not weaken the case for a declaration. It is the case. A name-based count is unreliable in both directions, so the only way to know where a half is missing is to declare the complete set and let aos-eval boundaries check compare it to what exists. Opening this issue without that check would have been a guess.

Ward's boundary axis is evidence, not permission

umbra and mcp-beaver bound what a caller may reach. Ward bounds what counts as having happened. From docs/agent-workflow.md:

Harness exit, a local commit, or a stale no-diff salvage branch is not delivery.

Each lane declares distinct evidence: remote main for direct landing, remote branch plus canonical PR URL plus submitted workflow state for PR landing, current CI plus review authorization plus QA verdict plus merged: true for director landing, and the named remote branch for branch-only.

The pairing writes itself. The out-half is Ward refusing to report delivery on insufficient evidence, which is the well-covered side. The in-half is Ward correctly reporting delivery when the evidence is genuinely there, and a Ward that reported failure unconditionally would pass an evidence-refusal suite perfectly while being useless.

The second kind, which no declaration can check offline

remote-branch-only is documented as "publish a remote branch and stop", with no PR or merge authority. In the code that constraint is prompt text. cmd/ward/agent.go:494:

For remote-branch-only workflows, the remote branch push is the finish line. Do not open a pull request and do not merge.

Every reference to workflowRemoteBranchOnly outside tests generates prompt material: the carry clause, the landing noun, the outcome status, and one review-gate skip. Nothing mechanically prevents that run from opening a PR.

This is by design rather than a defect. The README says so directly: roles, harnesses, and workflows "select" behavior and "None of those labels grants credentials, mounts, network, broker operations, or merge authority." Enforcement lives in the credential and broker layer, and the lane label is instruction.

The consequence for evaluation is the whole point of this issue. "Does an agent in a remote-branch-only lane actually stop at the branch" is a behavioral claim about a model reading a prompt clause. That is precisely what sirens-echo's board measures, cases authored against prompt clauses cited by line, and it cannot be answered by a declaration, a unit test, or an offline coverage tool.

So the split

Mechanical - merge eligibility, MergePullRequestRefusesFromReadOnlySurface, the clean-and-synced gate, delivery-evidence sufficiency per lane, AgentFrameworkChildCapabilityCannotSelectFixedWorkflow. Declarable in aos-eval.boundaries.v1, checkable offline with boundaries derive and boundaries check, no model and no grader.
Behavioral - the four lane clauses as an agent actually follows them. Needs the triple, and sirens-echo already proved the prompt-clause citation pattern including a test that fails when a citation drifts off its clause.

Do the mechanical half first. It is cheap, it costs no inference, and its output will scope the behavioral half.

Expect the first output to be a gap

Against sirens-echo's pilot board the same tool reported 56 derived slots, none authored. docs/aos-eval.md: "The first honest output in a new repo is usually a gap" and "A coverage report that cannot come back negative is decoration." Given what the directorMergeEligibility read turned up, Ward's gap will be smaller than the naming suggests and will land in places nobody has looked.

Complete when

A boundary declaration exists in aos-eval.boundaries.v1 shape, with origin naming where each rule actually lives so it cannot drift from its source.
Mechanical and behavioral boundaries are distinguished in that declaration, and the behavioral ones are marked as out of scope for the offline check rather than silently counted as covered.
aos-eval boundaries check runs as a just verb or in CI, and its current gap is recorded rather than suppressed.
A follow-up issue carries the behavioral half, scoped by what the mechanical pass leaves over.

Not in scope

The behavioral board itself. Also not the broker or credential layer's own correctness, which is where enforcement actually lives and is a separate question from whether the lane clause is followed.

Source

Opened after a portfolio-wide eval coverage read on 2026-08-18. docs/aos-eval.md in agentic-os names exactly two consumers, agent-compose and sirens-echo. Ward has none, and is the execution authority both of them run underneath.

## Outcome Declare Ward's boundaries in `aos-eval.boundaries.v1` shape, and split them by kind, because **Ward is the first repo in the portfolio that needs both halves of the eval stack rather than one**. The mechanical boundaries are declarable and offline-checkable today. The lane boundaries are claims about agent behavior and need a subject and a grader. Sibling issues: `umbra#305` and `mcp-beaver#84`. Ward differs from both and the difference is the substance of this issue. ## First, a correction to the premise those siblings ran on Read on `main` at depth 30: 997 test functions across 126 `_test.go` files, roughly 81 naming a refusal against roughly 16 naming a permit. On the sibling issues that ratio was the opening argument. **Then I checked the highest-stakes gate in the repo and it is properly paired.** `directorMergeEligibility` in `cmd/ward/agent_director_merge_test.go` carries four refusal tests by name and three unnamed positive-path assertions, at lines 521, 664, and 682, each wanting `true/""/729`. `directorMergeDecision` is paired the same way at line 38. **Nobody would learn that from the function names**, which is exactly the failure mode the siblings warn about, now demonstrated in the other direction. That does not weaken the case for a declaration. It is the case. A name-based count is unreliable in **both** directions, so the only way to know where a half is missing is to declare the complete set and let `aos-eval boundaries check` compare it to what exists. Opening this issue without that check would have been a guess. ## Ward's boundary axis is evidence, not permission umbra and mcp-beaver bound what a caller may reach. Ward bounds **what counts as having happened**. From `docs/agent-workflow.md`: > Harness exit, a local commit, or a stale no-diff salvage branch is not delivery. Each lane declares distinct evidence: remote `main` for direct landing, remote branch plus canonical PR URL plus submitted workflow state for PR landing, current CI plus review authorization plus QA verdict plus `merged: true` for director landing, and the named remote branch for branch-only. The pairing writes itself. The out-half is Ward refusing to report delivery on insufficient evidence, which is the well-covered side. **The in-half is Ward correctly reporting delivery when the evidence is genuinely there**, and a Ward that reported failure unconditionally would pass an evidence-refusal suite perfectly while being useless. ## The second kind, which no declaration can check offline `remote-branch-only` is documented as "publish a remote branch and stop", with no PR or merge authority. In the code that constraint is prompt text. `cmd/ward/agent.go:494`: > For `remote-branch-only` workflows, the remote branch push is the finish line. Do not open a pull request and do not merge. Every reference to `workflowRemoteBranchOnly` outside tests generates prompt material: the carry clause, the landing noun, the outcome status, and one review-gate skip. **Nothing mechanically prevents that run from opening a PR.** **This is by design rather than a defect.** The README says so directly: roles, harnesses, and workflows "select" behavior and "None of those labels grants credentials, mounts, network, broker operations, or merge authority." Enforcement lives in the credential and broker layer, and the lane label is instruction. The consequence for evaluation is the whole point of this issue. **"Does an agent in a `remote-branch-only` lane actually stop at the branch" is a behavioral claim about a model reading a prompt clause.** That is precisely what sirens-echo's board measures, cases authored against prompt clauses cited by line, and it cannot be answered by a declaration, a unit test, or an offline coverage tool. ## So the split * **Mechanical** - merge eligibility, `MergePullRequestRefusesFromReadOnlySurface`, the clean-and-synced gate, delivery-evidence sufficiency per lane, `AgentFrameworkChildCapabilityCannotSelectFixedWorkflow`. Declarable in `aos-eval.boundaries.v1`, checkable offline with `boundaries derive` and `boundaries check`, no model and no grader. * **Behavioral** - the four lane clauses as an agent actually follows them. Needs the triple, and sirens-echo already proved the prompt-clause citation pattern including a test that fails when a citation drifts off its clause. Do the mechanical half first. It is cheap, it costs no inference, and its output will scope the behavioral half. ## Expect the first output to be a gap Against sirens-echo's pilot board the same tool reported 56 derived slots, none authored. `docs/aos-eval.md`: "The first honest output in a new repo is usually a gap" and "A coverage report that cannot come back negative is decoration." Given what the `directorMergeEligibility` read turned up, Ward's gap will be smaller than the naming suggests and will land in places nobody has looked. ## Complete when * A boundary declaration exists in `aos-eval.boundaries.v1` shape, with `origin` naming where each rule actually lives so it cannot drift from its source. * Mechanical and behavioral boundaries are distinguished in that declaration, and the behavioral ones are marked as out of scope for the offline check rather than silently counted as covered. * `aos-eval boundaries check` runs as a `just` verb or in CI, and its current gap is recorded rather than suppressed. * A follow-up issue carries the behavioral half, scoped by what the mechanical pass leaves over. ## Not in scope The behavioral board itself. Also not the broker or credential layer's own correctness, which is where enforcement actually lives and is a separate question from whether the lane clause is followed. ## Source Opened after a portfolio-wide eval coverage read on 2026-08-18. `docs/aos-eval.md` in agentic-os names exactly two consumers, agent-compose and sirens-echo. Ward has none, and is the execution authority both of them run underneath.

coilyco-ops referenced this issue from coilyco-flight-deck/umbra

2026-08-18 22:31:02 +00:00

Declare umbra boundaries in aos-eval shape: the guard is tested for what it refuses, not for what it must still serve #305

coilyco-ops referenced this issue from coilyco-flight-deck/mcp-beaver

2026-08-18 22:31:05 +00:00

Declare mcp-beaver grant boundaries in aos-eval shape: deny-by-absence is tested, "everything you declared works" is not #84