The generated binary collapses every coded error to exit 1, so the exit-code taxonomy never reaches an orchestrator #341

New issue

Closed

opened 2026-08-30 04:16:18 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-30 04:16:18 +00:00

Owner

Found while demonstrating the mcp dialect from the released v0.186.0 binary. Pre-existing and not introduced by #337: it affects the spec and exec dialects identically, because the fault is in the shared main.go template.

Measured

A guarded CLI generated by specgen build, over a real MCP upstream, v0.186.0 on macOS:

exit=1 | policy refusal (allow guard)
   demo: argument path="/etc/shadow" is outside the allowed scope (allow path matches [^/var/log])
exit=1 | missing required input
   demo: stat-path requires --path
exit=3 | denied tool (never call)
   No help topic for 'read-text-head'
exit=0 | permitted call

The taxonomy says otherwise

pkg/exitcode at the same tag:

Success        = 0
Generic        = 1
PolicyDenied   = 2
UpstreamFailed = 3
Internal       = 4
UserError      = 5

A policy refusal should be 2 and a missing input 5. Both come back as 1.

Cause

http/specgen/codegen/codegen.go, the generated main:

func main() {
	if err := run(); err != nil {
		fmt.Fprintln(os.Stderr, "{{.Binary}}:", err)
		os.Exit(1)
	}
}

The template never imports pkg/exitcode, so the coded error every engine carefully constructs is printed and then discarded at the process boundary. grep -c exitcode over the template returns 0.

The engines are doing their part: opcore returns exitcode.New(exitcode.PolicyDenied, ...) and friends throughout. Only the last four lines throw the code away.

The exit 3 on an absent tool is urfave's own "no help topic" exit, not the taxonomy's UpstreamFailed. That it happens to be a taxonomy number is a coincidence, and a confusing one.

Why this matters more than a cosmetic code

docs/architecture.md sells the taxonomy as "a public exit-code taxonomy for orchestrators", and the generated binary is the main thing an orchestrator runs. As shipped, a caller cannot tell a policy refusal from a tool failure from a bad flag without parsing stderr prose, which is the thing exit codes exist to avoid.

It also makes a claim in #336 and #337 wrong. I wrote there that "the audit row and the exit-code taxonomy all bind unchanged" for an mcp leaf. The audit row does. The taxonomy is carried in the error and then dropped, for every dialect. Corrected on #336.

Do

Generated main unwraps the coded error and exits with its code, defaulting to Generic when the error carries none.
Decide what an unknown subcommand should exit. urfave's 3 currently collides with UpstreamFailed, which is worse than an uncoded 1.
A test asserting a policy refusal from a generated binary exits 2, since this is the kind of thing that regresses silently.

Acceptance

specgen build output returns 2 on a guard refusal, 5 on a missing required input, 0 on success, and no taxonomy code is produced by a path that did not mean it.

Found while demonstrating the mcp dialect from the released v0.186.0 binary. **Pre-existing and not introduced by #337**: it affects the spec and exec dialects identically, because the fault is in the shared `main.go` template. ## Measured A guarded CLI generated by `specgen build`, over a real MCP upstream, v0.186.0 on macOS: ``` exit=1 | policy refusal (allow guard) demo: argument path="/etc/shadow" is outside the allowed scope (allow path matches [^/var/log]) exit=1 | missing required input demo: stat-path requires --path exit=3 | denied tool (never call) No help topic for 'read-text-head' exit=0 | permitted call ``` ## The taxonomy says otherwise `pkg/exitcode` at the same tag: ```go Success = 0 Generic = 1 PolicyDenied = 2 UpstreamFailed = 3 Internal = 4 UserError = 5 ``` A policy refusal should be **2** and a missing input **5**. Both come back as 1. ## Cause `http/specgen/codegen/codegen.go`, the generated `main`: ```go func main() { if err := run(); err != nil { fmt.Fprintln(os.Stderr, "{{.Binary}}:", err) os.Exit(1) } } ``` The template never imports `pkg/exitcode`, so the coded error every engine carefully constructs is printed and then discarded at the process boundary. `grep -c exitcode` over the template returns 0. The engines are doing their part: `opcore` returns `exitcode.New(exitcode.PolicyDenied, ...)` and friends throughout. Only the last four lines throw the code away. The exit 3 on an absent tool is urfave's own "no help topic" exit, not the taxonomy's `UpstreamFailed`. That it happens to be a taxonomy number is a coincidence, and a confusing one. ## Why this matters more than a cosmetic code `docs/architecture.md` sells the taxonomy as "a public exit-code taxonomy for orchestrators", and the generated binary is the main thing an orchestrator runs. As shipped, a caller cannot tell a policy refusal from a tool failure from a bad flag without parsing stderr prose, which is the thing exit codes exist to avoid. It also makes a claim in #336 and #337 wrong. I wrote there that "the audit row and the exit-code taxonomy all bind unchanged" for an mcp leaf. The audit row does. The taxonomy is carried in the error and then dropped, for every dialect. Corrected on #336. ## Do 1. Generated `main` unwraps the coded error and exits with its code, defaulting to `Generic` when the error carries none. 2. Decide what an unknown subcommand should exit. urfave's 3 currently collides with `UpstreamFailed`, which is worse than an uncoded 1. 3. A test asserting a policy refusal from a generated binary exits 2, since this is the kind of thing that regresses silently. ## Acceptance `specgen build` output returns 2 on a guard refusal, 5 on a missing required input, 0 on success, and no taxonomy code is produced by a path that did not mean it.

coilyco-ops commented

2026-08-30 04:17:26 +00:00

Author

Owner

The audit row loses the code too, so this is two consumers, not one

Same session, same generated binary. The rows for the four invocations above:

{"ts":1788063341,"verb":"demo.ops.nodestats.stat-path","argv":[...,"--path","/etc/shadow"],"exit_code":1}
{"ts":1788063341,"verb":"demo.ops.nodestats.stat-path","argv":[...,"stat-path"],"exit_code":1}
{"ts":1788063341,"verb":"demo.ops.nodestats.stat-path","argv":[...,"--path","/var/log"],"exit_code":0}

Row 1 is a policy refusal and row 2 is a missing required input. Both record exit_code: 1, so the append-only log cannot distinguish "the guard refused this" from "the caller passed a bad flag" either. Reviewing the audit trail for refusals means grepping prose, the same way an orchestrator would have to.

That matters more than the process code on its own. The audit log is the durable record, and it is the artifact someone reads weeks later when the operator and the terminal are long gone.

So a fix belongs where the coded error is still intact, rather than only at os.Exit. verb.Wrap sees the error before the audit record is written, which is upstream of both consumers.

Worth checking as part of this: whether audit.Record already has somewhere to put a policy outcome distinct from the process exit code. If it does not, that field is part of the fix rather than a follow-up, because adding it later means the rows written in between are the ones that stay ambiguous.

## The audit row loses the code too, so this is two consumers, not one Same session, same generated binary. The rows for the four invocations above: ```json {"ts":1788063341,"verb":"demo.ops.nodestats.stat-path","argv":[...,"--path","/etc/shadow"],"exit_code":1} {"ts":1788063341,"verb":"demo.ops.nodestats.stat-path","argv":[...,"stat-path"],"exit_code":1} {"ts":1788063341,"verb":"demo.ops.nodestats.stat-path","argv":[...,"--path","/var/log"],"exit_code":0} ``` Row 1 is a policy refusal and row 2 is a missing required input. Both record `exit_code: 1`, so the append-only log cannot distinguish "the guard refused this" from "the caller passed a bad flag" either. Reviewing the audit trail for refusals means grepping prose, the same way an orchestrator would have to. That matters more than the process code on its own. The audit log is the durable record, and it is the artifact someone reads weeks later when the operator and the terminal are long gone. So a fix belongs where the coded error is still intact, rather than only at `os.Exit`. `verb.Wrap` sees the error before the audit record is written, which is upstream of both consumers. Worth checking as part of this: whether `audit.Record` already has somewhere to put a policy outcome distinct from the process exit code. If it does not, that field is part of the fix rather than a follow-up, because adding it later means the rows written in between are the ones that stay ambiguous.

coilyco-ops referenced this issue

2026-08-30 04:17:42 +00:00

upgrade specgen into a mcporter replacement #336

coilyco-ops referenced this issue from a commit

2026-08-30 05:08:32 +00:00

fix(exitcode): let the taxonomy reach the two consumers that read it

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-30 05:08:33 +00:00

Let the exit-code taxonomy reach the two consumers that read it #342