Workspace command execution and attachment ingest have no telemetry at all, so the audit record omits the effects that matter most #890

New issue

Closed

opened 2026-08-17 04:43:40 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-17 04:43:40 +00:00

Member

Found by Angie (ENG, claude seat) while building the tool-call mirror on #887, from a question Kai asked about what actually counts as a tool call.

What is covered, so the gap is precise

Everything the model can call is recorded, including the harness-internal surfaces, because they share one dispatch point. proxy.go:651 is the only place a model-requested tool is invoked, compositeSession.Call fans out by name, and RecordToolCall fires immediately after.

So all of these produce a metric and a mirrored record, told apart by the server field:

MCP roster servers - eco, forgejo, steam
scratchpad - scratch_list, scratch_read, scratch_write, scratch_search
fetch - fetch_url
harness - refresh_tools, handled inside mcpToolSession.Call
repo inventory

That half is fine. A harness-internal tool is not a second class of thing; it is the same call through the same seam.

The gap: effects that are not model tool calls at all

Two of them have no observability whatsoever. Not absent from the mirror - absent from spans, metrics, and logs.

$ grep -c "telemetry\.\|StartSpan\|Record" internal/community/workspace.go
0
$ grep -c "telemetry\.\|StartSpan\|Record" internal/community/attachmentingest.go
0

WardCommandRunner.Run executes a process. It builds an exec.CommandContext in a job workspace with an explicit environment, runs it, and bounds the output. Nothing records that it happened, what ran, how long it took, or what it exited with. jobexec.go adds nothing either.

For an audit record this is the inversion of what you would want. A read-only eco__get_market is fully recorded, and the harness running a command is not.

Attachment ingest performs network egress. It fetches a member's upload from cdn.discordapp.com and writes it into the requester's scratchpad. Bounded and confined, and unrecorded.

Lesser, and listed so the sweep is complete:

spillToolResult writes a file. It happens inside a tool call so it is attributed to one, but the write itself is not named.
Discord writes - the reply send, thread creation, reactions.
Thread prefill history reads.

Two questions, and only the first is mine

1. Should these emit telemetry at all? Yes, and that part is ordinary work. A command execution wants a span and an outcome the way a tool call has one, keyed on the same closed vocabulary: what ran, how long, what it exited with. No arguments, no output, no environment - those carry content and a token respectively.

2. Should they reach the Temporal mirror? That is a disclosure decision rather than an engineering one, and #887's whole design says a person makes it on purpose. ToolCallRecord is deliberately a fixed struct so widening what leaves this process to a third-party SaaS is a reviewed act. A command name is closer to content than mcp.tool.name is, and it deserves the same scrutiny the payload got.

Question 1 does not depend on question 2, and doing it first is what makes question 2 answerable with real data.

Acceptance for the part that is decided

A workspace command execution produces a span and a metric carrying what ran, its duration, and its exit status, with no arguments, no output, and no environment.
Attachment ingest produces the same for its fetch: outcome, byte count, duration. No filename, no content.
Both use the existing curated vocabulary rather than inventing labels.
Nothing new reaches the Temporal mirror without an explicit decision on question 2.

Related - #887 (the mirror and its payload rule), #621 (the sweep for capabilities that exist and do not run).

Found by Angie (ENG, `claude` seat) while building the tool-call mirror on #887, from a question Kai asked about what actually counts as a tool call. ## What is covered, so the gap is precise Everything the **model** can call is recorded, including the harness-internal surfaces, because they share one dispatch point. `proxy.go:651` is the only place a model-requested tool is invoked, `compositeSession.Call` fans out by name, and `RecordToolCall` fires immediately after. So all of these produce a metric and a mirrored record, told apart by the server field: * MCP roster servers - `eco`, `forgejo`, `steam` * `scratchpad` - `scratch_list`, `scratch_read`, `scratch_write`, `scratch_search` * `fetch` - `fetch_url` * `harness` - `refresh_tools`, handled inside `mcpToolSession.Call` * repo inventory **That half is fine.** A harness-internal tool is not a second class of thing; it is the same call through the same seam. ## The gap: effects that are not model tool calls at all Two of them have **no observability whatsoever**. Not absent from the mirror - absent from spans, metrics, and logs. ``` $ grep -c "telemetry\.\|StartSpan\|Record" internal/community/workspace.go 0 $ grep -c "telemetry\.\|StartSpan\|Record" internal/community/attachmentingest.go 0 ``` **`WardCommandRunner.Run` executes a process.** It builds an `exec.CommandContext` in a job workspace with an explicit environment, runs it, and bounds the output. Nothing records that it happened, what ran, how long it took, or what it exited with. `jobexec.go` adds nothing either. For an audit record this is the inversion of what you would want. A read-only `eco__get_market` is fully recorded, and the harness running a command is not. **Attachment ingest performs network egress.** It fetches a member's upload from `cdn.discordapp.com` and writes it into the requester's scratchpad. Bounded and confined, and unrecorded. Lesser, and listed so the sweep is complete: * `spillToolResult` writes a file. It happens inside a tool call so it is attributed to one, but the write itself is not named. * Discord writes - the reply send, thread creation, reactions. * Thread prefill history reads. ## Two questions, and only the first is mine **1. Should these emit telemetry at all?** Yes, and that part is ordinary work. A command execution wants a span and an outcome the way a tool call has one, keyed on the same closed vocabulary: what ran, how long, what it exited with. No arguments, no output, no environment - those carry content and a token respectively. **2. Should they reach the Temporal mirror?** That is a disclosure decision rather than an engineering one, and #887's whole design says a person makes it on purpose. `ToolCallRecord` is deliberately a fixed struct so widening what leaves this process to a third-party SaaS is a reviewed act. A command name is closer to content than `mcp.tool.name` is, and it deserves the same scrutiny the payload got. Question 1 does not depend on question 2, and doing it first is what makes question 2 answerable with real data. ## Acceptance for the part that is decided * A workspace command execution produces a span and a metric carrying what ran, its duration, and its exit status, with no arguments, no output, and no environment. * Attachment ingest produces the same for its fetch: outcome, byte count, duration. No filename, no content. * Both use the existing curated vocabulary rather than inventing labels. * Nothing new reaches the Temporal mirror without an explicit decision on question 2. **Related** - #887 (the mirror and its payload rule), #621 (the sweep for capabilities that exist and do not run).

coilyco-ops added the

labels

2026-08-17 04:44:02 +00:00

coilyco-ops referenced this issue from a commit

2026-08-17 05:00:57 +00:00

feat(telemetry): record the effects the harness caused itself

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-17 05:01:19 +00:00