coilyco-gaming/sirens-echo

Watch

Mirror tool-call metadata into Temporal Cloud from the telemetry seam, allowlisted and fire-and-forget #887

New issue

Closed

opened 2026-08-17 04:18:27 +00:00 by coilyco-ops · 1 comment

coilyco-ops commented

2026-08-17 04:18:27 +00:00

Member

Outcome

Deep emits a durable record of its tool-call trajectory into Temporal Cloud, carrying metadata only, hooked at the same place that starts OTel spans.

Design settled with Kai 2026-08-16. This is mirroring, not orchestration. Temporal observes the turn, it does not run it.

Why this shape rather than the one in #430

#430 records a decision to make the turn a workflow and every tool call an activity. That is control-flow integration and it carries two hazards this design does not:

Retry multiplication. A Temporal activity retry policy stacked on the agent-proxy fallback for sirens-echo/deepseek compounds, turning a 502 into a slow expensive 502. Mirroring keeps Temporal out of the control path, so nothing compounds.
Latency in the user-visible turn. An activity round trip per tool call sits in front of a Discord user. An async mirror does not. Dead air is the demo failure mode.

Mirroring is also the more honest claim. "We mirror our agent's tool-call trajectory into Temporal so there is a durable audit record" is true and specific. "Our harness runs on Temporal" would not have been.

The seam

internal/community/telemetry.go:

func (t *Telemetry) StartSpan(ctx, name, attributes ...attribute.KeyValue)

StartSpan is content-blind by construction. The only attribute it adds is sirens_echo.job.id. That makes it a safe place to hook.

It is a variadic passthrough, so it is not a safe place to copy from. Every other attribute comes from callers. Mirroring the attribute slice means mirroring whatever a caller passed, sight unseen, forever.

Requirements

Allowlist the emitted keys. Never pass through the attribute slice. The failure this prevents: someone later adds attribute.String("message.text", ...) to a span, and Discord member content starts flowing to a third-party SaaS with no change at the mirror and nothing to notice it. #385 already ruled that the disclosure footer never echoes tool arguments, for this same class of reason.
Use the existing metadata vocabulary as the allowlist. The Record* functions are already curated and carry zero content:
- RecordToolCall - mcp.server.name, mcp.tool.name, outcome
- RecordTurn, RecordModelCall - outcome
- RecordAdmission - outcome, transport
- RecordFailure - stage
- RecordAccess - reason
- StartJobSpan - job.id, job.kind, job.transport, job.attempt
RecordToolCall's triple plus timing is the payload this issue wants.
Never await it, never let it raise. StartSpan is on the hot path for every span in the harness. A mirror that blocks or propagates an error becomes a new way for turns to die silently, which is the shape of #137 and #190. Fire and forget, hard timeout, swallow the error, and count the drops so the silence is measurable.
Key off tool calls, not every span. For scale reference, agent-proxy logged roughly 68,900 http receive spans against about 4,300 real requests over 30 days. Hooking span-start indiscriminately points a firehose at a service that bills per action, and #431 already records per-turn cost being understated about ninefold.
No message content, no tool arguments, no tool results, no principal identifiers. #310 and #180 are open on principal-ID disclosure. Nothing in this mirror should be capable of carrying one.

Scope boundary

Deep only. Echo is out of scope.
The isolated owl-glass surface, not the demo Discord instance, until this is proven.
Deployment wiring is tracked separately in coilyco-bridge/deploy.
Credential provisioning is #444.

Acceptance

A tool call produces a Temporal Cloud record carrying server name, tool name, outcome, and timing.
Grepping the mirrored payload finds no message text, tool arguments, tool results, or user identifiers.
An induced Temporal outage leaves turns unaffected, and the dropped-mirror count is observable.
Mirror volume is roughly one action per tool call rather than one per span.
Adding a content-bearing attribute to any span does not change what the mirror emits.

Next owner

Engineer.

## Outcome Deep emits a durable record of its tool-call trajectory into Temporal Cloud, carrying **metadata only**, hooked at the same place that starts OTel spans. Design settled with Kai 2026-08-16. This is **mirroring, not orchestration.** Temporal observes the turn, it does not run it. ## Why this shape rather than the one in #430 https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/430 records a decision to make the turn a workflow and every tool call an activity. That is control-flow integration and it carries two hazards this design does not: * **Retry multiplication.** A Temporal activity retry policy stacked on the agent-proxy fallback for `sirens-echo/deepseek` compounds, turning a 502 into a slow expensive 502. Mirroring keeps Temporal out of the control path, so nothing compounds. * **Latency in the user-visible turn.** An activity round trip per tool call sits in front of a Discord user. An async mirror does not. Dead air is the demo failure mode. Mirroring is also the more honest claim. "We mirror our agent's tool-call trajectory into Temporal so there is a durable audit record" is true and specific. "Our harness runs on Temporal" would not have been. ## The seam `internal/community/telemetry.go`: ```go func (t *Telemetry) StartSpan(ctx, name, attributes ...attribute.KeyValue) ``` `StartSpan` is content-blind by construction. The only attribute it adds is `sirens_echo.job.id`. That makes it a safe place to hook. **It is a variadic passthrough, so it is not a safe place to copy from.** Every other attribute comes from callers. Mirroring the attribute slice means mirroring whatever a caller passed, sight unseen, forever. ## Requirements 1. **Allowlist the emitted keys. Never pass through the attribute slice.** The failure this prevents: someone later adds `attribute.String("message.text", ...)` to a span, and Discord member content starts flowing to a third-party SaaS with no change at the mirror and nothing to notice it. https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/385 already ruled that the disclosure footer never echoes tool arguments, for this same class of reason. 2. **Use the existing metadata vocabulary as the allowlist.** The `Record*` functions are already curated and carry zero content: * `RecordToolCall` - `mcp.server.name`, `mcp.tool.name`, `outcome` * `RecordTurn`, `RecordModelCall` - `outcome` * `RecordAdmission` - `outcome`, `transport` * `RecordFailure` - `stage` * `RecordAccess` - `reason` * `StartJobSpan` - `job.id`, `job.kind`, `job.transport`, `job.attempt` `RecordToolCall`'s triple plus timing is the payload this issue wants. 3. **Never await it, never let it raise.** `StartSpan` is on the hot path for every span in the harness. A mirror that blocks or propagates an error becomes a new way for turns to die silently, which is the shape of #137 and #190. Fire and forget, hard timeout, swallow the error, and count the drops so the silence is measurable. 4. **Key off tool calls, not every span.** For scale reference, agent-proxy logged roughly 68,900 `http receive` spans against about 4,300 real requests over 30 days. Hooking span-start indiscriminately points a firehose at a service that bills per action, and #431 already records per-turn cost being understated about ninefold. 5. **No message content, no tool arguments, no tool results, no principal identifiers.** #310 and #180 are open on principal-ID disclosure. Nothing in this mirror should be capable of carrying one. ## Scope boundary * Deep only. Echo is out of scope. * The isolated owl-glass surface, not the demo Discord instance, until this is proven. * Deployment wiring is tracked separately in `coilyco-bridge/deploy`. * Credential provisioning is https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/444. ## Acceptance * A tool call produces a Temporal Cloud record carrying server name, tool name, outcome, and timing. * Grepping the mirrored payload finds no message text, tool arguments, tool results, or user identifiers. * An induced Temporal outage leaves turns unaffected, and the dropped-mirror count is observable. * Mirror volume is roughly one action per tool call rather than one per span. * Adding a content-bearing attribute to any span does not change what the mirror emits. ## Next owner Engineer.

coilyco-ops added the

labels

2026-08-17 04:18:32 +00:00

coilyco-ops referenced this issue

2026-08-17 04:19:04 +00:00

Epic: Temporal Cloud orchestration — every tool call an activity, landing before August 19 #430

coilyco-ops referenced this issue

2026-08-17 04:32:24 +00:00

Provision the Temporal Cloud credential, which blocks nothing and is blocked by nothing #444

coilyco-ops referenced this issue from a commit

2026-08-17 04:36:58 +00:00

feat(mirror): send the tool-call trajectory to Temporal, metadata only

coilyco-ops referenced this issue from a pull request that will close it,

2026-08-17 04:37:33 +00:00

feat(mirror): send the tool-call trajectory to Temporal, metadata only #888

coilyco-ops referenced this issue

2026-08-17 04:39:54 +00:00

Provision the Temporal Cloud credential, which blocks nothing and is blocked by nothing #444

coilyco-ops referenced this issue

2026-08-17 04:40:30 +00:00

Decide whether the Temporal Cloud mirror is worth roughly $100/month once the trial credit lapses #889

coilyco-ops referenced this issue

2026-08-17 04:43:34 +00:00

Provision the Temporal Cloud credential, which blocks nothing and is blocked by nothing #444

coilysiren closed this issue

2026-08-17 04:43:35 +00:00

coilyco-ops referenced this issue

2026-08-17 04:43:41 +00:00

Workspace command execution and attachment ingest have no telemetry at all, so the audit record omits the effects that matter most #890

coilyco-ops referenced this issue

2026-08-17 04:47:40 +00:00

Provision the Temporal Cloud credential, which blocks nothing and is blocked by nothing #444

coilyco-ops referenced this issue

2026-08-17 05:01:19 +00:00

feat(telemetry): record the effects the harness caused itself #891

coilyco-ops referenced this issue

2026-08-17 05:09:58 +00:00

Provision the Temporal Cloud credential, which blocks nothing and is blocked by nothing #444

coilyco-ops commented

2026-08-17 05:15:36 +00:00

Author

Member

The `identity` field defeats requirement 5, and the allowlist cannot see it

Recorded by Olaf (ops seat). Found by leaking one, so this is demonstrated rather than theorized.

While probing Write permission against coilyco.gcdqf I started and terminated a throwaway workflow. Its termination event:

{
  "type": "workflowExecutionTerminatedEventAttributes",
  "reason": "ops write-permission probe, cleanup",
  "identity": "temporal-cli:kai@kais-macbook-pro-2.local"
}

A username and a hostname, written into a third-party SaaS, by a command that passed no such thing.

Why this defeats the design as written

Requirement 1 says allowlist the emitted keys and never pass the attribute slice through. Requirement 2 anchors that allowlist to the curated Record* vocabulary. Both are correct and neither helps here.

identity is not an attribute. It is SDK-level metadata that Temporal stamps onto workflow starts, signals, activity completions, and terminations, underneath whatever payload the caller constructs. An allowlist over RecordToolCall's mcp.server.name, mcp.tool.name, and outcome never sees it, because it is not in the same layer.

The consequence lands directly on the acceptance criteria:

Grepping the mirrored payload finds no message text, tool arguments, tool results, or user identifiers.

That criterion passes while the record carries a host identifier. The payload is clean. The envelope is not. A check written against the payload confirms exactly the wrong thing, and confirms it convincingly.

What Deep would actually emit

The Go SDK defaults Identity to <pid>@<hostname>. In sirens-deep the hostname is the pod name, so every mirrored record would carry something shaped like:

1@sirens-deep-5b6c89959c-87sq2

Not a user identifier, so requirement 5's exact wording survives. It is a host identifier and a ReplicaSet hash, which #310 and #180 are open about as a class, and which the fleet safety rule against host identifiers in durable artifacts covers directly. It also silently republishes cluster topology to a third party on every tool call.

The fix

Set Identity explicitly in the client options to a fixed constant, for example sirens-deep-mirror. One line, and it must be deliberate because the default is the leak.

Suggested additions to acceptance, since the current wording would not have caught this:

The mirror sets a constant Identity and never the SDK default.
Grepping the mirrored event, envelope included rather than payload alone, finds no hostname, pod name, username, or principal identifier.

The second matters more than the first. The current criterion is scoped to the payload, which is the one place the design already guarantees is clean.

Housekeeping

The probe workflow ops-write-probe-1 is terminated. I attempted to delete its history and could not confirm the deletion - temporal workflow delete hung with no output and the execution still lists as Terminated. It ages out with the namespace's 720h retention. The leaked value is a laptop hostname rather than anything operational, so this is untidy rather than harmful, but it is not cleaned up and should not be recorded as such.

## The `identity` field defeats requirement 5, and the allowlist cannot see it Recorded by Olaf (ops seat). Found by leaking one, so this is demonstrated rather than theorized. While probing Write permission against `coilyco.gcdqf` I started and terminated a throwaway workflow. Its termination event: ```json { "type": "workflowExecutionTerminatedEventAttributes", "reason": "ops write-permission probe, cleanup", "identity": "temporal-cli:kai@kais-macbook-pro-2.local" } ``` A username and a hostname, written into a third-party SaaS, by a command that passed no such thing. ## Why this defeats the design as written Requirement 1 says allowlist the emitted keys and never pass the attribute slice through. Requirement 2 anchors that allowlist to the curated `Record*` vocabulary. Both are correct and neither helps here. **`identity` is not an attribute.** It is SDK-level metadata that Temporal stamps onto workflow starts, signals, activity completions, and terminations, underneath whatever payload the caller constructs. An allowlist over `RecordToolCall`'s `mcp.server.name`, `mcp.tool.name`, and `outcome` never sees it, because it is not in the same layer. The consequence lands directly on the acceptance criteria: > Grepping the mirrored payload finds no message text, tool arguments, tool results, or user identifiers. **That criterion passes while the record carries a host identifier.** The payload is clean. The envelope is not. A check written against the payload confirms exactly the wrong thing, and confirms it convincingly. ## What Deep would actually emit The Go SDK defaults `Identity` to `<pid>@<hostname>`. In `sirens-deep` the hostname is the pod name, so every mirrored record would carry something shaped like: ``` 1@sirens-deep-5b6c89959c-87sq2 ``` Not a user identifier, so requirement 5's exact wording survives. It is a host identifier and a ReplicaSet hash, which https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/310 and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/180 are open about as a class, and which the fleet safety rule against host identifiers in durable artifacts covers directly. It also silently republishes cluster topology to a third party on every tool call. ## The fix Set `Identity` explicitly in the client options to a fixed constant, for example `sirens-deep-mirror`. One line, and it must be deliberate because the default is the leak. Suggested additions to acceptance, since the current wording would not have caught this: * The mirror sets a constant `Identity` and never the SDK default. * Grepping the mirrored **event**, envelope included rather than payload alone, finds no hostname, pod name, username, or principal identifier. The second matters more than the first. The current criterion is scoped to the payload, which is the one place the design already guarantees is clean. ## Housekeeping The probe workflow `ops-write-probe-1` is terminated. **I attempted to delete its history and could not confirm the deletion** - `temporal workflow delete` hung with no output and the execution still lists as `Terminated`. It ages out with the namespace's 720h retention. The leaked value is a laptop hostname rather than anything operational, so this is untidy rather than harmful, but it is not cleaned up and should not be recorded as such.

coilyco-ops referenced this issue

2026-08-17 23:46:23 +00:00

Epic: Temporal Cloud orchestration — every tool call an activity, landing before August 19 #430

coilyco-ops referenced this issue

2026-08-17 23:52:08 +00:00

Every mirrored trajectory closes TimedOut, so the audit record's normal state is indistinguishable from a broken one #930

coilyco-ops referenced this issue

2026-08-19 04:08:28 +00:00

The Temporal namespace shows 0 Completed and 60 TimedOut, which is by design and reads as broken on a Temporal stream #1041

coilyco-ops referenced this issue from a commit

2026-08-19 05:02:16 +00:00

feat(temporal): run the trajectory workflow, so a turn's record closes