Job-scoped telemetry and progress reporting

coilyco-ops commented

2026-08-12 11:11:27 +00:00

Member

Outcome

Telemetry is addressable by unit of work. "Show me run 47" is answerable, and a long job reports progress while it runs rather than only at the end.

Why

Telemetry today is service-scoped. Traces carry service.name = sirens-deep, logs carry k8s.namespace.name = sirens-deep (see coilyco-bridge/deploy#386 for why those differ). Both answer "how is the service doing." Neither answers "what happened in that run," which is the only question that matters once work is durable.

Scope

Item 13 - job-scoped telemetry.

A job id propagated as a span attribute and a log attribute on everything the job does.
The existing structured logging already promotes trace_id and span_id to row fields, so a job id joins them naturally and the correlation path already works.
A job's full trace and log history is retrievable by job id alone.

Item 14 - progress reporting.

Long-running jobs emit progress back to the originating transport during execution.
Progress updates edit or thread rather than spamming a channel. Mention safety applies: a progress update must not create notifications.
Rate-limited, so a chatty job cannot flood the origin.

Acceptance

Given a job id, its spans and logs are retrievable without knowing the service or namespace.
A long job shows intermediate progress at the origin.
Progress updates create no unintended notifications.

Depends on

#143 - there is no job id to attribute against until the job record exists.

Next owner

Engineer.

## Outcome Telemetry is addressable by unit of work. "Show me run 47" is answerable, and a long job reports progress while it runs rather than only at the end. ## Why Telemetry today is service-scoped. Traces carry `service.name = sirens-deep`, logs carry `k8s.namespace.name = sirens-deep` (see https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/386 for why those differ). Both answer "how is the service doing." Neither answers "what happened in that run," which is the only question that matters once work is durable. ## Scope **Item 13 - job-scoped telemetry.** * A job id propagated as a span attribute and a log attribute on everything the job does. * The existing structured logging already promotes `trace_id` and `span_id` to row fields, so a job id joins them naturally and the correlation path already works. * A job's full trace and log history is retrievable by job id alone. **Item 14 - progress reporting.** * Long-running jobs emit progress back to the originating transport during execution. * Progress updates edit or thread rather than spamming a channel. Mention safety applies: a progress update must not create notifications. * Rate-limited, so a chatty job cannot flood the origin. ## Acceptance * Given a job id, its spans and logs are retrievable without knowing the service or namespace. * A long job shows intermediate progress at the origin. * Progress updates create no unintended notifications. ## Depends on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/143 - there is no job id to attribute against until the job record exists. ## Next owner Engineer.

coilyco-ops referenced this issue

2026-08-12 11:12:55 +00:00

Job record: make a unit of work first-class, durable, and idempotent #143

coilyco-ops referenced this issue

2026-08-12 11:56:36 +00:00

Async job lifecycle: submit, poll, notify, cancel #144

coilyco-ops commented

2026-08-12 11:56:47 +00:00

Author

Member

Timing, and an overlap with #111

Direction from Kai, 2026-08-12 session. Batch-wide decisions are recorded on the entry point, #143.

Timing. The demo track owns the week to August 19. The batch starts August 20 and is the priority from then on. This unit begins once #143 lands, in parallel with #144 and the thread-binding half of #147.

No scope change. Item 13 (job-scoped telemetry) and item 14 (progress reporting) stand as written, including that a job id joins trace_id and span_id as a promoted row field, and that progress updates must create no unintended notifications.

Two overlaps worth planning around

Item 14 overlaps #111 substantially. #111 was decided the same session and builds a Discord progress element: a rich embed, edits on tool boundaries only, disclosure of stage plus server plus tool names but never arguments, and every terminal state resolving immediately through notifyFailure on a context detached from the turn deadline.

That is the same problem this unit's item 14 solves, one layer down — #111 reports turn progress, this reports job progress. #111 is deferred past August 19 and this unit starts after it, so item 14 should extend that element rather than build a second progress mechanism. The rate-limiting requirement in this issue and the edit-budget reasoning in #111 are the same constraint discovered twice.

Item 13 overlaps #139. That issue is landing per-server, per-phase MCP spans so a failing call is identifiable from telemetry alone. Both are attribute work on the same spans. If #139 lands first, as decided, this unit inherits a finer span structure to attach a job id to.

On the deploy-side note

The issue references coilyco-bridge/deploy#386 for why traces carry service.name and logs carry k8s.namespace.name. That asymmetry is unchanged by anything decided here — the job id has to be attached on both paths independently, which is the point of the acceptance criterion that a job's spans and logs be retrievable by job id alone, without knowing service or namespace.

## Timing, and an overlap with #111 Direction from Kai, 2026-08-12 session. Batch-wide decisions are recorded on the entry point, #143. **Timing.** The demo track owns the week to August 19. The batch starts August 20 and is the priority from then on. This unit begins once #143 lands, in parallel with #144 and the thread-binding half of #147. **No scope change.** Item 13 (job-scoped telemetry) and item 14 (progress reporting) stand as written, including that a job id joins `trace_id` and `span_id` as a promoted row field, and that progress updates must create no unintended notifications. ## Two overlaps worth planning around **Item 14 overlaps #111 substantially.** #111 was decided the same session and builds a Discord progress element: a rich embed, edits on tool boundaries only, disclosure of stage plus server plus tool names but never arguments, and every terminal state resolving immediately through `notifyFailure` on a context detached from the turn deadline. That is the same problem this unit's item 14 solves, one layer down — #111 reports turn progress, this reports job progress. #111 is deferred past August 19 and this unit starts after it, so **item 14 should extend that element rather than build a second progress mechanism.** The rate-limiting requirement in this issue and the edit-budget reasoning in #111 are the same constraint discovered twice. **Item 13 overlaps #139.** That issue is landing per-server, per-phase MCP spans so a failing call is identifiable from telemetry alone. Both are attribute work on the same spans. If #139 lands first, as decided, this unit inherits a finer span structure to attach a job id to. ## On the deploy-side note The issue references `coilyco-bridge/deploy#386` for why traces carry `service.name` and logs carry `k8s.namespace.name`. That asymmetry is unchanged by anything decided here — the job id has to be attached on both paths independently, which is the point of the acceptance criterion that a job's spans and logs be retrievable by job id alone, without knowing service or namespace.

coilyco-ops referenced this issue

2026-08-12 11:56:57 +00:00

Structured commands and thread-to-job binding #147

coilyco-ops referenced this issue

2026-08-12 11:58:33 +00:00

Attribution: every effect is traceable to the principal who requested it #151

coilyco-ops referenced this issue

2026-08-12 11:58:51 +00:00

Job record: make a unit of work first-class, durable, and idempotent #143

coilyco-ops referenced this issue

2026-08-12 12:03:03 +00:00

Attribution: every effect is traceable to the principal who requested it #151

coilyco-ops commented

2026-08-12 12:03:09 +00:00

Author

Member

The principal is a sibling attribute — build item 13 with #151

Direction from Kai, 2026-08-12 session.

#151 (attribution, item 6) settled its telemetry question: the requesting principal goes into the span as a first-class attribute, rather than being recovered by resolving a job id against the job store.

So item 13 here and #151's instrumentation are one pass, not two. This issue promotes a job id onto every span and log the job produces; #151 promotes the requesting principal the same way. Same mechanism, same code path, same review.

One difference in coverage worth building for deliberately: the job id exists only where a job exists, while the principal covers the whole request path — including turns that fail before a job is created, admission denials, and validation rejections. So the principal attribute is set earlier in the request lifecycle than the job id is. Do not attach them at the same point just because they are attached the same way.

The recorded value is the Discord user ID, not the handle. See #151 for the reasoning and for the retention question it opens.

Unchanged

This issue's own acceptance criterion stands as written — given a job id, its spans and logs are retrievable without knowing the service or namespace. Adding a principal attribute is additive to that, not a substitute for it.

## The principal is a sibling attribute — build item 13 with #151 Direction from Kai, 2026-08-12 session. #151 (attribution, item 6) settled its telemetry question: the requesting principal goes into the span as a first-class attribute, rather than being recovered by resolving a job id against the job store. **So item 13 here and #151's instrumentation are one pass, not two.** This issue promotes a job id onto every span and log the job produces; #151 promotes the requesting principal the same way. Same mechanism, same code path, same review. One difference in coverage worth building for deliberately: the **job id exists only where a job exists**, while the **principal covers the whole request path** — including turns that fail before a job is created, admission denials, and validation rejections. So the principal attribute is set earlier in the request lifecycle than the job id is. Do not attach them at the same point just because they are attached the same way. The recorded value is the Discord user ID, not the handle. See #151 for the reasoning and for the retention question it opens. ## Unchanged This issue's own acceptance criterion stands as written — given a job id, its spans and logs are retrievable without knowing the service or namespace. Adding a principal attribute is additive to that, not a substitute for it.

coilyco-ops referenced this issue from a commit

2026-08-12 12:10:31 +00:00

feat(jobs): submit, poll, notify, and cancel work that outlives its turn

coilyco-ops referenced this issue from a commit

2026-08-12 12:13:40 +00:00

feat(jobs): make telemetry addressable by unit of work

coilyco-ops closed this issue

2026-08-12 12:13:40 +00:00