Webhook-driven @coilyco-ops summons: ack-on-receipt, poll-for-reply, full o11y #375

New issue

Open

opened 2026-06-18 06:35:14 +00:00 by coilysiren · 1 comment

coilysiren commented

2026-06-18 06:35:14 +00:00

Owner

Why

The @coilyco-ops summon system today (#374, scripts/coilyco_ops_summon.py) is a 30s polling timer that dispatches Goose advisory and posts the answer. Three separate bugs already left it silent on a live summon (coilysiren/inbox#110): a missing token scope, a 403 on the permission gate, and wrong kai-server paths. All three are fixed now, but the shape of the thing means it will be down a large fraction of the time - Goose flakes, the model host flakes, the poll misses, the clone fails. And when it's down, the failure mode is the worst one: the human comments into a void and gets nothing back.

The fix is not "make the poller more reliable." It is "make silence impossible, and make the system observable enough to debug when it misbehaves" - because this is going to be buggy and is a prime o11y target.

What

A webhook-driven summon flow with an immediate acknowledgement and a bounded poll-for-reply, replacing the blind 30s timer's role as the primary path.

The webhook does not synchronously drive Goose ("does not push to claude"). It does the minimum fast work, returns 200, and hands off:

Receive - Forgejo issue_comment/issues webhook hits a tailnet-only receiver (no public surface; same network posture as the rest of the fleet). Verify the Forgejo HMAC signature, reject anything unsigned.
Gate - reuse the existing default-deny authorization (mention match + is_authorized + COILYCO_OPS_SUMMON_ALLOW allowlist) from coilyco_ops_summon.py. Drop non-mentions, unauthorized authors, and the bot's own comments. Idempotent per delivery/comment id.
Acknowledge immediately - post a reply comment ("🪿 summon received, Goose is working - expect a reply within ~N min, or a failure notice") before any Goose work. This is the core requirement: the human is never commenting into a void. Return 200 fast.
Dispatch out of band - enqueue the Goose advisory run (the existing run_goose throwaway-clone path, still advisory-only, never pushes). The webhook handler does not block on it.
Poll for the reply - watch for Goose's answer for X (configurable, default e.g. 15 min). On success, post the answer (or edit the ack into the answer). On timeout or error, post a visible failure comment with the reason and a correlation id. Silence is never an acceptable terminal state.

Keep the existing poller as a reconcile backstop (webhooks get missed/dropped) on a slow cadence, not the primary latency path.

Carry-over correctness bug to fix in the rework: the current poller's idempotency marks a comment "processed" (processed.add(comment["id"])) before the authorization check and Goose dispatch, and persists it at end of poll regardless of outcome. So any transient failure - a 403, a Goose crash, an exception mid-service - permanently burns that summon: the id is recorded as handled and the comment is never retried. Combined with "this fails a lot," that means failed summons silently never get answered. Idempotency must key on a successful reply posted, not on "seen." (Hit live: a 403 during gating recorded inbox#110's comment as processed, so the next clean run skipped it with 0 serviced.)

Observability - first-class, not bolted on

This is the explicit point of the issue. Treat the whole lifecycle as an o11y target.

Structured logs - one JSON event per stage (received, signature_verified, gated, authorized, acked, dispatched, goose_started, goose_finished, replied, timed_out, errored), each carrying a correlation id (Forgejo delivery id + comment id + repo#index) so a single summon is greppable end to end.
Metrics - counters (summons_received, acked, dispatched, replied, unauthorized, timed_out, failed{stage}) and histograms (ack_latency, goose_run_seconds, time_to_reply_seconds). Export to the fleet's existing metrics stack.
Traces - one OTEL trace per summon, spans across receive -> ack -> dispatch -> goose -> reply, exported to the existing tracing backend (Phoenix / o2r). The Goose subprocess gets its own child span.
Error handling - every external call (Forgejo API, git clone, goose run) wrapped; failures are categorized, surfaced as a comment (never swallowed into silence), retried with backoff where safe, and a repeated-failure / dead-letter condition fires the fleet's red-channel alert (Telegram).

Second carry-over bug: run_goose invokes goose run --no-session --system ... -t task without --no-profile, so it inherits whatever default Goose profile/extensions the host has. On the first live reply (inbox/#110 comment 15366) that meant Goose loaded the apps/extensionmanager toolset and had no file tools - it tried read and got "Tool 'read' not found." The advisory run needs the developer/file extensions explicitly (or the anti-thrash --no-profile + a pinned extension set, matching the goose-triage harness), or the answers are useless even when the plumbing works.

Open decisions for the implementer (flag, don't guess)

Receiver host + ingress: new systemd service + tailscale serve, or fold into an existing fleet service? Tailnet-only either way.
Metrics + trace backends: confirm Prometheus + Phoenix (or whatever the current fleet o11y stack is) and the export path.
X poll-window default and the ack copy.
Webhook-secret storage (SSM /forgejo/coilyco-ops/webhook-secret, SecureString) and the per-repo webhook registration (script, mirroring provision-coilyco-ops-bot.sh).
Does the webhook fully replace the 30s timer, or does the timer stay as a slow backstop? (Recommended: keep it slow as a backstop.)

Done when

A signed Forgejo webhook on an @coilyco-ops summon produces an ack comment within seconds, and either a Goose answer or an explicit failure comment within the poll window - never silence.
Logs, metrics, and traces for the full lifecycle are queryable, keyed by correlation id.
Security posture from #374 is preserved (default-deny auth, advisory-only Goose, no main mutation, tailnet-only, signature-verified ingress).

refs #374

## Why The `@coilyco-ops` summon system today (#374, `scripts/coilyco_ops_summon.py`) is a **30s polling timer** that dispatches Goose advisory and posts the answer. Three separate bugs already left it silent on a live summon (`coilysiren/inbox#110`): a missing token scope, a 403 on the permission gate, and wrong kai-server paths. All three are fixed now, but the shape of the thing means it **will be down a large fraction of the time** - Goose flakes, the model host flakes, the poll misses, the clone fails. And when it's down, the failure mode is the worst one: **the human comments into a void** and gets nothing back. The fix is not "make the poller more reliable." It is "make silence impossible, and make the system observable enough to debug when it misbehaves" - because this is going to be buggy and is a prime o11y target. ## What A **webhook-driven** summon flow with an immediate acknowledgement and a bounded poll-for-reply, replacing the blind 30s timer's role as the primary path. The webhook **does not synchronously drive Goose** ("does not push to claude"). It does the minimum fast work, returns 200, and hands off: 1. **Receive** - Forgejo `issue_comment`/`issues` webhook hits a tailnet-only receiver (no public surface; same network posture as the rest of the fleet). Verify the Forgejo HMAC signature, reject anything unsigned. 2. **Gate** - reuse the existing default-deny authorization (mention match + `is_authorized` + `COILYCO_OPS_SUMMON_ALLOW` allowlist) from `coilyco_ops_summon.py`. Drop non-mentions, unauthorized authors, and the bot's own comments. Idempotent per delivery/comment id. 3. **Acknowledge immediately** - post a reply comment ("🪿 summon received, Goose is working - expect a reply within ~N min, or a failure notice") **before** any Goose work. This is the core requirement: the human is never commenting into a void. Return 200 fast. 4. **Dispatch out of band** - enqueue the Goose advisory run (the existing `run_goose` throwaway-clone path, still advisory-only, never pushes). The webhook handler does not block on it. 5. **Poll for the reply** - watch for Goose's answer for `X` (configurable, default e.g. 15 min). On success, post the answer (or edit the ack into the answer). **On timeout or error, post a visible failure comment** with the reason and a correlation id. Silence is never an acceptable terminal state. Keep the existing poller as a **reconcile backstop** (webhooks get missed/dropped) on a slow cadence, not the primary latency path. **Carry-over correctness bug to fix in the rework:** the current poller's idempotency marks a comment "processed" (`processed.add(comment["id"])`) *before* the authorization check and Goose dispatch, and persists it at end of poll regardless of outcome. So any transient failure - a 403, a Goose crash, an exception mid-service - **permanently burns that summon**: the id is recorded as handled and the comment is never retried. Combined with "this fails a lot," that means failed summons silently never get answered. Idempotency must key on a *successful reply posted*, not on "seen." (Hit live: a 403 during gating recorded `inbox#110`'s comment as processed, so the next clean run skipped it with `0 serviced`.) ## Observability - first-class, not bolted on This is the explicit point of the issue. Treat the whole lifecycle as an o11y target. - **Structured logs** - one JSON event per stage (`received`, `signature_verified`, `gated`, `authorized`, `acked`, `dispatched`, `goose_started`, `goose_finished`, `replied`, `timed_out`, `errored`), each carrying a **correlation id** (Forgejo delivery id + comment id + repo#index) so a single summon is greppable end to end. - **Metrics** - counters (`summons_received`, `acked`, `dispatched`, `replied`, `unauthorized`, `timed_out`, `failed{stage}`) and histograms (`ack_latency`, `goose_run_seconds`, `time_to_reply_seconds`). Export to the fleet's existing metrics stack. - **Traces** - one OTEL trace per summon, spans across receive -> ack -> dispatch -> goose -> reply, exported to the existing tracing backend (Phoenix / o2r). The Goose subprocess gets its own child span. - **Error handling** - every external call (Forgejo API, git clone, `goose run`) wrapped; failures are categorized, surfaced as a comment (never swallowed into silence), retried with backoff where safe, and a repeated-failure / dead-letter condition fires the fleet's red-channel alert (Telegram). **Second carry-over bug:** `run_goose` invokes `goose run --no-session --system ... -t task` *without* `--no-profile`, so it inherits whatever default Goose profile/extensions the host has. On the first live reply (`inbox/#110` comment 15366) that meant Goose loaded the `apps`/`extensionmanager` toolset and had no file tools - it tried `read` and got "Tool 'read' not found." The advisory run needs the developer/file extensions explicitly (or the anti-thrash `--no-profile` + a pinned extension set, matching the `goose-triage` harness), or the answers are useless even when the plumbing works. ## Open decisions for the implementer (flag, don't guess) - Receiver host + ingress: new systemd service + `tailscale serve`, or fold into an existing fleet service? Tailnet-only either way. - Metrics + trace backends: confirm Prometheus + Phoenix (or whatever the current fleet o11y stack is) and the export path. - `X` poll-window default and the ack copy. - Webhook-secret storage (SSM `/forgejo/coilyco-ops/webhook-secret`, SecureString) and the per-repo webhook registration (script, mirroring `provision-coilyco-ops-bot.sh`). - Does the webhook fully replace the 30s timer, or does the timer stay as a slow backstop? (Recommended: keep it slow as a backstop.) ## Done when - A signed Forgejo webhook on an `@coilyco-ops` summon produces an **ack comment within seconds**, and either a Goose answer or an explicit failure comment within the poll window - **never silence**. - Logs, metrics, and traces for the full lifecycle are queryable, keyed by correlation id. - Security posture from #374 is preserved (default-deny auth, advisory-only Goose, no main mutation, tailnet-only, signature-verified ingress). refs #374

coilysiren referenced this issue

2026-06-18 06:45:31 +00:00

Let summoned @coilyco-ops Goose file/move issues (top use case: route issues out of the inbox) #376

coilysiren referenced this issue

2026-06-18 06:48:57 +00:00

Let summoned @coilyco-ops Goose file/move issues (top use case: route issues out of the inbox) #376

coilyco-ops added the

consult

labels

2026-06-24 05:38:37 +00:00

coilyco-ops commented

2026-06-24 05:38:37 +00:00

Member

Triage: P3 / consult - Webhook-driven summons; design + webhook infra first.

(goose-style pass; claude as the judgment engine, applied 2026-06-23)

**Triage:** `P3` / `consult` - Webhook-driven summons; design + webhook infra first. (goose-style pass; claude as the judgment engine, applied 2026-06-23)

coilyco-ops added

and removed

labels

2026-07-10 09:00:07 +00:00