coilyco-gaming/sirens-echo

Watch

forgejo.issue.failed fires silently and carries no diagnosable detail #89

New issue

Closed

opened 2026-08-10 23:24:05 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-10 23:24:05 +00:00

Member

Found while running a live battery against the deployed sirens-deep workload on 2026-08-10, image f869390e1225a94ab7479a70dcc57c918e42d220.

What happened

Over a 30 minute window of generated traffic, the sirens-deep namespace logged 14 forgejo.issue.failed errors. None of the prompts asked Deep to create, close, comment on, or label anything. Deep attempted issue creation on its own.

Nothing was created. The tracker shows no issues from this window, so the attempts failed closed, which is the safe direction.

The diagnosability problem

The entire log record is:

{"time":"2026-08-10T23:01:32.12373559Z","level":"ERROR","msg":"forgejo.issue.failed",
 "error_type":"forgejo_issue_failed","trace_id":"bdc8470e71c5d185e5fba737fa5567ff",
 "span_id":"4a44bf2cd13d4997"}

There is no status code, no response body, no message, no repository, no title, and no indication of which guardfile route was refused. error_type restates msg. An operator seeing 14 of these has no path to a cause without attaching to the process.

By contrast the sibling failure in the same window, turn.stage.failed, at least carries stage=model, which is what made #86 diagnosable.

Two things worth separating

The failure itself. Fourteen consecutive failures suggests something systematic rather than incidental. Candidates worth checking: the MCP guardfile refusing the call shape, the issue token, or a required field Deep is not supplying. The guardfile does permit issue creation on coilyco-gaming/sirens-echo, so a blanket denial would be surprising.

The silence. Deep attempts issue creation as a side effect of ordinary conversation. The captured response schema is {"reply": ..., "issue": ...}, so issue is populated by the model's own judgement. That is a write path on Kai's tracker driven by untrusted conversational input, and right now it neither succeeds nor explains itself.

Note on #76

#76 records the Sirens Deep profile as having mcp_servers: [] and "no automatic issue tracker". The deployed image contradicts both. It reaches the Eco MCP (observed tool calls to get_climate, get_government, get_economy, get_civics) and attempts Forgejo issue creation. Worth reconciling which is current.

Worth considering

Log the status code and response body on failure.
Decide deliberately whether an HTTP-only lane should be able to file issues at all.

Found while running a live battery against the deployed `sirens-deep` workload on 2026-08-10, image `f869390e1225a94ab7479a70dcc57c918e42d220`. ## What happened Over a 30 minute window of generated traffic, the `sirens-deep` namespace logged **14 `forgejo.issue.failed` errors**. None of the prompts asked Deep to create, close, comment on, or label anything. Deep attempted issue creation on its own. Nothing was created. The tracker shows no issues from this window, so the attempts failed closed, which is the safe direction. ## The diagnosability problem The entire log record is: ```json {"time":"2026-08-10T23:01:32.12373559Z","level":"ERROR","msg":"forgejo.issue.failed", "error_type":"forgejo_issue_failed","trace_id":"bdc8470e71c5d185e5fba737fa5567ff", "span_id":"4a44bf2cd13d4997"} ``` There is no status code, no response body, no message, no repository, no title, and no indication of which guardfile route was refused. `error_type` restates `msg`. An operator seeing 14 of these has no path to a cause without attaching to the process. By contrast the sibling failure in the same window, `turn.stage.failed`, at least carries `stage=model`, which is what made #86 diagnosable. ## Two things worth separating **The failure itself.** Fourteen consecutive failures suggests something systematic rather than incidental. Candidates worth checking: the MCP guardfile refusing the call shape, the issue token, or a required field Deep is not supplying. The guardfile does permit issue creation on `coilyco-gaming/sirens-echo`, so a blanket denial would be surprising. **The silence.** Deep attempts issue creation as a side effect of ordinary conversation. The captured response schema is `{"reply": ..., "issue": ...}`, so `issue` is populated by the model's own judgement. That is a write path on Kai's tracker driven by untrusted conversational input, and right now it neither succeeds nor explains itself. ## Note on #76 #76 records the Sirens Deep profile as having `mcp_servers: []` and "no automatic issue tracker". The deployed image contradicts both. It reaches the Eco MCP (observed tool calls to `get_climate`, `get_government`, `get_economy`, `get_civics`) and attempts Forgejo issue creation. Worth reconciling which is current. ## Worth considering * Log the status code and response body on failure. * Decide deliberately whether an HTTP-only lane should be able to file issues at all.

coilyco-ops commented

2026-08-10 23:25:36 +00:00

Author

Member

Root cause found. Traces answer what the logs could not.

Error spans from sirens-deep over the same window, grouped by URL and status:

http://sirens-deep-forgejo-mcp:8080/api/list_issue   404   17

Deep is calling /api/list_issue on the Forgejo MCP. That server serves MCP at /mcp, so the path does not exist and every call returns 404. The 17 failing forgejo.issue.ensure spans match the 17 404s, and those produce the 14 logged forgejo.issue.failed errors.

So this is not a guardfile denial, a token problem, or a missing field. Deep's Forgejo client is constructing a REST-style URL against an MCP endpoint.

That also explains why the failure looked systematic rather than incidental. It cannot succeed at all in this configuration.

The diagnosability half of this issue still stands on its own. A 404 against a wrong path is exactly the kind of cause a logged status code would have surfaced immediately, and instead it took a trace query to find.

Root cause found. Traces answer what the logs could not. Error spans from `sirens-deep` over the same window, grouped by URL and status: ``` http://sirens-deep-forgejo-mcp:8080/api/list_issue 404 17 ``` Deep is calling **`/api/list_issue`** on the Forgejo MCP. That server serves MCP at **`/mcp`**, so the path does not exist and every call returns 404. The 17 failing `forgejo.issue.ensure` spans match the 17 404s, and those produce the 14 logged `forgejo.issue.failed` errors. So this is not a guardfile denial, a token problem, or a missing field. Deep's Forgejo client is constructing a REST-style URL against an MCP endpoint. That also explains why the failure looked systematic rather than incidental. It cannot succeed at all in this configuration. The diagnosability half of this issue still stands on its own. A 404 against a wrong path is exactly the kind of cause a logged status code would have surfaced immediately, and instead it took a trace query to find.

coilyco-ops referenced this issue

2026-08-10 23:25:47 +00:00

Every turn makes one failing HTTP 400 call to each MCP endpoint #90

coilyco-ops referenced this issue from a commit

2026-08-10 23:29:47 +00:00

Add an error spans panel to the Sirens Deep console

coilyco-ops referenced this issue

2026-08-11 06:48:38 +00:00

fix: attribute the harness to the real ingress and make Forgejo failures diagnosable #101

coilyco-ops referenced this issue

2026-08-11 08:07:48 +00:00

Drop the issue field from the response envelope and let the model use the Forgejo tool it already has #102

coilyco-ops commented

2026-08-11 09:41:25 +00:00

Author

Member

Checked against current main (e0b3dc1). The diagnosability half is largely delivered, and the remaining ask conflicts with a contract the same PR established. Flagging rather than deciding it.

Already landed

#101 added ForgejoCallError and widened the log record. internal/community/agent.go:656 now emits mcp_tool, status, and tool_reported alongside error_type. So the record an operator sees is no longer just error_type restating msg, and the "which tool, what status" gap is closed.

The conflict

This issue asks to "log the status code and response body on failure." The status code is now there. The body is deliberately not, and that is written into the type:

// ForgejoCallError names the tool and status of a refused MCP call. Both are
// bounded metadata: the body is discarded and never enters this value.

internal/community/forgejo.go:39, with the discard itself at line 128. The house convention agrees: the model path logs status and response_bytes and never a body (internal/community/proxy.go:540), and #55 makes body-safety a hard requirement with tests that fail if a stack trace appears.

So adding the body reverses a decision made one commit ago, on the same safety axis. I am not doing that unilaterally.

Options

Leave it. Tool plus status plus tool_reported is probably enough to diagnose a systematic failure, and the body stays out of telemetry.
Server message only. Capture Forgejo's error-envelope message field when the body parses as one, never the raw body. Server-authored, bounded, still leaks nothing model-generated. My preference if the current fields prove insufficient.
Full body. Maximum diagnosability, reverses the contract.

Still open regardless

The second half of this issue is a decision, not an implementation: whether an HTTP-only lane should be able to file issues at all. issue is populated by the model's own judgement, so it is a write path on the tracker driven by untrusted conversational input. Worth splitting into its own issue, since it will outlive whichever logging option gets picked.

Checked against current `main` (`e0b3dc1`). The diagnosability half is largely delivered, and the remaining ask conflicts with a contract the same PR established. Flagging rather than deciding it. ## Already landed https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/pulls/101 added `ForgejoCallError` and widened the log record. `internal/community/agent.go:656` now emits `mcp_tool`, `status`, and `tool_reported` alongside `error_type`. So the record an operator sees is no longer just `error_type` restating `msg`, and the "which tool, what status" gap is closed. ## The conflict This issue asks to "log the status code and response body on failure." The status code is now there. The body is deliberately not, and that is written into the type: ```go // ForgejoCallError names the tool and status of a refused MCP call. Both are // bounded metadata: the body is discarded and never enters this value. ``` `internal/community/forgejo.go:39`, with the discard itself at line 128. The house convention agrees: the model path logs `status` and `response_bytes` and never a body (`internal/community/proxy.go:540`), and https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/55 makes body-safety a hard requirement with tests that fail if a stack trace appears. So adding the body reverses a decision made one commit ago, on the same safety axis. I am not doing that unilaterally. ## Options 1. **Leave it.** Tool plus status plus tool_reported is probably enough to diagnose a systematic failure, and the body stays out of telemetry. 2. **Server message only.** Capture Forgejo's error-envelope `message` field when the body parses as one, never the raw body. Server-authored, bounded, still leaks nothing model-generated. My preference if the current fields prove insufficient. 3. **Full body.** Maximum diagnosability, reverses the contract. ## Still open regardless The second half of this issue is a decision, not an implementation: whether an HTTP-only lane should be able to file issues at all. `issue` is populated by the model's own judgement, so it is a write path on the tracker driven by untrusted conversational input. Worth splitting into its own issue, since it will outlive whichever logging option gets picked.

coilysiren closed this issue

2026-08-11 09:53:33 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-gaming/sirens-echo#89

No description provided.

Rows
Columns