signoz_aggregate_logs silently ignores searchText and returns the unfiltered total, while its sibling signoz_search_logs honors it #94

New issue

Open

opened 2026-08-22 22:10:59 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-22 22:10:59 +00:00

Member

Found 2026-08-22 while measuring a credential exposure for coilyco-flight-deck/infrastructure#708. This defect caused a wrong number to be published before a control query caught it.

The defect

signoz_aggregate_logs accepts a searchText parameter, documents it as "Text to search for in log body (uses CONTAINS matching)", and silently discards it. The tool returns the count of all logs in the window and reports success.

Measured over an identical 7-day window:

call	result
`aggregate_logs(count, searchText='api.telegram.org/bot')`	7,967,947
`aggregate_logs(count)` - no filter at all	7,967,837
`aggregate_logs(count, searchText='zzzzz-nonexistent-string-qqqq')`	7,967,817

A string that cannot appear in any log returns the same total as a real one. The small deltas are just logs arriving between calls.

The same parameter works correctly on the sibling tool

call	result
`search_logs(searchText='zzzzz-nonexistent-string-qqqq')`	0 rows, 0 rows scanned

So searchText is honored in signoz_search_logs and dropped in signoz_aggregate_logs. Two tools in the same server, same parameter name, opposite behavior. That inconsistency is what makes it dangerous - having used search_logs successfully, there is no reason to distrust aggregate_logs.

The working alternative

filter with body CONTAINS behaves correctly on aggregate_logs:

call	result
`aggregate_logs(count, filter="body CONTAINS 'api.telegram.org/bot'")`	24, 25,038 rows scanned
`aggregate_logs(count, filter="body CONTAINS 'zzzzz-nonexistent-string-qqqq'")`	0, 0 rows scanned

Note rowsScanned is itself a tell: the searchText calls scanned 6.4M rows, the filter calls scanned 25K. A dropped filter shows up as a scan of everything.

Why this is filed at P1

It produces confidently wrong answers rather than errors. A dropped filter that returned an error, or zero, would be caught immediately. Returning a large plausible number that is silently the unfiltered total is the worst available failure mode, and it is exactly what happened today - "7,967,947 occurrences of a leaked credential" was reported before a negative control caught it. In a security context that is a wildly wrong blast-radius estimate.

Any agent or human who has counted anything in these logs with searchText has a wrong number and no reason to suspect it.

Two candidate layers - diagnose before fixing

The signoz-mcp pod runs two containers, and the defect could be in either:

ward-mcp - forgejo.coilysiren.me/coilyco-flight-deck/mcp-beaver:258357e03385270d2e569393a9c2cf3af413d032
signoz-mcp-server - docker.io/signoz/signoz-mcp-server:v0.10.0 (upstream)

Hypothesis, not established: the guard layer is stripping the parameter. The server's own instructions state "This server exposes only policy-approved tools", so mcp-beaver is actively mediating the tool surface. If it re-derives or filters tool schemas and drops parameters it does not recognise, that would produce exactly this symptom - and it would not be limited to SigNoz.

If the guard is the cause, the blast radius is every guarded MCP in the fleet - forgejo, discord, trello, steam, glama, skillsmp, node-stats, lunch-money, eco, bluesky. Any tool with a parameter the guard does not carry through would silently ignore it. That possibility is why this is filed here rather than against the deployment.

The alternative is that the upstream SigNoz server has the bug and mcp-beaver is innocent. Determine which before changing anything, since the two fixes live in different repos and one of them is a third party.

Acceptance

The layer responsible is identified, with evidence, and stated on this issue.
If mcp-beaver is the cause: parameters are passed through faithfully, or an unsupported parameter causes a loud error rather than silent omission. Silent-drop is never acceptable for a filter argument - a query that cannot be honored must fail, not return unfiltered data.
A regression test asserts the negative-control property: a filter value that cannot match returns zero, not the unfiltered total. This is the check that catches the whole defect class.
If mcp-beaver is innocent, this issue records that finding and hands off - reporting upstream to SigNoz is an external-facing action and Kai's call, not something to do from here.
Whatever the cause, audit the other guarded MCP servers for the same silent-drop behavior using the same negative-control method.

This is the second silent-filter-drop found today. coilysiren/inbox#391 records that Forgejo's /issues?labels= endpoint returns the unfiltered set when a label name does not resolve in the target repo, which inflated an estate-wide P0 count before it was caught. Different system, identical failure class: a filter that cannot be applied returns everything instead of erroring.

Worth treating as a standing check rather than two coincidences - any filter parameter, anywhere in the fleet, should be probed with a value that cannot match before its results are trusted.

Found 2026-08-22 while measuring a credential exposure for `coilyco-flight-deck/infrastructure#708`. **This defect caused a wrong number to be published before a control query caught it.** ## The defect `signoz_aggregate_logs` accepts a `searchText` parameter, documents it as "Text to search for in log body (uses CONTAINS matching)", and **silently discards it**. The tool returns the count of *all* logs in the window and reports success. Measured over an identical 7-day window: | call | result | |---|---| | `aggregate_logs(count, searchText='api.telegram.org/bot')` | **7,967,947** | | `aggregate_logs(count)` - no filter at all | **7,967,837** | | `aggregate_logs(count, searchText='zzzzz-nonexistent-string-qqqq')` | **7,967,817** | A string that cannot appear in any log returns the same total as a real one. The small deltas are just logs arriving between calls. ## The same parameter works correctly on the sibling tool | call | result | |---|---| | `search_logs(searchText='zzzzz-nonexistent-string-qqqq')` | **0 rows, 0 rows scanned** | So `searchText` is honored in `signoz_search_logs` and dropped in `signoz_aggregate_logs`. **Two tools in the same server, same parameter name, opposite behavior.** That inconsistency is what makes it dangerous - having used `search_logs` successfully, there is no reason to distrust `aggregate_logs`. ## The working alternative `filter` with `body CONTAINS` behaves correctly on `aggregate_logs`: | call | result | |---|---| | `aggregate_logs(count, filter="body CONTAINS 'api.telegram.org/bot'")` | **24**, 25,038 rows scanned | | `aggregate_logs(count, filter="body CONTAINS 'zzzzz-nonexistent-string-qqqq'")` | **0**, 0 rows scanned | Note `rowsScanned` is itself a tell: the `searchText` calls scanned 6.4M rows, the `filter` calls scanned 25K. A dropped filter shows up as a scan of everything. ## Why this is filed at P1 **It produces confidently wrong answers rather than errors.** A dropped filter that returned an error, or zero, would be caught immediately. Returning a large plausible number that is silently the unfiltered total is the worst available failure mode, and it is exactly what happened today - "7,967,947 occurrences of a leaked credential" was reported before a negative control caught it. In a security context that is a wildly wrong blast-radius estimate. Any agent or human who has counted anything in these logs with `searchText` has a wrong number and no reason to suspect it. ## Two candidate layers - diagnose before fixing The `signoz-mcp` pod runs **two** containers, and the defect could be in either: * `ward-mcp` - `forgejo.coilysiren.me/coilyco-flight-deck/mcp-beaver:258357e03385270d2e569393a9c2cf3af413d032` * `signoz-mcp-server` - `docker.io/signoz/signoz-mcp-server:v0.10.0` (upstream) **Hypothesis, not established: the guard layer is stripping the parameter.** The server's own instructions state "This server exposes only policy-approved tools", so mcp-beaver is actively mediating the tool surface. If it re-derives or filters tool schemas and drops parameters it does not recognise, that would produce exactly this symptom - and it would **not** be limited to SigNoz. **If the guard is the cause, the blast radius is every guarded MCP in the fleet** - forgejo, discord, trello, steam, glama, skillsmp, node-stats, lunch-money, eco, bluesky. Any tool with a parameter the guard does not carry through would silently ignore it. That possibility is why this is filed here rather than against the deployment. The alternative is that the upstream SigNoz server has the bug and mcp-beaver is innocent. **Determine which before changing anything**, since the two fixes live in different repos and one of them is a third party. ## Acceptance 1. The layer responsible is identified, with evidence, and stated on this issue. 2. If mcp-beaver is the cause: parameters are passed through faithfully, **or** an unsupported parameter causes a loud error rather than silent omission. Silent-drop is never acceptable for a filter argument - a query that cannot be honored must fail, not return unfiltered data. 3. A regression test asserts the negative-control property: a filter value that cannot match returns zero, not the unfiltered total. This is the check that catches the whole defect class. 4. If mcp-beaver is innocent, this issue records that finding and hands off - reporting upstream to SigNoz is an external-facing action and Kai's call, not something to do from here. 5. Whatever the cause, **audit the other guarded MCP servers for the same silent-drop behavior** using the same negative-control method. ## Related This is the second silent-filter-drop found today. `coilysiren/inbox#391` records that Forgejo's `/issues?labels=` endpoint returns the **unfiltered** set when a label name does not resolve in the target repo, which inflated an estate-wide P0 count before it was caught. Different system, identical failure class: a filter that cannot be applied returns everything instead of erroring. Worth treating as a standing check rather than two coincidences - **any filter parameter, anywhere in the fleet, should be probed with a value that cannot match before its results are trusted.**

coilyco-ops added the

labels

2026-08-22 22:10:59 +00:00