Find a cheap cloud model provider that does general-purpose work and images #148

New issue

Open

opened 2026-08-27 03:25:08 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-27 03:25:08 +00:00

Owner

Filed by Angie (ENG, claude seat) at Kai's request, out of coilyco-gaming/sirens-echo#168. Kai owns the provider selection; this records what the choice has to satisfy and how to verify a candidate without being fooled.

Why

Nothing currently routable can read an image. Measured through Agent Proxy on 2026-08-16:

Route	Accepted	Image actually seen
`sirens-echo/default` (ornith:35b)	No, HTTP 502	n/a
`sirens-echo/deepseek` (V4-Flash)	Yes, HTTP 200	No, silently dropped
`evaluation/deepseek-v4-flash`	Yes, HTTP 200	No, silently dropped
`evaluation/ministral-3-14b`	Yes, HTTP 200	Yes
`evaluation/ornith-35b`	No, HTTP 502	n/a

DeepSeek is ruled out by the vendor's own documentation rather than by the probe: api-docs.deepseek.com documents deepseek-v4-flash and deepseek-v4-pro and mentions no image, vision, multimodal, or image_url support anywhere, including the V4-Pro GA release of 2026-08-13. Third-party pages claiming otherwise contradict the vendor. The consumer web and mobile apps reportedly do accept uploads, which is consistent: the app is not the API, and Echo talks to the API.

ministral-3-14b does work, so this is not urgent in the sense of having no option at all. It is about having a cheap cloud option that covers general-purpose work and images together, rather than adding a second provider later for the image half.

What the candidate has to satisfy

OpenAI-compatible chat completions, because Agent Proxy and the Deploy-owned LiteLLM gateway speak that and nothing should be rebuilt to accommodate a provider.
Accepts image_url content parts in messages[].content arrays, and actually reads them. Those are different claims. See the verification note below.
General-purpose text quality sufficient for bounded Community work, since the same route serves the answering turn and the content classifier. They are one ProxyClient, so a route that reads images gives the classifier vision for free, and a route that does not leaves the classifier blind on image turns, which is the boundary bypass coilyco-gaming/sirens-echo#227 exists to prevent.
Cheap, in the sense #329 means it, with published per-token and per-image pricing.
Ordinary API-key authentication that can live in SSM.

How to verify a candidate, and why the obvious check fails

Do not trust the reply. A provider can accept the request, return HTTP 200, and silently discard the image, which is exactly what V4-Flash does. That looks like the feature working.

Two checks that cannot be faked:

Token delta. Send the same prompt with and without the image and compare usage.prompt_tokens. V4-Flash reported 132 both ways with an 8.5 KB PNG attached. A route that genuinely reads it moved 121 to roughly 705.
Unguessable content. The image must contain something the prompt does not imply, and the pass condition is the reply naming it. The probe used a 320x320 PNG, purple background, 7413 in white, TRIANGLE in yellow, and asked for the number, the word, and the background colour.

Payload shape, which is the ordinary one:

{"model": "<route>", "temperature": 0, "messages": [{"role": "user", "content": [
  {"type": "text", "text": "<prompt>"},
  {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}}]}]}

Worth knowing: on ornith the multipart array form itself is fine, text-only parts return 200 with identical token counts. Only the image part fails. So a candidate rejecting content arrays outright would be a different and worse problem.

Boundary

Same house rules as #329, restated so this issue stands alone:

Route through Agent Proxy and the Deploy-owned LiteLLM gateway. Do not call the provider directly from a harness or an evaluation.
Keep AOS role and harness routing model-opaque. A provider is a route, not a new harness.
Credentials in SSM, injected only into the owning deployment runtime, never tracked.
Not a fleet default. A new route is added; existing lanes move only by a separate decision.
Engineer owns implementation and local validation. Ops owns credential injection, deployment, and live verification.

Ordered work

Kai names the candidate provider and model. Nothing below can start first, which is why this is consult.
Confirm from the vendor's own documentation that the API, not just a consumer app, accepts image content parts.
Ops injects the credential and adds the route to LiteLLM behind Agent Proxy.
Run the two-part verification above and record raw responses and token counts.
If it passes, coilyco-gaming/sirens-echo#168 is unblocked on the model side and its request-builder work becomes buildable. Whether a Sirens lane actually points at it is a separate decision.

Done condition

A named provider and model, reachable through Agent Proxy, with recorded evidence that it both accepts and reads image content.
Or an explicit rejection with the reason and the next candidate named, so this does not quietly stall.

coilyco-gaming/sirens-echo#168 - allow reading images, the capability this unblocks. Carries the full probe writeup.
coilyco-gaming/sirens-echo#202 - the multimedia checklist. Static images, GIFs, links and embeds are in; audio and video are out.
coilyco-gaming/sirens-echo#227 - the content classifier must see image content. Media is not a boundary bypass.
#329 - the sibling program for a cost-controlled DeepSeek route, whose step 5 anticipates naming a next candidate when one does not work out. This is that, plus an image requirement.

Filed by Angie (ENG, `claude` seat) at Kai's request, out of `coilyco-gaming/sirens-echo#168`. **Kai owns the provider selection**; this records what the choice has to satisfy and how to verify a candidate without being fooled. ## Why Nothing currently routable can read an image. Measured through Agent Proxy on 2026-08-16: | Route | Accepted | Image actually seen | | --- | --- | --- | | `sirens-echo/default` (ornith:35b) | No, HTTP 502 | n/a | | `sirens-echo/deepseek` (V4-Flash) | Yes, HTTP 200 | **No**, silently dropped | | `evaluation/deepseek-v4-flash` | Yes, HTTP 200 | **No**, silently dropped | | `evaluation/ministral-3-14b` | Yes, HTTP 200 | Yes | | `evaluation/ornith-35b` | No, HTTP 502 | n/a | DeepSeek is ruled out by the vendor's own documentation rather than by the probe: `api-docs.deepseek.com` documents `deepseek-v4-flash` and `deepseek-v4-pro` and mentions no image, vision, multimodal, or `image_url` support anywhere, including the [V4-Pro GA release](https://api-docs.deepseek.com/news/news260813/) of 2026-08-13. Third-party pages claiming otherwise contradict the vendor. The consumer web and mobile apps reportedly do accept uploads, which is consistent: the app is not the API, and Echo talks to the API. `ministral-3-14b` does work, so this is not urgent in the sense of having no option at all. It is about having a **cheap cloud** option that covers general-purpose work and images together, rather than adding a second provider later for the image half. ## What the candidate has to satisfy * **OpenAI-compatible chat completions**, because Agent Proxy and the Deploy-owned LiteLLM gateway speak that and nothing should be rebuilt to accommodate a provider. * **Accepts `image_url` content parts** in `messages[].content` arrays, and **actually reads them**. Those are different claims. See the verification note below. * **General-purpose text quality** sufficient for bounded Community work, since the same route serves the answering turn and the content classifier. They are one `ProxyClient`, so a route that reads images gives the classifier vision for free, and a route that does not leaves the classifier blind on image turns, which is the boundary bypass `coilyco-gaming/sirens-echo#227` exists to prevent. * **Cheap**, in the sense #329 means it, with published per-token and per-image pricing. * Ordinary API-key authentication that can live in SSM. ## How to verify a candidate, and why the obvious check fails **Do not trust the reply.** A provider can accept the request, return HTTP 200, and silently discard the image, which is exactly what V4-Flash does. That looks like the feature working. Two checks that cannot be faked: 1. **Token delta.** Send the same prompt with and without the image and compare `usage.prompt_tokens`. V4-Flash reported 132 both ways with an 8.5 KB PNG attached. A route that genuinely reads it moved 121 to roughly 705. 2. **Unguessable content.** The image must contain something the prompt does not imply, and the pass condition is the reply naming it. The probe used a 320x320 PNG, purple background, `7413` in white, `TRIANGLE` in yellow, and asked for the number, the word, and the background colour. Payload shape, which is the ordinary one: ```json {"model": "<route>", "temperature": 0, "messages": [{"role": "user", "content": [ {"type": "text", "text": "<prompt>"}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}}]}]} ``` Worth knowing: on ornith the multipart array form itself is fine, text-only parts return 200 with identical token counts. Only the image part fails. So a candidate rejecting content arrays outright would be a different and worse problem. ## Boundary Same house rules as #329, restated so this issue stands alone: * Route through Agent Proxy and the Deploy-owned LiteLLM gateway. Do not call the provider directly from a harness or an evaluation. * Keep AOS role and harness routing model-opaque. A provider is a route, not a new harness. * Credentials in SSM, injected only into the owning deployment runtime, never tracked. * Not a fleet default. A new route is added; existing lanes move only by a separate decision. * Engineer owns implementation and local validation. Ops owns credential injection, deployment, and live verification. ## Ordered work 1. **Kai names the candidate provider and model.** Nothing below can start first, which is why this is `consult`. 2. Confirm from the vendor's own documentation that the API, not just a consumer app, accepts image content parts. 3. Ops injects the credential and adds the route to LiteLLM behind Agent Proxy. 4. Run the two-part verification above and record raw responses and token counts. 5. If it passes, `coilyco-gaming/sirens-echo#168` is unblocked on the model side and its request-builder work becomes buildable. Whether a Sirens lane actually points at it is a separate decision. ## Done condition * A named provider and model, reachable through Agent Proxy, with recorded evidence that it both accepts and reads image content. * Or an explicit rejection with the reason and the next candidate named, so this does not quietly stall. ## Related * `coilyco-gaming/sirens-echo#168` - allow reading images, the capability this unblocks. Carries the full probe writeup. * `coilyco-gaming/sirens-echo#202` - the multimedia checklist. Static images, GIFs, links and embeds are in; audio and video are out. * `coilyco-gaming/sirens-echo#227` - the content classifier must see image content. Media is not a boundary bypass. * #329 - the sibling program for a cost-controlled DeepSeek route, whose step 5 anticipates naming a next candidate when one does not work out. This is that, plus an image requirement.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-proxy#148

No description provided.

Rows
Columns