Add GIF reading support

coilyco-ops-gaming commented

2026-08-12 23:09:13 +00:00

Member

Summary

The bot currently lacks image/GIF support. Need to add capability to read and process GIFs when users send them with tags or in messages.

Context

User inquiry: "How do I send a gif with a tag"
Current state: No image support available
Status: Kai indicated this would be added within a few hours during volume test

Requirements

Enable bot to detect and read GIFs from user messages
Support tagging functionality for GIFs
Maintain existing response patterns when images are present

## Summary The bot currently lacks image/GIF support. Need to add capability to read and process GIFs when users send them with tags or in messages. ## Context - User inquiry: "How do I send a gif with a tag" - Current state: No image support available - Status: Kai indicated this would be added within a few hours during volume test ## Requirements - Enable bot to detect and read GIFs from user messages - Support tagging functionality for GIFs - Maintain existing response patterns when images are present

coilyco-ops referenced this issue

2026-08-13 04:07:40 +00:00

create a checklist of multi media support targets #202

coilyco-ops commented

2026-08-13 04:07:48 +00:00

Member

Design decision — confirmed target, but not first

Recorded by Delphi (design seat). Kai's decision, 2026-08-12.

GIF reading is a confirmed multimedia target. Full checklist: #202 — static images, GIFs, links and embeds are in; audio and video are explicitly out.

Sequencing note: static image support should land first. GIF reading implies frame sampling, which is meaningfully more work than reading a still, and the prompting user question — "How do I send a gif with a tag" — was arguably about Discord mechanics rather than about Echo interpreting animation. Static images cover the common community case (screenshots) at a fraction of the cost.

Scope check on the body's requirements. "Support tagging functionality for GIFs" is ambiguous and should be pinned down before anyone builds: it likely means Echo handles a GIF attached to a message that mentions it, which falls out of ordinary attachment reading. If it means something else, say so here.

Status correction for the record: the body notes "Kai indicated this would be added within a few hours during volume test." That did not happen, and this issue is the reason it is tracked. Worth stating so nobody assumes it shipped.

Constraints from the checklist that apply here: the content classifier must see GIF content (#227) — media is not a boundary bypass — and an unreadable attachment must produce a plain statement of that, never a confident answer about an image Echo did not process.

## Design decision — confirmed target, but not first Recorded by Delphi (design seat). Kai's decision, 2026-08-12. **GIF reading is a confirmed multimedia target.** Full checklist: https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/202 — static images, GIFs, links and embeds are in; audio and video are explicitly out. **Sequencing note:** static image support should land first. GIF reading implies frame sampling, which is meaningfully more work than reading a still, and the prompting user question — *"How do I send a gif with a tag"* — was arguably about Discord mechanics rather than about Echo interpreting animation. Static images cover the common community case (screenshots) at a fraction of the cost. **Scope check on the body's requirements.** "Support tagging functionality for GIFs" is ambiguous and should be pinned down before anyone builds: it likely means Echo handles a GIF attached to a message that mentions it, which falls out of ordinary attachment reading. If it means something else, say so here. **Status correction for the record:** the body notes *"Kai indicated this would be added within a few hours during volume test."* That did not happen, and this issue is the reason it is tracked. Worth stating so nobody assumes it shipped. Constraints from the checklist that apply here: the content classifier must see GIF content (https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/227) — media is not a boundary bypass — and an unreadable attachment must produce a plain statement of that, never a confident answer about an image Echo did not process.

coilysiren added the

🔒⚠️📦⚠️🔒 SANDBOXED 🔒⚠️📦⚠️🔒

label

2026-08-13 06:23:34 +00:00

coilyco-ops commented

2026-08-13 09:37:00 +00:00

Member

Same two blockers as static images, and one of them is not ours — Angie (ENG, claude seat). Research, not a claim. Posting here so this is not rediscovered separately from #168.

I traced the image path for 168 and the findings apply unchanged to GIFs, because nothing in the chain distinguishes the two.

1. The virtual-filesystem dependency is done. An upload lands in the requester's scratchpad under a reserved uploads/ path with confinement, per-file limit, quota and attribution, and scratch_read reaches it. That is the dependency both issues were waiting on.

2. The model transport is text-only, outbound. proxy.go carries type chatContent struct { Text string }. Its UnmarshalJSON tolerates text-part arrays coming back from gateways, which is not the same thing. image_url and ImageURL return zero matches outside an unrelated field, so nothing sends image content. This is real work in the request builder.

3. The route has to accept image content, and that is a deployment fact. Echo resolves to ornith:35b on ollama, Deep to deepseek-v4-flash. Whether either is multimodal is not establishable from this repository, and it gates item 2 completely.

The failure mode to design against is not a rejected request. It is the backend silently dropping the image and answering from the text, which looks exactly like the feature working.

What exists today is deliberate, and it is worth knowing before anyone removes it: prompt.go's attachmentSuffix tells the model an image arrived without claiming to have read it. That is the honest position, and stripping it before items 2 and 3 land would manufacture precisely the invented-capability defect this backlog has spent the night removing.

One thing specific to GIFs that static images do not have: a GIF is many frames. Even a multimodal route usually takes a single still, so "read this GIF" quietly becomes "read frame one", and whether that answers the member's question depends on the GIF. Worth deciding rather than discovering — a reply describing frame one as though it described the animation is a fresh way to be confidently wrong.

Not claiming. Item 3 is one question to Ops or Kai and everything else is unbuildable before it. Kai sequenced static images first on #202, so this should stay behind 168 regardless.

**Same two blockers as static images, and one of them is not ours — Angie (ENG, claude seat).** Research, not a claim. Posting here so this is not rediscovered separately from https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/168. I traced the image path for 168 and the findings apply unchanged to GIFs, because nothing in the chain distinguishes the two. **1. The virtual-filesystem dependency is done.** An upload lands in the requester's scratchpad under a reserved `uploads/` path with confinement, per-file limit, quota and attribution, and `scratch_read` reaches it. That is the dependency both issues were waiting on. **2. The model transport is text-only, outbound.** `proxy.go` carries `type chatContent struct { Text string }`. Its `UnmarshalJSON` tolerates text-part arrays coming back from gateways, which is not the same thing. `image_url` and `ImageURL` return zero matches outside an unrelated field, so nothing sends image content. This is real work in the request builder. **3. The route has to accept image content, and that is a deployment fact.** Echo resolves to `ornith:35b` on ollama, Deep to `deepseek-v4-flash`. Whether either is multimodal is not establishable from this repository, and it gates item 2 completely. **The failure mode to design against** is not a rejected request. It is the backend silently dropping the image and answering from the text, which looks exactly like the feature working. **What exists today is deliberate**, and it is worth knowing before anyone removes it: `prompt.go`'s `attachmentSuffix` tells the model an image arrived *without claiming to have read it*. That is the honest position, and stripping it before items 2 and 3 land would manufacture precisely the invented-capability defect this backlog has spent the night removing. **One thing specific to GIFs** that static images do not have: a GIF is many frames. Even a multimodal route usually takes a single still, so "read this GIF" quietly becomes "read frame one", and whether that answers the member's question depends on the GIF. Worth deciding rather than discovering — a reply describing frame one as though it described the animation is a fresh way to be confidently wrong. **Not claiming.** Item 3 is one question to Ops or Kai and everything else is unbuildable before it. Kai sequenced static images first on https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/202, so this should stay behind 168 regardless.

coilyco-ops referenced this issue

2026-08-13 09:41:25 +00:00

Index: the decisions this backlog is now waiting on, in one place #315

coilysiren removed the

🔒⚠️📦⚠️🔒 SANDBOXED 🔒⚠️📦⚠️🔒

label

2026-08-13 10:55:22 +00:00

coilyco-ops commented

2026-08-13 16:23:00 +00:00

Member

The detect half is built. The read half is blocked on a question nobody has answered — Angie (ENG, claude seat). Read at 64eedd1. Research, not a claim.

Your requirements list "detect and read GIFs". Those are two capabilities and only one of them is missing.

Detection ships today

attachmentTypes reads the Discord attachment's ContentType, defaults an unknown one to application/octet-stream, and puts the list on TranscriptEntry.Attachments. The prompt renders it, and the field's own comment records the reason:

Attachments carries media types only, never filenames or bytes. Without it a screenshot reads as text alone.

So a member posting a GIF today produces a turn where the model knows an image/gif arrived. It can acknowledge the GIF, ask what is in it, and answer around it. What it cannot do is see it.

That is worth knowing because it changes what "no image support" means. The bot is not blind to the fact of an image; it is blind to the content. A reply that says nothing about an obvious attachment is a doctrine gap rather than a missing capability.

The read half has a blocker with a name

#168 carries the question, and it has been sitting unanswered:

One question to Ops or Kai: does the configured route accept image content[...] Everything else is unbuildable until that is answered, and it is cheap to answer.

That is upstream of everything here. Sending image parts to a route that rejects them fails every turn carrying an attachment, which is worse than not seeing images at all.

Your Context section says "Kai indicated this would be added within a few hours during volume test." That was 2026-08-12. The thing standing between then and now is one question about the model route, not implementation effort.

Labelling `consult`

Blocked on 168's question, which is Ops' or Kai's. Nothing an agent can take end to end until the route answer exists.

One suggestion for whoever answers it. The media-type list is already flowing, so a first slice that needs no route change at all is doctrine: tell the model what to do when an attachment it cannot read arrives. "A member sent an image/gif and I cannot see it, ask them to describe it" is a better reply than silence, and it works today.

**The detect half is built. The read half is blocked on a question nobody has answered — Angie (ENG, claude seat).** Read at `64eedd1`. Research, not a claim. Your requirements list *"detect and read GIFs"*. Those are two capabilities and only one of them is missing. ## Detection ships today `attachmentTypes` reads the Discord attachment's `ContentType`, defaults an unknown one to `application/octet-stream`, and puts the list on `TranscriptEntry.Attachments`. The prompt renders it, and the field's own comment records the reason: > Attachments carries **media types only, never filenames or bytes**. Without it a screenshot reads as text alone. So a member posting a GIF today produces a turn where the model knows an `image/gif` arrived. It can acknowledge the GIF, ask what is in it, and answer around it. What it cannot do is see it. **That is worth knowing because it changes what "no image support" means.** The bot is not blind to the fact of an image; it is blind to the content. A reply that says nothing about an obvious attachment is a doctrine gap rather than a missing capability. ## The read half has a blocker with a name https://forgejo.coilysiren.me/coilyco-gaming/sirens-echo/issues/168 carries the question, and it has been sitting unanswered: > One question to Ops or Kai: **does the configured route accept image content**[...] Everything else is unbuildable until that is answered, and it is cheap to answer. That is upstream of everything here. Sending image parts to a route that rejects them fails every turn carrying an attachment, which is worse than not seeing images at all. Your Context section says *"Kai indicated this would be added within a few hours during volume test."* That was 2026-08-12. The thing standing between then and now is one question about the model route, not implementation effort. ## Labelling `consult` Blocked on 168's question, which is Ops' or Kai's. Nothing an agent can take end to end until the route answer exists. **One suggestion for whoever answers it.** The media-type list is already flowing, so a first slice that needs no route change at all is doctrine: tell the model what to do when an attachment it cannot read arrives. *"A member sent an image/gif and I cannot see it, ask them to describe it"* is a better reply than silence, and it works today.

coilyco-ops added the

autonomy

async-consult

label

2026-08-13 16:23:06 +00:00

coilyco-ops added the

priority

P2

label

2026-08-15 15:20:50 +00:00

coilyco-ops added the

role/engineer