Repository: transformers
Issue: Auto model & pipeline for image-text-to-image-text models
Labels: Feature request

### Feature request

This is a tracker issue for work on _interleaved_ in-and-out image-text generation.

There are now >= 5 open-source models that can do _interleaved_ image-text generation--and many more are expected to be released. Thus, it would now be practical & useful for us to (1) add native support for such models and (2) standardize the logic flow of data through processors and pipelines as done in https://github.com/huggingface/transformers/issues/31911 and https://github.com/huggingface/transformers/pull/32472

Model | Github | Notes | PR
--- | --- | --- | ---
[Anole](https://arxiv.org/abs/2407.06135) | https://github.com/GAIR-NLP/anole | - | https://github.com/huggingface/transformers/pull/32013
[Chameleon](https://arxiv.org/abs/2405.09818) | https://github.com/facebookresearch/chameleon | - | https://github.com/huggingface/transformers/pull/32013
[Llava-NeXT-Interleaved](https://arxiv.org/abs/2407.07895) | https://github.com/LLaVA-VL/LLaVA-NeXT | - | -
[Lumina-mGPT](https://arxiv.org/abs/2408.02657) | https://github.com/Alpha-VLLM/Lumina-mGPT | - | -
[Show-o](https://arxiv.org/abs/2408.12528) | https://github.com/showlab/Show-o | - | -
[Transfusion](https://arxiv.org/abs/2408.11039) | - | Not open-source (yet, perhaps) | -
[XGen-MM](https://arxiv.org/abs/2408.08872) | https://github.com/salesforce/LAVIS/tree/xgen-mm | The paper & the github repo don't actually demonstrate interleaved image-text generation yet, but they did train the model on such datasets & the model architecture(s) is perfectly suited for it | -
[Emu3](https://baai-solution.ks3-cn-beijing.ksyuncs.com/emu3/Emu3-tech-report.pdf?KSSAccessKeyId=AKLTgew6Kdg6RsK92QSfB2KLA&Expires=2591406552&Signature=6BvwfLVqvfww26Bhwvk3mG0FrL8%3D) | https://github.com/baaivision/Emu3 | The official repo only has demos for text-only & image-only generation, but the model seems to have been trained on text-image datasets | -

Initial work for Chameleon & Anole can be found here: https://github.com/huggingface/transformers/pull/32013 for reference.

Notes:
- We explicitly exclude models that can _only_ do text-only generation or image-only generation. We also exclude models that can do image-text generation but not in an interleaved manner.
- As I've demonstrated in my repo, _explicitly_ implementing the Finite State Machine (FSM) for switching between text-generation and image-generation modes as done in Chameleon's repo is not necessary. Implicitly implementing the FSM with Logits Processors suffices. Although more work is needed on finding the most efficient implementation.

TODOs:
- [ ] Add support for interleaved image-text generation with:
  - [x] Chameleon -> https://github.com/huggingface/transformers/pull/32013
  - [x] Anole -> https://github.com/huggingface/transformers/pull/32013
  - [ ] Lumina-mGPT
  - [ ] Show-o
  - [ ] Transfusion
  - [ ] XGen-MM
- [ ] Add auto model for image-text-to-image-text
  - [ ] [Optional] Add auto model for image-to-image-text
  - [ ] [Optional] Add auto model for text-to-image-text
- [ ] Add pipeline for image-text-to-image-text
  - [ ] [Optional] Add pipeline for image-to-image-text
  - [ ] [Optional] Add pipeline for text-to-image-text
- [ ] Benchmark different implementations of Logits Processors & FSMs for switching between text-generation and image-generation modes


### Motivation

1. To make benchmarking and evaluating models for interleaved image-to-text tasks saner
2. To continue work on Multimodal In-and-Out, Interleaved Structured Generation: https://github.com/leloykun/mmsg

### Your contribution

I've already started work on Chameleon & Anole here: https://github.com/huggingface/transformers/pull/32013

But I'm currently blocked by (1) not having enough time due to other responsibilities and (2) not having enough compute resources.

Any help would be appreciated!