T ext B ind: Multi-turn Interleaved Multimodal Instruction-following in the Wild (2024)

Abstract: Large language models with instruction-following abilities have revolutionized the field of artificial intelligence. These models show exceptional generalizability to tackle various real-world tasks through their natural language interfaces. However, their performance heavily relies on high-quality exemplar data, which is often difficult to obtain. This challenge is further exacerbated when it comes to multimodal instruction following. We introduce TextBind, an almost annotation-free framework for empowering LLMs with multi-turn interleaved multimodal instruction-following capabilities. Our approach requires only image-caption pairs and generates multi-turn multimodal instruction-response conversations from a language model. To accommodate interleaved image-text inputs and outputs, we devise MIM, a language model-centric architecture that seamlessly integrates image encoder and decoder models. Extensive quantitative and qualitative experiments demonstrate that MIM trained on TextBind achieves remarkable generation capability in multimodal conversations compared to recent baselines.

## introduction artificial intelligence (ai) has experienced a significant paradigm shift with the rise of large language models (llms). these models are capable of processing a wide range of natural language processing (nlp) applications through natural language interactions with users @xcite . recently, a number of efforts have been made to augment llms with visual perception and understanding abilities. prior work uses templatebased instruction-following datasets for training @xcite @xcite . these datasets comprise a variety of classic computer vision (cv) tasks, e.g., object detection, with each task being converted into an instructional format using a handful of human-written natural language instructions. however, classic cv tasks often represent manageable and focused abstractions or simplifications of real-world tasks @xcite , they generally fall short in representing the true variety and complexity of real-world tasks and capturing the lexical diversity of human language. for example, most of them are single-turn inquiries about a single input image, albeit a small fraction supports multi-turn textual interactions or multiple image inputs. consequently, the instruction-following capabilities of models trained on these datasets remain limited in open-world scenarios @xcite . this is reminiscent of the early development of instruction tuning in nlp, where public nlp tasks were initially adopted but eventually superseded by high-quality, diverse open-world instruction data @xcite . nevertheless, collecting such data for multimodal models can be extremely costly. in this paper, we address the above challenge by introducing textbind, an almost annotationfree framework for augmenting llms with multiturn interleaved multimodal instruction-following capabilities. the main idea is to represent images through their textual descriptions, e.g., captions, and utilize an llm to generate multi-turn instructions and responses. to ensure the coherence and meaningfulness of the constructed multi-turn conversations, we propose a series of strategies such as topic-aware image sampling and humanin-the-loop refinement of in-context demonstrations. textbind can harvest large-scale datasets given the abundance of public image-caption pairs. textbind provides examples of processing and generating arbitrarily interleaved image-text content. to accommodate interleaved image-text inputs and outputs, we devise mim, a multimodal model that emphasizes the reasoning abilities of llms and seamlessly integrates image encoder and decoder models. for a comprehensive evaluation of open-world multimodal instruction following, we also construct textbindeval, an evaluation set of textbind validated and refined by human annotators. to verify the effectiveness of textbind, we compare mim models trained on textbind and various existing datasets such as minigpt-4 @xcite and llava @xcite . the results demonstrate that textbind substantially outperforms these datasets across multiple evaluation metrics. additional evaluations on conventional vision-language, including mme @xcite , mmbench @xcite , and mm-vet @xcite , further confirm that textbind provides complementary benefits to existing multimodal datasets. for image generation quality, we compare the proposed mim with gill @xcite and sdxl @xcite , showcasing improved performance (e.g., 0.673 vs. 0.608 clip scores). the advantage is particularly pronounced in multi-turn scenarios. moreover, a holistic human evaluation further substantiates the superior performance of mim trained on textbind in contrast to baseline models in the context of open-world multiturn interleaved multimodal instruction-following. most interestingly, rather than requiring users to provide explicit commands and direct descriptions of the desired image, our model can spontaneously generate images in proper conversation contexts. our qualitative experiments show that mim trained on textbind can perform a wide range of tasks, including composing engaging stories inspired by a set of images, identifying commonalities and differences across multiple images, explaining concepts with vivid images, generating long coherent stories with illustrations, etc. examples can be found in appx. a. we hope textbind serves as an initial step towards building agi that can interact with humans flexibly in different modalities and broad real-world scenarios. compared to existing instruction-tuning data, the examples in textbind (1) generally exhibit greater task and lexicon diversity; (2) typically involve multiple images scattered throughout a multiurn conversation; (3) support multimodal output (image generation). ## textbind in this work, we seek to enhance the multi-turn instruction-following capabilities of a language model in the context of arbitrarily interleaved im-ages and text. constructing such datasets poses significant challenges: 1) it demands inventive thinking for devising high-quality visually-grounded instructions and their responses; 2) it requires specialized expertise to craft appropriate images. to tackle these issues, we introduce textbind, a method that predominantly resorts to existing textonly language models to produce the desired data. ## textbind data from gpt4 we apply textbind to gpt4 and the conceptual captions (cc3m) dataset @xcite as a case study. the details of the construction process can be found in appx. f. in this section, we present comprehensive analyses of the constructed dataset. statistics as depicted in tab. 1, our constructed dataset comprises 25, 629 conversations. the average number of turns per conversation is 3.36 (each turn is defined as a pair of instruction and response). hi! i'm trying to write a cartoon story for my kids featuring a grumpy clown and a lovable cupcake. can you help me craft this? of course! that sounds like a wonderful idea. we can start by establishing our characters. the grumpy clown could be someone who struggles finding joy in his own performances. that's a wonderful idea! how about the cute cupcake? the cupcake, on the other hand, is always upbeat and tries to help the clown find his happiness. despite being a tiny cupcake, it has a big heart and a contagious joy. interesting! what should be their adventure? they could go on a journey to find the ultimate joke that guarantees laughter. the lovable cupcake, with his relentless positivity and cheer, helps the grumpy clown see the lighter side of life and finally find his joy. the mean number of images in each conversation is 2.46. the length of each dialogue is about 285.90. diversity to understand the lexical and task diversity of our constructed data, we identify four types of required visual abilities: image creation, image comparison, intrinsic understanding, extrinsic understanding (see tab. 8 for detailed definitions). the distribution is shown in fig. 2b . we observe that a significant portion of conversations in our dataset focuses on more insightful and informative tasks, such as extrinsic understanding and image comparison. for topic diversity, we display three randomly sampled clusters in fig. 1 . the distribution of images across different turns is depicted in fig. 2c . we also compare the lexical diversity of our dataset and existing datasets in tab. 2. it can be seen that our datasets exhibit higher lexical diversity than existing datasets. ## experiments to verify the effectiveness of the proposed methods, we carry out quantitative evaluations against a set of recent baselines. our quantitative evaluations are divided into three parts: textual response generation, image generation, and a holistic evaluation of multimodal instruction-following. ## conclusion in conclusion, the introduction of the textbind framework has opened new doors for enhancing large language models with multi-turn interleaved multimodal instruction-following capabilities. by requiring only image-caption pairs, our approach significantly reduces the need for high-quality exemplar data, making it a more accessible and scalable solution for various real-world tasks. the mim architecture seamlessly integrates image encoder and decoder models, enabling the model to effectively handle interleaved image-text inputs and outputs. comprehensive quantitative and qualitative experiments demonstrate the remarkable performance of mim, trained on textbind, when compared to recent baselines in open-world multimodal conversations.