Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data (2024)

Abstract: The remarkable multimodal capabilities demonstrated by OpenAI’s GPT-4 have sparked significant interest in the development of multimodal Large Language Models (LLMs). A primary research objective of such models is to align visual and textual modalities effectively while comprehending human instructions.Current methodologies often rely on annotations derived from benchmark datasets to construct image-dialogue datasets for training purposes, akin to instruction tuning in LLMs. However, these datasets often exhibit domain bias, potentially constraining the generative capabilities of the models. In an effort to mitigate these limitations, we propose a novel data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. This approach harnesses the power of generative models, marrying the abilities of ChatGPT and text-to-image generative models to yield a diverse and controllable dataset with varied image content. This not only provides greater flexibility compared to existing methodologies but also significantly enhances several model capabilities. Our research includes comprehensive experiments conducted on various datasets using the open-source LLAVA model as a testbed for our proposed pipeline. Our results underscore marked enhancements across more than ten commonly assessed capabilities.

## introduction the launch of openai's chatgpt @xcite has marked a significant milestone in artificial intelligence (ai), showcasing the advanced capabilities of large language models (llms). these models, exemplified by @xcite , demonstrate exceptional versatility by handling not just images but also excelling in tasks once difficult to accomplish. this includes understanding humor within images and acting as intelligent agents @xcite , aspects that highlight its revolutionary potential. however, despite these notable achievements, a crucial aspect remains undisclosed: the specific mechanics underlying gpt-4, particularly concerning the seamless integration of multimodal information into llms. this knowledge gap has prompted a concerted research effort to address this puzzle. among the promising approaches, an emerging method receiving considerable attention involves the utilization of adapter-based techniques @xcite @xcite , which allow the training of a visual-to-text adapter that converts features from pre-trained visual models into llm tokens, showing promise in achieving results comparable to gpt-4. the effectiveness of adapter-based methods stems from their ability to leverage the extensive pre-existing knowledge in large visual models and llms. by focusing on training a lightweight adapter, these methods avoid the computational expense of training comprehensive models from scratch, thereby offering a more efficient pathway to enhancing llms' multimodal integration capabilities. a prerequisite for implementing these frameworks is the availability of paired vision-text image data. such datasets are essential for aligning visual and textual information, facilitating the llms' understanding of complex human instructions. analogous to instruction tuning in llms @xcite , this process is commonly referred to as visual instruction tuning. existing methods @xcite @xcite @xcite typically construct visual instruction tuning datasets by leveraging existing datasets, extracting information such as image captions, spatial locations, and categories to form dialogues. this approach maximizes resource utilization, creating a comprehensive and efficient training dataset for multimodal llms. despite the efficiency and simplicity of this approach to dataset construction, certain limitations still persist. existing large-scale vision-text datasets, such as laion @xcite and cc @xcite , often contain noise. consequently, training only a subset may inadequately align visual-text features for immediate user requirements. moreover, benchmark datasets @xcite @xcite often exhibit a domain bias, primarily in terms of image styles. for instance, prevalent datasets such as coco @xcite predominantly feature images from everyday life, while stylized images like cartoons are rarely represented. additional, these vision annotations may also constrain the types of dialogues generated from them. for example, almost none of the current datasets contain data to directly enhance the model's ability to comprehend jokes presented in the images, an impressive feature of gpt-4. moreover, as multi-image dialogues become increasingly integral to the practical application of multimodal llms, the current lack of comprehensive multi-image datasets further underscores the need for data enrichment in this domain. in addressing these limitations, we propose a novel data collection approach to enhance visual instruction tuning. building upon recent successes in generative ai, we leverage generative models to produce image-dialogue pair data for visual instruction tuning. more concretely, we employ chat-gpt to create data that includes image-generating prompts and content-based dialogues. we then utilize the text-to-image diffusion model, stablediffusion @xcite , to generate images based on these prompts. finally, the synthesized images and generated dialogues are employed to train multimodal llms. simultaneously generating both image content and dialogues enables the production of diverse training data, affording greater control over its nature and quality. this flexibility allows us to construct multi-turn dialogues and datasets involving multi-image reasoning, which are challenging to obtain from other benchmarks. additionally, our methodology can potentially integrate more advanced image generative models, such as dall-e 3 @xcite , to provide higher-level control over image contents like specifying complex spacial relations. this advanced control could generate more complex instructions to enhance image understanding capabilities. examples from our synthesized visual instruction tuning datasets are shown in figure 1 . building upon the flexible pipeline outlined above, users can tailor the generation of data to enhance specific capabilities based on their task requirements. furthermore, our method of generating both images and dialogues eliminates constraints on data volume, thereby facilitating the production for limitless scaling of the datasets. to demonstrate the effectiveness of our proposed pipeline, we conducted extensive experiments. our main contributions are as threefold: ## related work recent research @xcite efforts in multimodal large language models (llms) have yielded promising strategies to efficiently align the embeddings of other modalities with language tokens. this has made it possible to effectively utilize pre-trained encoders from other modalities and llms, which effectively reduces the computational burden and training time. while there are alternative research approaches that include training-free methods leverage expert models @xcite @xcite , these are not the focus of our work here. adapter-based llms represent a significant research direction, introducing methods to connect modalities through learnable interfaces with minimal training efforts. these approaches @xcite @xcite @xcite allow for the use of pre-trained modal encoders, reducing the need for training from scratch. variations include direct training of projection layers for embedding alignment and the use of learnable queries for extracting modality-specific information, as seen in models like llava @xcite and flamingo @xcite . innovations such as the llama-adapter @xcite and lavin @xcite have introduced lightweight and mixed-modality adapters, respectively, enhancing the field's diversity. visual instruction tuning datasets are crucial for training multimodal llms, focusing on aligning modalities and enabling instruction following. most existing methods @xcite @xcite @xcite @xcite rely on benchmark datasets for constructing visual instruction tuning datasets, which may be limited by the fixed categories in annotations. our method leverages well-trained image generation models to produce controllable image data, enhancing multimodal llm capabilities and allowing for the integration of advanced generative models for specific guidance forms, offering a more flexible and diverse approach to dataset construction. recently, the concept of employing synthesized multimodal datasets has been extended to the training of models for chart and video understanding @xcite , showcasing its broad applicability and potential in advancing model training across various domains. ## preliminary to assess the effectiveness of our data generation strategy, we chose the open-sourced llava @xcite as our multimodal llm model. it should be noted that our pipeline is model-agnostic, making it applicable for various models. this section serves as a foundation, briefly summarizing the llava model's design and training methods to prepare for a thorough exploration of our pipeline. the reader may refer to the original publication @xcite for detail. architecture. the llava model integrates vicuna-13b @xcite as the language model with a pre-trained clip visual encoder vit-l/14 @xcite for extracting visual features, transforming these features into language embedding tokens through a linear layer. this linear layer was updated in llava-1.5 @xcite with a two-layer mlp, replacing vicuna-13b with vicuna-13b-v1.5 and increasing input image size to 336x336. a detailed illustration of this model structure can be found in figure 2 . training and datasets. llava's training focuses on visual instruction tuning with data triplets: images, questions, and answers, aiming for predictive accuracy in an autoregressive manner. the training comprises two stages: the first emphasizes modality alignment using 595k image-text pairs, optimizing the linear layer with static visual encoder and llm weights. the second stage, using 158k multimodal dialogue data from coco, extends optimization to the llm's weights for comprehensive modality integration. llava-1.5 further enriches the dataset by incorporating additional data like region-level vqa @xcite @xcite and gqa (hudson and manning, 2019), expanding the second-stage dataset to 665k examples. ## methods this section outlines our dual-generation approach for creating visual instruction tuning datasets, which synthesizes images and their corresponding dialogues, as illustrated in figure 3 . we detail each component below. ## experiments in this section, we detail the experiments conducted to validate the effectiveness of our novel data collection approach for visual instruction tuning. we describe the training datasets, evaluation strategy, and both quantitative and qualitative outcomes. ## conclusion and future work in the rapidly evolving realm of large language models, efficiently integrating multimodal information is a key research focus. in this study, we introduced an innovative data collection method to enhance visual instruction tuning for multimodal models. compared to existing strategies, our approach uniquely combines image and dialogue generation, effectively addressing limitations found in benchmark datasets. by carefully crafting instruction templates, our method ensures high-quality training data covering a broad range of crucial capabilities for multimodal models and users can generate customized data based on their specific requirements. our research opens avenues for exploration. moving forward, we aim to leverage advanced generative models to enhance model abilities, including spatial comprehension and fine-grained recognition. with promising results from our dualgeneration method, forward-thinking data collection techniques are poised to play a significant role in the future of llm research.