What Matters in Training a GPT 4-Style Language Model with Multimodal Inputs? (2024)

Abstract: Recent advancements in GPT-4V have displayed remarkable multi-modal capabilities in processing image inputs and following open-ended instructions. Despite these advancements, there is considerable scope for enhancing open-source multi-modal LLMs, especially in terms of multi-modal understanding accuracy and instruction-following proficiency. In this paper, we conduct a comprehensive study on training GPT4-style models. We introduce Lynx a multi-modal LLM developed through a series of controlled experiments comparing various model variants. This process allowed us to identify and implement an optimal training strategy tailored for multi-modal LLMs. In addition to our model development, we propose a plug-and-play technique designed to augment the instruction-following capabilities of multi-modal LLMs. We have validated the performance of Lynx on multiple benchmarks. Results demonstrate that Lynx not only achieves strong image understanding accuracy but also excels in instruction-following tasks, paving the path for ongoing enhancements in multi-modal LLMs.

## introduction large language models (llms) @xcite @xcite @xcite @xcite have progressed rapidly in recent years and achieved impressive performance in language understanding and generalization. with instruction fine-tuning @xcite @xcite @xcite , llms can be further improved to follow openended instructions from non-expert users and serve as dialog-based assistants in our daily lives. leveraging powerful llms, recent studies have examined methods for adapting llms to multimodal inputs (e.g., images @xcite @xcite , videos @xcite @xcite , and audio @xcite ) and outputs (e.g., vision tasks @xcite , and robotic manipulation skills @xcite @xcite ). notably, gpt-4v (openai, 2023b) has astounded the world with its impressively versatile yet practical capabilities, such as generating descriptions, stories, poetry, advertisements, and codes given images, which were rarely observed in previous vision language models @xcite @xcite . however, it still remains a mystery that: how does gpt-4v obtain its impressive smartness? gpt-4v can analyze and generate descriptions for various types of images, including diagrams, text in images, maps, screenshots of software interfaces, illustrations, comics, and medical imagery. such capabilities likely depend on extensive annotated data and perhaps the integration of tools like ocr. this presents a substantial challenge for open-source multi-modal llms which are end-to-end models trained on open-source datasets with lower-quality annotations. despite these challenges, we have observed that current multi-modal language models usually suffer from hallucinations, such as generation of facts unrelated to the image inputs, and degraded instruction-following abilities compared to their text-only counterparts, as shown in figure 1 . though actively investigated recently, the open-source multi-modal llms are usually different in training data, training recipes, prompts, and evaluation benchmarks, which makes it challenging to identify which factors are crucial in achieving a strong baseline model. in this paper, we conduct a comprehensive study on training gpt4-style models. by implementing multiple model variants under controlled settings and conducting extensive experiments to draw reliable conclusions both quantitatively and qualitatively, our findings can be summarized as follows: our experiments with coyo700m @xcite , @xcite , and blipcapfilt @xcite demonstrate that pre-training on coyo700m and datacomp1b does not yield improved model performance. furthermore, it is crucial to avoid utilizing low-quality annotations during the instruction fine-tuning phase. we incorporate a range of open-source multimodal datasets, transforming them into an instruction-following format using manually crafted prompts, supplemented by additional prompts generated by gpt-4, producing 500 prompts for over 50 tasks in total. integrating nlp instruction data during training enables the model to handle a broader range of tasks, compensating for the typical limitations of open-source multi-modal datasets, which are predominantly focused on image captioning and visual reasoning. furthermore, our results also indicate that multimodal models based on vicuna-7b, which have undergone instruction fine-tuning, generally outperform those based on llama-7b. through our study, we present lynx, a multimodal llm with a three-stage training recipe. the first stage focuses on aligning vision and language, the second stage enhances the resolution of image inputs, and the final stage concentrates on instruction fine-tuning. at each stage, a different set of model parameters is trained using a varied mixture of image-text pairs, multi-modal datasets, and nlp instruction data, in order to fully exploit training data. furthermore, we propose a plug-and-play method aimed at boosting the instruction-following capabilities of multi-modal llms. our observations indicate that a more extended dialogue history can effectively unlock and amplify the instructionfollowing abilities inherent in llms. we recognize that most multi-modal tasks can be broken down into two steps: describing the image in details, and subsequently utilizing the capabilities of llms. consequently, during evaluation, we initiate the task by prompting the model to describe the image. this strategy consistently results in improved performance across a variety of multimodal instruction-following tasks, demonstrating the efficacy of our proposed method. experimental results show that lynx is a strong baseline for multi-modal llms, achieving state-ofart performance. 1 specifically, our model excels in mme's perception @xcite , demonstrating superior image understanding accuracy, and shows impressive results in visit-bench @xcite , which evaluates multi-modal instructionfollowing abilities. we also adopt the owleval test set proposed by mplug-owl @xcite for human evaluation, with lynx achieving state-of-art multi-modal instruction-following results. ## lynx lynx is a large language model that can take images and videos as inputs. in this section, we will introduce our lynx in detail, including the architecture (2.1) and three-stage training recipe(2.2). ## experiment in this section, we aim to answer the following questions by empirical studies: ## conclusions in this paper, we present lynx, a multi-modal large language model that can take as input images/videos and responses with open-ended natural languages. through extensive empirical study, we show that our model outperforms other existing open-source models both in multi-modal understanding accuracy and instruction-following capabilities. we also explore different factors that can affect the performance of a multi-modal large language model and conclude that: 1) the generative pretraining is much more sensitive to the quality of training data than previous methods such as contrastive training; 2) the abilities of instruction following are closely related to the number of different tasks and prompts used for training; 3) improving both the multi-modal understanding accuracy and instruction-following capabilities is important for multi-modal large language models; 4) by first prompting the model to describe the image, the instruction-following abilities can be improved. for future work, it is promising to scale up the model to a larger size (e.g. 30b and 65b llama @xcite ). moreover, a high-quality multi-modal dataset with diverse instructions is also needed to train such models. andy brock, soham de, samuel l smith, and karen simonyan. 2021. high-performance large-scale image recognition without normalization. in international conference on machine learning, pages 1059-1071. pmlr. tom brown, benjamin mann, nick ryder, melanie subbiah, jared d kaplan, prafulla dhariwal, arvind neelakantan, pranav shyam, girish sastry, amanda askell, et al. 2020. language models are few-shot learners. advances in neural information processing systems, 33:1877-1901. erhan bulbul, aydin cetin, and ibrahim alper dogru. 2018. human activity recognition using smartphones. in 2018 2nd international symposium on multidisciplinary studies and innovative technologies (ismsit), pages 1-6. ieee. minwoo byeon, beomhee park, haecheon kim, sungjun lee, woonhyuk baek, and saehoon kim. 2022. coyo-700m: image-text pair dataset. https: //github.com/kakaobrain/coyo-dataset . soravit changpinyo, piyush sharma, nan ding, and radu soricut. 2021. conceptual 12m: pushing web-scale image-text pre-training to recognize long-tail visual concepts. in proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 3558-3568. david chen and william b dolan. 2011. collecting highly parallel data for paraphrase evaluation. in proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190-200. guo chen, yin-dong zheng, jiahao wang, jilan xu, yifei huang, junting pan, yi wang, yali wang, yu qiao, tong lu, et al. 2023. videollm: modeling video sequence with large language models. arxiv preprint arxiv:2305.13292. xi chen, xiao wang, soravit changpinyo, aj piergiovanni, piotr padlewski, daniel salz, sebastian goodman, adam grycner, basil mustafa, lucas beyer, et al. 2022. pali: a jointly-scaled multilingual language-image model. arxiv preprint arxiv:2209.06794. xingyu chen, zihan zhao, lu chen, danyang zhang, jiabao ji, ao luo, yuxuan xiong, and kai yu. 2021. websrc: a dataset for web-based structural reading comprehension. arxiv preprint arxiv:2101.09465. xinlei chen, hao fang, tsung-yi lin, ramakrishna vedantam, saurabh gupta, piotr dollár, and c lawrence zitnick. 2015. microsoft coco captions: data collection and evaluation server. arxiv preprint arxiv:1504.00325. wei-lin chiang, zhuohan li, zi lin, ying sheng, zhanghao wu, hao zhang, lianmin zheng, siyuan zhuang, yonghao zhuang, joseph e. gonzalez, ion stoica, and eric p. xing. 2023. vicuna: an opensource chatbot impressing gpt-4 with 90%* chatgpt quality.