I nfi MM : Advancing Multimodal Understanding with an Open-Sourced Visual Language Model (2024)

Abstract: In this work, we present InfiMM, an advanced Multimodal Large Language Model that adapts to intricate vision-language tasks. InfiMM, inspired by the Flamingo architecture, distinguishes itself through the utilization of large-scale training data, comprehensive training strategies, and diverse large language models. This approach ensures the preservation of Flamingo’s foundational strengths while simultaneously introducing augmented capabilities. Empirical evaluations across a variety of benchmarks underscore InfiMM’s remarkable capability in multimodal understanding. The code can be found at: https://anonymous.4open.science/r/infimm-zephyr-F60C/.

## introduction recently, multimodal large language models (mllms) have shown a transformative evolution through the integration of pretrained vision encoders with large language models (llms). seminal contributions to this domain include flamingo @xcite , llava @xcite , blip-2 @xcite , and minigpt-4 @xcite . mllms demonstrate exceptional proficiency across a variety of tasks, including image captioning, visual question answering, and more complex activities such as generating code from images, converting image plots into markdown format tables, and simulating web browsing. for effective integration of pretrained vision encoders with large language models, careful design of vision-language connector modules is essential. these modules play a critical role in transforming and aligning visual tokens to formats compatible with large language models, as well as effectively leveraging these tokens. models like flamingo @xcite and blip-2 @xcite utilize perceiver resampler/q-former techniques, * corresponding authors offering flexibility and nuanced alignment with language counterparts. however, this approach can incur high computational costs and risk information loss @xcite . conversely, models such as llava and minigpt-v2 employ simpler multi-layer perception (mlp) strategies, reducing computational complexity but potentially sacrificing nuanced representation of visual data. in the utilization of the transformed tokens, architectures akin to flamingo employ cross-attention mechanisms, enabling nuanced interactions between token types without necessitating an expansion of the token sequence length. this method effectively manages computational load. conversely, llava-style models adopt a direct concatenation approach, which, while straightforward, leads to an augmentation in token sequence length and computational complexity. though efficient in the inference stage, few works adopt the flamingo-style architecture. open-flamingo @xcite and idefics @xcite are two reproductions of the flamingo. however, as they use less capable language models and limited training data, their performance could be improved. further we propose three-stage training strategies for vision-language alignment, vqa knowleage injection and unreshing conversation abality. we utilize a stronger vision encoder, language model, and higher-quality data to build a stronger model. we anticipate that this will foster development within the field. ## related work large language models (llms) have made significant advancements @xcite @xcite @xcite . these models are powerful in chatting and can finish many tasks only with different instructions. though impressive, these models are limited to only the language domain but not other modalities. they have subsequently been extensively utilized in multimodal tasks such as image-to-text generation and video-to-text generation @xcite @xcite @xcite @xcite , giving rise to a new class of models called multimodal large language model (mllm). flamingo @xcite leverages pretrained language models within the mllm framework, employing gated-cross attention to integrate visual information into textual sequences. in contrast, blip-2 @xcite , minigpt4 @xcite , and llava @xcite propose a novel approach by converting visual signals into soft tokens and directly integrating them into language models. utilizing gated-cross attention as a modality connector introduces more trainable parameters but can potentially reduce inference cost as the visual signal will not be turned into soft tokens, thereby not increasing the sequence length of large language models. while numerous open-source projects have emerged following the architectures of llava and blip-2, there needs to be more emphasis on the flamingo-style architecture. openflamingo @xcite and idefics @xcite represent two open-source models adopting the flamingo-style approach. however, due to constraints imposed by their language model and vision encoder, their capabilities could be more remarkable. in this study, we adopt the flamingo framework and harness a more potent combination of language model and vision encoder to construct a robust model. additionally, we employ higher-quality data for training, aiming to enhance the model's strength. these efforts will likely result in a more formidable model and contribute to the advancement of research in mllms. ## experiment we evaluate infimm across a diverse array of tasks. for image caption, we test our model with coco and flickr30k. for general vqa tasks, we leverage benchmarks such as okvqa @xcite , vqav2 @xcite and textvqa @xcite . on these dataset, we only evaluate the pretrained model in a zero-shot and few-shots manner. results can be found in we also assess the logical reasoning capabilities of our model by employing newly introduced benchmarks, including mm-vet @xcite , mme @xcite , mmbench @xcite , infimm-eval @xcite , and mmmu @xcite . notably, the mmmu @xcite presents challenging tasks that demand advanced subject knowledge and deliberate reasoning at a collegiate level. these tasks span diverse fields such as physics, chemistry, and biology. the mm-vet benchmark assesses the integrated capabilities of models. ## limitations although infimm demonstrates robust performance in vision-language modeling while maintaining a balanced computational load for processing multiple images, its efficacy is hampered by the constraint of limited image size, thereby restricting its ability to address complex visual content effectively. ## conclusion in this study, we introduce infimm, an advanced multimodal large language model that significantly advances the field of visual language understanding. infimm's architecture, inspired by flamingo and enhanced by our methodological innovations, demonstrates a delicate balance between computational efficiency and the capacity to handle nuanced visual-language tasks. evaluation on various benchmarks highlights infimm's remarkable ability to understand complex scenes and shows good reasoning ability. infimm represents a significant step forward in the multimodal understanding domain.