C a MML : Context-Aware Multimodal Learner for Large Models (2024)

Abstract: In this work, we introduce Context-Aware MultiModal Learner (CaMML), for tuning large multimodal models (LMMs). CaMML, a lightweight module, is crafted to seamlessly integrate multimodal contextual samples into large models, thereby empowering the model to derive knowledge from analogous, domain-specific, up-to-date information and make grounded inferences. Importantly, CaMML is highly scalable and can efficiently handle lengthy multimodal context examples owing to its hierarchical design. Based on CaMML, we have developed two multimodal models, CaMML-7B and CaMML-13B, that have shown exceptional performance across an array of benchmark datasets for multimodal tasks. Remarkably, CaMML-13B achieves the state-of-the-art performance on over ten widely recognized multimodal benchmark datasets, surpassing LLaVA-1.5 (13B) with a noticeable margin, without integration of any external resources. Moreover, we have conducted extensive ablative studies to inspect the inner workings of CaMML and performed qualitative analyses to showcase its effectiveness in handling real-world challenging cases. Code and models are available at: https://github.com/amazon-science/camml.

## introduction recently, large multimodal models (lmms) @xcite @xcite @xcite @xcite have demonstrated remarkable performance in a variety of tasks, including but not limited to visual question answering, image captioning, visual grounding, visual-language reasoning, optical character recognition, and visual entailment. notably, in certain benchmark assessments, these multimodal foundation models have even exceeded human-level performance @xcite . despite the impressive performance, their ability to make inferences is constrained by the knowledge encoded in the model parameters. the inflexible design of these models makes it challenging for them to generalize from contextual examples. for instance, llava-1.5 falls short in processing multiple images and attributes it to the lack of corresponding instruction-tuning training data @xcite . nevertheless, learning and making inferences through contextual examples are fundamental elements of our cognitive processes. human beings frequently tackle intricate problems by relying on past experiences and identifying analogous situations. taking inspiration from the cognitive process, we hypothesize that empowering large multimodal models with the capability to perceive and derive insights from analogous contextual examples can significantly streamline the inference process and lead to more precise predictions. nonetheless, the means to replicate the human cognitive processes in lmms remain unclear. as such, our goal is to empower multimodal foundational models to harness context-aware learning, thereby enhancing their ability to comprehend and adapt to previously unseen examples. identifying relevant context examples is relatively straightforward; this process can be facilitated using multimodal embedding models like image-bind @xcite or clip @xcite . this approach simulates the act of recalling similar situations from past experiences. yet, effectively and efficiently integrating the identified similar context samples into large models poses challenges, particularly given the potential variability in the number of context samples and interleaved modalities, resulting in lengthy and heterogeneous context input. to this end, we propose a context-aware multimodal learner, dubbed as camml, for lmms. camml acts as a crucial intermediary between the contextual examples and a large language model (llm). our approach is structured hierarchically, where the initial level establishes connections between the text and image modalities for each example through cross-attention mechanisms. this integration of text and image information enables a deeper understanding of the interleaved context. following this, another module takes the outputs of the first level and performs cross-attention between the contextual information and a predefined set of learnable, fixed-length tokens. the resulting output from this level is then used as fixed-length input for the llm, allowing the model to leverage the refined and context-aware information to perform complex multimodal understanding tasks. to summarize, we make the following contributions: ## related work large multimodal models the success of llms has sparked a burgeoning interest in scaling up multimodal models. one prevalent strategy involves the incorporation of vision encoders, such as vit @xcite and clip, into existing llms (e.g., llama @xcite , vicuna @xcite ). for example, llamaadapter @xcite inserts learnable adaption prompts into llama and make it to follow multimodal instructions for multimodal reasoning. blip2 @xcite bridges different modalities via q-former and employs a two-stage training method to bootstrap both representation learning and llm (e.g., opt @xcite , flan-t5 (chung et al., 2022)) generative learning capabilties. llava @xcite projects encoded image features to text token space using linear layers and shows state-of-the-art performance on a variety of multimodal tasks. another noteworthy series of strategy is to unify multimodal input data and pretrain the model from scratch, exemplified by ofa @xcite , perceiver @xcite , uni-perceiver @xcite , unival @xcite , and unified-io @xcite . these models map images, text, and other modalities into the same io space and use a causal language model objective for training. ## conclusion we present a new methodology called camml, a context-aware multimodal learner designed for the fine-tuning of large multimodal models. with camml, we build two multimodal models, camml-7b and camml-13b. camml empowers them to draw insights from analogous, domainspecific, and up-to-date context samples to make grounded inferences. moreover, it employs a lightweight multimodal perceivers to seamlessly integrate these context samples, enabling an efficient processing of lengthy context tokens. our proposed camml-13b model achieves state-ofthe-art results across more than ten prominent multimodal benchmarks, surpassing previous methods by a substantial margin. these achievements underscore camml's effectiveness in various multimodal applications.