How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning? (2024)

Abstract: The increase in parameter size of multimodal large language models (MLLMs) introduces significant capabilities, particularly multimodal in-context learning, where MLLMs enhance task performance without updating pre-trained parameters. However, this effectiveness hinges on the appropriate selection of in-context examples, a process currently biased towards visual data, overlooking textual information. More importantly, the area of supervised retrievers for retrieval of multimodal in-context learning, crucial for optimal in-context example selection, continues to be investigated. Our study provides an in-depth evaluation of the impact of textual information on the unsupervised selection of in-context examples in multimodal contexts, uncovering a notable sensitivity of retriever performance to the employed modalities. Based on the above finding, we introduce a novel supervised MLLM prompt retriever MSIER that leverages a trained retriever based on MLLM’s confidence to select examples, which enhances multimodal in-context learning efficiency. This approach is validated through extensive testing across three different tasks, demonstrating the method’s effectiveness. Additionally, we investigate the influence of modalities on our supervised retrieval method’s training and explore the transferability of the supervised prompt retriever. This exploration paves the way for future advancements, highlighting the potential for refined in-context learning in MLLMs through the strategic use of multimodal data. The public code is available at https://github.com/NUS-HPC-AI-Lab/Multimodal-ICL-Retriever.

## introduction the capability of large language models (llms) for in-context learning (icl) has acquired significant attention and exhibited remarkable efficacy across a diverse range of downstream tasks @xcite @xcite . icl aims to adapt the pre-trained model to achieve ## conclusion in this study, we conducted an extensive evaluation of textual information's role in unsupervised and supervised in-context example retrieval for multimodal in-context learning. we introduce a novel msier methodology that incorporates mllms' self-contained information to train a supervised prompt retriever. our experiments across three multimodal tasks demonstrate that integrating text modality and the foundational knowledge of mllms significantly enhances the efficiency of example selection, yielding substantial improvements over existing approaches. moreover, msier demonstrates high efficiency by outperforming benchmarks with far fewer examples, while also exhibiting strong transferability by remaining effective for larger models after training on smaller ones, thereby improving cost-efficiency for largescale inference. future research could delve into optimizing inter-modal interactions, leveraging our findings for more effective multimodal in-context learning retrieval strategies.