Visual In-Context Learning for Large Vision-Language Models (2024)

Abstract: In Large Visual Language Models (LVLMs), the efficacy of In-Context Learning (ICL) remains limited by challenges in cross-modal interactions and representation disparities. To overcome these challenges, we introduce a novel Visual In-Context Learning (VICL) method comprising Visual Demonstration Retrieval, Intent-Oriented Image Summarization, and Intent-Oriented Demonstration Composition. Our approach retrieves images via ”Retrieval & Rerank” paradigm, summarises images with task intent and task-specific visual parsing, and composes language-based demonstrations that reduce token count and alleviate cross-modal interaction problem. Experimental evaluations on five visual reasoning datasets demonstrate the effectiveness of our method. Moreover, our extensive experiments leverage information flow analysis to elucidate the effectiveness of our method, and investigate the impact of length and position of demonstrations for LVLM. The use of in-context unlearning further shows promise in resetting specific model knowledge without retraining.

## introduction large language models (llms) exhibit impressive reasoning abilities across various natural language tasks @xcite . researchers are actively investigating the extension of llm capabilities to address challenges in the visual domain by integrating llms with vision models @xcite . this endeavor has given rise to the development of large visual language models (lvlms). lvlms are designed to seamlessly fuse information from both images and text, enabling them to tackle more intricate tasks that demand a profound the image shows a person with their hands forming a heart shape, likely expressing love or affection. the person is standing near a body of w a t e r, w h i c h a d d s a s e r e n e a n d romantic atmosphere to the scene. comprehension of both modalities @xcite . the llm has a remarkable capability, known as in-context learning (icl), which involves providing llms with a limited amount of labeled data as demonstrations to improve their reasoning ability @xcite @xcite . this approach can significantly enhance the performance of llms in various nlp tasks, such as translation @xcite , sentiment classification @xcite , and question-answering @xcite . in the icl, llms can flexibly adjust their behavior based on the provided context, allowing them to understand and perform tasks with few labeled data. the success of icl has motivated research into extending the icl capabilities to lvlms. however, some studies @xcite find that while lvlms have icl capabilities, they are not as pronounced as those observed in llms. two factors lead to this difference: (1) as shown in figure 1 (a), as observed in previous research @xcite , visual-language interactions occur at deeper layers in lvlms, which highlights the difficulty of cross-modal interactions. in icl, label words aggregate information in shallow layers and subsequently distribute it in deeper layers @xcite . consequently, the issue of cross-modal interactions significantly impacts the icl capabilities of lvlms. (2) from figure 1 (b), our analysis reveals that visual features and llm embeddings occupy distinct spaces in lvlms. this observation underscores the inherent cross-modal gap present in lvlms. while many works @xcite are dedicated to enhancing the icl capabilities of llms, the challenges faced by lvlms in this regard differ substantially. this discrepancy arises from the difficulty in cross-modal interactions and inherent disparities in representation spaces within lvlms, which impose limitations on their icl performance. in this study, we present a novel visual in-context learning (vicl) method to enhance the icl capability of lvlms. vicl comprises visual demonstration retrieval, intent-oriented image summarization, and intent-oriented demonstration composition. for visual demonstration retrieval, we employ a pre-trained image encoder as a retriever to search for relevant candidate images for the provided image. subsequently, we rerank the retrieved candidates using textual descriptions of the provided image. moreover, lvlms perform intent-oriented image summarization, automatically extracting image summary with task intent and task-specific visual parsing from image-label pairs. in addition, intent-oriented demonstration composition uses language cues to create demonstrations that enhance icl of lvlms, replacing images with image summary in demonstrations. our method not only boosts in-context learning but also introduces in-context unlearning, allowing models to discard or reset specific knowledge through demonstration. the substitution of images with visual summaries significantly reduces the token count, enabling the concatenation of more demonstrations without encountering token limitations. in comparison to conventional visual-language interactions in standard visual icl approaches, our method solely relies on language interactions to facilitate effective demonstration understanding. our experiments across five image reasoning datasets evaluate our method's effectiveness, comparing lvlm performance using our approach against a baseline method. moreover, we employed information flow for interpretative analysis, to ver-ify the effectiveness of our method. furthermore, examined the influence of demonstrations and their sequence length on lvlm's icl capability. we investigate the importance of intent-oriented image summaries and the impact of demonstrations order. in addition, we explore the application of incontext unlearning, demonstrating its feasibility for unlearning scenarios without additional training. ## background generative language models could self-supervised learn knowledge from pre-training corpora @xcite @xcite . as the scale of model parameters and pre-training corpora expands, researchers have observed the emergent ability in large language models, enabling them to provide accurate answers merely by adjusting the input prompt without fine-tuning model parameters @xcite . the large language model is abstracted as a function denoted as llm(•). given an input prompt, denoted as p, it generates the corresponding output, denoted as o. in the most common case of question-answering, the prompt p is exactly the question q raised by the user. and the output o is expected to be the answer a to the question q. however, under the in-context learning paradigm, the input prompt p icl is carefully designed: considering the established efficacy of incontext learning on large language models, it was intuitive to extend the in-context learning approach to large visual-language models upon its emergence @xcite @xcite . in the context of a visual question answering task, the formulation of in-context learning on large visual-language model can be delineated as follows: ## visual in-context learning in this section, we elaborate on our approach vicl, which comprises three core components: visual demonstration retrieval, intent-oriented image summarization, and intent-oriented demonstration composition. the pipeline of vicl is shown in figure 2 . ## conclusion this paper has introduced the integration of in-context learning (icl) into large visual language models (lvlms), addressing challenges in cross-modal interactions and the distinct representation spaces. through visual in-context learning (vicl), incorporating visual demonstration retrieval, intent-oriented image summarization, and demonstration composition, lvlms show enhanced performance in understanding visual and textual information. in vicl, we have not only streamlined the in-context learning process but also introduced the concept of in-context unlearning, allowing lvlms to adjust their knowledge base dynamically without the need for retraining. our method shows effective in improving lvlms in processing multi-modal tasks. the extensive evaluations verify the effectiveness of our vicl method, highlighting its potential to bridge the gap between visual and linguistic modalities.