In-Context Compositional Generalization for Large Vision-Language Models (2024)

Abstract: Recent work has revealed that in-context learning for large language models exhibits compositional generalization capacity, which can be enhanced by selecting in-context demonstrations similar to test cases to provide contextual information. However, how to exhibit in-context compositional generalization (ICCG) of large vision-language models (LVLMs) is non-trival. Due to the inherent asymmetry between visual and linguistic modalities, ICCG in LVLMs faces an inevitable challenge—redundant information on the visual modality. The redundant information affects in-context learning from two aspects: (1) Similarity calculation may be dominated by redundant information, resulting in sub-optimal demonstration selection. (2) Redundant information in in-context demonstrations brings misleading contextual information to in-context learning. To alleviate these problems, we propose a demonstration selection method to achieve ICCG for LVLMs, by considering two key factors of demonstrations: content and structure, from a multimodal perspective. Specifically, we design a diversity-coverage-based matching score to select demonstrations with maximum coverage, and avoid selecting demonstrations with redundant information via their content redundancy and structural complexity. We build a GQA-ICCG dataset to simulate the ICCG setting, and conduct experiments on GQA-ICCG and the VQA v2 dataset. Experimental results demonstrate the effectiveness of our method.

## introduction compositional generalization, understanding unseen compositions by recombining known primitives, is a fundamental ability of human intelligence @xcite . deploying such ability in machine learning models has received increasing attention and significant progress in vision and language. nonetheless, most work @xcite @xcite focuses on boosting the compositional generalization ability of models by re-training or fine-tuning the models with explicit constraints, which is not applicable to large pre-trained models. recently, in-context learning with large visionlanguage models (lvlms) @xcite exhibits impressive performance on few-shot learning of various vision-language tasks. a question naturally arises: whether and how lvlms exhibit in-context compositional generalization (iccg)? recent work @xcite ) has revealed that in-context learning for large language models (llms) exhibits compositional generalization by selecting in-context demonstrations similar to a test case. however, selecting demonstrations based on similarity is not applicable to large vision-language models for iccg, because the inherent asymmetry between visual and linguis-tic modalities leads to much redundant information in the visual modality. such redundant information affects in-context learning in two aspects. firstly, the calculation of multimodal similarity may be dominated by redundant information, resulting in sub-optimal demonstration selection. secondly, redundant information in demonstrations brings misleading contextual information to in-context learning. for example, for a test case with an image that contains a dog in an indoor environment and a question "is the image indoors?", the cosine similarity of clip features @xcite is greater when a dog was included in the demonstration rather than an indoor environment, though the dog is redundant to answer the question, as shown in figure 1 @xcite . moreover, the more redundant information in the selected in-context demonstrations, the harder it will be for the model to answer the test case, as shown in figure 1 (b) . in this paper, we consider the asymmetry characteristics of multimodal data from both the content and structure perspectives, and propose a demonstration selection method for lvlms to achieve incontext compositional generalization. a diversitycoverage-based matching score is designed to select demonstrations with maximum coverage and less redundant information. specifically, for either a demonstration or a test case, we extract its primitives and construct constituent trees for its visual and linguistic information as its content and structure information, respectively. the calculation of the proposed matching score between a demonstration and a test case falls into two terms: (1) the content intersection and structure intersection between the demonstration and the test case at different modalities. (2) the symmetric difference between visual and linguistic content and the depth of the constituent tree, which model content redundancy and structural complexity of a demonstration, respectively. the first term is used to select diverse demonstrations with maximum coverage of the test case, while the second term is used to select demonstrations with less redundant information. in doing so, the redundant information mixed in in-context learning is reduced. to quantitatively evaluate the in-context compositional generalization ability of lvlms, we build a gqa-iccg dataset based on the gqa dataset @xcite . we first filter out samples from the val-balanced split of gqa that contain novel compositions of primitives seen in the train-balanced split of gqa, to construct the test set of the gqa-iccg. for each test case, the gqa-iccg contains a candidate demonstration set. the candidate set is constructed by randomly selecting 10 samples that contain each primitive in the test sample. this ensures that for each test case, there are various ways to select a set of in-context demonstrations that fully cover its primitives to satisfy the compositional generalization setting. we evaluated our method with four lvlms varies in parameters (3b to 9b), openflamingo @xcite , otter @xcite , fromage @xcite and idefics @xcite on our gqa-iccg dataset and the vqa v2 dataset @xcite . experimental results demonstrate the effectiveness of our method. to sum up, our contributions are as follows: (1) to the best of our knowledge, we are the first to investigate the in-context compositional generalization for large vision-language models, which is a promising few-shot paradigm. (2) we propose a demonstration selection method for large visionlanguage models to achieve in-context compositional generalization, taking both the content and structure of demonstrations into consideration.(3) we present a gqa-iccg dataset to quantitatively evaluate the in-context compositional generalization ability of large vision-language models. ## conclusion this work has presented a demonstration selection method to select in-context demonstrations, which can improve the iccg ability of lvlms. a diversity-coverage-based matching score is designed to reduce the impact of redundant visual information arising from the inherent asymmetry between visual and linguistic modalities. we have constructed a gqa-iccg dataset to enable the quantitative evaluation for the iccg ability of lvlms. experimental results demonstrate that content coverage/diversity/redundancy and structural coverage/diversity/complexity play an important role in demonstration selection.