C om CLIP : Training-Free Compositional Image and Text Matching (2024)

Abstract: Contrastive Language-Image Pretraining (CLIP) has demonstrated great zero-shot performance for matching images and text. However, it is still challenging to adapt vision-language pretrained models like CLIP to compositional image and text matching — a more challenging image and text matching task requiring the model’s understanding of compositional word concepts and visual components. Towards better compositional generalization in zero-shot image and text matching, in this paper, we study the problem from a causal perspective: the erroneous semantics of individual entities are essentially confounders that cause the matching failure. Therefore, we propose a novel training-free compositional CLIP model (ComCLIP). ComCLIP disentangles input images into subjects, objects, and action subimages and composes CLIP’s vision encoder and text encoder to perform evolving matching over compositional text embedding and subimage embeddings. In this way, ComCLIP can mitigate spurious correlations introduced by the pretrained CLIP models and dynamically evaluate the importance of each component. Experiments on four compositional image-text matching datasets: Winoground, VL-checklist, SVO, and ComVG, and two general image-text retrieval datasets: Flick30K, and MSCOCO demonstrate the effectiveness of our plug-and-play method, which boosts the zero-shot inference ability of CLIP, SLIP, and BLIP2 even without further training or fine-tuning. Our codes can be found at https://github.com/eric-ai-lab/ComCLIP.

## introduction image and text matching @xcite is a fundamental task for visionlanguage research that involves multimodal reasoning and multi-level visual and text concept alignment. recently, a growing number of pretrained * equally contributed. ## compositional image and text matching we first introduce the task of compositional image and text matching, where we are interested in improving the compositional understanding, more specifically, subject/object/predicate understanding of vision-language models. compositional image and text matching is a task focused on enhancing the understanding of compositional elements such as subjects, objects, and predicates within clip-like models. this task requires an appreciation of fine distinctions between texts and their underlying compositional structure, as illustrated in figure 1 with phrases like "man/hitting/sign." the model's ability to differentiate images that only vary by one conceptual element in their accompanying text highlights its comprehension of compositionality. we formally define this task as follows: given text prompts y (e.g., "a man is hitting a baseball") and a set of entities t @xmath0, where k denotes the total number of entities and e k represents the k-th entity, the model's objective is to match the text prompts with the corresponding images. the challenge lies in the inclusion of negative images that contain mismatched entities {e k } n @xmath1, where @xmath2. these negative images are designed to confuse the model, demanding a nuanced understanding of the entities within a sentence. simply relying on nouns or spurious relations would not succeed at this task. to evaluate how well the model grasps this concept of compositionality in texts and matches them with the right images, we introduce an additional we propose comclip to incorporate a causal view into the clip-like models. we briefly introduce the background of comclip in view of structured causal models in section 4.1. then, we present the overview of comclip pipeline in section 4.2. we introduce its critical components in depth in section 4.3 and 4.4. our objectives are: (i) we aim at disentangling visual input into subimages containing fine-grained compositional concepts. (ii) we intend to utilize those disentangled concepts to perform entity-level matching dynamically and mitigate the effect of spurious relations in the pretrained vision-language models learned during training. ## conclusion in this work, we observe that clip-like model could struggle in situations that require object, subject, and verb/predicate understanding when performing compositional image and text matching. based on this observation, we propose a training-free method for compositional image and text matching from the causal view, mitigating the effect of spurious relations and improving compositional generalization. we also propose a new dataset to facilitate future research in this direction. our method is plug-and-play and could be applied to other vision-language pretrained model. we hope that our simple yet effective training-free approach could boost the development of more interpretable and principled methods for the compositional image and text matching task. this appendix is organized as follows: