Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality (2023)

Abstract: Contrastively trained vision-language models have achieved remarkable progress in vision and language representation learning. However, recent research has highlighted severe limitations of these models in their ability to perform compositional reasoning over objects, attributes, and relations. Scene graphs have emerged as an effective way to understand images compositionally. These are graph-structured semantic representations of images that contain objects, their attributes, and relations with other objects in a scene. In this work, we consider the scene graph parsed from text as a proxy for the image scene graph and propose a graph decomposition and augmentation framework along with a coarse-to-fine contrastive learning objective between images and text that aligns sentences of various complexities to the same image. We also introduce novel negative mining techniques in the scene graph space for improving attribute binding and relation understanding. Through extensive experiments, we demonstrate the effectiveness of our approach that significantly improves attribute binding, relation understanding, systematic generalization, and productivity on multiple recently proposed benchmarks (For example, improvements up to \mathbf{18} % for systematic generalization, \mathbf{16.5} % for relation understanding over a strong baseline), while achieving similar or better performance than CLIP on various general multimodal tasks.

## introduction recent progress in contrastive learning using largescale image-text data for joint image-text representation learning has led to vision-language models (vlms) like clip @xcite and align @xcite that show remarkable zeroshot classification and retrieval capabilities. however, recent works have shown that these models struggle at compositional reasoning (yuksekgonul † work done while at meta. et @xcite @xcite . in particular, they struggle with binding correct attributes to the correct objects, understanding relations between objects, generalizing systematically to unseen combinations of concepts and to larger and more complex sentences. some works have made progress on this problem. @xcite show that hard negative mining of images and text during fine-tuning is a promising first step to improving compositionality. however, performance gains are highly dependent on how clean the training data is, and generalizing to unseen combinations of concepts remains a challenge. @xcite use llms for hard negative mining and @xcite explore using synthetic datasets to improve compositional understanding in vlms. synthetic datasets lead to a domain gap compared to natural datasets. we aim to develop a general-purpose approach for improving compositionality of all such contrastively trained vlms. in this paper, we consider a scene graph representation of the image and text. we observe that multiple sub-graphs of the text scene graph with different semantic complexities can be matched with the same image. performing this matching improves fine-grained and hierarchical understanding of text and thereby, of images. we achieve this by developing a scene graph-based text decomposition strategy that creates a scene graph for any given text, decomposing it into sub-graphs, and matching an image to multiple sentences derived from these sub-graphs (see fig. 2 for an overview). each sub-graph represents a distinct part of the image, aligning well with clip's original imagetext matching objective. focused on improving attribute binding and relation understanding, we develop novel hard negative graph creation strategies which helps vl contrastive learning. we provide a novel image-to-multi-text contrastive loss for matching individual images to multiple sentences. our approach of matching texts of different complexity (from coarse-grained to fine-grained) to the image leads to fine-grained and hierarchical text understanding. our resulting model is mosaiclip. our approach leads to significant improvements across compositionality benchmarks. for example, figure 1 b ) and c) shows that mosaiclip improves performance by 11.5% and 9.1% on crepe and aro dataset over a strong baseline and @xmath0. our contributions encompass: visio-linguistic compositionality: various studies have introduced benchmarks for assessing the compositional reasoning abilities of vision-language foundation models (vlms). for instance, winoground @xcite is a handpicked collection of 400 test cases, each comprising two images and two sentences. sentences have the same word content and differ in word-order. @xcite show that the winoground dataset tests additional challenges along with compositionality, including handling ambiguous image-text pairs and unusual examples. @xcite proposed the aro benchmark for probing vlms ability to understand attribute, relations, and word-order. @xcite proposed crepe for measuring two aspects of compositionality: systematic generalization and productivity. all benchmarks suggest that contrastively trained vlms have severe difficulty in compositional reasoning. as a remedy, negclip @xcite and teaching svlc @xcite ) create targeted rule-based and llm-guided hard negative sentences, syvic @xcite fine-tunes clip with million scale synthetic images-text pairs, for improving relational and attribute understanding. we observe that previous methods are either highly dependent on how clean the training data is, use expensive llm's for data augmentation or use synthetic datasets that require special solutions to resolve the synthetic-to-real domain gap. we hence develop a coarse-to-fine contrastive learning framework that matches images with texts of multiple complexities, which serves as a general-purpose solution to improve fine-grained and hierarchical text understanding, thereby improving compositionality. scene graphs are structured representations of visual scenes, consisting of objects, their attributes, and relationships between objects. scene graphs are beneficial for a range of tasks including image retrieval @xcite , image captioning @xcite , and image generation @xcite among others. ## conclusion we present a method to improve the compositional reasoning capabilities of contrastively trained large vision-language models. in particular, we provide a coarse-to-fine contrastive learning framework and a scene graph-based text decomposition strategy for matching subgraphs of the text scene graph having varying complexity to an image during contrastive learning. we also develop hard negative graph creation strategies focused on improving attribute binding and relation understanding capabilities. our techniques leads to significant improvements in compositional reasoning capabilities. we investigate the reasons for improved compositionality and present a novel finding based on language encoder tree-scores, suggesting that our models learn improved fine-grained and hierarchical text understanding, which is likely the key reason for improved vision and language compositionality of mosaiclip as compared to baselines.