Interpretable Composition Attribution Enhancement for Visio-linguistic Compositional Understanding (2024)

Abstract: Contrastively trained vision-language models such as CLIP have achieved remarkable progress in vision and language representation learning. Despite the promising progress, their proficiency in compositional reasoning over attributes and relations (e.g., distinguishing between “the car is underneath the person” and “the person is underneath the car”) remains notably inadequate. We investigate the cause for this deficient behavior is the composition attribution issue, where the attribution scores (e.g., attention scores or GradCAM scores) for relations (e.g., underneath) or attributes (e.g., red) in the text are substantially lower than those for object terms. In this work, we show such issue is mitigated via a novel framework called CAE (Composition Attribution Enhancement). This generic framework incorporates various interpretable attribution methods to encourage the model to pay greater attention to composition words denoting relationships and attributes within the text. Detailed analysis shows that our approach enables the models to adjust and rectify the attribution of the texts. Extensive experiments across seven benchmarks reveal that our framework significantly enhances the ability to discern intricate details and construct more sophisticated interpretations of combined visual and linguistic elements.

## introduction the field of vision-language research has made great advancements in recent years @xcite @xcite . vision-language foundation models, such as clip, have exhibited remarkable performance across a broad range of wellestablished evaluation tasks @xcite @xcite , 2021), directly or indirectly fostering progress in numerous areas, such as text-to-image generation @xcite , video recognition @xcite and multi-modal large language models @xcite . despite these advances, a notable limitation still persists: vlms such as clip exhibit significant challenges in understanding visio-linguistic concepts beyond object nouns, in particular relations and attributes @xcite @xcite . specifically, they struggle with understanding relations between objects, and binding correct attributes to the correct objects. for example, as illustrated in fig. 1 (left), given an image and two similar textual descriptions (containing the same set of words but composed differently), such as "the car is underneath the person" and "the person is underneath the car", humans can effortlessly discern the contextual differences between the two sentences. however, vlms tend to struggle, highlighting a significant challenge in compositional reasoning @xcite . to further investigate the factors impeding the compositional understanding capabilities of vlms such as clip, we employ various model attribution techniques, such as attention-based and gradcambased methods @xcite , to analyze the attribution scores assigned by the model to object and non-object words when performing image-text matching. as shown in fig. 1 (right), our investigation reveals a consistent pattern across four different attribution scores: the attribution scores for object words are significantly higher than those for relation and attribute words. for example, the mean attribution score of object tokens is 0.244, which is two times than the relation tokens (0.111). this indicates that the model disproportionately emphasizes object words, neglecting fine-grained details such as relations and attributes in the text. this phenomenon aligns with the recent studies @xcite which argued the presence of shortcuts in contrastive learning pretraining. specifically, the models distinguish the correct image-text pairs from distinctly incorrect ones through simple object recognition, without the need to comprehend finer-grained details such as relations and attributes in the texts. in this work, we further identify that the primary issue for compositional understanding is the unfair attribution for relation and attribute words. we refer to this as the issue of composition attribution deficiency. however, the existing methods to improve visiolinguistic compositional understanding are not designed to adjust the attribution for different texts. @xcite introduces captions with perturbed word order and nearest neighboring images into each batch, to force models to distinguish correct and hard negative samples. @xcite use llms for hard negative mining and cascante-bonilla et al. (2023) explore using synthetic datasets to compose hard negative samples. regardless of the methods of hard-negative mining, existing methods do not endow the models with proportionate attribution across different texts, neglecting the attribution issues. inspired by our observation, we propose a novel framework, named cae (composition attribution enhancement), to enhance the compositional understanding of vlms without constructing any hard negative samples explicitly. specifically, in addition to a task-specific loss, cae adds a new loss that aligns the attribution scores distribution of different types of text tokens during the train-ing process. this encourages the model to pay more attention to fine-grained details (relations or attributes) within the text beyond object nouns. we propose four instances of our framework: attentionbased, gradcam-based, perturbation-based, and gradient-based attribution. in each instance, the model's compositional understanding abilities is naturally improved. furthermore, our approach can be easily integrated with hard negative samples, leading to additional performance gains. we summarize our contributions as follows: vision-language compositionality. despite the impressive advancements achieved in vlms, recent studies @xcite @xcite show that existing vl models exhibit limited compositional reasoning abilities. yuksekgonul et al. (2022) argue that image encoder (clip) text encoder (clip) 𝑇 𝐼 a person rides on the horse composition attribution enhancement (attention/gradcam/gradient score) a person rides on the horse 0.23 0.08 0.05 0.25 enhancement a person rides on the horse 0.16 0.15 0.10 0.19 attribution tracing figure 2: overview of our method. image-text contrastive learning learns shortcuts and does not learn enough compositional information such as relation and attribute. to address this limitation, existing approaches mostly investigate how to augment the text captions or images in contrastive learning to enhance the ability of compositional understanding (yuksekgonul et al., 2022; singh et al., 2023; doveh et al., 2024; zhang et al., 2024; doveh et al., 2023; sahin et al., 2024; cascante-bonilla et al., 2023). yuksekgonul et al. (2022) firstly proposed a simple and straightforward fix: mining hard negatives, which can improve the model's performance. basu et al. (2023) enhances clip's visio-linguistic reasoning via introducing a distillation objective from text-to-image generative models such as stable-diffusion. ## method our approach employs an attribution method to derive attribution scores on the text, subsequently optimizing these attribution scores to enhance the model's capability for compositional reasoning. preliminary: consider a training example consisting of an image i and its corresponding caption t. contrastive loss clip consists of a text encoder f t : t -→ r d and an image encoder f i : i -→ r d to encode image and text into embedding space r d separately. the image-text similarity score is computed as: the image-to-text contrastive loss l i2t and textto-image contrastive loss l t2i are formulated as follows: t ) t @xmath0,t i ) , (3) l @xmath1,t )∈b log exp s(i,t ) i @xmath2,t ) . (4) formulation: firstly, we utilize a widely-used text scene graph parser @xcite to parse the caption t, extracting the relations, attributes and objects present within the text. with little cost, this process effectively categorizes which tokens in t pertain to relations or attributes, and which pertain to objects. note the parsing process is only applied to the training samples. the trained clip is used in the same way as the original one. for a vlm such as clip, the attribution score a i for each token t i in the caption indicates the contribution or importance of each token to the output image-text similarity. a higher magnitude of a i signifies a greater importance of t i to the final output. given our knowledge of the positions of object tokens and relation/attribute tokens in the text, we can obtain the attribution scores for these tokens. for each sample, we derive the object attribution score a obj by averaging the attribution scores of all object tokens. similarly, we obtain the compositional attribution score a comp for each sample by averaging the attribution scores of all relation/attribute tokens. for the entire batch, we define the proposed cae introduces an extra learning objective l attr that optimizes the text attribution score to encourage the model to pay more attention to relation or attribute tokens. an intuitive approach is to make the two items as close as possible, an idea that is also reflected in @xcite . therefore, we define the attribution loss as follows: in the following subsections, we introduce four instances with different attribution types. ## experiments datasets. for training, we use the approximately 110k image-text pairs from mscoco @xcite given that its captions are less noisy and provide a more detailed description of the relation and attribute content in the images. for evaluation, we use aro @xcite , sugar-crepe @xcite , vl-checklist @xcite , winoground @xcite , valse @xcite , svo-probes (hendricks and nematzadeh, 2021) and comvg @xcite . the details of these seven datasets are in the appendix b. implementation detail. we used the popular vit-b/32 openai clip @xcite as our model in all the experiments using the open-clip repository @xcite . we finetune it for 5 epochs with a batch size of 256. we use a cosine schedule with an initial learning rate of 5e-7 and use 50 steps for warm up. adamw (kingma and ba, 2014) optimizer is used with a weight decay of 0.2. all the experiments are conducted on an nvidia tesla v100 gpu. for details on the computational budget, please refer to appendix c. baseline. our approach is mainly compared against two distinct baselines: (i) a pre-trained clip model; (ii) a clip model fine-tuned on mscoco utilizing only the contrastive loss, devoid of our proposed attribution optimization loss. it is imperative to emphasize that the second baseline, (ii), plays a critical role in mitigating the influence of image-text pairs derived from mscoco during the finetuning process. ## discussion orthogonal to hard-negative mining as we know, the three main components of machine learning are data, model, and loss function @xcite . from the perspective of machine learning, existing approaches can primarily be classified into two categories: data-driven and designing new loss functions. a categorization of related work is shown in the appendix j. our method falls under the category of loss function, while hardnegative mining methods fall under the category of data-driven. therefore, as different lines of research, these two approaches improve the model from different dimensions. the hard-negative mining methods focus on constructing hard negative samples in a similar way as the vision-language compositionality benchmarks, which have significantly advanced the field of visual-language compositional understanding. also, from a machine learning perspective, better data and better loss functions often enable models to learn more effectively @xcite . this could be one reason why our approach not only enhances the performance of the baseline model, but also combines with state-of-the-art hard-negative mining methods to further boost their performance. relationship with vilt @xcite ) vilt proposes a unified, efficient, and simplest model architecture for vision-and-language pretraining, sharing a similar spirit with clip. both of them tried to achieve cross-modal object-level alignment through general pre-training, i.e., clip applied the image-text contrastive learning, while vilt applied the image-text matching (itm) and word patch alignment (wpa). our method introduces a generic framework to encourage the model to pay greater attention to relation and attribute words, which is orthogonal to the model architectures and learning objectives. this means our method possesses the potential to be combined with vilt. it is also worth noting that vilt designs a word patch alignment algorithm, using the inexact proximal point method for optimal transports. this technique gives us new ideas to augment our approach further to design a new instantiation of our method. the transportation-based heatmap for a relation or attribute token can be used to represent composition attribution. ## conclusion in this work, we present an intuitive and novel method to enhance the composition attribution and the compositional reasoning ability of contrastive vision-language models such as clip. extensive experiments across variant attribution methods and seven benchmarks show the effectiveness of our method. our method can be easily integrated with existing hard-negative mining techniques to further boost the performance. we hope our methods can provide useful insights to solve the compositional understanding dilemma of vlms and improve the semantic representations of texts.