Contrastive Visual and Language Learning for Visual Relationship Detection (2022)

Abstract: Visual Relationship Detection aims to understand real-world objects’ interactions by grounding visual concepts to compositional visual relation triples, written in the form of (subject, predicate, object). Previous works have explored the use of contrastive learning to implicitly predict the predicates from the relevant image regions. However, these models often directly leverage in-distribution spatial and language co-occurrences biases during training, preventing the models from generalizing to out-of-distribution compositions. In this work, we examine whether contrastive vision and language models pre-trained on large-scale external image and text dataset can assist the detection of compositional visual relationships. To this end, we propose a semi-supervised contrastive fine-tuning approach for the visual relationship detection task. The results show that fine-tuned models that were pre-trained on larger datasets do not yield better performance when performing visual relationship detection, and larger models can yield lower performance when compared with their smaller counterparts.

## introduction understanding the visual world is essential for many modern computer vision tasks, including visual question answering (vqa) @xcite , image captioning @xcite , and human-robot interaction @xcite . given the complexity of real world scenes, low-level object classification and recognition tasks are insufficient to solve these problems. instead, higher-level visual understanding and reasoning tasks are often required. visual relationship detection (vrd) aims to facilitate such understanding by bridging the gap between low-level visual information and high-level symbolic visual relation. previous work @xcite on vrd emphasized the use of spatial priors in the form of overlapping bounding boxes and language co-occurrence bias during training to achieve higher benchmark results. however, such explicit incorporation of priors during training often leads to biases that might not transfer to new visual relationship @xcite . to address these issues, current research leverage external linguistic commonsense knowledge from structured knowledge bases @xcite or unstructured natural language corpora @xcite by distilling the information down into low-dimensional distributed vector embeddings. these embeddings indirectly benefit the visual relationship detection task as they preserve the relevant topological structure from the knowledge base or the linguistic contextual information from raw language corpora. in this work, we focus on the later and examine whether contrastive embedding models learned from the abundant amount of unstructured image and text pairs from the web (see to this end, we examined the promise of robustness in the large pre-trained image and text deep neural encoders @xcite , focusing on learning joint visual and language embeddings for vrd. the present paper investigates the application of contrastive learning in this context. contrastive learning aims to learn representations by pulling together the matching (or positive) vector pairs, while pushing apart non-matching (or negative) vector pairs. we believe that such a contrastive approach, when paired with learning @xcite , will improve the joint representations and indirectly benefit the vrd task. thus, this work proposes a fine-tuning framework that extends existing vision and text encoders to classify predicates from the given ground truth object regions and object labels. the results obtained show that the amount of data used in the pre-training process do not necessarily impact the final performance of the vrd predicate classification (vrd-predcls) task when evaluated on models of the same size. ## related work this section presents a review of the work related to compositional grounding of visual concepts on language @xcite , with an emphasis on visual relationship detection through the use of contrastive learning. visual relationship detection aims to construct a symbolic representation from an unstructured scene by representing it as a set of visual relationship triples in the form of (subject, predicate, object) (e.g (person, riding, horse)). here, the subject and object are labels grounded to the salient image regions through bounding boxes, and the predicate is defined as a real-world interaction between these object pairs. however, due to the large number of potential real-world interactions, existing visual relation datasets including vrd @xcite and visual genome @xcite are often sparse and unbalanced, where common relations occur more frequently than rarer but plausible ones. to tackle this problem, existing work seek to incorporate linguistic knowledge as additional training features to the learning model. for example, @xcite have shown that leveraging pre-trained word embeddings can help the models learn linguistic statistical priors; similarly, @xcite show that distilling knowledge from external wikipedia datasets can improve the model's performance. other work have also directly targeted the bias nature of the benchmark by incorporating spatial information @xcite , and (subject, object) co-occurrence priors as learning features for the classifiers @xcite , improving the model's performance significantly. recent approaches have also tackled the effect of bias by using model-agnostic counterfactual prediction during inference such as total direct effect (tde) @xcite or by experimenting with different sampling strategies @xcite . in contrast, the present work does not use any predefined co-occurrence statistics, or spatial information, as learning features at the same time that it does not learn a neural discriminative classifier that directly predicts the predicate label from the input features. instead, we apply existing contrastive encoder-encoder architecture to construct visual and text embeddings that can be ranked using a cosine similarity scoring metric. datasets: visual genome @xcite is the most used dataset for vrd. it is composed of 108k images that have been labeled with 150 object and 50 predicate classes. the labeling strategy of visual genome is simple: first, volunteers were asked to provide captions for regions in images; then, these captions were transformed into (subject, predicate, object) triples. this strategy allowed to build a large-scale dataset with an average of 21 relations per image but at the cost of introducing various biases such as asymmetric relations (e.g. (hair, on, man) versus (man, has, hair)) or confusion between predicates (e.g. (man, wearing, shirt) and (man, wears, shirt)). visual genome and other datasets used for visual relationships detection such as vrd @xcite , suffer from a long-tail distribution of the predicates, as recurrently stated in previous work @xcite @xcite . nonetheless, visual genome has become the most used benchmark for vrd, mainly because of the large scale and diversity of its annotated data. contrastive learning aims to minimize a defined distance metric between the matching or positive embedding vector pairs while maximizing the distance between non-matching or negative embedding vector pairs. recent work on contrastive learning have shown that such discriminative learning approaches can (i) learn to ignore invariant features and spurious correlations through data augmentation and automatic negative sampling technique @xcite , and (ii) learn joint visual and language embeddings that can be used to perform zero-shot triple detection on a wide variety of tasks @xcite . in this work, we focus on use case (ii) which aims to transfer pretrained image and text embeddings to the visual relationship detection task. to this end, we built on top of an existing body of work on zero-shot transfer and multi-modal representational learning, with an emphasis on clip @xcite , a contrastive learning model that encodes image and text using abundant (image, caption) pairs from the internet. we believe that such contrastive encoder-encoder model gives a clearer separation of the visual embeddings and language embeddings compared to the traditional black-box neural fusion approaches @xcite , giving us more control over both the triples input and the final output embedding spaces. ## contrastive image and text matching this section describes the proposed architecture and outlines the details of the current implementation. the general architecture consists of three modules: (1) the visual module generates visual embeddings based on the extracted features from the (subject, object) union image regions, (2) the language module generates text embeddings of the concatenated (subject, predicate, object) string, and (3) the contrastive loss module that consists of the visual-language contrastive losses ensuring the consistency between the matching (visual, language) embedding pairs, while pushing apart the non-matching (visual, language) embedding pairs. here, we use the cosine similarity metric as the distance metric and the cross entropy loss as our main loss objective. ## results this section presents an evaluation of the performance of different fine-tuned openclip (wortsman et al., 2022) models on the visual genome dataset, from which 60,784 images were used for training and 26,466 images for testing. results marked with 1 have been computed by retraining the models using the pytorch implementation from @xcite . since the commonly used recall metric gives a biased view toward common relationships, we also present a visualization of how models of different sizes perform on individual predicate classes in figure 2 . ## conclusion visual relationship detection is the cornerstone of many modern machine learning tasks that require a comprehensive understanding of the visual scene. in this work, we investigated whether large neural encoders pre-trained on a large set of data could capture the natural web image-text pair distribution @xcite and assist the detection of visual relations. to this end, we proposed a contrastive fine-tuning technique that leverages this capability to perform visual relationship detection. while the results in moreover, based on results of the tests reported in this work, the success of contrastive learning techniques was highly dependent on the quality of the positive and negative sampling examples, and there was a limit to what automated sampling tech-niques can do without human intervention. without high-quality samples, these deep neural networks still learned spurious correlations and patterns, making it difficult for the models to generalize beyond the given domain. thus, future work may further explore better negative sampling and data augmentation techniques that incorporate external taxonomies or knowledge bases to avoid biases and spurious correlations stemming from the skewed training data distribution. finally, the test-time inference algorithm in this work becomes more expensive as the number of predicates p increases since the evaluation technique used measures the distance between the given image embedding and all textual embeddings (subject, predicate @xmath0,p) , object). however, this effect can be mitigated by pre-computing all sentence embeddings from the dataset and storing them in memory or in the file system. still, such approach may suffer from the i/o bottleneck where the embeddings have to be loaded into memory prior to computation. future work may also explore alternative strategies to leverage the generated embeddings in predicate prediction tasks.