R el CLIP : Adapting Language-Image Pretraining for Visual Relationship Detection via Relational Contrastive Learning (2022)

Abstract: Conventional visual relationship detection models only use the numeric ids of relation labels for training, but ignore the semantic correlation between the labels, which leads to severe training biases and harms the generalization ability of representations. In this paper, we introduce compact language information of relation labels for regularizing the representation learning of visual relations. Specifically, we propose a simple yet effective visual Relationship prediction framework that transfers natural language knowledge learned from Contrastive Language-Image Pre-training (CLIP) models to enhance the relationship prediction, termed RelCLIP. Benefiting from the powerful visual-semantic alignment ability of CLIP at image level, we introduce a novel Relational Contrastive Learning (RCL) approach which explores relation-level visual-semantic alignment via learning to match cross-modal relational embeddings. By collaboratively learning the semantic coherence and discrepancy from relation triplets, the model can generate more discriminative and robust representations. Experimental results on the Visual Genome dataset show that RelCLIP achieves significant improvements over strong baselines under full (provide accurate labels) and distant supervision (provide noise labels), demonstrating its powerful generalization ability in learning relationship representations. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/RelCLIP.

## introduction visual relationship detection needs to predict the relation label between a pair of localized objects (e.g. "man carrying bag"). based on such relationships we can construct a structural representation (i.e., scene graph) that regards the visual concepts within a scene as a whole and could benefit many downstream reasoning tasks, such as image retrieval (qi * equal contribution. † corresponding author. et @xcite , vision question answering @xcite , and visual commonsense reasoning @xcite . existing visual relationship detection models @xcite @xcite learn relation representations based on the visual information of object pairs under the supervision of numeric ids of relation labels. the learned representations often suffer from the highly changeful visual appearances of the instances from the same relation category as well as the fine-detailed visual difference between relation classes. as shown in fig. 1 , the representation of the sample "girl carrying surfboard" is very close to that of "man riding surfboard" in the visual embedding space, which may cause undesirable misclassification of relations. to alleviate this, we introduce language information of relation labels which is more compact than visual information to capture the semantic correlations among the labels, and regularize the representation learning of visual relationships via exploring visual-semantic alignment at relation level. by do-ing this, we can learn more robust and discriminative relationship representations. specifically, we propose a relationship detection framework that adapts contrastive language-image pretraining (clip) models @xcite to enhance the visual-semantic alignment during relationship learning, termed as relclip. the clip model is pre-trained on 400 million image-text pairs harvested from the web, and is proved to be good at image-level visual-semantic alignment. during the representation learning of clip, the objects within an image are considered to be independent, and it is difficult to capture the interaction between objects, e.g., visual relationships, so clip models can not be directly applied to visual relation detection. for example, as shown in fig. 2 , clip features only focus on individual objects but fail to activate the area that indicates the relationship between two objects. in contrast, our relclip can effectively focus on the interactive regions between objects that intuitively indicates the relations. to achieve relation-level visual-semantic alignment, we develop a novel relational contrastive learning (rcl) approach to match the visual and language embeddings of relationships, since naively using existing contrastive learning methods @xcite @xcite for relationship triplets (e.g. subject-predicate-object) may cause two severe issues. first, directly comparing the triplets may lead to trivial comparisons among all possible combinations of labels of subject, object and predicate, which is quite inefficient and greatly harms the robustness of the model. second, the synonymous relationships (e.g., "man riding bike" and "person riding bike") and the less informative negative ones (e.g. "man riding bike" and "bird sitting on branch") may be contrasted inappropriately and inefficiently. to address these issues, our rcl 1) decouples the triplet level comparison into dual contrastive objectives which compare the same relation instance in subject-predicate and predicateobject levels to reduce the amount of comparisons; 2) adopts a new semantic-aware active sampling strategy that excludes synonymous relationships and includes informative negative samples according to their semantic meaning. extensive experiments on the visual genome dataset show that relclip achieves significant improvements over strong baselines trained using whether human-annotated labels from full super-relclip (ours) person riding horse clip food sitting on table figure 2 : grad-cam visualization of relational activation. cross-modal pretraining models (e.g., clip) are powerful in image-level visual-semantic alignment but fail to capture the relationship between objects. in contrast, relclip can activate the interaction area between objects (e.g., "riding" and "sitting on") and achieve relation-level visual-semantic alignment. ## conclusion in this work, we develop relcilp, a simple yet effective visual relationship detection framework that successfully adapts cross-modal pre-training for improving relationship discrimination. to achieve this, we propose relational contrastive learning (rcl) that not only enables efficient comparisons via decoupling triplet-level contrast into subjectpredicate and predicate-object levels, but also ensures effective comparisons via a novel semanticaware sampling strategy. by collaboratively learning the semantic coherence and discrepancy from relationship instances, the model can generate more discriminative and robust relationship representations. our relclip significantly outperforms existing methods under both full and distant supervision, demonstrating the effectiveness and generalization ability of relational contrast powered by cross-modal pretraining models.