Learning Relation Alignment for Calibrated Cross-modal Retrieval (2021)

Abstract: Despite the achievements of large-scale multimodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the visual relation among the regions. The neglect of such relation consistency impairs the contextualized representation of image-text pairs and hinders the model performance and the interpretability. In this paper, we first propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. In response, we present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD and calibrate intra-modal self-attentions from the two modalities mutually via inter-modal alignment. The IAIS regularizer boosts the performance of prevailing models on Flickr30k and MS COCO datasets by a considerable margin, which demonstrates the superiority of our approach.

## introduction cross-modal retrieval, including image-text retrieval, video-text retrieval, etc., has long been an important downstream task in cross-modal representation learning. image-text retrieval (itr) aims at modeling the similarity of image-text pairs and recalling the most relevant one. it remains quite challenging due to the heterogeneity of the data and the semantic gap between two different modalities. to bridge this gap, neural networks are responsible for learning global representations of images a guy with a red shirt walking. denotes the self-attention stems from a query. the lower panel gives a bad case of inconsistent textual and visual relations. the region of "a red shirt" pays considerable attention to the region of the dog, which does not benefit the matching and is inconsistent with the self-attention of the corresponding phrase. and texts in a joint semantic space and aligning the images and texts with the same semantics @xcite . a straightforward way to enhance the alignment is to enforce the local matching between the object-oriented words and the corresponding image regions, and then leverage the object co-occurrence statistics @xcite in the pairs for inference. previous studies incorporate auxiliary knowledge source like scene graphs @xcite or object tags @xcite to explicitly indicate the cross-modal mapping. other researches try to establish fine-grained interaction on cross-modal attention to reinforce the focus from words to their most relevant regions, and vice versa @xcite @xcite @xcite . however, such word-region alignment at object level serves only as the basis because it mainly focuses on the local semantics but lacks the matching of global features like the intra-modal relation. the intra-modal relation refers to the correlation of items within a textual or visual sequence. more specifically, given a sentence and an image that describe the same scene and are highly matched, the correlation of the items in the textual sequence should also agree with the correlation of the corresponding items in the visual sequence. but such constraint of relation consistency is neglected in previous works, which hinders performance and interpretability of the models. to corroborate this, we conduct a case study on flickr30k entities dataset @xcite to probe the agreement of relation-level semantics in pre-trained models like uniter @xcite . we utilize the self-attention distribution as a representation of the intra-modal relations @xcite @xcite . as shown in figure 1 , the attention distributions grouped by the annotated object of the given text and image are in disagreement with each other. specifically, the attention distribution in the linguistic modality is reasonable. however, in the visual modality, the region "a red shirt" pays inappropriate attention to the region of the dog that doesn't appear in the text, which impairs the representation of this visual item, i.e., "a red shirt" under the condition of the corresponding text. such mismatched attention distributions suggest that the model represents the same concept with inconsistent semantics, which misleads the model to reduce the estimated similarity of the positive pairs and further leads to the wrong predictions that they are unmatched. what's even worse is that in practice, the input regions of the existing methods are extracted by a pre-trained object detector like faster r-cnn @xcite . the visual features are much noisier due to over-sampling @xcite , which necessitates a stronger regularizer to guide the alignment of the intra-modal relations. motivated by the above observations, we pro-mote the semantic alignment from object level to relation level. we leverage self-attention matrix to characterize the relation of items within one modality, and design intra-modal self-attention distance (isd), a novel metric to measure the consistency between textual and visual relations. our empirical analysis illustrates that the isd and the model performance on image-text retrieval are highly correlated, which verifies our hypothesis and inspires us to minimize the semantic distance between intramodal self-attentions in training. accordingly, we propose a new regularized training method called inter-modal alignment on intra-modal selfattentions (iais) to calibrate two intra-modal attention distributions mutually via inter-modal alignment, which helps learn better contextualized representations for image-text pairs. the model performance of image-text retrieval on flickr30k and ms coco datasets is improved by a considerable margin with iais, which demonstrates the superiority of our proposal. ## measuring semantic distance between intra-modal relations in this section, we present a formal definition of intra-modal relation alignment (section 2.1). such alignment requires extracting the visual and linguistic items corresponding to all objects and sorting them in the same order to make their self-attention distributions comparable. we first introduce the mechanism for multimodal attention calculation, and then present the method of attention weight extraction for constructing comparable intra-modal self-attentions (section 2.2). finally, we propose a metric named intra-modal self-attention distance (isd) to quantify the relation consistency. we conduct an empirical analysis on prevailing models to verify the correlation of the model performance and our metric (section 2.3). ## inter-modal alignment on intra-modal self-attentions (iais) in this section, we propose a new regularized training method, inter-modal alignment on intra-modal self-attentions (iais), for image-text retrieval. our goal is to enhance the semantic alignment of relations by minimizing the distance between two intra-modal self-attentions (isd). with the scattered items, foot_3 there are no object annotations and the region features extracted by faster r-cnn are much noisier @xcite , which results in difficulty in grouping the attention weights by ground-truth object. the isda thus cannot be used directly as the objective function to minimize. to tackle this problem, we regard the input sequence from one modality (e.g., the visual sequence v) as an anchor. for every item in the anchor sequence, we extract its corresponding representation from the other modality (e.g., one item or a collection of items in the linguistic sequence l) to reconstruct a mirrored sequence. after that, the items and their relations within the anchor sequence have a one-to-one correspondence with the items and relations within the mirrored sequence, which makes the intra-modal self-attentions derived from the two sequences comparable. in the next two subsections, we propose two methods, singular alignment and distributed alignment, to accomplish the attention extraction and reconstruction. the former establishes a one-to-one mapping between linguistic and visual attention weight, while the latter establishes a distributed mapping. besides, we design two losses l (s) iais and l (d) iais as a surrogate of the isda to measure the semantic distance between intra-modal self-attention matrices. finally, we incorporate the surrogate loss minimization as a regularization to calibrate intra-modal self-attentions mutually and achieve the relation-level alignment. ## related work in this section, we introduce the task of image-text retrieval and review the representative studies of image-text retrieval image-text retrieval (itr, @xcite , also known as image-text matching, is one of the popular and challenging languageand-vision (v+l) tasks. given image-text pairs, the prevailing approaches project them into a joint representation space, on which cosine or dot-product similarities are defined, and recall the most relevant one according to the similarity. ## conclusion in this paper, we promote the semantic alignment for cross-modal retrieval from the object level to the relation level. we propose a surrogate metric to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. furthermore, we present a regularized training method iais to calibrate intra-modal selfattentions mutually by minimizing the isd metric. our method improves both the performance and the interpretability of large-scale pre-trained models. note that, without object annotation in practice, the singular and distributed version of the iais loss only provides a coarse-grained attention distribution alignment. we leave the elaborate design of isda proxy function for future work.