An Anchor-based Relative Position Embedding Method for Cross-Modal Tasks (2022)

Abstract: Position Embedding (PE) is essential for transformer to capture the sequence ordering of input tokens. Despite its general effectiveness verified in Natural Language Processing (NLP) and Computer Vision (CV), its application in cross-modal tasks remains unexplored and suffers from two challenges: 1) the input text tokens and image patches are not aligned, 2) the encoding space of each modality is different, making it unavailable for feature comparison. In this paper, we propose a unified position embedding method for these problems, called AnChor-basEd Relative Position Embedding (ACE-RPE), in which we first introduce an anchor locating mechanism to bridge the semantic gap and locate anchors from different modalities. Then we conduct the distance calculation of each text token and image patch by computing their shortest paths from the located anchors. Last, we embed the anchor-based distance to guide the computation of cross-attention. In this way, it calculates cross-modal relative position embedding for cross-modal transformer. Benefiting from ACE-RPE, our method obtains new SOTA results on a wide range of benchmarks, such as Image-Text Retrieval on MS-COCO and Flickr30K, Visual Entailment on SNLI-VE, Visual Reasoning on NLVR2 and Weakly-supervised Visual Grounding on RefCOCO+.

## introduction transformer @xcite has shown excellent performance in natural language processing (nlp), computer vision (cv) as well as crossmodal tasks, including natural language inference @xcite , image classification @xcite , visual question answering @xcite and visual entailment @xcite , etc. nevertheless, transformer module lacks the capability to capture the ordering information of the input tokens * : equal contribution because of the limitation of its self-attention mechanism. therefore, incorporating explicit position representations is crucial to improve the performance of transformer-based models @xcite . generally, there are two mainstream position encoding methods in transformer-based nlp and cv models, i.e., absolute position embedding (ape) and relative position embedding (rpe). ape methods @xcite @xcite encode absolute positions of the input tokens with either trainable @xcite or fixed embedding @xcite . these position embeddings are added with the token embeddings, which are then passed to the selfattention layer to calculate the token relationship considering their positional information. it has been verified effective in a variety of nlp @xcite and cv @xcite tasks. on the other hand, rpe methods @xcite encode the pairwise distances of every two tokens. commonly, it directly interacts with the calculation of attention mechanism in different ways @xcite . compared with ape, rpe methods are superior to modeling the positional information of extremely long or variant-length sequences. as a result, in some span prediction tasks of nlp, rpe methods are shown to achieve more performance gains than ape ones @xcite . despite the success of the position embedding methods in unimodal tasks, its exploration in the field of cross-modal modeling is still limited. recent works on cross-modal tasks @xcite could be classified into two frameworks, 1) one-stage methods (fig. 1(a) ) which extract the cross-modal representation with a unified cross-modal encoder; 2) two-stage methods (fig. 1(b )), which have additional text encoder and image encoder. both of them adopt the position embeddings in a separate way, where the text and image position representations are embedded individually. in this way, the models can only learn position embedding in each modality separately while ignoring positional information between two tokens from different modalities. however, it is challenging to raise a unified method for cross-modal position embedding. firstly, the inputs from two modalities are embedded into different spaces, making the input embedding not comparable. secondly, since the text tokens and image patches are not aligned, the relative positions between two units from different modalities are meaningless. in this paper, we advocate a new perspective for effective cross-modal position encoding (shown in fig. 1(c )), called anchor-based relative position embedding (ace-rpe). it first computes alignment between text and image tokens to locate aligned pieces, which are called anchors in this paper. subsequently, the token-to-token (t2t) and patch-to-patch (p2p) relative position is calculated for unimodal ordering information. the relative position searching of arbitrary text token and image patch is then considered as a shortest path problem, containing three steps: 1) routing from given token and its nearby anchors; 2) routing from anchors and their located image patches, and 3) routing from the located patches to the given image patch. as illustrated in fig. 2 , the relative position of "a" and the image patch of the man is derived from three terms: the t2t relative position between "a" and the anchor "man", the relative position from anchor "man" to the image patch matching "man", and the relative position from the located image patch to the patch of human (obviously, 0 in this case). finally, we embed the anchor-based relative position to the self-attention calculation. further, we conduct extensive experiments to verify the effectiveness of the proposed ace-rpe compared to many strong baselines. the results demonstrate that our method can boost the performance of crossmodal transformers with a large margin. ## methods the overview of our backbone network is presented in fig. 3 , which contains a 6-layer visual transformer @xcite as the image encoder, a 6-layer linguistic transformer @xcite as the text encoder and a 6-layer crossmodal transformer. the anchor-based position embedding (ace-rpe) is proposed to leverage the cross-modal encoder with cross-modal positional information. it involves two key procedures: 1) learning the locating of cross-modal anchors; 2) ace-rpe calculation by incorporating anchor locating and t2t/i2i relative position. in this section, we first present the above procedures in detail (sec. 3.1 and sec. 3.2). then, we present the overall pre-training objectives of our method. a man in blue is sitting on a fake horse self-attention ffn self-attention ffn anchor locating flatten 6x 6x cross-attention mlp 6x ace-rpe itm mlm mim cross-modal encoder text / image encoder image [cls] text [cls] multi-group distance anchor detection t2t relative distance text tokens image patches i2i relative distance anchor-wise distance ace-rpe embedding tokenizer input text / image ## experiments in this section, we first provide numerical analyses of the proposed ace-rpe method compared with widely used baselines on 5 cross-modal tasks, including 6 benchmarks. then, we make a detailed ablation study to analyze the contribution of each component of the proposed ace-rpe method. ## conclusion in this paper, we present a cross-modal position embedding method, called ace-rpe, in which we first utilize an anchor locating method to learn to match the text words and the image patches. a man in blue is sitting on a fake horse. a man in blue is sitting on a fake horse the man is reading a newspaper while carrying umbrella an the man is reading a newspaper while carrying a umbrella. then, we compute physical distances between anchors and tokens from different modalities, which are applied for cross-modal fusion. we conduct comprehensive experiments to analyze the effectiveness of different components of ace-rpe as well as the performance under different modes and hyper-parameter settings. as we know, this work is the first to present position embeddings for cross-modal tasks, and the experimental results also demonstrate the superiority of our method.