Explore Better Relative Position Embeddings from Encoding Perspective for Transformer Models (2021)

Abstract: Relative position embedding (RPE) is a successful method to explicitly and efficaciously encode position information into Transformer models. In this paper, we investigate the potential problems in Shaw-RPE and XL-RPE, which are the most representative and prevalent RPEs, and propose two novel RPEs called Low-level Fine-grained High-level Coarse-grained (LFHC) RPE and Gaussian Cumulative Distribution Function (GCDF) RPE. LFHC-RPE is an improvement of Shaw-RPE, which enhances the perception ability at medium and long relative positions. GCDF-RPE utilizes the excellent properties of the Gaussian function to amend the prior encoding mechanism in XL-RPE. Experimental results on nine authoritative datasets demonstrate the effectiveness of our methods empirically. Furthermore, GCDF-RPE achieves the best overall performance among five different RPEs.

## introduction recently, the fully attention-based transformer model @xcite has achieved stateof-the-art results across a range of natural language processing (nlp) tasks, including reading comprehension @xcite , machine translation @xcite , natural language inference @xcite , unsupervised pretraining @xcite @xcite , etc. since the self-attention blocks in vanilla transformer are entirely invariant to sequence order, which is one of the most important features of natural language, how to explicitly encode position information is crucial for the current transformer based models. the original method is to use absolute position embedding (ape), such as pre-defined sinusoidal functions @xcite or fully data-driven learnable parameter embeddings @xcite , to integrate position information into contextual representation. although ape can significantly help the transformer model learn the contextual representation of the tokens at different positions, @xcite pointed out that the coupled method in ape is unreasonable. besides, ape itself also has many defects, such as the limitation of processing long sequences @xcite and the gradual loss of position information @xcite . to address the drawbacks mentioned above of ape, @xcite ; @xcite further proposed the relative position embedding (rpe), which incorporates carefully designed temporal bias term into the self-attention module to encode the relative distance between any two tokens. rpe has been proven to be more effective than ape, and thus it is adopted by many excellent pre-trained language models @xcite @xcite . despite the success of rpe, the existing methods are not perfect. although @xcite has made improvement to rpe, this improvement is only focused on the perspective of interaction rather than the perspective of encoding foot_0 . moreover, to the best of our knowledge, there is currently no unified and comprehensive evaluation of various rpes. since almost every rpe is proposed for specific tasks, it is unknown whether these rpes really have high universality and generalization ability. in this paper, we focus on the most widely adopted shaw-rpe @xcite and xl-rpe @xcite , and improve each of them from encoding perspective. concretely, for shaw-rpe, to overcome its weak ability to perceive the relative position at medium and long distance, we design an ingenious low-level fine-grained high-level coarse-grained (lfhc) embedding strategy without changing the number of parameters. for xl-rpe, we recognize the potential problems of its prior sinusoidal encoding functions under the relative position setting and propose a more reasonable encoding mechanism based on the gaussian cumulative distribution function (gcdf). we conduct a unified evaluation of five rpes on nine authoritative datasets, including language modeling, question generation, and text classification. the experimental results show that both lfhc-rpe and gcdf-rpe outperform their respective baseline, and gcdf-rpe achieves the best overall performance among the five methods 2 . (a) xl-rpe (b) gcdf-rpe figure 2: the prior encoding matrix of xl-rpe and gcdf-rpe. the horizontal axis represents dimension and the vertical axis represents relative position. ## experiments in this section, we evaluate the performance of five different rpes (t5 @xcite foot_5 , shaw, lfhc, xl, gcdf) on text classification, question generation and language modeling. ## conclusion and future work in this paper, we explore better rpes from encoding perspective for transformer models. for pure data-driven rpes, we propose lfhc-rpe to strengthen the sensitivity at medium and long relative positions. for rpes based on prior encoding mechanisms, we present gcdf-rpe with stronger generalization. extensive experimental results on nine datasets show the effectiveness of our methods. we leave adjusting our methods to different kinds of pre-trained language models as our future work.