Improve Transformer Models with Better Relative Position Embeddings (2020)

Abstract: The transformer model has demonstrated superior results on NLP tasks including machine translation and question answering. In this paper, we argue that the position information is not fully utilized in existing work. For example, the initial proposal of a sinusoid embedding is fixed and not learnable. In this paper, we first review the absolute position embeddings and existing relative position embedding methods. We then propose new methods to encourage increased interaction between query, key and relative position embeddings in the self-attention mechanism. Our most promising approach is a generalization of the absolute position embedding. Our method results in increased accuracy compared to previous approaches in absolute and relative position embeddings on the SQuAD1.1 dataset. In addition, we address the inductive property of whether a position embedding can be robust enough to handle long sequences. We demonstrate empirically that our relative embedding method can be reasonably generalized to and is robust in the inductive perspective. Finally, we show that our proposed method can be effectively and efficiently adopted as a near drop-in replacement for improving the accuracy of large models with little computational overhead.

## introduction the introduction of bert @xcite has lead to new state-of-the-art results on various downstream tasks such as question answering and passage ranking. variations of bert, including roberta @xcite , xlnet @xcite , albert @xcite and t5 @xcite have been proposed. at its core, bert is non-recurrent and based on self-attention; in order to model the dependency between elements at different positions in the sequence, bert relies on position embeddings. with bert, the input embeddings are the sum of the token embeddings, segment embeddings, and position embeddings. the position embedding encodes the absolute positions from 1 to maximum sequence length (usually 512). that is, each position has a learnable embedding vector. the absolute position embedding is used to model how a token at one position attends to another token at a different position. recent work suggested removing the next sentence prediction (nsp) loss with training conducted solely on individual chunks of text @xcite . in this setup, the notion of absolute positions can be arbitrary depending on chunk start positions. therefore, the association of a token to an absolute position is not well justified. indeed, what really matters is the relative position or distance between two tokens t i and t j , which is j -i. this phenomena has been realized and the relative position representation has been proposed in @xcite ; @xcite , in the context of encoder decoder machine translation and music generation respectively. @xcite has been modified in transformer-xl @xcite and adopted in xlnet @xcite . the relative position embedding in @xcite has been proven to be effective and thus it is adopted in @xcite . in this paper, we review the absolute position embedding from @xcite and the relative position embeddings in @xcite ; @xcite . our contributions are as follows. acquiring large gains with a small computation budget. in addition to the above work, @xcite proposed a novel method of adding location-awareness to the attention mechanism in the sequence to sequence framework for automatic speech recognition (asr). their work is related to this paper as both attempt to integrate a location information into the self-attention mechanism. ## position embeddings in this section, we review the absolute position embedding used in the original bert paper and the relative position embedding proposed in @xcite . we then propose a number of relative position embeddings, from simpler ones to more complex ones. we analyze the complexity of each embedding method. ## experiments we leverage the same data used to pre-train bert: bookscorpus (800m words) @xcite and english wikipedia (2.5b words) @xcite . following the setup from roberta @xcite , we leave out the next sentence prediction loss and only use one segment instead of the two segments proposed in bert @xcite during model training. we set the maximum input length to 512. similar to bert, we use a vocabulary size of 30k with wordpiece tokenization. we generate the masked input from mlm targets using whole word masking. the model updates use a batch size of 160 and adam optimizer with learning rate starting at 1e-4. our maximum batch size is 160 on an nvidia v100 instance (with 8gpus). following previous work @xcite @xcite , we evaluate on the general language un-derstanding evaluation (glue) benchmark @xcite and the stanford question answering dataset (squad1.1) @xcite . ## conclusion we proposed new relative position embedding methods to encourage more interactions between query, key and relative position embeddings in selfattention mechanism. our best proposed method is a generalization of the absolute position embedding and it leads to higher accuracy than the absolute and previous relative position embeddings on squad1.1. in addition, we demonstrated empirically that our relative embedding method is rea0 10 20 30 40 j 0 10 20 30 40 i 0.05 0.10 0.15 0.20 0.25 0.30 sonably generalized and robust from the inductive perspective. finally, we showed that our proposed method can be effectively and efficiently adopted as a drop-in replacement to boost the performance of large models with a small computational budget.