SHAPE : S hifted Absolute Position Embedding for Transformers (2021)

Abstract: Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues. The basic idea of SHAPE is to achieve shift invariance, which is a key property of recent successful position representations, by randomly shifting absolute positions during training. We demonstrate that SHAPE is empirically comparable to its counterpart while being simpler and faster.

## introduction position representation plays a critical role in selfattention-based encoder-decoder models (transformers) @xcite , enabling the self-attention to recognize the order of input sequences. position representations have two categories @xcite : absolute position embedding (ape) @xcite and relative position embedding (rpe) @xcite . with ape, each position is represented by a unique embedding, which is added to inputs. rpe represents the position based on the relative distance between two tokens in the self-attention mechanism. rpe outperforms ape on sequence-to-sequence tasks @xcite due to extrapolation, i.e., the ability to generalize to sequences that are longer than those observed during training @xcite . @xcite reported that one of the key properties contributing to rpe's superior performance is shift invariance 2 , the property of a function to not change its output even if its input is shifted. however, unlike ape, rpe's formulation 1 the code is available at https://github.com/ butsugiri/shape . ## position representations figure 1 gives an overview of the position representations compared in this paper. we denote a source sequence x as a sequence of i tokens, namely, @xmath0, . . . , x i ). similarly, let y represent a target sequence of j tokens @xmath1, . . . , y j ). ## experiments using machine translation benchmark data, we first confirmed that transformer trained with shape learns shift invariance (section 3.2). then, we compared shape with ape and rpe to investigate its effectiveness (section 3.3). ## analysis this section provides a deeper analysis of how the model with translation invariance improves the performance. we hereinafter exclusively focus on ape and shape because shape achieves comparable performance to rpe, and we were unable to train rpe on the interpolate dataset as explained in footnote 9. as discussed in section 3.3, figure 3 demonstrated that shape outperformed ape in terms of bleu score. however, bleu evaluates two concepts simultaneously, that is, the token precision via n-gram matching and the output length via the brevity penalty @xcite . thus, the actual source of improvement remains unclear. we hereby exclusively analyzed the precision of token prediction. specifically, we computed tokenwise scores assigned for gold references, and we then compared them across the models; given a sequence pair (x, y ) and a trained model, we computed a score (i.e., log probability) s j for each token y j in a teacher-forcing manner. here, a higher score to gold token means better model performance. we used the validation set for comparison. figure 4 shows the ratio that shape assigns a higher score to a gold token than ape, compared across for each position of the decoder. better extrapolation means better token precision figure 4a shows that shape outperforms ape, especially in the right part of the heat map. this area corresponds to sequences longer than those observed during training. this result indicates that better extrapolation in terms of bleu score means better token precision. interpolation is particularly effective for rare tokens as shown in figure 4b , shape consistently outperforms ape and the performance gap is especially significant in the low-frequency region (bottom part). this indicates that shape predicts rare words better than ape. one plausible explanation for this observation is that shape carries out data augmentation in the sense that in each epoch, the same sequence pair is assigned a different position depending on the offset k. rare words typically have sparse position distributions in training data and thus benefit from the extra position assignment during training. ## conclusion we investigated shape, a simple variant of ape with shift invariance. we demonstrated that shape is empirically comparable to rpe yet imposes almost no computational overhead on ape. our analysis revealed that shape is effective at extrapolation to unseen lengths and interpolating rare words. shape can be incorporated into the existing codebase with a few lines of code and no risk of a performance drop from ape; thus, we expect shape to be used as a drop-in replacement for ape and rpe.