Bridging the Gap between Position-Based and Content-Based Self-Attention for Neural Machine Translation (2023)

Abstract: Position-based token-mixing approaches, such as FNet and MLPMixer, have shown to be exciting attention alternatives for computer vision and natural language understanding. The motivation is usually to remove redundant operations for higher efficiency on consumer GPUs while maintaining Transformer quality. On the hardware side, research on memristive crossbar arrays shows the possibility of efficiency gains up to two orders of magnitude by performing in-memory computation with weights stored on device. While it is impossible to store dynamic attention weights based on token-token interactions on device, position-based weights represent a concrete alternative if they only lead to minimal degradation. In this paper, we propose position-based attention as a variant of multi-head attention where the attention weights are computed from position representations. A naive replacement of token vectors with position vectors in self-attention results in a significant loss in translation quality, which can be recovered by using relative position representations and a gating mechanism. We show analytically that this gating mechanism introduces some form of word dependency and validate its effectiveness experimentally under various conditions. The resulting network, rPosNet, outperforms previous position-based approaches and matches the quality of the Transformer with relative position embedding while requiring 20% less attention parameters after training.

## introduction the transformer @xcite revolutionized the field of neural machine translation before its wide adoption in numerous other tasks @xcite @xcite . using selfattention @xcite , the transformer computes high-level representations for each token as a weighted sum of the entire sequence, where the weights depend on the pairwise content interactions. however, recent work argues that results similar to the transformer can also be achieved by modeling self-attention weights based on positional instead of content information @xcite @xcite @xcite . often, these position-based methods are used with some form of gating mechanism that precedes or wraps the token-mixing operation @xcite @xcite . position-based self-attention alternatives often speed up the computation on commercial computing devices like gpu, but they can become more attractive from the perspective of using memristive crossbar arrays @xcite . recent advances in analog in-memory computation with memristive crossbar arrays have shown impressive efficiency improvements in the inference of deep learning models @xcite @xcite @xcite , up to 110 times better energy efficiency and 30 times better performance density compared to a tesla v100 gpu @xcite . however, such efficiency is obtained by storing weights of matrix-vector multiplications in the device rather than calculating them on the fly, which excludes the possibility of using attention to compute the weight matrix. with the goal of finding self-attention alternatives for machine translation that can be more easily used with memristive crossbar arrays, we compare existing position-based approaches and observe a significant quality loss when they use no form of gating. additionally, by scoring with a diverse set of metrics, we show that, even with gating, no existing approach can consistently match transformer results. while the role of gating to guide the information flow of neural networks is known @xcite , its importance for the performance of position-based approaches has yet to be explored. in this paper, we propose aposnet and rposnet, two position-based networks that leverage gating and compute self-attention weights based on the interactions of absolute and relative position representations. both differ slightly from the transformer baseline with relative position embeddings @xcite , which enables us to deliver insights into gating and its dependency on position information. in summary, we provide the following contributions: our gating reformulation suggests one probable reason, but we leave detailed investigations for future work (section 7.2). ## background neural machine translation is typically modeled with an encoder-decoder sequence-to-sequence @xcite transformer, which mainly consists of multi-head attention and feed-forward sub-layers. in the following, we introduce our notation, position-based token-mixing alternatives and the gating mechanism commonly used in modern architectures. ## reformulating the gating mechanism to better understand the implications of gating, we reformulate equation 4. we omit layer normalization for simplicity and will show in the appendix b that the general reformulation is unaffected if we apply layer normalization on v m . additionally, we leverage the gelu approximation σ g pv m q « v m σ s p1.702v m q, where σ s refers to the sigmoid function, and rewrite equation 4 as equation 5 shows that gating the context vector introduces the latent weights β nm p r d : in other words, the ratios of token-mixing weights for a query y n as computed by β are predetermined by the ratios across β 1 1...m . while we show in section 6 that this limitation is not problematic for self-attention, it may be part of the reason gating and relative position-based attention are not effective in cross-attention (see section 7.2). ## position-based attention in this section, we propose position-based attention, which determines the token-mixing weight connecting tokens x m and y n solely based on the position-position interactions between n and m. we pair position-based attention with the gating mechanism of section 2.3. ## results we compare translation quality of the base model configurations in ## related work the question of how to represent position and integrate it into the transformer architecture has been a vast research field that we briefly want to overview and connect to our approach. an extensive line of research focuses on improving position embeddings @xcite @xcite and their integration into the word vectors @xcite . this direction is mainly orthogonal to our approach, and many ideas and methods can be leveraged with position-based attention. we leave these investigations for future work and restricted to learnable @xcite and sinusoidal @xcite embeddings. a different line of research focuses on integrating position within the attention mechanism @xcite @xcite @xcite @xcite . they all improve over transformer models for various tasks by modifying word and position interactions within the attention matrix and introducing relative position representations as a scalar or vector. while they still rely on content-dependent attention weights, they showed the importance of relative position representations, which we also used in rposnet. however, we are interested in studying purely position-based self-attention approaches and how they can perform at least on par with the (content-based) transformer. additionally, we compare with @xcite as an upper bound since it leverages token-token interactions and was proposed for machine translation. ## conclusion we have introduced the gated token-mixing approaches aposnet and rposnet in order to find a high-quality self-attention alternative for machine translation whose attention weights can be pre-computed at inference time. although their token-mixing weights are position-based, the gating mechanism introduces content dependency in the form of latent weights β, as shown by our analysis. these weights capture token-token interactions and are crucial for the results of rposnet. in our experiments, we have compared aposnet and rposnet with existing position-based token-mixing approaches and found that rposnet outperforms all the position-based alternatives and performs on par with @xcite on most benchmarks while saving more than 20% of the self-attention parameters. moreover, the possibility of pre-computing rposnet's token-mixing weights paves the way for high-quality machine translation on specialized hardware accelerators.