Recurrent Attention for the Transformer (2021)

Abstract: In this work, we conduct a comprehensive investigation on one of the centerpieces of modern machine translation systems: the encoder-decoder attention mechanism. Motivated by the concept of first-order alignments, we extend the (cross-)attention mechanism by a recurrent connection, allowing direct access to previous attention/alignment decisions. We propose several ways to include such a recurrency into the attention mechanism. Verifying their performance across different translation tasks we conclude that these extensions and dependencies are not beneficial for the translation performance of the Transformer architecture.

## introduction since its introduction by @xcite , the transformer architecture has enabled state of the art results on nearly all machine translation (mt) tasks @xcite @xcite . compared to previous neural machine translation (nmt) approaches @xcite , it introduces many new concepts like self-attention, positional encoding and multi-head attention. however, the transformer still relies on the encoder-decoder attention mechanism introduced by @xcite to translate a source sentence into the target language. while for earlier nmt models, this attention mechanism was thoroughly investigated and many different variants were proposed @xcite @xcite , the same can not be said for the transformer. in the present work, we discuss the transformer encoder-decoder attention mechanism, propose different ways to enhance its capabilities and analyze the resulting systems. one particular design decision in the transformer attention mechanism catches the eye: when calculating the context vector in the current decoding step, there is no direct information flow coming from the previous steps. while earlier neural architectures explicitly incorporated the hidden state from the previous decoding step in the attention calculation @xcite and traditional count-based alignment models used higher order markov assumptions, the transformer relies on the self-attention mechanism and layer stacking to learn context dependencies. therefore we ask the questions if and how an explicit dependency on the previous attention decisions should be included in the transformer encoder-decoder attention mechanism. in order to provide an answer we propose numerous approaches towards modeling such an explicit dependency and report our findings across three language pairs. ## related work in recurrent network architectures @xcite the decoder state recurrently depends on the previous decoding step. many works have extended this by additionally adding an explicit recurrent dependency within the attention mechanism itself. feng et al. ( 2016 ) concatenate the attention context produced in the previous decoding step to the input of the attention mechanism. other approaches approximate a coverage value for every source position by accumulating the attention weights over all previous time steps, which is then included in the attention calculation @xcite . @xcite extend this idea by renormalizing the coverage using a fertility model that predicts how much attention a specific source word should receive. in a similar spirit @xcite explicitly bias the attention weights to be more focused on source positions that did not receive much attention yet. in contrast to network architectures with a recurrent decoder, the transformer @xcite is trained completely parallel and uses multi-head, additive cross-attention. this work tries to answer whether introducing a recurrent dependency can also benefit the transformer cross-attention. ## experimental setup we evaluate our approaches on three tasks: the wmt 2016 news translation @xmath0, the wmt 2018 news translation @xmath1, as well as the iwslt 2017 @xmath2. our training data consists of 612k (@xmath3: se times, europarl v8), 208k (@xmath4: se times) and 227k (@xmath5: ted talk) parallel sentences, which we preprocess using 20k byte-pair-encoding operations (8k for @xmath6. we train a 6-layer transformer for each task, similar to the 'base'-configuration of @xcite . all models are implemented in returnn @xcite . we tie the weights of all embedding/projection matrices and apply a dropout of 20% for @xmath7. the baseline models use a batch size of 9600, however gpu memory limitations allow a batch size of maximum 7600 for some experiments that add a recurrency to the decoder. we select the best checkpoint according to bleu on the development set and report case-sensitive bleu calculated with sacrebleu @xcite and ter with tercom @xcite 5 experimental results ## conclusion in this work we provide a detailed analysis on the encoder-decoder attention mechanism in the transformer architecture. we argue that -compared to previous attention formulations -there does not exist a direct link to the context produced in the earlier decoding steps. we propose different approaches to explicitly model this link and test the resulting systems on three machine translation tasks. the results show no significant improvements for any of the tested approaches. this leads us to the conclusion that the context information which is incorporated through self-attention is already sufficient for the given task of machine translation.