Dynamic Position Encoding for Transformers (2022)

Abstract: Recurrent models have been dominating the field of neural machine translation (NMT) for the past few years. Transformers have radically changed it by proposing a novel architecture that relies on a feed-forward backbone and self-attention mechanism. Although Transformers are powerful, they could fail to properly encode sequential/positional information due to their non-recurrent nature. To solve this problem, position embeddings are defined exclusively for each time step to enrich word information. However, such embeddings are fixed after training regardless of the task and word ordering system of the source and target languages. In this paper, we address this shortcoming by proposing a novel architecture with new position embeddings that take the order of the target words into consideration. Instead of using predefined position embeddings, our solution generates new embeddings to refine each word’s position information. Since we do not dictate the position of the source tokens and we learn them in an end-to-end fashion, we refer to our method as dynamic position encoding (DPE). We evaluated the impact of our model on multiple datasets to translate from English to German, French, and Italian and observed meaningful improvements in comparison to the original Transformer.

## introduction in statistical machine translation (smt), the general task of translation consists of reducing the input sentence into smaller units (also known as statistical phrases), selecting an optimal translation for each unit, and placing them in the correct order @xcite . the last step, which is also referred to as the reordering problem, is a great source of complexity and importance, which is handled with a variety of statistical as well as string-and treebased solutions @xcite . as evident in smt, the structure and position of words within a sentence is crucial for accurate translation. the importance of such information can also be explored in nmt and for transformers. previous literature, such as @xcite , demonstrated that source input sentences enriched with target order information have the capacity to improve translation quality in neural models. they showed that position encoding seems to play a key role in translation and this motivated us to further explore this area. since transformers have a non-recurrent architecture, they could face problems when encoding sequential data. as a result, they require an explicit summation of the input embeddings with position encoding to provide information about the order of each word. however, this approach falsely assumes that the correct position of each word is always its original position in the source sentence. this interpretation might be true when only considering the source side, whereas we know from smt @xcite that arranging input words with respect to the order of their target pairs can lead to better results. in this work, we explore injecting target position information alongside the source words to enhance transformers' translation quality. we first examine the accuracy and efficiency of a two-pass transformer (2pt), which consists of a pipeline connecting two transformers. the first transformer reorders the source sentence and the second one translates the reordered sentences. although this approach incorporates order information from the target language, it lacks end-to-end training and requires more resources than a typical transformer. accordingly, we introduce a better alternative, which effectively learns the reordered positions in an end-to-end fashion and uses this information with the original source sequence to boost translation accuracy. we refer to this alternative as dynamic position encoding (dpe). our contribution in this work is threefold: ## conclusion and future work in this paper, we first explored whether transformers would benefit from order signals. then, we proposed a new architecture, dpe, that generates embeddings containing target word position information to boost translation quality. the results obtained in our experiments demonstrate that dpe improves the translation process by helping the source side learn target position information. the dpe model consistently outperformed the baselines of related literature. it also showed improvements with different language pairs and dataset sizes.