Transformer with Syntactic Position Encoding for Machine Translation (2021)

Abstract: It has been widely recognized that syntax information can help end-to-end neural machine translation (NMT) systems to achieve better translation. In order to integrate dependency information into Transformer based NMT, existing approaches either exploit words’ local head-dependent relations, ignoring their non-local neighbors carrying important context; or approximate two words’ syntactic relation by their relative distance on the dependency tree, sacrificing exactness. To address these issues, we propose global positional encoding for dependency tree, a new scheme that facilitates syntactic relation modeling between any two words with keeping exactness and without immediate neighbor constraint. Experiment results on NC11 German→English, English→German and WMT English→German datasets show that our approach is more effective than the above two strategies. In addition, our experiments quantitatively show that compared with higher layers, lower layers of the model are more proper places to incorporate syntax information in terms of each layer’s preference to the syntactic pattern and the final performance.

## introduction over the past few years, end-to-end neural machine translation (nmt) has shown remarkable progress, achieving promising results on various machine translation tasks. although nmt models can perform well even only trained on parallel corpus, they have been found to still benefit from external linguistic knowledge (e.g., lexical analysis and parsing) @xcite , and the same finding was also observed in other tasks @xcite . therefore, several studies have started to to investigate how to incorporate dependency information into nmt based on transformer, which is currently the most advanced end-to-end machine translation backbone. there exist two broad categories of approaches to integrating a sentence's dependency structure into the transformer. the first category of approaches @xcite @xcite takes the dependency tree as a general graph structure data. a graph neural network or a sublayer with a similar mechanism was introduced at the bottom of the transformer's encoder to represent and encode the dependency relations among the words in the sentence. in this paradigm, however, a word focuses only on its immediate neighbors and ignores nodes that are multi-hop away from it, which may carry important context that helps disambiguate and enrich the current word's representation in terms of syntactic structure. in contrast, methods from the second category seek to model two words' syntactic relation relying on their distance from the dependency tree. the distance can be two words' relative depth on the tree or the length of the path between the two nodes @xcite . this strategy can be seen as extended relative position representations @xcite where the relative distance is defined based on the tree. the advantage of this strategy is that the syntactic relations between a word and all other words can be modeled, not just those of its immediate neighbors. nevertheless, this approach is also unsatisfactory because the distance on the dependency tree is a simple proxy for the syntactic relationship between two words and does not represent how a word reaches another word on the dependency tree. therefore, this type of approach has limitations in expressiveness. for these considerations, we propose to model the position of the word on the dependency tree in addition to its normal sequential position in the input text. we use the path from the root node to a word on the dependency tree to represent the global syntactic position of the word. this representation facilitates the modeling of syntactic relations between two words at arbitrary distances while enabling the model to learn more powerful correlation functions by exploiting dependency labels rather than simply relying on the distance of word pairs. our main contributions are as follows: ## related work previous studies showed that end-to-end nmt models could be further improved by combining source-side or target-side syntax structure into rnn encoder-decoder framework @xcite @xcite . since the transformer has become the de facto standard architecture for nmt, researchers have also begun to explore strategies to incorporate syntax information into transformer-based nmt. for phrase structures, @xcite used source-side linearized constituency parses to improve transformer-based nmt by multi-task learning. @xcite used neural syntactic distance @xcite for constituent parsing as input and output sequence. for dependency grammar, researchers mainly focused on how to integrate the dependency structure into the transformer. as mentioned in the introduction, there are two different viewpoints to look upon this problem. the first line of works extended the relative positional encoding @xcite defined on sequence to tree structures. @xcite defined the relative distance between two words on the dependency tree in terms of their relative depth. @xcite used the length of the shortest path between two nodes on the dependency tree as their relative distance, and the depth of each node as their absolute position, in addition to each word's normal sequential position. another perspective is to take the dependency tree as a general graph structure, focusing on local connection relations and apply methods from graph neural networks (or use similar mechanisms). @xcite use a graph-convolutional networks to encode the source syntax structure, and more recently, bugliarello and okazaki (2020) directly encode the dependency structure inside the transformer's selfattention module, achieving state-of-the-art results. by contrast, we neither treat the dependency tree as a simple sequential structure extension nor as general graph data. we seek to represent each word's role on the dependency tree, where the representation is tailored for the dependency grammar's tree structure, making it sensible to the syntactic connection with other words. our work is also inspired by researches on making transformer sensible to structured input. @xcite proposed a tree-structure positional encoding to make the transformer sensible to source code's tree structure in the code processing task. the main idea is to define the path from the root node to a leaf node as the leaf node's position on the tree. we extend this idea from unlabeled directed trees to labeled trees and break the limitation of the tree depth and the number of child nodes. moreover, we choose to use a sequence encoder instead of concatenating sub-vectors to compute the final path embedding, which utilizes the whole vector space and enables more powerful modeling of the interaction between two path embeddings. ## model in this section, we first recap the basic transformer architecture and then describe the proposed global syntactic position encoding mechanism. ## conclusion and future work in this paper, we propose a global position encoding scheme for the dependency tree. we leverage the dependency labels and represent each word's syntactic role on the dependency tree using the path, which enables efficient syntactic correlation modeling between any pair of words in a single layer. in the experiment, we show that our model outperforms other syntax augmentation strategies. in addition, we quantitatively analyze the model's reliance on syntax information and show that the model pay more weight and achieves better performance when we combine syntax information at lower layers. for future works, the following problems are worth noting: