Synchronous Syntactic Attention for Transformer Neural Machine Translation (2021)

Abstract: This paper proposes a novel attention mechanism for Transformer Neural Machine Translation, “Synchronous Syntactic Attention,” inspired by synchronous dependency grammars. The mechanism synchronizes source-side and target-side syntactic self-attentions by minimizing the difference between target-side self-attentions and the source-side self-attentions mapped by the encoder-decoder attention matrix. The experiments show that the proposed method improves the translation performance on WMT14 En-De, WMT16 En-Ro, and ASPEC Ja-En (up to +0.38 points in BLEU).

## introduction the transformer neural machine translation (nmt) model @xcite has achieved state-of-the-art performance and become the focus of many nmt studies. one of its characteristics is the self-attention mechanism, which computes the strength of relationships between two words in a sentence. transformer nmt has been improved by extending the self-attention mechanism to incorporate syntactic information @xcite @xcite @xcite . in particular, @xcite and @xcite have proposed dependencybased self-attentions, which are trained to attend to the syntactic parent for each token under constraints based on the dependency relations, for capturing sentence structures. existing syntaxbased nmt models, including their ones, use only monolingual syntactic information on either side or both. by contrast, synchronous grammars such as synchronous context-free grammars and synchronous dependency grammars, which are defined in two languages and generate sentence structures aligned across them, have been introduced into many smt models with the result of improving their translation performances @xcite @xcite . figure 1 shows an example of the dependency structures of source and target language sentences and their alignments foot_0 . inspired by synchronous dependency grammars, we aim to improve the performance of transformer nmt by incorporating the main idea of the synchronous dependency grammars (i.e., synchronizing sentence structures across two languages). as far as we know, neither the synchronous dependency grammars themselves nor their basic idea has yet been incorporated into nmt. this paper proposes a novel attention mechanism for transformer nmt, called "synchronous syntactic attention," which captures sentence structures aligned across two languages by the aligned self-attentions on the source-and targetside. the mechanism uses encoder-decoder attentions to map source-side syntactic self-attentions into a target language space based on @xcite 's observation that encoder-decoder attentions represent the alignments of source and target words. the mechanism is trained to maintain consistency between source-and target-side syntactic self-attentions according to an objective loss function that incorporates the difference between the target-side syntactic self-attentions and the mapped source-side syntactic self-attentions. we use dependency-based self-attention @xcite as source-and target-side syntactic self-attentions. ## transformer nmt model the transformer nmt model @xcite is an encoder-decoder model composed of the encoder that encodes source tokens @xmath0, f 2 , . . . , f i ) into hidden vectors and the decoder that generates target tokens @xmath1, e 2 , . . . , e j ) from the outputs of the encoder. the encoder and decoder consist of n enc encoder layers and n dec decoder layers, respectively. both the encoder layers and decoder layers are composed of multiple sub-layers, each of which includes a self-attention layer and a feed forward layer. the decoder layers additionally apply an encoder-decoder attention layer between the selfattention layer and the feed forward layer. the self-attention and encoder-decoder attention are calculated by a multi-head attention mechanism. the multi-head attention mha(q, k, v ) maps the d emb -dimension embedding space into h subspaces of the d k (= d emb h ) dimension and calculates attention in each subspace as shown in equations 1 to 3: and w @xmath2. in the selfattention, the previous layer's output is used as q, k, and v . in the encoder-decoder attention, the previous layer's output is used as q and the last encoder layer's output is used as k and v . note that, in training, the decoder's self-attention masks future tokens. ## dependency-based self-attention this section describes dependency-based selfattention (dbsa) @xcite , which is the baseline of our syntactic self-attention. dbsa captures dependency structures by extending the multi-head self-attention of the l dep -th layer of the encoder or decoder. let h be one of head of the l dep -th encoder layer's self-attention or the l dep -th decoder layer's self attention. an attention weight matrix a h , where each value indicates the dependency relationship between two words, is calculated by using the bi-affine operation in equation 4 : in a h , the probability of token q being the head of token t in a source/target sentence s (i.e., then, a weighted representation matrix m h , which includes dependency relationships in the source sentence or target sentence, is obtained by multiply- finally, m h is concatenated with the other heads and mapped to a d emb -dimensional matrix. in the decoder-side dbsa, future information is masked to prevent attending to unpredicted tokens in inference. the transformer nmt model with dbsa learns translation and dependency parsing at the same time by minimizing the objective function @xmath3, where l t is the translation loss and l dep is computed in equation 5 : ## proposed method: synchronous syntactic attention this section proposes a novel attention mechanism for transformer nmt, "synchronous syntactic attention," which captures sentence structures aligned across source and target languages. a transformer nmt model with the proposed attention is trained according to the objective function presented below as equation 6 : an example of synchronous syntactic attention attention (i.e., dbsa) and λ sync is a hyperparameter to control the influence of l sync . in particular, l sync is the differences between the encoder's self-attention, which is mapped into target language space by the encoder-decoder attention, and the decoder's self-attention. let e and d be the attention matrix a h of the l dep -th encoder layer's syntactic self-attention and that of the l dep -th decoder layer's syntactic self attention, respectively. the proposed method first maps e into the target language space by the encoder-decoder attention as shown by equation 7: next, the proposed method computes the mean squared error between d ′ and d as l sync as follows in equation 9 : figure 2 shows an example of the synchronous syntactic attention. the value in each cell indicates an attention score (i.e., an element of an attention weight matrix), and the darker cell represents a higher attention score. in all matrices, each row represents an attention distribution for each token (i.e., scores are normalized in a row direction). as can be seen in figure 2 , the english encoder's syntactic self-attentions e is mapped into the german encoder's syntactic self-attentions d ′ using the encoder-decoder attentions c and c ⊤ . then, the loss between the german encoder's syntactic self-attentions d ′ and the german decoder's syntactic self-attentions d is measured. when calculating the loss, the values of the masked elements in d ′ and d, such as d sprichst,du and d du,englisch? , are assigned to zero. ## related work the main characteristic of transformer nmt is attention mechanisms (i.e., self-attentions and encoder-decoder attentions). some researches have analyzed and/or improved the attention mechanisms of transformer nmt. for instance, @xcite analyzed encoder-decoder attentions in terms of word sense disambiguation, and @xcite analyzed self-attentions in terms of subject-verb agreement and word sense disambiguation. @xcite and @xcite revealed the behaviors of attention heads in terms of dependency relations. namely, @xcite observed that specific attention heads of the en-coder's self-attentions mark syntactic dependency relations. @xcite found that the confident heads play linguistically-interpretable roles like dependency relations. @xcite proposed a method for jointly learning to produce translations and alignments with a single transformer model and showed that encoder-decoder attentions emulate word alignments. based on their observations, our method maps the encoder's syntactic self-attentions into the target language space by using encoder-decoder attentions. @xcite extended a self-attention mechanism to encode the relative positions between two words in a sentence. omote et al. (2019) and wang et al. (2019b) proposed a selfattention mechanism to encode relative positions on source-side dependency trees. some researchers proposed syntax-aware selfattentions that are trained using dependency-based constraints. for instance, wang et al. (2019a) and bugliarello and okazaki (2020) proposed sourceside dependency-aware transformer nmt. wang et al. (2019a) created a constraint based on dependency relations between tokens to encoder self-attentions. bugliarello and okazaki (2020) also proposed parent-scaled self-attention, which multiplies an attention weight matrix by scores based on dependency relations. deguchi et al. 2019 proposed dbsa, which is applicable to both the encoder's and decoder's self-attentions and is extended to subword units. we used dbsa to implement source-and target-side syntactic attentions in transformer nmt. the main difference from the above-mentioned studies is that our work focuses on the incorporation of bilingual syntactic information into nmt. harada and watanabe (2021) incorporated synchronous phrase structure grammar into nmt. specifically, they proposed a syntactic nmt model that induces latent phrase structure and synchronizes the source-and target-side sentence structures. the difference with our model is that we synchronize dependency structures while they synchronize phrase structures. ## conclusions in this paper, we proposed a novel attention mechanism for transformer nmt, "synchronous syntactic attention," which captures sentence structures aligned across source and target languages by aligned self-attention. the synchronous at-tention mechanism trains syntactic self-attentions (dbsa) under a constraint that minimizes the loss between encoder's and decoder's self attentions, where the encoder's self attentions are mapped into the target language space by encoder-decoder attentions. since this method relies only on the constraint induced from the encoder's and decoder's self-attentions and encoder-decoder attentions, it does not require additional model parameters. the experiments show that the proposed method improves transformer nmt's translation performance (up to a 0.38 bleu point improvement).