Syntax-Based Attention Masking for Neural Machine Translation (2021)

Abstract: We present a simple method for extending transformers to source-side trees. We define a number of masks that limit self-attention based on relationships among tree nodes, and we allow each attention head to learn which mask or masks to use. On translation from English to various low-resource languages, and translation in both directions between English and German, our method always improves over simple linearization of the source-side parse tree and almost always improves over a sequence-to-sequence baseline, by up to +2.1 BLEU.

## introduction the transformer model for machine translation @xcite was originally defined as a mapping from sequences to sequences. more recent work has explored extensions of transformers to other structures: a tree transformer would be able to make use of syntactic information, and a graph transformer would be able to make use of semantic graphs or knowledge graphs. there have been a number of proposals for transformers on trees, including phrase-structure trees and dependency trees for natural languages, and abstract syntax trees for programming languages. one common strategy is to linearize a tree into a sequence @xcite . another strategy is to recognize that transformers are fundamentally defined not on sequences but on bags; all information about sequential order is contained in the positional encodings, so all that is needed to construct a tree transformer is to define new positional encodings on trees @xcite . in this paper, we present a third approach, which is to enhance the encoder's self-attention mechanism with attention masks @xcite , which restrict the possible positions an attention head can attend to. we extend this idea in two new ways. first, our attention masks are based on relationships among tree positions (for example, "is an ancestor of" or "is a descendant of") rather than sequence positions ("is left of" or "is right of"). second, instead of pre-assigning different masks to each attention head, we allow each attention head to learn separately which mask or masks to use. we experiment on machine translation of several low-resource language pairs (section 3). compared to linearization without masks, our method always improves accuracy, by up to +1.7 bleu (all bleu reported as percen t). compared with a sequence-tosequence baseline, our method improves accuracy by up to +2.1 bleu. on tasks where linearization hurts, our method is usually, but not always, able to turn the loss into a gain. ## methods like several previous approaches, we use linearized syntax trees. but whereas the usual linearization traverses a node both before and after its descendants, we use a preorder traversal of the tree. in other words, our linearization does not have closing brackets. our linearization does not have enough information to reconstruct the original tree; this information is contained in the attention masks, which we describe next. @xcite introduce the idea of using masks in a string transformer to allow attention heads to attend only to the left or only to the right. we apply this idea to tree transformers, with two modifications. first, instead of masking out the left or right context, we use masks based on the structure of the tree. second, instead of allocating a fixed number of heads to each mask, we let the model learn which mask(s) to use for each attention head. given a query @xmath0, key @xmath1, and value @xmath2, scaled dot-product attention is normally computed as the strength parameters are initialized to zero and learned by backpropagation with the rest of the model. in this way, each attention head can learn separately which mask or masks to use. it remains to define the masks m m . a mask can be defined for any imaginable string or tree relationship. because the model can always choose not to use a mask, we can add as many masks as we want. we use the following set: although none of the above masks overlap, there would be no problem with defining masks that do. please see figure 1 for an example. in (a) is an english tree; (b) shows the same tree after applying byte pair encoding (bpe) subword segmentation (see section 3 below); and (c) shows the relationships of all the nodes with the second np (the one dominating my father). ## https://github.com/tnq177/witwicky we score detokenized system outputs using casesensitive bleu against raw references (except on en-vi, where we use tokenized outputs and references), using bootstrap resampling @xcite ) for significance testing. ## conclusion in this paper, we've shown that syntax can be both helpful and easy to incorporate into low-resource neural machine translation. we introduced learnable attention masks for the transformer that allow each attention head to focus more narrowly on certain node relationships in the syntax tree, improving translation across a variety of low-resource datasets by up to +2.1 bleu.