Semantics-aware Attention Improves Neural Machine Translation (2022)

Abstract: The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel parameter-free methods for injecting semantic information into Transformers, both rely on semantics-aware masking of (some of) the attention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs. We further show an additional gain when using both semantic and syntactic structures in some language pairs.

## introduction it has long been argued that semantic representation can benefit machine translation @xcite . moreover, rnn-based neural machine translation (nmt) has been shown to benefit from the injection of semantic structure @xcite . despite these gains, to our knowledge, there have been no attempts to incorporate semantic structure into nmt transformers @xcite . we address this gap, focusing on the main events in the text, as represented by ucca (universal cognitive conceptual annotation; @xcite , namely scenes. ucca is a semantic framework originating from typological and cognitive-linguistic theories @xcite @xcite . its principal goal is to represent some of the main elements of the semantic structure of the sentence while disregarding its syntax. formally, a ucca representation of a passage is a directed acyclic graph where leaves correspond to the words of the sentence and nodes correspond to semantic units. the edges are labeled by the role of their endpoint in the relation corresponding to their starting point (see fig. 1 ). one of the motivations for using ucca is its capability to separate the sentence into "scenes", which are analogous to events (see fig. 1 ). every such scene consists of one main relation, which can be either a process (i.e., an action), denoted by p, or a state (i.e., continuous state), denoted by s. scenes also contain at least one participant (i.e., entity), denoted by a. for example, the sentence in fig. 1a comprises two scenes: the first one has the process "saw" and two participants -"i" and "the dog"; the second one has the process "barked" and a single participant -"dog". so far, to the best of our knowledge, the only structure-aware work that integrated linguistic knowledge and graph structures into transformers used syntactic structures @xcite @xcite @xcite . the presented method builds on the method proposed by @xcite , which utilized a universal dependencies graph (ud; @xcite of the source sentence to focus the encoder's attention on each token's parent, namely the token's immediate ancestor in the ud graph. similarly, we use the ucca graph of the source sentence to generate a sceneaware mask for the self-attention heads of the encoder. we call this method sasa (see §2.1). we test our model ( §2) on translating english into four languages. two that are more syntactically similar to english @xcite : german (en-de), russian (en-ru), and two that are much less so: turkish (en-tr) and finnish (en-fi). we selected these language pairs for their varied grammatical properties and the availability of reliable parallel datasets for each of them in the wmt benchmark. we find consistent improvements across multiple test sets for all four cases. in addition, we create a syntactic variant of our semantic model for better comparability. we observe that on average, our semantically aware model outperforms the syntactic models. moreover, for the two languages less similar to english (en-tr and en-fi), combining both the semantic and the syntactic data results in a further gain. while improvements are often small, at times the combined version outperforms sasa and udiscal (our syntactic variant, see §3) by 0.52 and 0.69 bleu points (or 0.46 and 0.43 chrf), respectively. we also propose a novel method for introducing the source graph information during the decoding phase, namely through the cross-attention layer in the decoder (see §2.2). we find that it improves over the baseline and syntactic models, although sasa is generally better. interestingly, for en-fi, this model also outperforms sasa, suggesting that some language pairs may benefit more from semantic injection into the decoder. overall, through a series of experiments (see §4), we show the potential of semantics as an aid for nmt. we experiment with a large set of variants of our method, to see where and in what incorporation method they best help. finally, we show that semantic models outperform ud baselines and can be complementary to them in distant languages, showing improvement when combined. ## models transformers have been shown to struggle when translating some types of long-distance dependencies @xcite , and when facing atypical word order @xcite . @xcite proposed ucca based preprocessing at inference time, splitting sentences into different scenes. they hypothesized that models need to decompose the input into scenes implicitly, and provide them with such a decomposition, as well as with the original sentence. they show that this may facilitate machine translation @xcite and sentence simplification @xcite in some cases. motivated by these advances, we integrate ucca to split the source into scenes. however, unlike sulem et al., we do not alter the sentence length in pre-processing, as this method allows less flexibility in the way information is passed, and as preliminary results in reimplementing this method yielded inferior results (see §a.5). instead, we investigate ways to integrate the split into the attention architecture. we follow previous work @xcite in the way we incorporate our semantic information. in their paper, @xcite introduced syntax in the form of a parent-aware mask, which was applied before the softmax layer in the encoder's self-attention. we mask in a similar method to introduce semantics. however, parent in the ucca framework is an elusive concept, given that nodes may have multiple parents. hence, we use a different way to express the semantic information in our mask, i.e., we make it scene-aware, rather than parent-aware. following @xcite , we divide the source sentence into scenes, using the sentence's ucca parse. we then define our scene-aware mask: intuitively, an attention head masked this way is allowed to attend to other tokens, as long as they share a scene with the current one. foot_0 figure 1 demonstrates two examples of such masks, accompanied by their ucca parse graphs and the segmentation into scenes from which these masks were generated. our base model is the transformer @xcite , which we enhance by making the attention layers more scene-aware. we force one foot_1 of the heads to attend to words in the same scene which we assume are more likely to be related than words from different scenes. as we replace regular self-attention heads with our scene-aware ones, we maintain the same number of heads and layers as in the baseline. ## experimental setting data preparation. first, we unescaped html characters and tokenized all our parallel corpora @xcite . next, we removed empty sentences, sentences longer than 100 tokens (either on the source or the target side), sentences with a source-target ratio larger than 1.5, sentences that do not match the corpus's language as deter-mined by langid lui and baldwin, 2012, and sentences that fast align @xcite considers unlikely to align (minimum alignment score of -180). then, for languages with capitalization, we trained true-casing models on the train set @xcite and applied them to all inputs to the network. finally, we trained a bpe model @xcite , jointly for language pairs with a similar writing system (e.g., latin, cyrillic, etc.) and separately otherwise, and then applied them accordingly. we trained our model on the full wmt16 dataset for the english-→german (en-de) task, using the wmt newstest2013 as development set. we also trained our models on a train set consisting of yandex corpus, news commentary v15, and wikititles v2 for the english-→russian (en-ru) task. in addition, we trained our models on the full wmt19 dataset (excluding paracrawl, in order to avoid noisiness in the data) for the english-→finnish (en-fi). finally, we trained on the full wmt18 dataset for the english-→turkish (en-tr) task. for the test sets, we used all the newstests available for every language pair since 2012, excluding the one designated for development. ## experiments we hypothesize that nmt models may benefit from the introduction of semantic structure, and present a set of experiments that support this hypothesis using the above-presented methods. ## conclusion in this work, we suggest two novel methods for injecting semantic information into an nmt transformer model -one through the encoder (i.e. sasa) and one through the decoder (i.e. sacra). the strength of both methods is that they both do not introduce more parameters to the model, and only rely on ucca-parses of the source sentences, which are generated in advance using an off-theshelf parser, and thus do not increase the complexity of the model. we compare our methods to previously developed methods of syntax injection, and to our adaptation to these methods, and find that semantic information tends to be significantly more beneficial than syntactic information, mostly when injected into the encoder (sasa), but at times also during decoding (sacra). moreover, we find that for sufficiently different languages, such as english and finnish or english and turkish, incorporating both syntactic and semantic structures further improves the performance of the translation models. future work will further investigate the benefits of semantic structure in transformers, alone and in unison with syntactic structure.