Passing Parser Uncertainty to the Transformer: Labeled Dependency Distributions for Neural Machine Translation (2022)

Abstract: Existing syntax-enriched neural machine translation (NMT) models work either with the single most-likely unlabeled parse or the set of n-best unlabeled parses coming out of an external parser. Passing a single or n-best parses to the NMT model risks propagating parse errors. Furthermore, unlabeled parses represent only syntactic groupings without their linguistically relevant categories. In this paper we explore the question: Does passing both parser uncertainty and labeled syntactic knowledge to the Transformer improve its translation performance? This paper contributes a novel method for infusing the whole labeled dependency distributions (LDD) of the source sentence’s dependency forest into the self-attention mechanism of the encoder of the Transformer. A range of experimental results on three language pairs demonstrate that the proposed approach outperforms both the vanilla Transformer as well as the single best-parse Transformer model across several evaluation metrics.

## introduction neural machine translation (nmt) models based on the seq2seq schema, e.g., @xcite ; @xcite ; @xcite ; @xcite , first encode the source sentence into a high-dimensional content vector before decoding it into the target sentence. several prior studies @xcite have pointed out that although nmt models may induce aspects of syntactic relations, they still cannot capture the subtleties of syntactic structure that should be useful for accurate translation, particularly by bridging long distance relations. previous work provides support for the hypothesis that explicit incorporation of source syntactic knowledge could result in better translation performance, e.g., @xcite ; @xcite . most models condition translation on a single best parse syn: arg max t p (t|s, syn) while most syntax-enriched strategies result in performance improvements, there are two noteworthy gaps in the literature addressing source syntax. firstly, none of the existing works conditions on the probability distributions over source syntactic relations. and secondly, none of the existing approaches conditions on the dependency labels, thereby conditioning only on the binary choice whether there is an unlabeled dependency relation between two words. @xcite ; @xcite ; @xcite showed that the whole dependency forest provides better performance than a single best parse approach. in this paper we go one step further and propose that a syntactic parser is more useful if it conveys to the nmt model also its remaining uncertainty, expressed as the whole probability distributions over dependency relations rather than a mere forest. to the best of our knowledge, there is no published work that incorporates a parser's distributions over dependency relations into the transformer model @xcite , let alone incorporating distributions over labeled dependency relations into nmt models at large. this paper contributes a generic approach for infusing labeled dependency distributions into the encoder's self-attention layer of the transformer. we represent a labeled dependency distributions as a three-dimensional tensor of parser probabilities, where the first and second dimensions concern word-positions and the third concerns the dependency labels. the resulting tensor is infused into the computation of the multi-head self-attention, where every head is made to specialize in a specific dependency class. we contribute empirical evidence that passing uncertainty to the transformer and passing labeled dependencies both give better performance than passing a single unlabeled parse, or an unlabeled/labeled set of dependency relations with uniform probabilities. ## related work the role of source syntactic knowledge in better reordering was appreciated early on during the statistical machine translation (smt) era. for example, @xcite propose that source language parses should play a crucial role in guiding the reordering within translation, and do so by integrating constituency labels of varying granularity into the source language. although, nmt encoders have been claimed to have the ability to learn syntax, work on rnns-based models shows the value of external source syntax in improving translation performance, e.g., @xcite , by refining the encoder component, leading to a combination of a tree-based encoder and a sequential encoder. noteworthy to recall here that the attention mechanism was originally aimed to capture all word-to-word relations, including syntacticsemantic relations. whereas, the work of @xcite has shown that a single unlabeled dependency parse, encoded utilizing graph convo-lutional networks (gcns), can help improve mt performance. @xcite and @xcite attempt to incorporate parse forests into rnns-based nmt models, mitigating parsing errors by providing more candidate options. however, these two works only rely on the binary (unlabeled) relations in all the sub-trees, ignoring the elaborate probability relations between word positions and the type of these relations. although the transformer @xcite is considered to have a better ability to implicitly learn relations between words than the rnns-based models, existing work @xcite shows that even incorporating a single best parse could improve the transformer translation performance. followup work @xcite provides similar evidence by changing the transformer's self-attention mechanism based on the distance between the input words of dependency relations, exploiting the single best unlabeled dependency parse. the work of @xcite suggests that the benefits of incorporating a single (possibly noisy) parse (using data manipulation, linearized or embedding-based method) can be explained as a mere regularization effect of the model, which does not help the transformer to exploit the actual syntactic knowledge. interestingly, @xcite arrive at a similar hypothesis, but they concentrate on exploring how to train one of the heads of the self-attention in the transformer for a combined objective of parsing and translation. the parsing-translation training objective focuses the self-attention of a single head at learning the distribution of unlabeled dependencies while learning to translate as well, i.e., the distribution is not taken as source input but as a gold training objective. by training a single head with syntax, they leave all other heads without direct access to syntax. our work confirms the intuition of @xcite regarding the utility of the parser's full dependency distributions, but in our model these distributions are infused directly into the selfattention while maintaining a single training objective (translation). furthermore, we propose that only when the full probability distribution matrices over labeled dependency relations is infused directly into the transformer's self-attention mechanism (not as training objective), syntax has a chance to teach the transformer to better learn syntax-informed self-attention weights. ## proposed approach a parser can be seen as an external expert system that provides linguistic knowledge to assist the nmt models in explicitly taking into account syntactic structure. for some sentences, the parser could be rather uncertain and spread its probability over multiple parses almost uniformly, but in the majority of cases the parser could have a rather sharp distribution over the alternative parses. therefore, simply passing a dependency forest amounts merely to passing all alternative parses accompanied with zero information on parser confidence (maximum perplexity) to the transformer nmt model, which does not help it to distinguish between the parsing information of the one input from that of another. this could increase the complexity of learning the nmt model unnecessarily. an alternative is then to use for each sentence a dependency distribution in the form of conditional probabilities, which could be taken to represent the degree of confidence of the parser in the individual dependency relations. furthermore, we propose that each dependency relation type (label), provides a more granular local probability distribution that could assist the transformer model in making more accurate estimation of the context vector. this might enhance the quality of encoding the source sentence, particularly because the transformer model relies on a weak notion or word order, which is input in the form of positional encoding outside the self-attention mechanism. note that the word-to-word dependency probabilities is not equivalent to using a distribution over dependency parses. this is because in some cases the word-to-word dependencies (just like word-toword attention) could combine together into general graphs (not necessarily trees). we think that using relations between pairs of words (rather than upholding strict tree or forest structures) fits well with the self-attention mechanism. ## experiments and analysis experimental setup we establish seven distinct sets of experiments, refer to parser: we employ an external dependency parser supar @xcite to automatically parse the source sentences. since this parser was trained using the biaffine method @xcite , we can extract dependency distributions by changing its source code. data: we evaluate the translation tasks for three language pairs from three different language families: english-chinese (@xmath0, english-italian (@xmath1, and english-german (@xmath2. we chose dev2010 and test2010 as our validation and test datasets from iwslt2017 @xmath3. in @xmath4, we randomly selected a 110k subset from the iwslt2015 dataset as training set and used dev2010 as validation set, tst2010 as test set. for training only, we first filtered out the source sentences that supar cannot parse and sentences that exceed 256 tokens in length. and then, we used supar foot_4 to parse each source language sentence to obtain the labeled dependency distributions and applied spacy foot_5 to tokenize the source and target languages, respectively. finally, we replaced words in the corpus with "<@xmath5, and for each minibatch sentences, added "<@xmath6,"<@xmath7, and for sentences with inconsistent lengths per mini-batch, added a corresponding number of "<@xmath8. hyperparameters: in the low-resource experiments, the batch size was 256, the number of layers for the encoder and decoder was 4, and the number of warm-up steps was 400. in the medium-resource experiments, their values were 512, 6, 4000, respectively. for the rest, we use the base configuration of the transformer @xcite : all experiments were optimized using adam @xcite (where β 1 was 0.9, β 2 was 0.98, ϵ was 10-9) and the initial learning rate was set to 0.0001, gradually reduced during training as follows: the number of heads in multi-head attention was set to 8 (16 in ldd layer), the dimension of the model was 512, the dimension of inner fullyconnected layers was set to 2048, and the loss function was the cross-entropy loss function. the checkpoint with the highest bleu-4 score on the validation set was saved for model testing during training. the number of epochs was set to 50 (one epoch represents a complete training produce). in order to prevent over-fitting, we set the dropout rate (also in our ldd layer) to 0.1. ## conclusion this paper presented a novel supervised conditional labeled dependency distributions trans-former network (ldd-seq). this method primarily improves the self-attention mechanism in the transformer model by converting the dependency forest to conditional probability distributions; each self-attention head in the transformer learns a dependency relation distribution, allowing the transformer to learn source language's dependency constraints, and generates attention weights that are more in line with the syntactic structures. the experimental outcomes demonstrated that the proposed method was straightforward, and it could effectively leverage the source language dependency syntactic structures to improve the transformer's translation performance without increasing the complexity of the transformer network or interfering with the highly parallelized characteristic of the transformer model.