Discriminative Reranking for Neural Machine Translation (2021)

Abstract: Reranking models enable the integration of rich features to select a better output hypothesis within an n-best list or lattice. These models have a long history in NLP, and we revisit discriminative reranking for modern neural machine translation models by training a large transformer architecture. This takes as input both the source sentence as well as a list of hypotheses to output a ranked list. The reranker is trained to predict the observed distribution of a desired metric, e.g. BLEU, over the n-best list. Since such a discriminator contains hundreds of millions of parameters, we improve its generalization using pre-training and data augmentation techniques. Experiments on four WMT directions show that our discriminative reranking approach is effective and complementary to existing generative reranking approaches, yielding improvements of up to 4 BLEU over the beam search output.

## introduction reranking models take a number of different output hypotheses generated by a baseline model and select one hypothesis based on more powerful features. before the recent re-emergence of neural networks, these models have been well studied for several nlp tasks including parsing @xcite and statistical machine translation @xcite . traditional statistical models (smt) based on n-gram counts made very strong independence assumptions where features would only capture very local context information to avoid sparsity and poor generalization. a large n-best list produced by these models would then be passed to a discriminatively trained reranker which leverages features engineered to capture more global context @xcite yielding significant improvements to the quality of the translations. on the other hand, modern neural models (nmt) make much weaker independence assumptions because predictions of standard sequenceto-sequence models depend on the entire source sentence as well as the target prefix generated. however, reranking may still be beneficial for two reasons: first, nmt systems are subject to exposure bias @xcite , i.e., models are never exposed to their own generations at training time, while a reranking model has been trained on model outputs. second, beam search with autoregressive models uses the chain rule to sum individual token-level probabilities to obtain a target sequence probability. however, individual probabilities are based on a limited amount of target context, while a reranking model can condition on the entire target context. indeed, recent generative reranking approaches applied to nmt, such as noisy-channel decoding (ncd, @xcite which leverages a pre-trained language model and a backward model, show strong improvements over beam search outputs, as demonstrated in recent wmt evaluations @xcite . in this paper, we explore whether training large transformer models using the reranking objective can further improve performance. our model, dubbed drnmt, takes as input the entire source sentence and an n-best list of output hypotheses to predict a distribution of sentence-level evaluation scores, such as bleu. 1 this setup is similar to earlier work with smt, except that the baseline model is an nmt model and the reranker is a big transformer architecture as opposed to a log-linear model on top of discrete or human engineered features. unfortunately, optimizing for the task of interest does not always lead to better performance. overfitting to the training set is a potential concern, as the reranker has hundreds of millions of parameters yet it receives only one gradient and weight update per source/target sentence pair as opposed to one per token as for standard nmt models. in our work, we mitigate overfitting in two ways. first, we leverage the success of pre-training by finetuning masked language models (mlm; @xcite ) which initializes the model with features trained on much more training data. second, we augment the original dataset with back-translated data (bt; @xcite . experiments show that drnmt can match the performance of a strong ncd baseline and that their combination leads to further improvements as measured by bleu, ter and also human evaluation. ## related work our method is inspired by the seminal work of @xcite and @xcite who introduced and popularized discriminative reranking to smt. besides using a weaker mt system to generate the n-best list, these works relied on a linear discriminator trained on human-designed features as opposed to a transformer taking the raw source sentence and hypothesis. most work using nmt has focused on generative reranking methods @xcite @xcite , where the reranker's parameters are optimized using a criterion which is different from the metric of interest. for instance, @xcite ; @xcite perform noisy-channel decoding where hypotheses are scored by linearly combining the output of the forward model, a target-side language model and a backward model which scores the source sentence given the hypothesis. these methods have shown remarkable improvements over the output of beam decoding, despite not being trained for the reranking task (except for the two or three hyperparameters of the linear combination of scores which are tuned on a validation set). another approach belonging to this class of methods is the one proposed by @xcite , which employs the scores from a masked language model (mlm). while this method employs a transformer architecture, it is still not trained for the task of interest. to the best of our knowledge, there is only concurrent work by @xcite which attempts at training discriminatively a reranker for nmt. they use a pair-wise margin loss on hypotheses sampled from the nmt, while we learn to rank the full n-best list produced by beam. their experiments also show that the reranker performs better when directly conditioned on the source sentence. however, they do not compare nor combine their method with ncd like we do. both their work and our work are however an extension of @xcite , who proposed to train a discriminator to improve neural language modeling. there is also a large body of literature on different ways to combine smt and nmt by using one to rerank the other, since smt is generally better at adequacy while nmt is better at fluency. for instance, @xcite uses an rnn discriminator to rerank the n-best list produced by a phrase-based smt. instead, @xcite does the opposite, using an smt discriminator to rerank an n-best list produced by an nmt. finally, our work is also related to recent attempts at using adversarial training to improve mt @xcite . unlike these approaches our method is much simpler because we do not update the parameters of the mt system generating the hypotheses. moreover, our discriminator is trained to predict the distribution of desired metric and it is used at decoding time to rerank, while gan-based mt would only retain the generator. ## model given a source sentence x, an nmt model generates a set of hypotheses u(x) = {u 1 , u 2 , ..., u n } in the target language. the goal of this work is to learn a reranker that produces higher scores for hypotheses of better quality, as defined in terms of a user-specified metric µ(u, r) such as bleu @xcite , where quality is measured with respect to a reference r. as illustrated in figure 1 , our reranker is a transformer architecture which takes as input the concatenation of the source sentence x and hypothesis @xmath0. the architecture includes also position embeddings and language embeddings, to help the model represent tokens that are shared between the two languages @xcite . the final hidden state corresponding to the start of sentence token ( s ) serves as the joint representation for (x, u); let us denote this feature vector as @xmath1. the reranker associates a scalar score @xmath2, u) by applying a one hidden layer neural network with d tanh hidden units to z, as default in the design of the "classification head" of roberta @xcite . the parameters of the reranker are denoted by θ and include the parameters of the transformer, all the embeddings and also the top projection block mapping the feature vector to the scalar score. each hypothesis u i in the set u(x) is therefore processed independently and yields a score o i . ## training and inference we train the reranker discriminatively, hence the name drnmt for discriminative reranker for nmt, by minimizing the kl-divergence between the target distribution and the model output distribution, d kl (p t ||p m ) @xcite . for each x, the model output distribution is a softmax over all n hypotheses in the n-best list: notice that we do not enforce any additional factorization. in particular, we do not assume that the score is computed auto-regressively. the target distribution is defined as a normalized distribution of the end metric µ(u i , r) which we assume to improve as it takes on larger values: the parameters of drnmt are then learned by minimizing the kl divergence over the training dataset. for a given training example, we have: we minimize this loss over the training set by stochastic gradient descent using standard backpropagation of the error, since all terms are differentiable. in order to alleviate overfitting, we employ dropout regularization @xcite , we pre-train the model @xcite and we also perform data augmentation by training on back-translated data (bt) @xcite . see §5.3 for details. at test time, generation proceeds by first having the nmt generate the n-best list, and then by applying the reranker to select the best hypothesis. since the score of the forward model is also available, unless otherwise specified we rerank using a weighted combination of both; this is dubbed as drnmt. in the experiments we also report results by adding all the other scores from ncd, namely the backward model score and the language model score. we denote this variant by "drnmt + ncd". whenever we combine scores from various models we tune the additional hyper-parameters controlling the weighted combination by random search on the validation set @xcite . ## experimental setup in this section we describe the datasets, baselines and model details. ## results in this section we report the main findings of our work. when optimizing for bleu as metric, the performance of drnmt and baselines for de-en, en-de, en-ta and ru-en is summarized in first, we notice that all methods improve over the beam search output with gains ranging from 1.0 to 4.1 bleu. however, there may be still room for improvement as the oracle performance suggests. the oracle is computed by selecting the best hypotheses based on bleu with respect to the human reference. of course, the oracle may be not achievable because of uncertainty in the translation task. third, drnmt performs on par (en-ta, en-de and ru-en) or better (de-en) than ncd, showing that discriminative reranking can be very competitive. note, that the reranker requires only one additional forward pass through the hypotheses generated by beam, while ncd requires two forward passes (one for the lm and one for the backward mt model). therefore, our reranker works at least as well as ncd while requiring roughly half of the compute. fourth, the discriminative reranker and ncd are complementary to each other, since combining both achieves the best performance overall across the three language directions, with gains between 0.9 bleu (de-en) and 0.2 (en-ta) compared to ncd, and an overall gain between 4.1 bleu (de-en) and 0.5 (en-ta) compared to the beam baseline. fifth, the gain brought by discriminative reranking can be better appreciated by comparing "fw + lm" and drnmt, as the major difference between the two approaches is the objective function used for training them (generative language modeling instead of prediction of the distribution of bleu scores). we can see that in all cases, discriminative reranking yields better translations, with gains between 0.2 and 2.3 bleu points depending on the language direction. finally, we notice that en-ta is a difficult lan- human evaluation: we randomly sample 750 sentences from the de-en test sets and collect human ratings. we perform a/b testing, where a rater can see the source sentence together with translated sentences from two systems. we conduct two rounds of human evaluation by comparing the proposed "drnmt + ncd" vs. "beam", and "drnmt + ncd" vs. "ncd". for each sentence, we collect three ratings (between 0 to 100) and average the scores, treating sentences with a score difference less than 5 as equally good. out of the 750 sentences, our proposed method generates better translation than beam on 149 sentences and is worse on 82 sentences, and it performs better than ncd on 123 sentences and worse on 108 sentences, corroborating the gains observed when measuring with bleu. next we show that drnmt works with other userspecified metrics, study how performance varies with the number of hypotheses and perform several ablation studies to better understand its critical components. ## conclusions reranking is effective for both smt and nmt. inspired by work done almost two decades ago @xcite , we studied discriminative reranking for nmt and found that it performs at least as well as the strongest generative reranking method we are aware of, namely noisy channel decoding (ncd) @xcite -as long as care is taken to alleviate overfitting. there is a subtle trade-off between improvements stemming from optimizing the end metric and addressing exposure bias on the one hand, and poor generalization and sample inefficiency of discriminative training on the other hand. in this study we regularize the reranker by using dropout, by pre-training on large corpora and by performing data augmentation. empirically, we found that ncd and our discrim-inative reranker are complementary to each other, yielding sizeable improvements over each other and the beam baseline. our reranker is computationally less demanding than ncd, since it consists of a single model while ncd requires scoring using two additional models. our reranker is also robust to the choice of the size of the n-best list and other hyper-parameters settings. in the future we plan to investigate better ways to alleviate sample inefficiency, as well as to design more effective architectures to score at the set level.