Generating Diverse Translation from Model Distribution with Dropout (2020)

Abstract: Despite the improvement of translation quality, neural machine translation (NMT) often suffers from the lack of diversity in its generation. In this paper, we propose to generate diverse translations by deriving a large number of possible models with Bayesian modelling and sampling models from them for inference. The possible models are obtained by applying concrete dropout to the NMT model and each of them has specific confidence for its prediction, which corresponds to a posterior model distribution under specific training data in the principle of Bayesian modeling. With variational inference, the posterior model distribution can be approximated with a variational distribution, from which the final models for inference are sampled. We conducted experiments on Chinese-English and English-German translation tasks and the results shows that our method makes a better trade-off between diversity and accuracy.

## introduction in the past several years, neural machine translation (nmt) @xcite @xcite @xcite based on the end-to-end model has achieved impressive progress in improving the accuracy of translation. despite its remarkable success, nmt still faces problems in diversity. in natural language, due to lexical, syntactic and synonymous factors, there are usually multiple proper translations for a sentence. however, existing nmt models mostly implement one-to-one mapping between natural languages, that is, one source language sentence corresponds to one target language sentence. although beam search, a widely used decoding algorithm, can generate a group of translations, its search space is too narrow to extract diverse translations. there are some researches working at enhancing translation diversity in recent years. @xcite and @xcite proposed to add regularization terms to the beam search algorithm so that it can possess greater diversity. @xcite and @xcite introduced latent variables into the nmt model, thus the model can generate diverse outputs using different latent variables. moreover, @xcite proposed to combine the structural characteristics of transformer and use the different weights between each head in the multi-head attention mechanism to obtain diverse results. in spite of improvement in balancing accuracy and diversity, these methods do not represent the diversity in the nmt model directly. in this paper, we take a different approach to generate diverse translation by explicitly maintaining different models based on the principle of bayesian neural networks (bnns). these models are derived by applying concrete dropout @xcite to the original nmt model and each of them is given a probability to show its confidence in generation. according to bayesian theorem, the probabilities over all the possible models under a specific training dataset forms a posterior model distribution which should be involved at inference. to make the posterior model distribution obtainable at inference, we further employ variational inference @xcite @xcite to infer a variational distribution to approximate it, then at inference we can sample a specific model based on the variational distribution for generation. we conducted experiments on the nist chinese-english and the wmt'14 english-german translation tasks and compared our method with different strong baseline approaches. the experiment results show that our method can get a good trade-off in translation diversity and accuracy with little train-ing cost. our contributions in this paper are as follows: ## background assume a source sentence with n words @xmath0, x 2 , ..., x n , and its corresponding target sentence with m words @xmath1, y 2 , ..., y m , nmt models the probability of generating y with x as the input. based on the encoder-decoder framework, nmt model θ encodes source sentence into hidden states by its encoder, and uses its decoder to find the probability of t-th word y t , which depends on the hidden states and the first t -1 words of target sentence y. the translation probability from sentence x and y can be expressed as: given a training dataset with source-target sentence pairs @xmath2, y * 1 ), ..., (x d , y * d )}, the loss function we want to minimize in the training is the sum of negative log-likelihood of equation 1: in practice, by properly designing neural network structures and training strategies, we can get the specific model parameters that minimize equation 2 and obtain translation results through the model with beam search. one of the most popular model in nmt is transformer, which was proposed by @xcite . without recurrent and convolutional networks, transformer constructs its encoder and decoder by stacking self-attention and fullyconnected network layers. self attention is operated with three inputs: query(q), key(k) and value(v) as: note that transformer implements the multihead attention mechanism, projecting inputs into h group inputs to generate h different outputs in equation 3, and these outputs are concatenated and projected into final outputs: we only give a brief description of transformer above. please refer to @xcite for more details. ## application to transformer previous sections show how to use concrete dropout to realize variational approximation of the posterior model distribution. in this section we will introduce the implementation in representing model distribution for transformer with aforementioned methods. ## related work researches in bayesian neural network have a long history, @xcite firstly proposes a variational inference approximation methods to bnn to minimize the minimum description length (mdl), then neal (1995) approximate bnn by hamiltonian monte carlo methods. in recent years, @xcite introduces the concept of variational inference, by approximating posterior distribution with model distribution, the model minimizes its mdl and reduces the model weight; and @xcite proposes an algorithm similar to @xcite , however, it uses mixture of gaussian densities as prior and achieved comparable performance with dropout in regularization. introduced by @xcite , dropout, which is easy to implement, works as a stochastic regularization to avoid over-fitting. and there are several theoretical explainations such as getting sufficient model combinations @xcite to train and augumenting training data @xcite . @xcite proposes that dropout can be understood as a bayesian inferences algorithm, and @xcite uses concrete dropout in updating dropout probabilities. also, the author implements the dropout methods to represent uncertainty in different kinds of deep learning tasks in @xcite . in neural machine translation task, lack of diversity is a widely acknowledged problem, some researches like @xcite investigate the cause of uncertainty in nmt, and some provide metrics to evaluate the translation uncertainty like @xcite ; @xcite . there are also other researches that put forward methods to obtain diverse translation. li et al. (2016 there are also a few papers in interpreting transformer model, @xcite suggests that some heads play a consistent role in machine translation, and their roles can be interpreted linguistically; also, they implement l 0 penalty to prune heads. @xcite shows that huge amounts of heads in transformer can be pruned, and the importance of head is cross-domain. also, @xcite shows that the layers in transformer are also able to be pruned: similar to our work, during training, they drop the whole layer with dropout and trained their probability; however, variational inference strategy is not used in their paper, and they take different kinds of inference strategies to balance performance and efficiency rather than sampling. ## conclusion in this paper, we propose to utilize variational inference in diverse machine translation tasks. we represent the transformer model distribution with dropout, and train the model distributions to minimize its distance to the posterior distribution under specific training dataset. then we generate diverse translations with the models sampled from the trained model distribution. we further analyze the correlations between module importance and trained dropout probabilities. experiment results in chinese-english and english-german translation tasks suggest that by properly adjusting trained modules and prior parameters, we can generate translations which balance accuracy and diversity well. in future work, firstly, since our model is randomly sampled from model distribution to generate diverse translation, it is meaningful to explore better algorithms and training strategies to represent model distribution and search for the most distinguishable results in model distribution. also, we'll try to extend our methods in a wider range of nlp tasks.