D IVERSE M ACHINE T RANSLATION WITH A S INGLE M ULTINOMIAL L ATENT V ARIABLE
Anonymous authors
Paper under double-blind review
A BSTRACT
There are many ways to translate a sentence into another language.
Explicit modeling of such uncertainty may enable better model ﬁtting to the data and it may enable users to express a preference for how to translate a piece of content.
Latent variable models are a natural way to represent uncertainty.
Prior work investigated the use of multivariate continuous and discrete latent variables, but their interpretation and use for generating a diverse set of hypotheses have been elusive.
In this work, we drastically simplify the model, using just a single multinomial latent variable.
The resulting mixture of experts model can be trained efﬁciently via hard-EM and can generate a diverse set of hypothesis by parallel greedy decoding.
We perform extensive experiments on three WMT benchmark datasets that have multiple human references, and we show that our model provides a better trade-off between quality and diversity of generations compared to all baseline methods.
I NTRODUCTION
Modeling uncertainty in the output distribution is a fundamental problem in machine learning.
Some prediction tasks, in particular those involving text generation, may admit many valid outputs.
However, in most cases the learner only observes one out of many correct outputs both at training and test time.
This raises questions about how to properly ﬁt the data distribution and how to properly measure such ﬁtting.
In this work, we investigate these questions for the task of machine translation (MT), however the arguments and methods are general and can equally be applied to other models and tasks.
Speciﬁcally, we aim at better modeling uncertainty for the purpose of effectively decoding a diverse set of hypotheses.
The motivation is twofold.
On one hand, we are interested in exploring the problem of modeling uncertainty, and we would like to better understand how different latent variable models capture uncertainty.
MT serves as a well deﬁned task for this investigation.
On the other hand, we are driven by the potential practical applications of such models for MT, which could enable end-users to express their preference about how to translate a piece of text.
Previous work has explored approaches for heuristically modifying the decoding algorithm to produce diverse hypotheses, e.g., by enforcing diversity constraints during beam search (Li & Jurafsky, 2016; Vijayakumar et al., 2016).
Neverthless, it seems more natural to consider explicit ways to represent the multi-modal nature of this task by introducing latent variables in the model.
Notably, Zhang et al.
(2016) proposed a variational version of NMT with a latent Gaussian variable (Kingma & Welling, 2013).
While they showed improvements due to the regularization effect of the Monte Carlo gradient estimate, it is unclear how well the corresponding latent space actually captures different output modes.
We propose a much simpler approach that introduces a single multinomial latent variable to capture uncertainty in the target sentence.
This model, which is a simpliﬁed instance of a mixture of experts (MoE) model (Jacobs et al., 1991), is much easier to train and interpret than previous approaches.
Our approach also enables fast parallel decoding from different latent values to generate a diverse set of hypotheses.
We evaluate our method on three WMT benchmark datasets that uniquely provide test sets with multiple human references.
These datasets enable us to quantitatively evaluate diversity, and we provide the ﬁrst systematic comparison of the various methods that produce multiple hypotheses.
Our results demonstrate that our proposed model is more robust and generates both high-quality and diverse translations, and compares favorably to variational NMT models, other variants of mixture of expert models, and diverse inference algorithms such as diverse beam search (Vijayakumar et al., 2016) and biased sampling (Graves, 2013; Fan et al., 2018).
Moreover, our study provides insights for best practices to avoid several failure modes of latent variable models, in particular when these are trained with modern architectures using dropout and shared parameters.
D IVERSE D ECODING IN NMT
Let x be a source sentence consisting of words ( x 1 , · · · , x L ) , and y =
( y 1 , · · · , y T ) a given reference (target) translation.
An NMT model has an encoder-decoder structure.
The encoder maps ( x 1 , · · · , x L ) to a sequence of representations h
= ( h 1 , · · · , h L ) that are then fed to the decoder, which generates an output sentence one word at a time conditioned on h .
At each time step, the decoder additionally conditions its output on the previous outputs, resulting in an auto-regressive factorization p ( y | x ; θ ) =
(cid:81)
Tt =1 p ( y t | y 1: t − 1 , x ; θ ) , where θ denotes the parameters of the network and is learned by maximum likelihood training.
At test time, approximate inference algorithms such as beam search are commonly used to decode the most likely translations.
Beam search decodes a sequence left-to-right while maintaining the
K best hypotheses at each step.
Although it is often effective in ﬁnding high quality hypotheses, these hypotheses typically have low diversity with only minor differences in the sufﬁx.
To remedy this, Vijayakumar et al.
(2016) proposed diverse beam search , where they perform beam search sequentially for G iterations, penalizing the selection of words used in previous iterations.
Another decoding strategy is to sample repeatedly from the model’s conditional distribution ˆ y t ∝ p ( ·| ˆ y 1: t − 1 , x ; θ ) .
Unfortunately, this approach usually produces diverse but low quality outputs.
Several heuristic methods have been proposed to obtain better quality, such as sampling with a temperature, biased sampling from the top k most likely words instead of all words at each step (Graves, 2013; Fan et al., 2018), oversampling K (cid:48) >
K outputs and retaining the top K , adding noise to the hidden state (Cho, 2016), etc.
L ATENT V ARIABLE NMT
Rather than search for diverse outputs at test time, which may be slow and difﬁcult to tune, we aim to explicitly model uncertainty during training.
We introduce a multinomial latent variable z taking values in { 1 , · · · , K } to capture various ways to translate the source sentence, and train a latent variable model p ( y | x ; θ )
=
(cid:80) Kz
=1 p ( z | x ; θ ) p ( y | z, x ; θ ) .
Intuitively, we want to push the uncertainty from p ( y | x ) into z , so that we can easily explore multiple modes during training and inference.
For each value of z , let us consider p ( y | z, x ; θ ) to be an expert and p ( z | x ; θ ) to be a gating function over experts, i.e., a source-dependent prior of using expert z .
Training proceeds by minimizing the negative log-likelihood loss function:
[Equation]
This is the Soft-Mixture of Experts (Soft-MoE) model.
In our work, the encoder and experts are based on the Transformer architecture (Vaswani et al., 2017), while the gating is implemented via mean-pooling over the encoder’s representations followed by a single layer feed-forward net to predict p ( z | x ; θ ) , similar to related work (Zhang et al., 2016).
3. O UR A PPROACH : H ARD -M O E
A well-known challenge in training Soft-MoE models is that experts “die” during training; in extreme cases the model collapses and only a single expert is trained (Eigen et al., 2014;
Shazeer et al., 2017b).
We address this by introducing two simplifying assumptions.
First, we assume that the prior p ( z | x ) = 1 /K is uniform and source independent.
This encourages the model to use multiple experts instead of collapsing into one with extreme p ( z | x ) values.
Second, we assume that a particular
Algorithm
Initialize model parameters θ repeat
Sample a mini-batch of m examples { ( x ( i ) , y ( i ) ) } mi =
Set the model in evaluation mode, turn off dropout and disable gradient computation for z in { 1 , · · · , K } do
Compute − log p ( y ( i ) | z, x ( i ) ; θ ) end for
Let z ( i ) =
arg min z − log p ( y ( i ) | z, x ( i ) ; θ )
Set the model in train mode, turn on dropout and enable gradient computation
Do another forward-backward pass to compute loss L = 1 m (cid:80) mi =1 − log p ( y ( i ) |
z ( i ) , x ( i ) ; θ ) and update θ by gradient descent until Convergence translation y is well explained by exactly one expert, that is p ( y
| z, x ) is large for only one value of z .
This forces experts to specialize on different subsets of the data.
Based on these assumptions, we can approximate the summation in Eq.
1 with the maximum term:
[Equation]
Taking the negative logarithm, and omitting the constant log K term, we get our loss function:
[Equation]
To train with this objective, for each mini-batch we enumerate all K possible values of z (typically
K = 10 ) and compute their losses: − log p ( y | z, x ; θ ) .
Next, for each ( x, y ) pair we select the value of z that yields the minimum loss, and use that value to back-propagate, i.e., only one expert receives gradients per sentence pair.
Since we make hard selections of the latent variable, we call this method Hard-Mixture of Experts (Hard-MoE).
This training algorithm can also be seen as a stochastic gradient descent version of the Hard-Expectation Maximization algorithm.
3. P RACTICAL C ONSIDERATIONS
So far our discussion has been very general.
Notice however that our motivation for training with a mixture of experts is different from existing MoE literature that typically aims to increase model capacity (Collobert et al., 2003;
Shazeer et al., 2017b; Gross et al., 2017).
Instead, we aim to better model multi-modal output distributions—a difference that has signiﬁcant practical implications.
First, even on large scale MT benchmarks, models are prone to overﬁt and require regularization (e.g., dropout, label smoothing, etc.); the standard modeling choice of using independently parameterized (decoder) experts is likely to exacerbate this overﬁtting.
Given our goal is not to increase model capacity, we choose instead to share almost all parameters among the experts, except for a single set of weights that we use to embed the different values of the latent variable.
Accordingly, we require only a negligible number of additional parameters over a baseline model.
More specifically, we found good results in pilot experiments by simply replacing the ﬁrst dummy beginningof-sentence or (cid:104) bos (cid:105) token of the decoder with the embedded representation of our latent variable.
This modeling choice further mitigates the latent variable collapse issue often reported in the MoE literature (Eigen et al., 2014;
Shazeer et al., 2017b), in which low quality experts are neglected and eventually “die” during training.
Instead, by sharing parameters, even unpopular experts receive some gradients throughout training.
Another potential problem is with dropout.
Dropout noise can mask the effect of the change of the latent variable, making assignments inconsistent and eventually leading to a degenerate model [Figure] that ignores the latent variable.
Therefore, in practice we decompose the minimization over z in
Eq.
3 into two separate steps.
First we set the model in evaluation mode, turn off dropout, disable gradient computation and perform K forward passes to identify the optimal assignment of z .
Once the optimal latent values are selected, we switch to train mode, turn on dropout, and do another forward-backward pass to update the parameters.
This is crucial; in Sec.
5.4 we show that if we instead leave dropout on during minimization, then the latent variable is ultimately ignored.
The whole process is formally described in Algorithm 1.
All the decoding strategies for p ( y | x ; θ ) mentioned in Sec.
2 can be applied to the latent variable model p ( y | z, x ; θ ) as well.
We adopt the most straightforward one: generating K hypotheses by ﬁrst enumerating z and then greedily decoding ˆ y t = arg max y p ( y | ˆ y 1: t − 1 , z, x ; θ ) .
This tests whether our latent variable captures different modes in p ( y | x ) , and we can generate hypotheses efﬁciently in parallel.
M ETHODOLOGY TO A SSESS Q UALITY AND D IVERSITY
In this section we describe the metrics we use to quantitatively assess the quality and diversity of a set of translation hypotheses.
We use BLEU (Papineni et al., 2002) as a basis for measuring the similarity of an ordered sentence pair ( u, v ) , where u is the reference and v is the hypothesis.
Suppose { y 1 , · · · , y N } are N reference translations of a source sentence x , and { ˆ y 1 , · · · , ˆ y K } are
K hypotheses.
We compute the following metrics (Ott et al., 2018a):
• Average oracle BLEU:
We pair each hypothesis ˆ y k with its best matching human reference that gives the highest sentence-level 4 BLEU: a k = arg max j BLEU ( y j , ˆ y k ) .
Then we compute the corpus-level BLEU of { ( y a k , ˆ y k ) } x ∈ data ,k ∈
[ K ] .
This measures the overall quality of a hypothesis set.
If this metric scores low, it implies that some generated hypotheses do not match any reference and the generation quality is poor, • Reference coverage:
We pair each hypothesis to its best matching reference (breaking ties randomly), count how many distinct references are matched to at least one hypothesis, and average this count over all sentences in the test set.
A low coverage number indicates that all hypotheses are close to a few references.
Instead, we would like a diverse set that covers most of the references.
• Pairwise BLEU:
We compare all hypotheses against each other, and compute the corpuslevel BLEU of { (ˆ y j , ˆ y k ) } x ∈ data ,j ∈ [ K ] ,k ∈
[ K ] \ j .
This metric measures the similarity among the hypotheses.
The more diverse the hypothesis set, the lower the pairwise BLEU.
Ideally, we would like a model with pairwise BLEU matching human pairwise BLEU.
In addition, we compute the average oracle BLEU and pairwise BLEU of human translations for comparison.
The average oracle BLEU is computed in a leave-one-out manner.
5 Namely, each y n is paired with the best of the rest: b n = arg max j (cid:54) =
n BLEU ( y j , y n ) , and we compute the corpuslevel BLEU of { ( y b n , y n ) } x ∈ data ,n ∈ [ N ] .
A toy illustration of how these metrics behave in different situations is shown in Fig.
E XPERIMENTS 5. D ATASETS
We test our latent variable models and baselines on three WMT datasets that have multiple human references (Ott et al., 2018a; Hassan et al., 2018).
We also use the small IWSLT’14 De-En dataset (Cettolo et al., 2014) for pilot experiments.
While IWSLT’14 does not have multiple references, pairwise BLEU lets us test whether the outputs are diverse or not.
WMT’17 English-German (En-De): We train on all available bitext excluding the ParaCrawl corpus and ﬁlter sentence pairs that have source or target longer than 80 words, resulting in 4.5M sentence pairs.
We use the Moses tokenizer (Koehn et al., 2007) and learn a joint source and target
Byte-Pair-Encoding (Sennrich et al., 2016) with 32K types.
We develop on newstest2013 and test on a 500 sentence subset of newstest2014 that has 10 references (Ott et al., 2018a).
WMT’14 English-French
(En-Fr): We borrow the setup of Gehring et al.
(2017) with 36M training sentence pairs and 40K joint BPE vocabulary.
We validate on newstest2012+2013, and test on asentence subset of newstest2014 that has 10 references (Ott et al., 2018a).
WMT’17 Chinese-English (Zh-En): We pre-process the training data following Hassan et al.
(2018) which results in 20M sentence pairs, 48K and 32K source and target BPE vocabularies respectively.
We develop on devtest2017 and report results on newstest2017 with 3 references.
IWSLT’14 German-English (De-En): We follow the setup of Edunov et al.
(2018) that has 160K training sentence pairs and 14K joint BPE vocabulary.
The test set here has only one reference.
5. E XPERIMENTAL D ETAILS
We build on the Transformer (Vaswani et al., 2017) implementation in the Fairseq toolkit.
The encoder and decoder have 6 blocks.
The number of attention heads, embedding dimension and innerlayer dimension are 16, 1024, 4096 on the WMT datasets and 4, 512, 1024 on the IWSLT dataset.
For variational NMT (Zhang et al., 2016) we use a Gaussian latent variable with K dimensions matching the number of categories of a multinomial latent variable (typically K = 10 ).
For baselines involving extra hyper-parameters such as diverse beam search (Vijayakumar et al., 2016) and top- k sampling, we perform grid search over these hyper-parameters and report the best result.
Models are optimized with Adam (Kingma & Ba, 2015) using β 1 = 0 .
9 , β 2 = 0 .
98 , and (cid:15) =
1 e −
and we use the same learning rate schedule as Ott et al.
(2018b).
We run experiments on DGX-1 machines with 8 Nvidia V100 GPUs and machines are interconnected by Inﬁniband.
We train models with 16-bit ﬂoating point operations and mini-batches of approximately 400K tokens for the WMT datasets and 25K tokens for IWSLT, following Ott et al.
(2018b).
5. R ESULTS
Table 1 summarizes our main results on three multi-reference large scale datasets, comparing our
Hard-MoE model with baselines that do not use a latent variable.
To better visualize these numbers, we plot pairwise BLEU in reverse order vs. average oracle BLEU in Fig.
The ideal algorithm generates diverse and high-quality translations, i.e., low pairwise BLEU (diverse) as well as high average oracle BLEU (high quality).
First, we observe that beam search’s average oracle BLEU is fairly close to human’s average oracle BLEU, implying a remarkably high generation quality.
However, it severely lacks diversity, as shown by the high pairwise BLEU near 80.
This suggests a pairwise BLEU avg oracle BLEU
#refs covered en-de en-fr zh-en en-de en-fr zh-en en-de en-fr zh-en
Sampling
Beam 73 .77 .
Diverse
[Figure] [Figure]
Human 35 .46 .25 .
[Figure] [Figure]
WMT’
[Figure] [Figure] [Figure] [Figure]
scenario similar to Fig.
1-B. In contrast, sampling is very diverse, typically even more so than human references on En-De and En-Fr, but has a very poor average oracle performance, as illustrated in Fig. 1-A. Diverse beam search increases the diversity of beam search to some extent, but pays a cost in translation quality.
Overall, Hard-MoE achieves the best trade-off between quality and diversity, measured in terms of both pairwise BLEU and reference coverage.
We also explore the impact of varying K on the WMT En-De dataset (Fig. 2, left ).
In general, when more hypotheses are generated (e.g., K = 20 ), they become more diverse but of worse quality.
The complete results from these experiments and the sampling top-
k baseline are provided in Table 5 in the appendix.
Finally, we observe that despite the improvement offered by Hard-MoE, there is still a large gap between the diversity of human references and system generations.
5. D ETAILED C OMPARISONS AND A BLATION S TUDY
In this section we compare the Hard-MoE model to other latent variable models, including variational NMT and Soft-MoE, on the IWSLT’14 De-En dataset with K = 2 .
As Table 2 shows, none of these alternatives is able to make effective use of the latent variable.
Variational NMT ignores the latent variable, demonstrated by its extremely high pairwise BLEU, which is a well-known failure of VAEs on text (Bowman et al., 2016).
7 Next, we tried several variants of Soft-MoE. If expert decoders have separate parameters (Lee et al., 2016), the model collapses and uses only one expert.
This is due to the “rich gets richer” effect, whereby only those experts that already explain the targets well receive gradients, causing the other experts to starve during training and eventually die.
While there are some “re-balancing” heuristics to alleviate these issues (Eigen et al., 2014; Shazeer et al., 2017a), training is difﬁcult and collapses are often still experienced.
If we instead share the encoder and decoder parameters and disable dropout, Soft-MoE works but severely overﬁts, resulting in low translation quality (low BLEU).
Finally, if we enable dropout during the latent variable selection process, then the model again gets 99 pairwise BLEU.
We surmise that this is due to dropout noise masking the effect of selection of different latent values. This pre- [Table]
[Table] vents the gating from training well and the latent variable embeddings from specializing [Table] causing the model to essentially ignore the latent variable.
We conﬁrm this empirically in the appendix.
5. Q UALITATIVE A NALYSIS
In this section we perform a qualitative analysis with the WMT’17 Zh-En dataset, which contains three reference human translations for each source sentence.
In Table 3 we show two source sentences, the corresponding reference translations
[Table]
and generated hypotheses for several approaches.
We observe that beam search tends to produce generations which only differ in the last few words.
Diverse beam search improves the diversity over beam search, but is not as diverse as Hard-MoE and may produce duplicate hypotheses (e
[Table]
g., if the diversity penalty is not sufﬁciently high).
Hard-MoE shows signiﬁcant diversity in wording, word order, clause structure, etc.
To investigate whether the latent variable in Hard-MoE learns different translation styles, we examine the hypotheses that each latent variable assignment generates.
For each value of z , we compute word frequencies of the corresponding generations and look for words whose frequency is signiﬁcantly different as we change the value of the latent variable.
We ﬁrst discover that for words like was , were and had , z = 1 ’s frequency is more than three times higher than z = 3 ’s; conversely, for has and says , z = 3 ’s frequency is more than twice higher than z = 1 ’s. Since Chinese does not have tense unless a time phrase is explicitly stated, we speculate that when translating into
English, the ﬁrst latent value tends to translate with past tense whereas the third latent value tends to translate with present tense.
Indeed we ﬁnd that this is a consistent behavior, as seen from the ﬁrst four examples in Table 4.
Similarly, we ﬁnd that different latent values exhibit different preferences for using this or that (see the fourth and ﬁfth examples in Table 4), % or per cent (see the last example in Table 4 and the ﬁrst example in Table 3), and so on. [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table]  R ELATED W ORK The training objective of our Hard-MoE approach is reminiscent of the Multiple Choice Learning (MCL) setting for ensemble learning (Guzman-Rivera et al., 2012), where the oracle loss is minimized for an ensemble of learners.
Lee et al.
(2016) develop this idea and propose a SGD based algorithm to train an ensemble of deep networks for object recognition tasks.
In another work, Lee et al.
(2015) introduce a tree structure parameter sharing of a CNN ensemble.
Our method shares the same motivation and underlying algorithm, but the architecture and the task are very different.
Moreover, our goal is to produce a diverse and high-quality set of translations, which we evaluate using multiple reference human translations and several quantitative metrics, as opposed to just evaluating the best matching hypothesis.
There have been several prior studies investigating uncertainty in machine translation.
Dreyer &
Marcu (2012) and Galley et al. (2015) introduced new metrics to address uncertainty at evaluation time.
Ott et al.
(2018a) inspected the sources of uncertainty and proposed tools to check ﬁtting between the model and the data distributions.
They also observed that modern conditional autoregressive NMT models do capture uncertainty, but only to a certain extent, and they tend to oversmooth probability mass over the hypothesis space.
In addition to the variational NMT model (Zhang et al., 2016), there are two extended models introducing latent variables in NMT.
Schulz et al.
(2018) considered a sequence of latent Gaussian variables to represent each target word in an auto-regressive fashion, as opposed to one latent variable representing the whole target sentence.
Kaiser et al.
(2018) proposed a similar model, except that they used groups of discrete multinomial latent variables.
In their qualitative analysis, Kaiser et al.
(2018) showed that the latent codes do affect the output predictions in interesting ways, but their focus was on speeding up regular decoding rather than producing a diverse set of hypotheses.
Concurrent to our work, He et al. (2018) consider the same problem and a similar approach to train a Soft-MoE model. They propose to sample a latent value from the posterior distribution to estimate the gradients efﬁciently, as opposed to we select a latent value via minimization as done in this work. Moreover, they do not consider evaluation datasets with multiple references and use weaker baseline models, which makes their empirical analysis less conclusive.
Besides machine translation, there are several works introducing latent variables for dialogue generation (Serban et al., 2017; Cao & Clark, 2017; Wen et al., 2017) and image captioning (Wang et al., 2017; Dai et al., 2017).
Our method departs from these VAE or GAN-based approaches and importantly, is much simpler.
It could also be applied to other text generation tasks as well.
C ONCLUSION
We propose a hard mixture of experts model to generate a diverse and high-quality set of hypotheses for machine translation.
This model is very simple and it can be used as a strong baseline for other approaches involving latent variables.
Although mixture of experts models are powerful, they are very brittle at training and are prone to several failure modes, such as using only a small subset of experts or learning undifferentiated experts, particularly when working with large, heavily regularized models as in NMT.
Our Hard-MoE approach makes several simplifying assumptions on top of the baseline MoE algorithm, which effectively mitigates these kinds of failures.
We report results from extensive experiments on three large-scale datasets with multiple references, and demonstrate that Hard-MoE achieves the best trade-off between quality and diversity.
Still there is a big gap between our model performance and human performance.
In the future, we would like to build upon this work to investigate models with richer and more structured latent representations
A D ETAILED R ESULTS
[Table]
Filtered sampling (top-
[Table]
In Table 5 we compare different approaches for generating diverse translations on the WMT’17 EnDe dataset.
We additionally compare each approach as we vary the quantity of desired translations ( K ).
We observe that sampling produces diverse but low quality outputs.
We can improve translation quality by restricting sampling to the top- k candidates at each output position ( k = 2 performed best), but translation quality is still worse than hard-MoE. Beam search produces the highest quality outputs, but with low diversity.
Diverse beam search provides a reasonable balance between diversity and translation quality, but the hard-MoE approach produces even more diverse and better quality translations.
Finally, besides unrestricted sampling, hard-MoE covers the largest number of references of any of the approaches evaluated.
B E FFECTS OF D ROPOUT D URING M INIMIZATION
[Figure]
In Section 5.4 we observed that it is crucial to turn off dropout when performing the minimization over z .
We speculate that the reason is that dropout noise dilutes the impact of the latent variable, causing the minimization process to select among the latent values at random.
Then it prevents different latent values from specializing and ultimately causes the model to ignore them.
To test this hypothesis, we show in Fig.
3 the effect of dropout noise on the minimization process at the beginning of training (i.e., with a randomly initialized model).
On the y-axis we plot the probability that the optimal value of z changes after applying dropout with different probabilities.
We see that as we increase the dropout probability, the optimal value of z is quickly corrupted— even with a dropout probability of 0.1 we observe a 45% chance that the optimal assignment of z changes.