M INIMUM D IVERGENCE VS . M AXIMUM M ARGIN : AN E MPIRICAL C OMPARISON ON S EQ 2S EQ M ODELS
A BSTRACT
Sequence to sequence (seq2seq) models have become a popular framework for neural sequence prediction.
While traditional seq2seq models are trained by Maximum Likelihood Estimation (MLE), much recent work has made various attempts to optimize evaluation scores directly to solve the mismatch between training and evaluation, since model predictions are usually evaluated by a task speciﬁc evalu ation metric like BLEU or ROUGE scores instead of perplexity
This paper puts this existing work into two categories , a minimum divergence and b ) maximum margin
We introduce a new training criterion based on the analysis of existing work, and empirically compare models in the two categories.
Our experimental results show that our new training criterion can usually work better than existing methods, on both the tasks of machine translation and sentence summarization
I NTRODUCTION
Sequence to sequence (seq2seq) models (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014;
Bahdanau et al., 2015; Luong et al., 2015) are a powerful end-to-end solution to sequence prediction since they allow a mapping between two sequences of different lengths.
However, Maximum
Likelihood Estimation (MLE) which is used as a standard training criterion in seq2seq has a series of drawbacks:
• Training and evaluation mismatch: during training, we maximize the log likelihood, while during inference, the model is evaluated by a different metric such as BLEU or ROUGE; • MLE fails to assign proper scores to different model outputs, which means that all incorrect outputs are treated equally.
Similar to SEARN (Daum´e et al., 2009) and DAGGER (Ross et al., 2011) in traditional models, many reinforcement learning (RL) techniques have been extended to seq2eq models recently to solve the mismatch between training and inference.
The key idea is to replace MLE with a criterion that gives task-speciﬁc scores to different model outputs.
Recent approaches to taking the evaluation metric into training criteria generally fall into one of the following two categories:
minimum divergence : Minimum divergence methods minimize some distance functions between the model output distribution and the probability distribution given by the ground truth, and by optimizing the divergence between the two distributions, the model output distribution will ideally be equal to the ground truth distribution.
maximum margin : During inference, in structured prediction, the goal is to simply ensure good structures have the highest predicted score.
That is, margin based learning differs from minimum divergence learning in the aspect that it only requires that the predicted score of the correct output should be higher than that of the others by a margin.
The main contribution of this paper can be summarized as: based on the analysis of existing work, we introduce a novel training criterion for seq2seq learning and show its effectiveness.
Based on our experimental results on the tasks of machine translation and sentence summarization, we conclude that some models that take the evaluation metric into consideration can improve over a strong MLE baseline by a large margin, and our new training criterion can usually get better results than existing work.
R ELATED W ORK
Seq2seq models were proposed and improved by Kalchbrenner & Blunsom (2013) and Sutskever et al.
With the attention mechanism (Bahdanau et al., 2015; Luong et al., 2015), seq2seq models achieve better results than traditional models in many ﬁelds including machine translation (Bahdanau et al., 2015; Luong et al., 2015) and text summarization (Chopra et al., 2016).
Before seq2seq models, previous work on optimizing task speciﬁc evaluation metric scores generally includes SEARN (Daum´e et al., 2009), DAGGER (Ross et al., 2011), Minimum Error Rate Training (MERT) (Och, 2003), and softmax-margin criterion (Gimpel & Smith, 2010).
Seq2seq model uses MLE as the training objective, which also has the problem of training and evaluation mismatch.
Ranzato et al.
(2016) incorporated the evaluation metric into training of seq2seq models and proposed the mixed incremental cross entropy REINFORCE (MIXER) training strategy, which is similar to the idea of Minimum Risk Training (MRT) (Smith & Eisner, 2006; Li & Eisner, 2009; Ayana et al., 2016;
Shen et al., 2016).
MIXER used decoder hidden states to predict the bias term to reduce the variance, while MRT renormalized the predicted probabilities.
Inspired by MIXER, Bahdanau et al.
(2017) extended the actor critic algorithm (Konda & Tsitsiklis, 2000; Sutton et al., 2000) to the seq2seq framework.
Sokolov et al.
(2016) and Kreutzer et al.
(2017) proposed the pairwise preference learning framework, which enables the model to learn from the difference between a pair of samples generated by the model.
Another popular model in the form of divergence minimization is the smoothed version of MERT algorithm proposed by Och (2003).
The smoothed MERT algorithm also minimizes the expected loss over samples from model outputs and is similar to MRT essentially.
Norouzi et al.
(2016) proposed Reward Augmented Maximum
Likelihood (RAML) which also optimizes the divergence by data augmentation, and RAML was further combined with maximum likelihood via α -divergence recently (Koyamada et al., 2017).
In contrast to the above divergence minimization based approaches, Wiseman & Rush (2016) used a margin based training objective and their search optimization process requires performing parameter updates or inference each time a new word is generated.
One main drawback of maximum margin is that given a sample subset of an arbitrary size, only two samples of the subset can contribute to the gradient.
The softmax-margin criterion proposed by Gimpel & Smith (2010) can alleviate this problem, in which all samples are used in loss and gradient computation.
However, Edunov et al.
(2018) shows that the softmax-margin criterion still cannot reach the performance of risk minimization.
As the latest work, Edunov et al.
(2018) is very similar to ours.
However, our interests are in ﬁnding better training criteria on a consistent setting, while they mainly focused on absolute performance improvement, using techniques like combination with token level loss, online vs. ofﬂine candidate generation, etc.
S EQUENCE TO S EQUENCE M ODEL
This paper considers the seq2seq model from Bahdanau et al.
and Luong et al.
(2015), which consists of an RNN encoder and an RNN decoder.
Due to the gradient vanishing problem, the vanilla RNN cell is replaced with a Gated Recurrent Unit (GRU) (Cho et al., 2014) or Long ShortTerm Memory (LSTM) (Hochreiter & Schmidhuber, 1997).
A bidirectional RNN is used as encoder, with one RNN reading the source sequence from beginning to end, and the other RNN reading in reversed order.
The bidirectional RNN encoder encodes the input sequence and gets a sequence of hidden states by concatenating the hidden states of each word in the two directions, which will be used as attention information in the decoder RNN.
The decoder RNN is initialized by passing the two ﬁnal hidden states of the encoder to a feed forward network.
The mixed information used as input in the ﬁrst step m 0 is initialized by a zero vector.
In decoding step i , using the input feeding approach described in Luong et al. (2015), the decoder
RNN reads the embedding of the word w i and the mixed information m i − 1 , to generate the hidden state of current step h i :
[Equation]
where f 1 is the RNN cell.
The decoder then generates the attention context c i from encoder using h i .
Since there are many different attention mechanisms, we omit the details here.
After getting c
i
, the model mixes c i and h i together by a feedforward network f 2 to generate m i , which is used as the mixed information in the next step:
[Equation]
Finally, m i is passed to a feedforward network to generate o i , and the softmax layer outputs the probability for each word at step i .
Suppose the target word in step i is v (cid:48) , the probability of the target word in step i is given by
[Equation]
For the sequence y = (cid:104) y 1 , y 2 , ..., y l (cid:105) , the probability p ( y | x ) is p ( y | x )
= p ( y 1 | x ) p ( y 2 | x , y 1 ) ... p ( y l | x , y 1: l − 1 ) .
Suppose y ∗ is the correct output, MLE actually maximizes the log likelihood of p ( y ∗ | x ) :
[Equation]
For a detailed explanation, please refer to Luong et al.
Now we describe the algorithm to construct a subset S from all possible output sequences which will be used in Section 4 & 5.
In seq2seq model, the n-best list can only be approximated by performing beam search, as used in
Wiseman & Rush (2016), or Monte Carlo (MC) sampling, as used in Ayana et al.
(2016) and Shen et al.
While using beam search may produce more accurate results, the time required for generating a candidate set of the same size as MC sampling using beam search is too long.
In this paper, we construct the approximated n-best list S using MC sampling proposed in Shen et al.
(2016) , and for completeness, we describe the sampling approach in Appendix 8.1 in details.
Please note that the sampling algorithm does not guarantee the size of the sample subset, since it is inefﬁcient to force the model to generate an exact number of distinct samples using GPU.
M INIMUM D IVERGENCE
In this section, we consider training criteria in the form of the following equation:
[Equation]
where p 1 and p 2 are two probability distributions given by the model or the ground truth, and D is a distance function, for example, Kullback Leibler (KL) divergence or cross entropy.
The baseline training criterion MLE can also be seen as minimum divergence training, where the cross entropy between the model output distribution and the one-hot encoded ground truth is minimized.
Now we will discuss training criteria in which the ground truth distribution contains information of evaluation metric scores.
4. R EWARD A UGMENTED M AXIMUM L IKELIHOOD
Suppose that { x , y ∗ } is a given input-output pair, the model learns parameters θ and gives prediction p ( y | x ; θ ) on the output space Y , the exponentiated payoff distribution (Norouzi et al., 2016)
q ( y | y ∗ ) is deﬁned as follows:
[Equation]
where Z ( y ∗ )
= (cid:80)
y ∈ Y exp
[ r ( y , y ∗ )
/τ ] is the normalization factor.
τ is a hyperparameter that controls the smoothness of the distribution, and r ( y , y ∗ ) is the score of prediction y given by the evaluation metric, e.g. BLEU (Papineni et al., 2002), or ROUGE (Lin, 2004)
The training objective of Reward augmented Maximum Likelihood (RAML)
(Norouzi et al., 2016) is deﬁned as
[Equation]
in which, the gradient is:
[Equation]
Equation (3) requires computing q ( y | y ∗ ) for all y and performing gradient descent on the whole Y .
However, due to the huge search space of Y and the high computational cost of r ( y , y ∗ ) , Equation (3) cannot be directly computed in an efﬁcient way.
Norouzi et al.
(2016) proposed using stratiﬁed sampling.
One can sample from negative edit distance or Hamming distance and then do importance reweighting to reweight the samples with task speciﬁc scores.
After sampling from q ( y | y ∗ ) , the model is trained to maximize log p ( y | x ; θ ) .
Another choice is to construct a sample subset S from the model outputs using MC sampling, and use Equation (1) to compute the probability q (cid:48)
( y | y ∗ ) in the sample subset.
The training criterion is then changed to
[Equation]
where
[Equation]
4. M INIMUM R ISK T RAINING
We ﬁrst introduce the REINFORCE algorithm (Williams, 1992), which is a famous policy gradient method in RL, and then compare it with the Minimum Risk Training (MRT) (Smith & Eisner, 2006;
Li & Eisner, 2009; Ayana et al., 2016;
Shen et al., 2016).
In REINFORCE, given a state x , an action y under state x , and model parameters θ , the policy gradient g is
[Equation]
where r ( y ) is the reward of action y , and B ( x ) is the baseline or control variate that is used to reduce the variance (without introducing bias).
In MRT, the training objective is the expectation of evaluation metric scores taken with respect to the model output distribution:
[Equation]
Also due to the huge search space, Equation (5) can only be estimated by sampling from the model outputs.
Suppose S is a sample subset from Y , p ( y | x ; θ ) is replaced by p (cid:48)
( y | x ; θ ; α ) :
[Equation]
where α is a hyperparameter that controls the smoothness of p (cid:48)
( y | x ; θ ; α ) .
We will later compare the effect of α here with τ in RAML.
The gradient of L MRT can be derived as:
[Equation]
B ) ] , with
[Equation]
If we regard the whole seq2seq framework as a one-step Markov Decision Process (MDP), in which the input sequence x is the state, the output sequence y is the action, and the evaluation metric score r is the reward, both REINFORCE and MRT actually optimize the same training criterion.
The difference between them is that REINFORCE requires only a single sample and is unbiased, while
MRT has multiple samples and the probability renormalization in Equation (6) introduces bias.
The expected loss objective in Kreutzer et al.
(2017) is also similar (and closer) to REINFORCE which uses one sample to estimate the gradient.
4. C OMPARISON OF RAML & MRT
The cross entropy is deﬁned as:
[Equation]
Starting from RAML, in terms of cross entropy, Equation (2) can be rewritten as:
[Equation]
The training criterion of RAML is essentially minimizing the the cross entropy loss from q ( y | y ∗ ) to p ( y | x ; θ ) .
Meanwhile, the training criterion of MRT in Equation (5) can be rewritten as:
[Equation]
in which, the second term is a constant as it has no gradient with respect to θ .
Thus we have:
[Equation]
Equation (9) actually shows that MRT minimizes the cross entropy loss from p ( y | x ; θ ) to q ( y | y ∗ ) .
By comparing the above two methods, we ﬁnd that both RAML and MRT are minimizing the cross entropy loss between the model output distribution and the exponentiated payoff distribution , but with different directions of D XENT .
Now we will discuss the effect of hyperparameter τ in RAML and α in MRT.
Cross entropy loss has a property that it is minimized if and only if the two probability distributions are equal.
Ideally, when the cross entropy loss in RAML is minimized, we have:
[Equation]
And in MRT 1 ,
[Equation]
When the two probability distributions are equal, we have:
[Equation]
During training of both RAML and MRT, the model is trying to predict the evaluation score r by multiplying log p with a constant (either τ in RAML or α in MRT).
4. H ELLINGER D ISTANCE M INIMIZATION
Now we introduce another popular distance metric, the Hellinger distance (Hellinger, 1909), which also belongs to the family of Csisz´ar f-divergence (Csisz´ar, 1963) and has been used in graphical models (Beykikhoshk et al., 2015) and clustering algorithms (Abdel-Azim, 2016; Ji et al., 2018).
Given two probability distributions p 1 and p 2 , let f and g denote the probability density functions of p 1 and p 2 .
The square of the Hellinger distance H ( p 1 , p 2 ) is deﬁned as:
[Equation]
The training criterion using squared Hellinger distance is:
[Equation]
with p (cid:48)
( y | x ; θ ) deﬁned in Equation (6) and q ( y | y ∗ ) deﬁned in Equation (1).
M AXIMUM M ARGIN
Let ∆ denote the loss given by the evaluation metric:
[Equation]
∆ is 0 when y = y ∗ and positive otherwise.
Let F be the predicted score of the model:
[Equation]
Different from minimizing the expected risk, another choice of training criterion is to optimize the evaluation score of the sequence with the highest predicted score:
[Equation]
where
[Equation]
For each input sequence x , we ﬁrst do inference to ﬁnd the output ¯ y that has the highest predicted score, then we train the model to maximize the score of ¯ y given by evaluation metric.
However, directly optimizing the training criterion that includes an arg max is not computationally tractable.
In structured prediction, this problem has been widely discussed in the training process of
Structural Support Vector Machine (SSVM) (Tsochantaridis et al., 2004; Joachims, 2006; Joachims et al., 2009).
Tsochantaridis et al.
(2004) proposed using margin and slack rescaling instead.
Margin Rescaling is based on the following inequality:
[Equation]
Similarly, Slack Rescaling is:
[Equation]
As Equation (13) is a tighter bound on ∆( ¯ y , y ∗ ) , slack rescaling generally gives better results than margin rescaling.
Both Equation (12) and Equation (13) can be optimized by subgradient descent.
Let (cid:98) y denote the sample that gives the max value.
The subgradient for margin rescaling is:
[Equation]
and similarly, the subgradient for slack rescaling is:
[Equation]
Since doing loss augmented inference on such a huge search space is again not realistic, approximate inference is still needed.
Equation (12) also has the name of structured hinge loss when used in
SSVM.
One may have found that Equation (13) is the same as the training criterion in Wiseman &
Rush (2016).
Besides, to make a fair comparison with other training criteria and make it applicable on large datasets, we replace the beam search with sample subset.
E XPERIMENTS
We evaluate the training criteria on two tasks in total.
The ﬁrst task is machine translation on a small dataset to explore the hyperparameter impact.
The second task is sentence summarization using large datasets.
For all the experiments, we use LSTM as RNN cell.
The attention mechanism used in our experiments is global attention with input feeding (Luong et al., 2015).
6. M ACHINE T RANSLATION
Due to space limit, experimental results on hyperparameter effects and training details are provided in Appendix 8.2.1.
Data We use the IWSLT 2014 German-English translation dataset, with the same splits as Ranzato et al.
and Wiseman & Rush (2016), which contains about 153K training sentence pairs, 7K validation sentence pairs and 7K test sentence pairs.
Sentences in the dataset are ﬁrst tokenized and then converted into lowercase.
During training, the maximum length for both inputs and outputs is restricted to 50.
Words that appear less than three times are replaced with a UNK token.
Models The encoder is a single-layer bidirectional LSTM with 256 hidden units for either direction, and the decoder LSTM also has 256 hidden units.
The size of word embedding for both encoder and decoder is 256.
We use a dropout rate (Srivastava et al., 2014) of 0.2 to avoid overﬁtting.
The gradient is clipped when its norm exceeds 1.0 to prevent gradient explosion and stabilize learning (Pascanu et al., 2013).
The batch size is set to 32 and the training set is shufﬂed at each new epoch.
All models are trained with the Adam optimizer (Kingma & Ba, 2015).
The sentence level
BLEU score used as reward or loss during training is smoothed by adding 1 to both numerator and denominator, as suggested by Chen & Cherry (2014).
Results For the two max margin methods, τ is set to 1 .
× 10 − 3 .
We use the sample subset version of
RAML, and τ for RAML is also set to 1 .
× 10 − 3 .
For Hellinger loss, α = 5 .
× 10 − 4 and τ = 0 .
The ﬁnal results on the test set are described in Table 1.
Max margin methods here improve the
MLE baseline less signiﬁcantly than Wiseman & Rush (2016) did, since the search optimization algorithm is not used in our experiments.
The training criterion of squared Hellinger distance gives the highest BLEU score on the test set.
[Table]
6. S ENTENCE S UMMARIZATION
The task of sentence summarization is: given a long sentence or a passage, the model will generate a short sentence that summarizes the long text.
For evaluation, following Rush et al.
(2015), we use the full length F1 variant of ROUGE score (Lin, 2004) as metric.
Training details are described in
Appendix 8.2.2.
Data We use the Gigaword corpus with the same preprocessing steps as in Rush et al.
In the Gigaword dataset, the ﬁrst sentence of each article is regarded as the source sequence, and the title of the article as the target sequence.
The training set consists of 3.7M sequence pairs, and the vocabulary size for both article and title is 50K, with the out-of-vocabulary words replaced by a uniﬁed UNK token.
The whole dataset is then converted into lowercase without tokenization.
During training, we use the ﬁrst 2K sequences of the dev corpus as validation set, and the size of the test set is also 2K. The maximum length for both input and output sequence is restricted to 50.
Models The encoder is a single layer bidirectional LSTM with 1000 hidden units for either direction.
The single layer decoder LSTM also has 1000 dimensions.
The word embedding size for both encoder and decoder is 500.
Dropout is not used in the sentence summarization task, and the gradient is clipped when its norm exceeds 1.0.
The batch size is set to 128.
The training set is shufﬂed at each new epoch, and we prefetch 20 batches and sort them by the length of target sequence to form new batches to speed up training.
We use ROUGE-2 as the reward or loss function.
Results Table 2 shows the results on the validation set and the test set.
We report ROUGE-1,
ROUGE-2, ROUGE-L and ROUGE-S4 score.
MRT outperforms the MLE baseline by a large margin, but is still slightly lower than the newly introduced Hellinger loss.
[Table]
MLE + BS 49.
MRT + BS 50.
Hellinger + BS 51.
D ISCUSSION
For training criteria in the form of divergence minimization , according to our experimental results,
MRT outperforms RAML.
As mentioned before, MRT and RAML minimize the cross entropy in two different directions.
By the deﬁnition of cross entropy loss in Equation (8), it is only minimized when p 1 = p 2 .
Optimizing the cross entropy in either of the two directions both guarantee consistency, i.e., the training procedure can ultimately recover the true probability distribution.
For a model that minimizes D XENT ( p 1 || p 2 ) , the model will sample from p 1 and maximize log p 2 , which means that if x has a high probability in p 1 , it will also have a high probability in p 2 , but if has a low probability in p 1 , p 2 ( x ) may still be high.
Thus in RAML which samples from exponentiated payoff distribution q and maximizes log p , sequences that have a high evaluation score will also have a high predicted probability.
However, samples that have a low q , e.g., a low BLEU or ROUGE, may still have a high p .
In contrast, MRT will ﬁrst sample from model outputs and ensure that a high BLEU or ROUGE score will be assigned to sequences that have a high probability in model output distribution p , at the cost that sequences that have low probabilities in p may still have high evaluation scores.
This can explain the difference between RAML and MRT.
The squared Hellinger distance can be regarded as the squared error between p 12 and q 12 , which is essentially a regression loss, while the cross entropy loss usually works better in classiﬁcation problems.
By optimizing the cross entropy loss, the model is trained to ﬁnd the candidate that has the highest probability and pays less attention to samples that have low probabilities, while by optimizing the squared Hellinger distance, the model learns to predict the evaluation metric score for every sample in the candidate set and pays equal attention to all of them.
Since seq2seq models generate a predicted sequence word by word, the target sequence which has the highest evaluation metric score (sample from q and maximize log p ) or the highest predicted probability (sample from p and maximize log q ) may not be reached by approximate inference algorithms like greedy search or beam search.
In this case, a model trained by optimizing the cross entropy loss may not work as expected, however, a model trained by optimizing the squared Hellinger distance can still make predictions normally since it pays equal attention to samples with low probabilities during training.
For maximum margin methods, slack rescaling is a tighter bound and the experimental result is slightly higher than margin rescaling on our small scale experiments.
However, slack rescaling still cannot achieve the same level of performance of MRT and Hellinger distance.
The reason may be that the upper bound is still not tight enough.
In the training process of maximum margin methods, only the target sequence and a negative sample selected from the candidate set are used to update model parameters, which may also explain their performance. A CKNOWLEDGMENTS
This paper was partially supported by National Key Research and Development Program of China (No. 2017YFB0304100), Key Projects of National Natural Science Foundation of China (U and 61733011), Key Project of National Society Science Foundation of China (No. 15-ZDA041),
The Art and Science Interdisciplinary Funds of Shanghai Jiao Tong University (No. 14JCRZ04).
A PPENDIX 8. T HE S AMPLING A LGORITHM
For completeness, we describe the MC sampling algorithm proposed by Shen et al.
(2016) and
Ayana et al.
(2016) here.
Algorithm
Input: source sequence x , model parameters θ , target sequence y ∗ sample size m , maximum length l
Output:
A set of samples S and corresponding probability P 1: S
= { y ∗ } ; P = { p ( y ∗ ) } ; 2: for i in 1 , ..., m − 1 do 3: sample
a sequence word by word given x and θ , stop sampling when length is l or meeting (cid:104) end-of-sequence (cid:105) symbol, and ﬁnally store the result and probability in y and q respectively 4: if y in S then 5: continue 6: end if 7:
S = S ∪ { y } 8:
P = P ∪ { q } 9: end for 10: return S , P
E XPERIMENTS 8.2. M ACHINE T RANSLATION
The MLE baseline is trained with a learning rate of 3 .
× 10 − 4 .
The model is trained for 20 epochs in total, and is evaluated every 1000 steps on the validation set.
τ for RAML For RAML, since we use BLEU instead of Hamming distance as evaluation metric, we ﬁrst show the BLEU scores on the validation set for hyperparameter τ in Table 3.
RAML-I refers to importance reweighting and RAML-S refers to sample subset RAML, as mentioned in Section 4.1.
Experimental results show that the BLEU score is not very sensitive to hyperparameter τ although τ varies from 1 .
× 10 − 7 to 1 .
× 10 − 1 , and setting τ larger will decrease the BLEU score compared with the MLE baseline.
We also ﬁnd that RAML-S tends to give slightly better results than RAML-I.
The learning rate for RAML is 1 .
× 10 − 6 .
IWSLT 2014 German-English translation evaluated on validation set using RAML criterion.
[Table]
α for MRT For MRT, since Shen et al.
(2016) did several experiments to study the effect of hyperparameters, the original hyperparameters are strictly followed in our experiments.
Hyperparameter α is set to 5 .
× 10 − 3 and the sample size is 100 .
The best BLEU score of MRT on the validation set is 30.55.
The learning rate is set to 1 .
× 10 − 5 .
α and τ for Hellinger loss As shown in Table 4, α = 5 .
× 10 − 4 , τ = 0 .
5 produces the best result.
The highest BLEU score on the validation set is 31.13, which is 0.58 higher than MRT.
The learning rate used in Hellinger loss is also 1 .
× 10 − 5 .
[Table]
Sample size for Hellinger loss For squared Hellinger loss, the effect of sample size is shown in
Table 5.
A larger sample size generally gives a higher BLEU score, however, due to the limit of
GPU memory, we cannot set sample size to a larger value.
[Table]
τ for max margin For the two max margin training criteria, the learning rate is 1 .
× 10 − 7 .
Table 6 shows the effect of τ : according to the results on the validation set, τ is not very sensitive to the
BLEU scores.
As shown in Table 6, BLEU scores given by slack rescaling are usually 0.1 higher than for margin rescaling, and we use τ = 1 .
× 10 − 3 for both of the two models.
[Table]
Details We use ROUGE-2 as reward or loss function.
For the MLE baseline, we train the model for 7 epochs in total.
For the ﬁrst 5 epochs, we use Adam with a learning rate of 3 .
× 10 − 4 , we then use SGD with a learning rate of 0 . 1 , and decay the learning rate by a factor of 0 .[-
-]
5 at the beginning of epoch 7.
For the other training criteria, we use the hyperparameters with the best performance described in
Section 6.1.
8.2. A DDITIONAL E XPERIMENTS
Details We implement the squared Hellinger distance criterion on convolutional seq2seq model (Gehring et al., 2017) and strictly follow the settings of IWSLT experiments in Edunov et al.
[Table]