Paper:
Existing machine translation decoding algorithms generate translations in a strictly monotonic fashion and never revisit previous decisions. As a result, earlier mistakes cannot be corrected at a later stage. In this paper, we present a translation scheme that starts from an initial guess and then makes iterative improvements that may revisit previous decisions. We parameterize our model as a convolutional neural network that predicts discrete substitutions to an existing translation based on an attention mechanism over both the source sentence as well as the current translation output. By making less than one modification per sentence, we improve the output of a phrase-based translation system by up to 0.4 BLEU on WMT15 German-English translation.
1 INTRODUCTION
Existing decoding schemes for translation generate outputs either left-to-right, such as for phrasebased or neural translation models, or bottom-up as in syntactic models (Koehn et al., 2003; Galley et al., 2004; Bahdanau et al., 2015). All decoding algorithms for those models make decisions which cannot be revisited at a later stage, such as when the model discovers that it made an error earlier on.
On the other hand, humans generate all but the simplest translations by conceiving a rough draft of the solution and then iteratively improving it until it is deemed complete. The translator may modify a clause she tackled earlier at any point and make arbitrary modifications to improve the translation.
It can be argued that beam search allows to recover from mistakes, simply by providing alternative translations. However, reasonable beam sizes encode only a small number of binary decisions. A beam of size 50 contains fewer than six binary decisions, all of which frequently share the same prefix (Huang, 2008).1
In this paper, we present models that tackle translation similar to humans. The model iteratively edits the target sentence until it cannot improve it further. As a preliminary study, we address the problem of finding mistakes in an existing translation via a simple classifier that predicts if a word in a translation is correct (§2). Next, we model word substitutions for an existing translation via a convolutional neural network that attends to the source when suggesting substitutions (§3). Finally, we devise a model that attends both to the source as well as to the existing translation (§4). We repeatedly apply the models to their own output by determining the best substitution for each word in the previous translation and then choosing either one or zero substitutions for each sentence. For the latter we consider various heuristics as well as a classifier-based selection method (§5). Our results demonstrate that we can improve the output of a phrase-based translation system on WMT15 German-English data by up to 0.4 BLEU (Papineni et al., 2002) by making on average only 0.6 substitutions per sentence (§6). Our approach differs from automatic post-editing since it does not require post-edited text which is a scarce resource (Simard et al., 2007; Bojar et al., 2016). For our first model (§3) we merely require parallel text and for our second model (§4) the output of a baseline translation system.
∗Roman was interning at Facebook for this work. 125 = 32 < 50 < 26 = 64
2 DETECTING ERRORS
Before correcting errors we consider the task of detecting mistakes in the output of an existing translation system.
In the following, we use lowercase boldface for vectors (e.g. x), uppercase boldface for matrices (e.g. F) and calligraphy for sets (e.g. X ). We use superscripts for indexing or slicing, e.g., xi, Fi,j , Fi = (Fi,1, . . . ,Fi,|F
i|). We further denote x as the source sentence, yg as the guess translation from which we start and which was produced by a phrase-based translation system (§6.1), and yref as the reference translation. Sentences are vectors of indices indicating entries in a source vocabulary X or a target vocabulary Y . For example, x = (x1, . . . ,x|x|) ∈ X |x| with X = {1, . . . , |X |}. We omit biases of linear layers to simplify the notation.
Error detection focuses on word-level accuracy, i.e., we predict for each token in a given translation if it is present in the reference or not. This metric ignores word order, however, we hope that performance on this simple task provides us with a sense of how difficult it will be to modify translations to a positive effect. A token yig in the candidate translation yg is deemed correct iff it is present in the reference translation: yig ∈ yref. We build a neural network f to predict correctness of each token in yg given the source sentence x: f(x,yg) ∈ [0; 1]|yg| , where f(x,yg)i estimates P ( yig ∈ yref ) .
Architecture. We use an architecture similar to the word alignment model of Legrand et al. (2016). The source and the target sequences are embedded via a lookup table that replace each word type with a learned vector. The resulting vector sequences are then processed by alternating convolutions and non-linearities. This results in a vector S (x)i representing each position i in the source x and a vector T (yg)
j representing each position j in the target yg. These vectors are then compared via a dot product. Our prediction estimates the probability of a target word being correct as the largest dot product between any source word and the guess word. We apply the logistic function σ to this score,
f(x,yg) i = σ ( max
16j6|x|
[ S(x)T(yg) T ]j,i) .
Training. At training time we minimize the cross-entropy loss, with the binary supervision 1 for yig ∈ yref, 0 otherwise.
Testing. At test time we threshold the model prediction f(x,yg)i to detect mistakes. We compare the performance of our network to the following baselines:
1. Predicting that all candidate words are always correct fcor ≡ 1, or always incorrect fwrong ≡ 0.
2. The prior probability of a word being correct based on the training data fstat(y) = (P [y ∈ yref | y ∈ yg] > 0.5).
We report word-level accuracy metrics in Table 1. While the model significantly improves over the baselines, the probability of correctly labeling a word as a mistake remains low (62.71%). The task of predicting mistakes is not easy as previously shown in confidence estimation (Blatz et al., 2004; Ueffing & Ney, 2007). Also, one should bear in mind that this task cannot be solved with 100% accuracy since a sentence can be correctly in multiple different ways and we only have a single reference translation. In our case, our final refinement objective might be easier than error detection as we do not need to detect all errors. We need to identify some of the locations where a substitution could improve BLEU. At the same time, our strategy should also suggest these substitutions. This is the objective of the model introduced in the next section.
3 ATTENTION-BASED MODEL
We introduce a model to predict modifications to a translation which can be trained on bilingual text. In §5 we discuss strategies to iteratively apply this model to its own output in order to improve a translation.
Our model F takes as input a source sentence x and a target sentence y, and outputs a distribution over the vocabulary for each target position, F(x,y) ∈ [0, 1]|y|×|Y| . For each position i and any word j ∈ Y , F(x,y)i,j estimates P(yi = j |x,y−i), the probability of word j being at position i given the source and the target context y−i = ( y1, . . . ,yi−1,yi+1, . . . ,y|y| ) surrounding i. In other words, we learn a non-causal language model (Bengio et al., 2003) which is also conditioned on the source x.
Architecture. We rely on a convolutional model with attention. The source sentence is embedded into distributional space via a lookup table, followed by convolutions and non-linearities. The target sentence is also embedded in distributional space via a lookup table, followed by a single convolution and a succession of linear layers and non-linearities. The target convolution weights are zeroed at the center so that the model does not have access to the center word. This means that the model observes a fixed size context of length 2k for any target position i, y−i|k = ( yi−k, . . . ,yi−1,yi+1, . . . ,yi+k
) where 2k + 1 refers to the convolution kernel width. These operations result in a vector Sj representing each position j in the source sentence x and a vector Ti representing each target context y−i|k.
Given a target position i, an attention module then takes as input these representation and outputs a weight for each target position
α(i, j) = exp
( Sj ·Ti )∑|x| j′=1 exp (S j′ ·Ti) .
These weights correspond to dot-product attention scores (Luong et al., 2015; Rush et al., 2015). The attention weights allow to compute a source summary specific to each target context through a weighted sum,
a ( y−i|k,x ) = |x|∑ j=1 α(i, j) Sj
Finally, this summary a(y−i|k,x) is concatenated with the embedding of the target context y−i|k obtained from the target lookup table,
L ( y−i|k ) = { Lj , j ∈ y−i|k } and a multilayer perceptron followed by a softmax computes F(x,y)i from a(y−i|k,x), L(y−i|k). Note that we could alternatively use Ti instead of L(y−i|k) but our preliminary validation experiments showed better result with the lookup table output.
Training. The model is trained to maximize the (log) likelihood of the pairs (x,yref) from the training set.
Testing. At test time the model is given (x,yg), i.e., the source and the guess sentence. Similar to maximum likelihood training for left-to-right translation systems (Bahdanau et al., 2015), the model is therefore not exposed to the same type of context in training (reference contexts from yref) and testing (guess contexts from yg).
Discussion. Our model is similar to the attention-based translation approach of Bahdanau et al. (2015). In addition to using convolutions, the main difference is that we have access to both left and
right target context y−i|k since we start from an initial guess translation. Right target words are of course good predictors of the previous word. For instance, an early validation experiment with the setup from §6.1 showed a perplexity of 5.4 for this model which compares to 13.9 with the same model trained with the left context only.
4 DUAL ATTENTION MODEL
We introduce a dual attention architecture to also make use of the guess at training time. This contrasts with the model introduced in the previous section where the guess is not used during training. Also, we are free to use the entire guess, including the center word, compared to the reference where we have to remove the center word.
At training time, the dual attention model takes 3 inputs, that is, the source, the guess and the reference. At test time, the reference input is replaced by the guess. Specifically, the model Fdual(x,yg,yref) ∈ [0; 1]|yref|×|Y| estimates P ( yiref |x,yg,y −i ref ) for each position i in the reference sentence.
Architecture. The model builds upon the single attention model from the previous section by having two attention functions a with distinct parameters. The first function asource takes the source sentence x and the reference context y−iref to produce the source summary for this context asource ( y−i|k,x ) as in the single attention model. The second function aguess takes the guess sentence yg and the reference context y−iref and produces a guess summary for this context aguess ( y−i|k,yg ) . These two summaries are then concatenated with the lookup representation of
the reference context L ( yref −i|k) and input to a final multilayer perceptron followed by a softmax. The reference lookup table contains the only parameters shared by the two attention functions.
Training. This model is trained similarly to the single attention model, the only difference being the conditioning on the guess yg.
Testing. At test time, the reference is unavailable and we replace yref with yg, i.e., the model is given (x,yg,y −i|k g ) to make a prediction at position i. In this case, the distribution shift when going from training to testing is less drastic than in §3 and the model retains access to the whole yg via attention.
Discussion. Compared to the single attention model (§3), this model reduces perplexity from 5.4 to 4.1 on our validation set. Since the dual attention model can attend to all guess words, it can copy any guess word if necessary. In our dataset, 68% of guess words are in the reference and can therefore be copied. This also means that for the remaining 32% of reference tokens the model should not copy. Instead, the model should propose a substitution by itself (§6.1). During testing, the fact that the guess is input twice (x,yg,y −i|k g ) means that the guess and the prediction context always match. This makes the model more conservative in its predictions, suggesting tokens from yg more often than the single attention model. However, as we show in §6, this turns out beneficial in our setting.
5 ITERATIVE REFINEMENT
The models in §3 and §4 suggest word substitutions for each position in the candidate translation yg given the current surrounding context.
Applying a single substitution changes the context of the surrounding words and requires updating the model predictions. We therefore perform multiple rounds of substitution. At each round, the model computes its predictions, then our refinement strategy selects a substitution and performs it unless the strategy decides that it can no longer improve the target sentence. This means that the refinement procedure should be able to (i) prioritize the suggested substitutions, and (ii) decide to stop the iterative process.
We determine the best edit for each position i in yg by selecting the word with the highest probability estimate: yipred = arg maxj∈Y F (x,yg) i,j . Then we compute a confidence score in this prediction s(yg,ypred) i , possibly considering the prediction for the current guess word at the same position.
These scores are used to select the next position to edit, i? = arg maxi s(yg,ypred)i and to stop the iterative process, i.e., when the confidence falls below a validated threshold t. We also limit the number of substitutions to a maximum of N . We consider different heuristics for s,
• Score positions based on the model confidence in yipred, i.e., sconf(yg,ypred) i = F(x,yg) i,yipred .
• Look for high confidence in the suggested substitution yipred and low confidence in the current word yig: spr(yg,ypred) i = F(x,yg) i,yipred × ( 1− F(x,yg)i,y i g ) .
• Train a simple binary classifier taking as input the score of the best predicted word and the current guess word: scl(yg,ypred)i = nn ( logF(x,yg) i,yipred , logF(x,yg) i,yig ) , where nn
is a 2-layer neural network trained to predict whether a substitution leads to an increase in BLEU or not.
We compare the above strategies, different score thresholds t, and the maximum number of modifications per sentence allowed N in §6.2.
6 EXPERIMENTS & RESULTS
We first describe our experimental setup and then discuss our results.
6.1 EXPERIMENTAL SETUP
Data. We perform our experiments on the German-to-English WMT15 task (Bojar et al., 2015) and benchmark our improvements against the output of a phrase-based translation system (PBMT; Koehn et al. 2007) on this language pair. In principle, our approach may start from any initial guess translation. We chose the output of a phrase-based system because it provides a good starting point that can be computed at high speed. This allows us to quickly generate guess translations for the millions of sentences in our training set.
All data was lowercased and numbers were mapped to a single special “number” token. Infrequent tokens were mapped to an “unknown” token which resulted in dictionaries of 120K and 170K words for English and German respectively.
For training we used 3.5M sentence triples (source, reference, and the guess translation output by the PBMT system). A validation set of 180K triples was used for neural network hyper-parameter selection and learning rate scheduling. Finally, two 3K subsets of the validation set were used to train the classifier discussed in §5 and to select the best model architecture (single vs dual attention) and refinement heuristic.
The initial guess translations were generated with phrase-based systems trained on the same training data as our refinement models. We decoded the training data with ten systems, each trained on 90% of the training data in order to decode the remaining 10%. This procedure avoids the bias of generating guess translation with a system that was trained on the same data.
Implementation. All models were implemented in Torch (Collobert et al., 2011) and trained with stochastic gradient descent to minimize the cross-entropy loss.
For the error detection model in §2 we used two temporal convolutions on top of the lookup table, each followed by a tanh non-linearity to compute S(x) and T(yg). The output dimensions of each convolution was set to 256 and the receptive fields spanned 5 words, resulting in final outputs summarizing a context of 9 words.
For the single attention model we set the shared context embedding dimension dimSj = dimTi = 512 and use a context of size k = 4 words to the left and to the right, resulting in a window of size 9 for the source and 8 for the target. The final multilayer perceptron has 2 layers with a hidden dimension of 512, see §3). For the dual attention model we used 2-layer context embeddings (a convolution followed by a linear with a tanh in between), each having output dimension 512, context of size k = 4. The
final multilayer perceptron has 2 layers with a hidden dimension of 1024, see §4). In this setup, we replaced dot-product attention with MLP attention (Bahdanau et al., 2015) as it performed better on the validation set.
All weights were initialized randomly apart from the word embedding layers, which were precomputed with Hellinger Principal Component Analysis (Lebret & Collobert, 2014) applied to the bilingual co-occurrence matrix constructed on the training set. The word embedding dimension was set to 256 for both languages and all models.
6.2 RESULTS
Table 2 compares BLEU of the single and dual attention models (F vs Fdual) over the validation set. It reports the performance for the best threshold t ∈ {0, 0.1, . . . , 1} and the best maximum number of modifications per sentence N ∈ {0, 1, . . . , 10} for the different refinement heuristics. The best performing configuration is Fdual with the product-based heuristic spr thresholded at t = 0.5 for up to N = 5 substitutions. We report test performance of this configuration in table 3. Tables 4, 5 and 6 show examples of system outputs. Overall the system obtains a small but consistent improvement over all the test sets.
Figure 1 (left) plots accuracy versus the number of allowed substitutions and Figure 1 (right) shows the percentage of actually modified tokens. The dual attention model (§4) outperforms single attention (§3). Both models achieve most of improvement by making only 1-2 substitutions per sentence. Thereafter only very few substitutions are made with little impact on BLEU. Figure 1 (right) shows that the models saturate quickly, indicating convergence of the refinement output to a state where the models have no more suggestions.
To isolate the model contribution from the scoring heuristic, we replace the scoring heuristic with an oracle while keeping the rest of the refinement strategy the same. We consider two types of oracle: The full oracle takes the suggested substitution for each position and then selects which single position should be edited or whether to stop editing altogether. This oracle has the potential to find the largest BLEU improvement. The partial oracle does not select the position, it just takes the heuristic suggestion for the current step and decides whether to edit or stop the process. Notice that both oracles have very limited choice, as they are only able to perform substitutions suggested by our model.
Figure 2 reports the performance of our best single and dual attention models compared to both oracles on the validation set; Figure 3 shows the corresponding number of substitutions. The full and partial oracles result in an improvement of +1.7 and +1.09 BLEU over the baseline in the dual attention setting (compared to +0.35 with spr).
In the single-attention setup the oracles yields a higher improvement (+2.37 and +1.3) and they also perform more substitutions. This supports our earlier conjecture (§4) that Fdual is more conservative and prone to copying words from the guess yg compared to the single attention model. While helpful in validation, the cautious nature of the dual model restricts the options of the oracle.
We make several observations. First, word-prediction models provide high-quality substitutions ypred that can lead to a significant improvements in BLEU (despite that both oracles are limited in their choice of ypred). This is supported by the simple heuristic sconf performing very close to more sophisticated strategies (Table 2).
Second, it is important to have a good confidence estimate on whether a substitution will improve BLEU or not. The full oracle, which yields +1.7 BLEU, acts as an estimate to having a realvalued confidence measure and replaces the scoring heuristic s. The partial oracle, yielding +1.09 BLEU, assesses the benefit of having a binary-valued confidence measure. The latter oracle can only prevent our model from making a BLEU-damaging substitution. However, confidence estimation is a difficult task as we found in §2. Finally, we demonstrate that a significant improvement in BLEU can be achieved through very few substitutions. The full and partial oracle modify only 1.69% and 0.99% of tokens, or 0.4 and 0.24 modifications per sentence, respectively. Of course, oracle substitution assumes access to the reference which is not available at test time. At the same time, our oracle is more likely to generate fluent sentences since it only has access to substitutions deemed likely by the model as opposed to an unrestricted oracle that is more likely to suggest improvements leading to unreasonable sentences. Note that our oracles only allow substitutions (no deletions or insertions), and only those that raise BLEU in a monotonic fashion, with each single refinement improving the score of the previous translation.
7 CONCLUSION AND FUTURE WORK
We present a simple iterative decoding scheme for machine translation which is motivated by the inability of existing models to revisit incorrect decoding decisions made in the past. Our models improve an initial guess translation via simple word substitutions over several rounds. At each round, the model has access to the source as well as the output of the previous round, which is an entire translation of the source. This is different to existing decoding algorithms which make predictions based on a limited partial translation and are unable to revisit previous erroneous decoding decisions.
Our results increase translation accuracy by up to 0.4 BLEU on WMT15 German-English translation and modify only 0.6 words per sentence. In our experimental setup we start with the output of a phrase-based translation system but our model is general enough to deal with arbitrary guess translations.
We see several future work avenues from here. Experimenting with different initial guess translations such as the output of a neural translation system, or even the result of a simple dictionary-based word-by-word translation scheme. Also one can envision editing a number of guess translations simultaneously by expanding the dual attention mechanism to attend over multiple guesses.
So far we only experimented with word substitution, one may add deletion, insertion or even swaps of single or multi-word units. Finally, the dual-attention model in §4 may present a good starting point for neural multi-source translation (Schroeder et al., 2009).
ACKNOWLEDGMENTS
The authors wish to thank Marc’Aurelio Ranzato and Sumit Chopra for their advice and comments.
A EXAMPLES

Review:
This paper proposes a model for iteratively refining translation hypotheses. This has several benefits, including enabling the translation model to condition not only on “left context”, but also on “right context”, and potentially enabling more rapid and/or accurate decoding. The motivation given is that often translators (and text generators generally) use a process of refinement in generating outputs.
This is an important idea that is not currently playing much of a role in neural net models, so this paper is a welcome contribution. However, while I think this is an important first step, I do feel that the lack of in depth analysis suggests this paper is not quite ready for a final publication version. For example, there are many possible connections to prior work in NLP, MT, and other parts of ML that could better contextualize this work (see specifics below). More substantively, the model in Section 3 could be interpreted as a globally normalized, undirected (~CRF) translation model trained using a pseudo-likelihood objective. In this analysis, the model squarely back in the context of traditional discriminative translation models which used “undirected” features, and the decoding algorithm then looks more like a standard greedy hill-climbing algorithm (albeit with an extra heuristic model for selecting which variable to update), which is also nothing unfamiliar.
My second criticism the limitations of the model are not well discussed. For example, the proposed editing procedure cannot obviously remove or insert a word from a translation. While I think this is a reasonable assumption than can be made for the sake of tractability, it is very unfortunate since missing or extra words (esp. function words) are a common problem in the baseline models that are being used. Second, the standard objections to absolute positional models (vs. relative positional models) seem particularly crucial to bring up in this work, especially since they might make some of the design decisions a bit more justifiable.
Overall, this is an initial step in an interesting direction, but it needs more thorough analysis to demonstrate its value. A more thorough analysis will also likely suggest some important model variants (for example: is a global translation model really the goal? or is a post-editing model that fixes outputs with more complex operations more ideal?)
Related work:
I think that more could be done to put this work in the context of what has come before and what is currently going on in other parts of ML. The idea of iterative refinement has been proposed in other problems that have complex output spaces, for example the DRAW model of Gregor et al. and the conditional adversarial network models used to refine images proposed recently by Isola et al. In NLP, there have been several (stochastic) hill climbing approaches that have been proposed, such as the work on parsing by Zhang and Lei et al. (2014) who use random initial guesses and then do greedy hill climbing using a series of local refinements, the structured prediction cascades of Weiss and Taskar (2009) (not to mention general coarse-to-fine modeling strategies). Finally, in MT, Arun et al. (2009) who use a Gibbs sampler to refine an initial guess to do decoding with a more complex model. The use of an explicit error model is rather novel in the context of correction, but I would point out that although the proposed architecture is different, the discriminative word lexicon models of Mauser et al. (2009) and the neural version of the same by Ha et al. (2014) are similar in spirit. There have also been a number of papers on “automatic post editing”, including the shared task at WMT2016, and there are not only standard test sets and baselines, but also datasets that could actually be used to train a post editing model with human-generated data. Minimally using the techniques they described could be a useful foil for the models presented in this paper.
“the target sentence is also embedded in distributional space via a lookup table” I think “distributional space” is a bit unclear. Maybe “the target sentence is represented in terms of distributed word representations via a lookup table” or something like that. “distributional” suggests that the representations are derived from how the words are distributed in the corpus, whereas you are learning these representations on this task which isn’t modeling their distribution except only very indirectly.
Section 3 Model: In Section 3, the model computes the distribution over target word types at an absolute position i in the output sentence, given the target language context and the source language context. It is introduced as the model that is used to refine an existing hypothesis, but it is not immediately clear that the training data for this model (at least in this section) are the gold standard translations- “training set” could be interpreted in variety of ways. This becomes clearer when reading later in the paper, but it’s a bit less clear when reading from the beginning for the first time.
The use of a fixed sized window for representing the target word in context also seems to make something like a model 1 assumption since only the lexical features (and not any “alignment” or “positional” features) determine the attention. This should be clarified since it will make the assumptions of the model more transparent (and also suggest possible refinements to the model, e.g., including (representations) of i and j as components of S^j and T^i, which would allow model 2/3-like responses to be learned- although by leaving them out, the model might behave a bit more like a relative positional model than an absolute positional model, which is probably attractive).
Finally, some discussion for why a fixed window is used to represent the target sentence is worth including (since a global context is apparently used to represent the source sentence).
The relationship between this training objective and pseudo likelihood (PL; Besag, 1975) might be worth mentioning. Since I believe this is just a PL objective for a certain global model, this suggests alternative decoding algorithms, or certainly a different analysis of the proposed decoding objective.
The section 4 model conditions on the true context of a position in the true target, the current target guess, and the source. I don’t completely understand the rationale for this model since at test time only two of these variables are available, and the replacement of y_ref with y_g seems hard to justify.