Neural Interactive Translation Prediction (2016)

Abstract: We present an interactive translation prediction method based on neural machine translation. Even with the same translation quality of the underlying machine translation systems, the neural prediction method yields much higher word prediction accuracy (61.6% vs. 43.3%) than the traditional method based on search graphs, mainly due to better recovery from errors. We also develop efficient means to enable practical deployment.

## interactive translation prediction interactive translation prediction leaves the translator in charge of writing the translation and places the machine translation system in an assisting role. rather than having a translator postedit machine translated output, the system actively makes suggestions as the translator writes their translation. this modality is similar to an auto-complete function. whenever the translator diverges from the suggestion (by typing a word that differs from the model's suggestion), the system recalculates (taking the translator's input into account) and generates new suggestions. implementations of interactive translation can be found in the casmacat foot_0 (see figure 1 ) and figure 1 : interactive translation prediction in casmacat: the system suggests to continue the translation with the words mehr als 18, which the user can accept by pressing the tab key. lilt foot_1 computer aided translation tools. this interaction mode is preferred by translators over post-editing @xcite . the goal of interactive translation prediction is to offer suggestions that the translator will accept. existing approaches with statistical machine translation use the static search graph produced by the machine translation system. the system attempts to match the partial translator input (called the prefix) to the search graph, using approximate string matching techniques (minimal string edit distance) when an exact match cannot be found. as a baseline, we use a statistical machine translation system for interactive translation prediction that closely follows @xcite and @xcite . the prefix could be matched by constraint decoding, however, at a much higher computational cost. initial work on interactive translation prediction can be found in the transtype and transtype2 projects @xcite @xcite . our current work focuses on how to produce suggestions; for various approaches to interaction modalities, see sanchis ## neural machine translation the use of neural network methods in machine translation has followed their recent success in computer vision and automatic speech recognition. motivations for their use include better generalization of the statistical evidence (such as the use of word embeddings that have similar representations for related words), and more powerful non-linear inference. the current state-of-the-art neural machine translation approach @xcite ) consists of: we describe a fairly general neural machine translation approach in order to motivate its use in the interactive translation prediction setting, illustrated in figure 2 . for more details, see @xcite , whose notation this section follows, or edinburgh's wmt 2016 submission @xcite , whose system we use in our experiments. target words y t are emitted from hidden states s t . the hidden state s t is informed by the input sequence, weighted by an attention mechanism α t,1 , ..., α t,t . the source language sequence x 1 , ..., x t is encoded as the hidden states h of two recurrent neural networks. at each time step t, the standard decoder computes the conditional probability of generating a word y t given the input sentence x. this is defined to be: during encoding, annotations h t were produced for each word x t in the input sentence @xmath0, • • • , x t ). these h t were produced by concatenating the forward and backward hidden states produced for each word by the forward and backward rnns, respectively. the context vector c t in equation 1 is a weighted average of the annotations. first, weights α @xmath1, where e @xmath2, h j ) can be thought of as an alignment model (parameterized as a neural network and jointly trained with the rest of the system). the weight α tj can be interpreted as the probability that y t is aligned to x j , resulting in soft alignments used by the system's attention mechanism to weight the focus of the context vector. the context vector is then computed as c @xmath3. as indicated above, decoding in this attention-based neural machine translation approach proceeds word by word. at each step of the decoding process, a probability distribution over possible next words is computed. this is conditioned on the previous word, the context vector, and the hidden state. the highest scoring word is selected and used in the conditioning context for the next step. alternatively, similar to beam search in traditional statistical machine translation decoding, the top n next words may be considered and competing hypotheses with different output words maintained. each of the hypotheses (consisting of a word sequence and a hidden state, and ranked by the combined word translation probabilities) is extended at the next decoding step. there are various choices for the exact design of the recurrent neural networks used in the encoder and decoder. going beyond basic approaches that use a simple hidden layer, more complex designs such as long short term memory (lstm) cells or gated recurrent units (gru) may be employed. ## neural interactive translation prediction the decoding process for neural translation models points to a straightforward implementation of interactive translation prediction. instead of using the model's predictions in the conditioning context for the next step, the words in the prefix provided by the translator can be used. hence, the next word prediction is conditioned on the choice of the translator, rather than the prediction of the model. during decoding for translation (as described above), the model's predictions {ŷ 1 , • • • , ŷt-1 } are fed back into the model to produce the next predicted word. in order to do interactive prediction, we instead feed the true prefix {y * 1 , • • • , y * t-1 } produced by the translator back into the model. thus we redefine the conditional probability of generating a word y t to be: in this work, we present two variations on neural interactive translation prediction: while beam search is known to produce better bleu scores than models without beam search, it is also more computationally expensive. we demonstrate that it performs well on the interactive translation prediction task, but note that full beam search is too slow for use in a live system. passing the translator prefix into the system (when the translator diverges from the predicted sequence) may produce subsequent errors in the translation, for instance by causing the attention mechanism to be misfocused. we show that the system is often able to recover from these errors, but that it occasionally results in incoherent sequences of suggestions. however, we show that the sequences of rejected suggestions produced by the neural systems tend to be shorter than those produced by the traditional search graph based systems. ## experimental setup since a user study that collects sufficient data for the various settings of our methods would be too costly, we resort to a simulation where a preexisting human translation is used in place of the translator's input to the interactive translation system. we do this by treating the preexisting human translation as though it is being typed live, one word (or letter) at a time, by a translator interacting with our prediction system. while we expect that in practical use, the human translator may match the machine translation's suggestions more closely (for example, by accepting synonyms which we score as "wrong" as they are not exact matches), we can nevertheless compare methods based on their prediction accuracy against the human translation relative to one another. data our experiment is carried out on the german-english data sets foot_2 made available for the shared news translation task of the conference for machine translation (wmt). the data consists of a 115 million word parallel corpus (europarl, news commentary, commoncrawl), and about 75 billion words of additional english monolingual data (ldc gigaword, monolingual news, monolingual commoncrawl). we use the official 2999 sentence test set (average sentence length 23 tokens) to measure the accuracy of our methods. ## results see figure 3 for an example of the neural interactive translation prediction model's output for one sentence. the figure displays the correct word choices (taken from the reference translation), the model's prediction (using the prefix of the reference translation as conditioning context), and the most probable word choices according to the model's probability distribution. some of the failures are near-synonyms (numbers instead of levels, or increasing instead of rising) that we might expect would be accepted by real human users of the system. for the purposes of our evaluation, we count even these near-synonyms as incorrect (as they are not exact string matches). however, it is worth noting the prevalence of these near-synonyms: in the neural versions, we find that 21.0% of incorrect predictions (22.4% with beam search) are input: das unternehmen sagte, dass es in diesem monat mit bewerbungsgesprächen beginnen wird und die mitarbeiterzahl von oktober bis dezember steigt. ## speed considerations we have shown that the neural method delivers superior prediction accuracy, but is it fast enough? to be used in an interactive user interface, the method has to quickly produce alternative sentence completion predictions. a common time limit in human computer interaction is 100 milliseconds for the system's response. any longer feels sluggish and annoying to the user. ## conclusion in this paper we demonstrate that neural machine translation systems can effectively be applied to interactive translation prediction, improving upon the performance of traditional methods. we show that they recover well from errors, have shorter sequences of incorrect predictions, and produce more near-synonyms of the correct answers. finally, we demonstrate a method that allows for practical deployment given real-life time constraints.