Ranking suggestions for black-box interactive translation prediction systems with multilayer perceptrons (2016)

Abstract: The objective of interactive translation prediction (ITP), a paradigm of computer-aided translation, is to assist professional translators by offering context-based computer-generated suggestions as they type. While most state-of-the-art ITP systems are tightly coupled to a machine translation (MT) system (often created ad-hoc for this purpose), our proposal follows a resourceagnostic approach, one that does not need access to the inner workings of the bilingual resources (MT systems or any other bilingual resources) used to generate the suggestions, thus allowing to include new resources almost seamlessly. As we do not expect the user to tolerate more than a few proposals each time, the set of potential suggestions need to be filtered and ranked; the resource-agnostic approach has been evaluated before using a set of intuitive length-based and position-based heuristics designed to determine which suggestions to show, achieving promising results. In this paper, we propose a more principled suggestion ranking approach using a regressor (a multilayer perceptron) that achieves significantly better results.

## introduction translation technologies are frequently used to assist professional translators. common approaches use machine translation (mt) @xcite or translation memories @xcite to produce a first (and usually inadequate) prototype of the translation which is then edited by the professional translator in order to produce a target-language text that is adequate for publishing. in both scenarios, the suggestion may be considered by the professional translators as a source of inspiration: they will assemble the final translation on some occasions by accepting and rearranging parts of the proposal, or on other occasions by introducing their own words when an appropriate equivalent is not found in the suggestion. our approach, described in a previous work @xcite , follows a different paradigm known as interactive translation prediction (itp), which, instead of presenting a translation proposal that gets reshaped by the target-language sentence formed in the translator's mind, focuses on offering translation suggestions as the translation is carried out. most state-ofthe-art itp approaches obtain the suggestions by means of a modified (or tailor-made) statistical machine translation system (smt) @xcite ) that is able to provide additional information (such as word alignments, alternative translations, and scores or probabilities for the translation). these systems are able to leverage more information from the bilingual resource than if it were used unmodified as a black-box, but doing so they inherit the common requirements of smt, namely, the dependency on the availability of extensive parallel corpora. it is worth noting that integrating other resources of bilingual information would be almost impossible in this kind of systems, as the itp tool needs the additional information obtained from the underlying smt engine. unlike those previously described, the approach described here is able to use any bilingual resource capable of delivering one or more translations into target language, regardless of how they are obtained and without the need to modify the resource; suggestions are created by generating all possible sub-segments of words in the source-language sentence (up to a given length) and then querying the available bilingual resources for their translations. the nature of these bilingual resources is not limited to mt systems, but they may also include translation memories, dictionaries, catalogues of bilingual phrases, or any combination of them. a neuralbased machine learning algorithm trained on features extracted from the source sentence, from the current prefix of the target sentence, and from the translated sub-segments is used to rank and select which suggestions to show at each time step. not having to rely on the inner workings of each system allows us to integrate new resources without modifying how the itp system works; similarly, we do not need to modify the resources in any way. both these features make it possible to use any resource the professional translator has access to in a seamless way. these translated sub-segments are then offered to the user as the translation is being typed with the objective of saving keystrokes and, hopefully, time. in previous works we have explored the performance of our approach considering rule-based mt systems @xcite and in-domain and out-of-domain smt systems @xcite , using a naïve strategy based on a number of intuitive heuristics to rank and select a few suggestions that are shown to the user. in this paper, however, we aim to improve the strategy by using machine learning techniques. the improvement presented here is twofold: on the one hand, more coherent and principled models are introduced; on the other hand, the results we achieve are significantly better. the remainder of the paper is organised as follows. after reviewing the state-of-the-art of itp in section 2, we describe our method for generating and ranking translation suggestions from bilingual resources in section 3, emphasizing the differences between the sounder approach presented in this paper and the former heuristic ranking method. we then introduce in section 4 our experimental set-up and show the results of its evaluation. finally, we discuss the results and propose future lines of research in section 5. ## related work the systems which have most significantly contributed to the field of itp are those built in the pioneering transtype project @xcite , and its continuation, the transtype2 project @xcite . these systems observe the current partial translation already typed by the user and, by exploiting an embedded statistical mt engine, propose one or more completions that are compatible with the sentence prefix entered by the user. the proposals offered may range from one or several words to a completion of the remainder of the target sentence. an automatic best-scenario evaluation with training and evaluation corpora belonging to the same domain @xcite showed that it might theoretically be possible to use only 20-25% of the keystrokes in comparison with the unassisted translation for english-spanish translation (both directions) and around 45% for english-french and english-german. a number of projects continued the research where transtype2 had left off. caitra @xcite is an itp tool which uses both the phrase table and the decoder of a statistical mt system to generate suggestions. researchers at the universitat politècnica de valència have also made significant improvements to a transtype2-style system such as allowing users to accept discontinuous segments of the suggested translation @xcite . the casmacat project @xcite followed the same line of research, improving itp using active and on-line learning @xcite . commercial translation memory systems also integrate some form of itp as one of their basic features (see, for example, sdl trados autosuggest 2.0 foot_0 ), and new translation tools such as lilt @xcite focus on delivering itp on an user-friendly web interface. ## method as already described in other articles @xcite , our method starts by generating all possible whole-word sub-segments of the source-language sentence s of lengths @xmath0, l], where l is the maximum source sub-segment length measured in words. 2 the resulting sub-segments are then translated by means of a bilingual resource (or a combination of bilingual resources). the set of potential proposals p s for sentence s is made up of suggestions p, which in turn are made up of a target-language sub-segment t p and the starting b p and ending e p positions of the corresponding sub-segment in s. these suggestions are then offered as the translation t is being typed. 3 let pr(x) be the character-level set of prefixes of a string x, t k the k-th word of t , and @xmath1...t k-1 ŵ the partially translated sentence where ŵ ∈ pr(t k ) is the currently typed prefix of t k ; we define the set of compatible suggestions p s compatible as for example, given @xmath2, with @xmath3, the set of potential proposals will be p @xmath4, "my tailor", "tailor", "tailor is", "is", "is healthy", "healthy"}; with @xmath5, the set of compatible suggestions would be p s @xmath6, "tailor is"}. as studied in @xcite , the number of compatible suggestions depends not only on the value of l, but also on the specific word prefix; for example, when users type the letter d when translating a long sentence into spanish, they will probably obtain a significant number of suggestions starting with de foot_3 originating from sub-segments located in different source positions. obviously, only suggestions originating from the part of the source sentence currently being translated may be useful, but this position is difficult to determine unambiguously. as we do not expect users to tolerate a long list of suggestions, more elaborated strategies are needed both to rank suggestions and to reduce the list to a manageable size. ## experimental setup and results as explained in section 3.2, four different configurations are trained. training, development and test data are extracted from a corpus with 15,000 english-spanish sentences extracted from europarl @xcite version 7, a collection of proceedings from the european parliament; 11,000 sentences were used as training set, 1,000 as development or validation set and the remaining 3,000 as test set. as a bilingual mt engine capable of providing the translation of the sub-segments we used the free/open-source statistical mt system moses @xcite trained over 155,760 independent sentences from the same corpus, following the standard procedure for training a baseline system. 11 the evaluation was conducted for the translation of texts from english to spanish. to generate the training set, using a source sentence and a reference, an automatic system iteratively considers the first letter of each word and evaluates all the possible suggestions; those that fully match the following words of the reference are tagged as usable in that context, and those that would be part of the greedy suboptimal sequence of actions that leads to the best performance are tagged as winning in that context. in order to measure the performance of each configuration, we use the keystroke ratio (ksr), that is, the ratio between the actual number of keys pressed for typing the translation and the length in characters of the translation, as described by @xcite . accepting a suggestion, no matter its rank, costs one keystroke. it is worth noting that this metric does not measure the time or effort needed to read the suggestions, and does not penalize the offering of inappropriate suggestions in any way. the automatic evaluation system foot_6 is similar to the one described in the previous work @xcite : it tries to emulate the behaviour of a user that, given a source sentence s, mentally generates a translation t , foot_7 then proceeds to type it, reading every offered suggestion and accepting the longest one that matches exactly t , if any. suggestions need to be full-word translations: if t k is "thesaurus", the suggestion "the" will not be accepted. the system types the first letter of the next word in t , evaluates all the suggestions, and either chooses the longest one that matches with t or types the rest of the word, foot_8 until t is completed. 15 the models have been tested with different limits for the maximum number of suggestions @xmath7, 8], sorted in descending order according to the multilayer perceptron output value. all the multilayer perceptron configurations have three layers: input, hidden and output. those trained with the reduced feature set only have two input neurons; perceptrons dealing with the full feature set have 75 input units. we will explore different values for the number of units in the hidden layer. all configurations have just one output neuron, and are fully connected. all the neurons but the output neuron have logistic activation functions; as we are estimating a regression function, the output neuron has the identity activation function. for the training procedure, we use backpropagation with mean squared error (mse) as the error function to optimize, and no momentum or regularization of any kind. as neural networks have trouble dealing with local minima @xcite , each perceptron has been trained five times with different random initializations. as hyperparameters, we explored different values for the learning rate and the number of neurons in the hidden layer. all the configurations were tested with learning rates α ∈ {10 -2 , 10 -3 , 10 -4 }. the configurations using the reduced feature set were trained and tested with @xmath8, 4, 8, 16} neurons in the hidden layer; the ones using the full feature set used figure 3 : ksr for the multilayer perceptron hyperparameter configuration trained using the full feature set and the winning suggestions set that got the best mse. the system beats the baseline in a statistically significant way for every tested value of the maximum number of suggestions m . @xmath9, 8, 32, 128} neurons in the hidden layer. we will also use the previous intuitive heuristic approach, as described in subsection 3.1 (referred to as heuristic onwards) as a baseline. paired bootstrap resampling @xcite is performed between the different models (including the heuristic approach) using 1000 iterations and @xmath10.05; the best statistically significant ksr values achieved will be denoted with a circle. results show that the perceptrons trained with the winning suggestions and full feature set (figure 3 ) perform notably better than the ones trained with the reduced set (figure 4 ). in every figure, the line "minimum ksr" denotes the best ksr we can achieve offering all the suggestions using the current methodology, and circled points denote the best statistically significant ksr. while several systems trained with the full feature set performed statistically significantly better than the heuristic approach, no reduced feature set system manages to outperform the heuristic baseline with @xmath11, 2]. those trained with the usable suggestions perform similarly to the ones trained with the winning suggestions: the best performing systems are not statistically significantly better or worse than the best ones using winning suggestions for most values of the maximum number of suggestions m , but, overall, less configurations manage to beat the baseline. figure 5 compares the performance of the best system for each configuration. figure 4: ksr for the multilayer perceptron hyperparameter configuration trained using the reduced feature set and the winning suggestions set that got the best mse. the system beats the baseline in a statistically significant way for every tested value of m , but for @xmath12, 2, where it achieves significantly worse ksr. figure 5 : ksr for the best multilayer perceptron hyperparameter configuration for each training set. the best models perform similarly regardless if predicting winning or usable suggestions, even though the mse is higher for the ones predicting usable suggestions. best statistical significance marks those values of the maximum number of suggestions m where the model using the full feature set with winning suggestions is statistically significantly better than the rest; 2-best statistical significance marks those where it is not significantly better or worse than the model using the full feature set with usable suggestions, but both are statistically significantly better than the rest. ## conclusions and future work resource-agnostic itp is a low-cost approach for computer-aided translation. we aim to provide a competitive alternative to postediting that can easily integrate any bilingual resource the user has access to. the naïve distance-based ranking described in @xcite provided results comparable to glass-box itp. we managed to significantly improve it employing a sounder machine-learned model that manages to obtain keystroke savings in the range of 25-45% when offering up to 4 suggestions to the user. it is important to evaluate this model with more language pairs, specially those which present special interest deemed their grammatical or lexical differences. also, selecting which features are more representative can further improve the performance of the models while reducing the processing power needed to train and test the model. there are other itp systems being currently developed, namely thot. (ortiz-martínez and casacuberta, 2014) although it is addressing a different problem by using an ad-hoc smt system, it assists the user in a similar way. a comparison of the performance achieved will be conducted to contextualize our method. for further improving the results attained, we plan to use a distortion model as the one proposed by @xcite , that can be used to predict which source words will be translated next, integrating this information as a feature for the multilayer perceptron, hopefully complementing our light alignment model described in section 3.3. we also plan to use a simplified language model to give a rough estimate of the likelihood of the concatenation of t and a given suggestion. we also plan to explore the impact of simultaneously using different black-box bilingual resources. different strategies will be evaluated in order to integrate the available resources: combining multiple black-box mt systems as described in @xcite ; using confidence-based measures in order to select the most promising translations as performed by @xcite ; predicting the best candidates for the translation of each particular subsegment by using only source-language information, thus avoiding the need to consult every available resource, as explored by @xcite ; or letting the multilayer perceptron manage the different suggestions, devising new features if needed. finally, we also aim to find evaluation metrics that improve the correlation with professionals by mimicking how a professional translator would work with the tool. currently, we only choose suggestions that perfectly fit what the professionals are going to type; a translator could however accept a partially matching suggestion and then replace the mismatching part, or even accept a suggestion that does not match the planned translation, adapting it to the new prefix.