Letting a Neural Network Decide Which Machine Translation System to Use for Black-Box Fuzzy-Match Repair (2018)

Abstract: While systems using the Neural Network-based Machine Translation (NMT) paradigm achieve the highest scores on recent shared tasks, phrase-based (PBMT) systems, rule-based (RBMT) systems and other systems may get better results for individual examples. Therefore, combined systems should achieve the best results for MT, particularly if the system combination method can take advantage of the strengths of each paradigm. In this paper, we describe a system that predicts whether a NMT, PBMT or RBMT will get the best Spanish translation result for a particular English sentence in DGT-TM 20161. Then we use fuzzy-match repair (FMR) as a mechanism to show that the combined system outperforms individual systems in a black-box machine translation setting.

## introduction natural language processing (nlp) systems designed to do the same task often belong to different methodological paradigms. at any time in history, the best-scoring systems may tend to come from a particular paradigm. for example, in machine translation (mt), the current dominant paradigm is neural network-based mt (nmt). the previously dominant paradigm was phrase based mt (pbmt), and so on. when comparing mt results for different types of input, systems from certain paradigms perform better on certain types of input c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 https://ec.europa.eu/jrc/en/language-technologies/dgttranslation-memory and vice versa @xcite . in some cases nmt suffers more than other paradigms @xcite . thus, it may be premature to completely abandon "old" methods in favor of "new" ones. newer methods, especially nmt, tend to achieve higher bleu scores than previous methods including pbmt and rule-based mt (rbmt) systems. however, professional translators and users of computer-assisted translation (cat) tools seem to prefer pbmt output for particular sentences @xcite . many recent systems (e.g., participants in wmt17 @xcite ) use nmt, because it obtains higher scoring results, but does not require time-consuming procedures like feature generation or quality estimation (qe) to achieve quality mt translations. cat tools, and other systems using black-box mt, could benefit from a way of predicting which mt system will perform the best at translating a particular source segment. such systems which typically use only one mt tool to translate all input could benefit from selectively using the output of multiple systems in this way. this paper describes a series of experiments that attempt to take advantage of the strengths of alternative systems and combine system output to produce the best result. first, we describe our system, select, which uses a neural-network based approach to predict which system provides the best output for translating a particular english sentence to spanish using: nematus @xcite ), an nmt system; moses @xcite ), a pbmt system; and apertium @xcite , an rbmt system. then we use the mt system predicted to be the best 2 to improve pre-vious work on fuzzy-match repair (fmr), an approach that uses black-box machine translation as its primary method for translating sub-segments to be repaired @xcite . most previous hybrid approaches to mt focus on ways to combine individual translations from different mt systems. in contrast, our system uses multiple predictive model types to choose the optimal sentence-level translations, without previous knowledge of the internal workings of the mt system. select predicts which of 3 translation systems will produce the best translation. select uses the performance differences seen on various tasks with different data where typically one mt system, be it rule-based, phrase-based, or neural, outperforms the other systems, to improve results by providing sentence-level predictions where often times differences in mt system quality can occur depending on the data. in a professional setting, mt systems may have a higher cost due to quality performance issues and it would make sense that a translator has the most appropriate translation at hand when using the mt tool. relying on a single mt system could be costly as shown in previous investigations @xcite . we propose a prediction system that integrates easily into any system that uses blackbox mt. black-box mt systems would use the mt engine that select predicts using a pre-translation performance metric. select accepts any source sentence input s and produces a translation σ in a transparent way by predicting beforehand the system to use and querying the black-box mt system with the ideal (best-predicted) mt engine to use. our goal is to measure how well mainstream mt systems perform and compare their differences for commercial use situations where often times translation quality should be determined beforehand to determine economic value. ## methodology our work uses a predictive classifier to determine the best mt system for translation when used as a black box such that no prior knowledge of the internal workings of the mt system is necessary. it will allow any system with the ability to call a translate() method access to sentence-level quality without the use of more complex paradigms such as quality estimation. combining several mt systems via our blackbox method should achieve higher scores than just using one mt system. fmr @xcite is a recent example of a black-box translate() method that uses one mt system, apertium. their work assumes no dependency on other parts of the mt system. here, we use the work from @xcite to show the advantage of having a mechanism to predict the best mt system to use before actually calling the translate() method. our work intrinsically compares 3 open-source mt systems using 3 different classifier models: 1) recurrent neural network (rnn), 2) fasttext foot_1 classification, and 3) logistic regression (lr) described in section 5.1.1. each model is created to predict sentence-level quality using bleu. then, we use both bleu and word-error rate (wer) as a performance measurement to determine which model to use in fmr. wer for our experiments is considered as the word-based edit distance between the reference translation and the system translation often called levenshtein distance @xcite . our model is somewhat similar to a quality estimation model but based on mt engines alone. the prediction model is part of a bigger system that when given a new sentence s and a set of systems: {m t 01 , m t 02 , m t 03 , ...} derives a translation by selecting an mt system based on training data. our hypothesis is that a system that can determine which mt engine to use before actually having the system translation should perform better and offer the best value for the translator or cat tool user. 4 after establishing that select can select translation engines in a fashion that is beneficial to the user, we evaluate, with wer, select's choices using a system that uses black-box mt, fuzzy-match repair @xcite . when the fmr system needs to translate any segment, whether an entire sentence or sub-segment of a sentence, it calls upon select to determine which engine to use for the source sentence to be translated. then, fmr calls its translate() method with the mt engine suggested. we test select in this paper using: apertium, moses, and nematus. our experiments measure wer from fmr when using select. we aim to improve upon previous results @xcite by choosing the best predicted mt system for each translation in fmr. ## results we provide results of 2 experiments: experiment 1 measures the accuracy of the predictive models in select using bleu and wer as evaluation metrics. experiment 2 uses select as an agnostic predictor to choose an mt system for fmr. for experiment 1, we use 20321 sentences to test the 3 mt systems (apertium, moses, and nematus) with 3 types of classification (rnn, fasttext, and logistic regression). for even more details about our predictive models, we present the accuracy of our models in isolation on the 20321 test sentences. our fasttext system. for example, had a 19.04 improvement over the bleu lower-bound of (90.2% of the potential difference) and a 14.36 improvement over the wer lower-bound (83.4% of the potential difference), in both cases, this is significantly more than the average of the upper and lower bounds (29.53 bleu score and 55.31 wer). the ensemble system (using fasttext) also out-performs the best individual system (nematus) by .65 blue and 2.22 wer. the average between the upper and lower bounds is a good baseline to beat, to demonstrate that our system is successful at predicting the correct high-scoring system most of the time. however, being the best system gives the results practical value. we observe that nematus is more likely to correctly handle polysemous words (should english march be translated to spanish as marzo (the month) or marcha (the action)). however, some of nematus' errors involve seemingly arbitrary translations of words or the addition of arbitrary words. for example, the english "identification numbers" is correctly translated as "números de identificación" by apertium, but nematus translates it as identificación de identificación (moses translates it nearly correctly, but leaves off the "s" in "números"). similarly, apartium correctly translates the english "saffron" as azafrán, whereas moses leaves it untranslated ("saffron") and nematus translates it mysteriously as "lágrimas de los perros". ## conclusion our experiments show that select can be used to increase performance in black-box mt tools. se-lect is agnostic to other processes in a typical mt pipeline and does not require underlying process changes in current black-box mt systems. se-lect only requires access to a command-line utility that accepts a sentence as input to select the best mt system. the work presented in section 5.1.1 also helps explain how well various models perform for black-box systems. baseline mt systems are combined with a predictive model to create a non-traditional ensemble for improving translations from tools using black-box translation. in our experiments, fasttext outperformed other models as measured by bleu and wer. there are surely more prediction models (non-baseline) that could perform better but we leave that for future work.