A Comparison of Statistical and Neural MT in a Multi-Product and Multilingual Software Company - User Study (2018)

Abstract: Over the last 4 years, Infor has been implementing machine translation (MT) in its translation process. In this paper, the results of both statistical and neural MT projects are provide to give an insight in the advantages and disadvantages of MT use in a large company. We also offer a look into the future of MT within our company and to strengthen the implementation of MT in our translation process.

## introduction in the last few years, we have seen a change of direction regarding machine translation approaches. in different domains, more research is being focussed on neural machine translation (nmt) in comparison to phrase-based statistical machine translation: in both the research environment @xcite and commercial companies like google @xcite and microsoft @xcite nmt is increasingly important. in the context of commercial translations, the continuous improvement of (n)mt has not passed unnoticed. more and more language service providers (lsps) are implementing machine translation into their translation workflows and in addition, translation teams in large companies are investing in machine translation as part of their translation processes. as a large global software development company, infor 1 translates its products into many languages. this paper summarizes the results of the c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 http://www.infor.com/ investigations into the potential benefits of machine translation for a company with many products, many target languages and very different translation circumstances per product. this study consists of two main parts: smt and nmt. first, we give a description of our experiments, after which the results of the experiments are described. lastly, the results and impact on our company are discussed. we had 2 main goals for this user study: to find out the current importance of (s)mt in our company and the potential benefits of moving to nmt in the future. these goals are discussed in section 4. ## results normally, machine translation results are expressed using evaluation scores like meteor, bleu and/or hter. however, as these metrics generally do not correlate with linguists' findings @xcite , we chose to only report the number of machine translated segments (that were used in the translation projects) and the qualitative analyses of our linguists. both the linguist reviews and the number of machine translated sentences gave us an indication of the usefulness of mt in translation projects. ## discussion in this section, the results of the smt and nmt experiments are discussed. ## conclusion and outlook in this paper, we have discussed the outcomes of statistical (smt) and neural (nmt) machine translation experiments that we have conducted at infor. with a total of over 2 million machine translated words, smt has become a significant factor in product translations. smt has been used for 10 products with up to 12 languages. tests showed that smt produced workable translations on language pairs that are closely related, and we needed handwritten auto-post-editing scripts to improve the output quality. a first test with nmt has shown that nmt performs better on all languages tested (dutch, german and russian) than smt. the purpose of the experiments was to determine the significance of mt in our workflow and whether nmt is the next step to take. based on the number of machine translated words in the last few years, we now have a good understanding of the type of projects in which mt is of use, and it has already impacted the costs of translation projects in which mt was used. we have also seen that nmt scores higher than smt according to our linguists, which is a clear indication that nmt is the next step in improving our mt process. with a potential of many more products to translate and many more languages to translate to, we will start experimenting with nmt in the same way that we did with smt.