How to Move to Neural Machine Translation for Enterprise-Scale Programs - An Early Adoption Case Study (2018)

Abstract: While Neural Machine Translation (NMT) technology has been around for a few years now in research and development, it is still in its infancy when it comes to customization readiness and experience with implementation on an enterprise scale with Language Service Providers (LSPs). For large, multi-language LSPs, it is therefore not only important to stay up-to-date on latest research on the technology as such, the best use cases, as well as main advantages and disadvantages. Moreover, due to this infancy, the challenges encountered during an early adoption of the technology in an enterprise-scale translation program are of a very practical and concrete nature and range from the quality of the NMT output over availability of language pairs in (customizable) NMT systems to additional translation workflow investments and considerations with regard to involving the supply chain. In an attempt to outline the above challenges and possible approaches to overcome them, this paper describes the migration of an established enterprise-scale machine translation program of 28 language pairs with post-editing from a Statistical Machine Translation (SMT) setup to NMT.

## introduction the idea of using recurrent neural networks for machine translation was first presented by @xcite , followed soon after by @xcite and @xcite . in a mere three years from those papers, nmt systems were outperforming smt systems for several translation tasks at the association for computational linguists' conference on machine translation (wmt). at the same time, large translation providers such as systran, google and microsoft announced deployments of nmt systems for public consumption. the combination of these factors quickly made both buyers and providers of translation services aware of the new opportunities. the rapid emergence of nmt has necessitated that lsps focus on many new areas, including: qualitative evaluation of individual nmt systems, comparing translation quality and productivity of nmt and smt systems, implementation and deployment of nmt systems, and building customized nmt systems for specific domains and/or clients. ## contextualization since the deployment of machine translation technology for commercial use, and especially the breakthrough of statistical mt solutions, requests for mt as part of regular translation programs have constantly been on the rise. an explosion in the amount of content published as well as increasing pressure to publish content fast and simultaneously in different target markets and languages have caused clients to look into alternative, cheaper options and lsps to adjust their translation workflows and processes. the continually improving quality of mt systems and new developments such as nmt add to this demand. as a major global lsp, we count a range of big global companies among our end clients, for whom we typically provide ongoing, on-demand translation services into 20+ languages, covering various content types (= enterprise-scale translation program). it is our role to advise our clients on new developments in (mt) technology, opportunities for automation and workflow improvement as well as cost and time savings in their translation needs. in our case, we do not work with one specific mt provider, but recommend the mt solution we consider the best fit for a given end client, based on their specific needs and setup. the arrival of nmt therefore requires us to reevaluate existing mt programs as well as the mt solutions offered by different providers we work with. ## planning for nmt for enterprise-scale programs since its breakthrough, nmt quickly showed great promise to be able to deliver noticeably higher quality raw machine translations, especially for historically challenging and expensive translation pairs like english-japanese. in our planning, we therefore started to evaluate a range of the then available, initially generic nmt systems for their qualitative performance on a subset of languages. the evaluation of these generic nmt systems was performed with suitable test content from clients that gave us their permission to use their content for this purpose. we compared these generic systems with the existing, customized smt solutions that were in place for the respective client programs, using automatic scoring for bleu, gtm, nist, meteor, precision, recall, ter and edit distance (levenshtein foot_1 ), a postediting test and human evaluations (see 3.2 evaluation methodology for details). while generic nmt frequently outperformed customized smt on various metrics, the results were inconsistent across content types and languages. lacking lexical coverage from the generic systems added to this picture, with some languages benefitting more from the increased fluency and grammatical accuracy of the nmt system (e.g. japanese) while other languages seemed to struggle more with the terminological inaccuracies (e.g. german), at least from a human evaluation viewpoint. selected results from this study were presented during the 2017 machine translation summit in nagoya, japan, the 2017 school of advanced technologies for translators in trento, italy, and with the translation automation user society's (taus) mt user group @xcite . while results were still mixed at this early stage, they showed that, for some languages, already the generic nmt systems were performing equally well when compared with the established, customized smt systems. with mt providers starting to make customizable nmt solutions available and the promise in relation to an even better performance from these, we then progressed to direct comparisons on custom smt to custom nmt, partly in the form of official client pilot projects. in the following paragraphs, we outline the different phases in the pilot, evaluation and subsequent migration to a customized nmt solution for a translation program of 28 languages. ## acknowledgement we would like to thank our colleagues elaine o'curran, alex yanishevsky, naoko miyazaki and david landan for their contribution.