Integrating MT at S wiss Post’s Language Service: preliminary results (2018)

Abstract: This paper presents the preliminary results of an ongoing academia-industry collaboration that aims to integrate MT into the workflow of Swiss Post’s Language Service. We describe the evaluations carried out to select an MT tool (commercial or open-source) and assess the suitability of machine translation for post-editing in Swiss Post’s various subject areas and language pairs. The goal of this first phase is to provide recommendations with regard to the tool, language pair and most suitable domain for implementing MT.

## introduction nowadays, the production environments of many companies incorporate mt for various reasons: it might be upon request of a client, an initiative to add new services to a company's assets or an attempt to cut costs and shorten delivery times. the technology can be developed by a third party or inhouse, each solution having its own pros and cons. swiss post's language service would like to integrate mt in their workflow in different contexts, ranging from gisting to professional postediting, thereby allowing for reduced turnaround times. hence, in collaboration with the university c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. the source data used to train and test the different systems for the various language pairs are almost parallel, making it possible to compare results across less-studied pairs. in addition, when designing our experimental setting, we chose to put the focus on users, namely swiss post's professional translators, providing them with specific training before involving them in the evaluation process. we are convinced that when reorganizing the traditional workflow of professional translators, it is important to give them an active role in the change in order to foster acceptance and avoid biased evaluation due to reluctant mt users. the paper is structured as follows: we first describe the available data for the various languages and subject areas (section 2), then explain how we selected the mt engine (section 3). we then present how the suitability of the mt for pe was assessed by swiss post's in-house translators (section 4) and discuss the results (section 4.4), before concluding (section 5). pérez-ortiz, sánchez-martínez, esplà-gomis, popović, rico, martins, van den bogaert, forcada (eds.) proceedings of the 21st annual conference of the european association for machine translation, p. 281-286 alacant, spain, may 2018. ## conclusion and future work we have presented the preliminary results of a project that aims to integrate mt into the workflow of swiss post's language service. the first part of the study was devoted to choosing between the commercial microsoft translator hub system and an in-house trained moses solution, both trained using language service's material. we decided to proceed with the latter, trained on mtm, since automatic scores were systematically better with this system and training configuration. this allowed us to use just one system per language pair. in the second part of the study, a human evaluation was carried out to assess the percentage of raw mt sentences perceived as suitable for professional post-editing. a sample of swiss post language service's professional translators was actively involved in this task. the outcomes of the evaluation were overall better for the subject area "process manuals" (pn). de-it evaluators assessed the highest percentage of usable sentences (with or without changes). more agreement among evaluators was also reported for this language pair. however, we sometimes found contradictions between human results and automatic scores, for instance in de-en, likely due to the fact that we only had two evaluators for this language pair. furthermore, gb scored worse with automatic metrics, but was the second best subject area, according to human evaluation. further investigation is required to discover the reasons behind this inconsistency between human ratings and automatic scores. all in all, we consider our results to be satisfactory: a percentage of usable sentences ranging from 84% (de-fr) to 96% (de-it) is a good threshold to start working with mt in a professional context. as for de-en, the 62.80% obtained suggests that in this case, raw mt output might be suitable, but to a lesser extent, so further work should be done in this direction. in the next phase, we will carry out a productivity test with the translators, in order to determine if implementing mt into language service's workflow could actually be cost effective. these tests will first involve the highest scored domain (pn), since we believe that a gentle introduction to mt as new working tool is necessary to make the most of it. finally, once translators are used to the new workflow, we would like to carry out a comparative evaluation of our pbmt system with the neural baseline we are currently training. this will allow us to compare both translators' productivity and satisfaction when using different mt architectures.