A Quality Estimation and Quality Evaluation Tool for the Translation Industry (2022)

Abstract: With the increase in machine translation (MT) quality over the latest years, it has now become a common practice to integrate MT in the workflow of language service providers (LSPs) and other actors in the translation industry. With MT having a direct impact on the translation workflow, it is important not only to use high-quality MT systems, but also to understand the quality dimension so that the humans involved in the translation workflow can make informed decisions. The evaluation and monitoring of MT output quality has become one of the essential aspects of language technology management in LSPs’ workflows. First, a general practice is to carry out human tests to evaluate MT output quality before deployment. Second, a quality estimate of the translated text, thus after deployment, can inform post editors or even represent post-editing effort. In the former case, based on the quality assessment of a candidate engine, an informed decision can be made whether the engine would be deployed for production or not. In the latter, a quality estimate of the translation output can guide the human post-editor or even make rough approximations of the post-editing effort. Quality of an MT engine can be assessed on document- or on sentence-level. A tool to jointly provide all these functionalities does not exist yet. The overall objective of the project presented in this paper is to develop an MT quality assessment (MTQA) tool that simplifies the quality assessment of MT engines, combining quality evaluation and quality estimation on document- and sentence- level.

## mtqa tool overview the mtqa is designed as a standalone tool and an api that can be used by users or invoked by other tools. behind the user interface lies a distributed architecture which operates in two modes. intermediate and final results are displayed to the user; final results are made available for download. mtqev is a human-driven quality assessment module, in which one or more mt systems' quality is evaluated based on a human-generated translation reference. mtqev is typically used to compare already-in-use and new mt models by means of comparing their translation to the human gold standard, using automatic metrics, such as ter @xcite , bleu @xcite , chrf @xcite and others. in our mtqa tool, this mode shall be used to take an informed decision on a business level about the models to be deployed in production for different language combinations and domains. mtqe (machine translation quality estimation) is the process of predicting the quality of an mt system without human intervention or reference translations. mtqe can be at a word, sentence, or document level. in the case of document and sentence level, which are of interest for our project, the task is typically to predict a score that corresponds to a target evaluation criteria or metric. mtqe is the second mode the tool will be able to work in. instead of comparing the mt output with an existing human translation, this mode will be used at the beginning of each translation project to evaluate the quality of the output by predicting the approximate number of changes a given mt output should undergo to reach acceptable quality. this mode shall be used to evaluate the usability of mt models for each project, in order to choose the best possible starting point for the pe (postediting) and therefore to better allocate time and resources. both modes will be able to work both on document level and on segment level, so that the endusers, e.g., the project managers, will be able to choose the level of granularity they want to get to take a well-informed decision. to facilitate the use of the tool across all business workflows and for each use case, both modes will be integrated and independent from the other language technology tools that lsps usually work with. for mtqev mode we employ the metrics ter, bleu and chrf. for mtqe we first build neural qe models with the data described in section 3; we then employ these models to score input data. ## working with industry data an lsp could use either publicly available mt engines, or proprietary mt engines, trained specifically for the given translation use case, which would employ data, usually provided by the lsp to train a domain-specific and use-case-specific mt engine with highest quality. the data an lsp usually translates is proprietary and cannot be publicly accessible, even for research purposes. a collaboration such as the one this project is based on, between orbital14 and tilburg, allows researchers and industry to work together on real use cases and proprietary data. within the scope of this project, we exploit data that has been translated via trained and generic mt engines and was post-edited by aglatech14 in the context of several translation projects. these data allow us to experiment with and build effective qe models that can be employed in the mtqa tool. for this project we employ english-italian data from the patent domain. to this end, two types of data were provided by aglatech14: (i) data that had been post-edited, in which case three documents were provided, source, mt output, and a post-edited version of the output (s-mt-pe); and (ii) data that had been translated by professional human linguists in the original project, in which case source and (human) translation (s-t) were provided. to train our qe models we employed three different data sets: (i) the s-mt-pe documents; (ii) the s-t documents for which we translated the source using aglatech14's mt engines and generated an s-mt-pe* corpus; and (iii) the open data sets binqe @xcite and es-cape @xcite . for all data we computed the ter score between the mt and the postedited or translated reference. this we used as target labels for our mtqe models.