APE - QUEST : an MT Quality Gate (2020)

Abstract: The APE-QUEST project (2018–2020) sets up a quality gate and crowdsourcing workflow for the eTranslation system of EC’s Connecting Europe Facility to improve translation quality in specific domains. It packages these services as a translation portal for machine-to-machine and machine-to-human scenarios.

## overview the ape-quest project (automated postediting and quality estimation) is funded by the ec's cef telecom programme (project 2017-eu-ia-0151) which started in october 2018 and runs until september 2020. the project offers a quality gate connected to the etranslation mt system, developed by the directorate-general for translation and provided by the cef automated translation building block of the directorate-general for communications networks, content and technology (dg cnect) as a service to digital service infrastructures (dsis) of the ec and to public administrations of member states. the project consortium consists of two companies, crosslang (coordinator) and unbabel, and the university of sheffield. ape-quest consists of a quality gate which injects qe and ape into the translation workflow. the main goals of this injection are (1) to reach the desired translation quality in an efficient and reliable way using @xcite to create data aggregation opportunities by making translations and post-edits "locally owned", as data is generated and curated at the end user's site, in accordance with the similar main principle of the ec's elrc action. 1 the ape-quest project focuses on mature technologies by integrating systems for mt, qe and ape. tests involve three language pairs (english to portuguese, french and dutch) and three domains (legal-domain text, procurement and online dispute resolution). ## architecture the workflow consists of three tiers: (1) mt output with acceptable quality flows directly to the end user or connected system, (2) moderatequality mt is enhanced through ape, and (3) low-quality mt is sent to a workflow for human pe. the input consists of text snippets (messages) or full text documents from the project's stakeholders, such as dsis, public services in member states, and organisations involved in cef telecom projects adopting etranslation. the input is provided through an api or user interface and is segmented into sentences to allow for routing sentences to different tiers based on the detected quality. the pe workflow allows for collecting user data for system improvement, i.e. data for adapting the mt system and re-training qe and ape systems. the components of the workflow have been successfully integrated and it is fully operational. it is interoperable with the etranslation system, it is conformant with the ec's edelivery building block in order to ensure secure and reliable data transfer, and it contains a portal-style front end (an adaptation of matecat). 2 ## outcome the human pe data created in the framework of the project is available in the elrc-share repository, 3 which collects, prepares and shares language resources. this data consists of around 30k tuples including a source sentence, a corresponding neural mt output, a post-edited version created by a professional translator, and the original reference translation crawled from parallel 2 http://www.matecat.com 3 name of the resource: "post-editing corpus english to dutch/french/portuguese, legal domain" language websites in the legal domain. all data were anonymized. tests with the quality gate are ongoing in order to determine qe thresholds. the ape component for these tests is based on neural copycat networks, while the open-source framework openkiwi is used for qe. both are described in @xcite . the thresholds aim at establishing a trade-off between automatic processing (mt and ape) and human pe, in terms of translation speed, quality and cost. the tests involve two use cases, i.e. dissemination and assimilation of translated texts.