QUARTZ : Quality-Aware Machine Translation (2022)

Abstract: This paper presents QUARTZ, QUality-AwaRe machine Translation, a project led by Unbabel which aims at developing machine translation systems that are more robust and produce fewer critical errors. With QUARTZ we want to enable machine translation for user-generated conversational content types that do not tolerate critical errors in automatic translations.

## introduction despite the progress in the fluency of machine translation (mt) systems, critical translation errors are still frequent, including deviations in meaning through toxic or offensive content, hallucinations, mistranslation of entities with health, safety, or financial implications, or deviation in sentiment polarity or negation. these errors occur more often when the source sentence is out of domain or contains typos, abbreviations, or capitalized text, all common with user-generated content. this lack of robustness prevents the use of mt systems in practical applications where the above errors cannot be tolerated. quartz aims to build reliable, quality-aware mt systems for user-generated conversational data. the project will address the limitations above by: (a) developing quality metrics capable of detecting critical errors and hallucinations; (b) endowing mt systems with a confidence (quality) © 2022 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 funding from the european union's horizon 2020 research and innovation program under grant agreement no 951847 score, and fine-tuning pre-trained mt models to the domains in which they will be used through quality-driven objectives. this will be done by leveraging post-edited data and quality annotations produced by the unbabel community and building upon the state-of-theart, open-source quality estimation technology already existing at unbabel: openkiwi @xcite and comet @xcite . from a product perspective, focus will be given to conversational, user-generated data in a multilingual customer service scenario (email or chat involving a customer and an agent), in which unbabel has renowned expertise and existing technology validated by existing customers. the solution aims to eliminate language barriers in the highly multilingual european market. ## mt and translation quality the current state of the art in mt is based on autoregressive sequence-to-sequence models trained with maximum likelihood and teacher forcing. this objective encourages the model to assign high probability to reference translations, but does not account for the severity of translation mistakes of the hypotheses generated. this leads to exposure bias, vulnerability to adversarial attacks, and no control for hallucinations, harmful content, and biases @xcite , hampering the responsible use of nmt for user-generated conversational content. project overview qualitative evaluation carried out by translators (post-editors and annotators) provides a human feedback loop that can generate large amounts of data with information about translation errors, their severities, and detailed quality annotations. the main methodology used to evaluate translations according to different aspects of translation quality is the industry-figure 1 : in quartz quality estimation systems will interact directly with the machine translation system during the decoding phase to avoid critical errors. words marked in red are considered errors. related work prior work on minimum bayes risk (mbr) decoding paves the way to tune mt systems towards a given metric, but so far this has been done with purely lexical metrics such as bleu @xcite or neural metrics that do not capture severity and biases @xcite . the main difference between quartz and previous work is going beyond lexical metrics in incorporating quality scores for generating automatic translations.