NICT K yoto Submission for the WMT ’21 Quality Estimation Task: Multimetric Multilingual Pretraining for Critical Error Detection (2021)

Abstract: This paper presents the NICT Kyoto submission for the WMT’21 Quality Estimation (QE) Critical Error Detection shared task (Task 3). Our approach relies mainly on QE model pretraining for which we used 11 language pairs, three sentence-level and three word-level translation quality metrics. Starting from an XLM-R checkpoint, we perform continued training by modifying the learning objective, switching from masked language modeling to QE oriented signals, before finetuning and ensembling the models. Results obtained on the test set in terms of correlation coefficient and F-score show that automatic metrics and synthetic data perform well for pretraining, with our submissions ranked first for two out of four language pairs. A deeper look at the impact of each metric on the downstream task indicates higher performance for token oriented metrics, while an ablation study emphasizes the usefulness of conducting both self-supervised and QE pretraining.

## introduction this paper describes the nict kyoto submission to the wmt'21 quality estimation (qe) shared task. we participated in task 3 "critical error detection" involving four language pairs, namely english-chinese, english-czech, english-japanese and english-german. a critical error is defined as a translation error falling into one of the following five categories: toxicity, health or safety risk, named entity, sentiment polarity and number or unit deviation. 1 the objective of the task is to classify a sequence pair, composed of a sentence in the source language and its automatic translation in the target language, in a binary fashion whether it contains or not at least one of the five types of critical errors. this task differs from the other qe tasks as not all translation errors should be detected but only critical ones. labels were produced by majority vote over three annotators for each pair leading to two possible classes: err (or class 1) when at least one critical error is spotted and no (or class 0) when no critical errors are present. our approach relies mainly on qe model pretraining leveraging a large amount of synthetic data produced using parallel corpora and mt systems. because annotating translations for critical error is costly, we propose to pretrain a model on translation quality scores computed with automatic metrics. to capture multiple translation error granularities during pretraining, we employ multiple metrics and evaluate their performance individually on the downstream task. additionally, we pretrain the qe model jointly on all wmt qe shared tasks language pairs as a data augmentation method. transfer learning is then conducted for each language pair by finetuning the pretrained model on the downstream task with the officially released training data annotated with critical errors. the remainder of this paper is organized as follows. in section 2, we introduce our approach involving multimetric and multilingual pretraining. in section 3, the data, tools and training procedure are presented, followed by the experimental results and their analysis in section 4, before the conclusion in section 5. ## multimetric & multilingual pretraining multilingual pretrained masked language models (lms) were shown to perform well in several downstream natural language processing tasks @xcite @xcite . starting from an xlm-r checkpoint @xcite , we performed continued (or intermediate) training @xcite with large amount of automatically translated source language texts (thereafter called synthetic data), replacing the masked lm objective with qe oriented ones. because xlm-r is multilingual and all languages in this model share a common vocabulary of sub-words, we decided to conduct qe pretraining on the 11 language pairs from all subtasks of wmt'21 qe. these language pairs all share english, whether on the source or target side, and this method can be seen as a data augmentation approach to increase vocabulary coverage. the objective of qe task 3 is to classify sentence pairs in a binary fashion. formally, given a source sequence s and its translation t, we want to learn a function f : f θ (s, t) → y where @xmath0, 1} is the class associated with the sequence pair (s, t) and θ represents the model parameters. while finetuning a pretrained model on the official qe task 3 data allows us to directly learn model parameters approximating y given (s, t), we do not have such classes for synthetic data. we decided to use mt automatic metric scores as objective instead, assuming that critical error classes could correlate with translation quality scores at least in extreme cases (e.g. no translation errors also means no critical errors). several automatic metrics are used by the research community to evaluate the performance of mt systems by measuring translation accuracy against a human-produced reference at different granularity levels. we opted for metrics capturing quality information at the character (chrf @xcite ), token (ter @xcite ) and token n-gram (bleu @xcite ) levels. for the latter, the smoothed sentence-level bleu was chosen @xcite . in addition to sentence-level metrics, token-level binary tags were also extracted following the usual procedure to determine post-editing effort @xcite ). 2 to allow for sentence-level qe predictions, we added a feed-forward layer on top of xlm-r for each of the three metrics employed without parameter sharing, following: all losses are linearly summed with equal weights before back-propagation. the parameters of the classification and regression heads are optimized along with xlm-r. ## data and tools this section presents the data used in our experiments, including the synthetic data produced for pretraining and the official qe task 3 corpora, along with the tools required to train our models and the procedure employed for both pretraining and finetuning. ## results and analysis we present in this section the main results obtained on the official shared task test set as reported by the organizers, followed by an analysis with ablation study and various pretraining objectives. ## conclusion this paper presented the nict kyoto submission for the wmt'21 qe task 3 "critical error detection". our submissions were ranked first for two out of four language pairs. our approach relies mainly on model pretraining with large amount of synthetic data, followed by finetuning on the official data released for the shared task. we proposed a novel qe pretraining approach which allows for a multimetric learning objective based on relatively cheap to compute mt automatic metrics. an analysis of each automatic metric used during qe pretraining shows the complementarity of metrics both at level of sentences and words. the ablation study emphasized the usefulness of both self-supervised and qe pretraining. future work focuses on exploring additional metrics and their performance on various downstream qe tasks.