ICL ’s Submission to the WMT 21 Critical Error Detection Shared Task (2021)

Abstract: This paper presents Imperial College London’s submissions to the WMT21 Quality Estimation (QE) Shared Task 3: Critical Error Detection. Our approach builds on cross-lingual pre-trained representations in a sequence classification model. We further improve the base classifier by (i) adding a weighted sampler to deal with unbalanced data and (ii) introducing feature engineering, where features related to toxicity, named-entities and sentiment, which are potentially indicative of critical errors, are extracted using existing tools and integrated to the model in different ways. We train models with one type of feature at a time and ensemble those models that improve over the base classifier on the development (dev) set. Our official submissions achieve very competitive results, ranking second for three out of four language pairs.

## introduction critical error detection (ced) is a new task which has been introduced in the wmt21 quality estimation (qe) shared task. 1 the purpose of ced is to address a challenging problem in machine translation (mt): translations produced by state-of-the-art mt systems can be grammatical and fluent but do not always retain the meaning of the source text. more importantly, incorrect translations can be misleading and even have catastrophic consequences such as health, safety, legal, or financial implications. however, these can be hard errors to capture by general qe architectures, which have been shown to be prone towards relying mainly on the translated sentence @xcite . according to the shared task definition, a critical translation error is a type of error that occurs when the meaning of the translation deviates from source sentence in a critical way. the task data (section 2.1) includes five categories of such errors: deviation in toxicity (tox), in named entities 1 http://statmt.org/wmt21/quality-estimation-task.html (nam), in sentiment polarity or negation (sen), or in numbers (num), or introduction of health or safety risks (saf). the baseline model for this task utilises the xlm-roberta @xcite for sequence classification model, following the mono-transquest architecture proposed by @xcite . inspired by the fact that these five critical error types refer to specific linguistic phenomena, we aim to bring additional information to the models on the presence of such phenomena. the intuition is that sentences containing certain types of linguistic features, such as named entities or dates, are more likely to lead to errors. therefore, we first process the dataset to extract features reflecting the sentences' toxicity, sentiment and named entities, using off-the-shelf toolkits or apis (section 2.2). we then enhance the baseline architecture with this additional information. 2 we experiment with two approaches to take the additional features into account, at token and hidden state levels. we build multiple models taking one type of feature at a time and finally ensemble "promising" models. promising models are those that lead to improvements over the baseline on the dev set (section 2.3). our results comparing different features show that some of the features are indeed useful, but there is no general pattern that applies to all language pairs (section 3.1). the official submission, which uses an ensemble of the models that lead to improvements over the baseline on the dev set for each language shows that ensembling only models with promising features are better than ensembling models with all kinds of features (section 3.2). upon manual inspection, we observed that additional features indeed help the model to make predictions but this is subject to the accuracy of features (section 3.3). ## conclusions this paper describes our submission to sentencelevel ced task in wmt21. our work extends the baseline monotransquest architecture by exploring feature engineering and model ensembling, as well as weighted sampling to deal with imbalanced datasets. potentially due to the skewed distribution of labels in the dataset, the model performance varies substantially over different runs. however, our results averaged over multiple random seeds show that our feature engineering and ensembling lead to large improvements over the baseline. our official submission achieves the 2nd position in en-cs, en-de, en-ja, and the 4th postion in en-zh.