TMUOU Submission for WMT 20 Quality Estimation Shared Task (2020)

Abstract: We introduce the TMUOU submission for the WMT20 Quality Estimation Shared Task 1: Sentence-Level Direct Assessment. Our system is an ensemble model of four regression models based on XLM-RoBERTa with language tags. We ranked 4th in Pearson and 2nd in MAE and RMSE on a multilingual track.

## introduction quality estimation (qe) is a task of estimating translation quality without reference sentences @xcite @xcite . automatic evaluation metrics based on reference sentences, such as bleu @xcite , have contributed to improving translation quality on benchmark datasets. however, in situations where machine translation (mt) is actually used, these metrics are sometimes unable to assess the translation quality owing to the lack of reference sentences. the development of qe methods that are well correlated with manual evaluations enable users to decide whether to use the translation results as is, post-edit the results, or employ other machine translations. at the conference on machine translation (wmt), there have been conducted several qerelated competitions such as the qe task @xcite for estimating post-edit rate hter @xcite and the qe as a metric task @xcite for relative evaluations of translation quality. this year, the wmt qe task held a new competition @xcite on absolute evaluations of translation quality. in task 1, sentences are annotated with direct assessment (da) scores as in the metrics task @xcite . 1 tokyo metropolitan university and osaka university we have been working on the metrics task with an approach that uses pre-trained sentence encoders @xcite . @xcite employed infersent @xcite , quick-thought @xcite , and universal sentence encoder @xcite as encoders, and achieved the highest performance in all to-english language pairs of wmt18 metrics shared task @xcite . subsequently, @xcite employed bert @xcite as an encoder to further improve the correlation with manual evaluations. in this study, we apply similar approaches to the qe task. however, to support both source and target languages, we employ xlm-roberta foot_0 @xcite , a pre-trained multilingual sentence encoder. ## wmt20 qe shared task 1 in the wmt20 qe task 1 (sentence-level direct assessment), participants predict translation quality at the sentence level from pairs of source and mt output sentences. this task provides datasets for seven language pairs and sets up a multilingual track for a language-independent approach. ## tmuou system our system is an ensemble model of four regression models based on xlm-roberta @xcite with language tags. we first explain each base model in section 3.1, and then introduce the ensemble model in section 3.2. finally, section 3.3 describes the implementation details. ## results ## conclusions we describe the tmuou submission for the wmt20 shared task on quality estimation. our system is an ensemble model based on xlm-roberta, which takes into account averaged token vectors and language identifiers to improve performance. in the official evaluation, we ranked 4th in pearson and 2nd in mae and rmse on a multilingual track.