Welocalize- ARC / NKUA ’s Submission to the WMT 2022 Quality Estimation Shared Task (2022)

Abstract: This paper presents our submission to the WMT 2022 quality estimation shared task and more specifically to the quality prediction sentence-level direct assessment (DA) subtask. We build a multilingual system based on the predictor–estimator architecture by using the XLM-RoBERTa transformer for feature extraction and a regression head on top of the final model to estimate the z -standardized DA labels. Furthermore, we use pretrained models to extract useful knowledge that reflect various criteria of quality assessment and demonstrate good correlation with human judgements. We optimize the performance of our model by incorporating this information as additional external features in the input data and by applying Monte Carlo dropout during both training and inference.

## introduction machine translation quality estimation (mtqe) is the task of automatically estimating the quality of the mt output without using reference translations or any other human input @xcite @xcite . mtqe has many use cases and can be applied in various settings @xcite . it can be used to estimate the post-editing effort, to rank and compare outputs of different mt systems or to classify the segments that need post-editing. it can also be used to estimate the quality of the final translations as well as to filter out noisy segments from translation memories or training datasets. mtqe techniques usually have multiple granularity levels and can be applied to a word, phrase, sentence or even to an entire document. such systems are highly efficient when a vast amount of machine translated segments need to be evaluated in less time, with less effort and lower costs compared to traditional evaluation techniques. the wmt 2022 quality estimation shared task includes the following separate tasks: quality pre-diction, explainable qe and critical error detection. our team participated in the quality prediction sentence-level direct assessment (da) subtask with a multilingual mtqe system. specifically, we developed a cross-lingual mtqe system following the predictor-estimator architecture @xcite . we used the large-scale pretrained xlm-roberta (xlm-r) foot_0 model @xcite for feature extraction, similarly to @xcite . we combined the model's output with additional external features that demonstrate good correlation with the target variable. we then used the concatenated vector as input to our final mtqe regression model. our regressor is a feed-forward neural network with a linear output layer used to estimate the z-standardized da labels. ## quality prediction: sentence-level direct assessment the quality prediction task of the wmt 2022 quality estimation shared task consists of a sentencelevel and a word-level subtask. using the provided annotated training data, the objective of the sentence-level direct assessment subtask is to develop a system that automatically estimates a quality score for each provided sentence pair which is highly correlated with human-generated z-standardized da values. ## experimental results in this section we present the performance of our model on the provided test dataset for the wmt 2022 shared task on quality evaluation for the prediction of sentence-level direct assessments. in particular, our model outperformed the baseline system in terms of spearman and pearson correlation in all the multilingual and bilingual tasks, in which we participated, as illustrated in the tables 3 and 4 respectively. based on the official results, it seems that the lowest performing language pair, for both our model and the baseline, is english-japanese while the highest performing one is khmer-english. we did not further examine the reasons of this pattern, as we considered this exercise out of the scope of our study. in a future work, it would be useful to investigate whether certain factors contribute to this pattern (such as the source and target language complexity, the writing script, the performance of the pretrained models used to generate the features for each language, or even the content of the test dataset). it is also worth mentioning that for most language pairs of the test sets, as illustrated in the the results of the test set from the official leaderboard 6 for each language pair, in which we participated, can be found in the official results of the wmt 2022 qe task 1 -sentence-level direct assessment. in these tables, our proposed model is compared to the baseline system in terms of rmse, mae, spearman and pearson correlation coefficient. ## conclusions this paper presents our submission to the wmt 2022 quality estimation task 1 on sentence-level direct assessment. we introduce a model trained based on the predictor-estimator architecture using the xlm-roberta 1 for feature prediction and a regression head to finally estimate the z-standardized da values. we suggest the use of additional external features that reflect different criteria of human judgements and multiple levels of translation quality. these features exhibit good correlation with the target variable and consequently with human annotations. our approach is applicable in multilingual settings even with languages or writing scripts not explicitly seen during the training of the mtqe model. our system demonstrates competitive results and a strong correlation with human judgements of quality assessment outperforming the baseline system in terms of both spearman and pearson correlation coefficient.