IOL Research’s Submission for WMT 2023 Quality Estimation Shared Task (2023)

Abstract: This paper presents the submissions of IOL Research in WMT 2023 quality estimation shared task. We participate in task 1 Quality Estimation on both sentence and word levels, which predicts sentence quality score and word quality tags. Our system is a cross-lingual and multitask model for both sentence and word levels. We utilize several multilingual Pretrained Language Models (PLMs) as backbones and build task modules on them to achieve better predictions. A regression module on PLM is used to predict sentence level score and word tagging layer is used to classify the tag of each word in the translation based on the encoded representations from PLM. Each PLM is pretrained on quality estimation and metrics data from the previous WMT tasks before finetuning on training data this year. Furthermore, we integrate predictions from different models for better performance while the weights of each model are automatically searched and optimized by performance on Dev set. Our method achieves competitive results.

## introduction quality estimation (qe) is the task of predicting the quality of a target machine translation without using reference texts or human inputs @xcite . since machine translation is in high demand nowadays, the development of qe system becomes crucial for the broad application of machine translation. in wmt 2023 quality estimation shared task, there are two tasks: quality estimation and fine-grained error span detection. this paper describes our submission to task 1 quality estimation in both sentence and word levels in detail. considering the powerful capability and widely used in previous qe tasks of pretrained language models (plms) @xcite , our method utilizes different multilingual plms to encode source-translation sentence pairs and predict sentence-level scores or word-level tags. such plms are pretrained on various languages which could show incredible ability when trained qe models are transferred to unseen language pairs. meanwhile, extra task modules are added to plms to boost the interaction between source and translation sentences to make better predictions. also, it is common that using other task data similar to qe can further improve the performance of qe. according to the results from previous years' qe tasks, we use the data from qe and metrics tasks from previous years' wmt tasks, as well as automatic post-editing (ape) data, to pretrain plms before training on data of this year. moreover, ensemble methods of different models are explored in sentence and word level tasks. for sentence level, we sum scores with weights from different models which are filtered by the performance on the dev set. as for word level, we use voting or weighted sum of tag probabilities to get the final predicted tags. taking zero-shot language pairs into account, we choose the best model evaluated on other language pairs to test if they can generalize to unseen language pairs. 2 quality estimation task 2.1 task description wmt 2023 quality estimation task 1 contains two tasks. the sentence level task aims at predicting a quality score for translation and the word level task is to classify a quality tag for each word in translation. both tasks have zero-shot language pairs to test the generalization ability of qe models and use the same source-translation pairs for each language pair. sentence level there are two types of quality scores. one is the direct assessments (da) score which is given by human annotators for each source-translation pair. the other is the multidimensional quality metrics (mqm) score which is defined and computed under mqm methodology. train_sent train_word train_mtl 224195 263184 105992 ## conclusion this paper describes our work for wmt 2023 quality estimation task 1 on both sentence level and word level. with the help of plms and extra data, we can train better representations of source text and its translation for quality estimation task. we also experiment with diverse combinations of plms, task modules, and pretraining datasets. we find that qe systems for certain language pairs need to adopt particular combinations to acquire improvement, which reveals that there are distinct characteristics between languages. such features make it hard to build one model for all languages, especially those without labeled data. the multitask training approach shows obvious improvements and prevents models from overfitting. besides, the score refinement trick does not always give us positive feedback which suggests the number range is not the only factor to train on da and mqm data properly. as expected, the ensemble method makes the predictions have a higher correlation with the ground truth. for future work, we will explore more profitable pretraining techniques for quality estimation and efficient modules that work well for various language pairs.