Translation Quality Estimation by Jointly Learning to Score and Rank (2020)

Abstract: The translation quality estimation (QE) task, particularly the QE as a Metric task, aims to evaluate the general quality of a translation based on the translation and the source sentence without using reference translations. Supervised learning of this QE task requires human evaluation of translation quality as training data. Human evaluation of translation quality can be performed in different ways, including assigning an absolute score to a translation or ranking different translations. In order to make use of different types of human evaluation data for supervised learning, we present a multi-task learning QE model that jointly learns two tasks: score a translation and rank two translations. Our QE model exploits cross-lingual sentence embeddings from pre-trained multilingual language models. We obtain new state-of-the-art results on the WMT 2019 QE as a Metric task and outperform sentBLEU on the WMT 2019 Metrics task.

## introduction the translation quality estimation (qe) task @xcite aims to evaluate the quality of a translation based on the translation and the source sentence without using reference translations. the qe task includes word-level qe, sentence-level qe, document-level qe and qe as a metric tasks. the qe as a metric task requires qe models to score a translation on the sentence level similar to the sentence-level qe task, but these two tasks are different as the goal of the sentence-level qe task @xcite is to predict the percentage of edits needed to fix the translation for post-editing purposes while the goal of the qe as a metric task is to estimate the general quality of the translation like machine translation (mt) evaluation metrics, such as bleu @xcite and meteor @xcite , except without using reference translations. supervised learning of the qe as a metric task requires human evaluation of translation quality as training data. human evaluation of translation quality is generally very costly and can be performed in different ways, such as direct assessment (da: requiring human assessors to assign an absolute score to a translation) @xcite @xcite or relative ranking (rr: requiring human assessors to rank different translations) @xcite . since the qe as a metric task requires qe models to assign an absolute score to a translation, da human evaluation data can be straightforwardly used as training data for the qe as a metric task. in order to also make use of the rr human evaluation data, we propose a multi-task learning qe model that jointly learns two tasks, score a translation and rank two translations. multi-task learning of these two closely related tasks enables us to use both da and rr human evaluation data for training the qe model and improve performance compared to learning these two tasks separately. our model performs translation quality estimation based on cross-lingual sentence embeddings from pretrained multilingual language models @xcite and does not need reference translations. we obtain new state-of-the-art results on the wmt 2019 qe as a metric task and outperform sentbleu on the wmt 2019 metrics task @xcite . a number of previous works also used sentence embeddings for evaluating translation quality @xcite @xcite . however, @xcite ; @xcite 's models only learn to score a translation and @xcite 's model only learns to rank two translations while our model jointly learns to score a translation and rank two translations in order to make use of different types of human evaluation data for model training. in addition, @xcite ; @xcite ; @xcite 's models use the reference translation for evaluating translation quality while our qe model does not require reference translations. there are existing qe models @xcite that do not need the reference translation and perform translation quality estimation based on cross-lingual word/sentence embeddings, but these qe models give relatively poor and unstable results for different language pairs @xcite while our qe model achieves more robust and better results. in addition, lo (2019); yankovskaya et al. (2019 )'s qe models only score a translation while our qe model jointly learns to score a translation and rank two translations via multi-task learning. ## our approach we propose a multi-task learning qe model that jointly learns two tasks: score a translation and rank two translations. our qe model is based on cross-lingual sentence embeddings from multilingual bert (m-bert) @xcite . to compute the sentence embedding for a given sentence, we feed this sentence into m-bert and then perform mean pooling over the output of m-bert to obtain fixedsize sentence embedding. we fine-tune m-bert for the qe tasks. the scoring task to score a translation t given the source sentence s, we use the cosine similarity between the source sentence embedding s and the target sentence embedding t as the score of the translation. foot_0 equation 1 gives the loss function for the scoring task, where y human (0 ≤ y @xmath0. the ranking task to rank two translations t 1 and t 2 given the source sentence s, we compute euclidean distances between source and target sentence embeddings euc dis s, t 1 and euc dis s, t 2 . the translation that has a smaller euclidean distance with the source is predicted to be the better translation. foot_1 equation 2 gives the loss function for the ranking task, where t b is the better translation and t w is the worse translation according to the human ranking. by minimizing l rank , we want euc dis s, t b to be at least ε less than euc dis s, t w . we tuned ε on the development set and finally set ε = 1 in our experiments. foot_2 multi-task learning we train our qe model to jointly learn the scoring task and the ranking task via multi-task learning. each training step includes two training batches: one training batch for the scoring task and one training batch for the ranking task. direct assessment (da) human evaluation data which requires human assessors to assign an absolute score to a translation is used as training data for the scoring task; relative ranking (rr) human evaluation data which requires human assessors to rank different translations is used as training data for the ranking task. the main advantage of multi-task learning for these two closely related tasks is that we can use both da and rr human evaluation data for training the qe model and improve performance compared to learning these two tasks separately. we test our method on the wmt 2019 qe as a metric task which requires qe models to assign an absolute score to a translation. we show that, on the qe as a metric task, our multi-task learning method can achieve significantly better results compared to only training the qe model to learn the scoring task with da human evaluation data. ## conclusion this paper presents a multi-task leaning qe model that jointly learns two tasks, score a translation and rank two translations. the scoring and ranking results performed by human assessors can be used as training data for learning the scoring and ranking tasks respectively. multi-task learning of these two closely related tasks enables us to make use of both types of human evaluation data for model training and improve performance compared to learning these two tasks separately. our model obtains new state-of-the-art results on the wmt 2019 qe as a metric task and outperforms sentbleu on the wmt 2019 metrics task.