Machine Translation Reference-less Evaluation using Y i S i-2 with Bilingual Mappings of Massive Multilingual Language Model (2020)

Abstract: We present a study on using YiSi-2 with massive multilingual pretrained language models for machine translation (MT) reference-less evaluation. Aiming at finding better semantic representation for semantic MT evaluation, we first test YiSi-2 with contextual embed- dings extracted from different layers of two different pretrained models, multilingual BERT and XLM-RoBERTa. We also experiment with learning bilingual mappings that trans- form the vector subspace of the source language to be closer to that of the target language in the pretrained model to obtain more accurate cross-lingual semantic similarity representations. Our results show that YiSi-2’s correlation with human direct assessment on translation quality is greatly improved by replacing multilingual BERT with XLM-RoBERTa and projecting the source embeddings into the tar- get embedding space using a cross-lingual lin- ear projection (CLP) matrix learnt from a small development set.

## introduction the machine translation quality estimation as a metric (qe as a metric) task was first introduced in wmt 2019 @xcite to encourage the exploration of reference-less evaluation metrics. qe as a metric task shifts the use case of the qe systems from assisting professional translators to estimate post-editing efforts to assisting mt developers or general mt users to discriminate the translation quality of different mt systems without the presence of a human reference translation. yisi-2, the reference-less variants of the yisi metric @xcite , was the only metric who participated in evaluating all the translation directions in wmt 2019 qe as a metric shared task. the qe as a metric task is very similar to task 1 (sentence-level direct assessment) of wmt20's quality estimation shared task where metric performance is evaluated in terms of correlation at the sentence-level with human direct assessment scores on translation quality. the subtle but crucial difference between the wmt20 qe task 1 and the qe as a metric task is that qe systems for the former task is trained specifically to estimate the quality of a single mt system whereas qe metrics for the latter task is generalized for multiple machine translation systems. the qe systems for wmt20's qe task 1 have access to the mt system that generate the translations while the referenceless metrics for the latter task have no information on the mt systems being evaluated. in wmt 2019 metrics shared task, pretrained multilingual bert @xcite was used in yisi for both mt reference-based (yisi-1) and reference-less (yisi-2) evaluation in all tested translation directions where monolingual pretrained bert model was not available for the target language (such as czech, german, etc.). since then, another massive multilingual pretrained language model, xlm-roberta @xcite , has been published. we evaluate the use of contextual embeddings extracted from each of the intermediate layers of the two models in mt reference-less evaluation. in addition, despite using the same pretrained embedding model of last year, yisi-2 showed a significant performance degradation when comparing to yisi-1. for example, segment-level correlation with human direct assessment for evaluating en-@xmath0.475 (yisi-1) to 0.069 (yisi-2). this shows that the cross-lingual semantic representation in pretrained multilingual bert is not as accurate as the monolingual semantic representation for each language. in other words, we observed the language clustering effect where a clear segregation of vector subspace among different languages in the multilingual contextual em-bedding model. inspired by @xcite , we employ a weakly-supervised bilingual mapping learnt from a small development set that transforms the contextual embeddings of the source sentence to the target subspace for better cross-lingual semantic similarity evaluation. in this paper, we show that yisi-2's correlation with human direct assessment on translation quality is greatly improved by replacing multilingual bert with xlm-roberta large using the optimal intermediate layer (7 th layer count from the last) and projecting the source embeddings into the target embedding space using a cross-lingual linear projection matrix learnt from a small development set. ## yisi-2 yisi @xcite is a unified semantic mt quality evaluation and estimation metric for languages with different levels of available resources. yisi-1 measures the similarity between a machine translation and human references by aggregating weighted distributional (lexical) semantic similarities, and optionally incorporating shallow semantic structures. improvements in yisi-1 for wmt 2020 metrics shared task is detailed in @xcite . yisi-2 is the bilingual, reference-less version, which uses bilingual word embeddings to evaluate cross-lingual lexical semantic similarity between the input and mt output. ## results we use wmt 2019 metrics task evaluation set @xcite for our development experiments. the official human judgments for translation quality of wmt 2019 were collected using reference-based direct assessment. since we use exactly the same correlation analysis as the official metrics shared evaluation and the 2019 version of yisi performed consistently well among participants in wmt 2019, we only compare our results with the 2019 version of yisi and bleu. our results are directly comparable with those reported in @xcite . ## conclusion we have presented an improved version of yisi-2 that uses xlm-roberta and a cross-lingual linear projection of the source embedding to the target language subspace to better capture the semantic representation across languages. our results show that yisi-2 correlates better with human judgement on evaluating translation quality than bleu for most of the evaluation conditions. this improved version of yisi-2 is submitted to the wmt 2020 metrics shared task qe as a metric track. for evaluating inuktitut↔english where one of the language (inuktitut) is not covered by xlm-r, we build our own xlm cross-lingual language model with the parallel training data. potential research directions definitely include improving massive multilingual pretrained language model to close the performance gap between yisi-1 and yisi-2 and expanding the language coverage of these models in post-hoc and unsupervised manner.