Extended Study on Using Pretrained Language Models and Y i S i-1 for Machine Translation Evaluation (2020)

Abstract: We present an extended study on using pretrained language models and YiSi-1 for machine translation evaluation. Although the recently proposed contextual embedding based metrics, YiSi-1, significantly outperform BLEU and other metrics in correlating with human judgment on translation quality, we have yet to understand the full strength of using pretrained language models for machine translation evaluation. In this paper, we study YiSi-1’s correlation with human translation quality judgment by varying three major attributes (which architecture; which inter- mediate layer; whether it is monolingual or multilingual) of the pretrained language mod- els. Results of the study show further improvements over YiSi-1 on the WMT 2019 Metrics shared task. We also describe the pretrained language model we trained for evaluating Inuktitut machine translation output.

## introduction recent research on large-scale evaluation of automatic machine translation (mt) evaluation metrics @xcite @xcite showed that the newly proposed contextual embedding based metrics, like yisi-1, bertscore @xcite and esim @xcite , significantly outperform bleu @xcite and other metrics in correlating with human judgment on translation quality. yisi-1 and bertscore use contextual embeddings extracted from the pretrained language model, @xcite , as-is without further fine-tuning or fitting to existing labeled data predictions. although finetuning the pretrained language models for specific downstream tasks show improvements in many cases, using the pretrained language models without fine-tuning makes the mt evaluation metrics more portable to languages without labeled data and the resulted metric scores are comparable to each other over time across systems. thus, instead of spending efforts into fine-tuning the pretrained language models for mt evaluation, we focus on finding the most optimal way (which architecture; which intermediate layer; whether it is a monolingual or multilingual model) to utilize them as-is. zhang et al. ( 2020 ) investigated into a few aspects (architecture and layer) of the use of contextual embeddings in text generation evaluation. they evaluated several model architectures of different sizes, such as bert @xcite , roberta @xcite , xlnet @xcite and xlm @xcite . as more pretrained language models that cover more languages are released since then, we extend the study on yisi-1 to include more pretrained language models and also compare the effect of using monolingual pretrained models versus multilingual pretrained models. in this paper, we experiment on different settings of yisi-1 in the wmt 2019 metrics shared task, integrating it with different transformerbased @xcite contextual language models in both monolingual or multilingual, such as bert @xcite , albert @xcite , bart @xcite , roberta @xcite , xlm-roberta @xcite and xlnet @xcite , using different intermediate layers. we show that yisi-1's correlation with human judgment on translation quality is improved by using the results of this study. ## yisi yisi @xcite is a unified semantic mt quality evaluation and estimation metric for languages with different levels of available resources. yisi-1 measures the similarity between a machine translation and human references by aggregating weighted distributional (lexical) semantic similarities, and optionally incorporating shallow semantic structures. yisi-0 is the degenerate version of yisi-1 that is ready-to-deploy to any languages by using longest common character substring, instead of cosine similarity of contextual embeddings, to measure lexical similarity. yisi-2 is the bilingual, reference-less version, which uses bilingual word embeddings to evaluate cross-lingual lexical semantic similarity between the input and mt output, and optionally incorporating shallow semantic structures. improvements in yisi-2 for wmt 2020 metrics shared task is detailed in @xcite . ## experiments and results we use wmt 2019 metrics shared task evaluation set @xcite for our development experiments. the official human judgments for translation quality of wmt 2019 were collected using reference-based direct assessment. since we use exactly the same correlation analysis as the official metrics shared evaluation and the 2019 version of yisi performed consistently well among participants in wmt 2019, we only compare our results with the 2019 version of yisi and bleu. our results are directly comparable with those reported in @xcite . ## conclusion we have presented an extend study of the pretrained language models used in yisi-1 for machine translation evaluation. from this study, we conclude that for the best performance of yisi-1: 1) when evaluating mt output in english, it is recommended to use the contextual embeddings extracted from layer -6 of roberta large ; 2) when evaluating mt output in languages where monolingual pretrained model in the same or general domain is available, it is recommended to use the contextual embeddings extracted from those models; and finally 3) when evaluating mt output in languages only covered by multilingual pretrained language models, it is recommended to use the contextual embeddings extracted from layer -7 of xlm-roberta. this improved version of yisi-1 is submitted to the wmt 2020 metrics shared task. for evaluating inuktitut↔english where one of the language (inuktitut) is not covered by any released pretrained langauge model, we build our own xlm cross-lingual language model with the parallel training data.