S ent S im: Crosslingual Semantic Evaluation of Machine Translation (2021)

Abstract: Machine translation (MT) is currently evaluated in one of two ways: in a monolingual fashion, by comparison with the system output to one or more human reference translations, or in a trained crosslingual fashion, by building a supervised model to predict quality scores from human-labeled data. In this paper, we propose a more cost-effective, yet well performing unsupervised alternative SentSim: relying on strong pretrained multilingual word and sentence representations, we directly compare the source with the machine translated sentence, thus avoiding the need for both reference translations and labelled training data. The metric builds on state-of-the-art embedding-based approaches – namely BERTScore and Word Mover’s Distance – by incorporating a notion of sentence semantic similarity. By doing so, it achieves better correlation with human scores on different datasets. We show that it outperforms these and other metrics in the standard monolingual setting (MT-reference translation), a well as in the source-MT bilingual setting, where it performs on par with glass-box approaches to quality estimation that rely on MT model information.

## introduction automatically evaluating machine translation (mt) as well as other language generation tasks has been investigated for decades, with substantial progress in recent years due to the advances of pretrained contextual word embeddings. the general goal of such evaluation metrics is to estimate the semantic equivalence between the input text (e.g. a source sentence or a document) and an output text that has been modified in some way (e.g. a translation or summary), as well as the general quality of the output (e.g. fluency). as such, by definition metrics should perform some forms of input-output comparisons. however, this direct comparison has been proven hard in the past because of the natural differences between the two versions (such as different languages). instead, evaluation metrics have resorted to comparison against one or more correct outputs produced by humans, a.k.a. reference texts, where comparisons at the string level are possible and straightforward. a multitude of evaluation metrics have been proposed following this approach, especially for mt, the application we focus on in this paper. these include the famous bleu @xcite and meteor @xcite for machine translation, rouge @xcite for summarization, and cider @xcite for image captioning. these traditional metrics are based on simple-word, n-gram matching mechanisms or slight relaxations of these (e.g. synonyms) which are computationally efficient, but suffer from various limitations. in order to overcome the drawbacks of the traditional string-based evaluation metrics, recent work @xcite @xcite @xcite has investigated metrics that perform comparisons in the semantic space rather than at the surface level. notably, applications of word mover's distance (wmd; @xcite , such as wmdo @xcite , vifidel @xcite and moverscore @xcite , which compute similarity based on continuous word embeddings using pretrained representations. these have been shown to consistently outperform previous metrics on various language generation evaluation tasks. however, these metrics have two limitations: (i) they still rely on reference outputs, which are expensive to collect, only cover one possible correct answer, and do not represent how humans do evaluation; (ii) they are bag-of-embeddings approaches which capture semantic similarity at the token level, but are unable to capture the meaning of the sen-tence or text as a whole, including correct word order. in this paper, focusing on mt, to address these limitations we first posit that evaluation can be done by directly comparing the source to the machine translation using multilingual pretrained embeddings, such as multilingual bert, avoiding the need of reference translations. we note that this is different from quality estimation (qe) metrics @xcite , which also compare source and machine translated texts directly, but assume an additional step of supervised learning against human labels for quality. second, we introduce sentence semantic similarity (sss) , an additional component to be combined with bag-of-embeddings distance metrics such as bertscore. more specifically, we propose to explore semantic similarity at the sentence levelbased on sentence embeddings @xcite @xcite -and linearly combine it with existing metrics that use word embeddings. by doing so, the resulting metrics have access to word and compositional semantics, leading with improved performance. the combination is a simple weighted sum, and does not require training data. as a motivational example, consider the case in our main contributions are: ## related work various natural language generation tasks, including machine translation, image captioning, among others, produce sentences as output. these are evaluated either manually or automatically by comparison against one or multiple reference sentences. a multitude of metrics have been proposed for the latter, which perform comparisons at various granularity levels, from characters to words to embedding vectors. the goal of such metrics is to replace human judgements. in order to understand how well they fare at this task, metrics are evaluated by how similar their scores are to human assigned judgements on held-out datasets. for absolute quality judgements, pearson correlation is the most popularly used metric for such a comparison @xcite . there is also a bulk of work on metrics that take a step further to optimize their scores using machine learning algorithms trained on human scores for quality @xcite . they often perform even better, but the reliance on human scores for training, in addition to reference translations at inference time, makes them less applicable in practice. a separate strand of work that relies on contextual embeddings is that of quality estimation @xcite @xcite . these are also trained on human judgements of quality, but machine translations are compared directly to the source sentences rather than against reference translations. in addition to embeddings for words, embeddings for full sentences have been shown to work very well to measure semantic similarity. these are extracted using transformer models that are specifically trained for capturing sentence semantic meanings using bert, roberta, and xlm-roberta embeddings @xcite @xcite and provide state-of-art performance pretrained models for many languages. 1 in this paper, we take inspiration from these lines 1 https://github.com/ukplab/sentence-transformers of previous works to propose unsupervised metrics that combine word and sentence semantic similarity and show that this can be effective for both mtreference and source-mt comparisons. ## method in this section, we first describe in more detail the metrics that we have used in our experiments, namely semantic sentence cosine similarity, wmd and bertscore. then we present our simple approach to linearly combine these metrics. ## experiment setup in this section, we describe two types of experimental scenarios, monolingual and crosslingual evaluation, as well as the three datasets and pretrained embeddings we used. ## results the evaluation results are presented in this section. our code and data can be found on github 2 . ## conclusions in this paper, we propose to combine sentence-level and token-level evaluation metrics in an unsupervised way. in our experiments on a number of standard datasets, we demonstrate that this combination is more effective for mt evaluation than the current state-of-the-art unsupervised token-level metrics, substantially outperforming these as well as sentence-level semantic metrics on their own. the sentence level metric seems to capture higherlevel or compositional semantic similarity, which complements the token-level semantic similarity information. we also show that this combination approach can be applied both in the standard monolingual evaluation setting, where machine translations are compared to reference translations, and in a crosslingual evaluation setting, where reference translations are not available and machine translations are directly compared with the source sentences. in future work, we will aim to improve the crosslingual metric and explore other types of multilingual embeddings for better mapping across different languages.