REUSE : RE ference-free U n S upervised Quality Estimation Metric (2022)

Abstract: This paper describes our submission to the WMT2022 shared metrics task. Our unsupervised metric estimates the translation quality at chunk-level and sentence-level. Source and target sentence chunks are retrieved by using a multi-lingual chunker. The chunk-level similarity is computed by leveraging BERT contextual word embeddings and sentence similarity scores are calculated by leveraging sentence embeddings of Language-Agnostic BERT models. The final quality estimation score is obtained by mean pooling the chunk-level and sentence-level similarity scores. This paper outlines our experiments and also reports the correlation with human judgements for en-de, en-ru and zh-en language pairs of WMT17, WMT18 and WMT19 test sets.

## introduction quality estimation (qe) is an essential component of the machine translation workflow as it assesses the quality of the translated output without conferring reference translations @xcite . high quality reference translations are often hard to find, qe helps to evaluate the translation quality based on the source sentences. recently qe has emerged as an alternative evaluation approach for nmt systems @xcite . recently, many researchers have been working on qe, as a part of quality estimation shared task, several qe systems @xcite @xcite were evaluated in wmt conference @xcite . however, most of the quality estimation systems are supervised i.e., the model regresses on the human judgements. often, human assessments are not available and it is very difficult to procure high quality human judgements. this motivated our research to emerge with an unsupervised quality estimation system. also, qe is usually performed at different granularity (e.g., word, sentence, document) @xcite ; in this work, we focus on the chunk-level and sentence-level similarity. the final qe score of the target sentence is obtained by mean pooling the chunk similarity scores and sentence similarity scores. overall, our main contribution is as follows: ## reuse we propose reuse, a reference-free unsupervised quality estimation metric that evaluates a machine translated output based on the corresponding source sentence regardless of the reference. figure 2 depicts the high-level architecture of our model. the chunks of source and hypothesis are acquired from the multilingual chunking model. further chunk-wise subword contextual bert embeddings are mean-pooled to obtain the chunk-level embeddings. meanwhile, labse model @xcite is used for the sentence-level embeddings. using these embeddings, we compute chunk-level similarity and sentence-level similarity, finally combine them by averaging chunk-and sentence-level similarity scores 1 . we discuss the working details of our system in the following sections. 1 reuse score ranges between 0-1. ## conclusion in this paper, we describe our submission to the wmt22 metrics shared task (qe-as-a-metric). our submission includes segment-level and systemlevel quality estimation scores for sentences of three language pairs chinese-english (zh-en), english-russian (en-ru) and english-german (ende). we evaluate this year's test set using our unsupervised, reference-free metric -reuse, that provides a quality estimation score by evaluating a hypothesis against the source sentence. reuse estimates the translation quality by combining chunklevel similarity score and sentence-level similarity score, leveraging multilingual bert embeddings. we performed our experiments on testsets of wmt17, wmt18, wmt19 and it has been emperically observed that the combination of chunkand sentence-level similarity scores performed better in terms of agreement with human assessments. potential research directions definitely include improving the multilingual chunking model. as part of future work, we aim to further experiment and emerge with such effortless efficient unsupervised approach to estimate the translation quality and exhibit higher agreement with humans.