Unsupervised Embedding-based Metric for MT Evaluation with Improved Human Correlation (2022)

Abstract: In this paper, we describe our submission to the WMT22 metrics shared task. Our metric focuses on computing contextual and syntactic equivalences along with lexical, morphological, and semantic similarity. The intent is to capture the fluency and context of the MT outputs along with their adequacy. Fluency is captured using syntactic similarity and context is captured using sentence similarity leveraging sentence embeddings. The final sentence translation score is the weighted combination of three similarity scores: a) Syntactic Similarity b) Lexical, Morphological and Semantic Similarity, and c) Contextual Similarity. This paper outlines two improved versions of MEE i.e., MEE2 and MEE4. Additionally, we report our experiments on language pairs of en-de, en-ru and zh-en from WMT17-19 testset and further depict the correlation with human assessments.

## introduction neural machine translation (nmt) systems have emerged with an increased research interest in recent times and significantly enhanced the mt quality. however, the mt research community still relies mainly on antiquated metrics and no new, universally adopted standard metric has emerged. in the last few years, research in machine translation (mt) evaluation has made significant progress. a metrics-shared task is held annually at the wmt conference, where new evaluation metrics are proposed and those which correlates highly with human judgements are presented from the pool of newly defined metrics. neural-based metrics largely dominated the last two years of the wmt metrics task @xcite @xcite . nevertheless, n-gram based or lexical-based metrics remain popular as automatic mt evaluation metric due to their ag-ile and light-weighted nature. traditionally, automatic metrics for evaluating mt quality have relied on estimating the similarity between machine outputs and reference sentences in the target language. however, advanced nmt methods yield high-quality translations that might have lexical, morphological, syntactic variations and different word choices having similar meanings. typically, the machine output diverges from monotonic lexical transfer between the source and target languages. widely used evaluation metrics rely on basic, lexical-level features as they calculate the surface similarity between the hypothesis and reference sentences by counting the number of matching n-grams @xcite . metrics relying on n-gram overlap cannot appropriately capture morphological, syntactic and semantic variations as they are sensitive to only lexical variations. meteor @xcite @xcite @xcite captures semantic variations but it is highly dependent on language specific tools . hence, there is huge requirement for a robust, understandable, easy to use automatic mt evaluation metric which captures all the linguistic features to evaluate like humans. the better evaluation metric will be highly helpful to the development of better mt systems @xcite . in this paper, we present our submission to the wmt2022 metrics shared task. we evaluate the translations of english-german (en-de), english-russian (en-ru) and chinese-english (zh-en) language pairs. however, the proposed metric is language independent and supports 100+ languages. here, our submission includes scores of three metrics mee @xcite , mee2 and mee4 (mee2 and mee4 are extended versions of mee). we have evaluated the testsets of wmt17 @xcite , wmt18 @xcite and wmt19 @xcite , for the same language pairs (en-de, en-ru and zh-en) and reported the correlation with human assessments. the empirical results conclude that mee4 shows better agreement with humans. ## conclusion and future work in this paper, we present our participation to the wmt22 metrics shared task. our submission includes segment-level and system-level scores for sentences of three language pairs chinese-english (zh-en), english-russian (en-ru) and english-german (en-de). we evaluate this year's test set using our unsupervised, reference-based metrics: mee2 and mee4. both the metrics are extended versions of mee with improved correlation. from the last year's findings, it was evident that mee2 was one among the better performing metrics as it was highlighted in the top significant cluster @xcite . however, this year we present mee4 along with mee2 and mee4 has proved to perform better in terms of correlation with humans when evaluated on testsets of wmt17, wmt18 and wmt19. we observe that this improvement in agreement to human experts level judgements is due to assigning more weightage to context information (sentence level semantics) when compared language pair #sentences #systems en-ru 33988 82 en-de 97002 164 zh-en 73668 192