MEE 4 and XL sim : IIIT HYD ’s Submissions’ for WMT 23 Metrics Shared Task (2023)

Abstract: This paper presents our contributions to the WMT2023 shared metrics task, consisting of two distinct evaluation approaches: a) Unsupervised Metric (MEE4) and b) Supervised Metric (XLSim). MEE4 represents an unsupervised, reference-based assessment metric that quantifies linguistic features, encompassing lexical, syntactic, semantic, morphological, and contextual similarities, leveraging embeddings. In contrast, XLsim is a supervised reference-based evaluation metric, employing a Siamese Architecture, which regresses on Direct Assessments (DA) from previous WMT News Translation shared tasks from 2017-2022. XLsim is trained using XLM-RoBERTa (base) on English-German reference and mt pairs with human scores.

## introduction in recent times, there has been a growing interest in neural machine translation (nmt) systems, leading to significant improvements in machine translation (mt) quality. over the past few years, the field of mt evaluation has seen substantial advancements. each year, the wmt conference hosts a metrics-shared task, where new evaluation metrics are introduced and those demonstrating a strong correlation with human judgments are highlighted from the array of newly devised metrics. in the last three years of the wmt metrics task @xcite @xcite , neuralbased metrics have predominantly taken the lead. nevertheless, n-gram-based and lexical-based metrics @xcite continue to be favored as automatic mt evaluation tools due to their flexibility and efficiency. as a result, this year we participated in the metrics shared task, evaluating machine translation out-puts using two types of metrics: an unsupervised metric and a supervised metric. unsupervised metric: our unsupervised metric, mee4 @xcite , relies on a combination of lexical and embedding similarity measures. notably, mee4 demonstrated strong performance in the previous year's shared task @xcite , surpassing several baseline metrics such as bertscore @xcite , bleu @xcite , f101spbleu @xcite , and chrf @xcite . in our efforts to improve its performance further this year, we conducted experiments with two different sentence embedding models: labse @xcite and the stsb-xlmr-multilingual foot_2 . interestingly, our findings indicated that mee4, when equipped with labse as the sentence embedding model, exhibited superior performance compared to the alternatives. supervised metric: unlike the existing neural models which are huge in size, our goal was to build a more compact supervised training model (xlsim) that offers improved performance. to achieve this, we created a sentencetransformer model by combining a pre-trained transformer model with a pooling layer. this hybrid approach enables the generation of sentence embeddings, which can be compared using cosine similarity to assess similarity between sentences. ## mee4 mee4 is an improved version of mee focusing on computing contextual and syntactic equivalences, along with lexical, morphological, and semantic similarity. the goal is to comprehensively evaluate the fluency and adequacy of mt outputs while also considering the surrounding context. fluency is determined by analyzing syntactic correlations, while context is evaluated by comparing sentence similarities using sentence embeddings. the ultimate score is derived from a weighted amalgamation of three distinct similarity measures: a) syntactic similarity, which is established using a modified bleu score. b) lexical, morphological, and semantic similarity, quantified through explicit unigram matching. c) contextual similarity, gauged by sentence similarity scores obtained from the language-agnostic bert model @xcite . in our experiments this year, we made adjustments to mee4 while maintaining the same underlying architecture. specifically, we computed the evaluation scores using a different sentence embedding model. in addition to our previous choice, we utilized the stsb-xlm-r-multilingual model. this particular sentence-transformers model is designed to map sentences and paragraphs into a 768-dimensional dense vector space, making it suitable for various tasks such as clustering and semantic search. it's worth highlighting that the version of xlm-r (conneau et al., 2020) we employed is considered a state-of-the-art model for multilingual semantic textual similarity (sts) @xcite . ## xlsim: mt evaluation metric based on siamese architecture xlsim is a supervised reference-based metric that regresses on human scores provided by @xcite . using a cross-lingual language model xlm-roberta-base foot_3 @xcite , we train a supervised model using a siamese network architecture with cosinesimilarityloss. ## conclusion and future work in this paper, we describe our submissions to the wmt23 metrics shared task. our submission in-cludes segment-level and system-level translation evaluation scores for sentences of three language pairs english-german (en-de), chinese-english (zh-en) and hebrew to english (he-en). we evaluate this year's test set using: a)two unsupervised metrics, mee4 and mee4_stsb_xlm. these metrics are based on lexical and embedding similarity match that evaluates the translation on various linguistic features (syntax,lexical, morphology, semantics and context) ; b) a supervised metric, xlsim that learns on en-de wmt da human evaluation data from 2017-2022. it is observed that all the three metrics displayed a positive correlation (>0.5) with the baseline metric comet. certainly, there are promising research directions to explore, especially in the realm of metric enhancement. in our future work, we intend to delve deeper into these areas: mee4 metric improvement: one of our primary objectives is to refine and enhance mee4, seeking more efficient approaches that can better estimate translation quality while achieving higher agreement with human judgments. this might involve exploring novel techniques in sentence embedding, fine-tuning, or leveraging additional linguistic information. xlsim enhancement: for xlsim, we plan to boost its performance by optimizing the training data. this involves ensuring that it is trained on a more diverse set of languages and data to improve its cross-lingual capabilities. simultaneously, we aim to maintain its compactness and ensure it remains trainable with fewer computational requirements. these future research directions hold the potential to contribute significantly to the field of machine translation evaluation, ultimately leading to more robust and accurate metrics that align closely with human assessments.