Semantically-Informed Regressive Encoder Score (2023)

Abstract: Machine translation is natural language generation (NLG) problem of translating source text from one language to another. As every task in machine learning domain it requires to have evaluation metric. The most obvious one is human evaluation but it is expensive in case of money and time consumption. In last years with appearing of pretrained transformer architectures and large language models (LLMs) state-of-the-art results in automatic machine translation evaluation got a huge quality step in terms of correlation with expert assessment. We introduce MRE-Score, seMantically-informed Regression Encoder Score, the approach with constructing automatic machine translation evaluation system based on regression encoder and contrastive pretraining for downstream problem.

## introduction wmt metrics shared task @xcite is a machine learning competition where participants have to construct an automatic evaluation system for machine translation for several language pairs. for wmt23 metrics shared task 1 , three language pairs are considered: english-german (ende), chinese-english (zh-en), and hebrew-english (he-en). for each source sentence, there is a corresponding target machine-translated text and a reference human translation. the main goals of this competition are: within the wmt23 metrics competition, our investigation focuses on the approach of constructing evaluation models to solve the regression problem based on expert degrees. specifically, we construct regression models using a pretrained transformer encoder from the mt0 model family @xcite , both vanilla models and with additional contrastive representation tuning. mt0 is finetuned version of mt5 @xcite , multilingual transformer model, which demonstrated capabilities of crosslingual generalization to unseen tasks and languages. similar approaches demonstrated the best results in wmt21 @xcite and wmt22 @xcite shared tasks. we release our code and pre-trained models openly to foster further research. 2 2 related work evaluation metrics in nlg evaluation, one can differentiate four types of approaches for model construction: (1) classical lexical overlap models, on the one hand, and llm-based approaches based on (2) unsupervised matching, (3) regression, and (4) zero-shot prompting, on the other hand. classical lexical overlap methods measure overlap between source, reference, and target sentence ngrams @xcite @xcite . modern unsupervised matchingbased approaches use large language model (llm)based encoders such as bert to compute the semantic similarity between reference and hypothesis texts @xcite or between source and hypothesis texts @xcite . in modern regression approaches, models are fine-tuned to predict human evaluation scores. generally, they consist of a transformer encoder model and a regression head. as input, they can use late binding with source, reference, and target texts or different concatenation combinations @xcite . llm-based zeroshot approaches use prompt engineering for llms with the expectation of a score in the generation output @xcite . in some research, attempts are made to predict evaluation scores as a weighted sum of digit tokens, where the weights are token probabilities from a markov chain probability model @xcite . the previous winner of the wmt metrics shared task competition was the proprietary met-ricx @xcite model, which used a regression approach. one of the state-of-the-art models in machine translation is gpt-4 with zeroshot scoring. however, due to the time consumption of its inference and the closeness of regression approaches with relatively small backbones (e.g., comet used the base version of xlm-roberta @xcite with 2.5b parameters), task-specific nlg evaluation with sophisticated tricks with vector representation and training datasets may provide better results. contrastive learning contrastive learning for nlp problems is a popular pretraining approach for improving results in downstream tasks. for example, the recent e5 model @xcite is pretrained in a contrastive manner using a curated large-scale text pair dataset to solve various tasks that require a single-vector representation of texts, both after finetuning and in a zero-or few-shot manner. another work that investigated contrastive learning for extrapolating vector representations for different tasks is instructor @xcite . this model incorporates instructions in contrastive learning and achieved good results for tasks that were unseen during pretraining. the idea of knowledge transfer in the latent space may provide improvements with clean datasets and an appropriate fitting process. ## results and discussion we trained and tested each configuration on our test data using the kendall-𝜏 correlation metric. the results in ## conclusion this paper presents our experiments with semantically informed architectures with a regression head. this led us to conclude that the additional awareness of the encoder and extra pretraining may positively affect the model quality in these conditions. in the future, it would be possible to explore other ways to inform the model and conduct experiments with larger versions of our implemented architectures.