HIT - MI & T Lab’s Submission to E val4 NLP 2023 Shared Task (2023)

Abstract: Recently, Large Language Models (LLMs) have boosted the research in natural language processing and shown impressive capabilities across numerous domains, including machine translation evaluation. This paper presents our methods developed for the machine translation evaluation sub-task of the Eval4NLP 2023 Shared Task. Based on the provided LLMs, we propose a generation-based method as well as a probability-based method to perform evaluation, explore different strategies when selecting the demonstrations for in-context learning, and try different ensemble methods to further improve the evaluation accuracy. The experiment results on the development set and test set demonstrate the effectiveness of our proposed method.

## introduction as the output quality of the machine translation systems has been improved, the evaluation of translation outputs has become more challenging and critical. on one hand, human evaluations of these outputs are often time-consuming and laborious; on the other hand, previous automatic metrics such as bleu @xcite are becoming less reliable with little remaining correlation with human judgments @xcite . as a result, the demand for next generation of automatic evaluation is stronger than ever. large language models (llms), especially generative pre-trained transformer (gpt) models @xcite , have led to a revolution of research in natural language processing, including machine translation evaluation. metrics like gemba @xcite explore the prompting of gpt models like @xcite and gpt4 (openai, 2023) directly leveraged as metrics. error analysis prompting @xcite proposes to generate human-like mt evaluations with the help of llms by combining chain-of-thoughts @xcite and error analysis @xcite . besides, other work also uses llms to calculate the conditional probability of the generated text as the evaluation results @xcite . this paper describes our submission to the machine translation evaluation sub-task of the eval4nlp 2023 shared task @xcite . participants of this task are required to prompt the llms specified by the organizers as metrics for machine translation, without any fine-tuning on the selected llm. in our work, on the basis of four llms provided by the organizers, we propose a generation-based method that directs the llm to score the translated sentence directly by generation, and a probability-based method that calculate the conditional probability of the translated sentence. we also explore different demonstration selection strategies for in-context learning @xcite , including bucket-based selection and similarity-based selection. what's more, we try different ensemble methods, including averagingbased ensemble and multi-agent ensemble, to further improve the performance. experiments on the development and test set shows that we obtain competitive results in this year's shared task, verifying the effectiveness of our proposed methods. our contributions are summarized as follows: ## conclusion in this paper, we describe our submission to eval4nlp 2023 shared task. we propose to apply llm on translation evaluation by generation-based method and probability-based method, and explore different demonstration selection strategies for incontext learning. different emsemble methods are also used to obtain the final score. a surprising result of our work is the remarkable performance of probability-based method, which performs better while requiring less token as input. in the future, we would explore more information from the inside of llms as the evidence to evaluate text quality.