Towards Large Language Model driven Reference-less Translation Evaluation for E nglish and I ndian Language (2023)

Abstract: With the primary focus on evaluating the effectiveness of large language models for automatic reference-less translation assessment, this work presents our experiments on mimicking human direct assessment to evaluate the quality of translations in English and Indian languages. We constructed a translation evaluation task where we performed zero-shot learning, in-context example-driven learning, and fine-tuning of large language models to provide a score out of 100, where 100 represents a perfect translation and 1 represents a poor translation. We compared the performance of our trained systems with existing methods such as COMET, BERT-Scorer, and LABSE, and found that the LLM-based evaluator (LLaMA2-13B) achieves a comparable or higher overall correlation with human judgments for the considered Indian language pairs (Refer figure 1).

## introduction the field of natural language processing (nlp) and artificial intelligence (ai) has been transformed by the rapid advancements in large language models (llms), as they have demonstrated their capabilities in a wide range of natural language processing tasks, including open/close question answering, summarization @xcite , code completion, and code debugging @xcite @xcite , etc. these models have significantly impacted nlp applications by enhancing various aspects of language understanding, generation, and analysis @xcite . as llms continue to evolve and improve, they hold immense potential for further advancements in nlp, paving the way for more sophisticated and intelligent solutions @xcite . translation, on the other hand, plays a vital role in bridging the gap between different languages, enabling effective communication. with the advancement of machine translation systems @xcite , there has been a significant shift in how translations are produced @xcite . translation engines have made great progress in understanding languages and generating translations, but they still face numerous challenges in accurately conveying the intended meaning @xcite @xcite . while these systems have made remarkable progress, their evaluation remains an indispensable component in ensuring the accuracy and quality of translations produced by them @xcite . to date, human involvement has been considered one of the most reliable and effective ways to evaluate translations @xcite @xcite . however, this requires a significant investment of time, expertise, and financial resources. the cost and effort involved can limit the frequency and scale at which human involved evaluation can be conducted. therefore, it is crucial to develop a scalable, replicable, and efficient automatic method that mimics human evaluation to ensure reliable and effective translation assessment. recent developments in the field of automatic translation evaluation have demonstrated that techniques utilizing multilingual embeddings have a tendency to outperform other traditional approaches and display the strongest correlation with human assessments @xcite . notable instances of these techniques include bertscore @xcite and comet @xcite @xcite . to further improve the effectiveness of automatic translation evaluation, it is reasonable to investigate methodologies that leverage large language models, considering their notable capability for comprehension. in this work, we aim to assess the capability of llms and utilize them for reference-less translation evaluation involving english and indian languages. the research questions that we pose are as follows: in this study, our objective is to gain a clearer understanding of the reference-less translation evaluation capabilities of several popular large language models. specifically, we focus on opt @xcite , bloom @xcite , llama-1 foot_0 , mpt foot_1 , falcon @xcite , llama-2 @xcite for translating english to five indian languages: hindi, gujarati, marathi, tamil, and telugu. reference-less translation evaluation generally involves assessing the translation of a given source language sentence without a specific reference translation. in this evaluation, an automatic reference-less evaluation system assigns a score to the translation, which is similar to the direct assessment (da) score @xcite , commonly used in human translation evaluation. this score ranges from 1 to 100, with 1 representing a poor translation and 100 representing a perfect translation. in our work, we initially evaluate the reference-less translation evaluation capabilities of the raw large language models mentioned above in both zero-shot @xcite and in-context learning foot_2 @xcite scenarios. subsequently, we employ the lora parameter-efficient fine-tuning technique @xcite , as well as full fine-tuning, to refine the selected base llm models. additionally, considering that llms are known for their multi-task learning abilities @xcite , we explore the fine-tuning of llms on translation corpora alongside translation evaluation corpora to determine if translation-based fine-tuning improves overall translation evaluation performance. the key findings of our study, presented in figure 1 , highlight the performance of our fine-tuned llm-based reference-less translation evaluation models compared to various wellknown reference-less evaluation methods, such as comet, labse, and bert-scorer, based on their spearman correlation with human judgments. our findings emphasize the significant potential of large language models for reference-less translation evaluation tasks involving english and indian languages. raw llms do not inherently possess the capabilities for translation evaluation as they do not provide a score as an evaluation outcome. however, our multi-lingual llm based lora-fine-tuned models (llama-2-7b, llama-2-13b and mistral-7b) demonstrate competitive or superior correlation with human judgments compared to existing reference-less methods like comet under same training and evaluation configurations. we did not observe any additional benefits when we perform translation evaluation fine-tuning with translation fine-tuning under the multi-task setting. the results suggest that fine-tuned llms hold promise for translation evaluation in the targeted reference-less translation evaluation task. our study represents an essential and pioneering milestone in assessing and enhancing the reference-less translation evaluation capabilities of llms, involving english and indian languages. ## related work there are two main categories of automatic machine translation (mt) metrics: string-based metrics and pretrained models based metrics. in the following sub-section, we discuss them briefly one by one. ## large language models language modeling, a well-established task in the field of natural language processing, has garnered significant attention over the years @xcite . this task involves predicting the probability of the next token in a sequence of words. transformers have emerged as the fundamental architecture underlying many existing large language models @xcite . transformers based auto-regressive models like gpt @xcite @xcite have played a crucial role in advancing natural language processing (nlp). in this work, we used following base llm models to check how we can utilise them for machine translation evaluation involving english and indian languages. opt-6.7b foot_3 : the opt-6.7b @xcite model has been extensively trained on the objective of causal language modeling (clm) using english text with 6.7 billion parameters. bloom-7b 5 : bloom (scao et al., 2022) was the first largest multilingual large language model with causal language modeling objective and supports 46 languages and 13 programming languages. it has 7,069,016,064 parameters. llama-7b 6 : llama is a collection of foundation language models ranging from 7b to 65b parameters. in our experiments we evaluated llama model with 7b parameters where 4096 is the em-bedding dimensions and 32 layers and 32 attention head. mpt-7b foot_6 : similar to above models mpt-7b model is trained on a large amount of data 1t tokens on causal language modeling objective. falcon foot_7 : falcon @xcite is another large language model trained on causal language modeling (clm) objective. here, we utilised falcon-7b model which is a 7b parameters for our experiments. llama-2-7b foot_8 and llama-2-13b foot_9 : llama 2 based models @xcite are also trained on causal language modeling (clm) objective and pretrained on 2 trillion tokens of data from publicly available sources. in our experiments we have experimented with 7b and 13b llama-2 models. llama-2-7b network has 32 layers and 32 attention heads while llama-2-13b has 40 layers and 40 attention heads. mistral-7b foot_10 : mistral-7b large language model (llm) @xcite is a pre-trained on causal language modeling (clm) objective with 7 billion parameters. it uses sliding window attention (swa) to handle longer sequences at smaller cost and grouped-query attention (gqa) for faster inference which reduces the memory requirement during decoding. it has 4096 embedding dimension, 32 layers and 32 attention heads with context length of 8192. ## reference-less translation evaluation on raw llm to evaluate the effectiveness of the llms mentioned above for translation evaluation tasks without reference, we conducted two different experiments. the first involved assessing the performance of the pre-trained (raw) llm for reference-less translation evaluation. in the second experiment, we performed example-based in-context learning for the same purpose. both experiments were carried out using translation evaluation data mentioned in the section 5.1. as part of our experimental setup, we configured a prompting pipeline depicted in figure 2 . this pipeline involved using a prompt generator to generate specific prompts for the source and target language pairs for source and translation text. subsequently, an llm call is triggered to generate a response, which was then processed by a reply parser to obtain the actual translation. to ensure high-throughput and memory-efficient inference and serving for llms, we utilized the vllm library 12 (kwon et al., 2023). we conducted all experiments using a temperature parameter of 0, which ensures that the model behaves deterministically. by setting the temperature to 0, the model is constrained to select the word with the highest probability, effectively limiting its choice to the most likely option @xcite . all of our experiments are conducted using this library, and the models are deployed on a100, 40gb gpus. ## results and discussion in order to determine the performance of the llm models, we examined the correlation between human judgments and metric output scores. this correlation served as the primary evaluation factor for identifying the most effective fine-tuned model. to conduct this analysis on the aforementioned development corpora, we utilized spearman's rank correlation coefficient, pearson correlation coefficient, and kendall's rank correlation coefficient. to calculate these correlations, we employed the scipy library 20 . 6.1 zero shot vs icl based reference-less translation evaluation over raw llms overall, the correlation scores with human evaluation for both raw llms and in context learning (icl) based llms in reference-less translation evaluation are substandard. all llms exhibited poor performance, which could be attributed to a lack of understanding or knowledge of the translation evaluation task. in our manual analysis, we observed that llms consistently provide a common number close to 100, regardless of the transla-english to ins average across languages co-relation scores s p k comet-qe 0.4568 0.5236 0.32042 llama-2-7b-adpt 0.45322 0.52096 0.3371 llama-2-13b-adpt 0.4574 0.53744 0.3437 mistral-7b-adpt 0.44538 0.50744 0.33622 llama-2-7b-adpt-full 0.43678 0.4945 0.33852 llama-2-13b-adpt-full 0.36608 0.32434 0.28216 llama-2-7b-adpt-trans 0.39814 0.4643 0.2972 llama-2-13b-adpt-trans 0.4429 0.50322 0.33116 mistral-7b-adpt-trans 0.41762 0.47528 0.31674 for in-context learning, llms simply mimic the example translation evaluation scores. therefore, in response to our initial question, it can be concluded that raw llms do not possess inherent translation evaluation capabilities, either with zero-shot or example-driven contextual learning. refer to the appendix for examples and correlation details. ## limitations we conducted all our experiments using highperformance gpus, specifically the a100-40gb, which may not be readily available for everyone to reproduce these experiments and obtain the same results due to compute limitations. to address this constraint, our goal is to make all outputs, including model outputs and results, openly accessible foot_16 for further research. ## conclusion and future work our experiments and results indicate that fine-tuned llms show promise for translation evaluation in the targeted reference-less translation evaluation task. the findings call for further analysis and understanding of translation evaluation tasks involving large language models. as part of future work, we plan to incorporate additional indian languages for the reference-less translation evaluation task. furthermore, we aim to utilize large language models for reference-driven translation evaluation, encompassing english and a wider range of indian languages. overall, our study represents a significant milestone in assessing and enhancing the reference-less translation evaluation capabilities of llms, specifically in the context of english and indian languages. for the future of automatic translation evaluation, we strongly advocate for solutions based on large language models as the primary method of evaluation. we have also released our models and predictions for future research.