LTRC _ IIITH ’s 2023 Submission for Prompting Large Language Models as Explainable Metrics Task (2023)

Abstract: In this report, we share our contribution to the Eval4NLP Shared Task titled “Prompting Large Language Models as Explainable Metrics.” We build our prompts with a primary focus on effective prompting strategies, score-aggregation, and explainability for LLM-based metrics. We participated in the track for smaller models by submitting the scores along with their explanations. According to the Kendall correlation scores on the leaderboard, our MT evaluation submission ranks second-best, while our summarization evaluation submission ranks fourth, with only a 0.06 difference from the leading submission.

## introduction with groundbreaking advancements in unsupervised learning and scalable architectures, the possibilities and associated risks, of automatically generating audio, images, videos, and text have become incredibly daunting. conducting human evaluations of such content is not only costly but often logistically challenging. consequently, there is a pressing need for automatic metrics that can reliably assess the quality of generation systems and their outputs. presently, the state-of-the-art metrics for evaluating natural language generation (nlg) systems still fall short of replicating the proficiency of human experts. these metrics primarily rely on neural language models and typically yield a single quality score at the sentence level. this singular score makes it arduous to explain their internal decision-making processes and their resulting assessments @xcite . the introduction of apis for large language models (llms), such as chatgpt, and the recent opensource availability of llms like llama have ig-nited a surge in nlp research, including the development of llm-based metrics @xcite . noteworthy examples include gemba @xcite , which delves into using prompts with chatgpt (openai, 2023a) and gpt4 (openai, 2023b) directly as metrics, and instructscore @xcite , which takes a different approach by fine-tuning a llama model to provide a detailed error diagnosis of machinetranslated content. it is important to note that current research lacks systematic evaluation of potential prompts and prompting techniques for metric usage. this includes approaches that involve instructing a model or having the model explain a task on its own. additionally, there is a scarcity of assessments regarding the performance of recent open-source llms, despite their critical role in enhancing the reproducibility of metric research compared to closedsource alternatives. this year's eval4nlp shared task (leiter et al., 2023b) addresses these gaps, providing opensource, pre-trained llms ( the shared task aims to achieve the following objectives: our submission aligns with these objectives. we attain these goals by utilizing the orca_mini_v3_7b (mathur, 2023) model and crafting prompts through a combination of fine-grained and chain-of-thought prompting strategies. additionally, we have adapted 4-bit quantization to optimize model loading. we submit reference-free a) segment-level quality scores for all the language pairs (en-de, en-zh, enes) listed under the mt evaluation task and b) summary-level quality scores for all the documents provided. ## system description we opted for orca_mini_v3_7b among the provided llms due to its smaller size, which accommodated our resource constraints. we encountered challenges when attempting to load other llms. we curated prompts using a blend of fine-grained and chain-of-thought prompting strategies. furthermore, using bitsandbytes foot_0 we employed 4-bit quantization to enhance model loading efficiency and considered max tokens as 512 during inference (refer appendix 7 for computation details). our submission includes: a) summary-level quality scores for all the documents provided in the task. b) segment-level quality scores for language pairs (en-de, en-zh, en-es) in the mt evaluation task, without relying on references. the summary-level scores and segment-level scores lies in the range of 0-100, where 0 is the least score that can be awarded to a bad translation/summary and 100 is the highest score that can be assigned to a perfect translation/summary. ## results & analysis an interesting point to observe is that our submissions have surpassed most of the submissions made in the large model track except nllg for en-de and en-es, and mysterytest for en-es.