Little Giants: Exploring the Potential of Small LLM s as Evaluation Metrics in Summarization in the E val4 NLP 2023 Shared Task (2023)

Abstract: This paper describes and analyzes our participation in the 2023 Eval4NLP shared task, which focuses on assessing the effectiveness of prompt-based techniques to empower Large Language Models to handle the task of quality estimation, particularly in the context of evaluating machine translations and summaries. We conducted systematic experiments with various prompting techniques, including standard prompting, prompts informed by annotator instructions, and innovative chain-of-thought prompting. In addition, we integrated these approaches with zero-shot and one-shot learning methods to maximize the efficacy of our evaluation procedures. Our work reveals that combining these approaches using a “small”, open source model (orca_mini_v3_7B) yields competitive results.

## introduction large language models (llms) have revolutionized the field of natural language processing (nlp) by demonstrating remarkable proficiency in a multitude of generative tasks @xcite . beyond their capabilities in text generation, llms offer the potential to automate the evaluation of generated text, particularly in domains such as machine translation and summarization. previous research efforts have explored llmbased evaluation metrics, yielding promising results. notable examples include the development of metrics like the gemba metric for translation quality assessment @xcite , work on the effectiveness of llms as an alternative to human evaluation for nlp tasks by @xcite , and the instructscore metric for summarization evaluation @xcite . however, a significant gap exists in the systematic evaluation and exploration of prompting techniques available for metric usage with llms. in fact, there is scant work in this area to date. excep-tions include the work of @xcite for dialogue evaluation, @xcite and gemba for mt evaluation, g-eval, a chain-ofthought based framework for the evaluation of generated texts that leverages gpt-4 @xcite , and gptscore for text generation evaluation @xcite . this paper presents our contribution to addressing this gap through our participation in the prompting large language models as explainable metrics shared task @xcite , which was conducted as part of the 4th workshop on evaluation & comparison of nlp systems, hosted at aacl 2023. we delve into various prompting approaches and techniques, offer a comprehensive overview of the results we have obtained, and provide a thorough analysis of our findings (our team name is competitionentrants). we exclusively participated in the small models track, focusing on models with parameters less than or equal to 25 billion, for the summarization task. consequently, all our experiments and reported results are derived from the orca_mini_v3_7b model. among our various approaches, the best-performing one, employing a standard prompt in a zero-shot setting, achieved a score of 0.44 during the test phase of the shared task. while this performance is notable, it places us marginally behind the leaderboard's highest score of 0.50 by a margin of 0.06. we also find that chain-of-thought (cot) prompting @xcite also aids in explicating the evaluation task to the model. this technique holds the potential to enhance the interpretability and explainability of quality estimation models. ## task description the primary objective of the shared task is to investigate prompt-based methodologies for llms in the development of automated quality metrics in a reference-free setup tailored to natural language generation tasks, specifically summarization and machine translation. these quality scores are constructed using fine-grained scoring or error labels. the underlying rationale behind reference-free evaluation metrics is to provide assessment scores that are comparable to those of reference-based metrics while reducing reliance on often noisy and costly reference labels. a typical reference-free metric operates by taking a source (src) and a hypothesis (hyp) as inputs and subsequently generates a metric score, thereby providing an alternative to traditional reference-based evaluation methodologies. ## approaches three main classes of strategies are employed to enhance prompt effectiveness and interpretability for evaluating generated summaries. the first strategy, core prompts, encompasses three one-step methods for generating prompts. the first two borrow from existing literature, and the final uses an llm to simply generate a prompt from scratch. next, in section 3.2, we introduce three methods (one manual, two automatic) to take prompts in section 3.1 and further refine them. this is akin to paraphrasing in bulk. finally, in section 3.3, we outline two simple approaches for further refining prompts generated earlier. in total, the three classes span 8 different approaches, and approaches can be combined across the classes. ## results among the approaches, prompt p1, which employs a standard manual prompt in a zero-shot setting with a grading scale ranging from 1 to 5, emerges as the top performer, achieving a notable score of 0.3211 on the development dataset. this was surprising as this is essentially one of the most straightforward approaches to try. however, it is perhaps unsurprising as the direct assessment baseline provided by the shared task is also a simple manual prompt in a zero-shot setting @xcite . following closely behind, we find approaches that leverage annotator instructions as seed prompts (p12) and prompts generated using llms (p14 and p10), all in zero-shot settings, also stand out. these prompts, in contrast to standard ones, contain a wealth of details about the evaluation metrics, offering intricate guidance to the model. however, our exploration of a one-shot setting indicates that this approach does not yield as promising results (as much as 0.1 behind our best approach combination). further experimentation with the choice of examples provided to the model may be warranted to enhance its performance. notably, the incorporation of chain-of-thought prompting appears to be a beneficial strategy, as evidenced by the strong performance observed (our third, fourth, and fifth-best experiments). on the lower end of the result scores, we find standard prompts (p5, p4, p7, p3, p8) that utilize grading scales with exceptionally high precision or qualitative labels as quality measures. these prompts, while designed with attention to detail, exhibit comparatively lower scores in the evaluation, suggesting the importance of striking a balance between precision and other factors when designing prompts for quality estimation tasks. prompt p6 achieves the highest kendall correlation coefficient on the test set with a score of 0.4423. furthermore, p1 and p2 also achieve competitive scores of 0.4419 and 0.4422 respectively. ## discussion the evaluation results we have presented furnish compelling evidence regarding the proficiency of llms in the domain of quality estimation for summarization. our findings underscore the capacity of these models to provide valuable insights into the quality of generated outputs, thereby contributing to the advancement of evaluation methodologies within the field of nlp. ## conclusion in this paper, we have presented our system entry for the eval4nlp 2023 shared task on prompting large language models as explainable metrics. our approach revolves around assessing the capabilities of the small orca_mini_v3_7b model in the reference-free evaluation of generated summaries. through a series of extensive experiments, we have explored the model's adaptability and responsiveness to various prompting techniques, including zero-shot, one-shot, and chain-of-thought prompts. these experiments were conducted using meticulously designed prompts that emphasize key evaluation criteria. our findings suggest that the use of a standard prompting (when using a small model) is performant while the chain-of-thought prompt-based approach holds significant promise for evaluating the quality of generated summaries. however, we do note that slight variations in the underlying prompt text could swing performances rather widely, so more research is definitely required to understand this phenomenon.