Which is better? Exploring Prompting Strategy For LLM -based Metrics (2023)

Abstract: This paper describes the DSBA submissions to the Prompting Large Language Models as Explainable Metrics shared task, where systems were submitted to two tracks: small and large summarization tracks. With advanced Large Language Models (LLMs) such as GPT-4, evaluating the quality of Natural Language Generation (NLG) has become increasingly paramount. Traditional similarity-based metrics such as BLEU and ROUGE have shown to misalign with human evaluation and are ill-suited for open-ended generation tasks. To address this issue, we explore the potential capability of LLM-based metrics, especially leveraging open-source LLMs. In this study, wide range of prompts and prompting techniques are systematically analyzed with three approaches: prompting strategy, score aggregation, and explainability. Our research focuses on formulating effective prompt templates, determining the granularity of NLG quality scores and assessing the impact of in-context examples on LLM-based evaluation. Furthermore, three aggregation strategies are compared to identify the most reliable method for aggregating NLG quality scores. To examine explainability, we devise a strategy that generates rationales for the scores and analyzes the characteristics of the explanation produced by the open-source LLMs. Extensive experiments provide insights regarding evaluation capabilities of open-source LLMs and suggest effective prompting strategies.

## introduction as large language models (llms) like gpt-4 continue to advance rapidly, the natural language generation (nlg) capability is approaching a level of expertise comparable to that of a human. as a result, the precise evaluation of nlg has become increasingly paramount. however, traditional similarity-based metrics like bleu @xcite and rouge @xcite , which are widely used in nlg evaluations, tend to show a discrepancy from human assessments @xcite . additionally, the reliance on reference texts for these metrics can hinder an accurate assessment of nlg quality, particularly for open-ended generation tasks. recent research has introduced methodologies that leverage llms as nlg evaluators, showcasing the potential of llm-based metrics. these approaches are motivated from findings in recent research which revealed that llm can directly evaluate nlg capabiltiy harnessing knowledge retained during the pre-train @xcite . these metrics have demonstrated notable correlation @xcite @xcite with human evaluations to learned evaluators @xcite . concurrently, recent advancement of llms such as llama @xcite , vicuna @xcite , and orca @xcite , has paved a way for research on nlg evaluations utilizing open-source llms @xcite . however, there are few comprehensive studies that systematically evaluate the vast amount of possible prompts and prompting techniques for llm-based metrics. especially, research assessing the capabilities of open-source llms in the context of llm-based metrics is even more scarce. given the importance of enhancing the reproducibility of llm-based metrics in metric research, there is a clear need for studies that explore effective prompts and prompting techniques specifically for open-source llms @xcite . in this work, we conduct a thorough exploration of various prompts and prompting techniques for effective deployment of open-source llms as metrics: analyze them in terms of prompting strategy, score aggregation, and explainability. within the scope of prompting strategies, we compare the effectiveness of human and model instruction templates for nlg evaluation. in addition, we explore granularity in score assignment to accurately evaluate nlg quality. additionally, we gauge the influence of the open-source llm's in-context learning (icl) capability @xcite in nlg evaluation by employing various types of demonstrated examples. for score aggregation, we compare three methodologies to discern the optimal strategy for aggregating nlg quality scores. to infer the explainability of opensource llms, we generate rationale when computing scores. these comprehensive experiments on prompting techniques for llm-based metrics provide insights into the evaluation capabilities of open-source llms and guidelines for effective prompting strategies. furthermore, we provide insights derived from analysis of the features embedded in prompts and behaviors of open-source llms as llm-based metrics. additionally, we report our strategies and outcomes applied to the test set of summarization track in eval4nlp 2023 shared task. ## summarization track the summarization track of eval4nlp 2023 shared task @xcite aims to propose a reference-free metric for summarization. specifically, reference-free metric evaluates a given summary using only the provided source sentence or paragraph without additional human-written references. the objective of shared task is to develop llm-based metrics by exploring effective prompting strategies for open-source llms. ## method in this section, we address the prompting strategies and score aggregation methods, as well as approaches to assess the explainability of open-source llms. ## conclusion in this work, we conduct a systematic analysis of effective prompting techniques and strategies for llm-based metrics in nlg evaluation. our comprehensive experiments reveal that providing clear and straightforward instructions, akin to those explained to humans, proves to be more effective. furthermore, we examine various score aggregation methods to achieve effective score assignments and show the potential for enhancing explainability within open-source llms. additionally, we explore performance change relative to model size and scrutinize the influence of various elements within the prompt template. we hope that our research findings will furnish valuable insights for 8 a experimental setup library version guidance 0.0.64 vllm 0.1.7 torch 2.0.1 for optimized inference with open-source llms, we employ guidance and vllm libraries. the libraries and their respective versions used for the experiments can be found in