Exploring Prompting Large Language Models as Explainable Metrics (2023)

Abstract: This paper describes the IUST NLP Lab submission to the Prompting Large Language Models as Explainable Metrics Shared Task at the Eval4NLP 2023 Workshop on Evaluation & Comparison of NLP Systems. We have proposed a zero-shot prompt-based strategy for explainable evaluation of the summarization task using Large Language Models (LLMs). The conducted experiments demonstrate the promising potential of LLMs as evaluation metrics in Natural Language Processing (NLP), particularly in the field of summarization. Both few-shot and zero-shot approaches are employed in these experiments. The performance of our best provided prompts achieved a Kendall correlation of 0.477 with human evaluations in the text summarization task on the test data.

## introduction summarization is crucial for quickly understanding large textual documents. the goal of text summarization is to condense lengthy documents into a concise, coherent, and easily understandable format while retaining the essential information from the original source. however, assessing the quality and performance of summarization systems has proven to be a challenging task. commonly used evaluation methods for summarization, such as rouge scores @xcite , have certain limitations. they fail to capture the overall quality, coherence, and interpretability of summaries. additionally, they rely on human-generated reference summaries, which are time-consuming and subjective. given the limitations of traditional evaluation approaches, it is important to explore alternative evaluation methods that leverage the capabilities of llms and offer explainable metrics. llms, such as @xcite and llama @xcite , have shown remarkable summarization capabilities. they can generate coherent and contextually grounded summaries. this makes them ideal for evaluation purposes. llms provide both interpretability and inherent summarization abilities. they can generate explanations and reasoning for their predictions, giving evaluators a deeper understanding of system strengths and weaknesses. moreover, llms reduce the dependency on goldstandard reference summaries. by using llms as evaluators, we can generate comparative summaries and objectively assess system-generated summaries. this eliminates potential biases from human references. in summary, using llms as explainable metrics in summarization evaluation offers several benefits. it overcomes the limitations of traditional methods, provides interpretability, and reduces reliance on human-generated references. this emerging field of research holds promise for a more comprehensive and objective assessment of summarization systems. the main contribution of this paper is the investigation of various prompt-based methods for explainable evaluation of summarization tasks. we explore both few-shot and zero-shot approaches in our experiments. the best performance prompt follows the zero-shot strategy and is introduced in the paper under the name p1. in this prompt, the criteria for assessing the quality of summaries are described (e.g., how well the main idea of the main document is captured in the summary). this prompt achieves a kendall correlation score of 0.477, outperforming other methods in comparison. our conducted experiments highlight the promising potential of llms as evaluation metrics in the field of nlp, with a specific focus on summarization. ## related work several recent studies have focused on utilizing llms for the evaluation of several different tasks in nlp (e.g., text generation, machine translation, and summarization). gptscore @xcite is a novel framework for evaluating generated texts using large pre-trained language models, particularly gpt-3. the framework leverages the emergent abilities of generative pre-trained models, such as zero-shot instruction, to score generated texts. gptscore operates under the assumption that a large pre-trained language model is more likely to assign higher probabilities to high-quality generated text when provided with adequate instruction and context. by leveraging the power of gpt-3, gptscore aims to assess the quality of generated text based on the model's generative capabilities. in a similar vein, @xcite conducted a preliminary survey on using chatgpt, a variant of the gpt model, as a natural language generation (nlg) evaluator. the study explores the potential of chatgpt in evaluating the quality of generated text in various nlg tasks. in the context of translation quality assessment, gemba @xcite ) is introduced as a gpt-based metric that can effectively evaluate translations with or without a reference translation. the evaluation focuses on zero-shot prompting and involves comparing four prompt variants in two modes, depending on the availability of a reference. results from the evaluation demonstrate that gemba achieves state-of-the-art accuracy when compared to mqm-based human labels, as evidenced by the wmt22 metrics shared task. instructscore @xcite is an open-source and explainable evaluation metric for text generation. this model fine-tunes the llama model to predict a fine-grained error diagnosis of machine translated content. this work presents a novel framework for explainable text generation evaluation, addressing the limitations associated with black-box metrics and showcasing the potential of llms to provide meaningful and interpretable evaluations. g-eval @xcite , is a framework that utilizes llms with chain-of-thoughts (cot) and a form-filling paradigm to assess the quality of nlg outputs, specifically in text summarization and dialogue generation tasks. the experiments demonstrate that g-eval, utilizing gpt-4 as the backbone model, achieves a high spearman correlation of 0.514 with human evaluations in the text summarization task, outperforming previous methods significantly. ## task description the topic of eval4nlp shared task @xcite is to provide explainable metrics for summarization and machine translation evaluation by prompting llms. the main goal is to investigate different prompting methods (e.g., zero-shot, fewshot, chain of thought, fine-grained, majority vote, self-refinement), therefore, fine-tuning the llms is not allowed. also, a number of llms are allowed to be used. the shared task has two tracks based on the model sizes (one for models bigger than 25b parameters, and one for smaller models). this work has been done on the summarization task and using small models. in the following, the dataset and the evaluation metric are explained. ## methodology in this research, we have used pre-trained orca_mini_v3_7b (tuned llama2-7b model) @xcite on the huggingface transformers foot_1 . we employed two strategies, zero-shot and few-shot, for constructing prompts. the zero-shot strategy included the combination of evaluation criteria for the quality of summarization in the form of questions or detailed explanations provided to the model. there are examples of summarization evaluation written in the few-shot strategy. in this way, the main document, the summarized document, and the scores received are mentioned precisely. to assess summarization quality via prompting an llm, the following parameters are needed: ## results and analysis we experiment with six different distinct prompt types. one of them is few-shot (p6) and the rest are zero-shot. the execution of each prompt takes approximately 1 hour. if we also include the explanation of the results in the output, each execution of the test data takes close to 4 hours. based on the kendall measure (which serves as the primary evaluation metric), the best result is associated with p1. this prompt follows the zero-shot strategy. in this prompt, some of the summeval criteria are also mentioned. additionally, p1 achieves the highest value in the spearman measure and serves as the final strategy for the test phase. the results of p2, p3, and p4 are very close to each other. the reason for the difference observed prompt p1: score the effectiveness of the summarization by comparing the key points and overall coherence of the summarized with the main document. checked whether the summary captures the main ideas, maintains the intended tone and style, and provides a concise yet comprehensive overview of the main document. score the summarization with respect to the summarized document on a continuous scale from 0 to 100, where a score of zero means irrelevant, factually incorrect and good readable and score of 100 means relevant, factually correct, no readability summarized. also explain your process to get this score to summary. ## conclusion in this paper, we have investigated different prompts to define explainable evaluation metrics for summarization using llms. the experiments conducted indicate that llms have great potential as evaluation metrics in nlp tasks, especially summarization. in these experiments, both the few-shot and zero-shot approaches were used. our best prompt achieved a kendall correlation of 0.477 compared to the human score. in future work, other prompt strategies, such as the chain of thought, can also be explored. experiments can also be repeated with existing prompts and other language models and compare the results obtained to determine the effect of the language model on changing the quality of the output.