One Prompt To Rule Them All: LLM s for Opinion Summary Evaluation (2024)

Abstract: Evaluation of opinion summaries using conventional reference-based metrics often fails to provide a comprehensive assessment and exhibits limited correlation with human judgments. While Large Language Models (LLMs) have shown promise as reference-free metrics for NLG evaluation, their potential remains unexplored for opinion summary evaluation. Furthermore, the absence of sufficient opinion summary evaluation datasets hinders progress in this area. In response, we introduce the SUMMEVAL-OP dataset, encompassing 7 dimensions crucial to the evaluation of opinion summaries: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity. We propose OP-I-PROMPT, a dimension-independent prompt, along with OP-PROMPTS, a dimension-dependent set of prompts for opinion summary evaluation. Our experiments demonstrate that OP-I-PROMPT emerges as a good alternative for evaluating opinion summaries, achieving an average Spearman correlation of 0.70 with human judgments, surpassing prior methodologies. Remarkably, we are the first to explore the efficacy of LLMs as evaluators, both on closed-source and open-source models, in the opinion summary evaluation domain.

## introduction opinion summarization systems predominantly use traditional metrics such as rouge @xcite and bertscore @xcite for automatic evaluation, however, they have been shown to have poor correlations with human judgments @xcite . moreover, these metrics fall short of comprehensively evaluating opinion summaries. additionally, obtaining reference-based datasets at a large scale is an expensive process. recently, large language models (llms) have been utilized as reference-free evaluators for natural language generation (nlg) outputs @xcite @xcite . the idea is to prompt a powerful llm such as chatgpt-3.5/gpt-4 to evaluate an output on certain criteria. however, their suitability has not been explored at all for evaluating opinion summaries. moreover, these approaches have been tested only on closed-source models (chatgpt-3.5/gpt-4) primarily because of the limitations of the open-source models in following instructions and producing the desired output @xcite . ## methodology we describe our dimension independent and dependent prompts and the model scoring function. ## summeval-op benchmark dataset we created the summeval-op benchmark dataset for evaluating the opinion summaries on 7 dimensions. in this section, we discuss the dataset used, opinion summary evaluation metrics, annotation details, and its analysis. ## experiments we discuss the available benchmark dataset for opinion summary evaluation, the summarization models used for opinion summary generation, baseline metrics, and the implementation details. ## results and analysis g-eval vs. op-i-prompt vs. op-prompts. for open-source models, overall, we observe that op-i-mistral performs the best, followed by op-mistral and then g-eval-mistral. figure 4 shows the performance of different prompt approaches over @xmath0. as we increase the number of generations we generally observe an improvement in the correlation. op-i-mistral shows an improvement against g-eval-mistral across all 7 dimensions and by a large margin specifically for aspect coverage, sentiment consistency, and specificity.