Understanding Large Language Model Based Metrics for Text Summarization (2023)

Abstract: This paper compares the two most widely used techniques for evaluating generative tasks with large language models (LLMs): prompt-based evaluation and log-likelihood evaluation as part of the Eval4NLP shared task. We focus on the summarization task and evaluate both small and large LLM models. We also study the impact of LLAMA and LLAMA 2 on summarization, using the same set of prompts and techniques. We used the Eval4NLP dataset for our comparison. This study provides evidence of the advantages of prompt-based evaluation techniques over log-likelihood based techniques, especially for large models and models with better reasoning power.

## introduction transformer-based language models have revolutionized the field of natural language processing (nlp), particularly in the area of language generation. however, the improved language generation capabilities of these models have also exposed the limitations of traditional lexical evaluation metrics, such as perplexity, bleu @xcite , and rouge @xcite . these metrics are often unable to accurately assess the quality of generated text, especially when it is creative or informative. in response, researchers have developed a wide range of new automatic evaluation models, such as bleurt @xcite , bertscore @xcite , and bartscore @xcite . these models typically rely on a combination of lexical and semantic features to assess the quality of generated text, and some of them also take into account the golden reference annotation. recent large language models (llms) like palm @xcite , gpt-3.5, and gpt-4 (openai, 2023) have taken language generation capabilities to a new level, making it difficult to distinguish between machine-generated and human-written text. this has led to the use of llms for a variety of more complex tasks, such as summarizing entire research papers, even when the ground truth is not known. the increased complexity of these tasks has spurred interest in using llms themselves for model evaluation. prompt-based and log-likelihood-based evaluation are two widely used approaches for automatic evaluation of large language models (llms). however, it is unclear which approach works better with different model sizes, as previous studies have used these approaches on mutually exclusive sets of models. in this paper, we evaluate multiple llm models of different sizes using both prompt-based and log-likelihood-based evaluation on the eval4nlp dataset @xcite as part of the eval4nlp shared task @xcite . we experiment with three models from the llama @xcite and llama2 @xcite family, which are allowed in the eval4nlp 2023 shared task. our results show that prompt-based evaluation generally outperforms log-likelihood-based evaluation for all model sizes. this is likely because prompt-based evaluation is more directly aligned with the tasks that llms are typically used for, such as generating text, translating languages, and answering questions. our findings suggest that prompt-based evaluation is a more reliable and informative approach for evaluating llms of all sizes. ## dataset and task description the summarization track of the eval4nlp task involved predicting an overall score for a modelgenerated summary of a source text. the competition required participants to use only a limited set of models without fine-tuning, meaning that the #of examples train 320 dev 1280 test 825 ## experiments the competition allowed only variants of the 13b llama and llama2 models, as well as quantized versions of llama or llama2 models with 60b+ parameters. our main aim was to compare prompt-based evaluation and log-likelihoodbased evaluation techniques across different model sizes. therefore, we decided to work with the noushermes-13b @xcite and platypus-70b @xcite models. however, since these two models belong to different llama families, we also included the results obtained using the ocra-13b model @xcite , which is based on llama2, for a fair comparison. we experimented with two different approaches as follows: ## results comparing the likelihood based scores for the platypus-13b model across the single scoring and the 2 different prompts sets for average scoring from the likelihood performance of llama2 based models is consistently worse than those of llama based models across all settings. the performance of ocra-13b model is similar to the noushermes-13b model in case of likelihood based approach. but considering that prompt based scores are reversed for the two, it seems llama2 based models are generally worse than llama based models in the case of likelihood. we believe that one of the reasons for this could be that llama2 based model's generation distribution might be different. i.e. it might consider most of the summaries to be average in nature resulting in low likelihood. fur-ther analysis and experiments with other instruction tuned model might be required to understand if other llama2 based models also have similar results. for the prompt based evaluations we can see that using a single prompt to get a score led to performance degradation across all the three models. this shows that the use of a complex prompt makes the reasoning process difficult for the model. the performance of llama2 based ocra-13b model is much better than the llama based noushermes model. the performance different between the two models is vastly different. the two reasons for this could be (a) ocra is a llama2 based model or (b) different instrucion tuning data used for the two models. we believe the first to be true as it is eviden from the huggingface leaderboards, where llama2 based models are consistentl ranked higher than llama based models. lastly the quantized platypus-70b model surpasses the performance of ocra-13b model in the scoring based approach showing that bigger models tend to improve performance, even if it has been quantized down to 4-bits. we tested the best models across both the settings i.e. the likelihood and the prompt based approach on test dataset. all the submission were made under the team name of beginners. noushermes-13b model achieved the best results using the likelihood based approach with a score 0.38 on test data. a single prompt was used as shown in appendix d with the submission id 20138. the platypus-70b model achieved the best score score in the prompt based approach. it got a score of 0.44 on test data by averaging the scores obtained using four different prompts for four different aspects (consistency, fluency, relevance, coherence) with submission id 20254. ## conclusion prompt-based evaluation technique outperforms log-likelihood-based evaluation technique in text summarization evaluation. however, evaluating single summaries is challenging, as there are many different aspects to consider, and some aspects may be more important than others. averaging scores from different aspects improves performance, suggesting that there are other evaluation aspects that we did not consider. llama2 based models seem better at reasoning and making decisions, even with low likelihood scores. therefore, combin-ing prompt-based evaluation with llama2 based models may further improve text summarization evaluation results.