Reference-Free Summarization Evaluation with Large Language Models (2023)

Abstract: With the continuous advancement in unsupervised learning methodologies, text generation has become increasingly pervasive. However, the evaluation of the quality of the generated text remains challenging. Human annotations are expensive and often show high levels of disagreement, in particular for certain tasks characterized by inherent subjectivity, such as translation and summarization.Consequently, the demand for automated metrics that can reliably assess the quality of such generative systems and their outputs has grown more pronounced than ever. In 2023, Eval4NLP organized a shared task dedicated to the automatic evaluation of outputs from two specific categories of generative systems: machine translation and summarization. This evaluation was achieved through the utilization of prompts with Large Language Models. Participating in the summarization evaluation track, we propose an approach that involves prompting LLMs to evaluate six different latent dimensions of summarization quality. In contrast to many previous approaches to summarization assessments, which emphasize lexical overlap with reference text, this method surfaces the importance of correct syntax in summarization evaluation. Our method resulted in the second-highest performance in this shared task, demonstrating its effectiveness as a reference-free evaluation.

## introduction text summarization is a natural language processing (nlp) task that aims to condense a given text into a shorter version while retaining its most essential information. it plays a crucial role in information retrieval, content extraction, and document management. automatic summarization systems, whether extractive (selecting and rearranging existing sentences) or abstractive (generating novel sentences), offer significant advantages in various domains such as news articles, legal documents, academic papers, and online content. the ability to generate concise and coherent summaries enhances information accessibility, facilitates quicker decision-making, and improves user experience in an era of information overload @xcite . a good summary plays a pivotal role in information processing and communication across various domains. it serves as a concise yet comprehensive representation of a larger body of text, distilling the core ideas, key information, and essential insights. the importance of a good summary lies in its ability to save time and effort for readers, enabling them to grasp the main points quickly and make informed decisions without delving into extensive documents or articles. a well-crafted summary is not merely a condensation of content; it is a bridge between complex information and its audience, ensuring that knowledge is accessible and actionable. evaluating the output of summarization systems is of paramount importance to ensure their effectiveness and utility. it involves assessing key factors like coherence, informativeness, and fluency. adequate evaluation frameworks help researchers and practitioners to fine-tune algorithms, identify areas for improvement, and compare different summarization methods @xcite . a comprehensive evaluation not only facilitates the development of robust summarization algorithms but also guides their practical applications in realworld scenarios, addressing the increasing need for efficient content summarization in the digital age. numerous well-established evaluation metrics, as detailed in section 2, are typically employed to assess the quality of generated summaries compared to reference summaries. these metrics include, but are not limited to, rouge (recall-oriented understudy for gisting evaluation), bleu (bilingual evaluation understudy), meteor, bertscore, and moverscore, among others. the majority of the metrics employed for the evaluation of generated summaries share a com-mon requisite; namely, the availability of reference summaries. although reference-based evaluation methods can offer valuable insights into the performance of summarization systems, they come with inherent limitations. one significant challenge is the subjectivity of reference summaries. summarization tasks often involve multiple valid ways to condense and express content, leading to diverse reference summaries for the same source text. consequently, reliance on a limited set of references can introduce bias and fail to capture the full spectrum of acceptable summarization outputs @xcite . another problem with reference-based evaluation is the issue of task-specific references. creating reference summaries requires significant human effort, making it impractical to amass a large and diverse reference set for every possible source text. as a result, reference summaries may not adequately cover the variety of linguistic styles, domain-specific terminologies, or nuances in summarization needs, leading to biased evaluations that favor systems generating summaries similar to the available references. furthermore, most reference-based metrics primarily hinge on the presence or absence of specific words within generated summaries as the core element of their evaluation criteria. nevertheless, other critical factors, such as coherence, readability, fluency, and consistency, among others, have been recognized as pivotal elements in the usability of text summaries @xcite . these essential aspects of summary assessment can be regarded as latent dimensions in the overall quality assessment. to overcome the previously discussed challenges and in light of the recent advancements in large language models (llms) and their widespread applicability, the eval4nlp workshop organized a shared task. this task was specifically designed to investigate whether llms can be used to evaluate text summarizes, solely on the basis of the original text. with this aim in mind, the organizers provided a list of six llms sourced from hugging face, as outlined in section 5. these models are diverse in their parameter counts and training data. we participate in this challenge by designing different types of prompts focusing on the latent dimensions of the evaluation process. we conducted various experiments combining different prompts with the six available llms -including both large and small models -and evaluated the results on the training and validation sets to develop the final methodology. the final evaluation on the test set revealed that our best proposed prompt, coupled with a smaller llm, achieved a notable kendall τ correlation value of 0.49. this outcome positioned our system as the second-best performer in the competition. the remainder of this paper is structured as follows: we commence with a review of related work in section 2. section 3 is dedicated to the dataset employed in our experiments, providing an overview of its characteristics. subsequently, in section 4, we delve into the solutions implemented. in section 5, we elaborate on the experimental framework and present the results obtained. lastly, we conclude the paper in section 6 with a discussion of our findings and areas for future work. ## related work the quality evaluation of textual data generated in the era of natural language processing has always been seen as a difficult task because of the inherent complexity and diversity of textual data @xcite . the fact that a single idea can be expressed in multiple ways poses a challenge for reference-based methods, as they cannot cover all possible scenarios comprehensively, besides the costs of preparing the references for the evaluation. on the other hand, creating dependable referencefree metrics is not a straightforward endeavor and can be problematic as they must be able to correctly evaluate the different summaries generated from a same source text.. traditional metrics of summarization quality have also failed to take into account important aspects such as coherence, fluency, and consistency @xcite . various reference-based evaluation metrics are frequently used in text generation tasks. some of the important ones are as follows: rouge stands as a widely adopted metric in the assessment of summarization quality. it quantifies the degree of overlap in n-grams between the generated summary and the reference summary. rouge is computed for various word n-gram sizes, such as 1-gram, 2-gram, and 3-gram, and the resulting scores are aggregated to produce a comprehensive evaluation score @xcite . bleu is another reference-based metric used to assess the quality of machine-generated text summaries by measuring how closely they match human-written reference summaries. it quantifies the precision of n-grams in the machine summary that also appear in the reference summary, providing a score that indicates the summary's accuracy and fluency @xcite . though bleu and rouge both evaluate language quality, they diverge in their emphasis and methodology. bleu places a primary focus on precision, whereas rouge prioritizes recall as its key metric. meteor (metric for evaluation of translation with explicit ordering) is a text summarization metric that evaluates the quality of machinegenerated summaries by considering a variety of linguistic aspects, including unigram matching, stemming, synonyms, and word order. it provides a comprehensive measure of overall summary quality and can account for different ways of expressing the same information, making it a robust evaluation metric for text summarization @xcite . moverscore @xcite and chrf @xcite ) assess the quality of generated summaries by comparing character n-grams between the generated summary and human reference summaries. chrf accounts for both precision and recall and is particularly useful for languages with complex morphology and word forms @xcite . moving away from the reference-based approach, @xcite have introduced new metrics that rely on question-answering and demonstrated their positive outcomes when employed as rewards in a reinforcement learning setting. importantly, these metrics do not depend on human references and can be computed directly from the text to be summarized. in another study by @xcite , the authors explored the viability of llms, focusing on chatgpt and the text-davinci series models, for reference-free text quality assessment. they conducted a comparative analysis of various techniques for evaluating text quality and identified the utilization of an explicit score generated by the gpt model as the most efficacious and consistent approach. they also discussed prompt design as an important factor influencing quality of scores generated by gpt model. bertscore is another reference-free text summarization metric that leverages bert (bidirectional encoder representations from transformers) embeddings to measure the similarity between the machine-generated summary and human reference summaries. it considers contextual information and semantic similarity, providing a more nuanced and accurate evaluation of summary quality @xcite . @xcite , introduces a novel approach by advocating the direct utilization of pretrained natural language inference (nli) models as evaluation metrics. furthermore, they developed a novel preference-based adversarial test suite for machine translation and summarization metrics. with this approach, there is no need for human annotators and it is particularly well-suited for reference-free evaluation. additionally, their research findings indicate that nli metrics exhibit strong performance in the context of summarization but yield results below the established standard metrics in the domain of machine translation. in the study conducted by @xcite , gemba, an assessment method based on gpt technology, was introduced. the researchers conducted an evaluation of their metrics by comparing them to the metrics included in the wmt22 metrics shared task. remarkably, their approach demonstrated state-of-the-art performance on the mqm 2022 test set across three distinct language pairs: english to german, english to russian, and chinese to english. @xcite , did a comprehensive analysis of the potential of large language models in the context of machine translation evaluation through score prediction. they introduced a novel prompting technique known as automqm, which effectively harnesses the multidimensional quality metrics (mqm) framework for the purpose of achieving interpretable machine translation (mt) evaluation using large language models (llms). a study by @xcite aimed to assess the alignment of current reference-free evaluation metrics with human preferences when ranking summarization systems. they focused on two principal categories of metrics: quality and factuality metrics. within the quality metrics, they examined supert @xcite , which assesses the quality of generated summaries by contrasting them with automatically identified pivotal sentences from the input, along with blanc @xcite , which scrutinizes summaries via language understanding tasks. the second category of metrics is specifically designed to gauge the presence of inaccuracies in generated summaries concerning the source article. ermakova et al. (2019) provided a comprehensive overview of existing metrics for summary evaluation. they pointed out various limitations in these existing evaluation frameworks and introduced an automatic evaluation framework that eliminates the need for human annotations. they categorized the evaluation metrics into informative metrics like rouge and readability metrics including coherence, conciseness, content, grammar, recall, pithiness etc. @xcite conducted another extensive survey of the currently available automatic evaluation metrics in the domain of natural language generation (nlg). they subsequently introduced a systematic taxonomy to categorize these evaluation metrics, with the categorization structured around the methodologies they employ. @xcite , showed that in-context learning can serve as a viable alternative to fine-tuned evaluation metrics for assessing nlg tasks. by employing a limited set of examples, in-context learning evaluators can achieve, and in some cases surpass, the current state-of-the-art performance in multi-dimensional evaluation. this approach's robustness is evident across various in-context examples. furthermore, the research reveals a strong alignment between in-context learning evaluators and human judgments when evaluating summaries generated by gpt-3. the present study shares similarities with the previously discussed reference-free evaluation metrics in that it operates without the need for reference summaries. however, unlike other approaches that entail intricate configurations, the model introduced here solely relies on straightforward prompts used with pre-trained llms. ## data and evaluation in the eval4nlp 2023 shared task, the dataset provided for the summarization track comprises training and validation subsets, each containing source texts along with their corresponding summaries. these summaries have been generated by a summarization model that was trained on the cnn/dailymail dataset, as documented by @xcite . notably, the training dataset includes associated scores for each generated summary relative to its source text, which are intended for use in the system development process. furthermore, the organizers have also introduced a test set, which encompasses sentences and paragraphs extracted from english wikipedia pages created subsequent to the date of july 15, 2023 (i.e., beyond the llama2 training cutoff) @xcite . for a comprehensive overview of the dataset, including key statistics, please refer to the validation and test data sets do not include explicit score annotations, necessitating participants to submit their results on the shared task page hosted on codabench 1 the evaluation process in this study adheres to the metrics established in the wmt22 competition, as described by @xcite , and employs segment-level kendall correlation as the primary evaluation metric. in the realm of statistics, the kendall rank correlation coefficient, commonly known as kendall τ coefficient, is a statistical measure employed to assess the ordinal association between two measured variables. a τ test, which is a non-parametric hypothesis test used to determine statistical dependence based on the τ coefficient, is employed for this purpose. the ranking of systems in the shared task will be determined by their kendall correlation scores on the test set, with the highest correlation indicating superior performance. ## solution irrespective of the type of summarization, whether it pertains to single or multi-document summarization or falls within the categories of abstractive or extractive summarization, certain fundamental criteria must be met by any generated summary. as highlighted by ter @xcite , five of these criteria include: (1) coherence (does information flow logically from one sentence to the next?), (2) completeness (does the summary capture the most important information from the text?), (3) conciseness (is the summary brief and to the point?), ( 4 ) consistency (does the information in the summary align with that in the original text), and (5) readability (is the summary written in a clear and understandable manner?). additionally, adhering to the conventions of correct language syntax stands as an imperative prerequisite, representing a sixth criterion complementing the other aforementioned factors for any text generated for various purposes. in our approach to the eval4nlp shared task, we devised straightforward prompts encompassing the six latent dimensions mentioned above. these ## experiments in line with the prompt design outlined in section 4, we leveraged the computational resources offered by the canada digital alliance to apply the designated models with diverse prompts across both the training and validation datasets. the organizers categorized models with parameters fewer than 25b as "small" and the rest as "large" models. we conducted experiments across all these models, and the performance variations, as indicated in when evaluated on the training data, the best performance was achieved by the following prompts (in ranked order): p7, p2, p1, p5, p6, p4, and p3. in contrast, for the validation data, a slightly different order emerged, with p5, p2, p6, p7, p4, p1, and p3 being more effective. this variation is reasonable given that the source texts and generated summaries for the two datasets originate from different sources. subsequently, we proceeded to apply certain model-prompt combinations that had demonstrated promising results during the training and validation phases to the released test data. the performance of these selected model-prompt pairs, as evaluated by the organizers on the test data, is presented in upon comparing the similarity between the results from the validation and test sets, it becomes evident that the test set exhibits greater similarity to the validation data rather than the training data. these results confirm that the utilization of large-scale language models (i.e. the llms with an extensive parameter count) without fine-tuning does not consistently yield high performance in the context of evaluation score generation tasks. in addition, the best results were achieved using the prompt for syntax, emphasizing the significance of this latent dimension in the quality of the generated summaries. syntax is largely overlooked by reference-based metrics that focus on lexical overlap between the generated summary and a reference summary; however, our results suggest that it plays an important role in evaluation. the secondhighest score was achieved using the prompt for completeness, consistent with the idea that a summary should include the most salient points from the original text. it is worth highlighting that regulatory constraints imposed on participants prevented us from exploring the possibility of combining the scores from various prompts and models during our experimental phase. however, by employing a solitary model, we achieved a notable second-place ranking in the competition. ## conclusion the assessment of summarization system outputs is vital to ascertain their efficiency and usefulness. traditional approaches to summarization evaluation involve comparing the generated text with human-written reference summaries. however, the constraints associated with reference-based metrics encourage the researchers and practitioners to seek reference-free metrics for the evaluation and comparison of various summarization methods. with the objective of formulating effective prompts for utilization along with llms, the eval4nlp organized a collaborative initiative. the primary goal of this endeavor was to systematically examine the potential utility of llms in the evaluation of text summaries, relying exclusively on the source text. in this study, we actively engaged in the development of prompts tailored to each of the six latent dimensions (i.e. completeness, conciseness, readability, coherence, consistency and syntax) found to be relevant to summary evaluation. one specifically devised prompt, centered on the syntactic assessment of generated summaries, garnered a noteworthy score of 0.49 in terms of kendall τ , thereby securing the second-highest position among performance evaluation systems. our primary focus in the present work involved the utilization of individual llms. nevertheless, we acknowledge that the collaborative use of various models presents a promising avenue for potential performance enhancement, which we consider as a valuable direction for future investigations.