Less is More for Long Document Summary Evaluation by LLM s (2024)

Abstract: Large Language Models (LLMs) have shown promising performance in summary evaluation tasks, yet they face challenges such as high computational costs and the Lost-in-the-Middle problem where important information in the middle of long documents is often overlooked. To address these issues, this paper introduces a novel approach, Extract-then-Evaluate, which involves extracting key sentences from a long source document and then evaluating the summary by prompting LLMs. The results reveal that the proposed method not only significantly reduces evaluation costs but also exhibits a higher correlation with human evaluations. Furthermore, we provide practical recommendations for optimal document length and sentence extraction methods, contributing to the development of cost-effective yet more accurate methods for LLM-based text generation evaluation.

## introduction the evaluation of text generation plays a crucial role in the development of high-quality text generation systems @xcite . however, the alignment of automatic evaluation metrics with human judgment remains a challenging task @xcite . recently, large language models (llms) have shown promising results in this regard @xcite @xcite , demonstrating a strong correlation with human evaluations. despite their effectiveness, they face challenges such as high computational cost and the lost-in-the-middle problem @xcite where important information in the middle of long documents is often overlooked for long document summary evaluation. ## methods summarization evaluation metrics assign a rating ŝ to a model-generated summary ŷ. the higher the correlation corr(ŝ, s) between this score ŝ and the human judgment score s, the better the evaluation metric is. to assign a rating ŝ, existing studies use either the reference summary y or the input document x, as well as the generated summary ŷ. to use llms as evaluators, previous approaches commonly use the model-generated summaries ŷ, and the source document x as inputs, where ŝ = f (x, ŷ), but the extract-then-evaluate method comprises two steps to use llms as illustrated in figure 1: (1) extract important sentences for summary evaluation from the long source document x until it reaches the pre-defined length n , and compose a short but information-dense document x ′ . (2) evaluate the quality of the summary ŷ by prompting llms @xcite . design prompts 2 that can take both the extracted source document x ′ and summary ŷ as inputs and generate a rating scale s as output: ŝ = f (g extract (x), ŷ) to extract sentences, we considered the following approaches: this is considered a strong baseline for extractive summarization @xcite . ## discussion how are extracted sentences distributed? we analyzed the positions of sentences extracted by each method. figure 2 displays the distribution of sentence positions when limiting the length to 1024 tokens. for the scientific domain (i.e., arxiv and pubmed), rouge-based methods tend to extract sentences from positions similar to the lead, suggesting that important information is mostly located at the beginning of these documents. in contrast, for the general domain (i.e., gov-report and squality), rouge-based methods tend to extract sentences not only from the beginning but also from various positions throughout documents, indicating that important information might be distributed throughout documents. meanwhile, model-based methods (i.e., bertscore and nli) extract sentences from various positions within the document, regardless of the dataset. how long is the optimal document length? figure 3 shows the relationship between pearson correlation and the length of documents for various datasets and evaluation criteria. the dashed lines correspond to the full document setting. we observed a strong correlation within the document length range of 1000 to 2000 tokens across all datasets. notably, extracted documents should generally be longer than the summaries, while long documents pose the lost-in-the-middle challenges for llms @xcite , causing the correlation curves to initially rise and then decline. which sentence extraction method is the best? as shown in figure 3 (more detailed numbers can be found in ## conclusion in this study, we proposed the extract-then-evaluate method for evaluating long document summaries using llms. our findings demonstrated that this approach not only reduces evaluation costs but also aligns more closely with human evaluations compared to existing automatic metrics. furthermore, we provided practical recommendations for optimal document length and sentence extraction methods, contributing to the development of more efficient and cost-effective methods for text generation evaluation using llms.