References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation (2025)

Abstract: Human language production exhibits remarkable richness and variation, reflecting diverse communication styles and intents. However, this variation is often overlooked in summarization evaluation. While having multiple reference summaries is known to improve correlation with human judgments, the impact of the reference set on reference-based metrics has not been systematically investigated. This work examines the sensitivity of widely used reference-based metrics in relation to the choice of reference sets, analyzing three diverse multi-reference summarization datasets: SummEval, GUMSum, and DUC2004. We demonstrate that many popular metrics exhibit significant instability. This instability is particularly concerning for n-gram-based metrics like ROUGE, where model rankings vary depending on the reference sets, undermining the reliability of model comparisons. We also collect human judgments on LLM outputs for genre-diverse data and examine their correlation with metrics to supplement existing findings beyond newswire summaries, finding weak-to-no correlation. Taken together, we recommend incorporating reference set variation into summarization evaluation to enhance consistency alongside correlation with human judgments, especially when evaluating LLMs.

## introduction human-written texts vary widely in terms of length, style, communicative intent, lexical/syntactical choices, and numerous other dimensions @xcite @xcite . such variation poses a significant challenge in the evaluation of summarization systems @xcite . traditional summarization metrics typically rely on comparing * equal contribution. system outputs to one or more references, treating these references as a "gold standard". although the limitations of reference-based metrics have long been acknowledged @xcite @xcite @xcite , they remain widely popular due to their simplicity, low compute requirements, relative ease of adaptation to different languages, and reproducibility. the assumption behind the use of referencebased metrics is that system outputs that are more similar to the reference(s) are better, due to their "human-likeness" @xcite . however, the significant variation in human-written summaries implies that evaluating system outputs against a single or limited set of references has inherent drawbacks. previous research has extensively looked at correlations between metrics and human judgments in summarization @xcite , further exploring the use of multiple references to improve such correlations @xcite @xcite as well as the interpretability and efficiency aspects of such automatic metrics @xcite . however, a much less studied question is the extent to which automatic metrics are sensitive to the choice of human-written reference summaries, as shown in figure 1 . in other words, are these metrics stable across different plausible gold-standard refer-ences? if metric scores vary significantly with the selected reference(s), this variation calls into question the reliability of many evaluation practices in the field. in this work, we quantify the impact of reference choice on automatic evaluation metrics for summarization. our contributions are as follows: our analysis reveals that few metrics tend to show both reasonable correlation with human judgments and robustness to the reference sets, especially when scoring llm outputs. the code is available at https://github.com/ mainlp/references-matter . metrics meta-evaluation. meta-evaluation of summarization metrics typically focuses on the extent to which they can be used as a proxy for human evaluation. @xcite examined the validity of automatic scores for nlg tasks, while @xcite focused on rouge and its correlation with humans. @xcite showed that metrics with reasonable correlation on lower-quality outputs tend to diverge when output quality increases. @xcite demonstrated the idiosyncrasies of automatic evaluation metrics, noting that high correlation with human judgments is not sufficient to characterize their reliability. @xcite performed a large-scale meta-evaluation of summarization metrics, and found that most metrics have low correlation with human judgments on coherence, while relevance is weakly or moderately correlated. @xcite performed a meta-analysis of reference-based, reference-free, and llm-based summarization metrics focusing on eight languages from four typological families, showing low correlation. they also observed that off-the-shelf llms as judges still lag behind other metrics. while most existing research focused on correlation to human scores, @xcite addressed the challenge of evaluation when a limited number of references is available. they proposed leveraging llms to diversify the references, expanding the evaluation coverage and improving the correlation with humans. their results show that increasing the number of references significantly enhances the reliability of existing evaluation metrics in terms of correlation. however, since llm outputs tend to show less variability and follow distinct patterns compared to humanproduced content @xcite @xcite , relying on them to replace human references might introduce biases. beyond summarization, evaluation of reference variability has also been conducted in tasks such as machine translation @xcite @xcite and image captioning @xcite . ## experimental setup to quantify the impact of human-written references on the scores of automatic metrics, we leverage multiple elements. for datasets, we use summeval @xcite , gumsum @xcite , and duc2004 @xcite , which contain multiple humanwritten summaries ( §3.1), to assess how different reference summaries affect metric performances. next, to assess summarization models, we use the existing outputs provided by @xcite for summeval. as these outputs predate llms, we additionally collect outputs using llms ( §3.2) for all three datasets. lastly, to compute the correlations with humans, we use the human judgments available in summeval and gather new human ratings for gumsum on both human and llm-generated summaries ( §3.3). we prioritize gumsum over duc2004, as it includes multiple genres beyond news data. our metric selection is outlined in §3.4. details on data licensing and codebase are provided in appendix a. ## reference variability and metric robustness reference-based metrics assume that more human-like outputs deserve higher scores. however, human summaries are very diverse. this section examines how metrics fluctuate with different human references. by analyzing metric robustness, we aim to understand how conclusions about models, drawn from reference-based metrics, might change when different sets of human-written references are used, thereby undermining evaluation reliability. ## conclusions in this work, we have investigated how reference sets impact the reliability of referencebased summarization metrics. our analysis across three multi-reference datasets reveals that, despite their popularity, token-matching metrics such as rouge are highly sensitive to the reference(s). this sensitivity leads to instability in system rankings, particularly when only a small number of references are available, which is typical in summarization datasets. in these situations, we thus recommend avoiding such metrics, echoing earlier calls for caution @xcite , in favor of model-based or reference-free alternatives. we find that increasing the number of reference summaries consistently improves both the stability of metric scores and their alignment with human judgments. this might be explained by the possibility of representing a larger human diversity from the one hand, and from that of limiting annotator bias on the other. in these conditions, n-gram-based metrics such as rouge-n become more reliable. our findings highlight the need to incorporate reference set variation into evaluation frameworks. future metric development should explicitly account for this variation. while we have not specifically studied the dimensions of diversity in human references besides their lexical, syntactical, and semantic variation from @xcite , we believe this is an important area of investigation. we speculate that the main challenge would be to collect a reference set that is "diverse enough" to represent human production for a fixed number of references. future work in this direction needs to identify and characterize the relevant dimensions of diversity. these dimensions might be lexical, stylistic, intentional, or even sociolinguistic (cutting across the previously mentioned dimensions, @xcite . how to collect such references and whether specific guidelines should be adopted is also an open problem. future work should also explore the role of genre. we thus advocate for the creation of larger, more diverse multi-reference datasets, as well as for metric designs that are inherently robust to reference variability. such efforts will be key to ensuring fairer and more reliable and human-aligned evaluation practices in summarization in the llm era and beyond.