Real-World Summarization: When Evaluation Reaches Its Limits (2025)

Abstract: We examine evaluation of faithfulness to input data in the context of hotel highlights—brief LLM-generated summaries that capture unique features of accommodations. Through human evaluation campaigns involving categorical error assessment and span-level annotation, we compare traditional metrics, trainable methods, and LLM-as-a-judge approaches. Our findings reveal that simpler metrics like word overlap correlate surprisingly well with human judgments (r=0.63), often outperforming more complex methods when applied to out-of-domain data. We further demonstrate that while LLMs can generate high-quality highlights, they prove unreliable for evaluation as they tend to severely under- or over-annotate. Our analysis of real-world business impacts shows incorrect and non-checkable information pose the greatest risks. We also highlight challenges in crowdsourced evaluations.

## introduction instruction-tuned large language models (llms) have become ubiquitous in natural language processing @xcite . they proved very capable and versatile, with simple prompting achieving various tasks, avoiding the need to produce costly in-domain training datasets required for most previous approaches @xcite . however, llms are known to have issues with faithfulness of their outputs to the input, where they produce text not grounded in the input prompt (hallucinations; @xcite . evaluating faithfulness is therefore crucial, especially without human-written references, which are expensive and risk leakage into llms' training data @xcite . however, traditional evaluation metrics for nlp show low correlation with human judgments @xcite and overt reliance on surface similarities (gehrmann * work done while on internship at trivago. @xcite . despite this, most works in the field still rely on them @xcite . a new alternative is using llms themselves to evaluate generated outputs @xcite . while promising, llms may show self-bias @xcite and performance may vary across domains. we focus on faithfulness evaluation of text summarization using a case study of generating short highlights from hotel descriptions and reviews @xcite . we are concerned with the following research questions: (1) how well do faithfulness metrics generalize to unseen domains? (2) can we evaluate faithfulness in a referenceless scenario? (3) can llms be used as judges in this setting? and (4) what is the estimated business impact of these errors? in response to these questions, we present the following contributions: hotel highlights @xcite are brief llm-generated summaries that capture unique features of hotels based on their descriptions and reviews. these highlights help travelers select appropriate accommodations without reading numerous reviews and lengthy descriptions. consequently, ensuring highlight accuracy is essential. the highlights intentionally present hotels in a positive light, with subjective phrases such as 'a local gem' that, while not explicitly supported by the input text, are not considered errors. this characteristic presents additional evaluation challenges. the objective of the task is not to produce a single gold-standard summary, but rather to generate a diverse set of approximately 10 highlights that can be ranked and sampled. we therefore focus primarily on verifying whether the information in each highlight is properly grounded in the source description. ## discussion gold human annotation? after investing £800 in crowdsourced span annotations and receiving poor-quality results despite multiple precautions (pre-filtering participants, conducting pilots, and offering payment bonuses for attention checks), we conclude that alternative approaches may be more cost-effective for difficult tasks. these include evaluating smaller output samples with larger annotator groups, or whitelisting trusted annotators who pass multiple qualification tests @xcite . we refer to @xcite for more observations on human evaluation of hallucinations. some llms pass attention checks gemma3 and o3-mini both passed the attention check that was designed to be simple and visible, yet only 33% of crowdworkers passed it. this shows that llms can capture the more visible errors, but still have room for improvement in subtler errors. running gemma3 without validation would underestimate the amount of non-checkable errors which have a higher business impact. similarly, relying on gpt-4o without the context of validation would significantly overestimate the seriousness of incorrect errors. @xcite found that most trainable summarization metrics work more reliably on older model outputs compared to the new ones. indeed, we can see that the best trainable metric is the modernbert nli model from 2024 @xcite . on the other hand, bertscore @xcite , alignscore @xcite , as well as the older nli models @xcite performed worse. ## related work the validation of automatic metrics against human judgment is an active area of research, with broad consensus that metrics cannot fully substitute for human evaluation @xcite . this challenge has intensified with recent models, whose outputs often fall into metrics' blind spots @xcite . however, few works evaluate the real-world impact of nlp systems @xcite . @xcite validated automatic metrics against human judgment for medical consultation note generation, finding -consistent with our work -that human insights are essential for assessing practical utility, and that simple metrics retain significant value. similarly, @xcite employed auxiliary tasks such as question-answering to evaluate summary usefulness, showcasing alternative approaches to intrinsic metrics. ## conclusion addressing our research questions, we demonstrated that: (1) nli entailment and simple statistical metrics achieve moderate correlation with human judgments and are thus the best out-of-thebox options for measuring faithfulness; (2) referenceless evaluation can be effective when validated properly; (3) while llms excel at generating hotel highlights, they prove unreliable as evaluators of content faithfulness; and (4) non-checkable and incorrect information have the highest potential for negative business impact. we believe that real-world evaluation of tasks that emerged with the rise of llms and few-shot prompting should be studied carefully. current evaluation methods are insufficient for automated quality assurance, and the errors that go unnoticed are likely to cause a negative business impact.