L ong E val: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization (2023)

Abstract: While human evaluation remains best practice for accurately judging the faithfulness of automatically-generated summaries, few solutions exist to address the increased difficulty and workload when evaluating long-form summaries. Through a survey of 162 papers on long-form summarization, we first shed light on current human evaluation practices surrounding long-form summaries. We find that 73% of these papers do not perform any human evaluation on model-generated summaries, while other works face new difficulties that manifest when dealing with long documents (e.g., low inter-annotator agreement). Motivated by our survey, we present LongEval, a set of guidelines for human evaluation of faithfulness in long-form summaries that addresses the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can we minimize annotator workload while maintaining accurate faithfulness scores? and (3) Do humans benefit from automated alignment between summary and source snippets? We deploy LongEval in annotation studies on two long-form summarization datasets in different domains (SQuALITY and PubMed), and we find that switching to a finer granularity of judgment (e.g., clause-level) reduces inter-annotator variance in faithfulness scores (e.g., std-dev from 18.5 to 6.8). We also show that scores from a partial annotation of fine-grained units highly correlates with scores from a full annotation workload (0.89 Kendall’s tau using 50% judgements). We release our human judgments, annotation templates, and software as a Python library for future research.

## introduction human judgments are considered the gold standard for evaluating model-generated sum-maries @xcite and generated text more broadly @xcite . unfortunately, human evaluation tends to be labor-intensive, expensive to scale, and difficult to design. this is problematic as a large number of judged examples is needed to draw statistically significant conclusions about system performances @xcite or correlations between human judgments and automatic metrics @xcite . human evaluation is especially challenging when long sequences of generated text need to be evaluated, due to the inherent subjectivity in the task @xcite @xcite . to better understand the challenges of human evaluation on long-form summaries (150 words or longer), we first conduct a comprehensive survey of 162 publications and preprints on long-form summarization (section 2). we find that 119 papers (73%) do not perform human evaluation on long-form summaries, while the remaining papers deviate significantly from suggested best practices for reproducibility @xcite . current human evaluation setups lack standardization in their design decisions (such as annotation granularity), some of which can significantly impact inter-annotator agreement (section 3.1). finally, 20 papers explicitly mention human evaluation is expensive, difficult, and time-consuming due to the long length of summaries and source documents. to move towards a more consistent and efficient human evaluation, we present longeval, a set of guidelines for human evaluation of faithfulness in long-form summarization (section 3). we empirically evaluate longeval using human annotation studies on two long-form summarization datasets: squality @xcite and pubmed @xcite . we provide an overview of our main research questions and findings in figure 1 and enumerate them here: …. he recognized her as old hazeltyne 's daughter harriet , no doubt come to see justice done . she did n't have the hothouse -flower look asa would have expected in a girl whose father owned the most valuable of the planetary franchises . she was not afraid to meet his eye , the eye of a judicially certified criminal . there was , perhaps , a crease of puzzlement in her brow , as if she had thought crimes were committed by shriveled , rat -faced types , and not by young biological engineers who still affected crewcuts . tom dorr , hazeltyne 's general manager , was her escort . asa felt certain , without proof , that dorr was the man who had framed him for the charge of grand theft by secreting a fresh slider egg in his laboratory . the older man stared at asa coldly as he was led out of the courtroom and down the corridor back to jail . jumpy , asa 's cellmate , took one look at his face as he was put back behind bars . " guilty , " jumpy said ….asa took four steps to the far wall of the cell , stood there briefly with his head bent and turned to face jumpy . " nope , " asa said softly . " i 'm going into a conversion tank . i 'm going to be a muck man , jumpy . i 'm going out to jordan 's planet and hunt slider eggs . " " smuggling ? it wo n't work . " asa did n't answer . the hazeltyne company had gone after him because he had … asa graybar is a biological engineer who studies keeping slider eggs alive and he is accused of a crime at the opening of the story . he thinks he was framed by tom dorr , hazeltyne 's general manager . he was offered one year as a " changeling " on another planet or 5 years in rehabilitation on earth . he elects to do the one year , and thinks that he will get into smuggling slider eggs on jordan 's planet ….. rq1: can inter-annotator agreement be improved while evaluating faithfulness of long-form summaries via fine-grained annotations? finding: annotating faithfulness of individual summary clauses and aggregating them leads to significantly higher inter-annotator agreement, compared to the dominant paradigm of evaluating whole summaries at once via likert ratings (std-dev 18.5 to 6.8 on squality). rq2: can we reduce annotator workload by partially annotating a long summary while maintaining accurate faithfulness scores? finding: despite annotating a fraction of summary clauses, faithfulness scores under a reduced workload maintain high correlation with those from a full workload (0.89 kendall's τ at 50% workload). rq3: do humans benefit from automatically aligning summary units to relevant sentences in the source document? finding: unlike suggestions in prior work on shortform summarization @xcite , aligning parts of the summary to source document is only useful when the summary is highly extractive or mostly correct. overall, our contributions are: ## survey of human evaluation practices before discussing longeval, we first attempt to understand current human evaluation practices in long-form summarization through a comprehensive survey of 162 papers. our survey reveals several concerning trends: absence of human evaluation, non-reproducible experimental setups, lack of standardization, and complaints of long summaries being challenging and expensive to evaluate. these results show an urgent need to develop more efficient and standardized human evaluation protocols. ## related work a large body of recent work has focused on new automatic evaluation methods for summarization via nli-based algorithms @xcite or qa-based algorithms @xcite . our work focuses on the much less studied area of human evaluation, the gold standard for developing automatic metrics. a notable effort in this space is the pyramid method @xcite , along with work improving pyramid efficiency @xcite . efficient pyramid-like protocols have been used to collect large-scale datasets human judgments @xcite in short-form news summarization tasks like cnn/dm. while these efforts focus on salience evaluation and assume access to multiple references, our work focuses on faithfulness and operates in a reference-free setting. moreover, we focus on long-form summarization tasks like squality and pubmed, which are much more challenging and expensive to evaluate. evaluating summary faithfulness relates to fact verification @xcite , where claim sentences are checked against a large knowledge source (wikipedia). prior work @xcite attempts to simplify the human fact checking process by methods like knowledge source snippets @xcite , similar to hint highlights ( §3.3). faithfulness in summarization differs from fact verification in three ways: (1) summaries are paragraph-long and contextual compared to single sentence stand-alone claims in fact verification; ## conclusion we present the longeval guidelines, a set of recommendations for moving towards standardized human evaluation of long-form summarization. we empirically analyze each recommendation on two datasets. overall, we find that (1) fine-grained annotations have lower inter-annotator variance than coarse-grained annotations; (2) partially annotating a summary reduces annotator workload while maintaining accuracy; (3) highlighting hints in the source document has limited usefulness for evaluating long-form summaries. as future work, we plan to conduct experiments on other aspects of summarization evaluation like salience and coherence.