Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains (2024)

Abstract: Recent work has shown that large language models (LLMs) are capable of generating summaries zero-shot—i.e., without explicit supervision—that, under human assessment, are often comparable or even preferred to manually composed reference summaries. However, this prior work has focussed almost exclusively on evaluating news article summarization. How do zero-shot summarizers perform in other (potentially more specialized) domains?In this work we evaluate zero-shot generated summaries across specialized domains including: biomedical articles, and legal bills (in addition to standard news benchmarks for reference). We focus especially on the factuality of outputs. We acquire annotations from domain experts to identify inconsistencies in summaries and systematically categorize these errors. We analyze whether the prevalence of a given domain in the pretraining corpus affects extractiveness and faithfulness of generated summaries of articles in this domain. We release all collected annotations to facilitate additional research toward measuring and realizing factually accurate summarization, beyond news articles (The dataset can be downloaded from https://anonymous.4open.science/r/zero_shot_faceval_domains-9B83)

## introduction modern llms now offer strong zero-shot summarization performance, and even surpass fine-tuned models according to human assessments @xcite . indeed, zero-shot summaries are sometimes deemed comparable in quality to reference summaries @xcite . past evaluative work, however, has focused nearly exclusively on news article summarization, a domain in which there is no shortage of available training data. but zero-shot summarization is perhaps most appealing in niche domains where acquiring training data with which to fine-tune summarization models is sparse and may be prohibitively expensive to collect. recent work @xcite suggests the promise of zeroshot summarization in such domains. however, there has not yet been a comprehensive investigation of the factuality of model outputs produced in zero-shot summarization across multiple domains (i.e., beyond news). here we address this gap, and compare the quality of zero-shot summaries generated in niche domains (law, medicine) to those generated for news articles. in evaluating these models, we center the consistency and faithfulness of summaries generated by llms with respect to the input (source) document. inconsistencies within summaries have long posed a challenge @xcite , motivating approaches intended to mitigate this issue @xcite , and for automated evaluation of factuality @xcite @xcite @xcite . here we systematically assess the factual accuracy of zero-shot summarizers across a diverse set of specialized domains. specifically, we look to answer four major questions. ( 1 ) what is the prevalence of errors in zeroshot summaries across various domains, and how does this compare to established results on news summarization tasks? (2) are the types of errors observed in these niche domains different from what has been seen in news article summarization? (3) what is the relationship between the frequency of domains in training corpora and the likelihood of model hallucinations in these domains? (4) are existing automatic systems for factual evaluation reliable across multiple domains? to answer these questions, we enlist expert annotators to manually evaluate the outputs from two representative zero-shot summarization systems-gpt-3.5 (gpt-3.5-turbo-0301; @xcite and flan-t5-xl (chung et al., 2022)-across standard and niche summarization datasets. specifically, we evaluate (zero-shot) summaries of medical and legal documents, as well as news articles for reference. in general, we find that the proportion of factual inconsistencies in summaries varies considerably across domains, calling into question the community focus on news summarization datasets specifically. further, we find evidence that the prevalence of articles in pretraining data from a given domain may correlate with the factuality of summaries of articles from the same. we speculate that this may be due to the model introducing content implicit in its weights in such cases (whereas it may have less "knowledge" in niche domains), although this would need to be validated in future work. ## manual evaluations of summaries data we use xsum @xcite and @xcite for news, as well as niche domains like pubmed (medicine; @xcite ) and legal bills (law; kornilova and eidelman 2019) for comparison. we select articles shorter than 4096 tokens from the test sets to accommodate model token limitations, resulting in approximately 22,000 articles for news, 3,000 for billsum, and 200 for pubmed. we randomly (i.i.d.) sample 50 articles from each domain. we provide more data statistics in appendix a.1 ## results how prevalent are errors across domains? figure 1a shows the average proportion of sentences marked as inconsistent (with respect to the corresponding input) in summaries generated by gpt-3.5 @xcite and flan-t5 xl @xcite for three domains: news, medical, and legal. perhaps surprisingly, we observe a higher prevalence of inconsistencies for news articles, as compared to the specialized domains of medicine and law. while flan-t5 introduces more errors than gpt-3.5 overall, the trends are analogous. error categories across domains we next characterize the distribution of error categories in factually inconsistent summaries generated by models across the domains considererd. figure 1b reports the distribution of error categories for both models. 3 there are more extrinsic errors introduced in the news domain compared to the niche domain datasets. we include "mixed" errors for cases where errors were classified as different types (intrinsic/extrinsic) by annotators. the news domain has a higher frequency of such cases. reviewing these, we find that they include cases where the summary both misinterprets source information and where it introduces new information. we provide examples in appendix a.5. an "other" option is available to annotators, along with a comment box for capturing miscellaneous errors. annotator comments highlight instances where there is no clear misunderstanding but instead a misleading overall impression, such as the over-generalization of specific information in the summary how extractive are summaries, and how does this relate to factuality? we investigate the relationship between extractiveness (i.e., degree of copying) and factual accuracy across domains. specifically, we take the proportion of 3-gram sequences in the summary that are also present in the source for each source-summary pair as a proxy measure for extractiveness. figure 2 reveals that there is a comparable level of copying across different models and domains. however, models tend to copy more often when summarizing articles in the pubmed dataset; this could explain the lower frequency of errors in this domain, since extractive summaries are unlikely to t5 ). this likely explains the greater factuality observed in this domain, and may reflect unfamiliarity with the domain (see figure 3 ). "hallucinate" by definition. we calculated spearman rank correlations between 3-gram overlaps and factuality scores for article-summary pairs. the correlations for the news, billsum, and pubmed domains are 0.61, 0.38, and 0.16 respectively. domain representation in pretraining corpora and its relation to factuality. one possible explanation for the higher proportion of factual errors in news datasets compared to specialized domains is that general news has greater representation in the training data. as a proxy to measure model exposure to articles belonging to these domains we prompt llms to generate overviews of articles based on titles only (headlines for news articles, bill titles for billsum, and study titles for pubmed). we use the template "generate a comprehensive overview of the following topic: [title]" to generate text for each article title, assessing llms' memorization. we speculate that increased exposure to an article topic in training data should enable llms to reproduce more content present in the original article (as seen with popular celebrities/events, for instance). we assess information overlap between the generated text and original article using rouge-l recall, favoring it over embedding based metrics because it emphasizes longest common subsequences based on exact word matches, which makes it suitable for measuring memorization. this is also preferable for content containing specialized terminology like pubmed abstracts and legal articles. figure 3 shows that gpt-3.5 and flan-t5-xl have higher rouge-l recall scores for news, suggesting that these models have had more exposure to news topics; this could explain the increased extrinsic error rate in news summaries. furthermore, in appendix a.7, we show similar trends using an alternative approach to measure domain representation by directly querying the pretraining corpus with article titles, and using the number of retrieved articles as a proxy for representation. are existing automatic systems for factual evaluation reliable across different domains? prior research has focused on creating automated metrics for evaluating factuality of generated summaries using question answering @xcite , natural language inference (nli; @xcite , dependency entailment @xcite , and classification methods @xcite . the performance of these metrics has been assessed almost exclusively on evaluation benchmarks comprising model-generated summaries annotated for factuality in the news domain @xcite @xcite @xcite @xcite . the effectiveness of such automated factuality metrics outside of news is underexplored. to address this, we use our annotated dataset to examine the performance of qafacteval @xcite , questeval @xcite and summac variations @xcite across all three domains. the results in ## conclusions we analyzed zero-shot summarization abilities of two llms, focusing on factuality. surprisingly, inaccuracies were more likely to be introduced in summaries of news articles compared to legal and biomedical domains. specifically, in this domain we observed more extrinsic errors-i.e., hallucinations of content not mentioned in the sourcewhereas errors in specialized domains were typically related to an apparent "misunderstanding" of concepts in the source. we hypothesize that the discrepancy could result from a higher proportion of news articles in the model's pretraining data, supported by preliminary evidence. additionally, we observed lower spearman rank correlations between automated metrics and human annotations in specialized domains compared to news articles, highlighting the necessity for manual evaluations or the development of new metrics for diverse benchmarks.