Understanding Factuality in Abstractive Summarization with FRANK : A Benchmark for Factuality Metrics (2021)

Abstract: Modern summarization models generate highly fluent but often factually unreliable outputs. This motivated a surge of metrics attempting to measure the factuality of automatically generated summaries. Due to the lack of common benchmarks, these metrics cannot be compared. Moreover, all these methods treat factuality as a binary concept and fail to provide deeper insights on the kinds of inconsistencies made by different systems. To address these limitations, we devise a typology of factual errors and use it to collect human annotations of generated summaries from state-of-the-art summarization systems for the CNN/DM and XSum datasets. Through these annotations we identify the proportion of different categories of factual errors and benchmark factuality metrics, showing their correlation with human judgement as well as their specific strengths and weaknesses.

## introduction factuality is defined as a measure of "whether eventualities are characterized as corresponding to facts, possibilities, or situations that do not hold in the world" @xcite . in summarization, this "world" is the article, which is taken as ground-truth, and the output summary must be faithful to the article's facts. despite advancements in neural abstractive summarization @xcite @xcite , ∼30% of summaries have factual inconsistencies @xcite . with summarization being an integral component of information consumption, this highlights a need for ensuring summarization systems are factually consistent and developing methods for evaluating them. common evaluation metrics for summarization based on n-gram overlap -bleu, rouge, and meteor @xcite @xcite -are insufficient to measure the factual correctness of summaries and fail to correlate with the human judgements of factuality @xcite . more recent metrics proposed to improve the evaluation of summarization factuality @xcite @xcite cannot be compared due to the lack of common benchmarks. more critically, while these approaches differ in the way they model factuality, they all consider factuality as a binary concept, labeling summaries of any length as factual or non-factual. they do not provide any finegrained understanding of the factual errors made by different systems that could serve as an actionable feedback on a system's limitations. the binary factuality of a text can be difficult to determine. @xcite show relatively low crowd-expert agreement, indicating the presence of subjectivity in the annotation process. moreover, not all factual errors are equally important and the number of errors can have a significant impact on the perceived factuality of a text. this suggests that non-factuality should be modeled as a multidimensional construct and not a label. in this work, we propose a linguistically motivated typology of factual errors for fine-grained analysis of factuality in summarization systems ( §2). our typology is theoretically grounded in frame semantics @xcite and linguistic discourse theory @xcite . it provides several benefits. first, we find that decomposing the concept of factuality in (relatively) well-defined and grounded categories makes the final binary decision more objective leading to near perfect agreement between crowd and expert annotators (κ = 0.86). second, this approach provides some measure of the degree of non-factuality both in terms of the quantity and the category of factual violations that appear figure 1 : we propose a linguistically grounded typology of factual errors. we select crowd workers to annotate summaries from two datasets according to this typology achieving near perfect agreement with experts. we collect frank, the resulting dataset, to benchmark factuality metrics and state-of-art summarization systems. in the text. this typology also provides us with the means to categorize the types of errors made by summarization systems, helping us gain deeper insights than simply categorizing content as factual or hallucinated. we define an annotation protocol of factuality based on our typology and collect a dataset of human judgements over a diverse set of model generated summaries on the cnn/dm @xcite and xsum @xcite datasets ( §3). through this dataset, we aim to both assess the factuality of summarization systems and benchmark recently proposed factuality metrics. in §4 we discuss various state-of-art models and show a detailed analysis of the factual errors they make. finally, in §5 we evaluate multiple summarization metrics against our benchmark and show their strengths and weaknesses in detecting specific types of factual errors. figure 1 shows an overview of this work. ## typology of factual errors previous studies of factuality in summarization only distinguish factual and hallucinated content @xcite and provide limited insights on the fine-grained types of factual errors. in the simplest case, factual errors appear within a single proposition. however, as summaries include several sentences, discourse markers describe relations across propositions. these cross-sentence links, such as causality or temporal ordering, can introduce inconsistencies with the article. furthermore, information in the summary should be verifiable given the article. this understanding outlines different levels of linguistic structure where factual mistakes can arise in summaries: at the semantic frame level, at the discourse level, or because the content cannot be verified. below we define a typology of factual errors further detailing these three levels. this typology is theoretically grounded in frame semantics @xcite @xcite and linguistic discourse analysis @xcite . examples for each category are shown in ## dataset creation beyond theoretical grounding, we empirically verify our typology through large scale human annota-tions of five abstractive summarization models on the cnn/dm dataset and four on the xsum dataset. through our dataset, we aim to have a broad coverage of different types of errors made by neural summarization systems, with human judgements on their fine-grained factuality errors. annotation data for the annotation, we include model summaries from cnn/dm and xsum datasets as they present different characteristics. cnn/dm summaries are longer, with three sentences on average, while xsum has only single sentence summaries. having longer summaries is crucial to identify discourse level errors. on the other hand, xsum summaries are more abstractive and include more factual errors on average @xcite . for a diverse set of model summaries, we collect publicly available model outputs from different summarization models with differing factuality capabilities. for the cnn/dm dataset, we use model outputs from a lstm seq-to-seq model (s2s) @xcite , a pointer-generator network (pgn) model @xcite , a bottom-up summarization (bus) model @xcite , a bert based extractive-abstractive model (bertsum) @xcite and a jointly pretrained transformer based encoder-decoder model bart @xcite . for the xsum dataset, we collect model outputs from a topic-aware cnn model @xcite , a pointer-generator network (pgn) model, a randomly initialized (transs2s) @xcite and one initialized with bert-base (berts2s) @xcite . 2 details of the models used are provided in §a.1. annotation collection using the above model generated summaries, we collect human annotations from three independent annotators for 250 articles from each dataset (with a total of 1250 model outputs on cnn/dm and 1000 on xsum). we annotate each sentence of a summary to break the judgement of factuality into smaller units. we present sentences in the context of the entire summary to identify discourse errors spanning multiple sentences. annotations are a two step process: for each sentence in the summary, the annotator first selects whether the sentence is factual, and if marked not factual, identifies the category of each error based on our typology. 3 a sentence can be annotated with more than one category of errors to account for multiple errors within a sentence. we conduct the annotation task on the amazon mechanical turk (mturk) platform. to achieve high quality crowd-sourced annotations, we build an intuitive interface foot_3 which combines: inter-annotator agreement: we report interannotator agreement in terms of fleiss kappa κ @xcite . following @xcite , we report the percentage p of annotators that agree with the majority class. each datapoint in our dataset corresponds to a sentence in a summary. we compute agreement on all 4942 annotated sentences. on the annotation of whether a sentence is factual or not we obtain κ = 0.58, with @xmath0. as a comparison, @xcite reports @xmath1.7% average agreement. when all three annotators agree that a sentence is not factual, we obtain κ = 0.39 with @xmath2.9% of annotators agreeing with the majority class on the eight category annotation (seven categories of errors and "other") which indicate a moderate agreement. agreement with domain expert: we measure agreement between the majority class of the three 1 . ## summarization model analysis we evaluate the performance of different summarization models in terms of factuality. figure 2 visualizes the percentage of summaries with factual errors for each category model and dataset, with a breakdown of proportion of different error types within each. a summary is considered incorrect if it contains at least one sentence with a factual error. a sentence contains a factual error if the majority of annotators indicate the presence of an error (here we do not consider annotations where all three annotators disagree on the category). how factual are generated summaries across different datasets? from our annotations, we observe that 60% of the summaries that were annotated contain at least one factual error. from figure 2 , we see that the xsum dataset has more factually incorrect model summaries (92%) than cnn/dm (43%). it poses more significant challenges in terms of factuality as all models @xmath3, with the best model (berts2s) producing 83% wrong summaries. on the cnn/dm dataset, while state-of-the-art pretrained models like bertsum and bart have better factuality numbers, the percentage of factually incorrect summaries is still high (23% for bertsum and 27% for bart). the proportion of errors across different categories vary widely between the two datasets. for the cnn/dm dataset, the most frequent classes of errors are entity error (ente) and coreference error (corefe). for the xsum dataset they are out of article error (oute) and entity error (ente). note that there are no discourse errors (corefe, linke) in the xsum dataset because the data only contains single sentence summaries. additionally, we observe that othe makes up a very small percentage (∼ 1%) of errors overall showing that our typology is complete with most errors being mapped to one of our existing categories. how factual are generated summaries across different models? from figure 2 , we observe that lstm based models like s2s and bus generate many incorrect summaries. interestingly, pgn on cnn/dm has fewer summaries with factual errors (26%) compared to s2s (74%) and bus (62%) potentially due to the extractive nature of cnn/dm and the copy based objective in pgn. pgn has been previously shown to produce highly extractive summaries on cnn/dm copying large portions of text (often entire sentences) @xcite . on the more abstractive dataset xsum, pgn @xmath4. we also observe that large-scale pretrained models improve factuality on both datasets, as also noted by @xcite , with more significant gains on cnn/dm. on cnn/dm, bertsum and bart display half the error rate of bus. in contrast, on xsum, berts2s improves over non-pretrained models by ∼ 10% only, showing that xsum poses a significant challenge for factuality even in pretrained models. different models also exhibit different distributions in the error categories. lstm based mod-els have higher proportion of grammatical errors (grame) while transformer and cnn based models have a lower proportion. for pretrained transformer models, we observe that the improved error-rate on the cnn/dm dataset can be attributed to improvements at the frame level (prede, ente, circe) while the discourse level errors still remain a challenge. errors corefe, linke account for a higher proportion of errors in bertsum and bart compared to the other models. ## factuality metric evaluation we propose the frank dataset resulting from the human annotation study as a common benchmark to assess different factuality metrics. we provide an evaluation protocol of factuality metrics, which controls for dataset biases, and a fine grained analysis of the strengths of each metric. ## conclusion in this work we provide a linguistically grounded typology of factual errors which we use to collect frank, a dataset of human annotations of 2250 summaries covering both cnn/dm and xsum datasets. we use frank to assess the factuality of summarization systems and benchmark recently proposed factuality metrics highlighting the types of errors they can capture. with the frank benchmark we have started moving away from a summary-level binary understanding of factuality.