Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization (2023)

Abstract: Neural models for abstractive summarization tend to generate output that is fluent and well-formed but lacks semantic faithfulness, or factuality, with respect to the input documents. In this paper, we analyze the tradeoff between abstractiveness and factuality of generated summaries across multiple datasets and models, using extensive human evaluations of factuality. In our analysis, we visualize the rates of change in factuality as we gradually increase abstractiveness using a decoding constraint, and we observe that, while increased abstractiveness generally leads to a drop in factuality, the rate of factuality decay depends on factors such as the data that the system was trained on. We introduce two datasets with human factuality judgements; one containing 10.2k generated summaries with systematically varied degrees of abstractiveness; the other containing 4.2k summaries from five different summarization models. We propose new factuality metrics that adjust for the degree of abstractiveness, and we use them to compare the abstractiveness-adjusted factuality of previous summarization works, providing baselines for future work.

## introduction summarization is the task of generating a semantically faithful, well-formed and concise text representation of the input. automatically generated summaries have traditionally been extractive @xcite @xcite @xcite , leading to issues with readability and coherence, as different extracted fragments may not fit well when taken out of their original contexts @xcite . researchers have also invested in methods for abstractive summarization, aiming to paraphrase the input documents' main points figure 1 : three successively more abstractive summaries generated from the same input article, with mint abstractiveness scores (section 2.1) of 46.1%, 67.2%, 79.5%. fragments extracted from the input are marked from red (longer fragments) to yellow (shorter fragments). the bottom summary has factual errors. without borrowing their exact lexical expressions @xcite @xcite @xcite @xcite . abstractive summaries generated by today's neural models tend to be fluent and well-formed, but lack semantic faithfulness @xcite . observed rates of factual errors in abstractive summaries have ranged from 30% to over 75% @xcite . the research community is developing automatic factuality metrics @xcite @xcite @xcite and methods that attempt to increase factuality @xcite @xcite @xcite . however, the factuality problem of abstractive summaries cannot be well understood without considering the degree of abstractiveness of a given summary: any summary is on a spectrum between extractive and abstractive @xcite . summaries that are extractive to a larger extent tend to be more factual since copying text from the input into the summary rarely introduces factual errors while the task of paraphrasing, which results in summaries that are more abstractive, is harder and prone to semantic errors. as an example, figure 1 shows part of a washington post article and three summaries with increasing abstractiveness, which we have generated using our abstractiveness constraints (section 2.2). the first two summaries are correct, but the third, most abstractive, summary has factual errors, misinterpreting the input. few authors have discussed this connection explicitly. @xcite observe that abstractive summaries consisting of concatenated extracted fragments tend to be more factual than those created by more complex fusion. @xcite observe that models trained on the more extractive cnn/dm dataset @xcite create more factual summaries than models trained on the more abstractive xsum dataset @xcite . we show that such models differ in factuality even when we bias them to generate summaries that have similar levels of abstractiveness. our analysis (section 4) situates summarization models on the spectrum outlined in figure 2 , where factual summaries range from "trivially factual" (extractive) to truly "paraphrasing" (abstractive). we make the following contributions: ## factuality we now describe metrics for factuality, before we can describe the relationship between abstractiveness and factuality (section 4). by factuality of a summary y, we mean factual consistency with the input x, rather than objective factuality or universal truth. measuring factuality automatically is an active area of research @xcite . factuality is most naturally measured by human annotators; we describe our setup for human factuality annotation first, then move to automatic metrics. ## abstractiveness-factuality tradeoff the metrics for factuality and abstractiveness along with the abstractiveness constraints allow us to systematically explore the relationship between abstractiveness and factuality. we can control abstractiveness and observe the effect on factuality, i.e., we can vary the amount of lexical overlap between input and generated summary and observe the extent to which the summary preserves the input semantics. ## conclusions we presented new metrics and datasets for evaluating the relationship of abstractiveness and factuality. as part of our analysis, we presented abstractiveness constraints, which can bias a summarization model to increase or decrease the level of abstractiveness while generating summaries, using nonlinear penalties or rewards based on the length of summary fragments extracted from the source. through automatic and human factuality evaluations, including 10.2k human factuality judgements of summaries with systematically varied abstractiveness, we shed light on how abstractiveness interacts with factuality, across multiple datasets and models. we proposed new metrics to measure the tradeoff, including f@50 and mint-adjusted factuality rates, such as µdae and µfactcc, and we established baselines for future research.