Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization (2022)

Abstract: Despite recent progress in abstractive summarization, systems still suffer from faithfulness errors. While prior work has proposed models that improve faithfulness, it is unclear whether the improvement comes from an increased level of extractiveness of the model outputs as one naive way to improve faithfulness is to make summarization models more extractive. In this work, we present a framework for evaluating the effective faithfulness of summarization systems, by generating a faithfulness-abstractiveness trade-off curve that serves as a control at different operating points on the abstractiveness spectrum. We then show that the Maximum Likelihood Estimation (MLE) baseline as well as recently proposed methods for improving faithfulness, fail to consistently improve over the control at the same level of abstractiveness. Finally, we learn a selector to identify the most faithful and abstractive summary for a given document, and show that this system can attain higher faithfulness scores in human evaluations while being more abstractive than the baseline system on two datasets. Moreover, we show that our system is able to achieve a better faithfulness-abstractiveness trade-off than the control at the same level of abstractiveness.

## introduction generating abstractive summaries of documents has been a long-standing goal of summarization. while there has been tremendous progress towards this goal @xcite @xcite , abstractive summarization systems still suffer from faithfulness errors @xcite , generating information that is not present in the original text. this has led to an increased research in faithfulness evaluation of summarization systems @xcite @xcite as well as methods to improve faithfulness of generated summaries @xcite . intuitively, one straightforward way of improving faithfulness of generated summaries is to copy a larger amount of content from the source article (i.e. more extraction). thus, any methods that increase the level of extractiveness, whether intentionally or not, would improve faithfulness. without reported extractiveness, it is unclear whether prior improvements mainly arise from increased extractiveness. we argue that in order to make progress in abstractive summarization, it is important to tease apart faithfulness improvements due to increased extractiveness versus improvements due to improved abstraction. in order to tease this apart, we develop a framework for evaluating progress in faithfulness, by considering the effective faithfulness, i.e. the improvement in faithfulness over a baseline system (control) operating at the same level of extractiveness. in particular, we split the training examples into different groups by the extractiveness of the summary, and train the control models on each group. each of these models corresponds to a specific tradeoff between abstractiveness and faithfulness, forming a trade-off curve indicating how much faithfulness can be improved solely by increasing extractiveness. systems that improve effective faithfulness should lie above this curve. using this framework, we show that the improved faithfulness of recently proposed methods comes mainly from an increased extractiveness. we then conduct further analysis to explore whether it is possible to have a system that can be both more abstractive and more faithful than the baseline system. we train a selector on a small set of humanannotated data that, given a set of output summaries with varying levels of extractiveness, picks the most abstractive output that is faithful to the source. our proposed system is both more abstractive and more faithful than the baseline. moreover, we show that we use coverage defined in @xcite to measure extractiveness, where summaries with higher coverage are more extractive. we observe that automated metrics of faithfulness are positively correlated with extractiveness. to summarize, our contributions are as follows: ## dataset we conduct our study on two english abstractive summarization datasets, one from the news domain, and one from a non-news domain. for the news domain dataset, we decided against using the popular cnn/dailymail dataset since its reference summaries tend to be very extractive @xcite , making it a poor choice for studying faithfulness in abstractive summarization. similarly, we also decided against using xsum, another popular news summarization dataset, since almost 77% of the gold reference summaries contain hallucinations @xcite . instead, we opted for gigaword and wikihow, which are datasets with substantial abstraction without as much hallucination problems as xsum. gigaword reference summaries have substantially less hallucinations than xsum @xcite , and wikihow summaries tend to be of a higher quality since they are written and curated by humans @xcite . wikihow @xcite is a dataset of how-to articles covering a diverse set of topics, collected from the wikihow.com website. each article contains several paragraphs detailing step by step instructions for a procedural task. there are about 12m such paragraphs in the dataset, paired with a one sentence summary. gigaword @xcite is a headline generation dataset that contains around 4m examples, extracted from news articles that were collected as part of the gigaword corpus @xcite . the model is tasked with generating the headline of the article given the first sentence. ## evaluating effective faithfulness given that extractiveness is confounded with faithfulness, we propose a framework for evaluating effective faithfulness, which takes into account the extractiveness of a system. in order to do this, we first need to determine the faithfulness of a system operating at a given level of extractiveness. we call this the faithfulness-abstractiveness tradeoff and we describe it further in §4.1. the effective faithfulness of a system is then simply the relative difference between the faithfulness score assigned to the system, and the score of a system operating with the same average extractiveness according to the trade-off curve. ## implications and limitations recent studies that propose methods to improve faithfulness evaluate progress by conducting human evaluation on generated summaries and check whether the faithfulness scores are higher for their proposed system as compared to their baselines. we show that there is a strong relationship between the extractiveness and faithfulness of generated outputs (i.e., more extractive outputs tend to be more faithful), and therefore we cannot simply disregard extractiveness in faithfulness evaluation. we propose that we should instead be measuring effective faithfulness and introduce a framework that takes into account the faithfulnessabstractiveness trade-off curve that is generated by training control models at different points in the abstractiveness spectrum. we demonstrate the importance of measuring effective faithfulness by showing that recently proposed methods that improve faithfulness over the baseline fails to consistently improve over a simple control operating at the same level of abstractiveness. we argue that measuring effective faithfulness is important since our goal is to build abstractive, faithful summarization systems. if the objective was to optimize for faithfulness alone, we could do so by simply building more extractive systems (such as the q4 model we trained above). limitations. note that this method relies on some diversity in the extractiveness of reference summaries, since we rely on sub-sampling to train models for the control. it is less likely to be effective for datasets with very little variation in the extractiveness of the generated summaries. however, in general, we see significantly more faithfulness problems for datasets with higher diversity of abstractiveness. therefore, we suggest to account for the faithfulness-abstractiveness trade-off for such datasets in future work. ## acknowledgements this work was partly supported by the office of the director of national intelligence (odni), intelligence advanced research projects activity (iarpa), via contract #fa8650-17-c-9117, samsung advanced institute of technology (next generation deep learning: from pattern recognition to ai), and a collaborative grant from amazon to the columbia center for artificial intelligence entitled "extremely abstractive summarization". the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the funding agencies. we further thank the anonymous reviewers and the stanford nlp group for their helpful feedback.