Multi-View Source Ablation for Faithful Summarization (2023)

Abstract: In this paper, we present MuFaSSa (Multi-view Faithfulness Scoring via Source Ablation), a metric for evaluating faithfulness of abstractive summaries, and for guiding training of more faithful summarizers. For evaluation, MuFaSSa employs different strategies (e.g., masking entity mentions) to first remove information from the source document to form multiple ablated views. Then, the faithfulness level of each token in a generated summary is measured by the difference between the token generation probabilities when given the original document and the ablated document as inputs to trained summarizers. For training, MuFaSSa uses a novel word truncation objective that drops unfaithful tokens located by MuFaSSa in both the decoder input and output. Alignments with human-annotated faithfulness labels on AggreFact show that MuFaSSa is comparable to or better than existing metrics built on classifiers or QA models pre-trained on other tasks. In experiments on summarization with XSum and CNN/DailyMail, models trained with word truncation using MuFaSSa outperform competitive methods according to both automatic faithfulness metrics and human assessments.

## introduction automatic text summarization systems have made great strides with the use of large pre-trained models, which enable more precise identification of salient content in the document and generation of summaries with human-level fluency @xcite . however, modelgenerated summaries frequently contain unfaithful information that either contradict the source text or cannot be verified @xcite , creating risks in real-world deployment of automatic text summarization models and motivating the development of models targeting more faithful summaries @xcite . figure 1 : mufassa estimates the faithfulness level of each summary token as the difference between probabilities given by trained summarizers with the original source document and the ablated document chosen for the token (e.g., shuffling tokens for verbs). the large difference for "make" indicates it is faithful to the document, while the small difference for "kezia" indicate an unfaithful token. as overlap-based metrics such as rouge @xcite struggle to reflect the faithfulness level of generated summaries @xcite , a number of model-based faithfulness metrics have been introduced. these metrics leverage external textual entailment @xcite and question answering models @xcite to measure the degree to which claims in the summary align to information in the source text. yet, there remains substantial room for improvement @xcite . moreover, despite being relevance or complementary to each other, for building faithful summarization systems, faithfulness metrics are rarely exploited and researchers mainly resort to more complex training routines @xcite or model architectures @xcite . to this end, our work introduces a faithfulness metric that (1) more accurately estimates summary faithfulness levels; and (2) can be easily integrated into training objectives to produce more faithful summarization systems. in our method, which we call mufassa (multi-view faithfulness scoring via source ablation), multiple ablated source documents are constructed by masking entities, shuffling tokens, and discarding all tokens in the original source document, to remove crucial information for the generation of different content in the summary, as shown at the top of figure 1 . since the ablated sources do not include sufficient information for generating the corresponding summary, the differences between token output probabilities given by the original input and each ablated input should be high for the faithful content and low for the unfaithful one (e.g., "make" and "kezia" in figure 1 ). we then aggregate the differences to obtain the summary faithfulness score. additionally, to train faithful summarization systems, we adapt loss truncation @xcite and nullify losses on summary tokens that are deemed less faithful by mufassa during training. compared to using training losses for detecting unfaithful content in the original loss truncation, mufassa provides a more accurate estimation of token faithfulness in training samples, and more faithful summaries can therefore be produced while maintaining informativeness. we further design word truncation, to drop the generation dependency on less faithful words in the auto-regressive decoder by completely removing them from the decoder input. 1 two sets of experiments are conducted to show the effectiveness of mufassa at evaluating and training faithful summarizers. first, we compare with existing faithfulness metrics on ag-grefact @xcite , a curated benchmark for meta evaluation of faithfulness metrics, where mufassa obtains the best average performance. we then leverage mufassa during model training on xsum @xcite and @xcite . com-pared to baselines and recent models built with augmented data or more complex training objectives, mufassa-trained models produce summaries with competitive or better faithfulness while maintaining the coverage of salient document information, according to both automatic faithfulness metrics and human judgments. in summary, we make the following contributions: in this work we pursue an alternative approach that detects unfaithful outputs by analyzing differences in token probabilities when conditioning on different views of the source document. this approach was first proposed by @xcite , whose coco metric measures probability differences on pre-specified sets of key terms in the output of models conditioned on partially masked sequences. our proposed metric, mufassa, builds upon coco in two ways. first, we eschew the need for key terms, instead providing an approach for assessing faithfulness of different types of tokens (e.g., entities, relations). this not only makes mufassa easier to use, but, as we will see in section 4, also results in better performance. secondly, we introduce a strategy for incorporating these metrics into training, and demonstrate in section 5.2 that this training scheme produces more faithful summarizers. ## mufassa: multi-view information ablation in this section, we first introduce the formulation of faithfulness estimation by mufassa ( §3.1) and the construction of inputs with different information ablated ( §3.2). we then describe how mufassa can be incorporated into model training through loss truncation and our proposed word truncation ( §3.3). we fine-tune bart @xcite for all summarization models in this paper. ## metric experiments we first test how well mufassa agrees with human judgments on summary faithfulness. ## summarization experiments to verify the effectiveness of our methods to produce more faithful summaries, we train summarizers on popular summarization datasets with our proposed loss truncation and word truncation methods equipped with mufassa. ## additional experiments in this section, we inspect the effects of important design choices in mufassa ( §6.1). furthermore, to show the possibility of applying mufassa to other tasks, we train data-to-text generation models with our proposed methods ( §6.2). ## conclusion we studied improving faithful summary evaluation and generation. our proposed method, mufassa, estimates the faithfulness level of a summary token by the decrease in its generation probability after ablating crucial information from the source document. multiple ablation strategies are used by mufassa for different summary tokens to achieve accurate faithfulness estimation. we also designed word truncation for improved integration of mufassa into model training. experiments on aggrefact show that mufassa better aligns with human faithfulness labels than existing metrics. when used for highlighting less faithful tokens during summarizer training, mufassa leads to summaries with enhanced faithfulness, which is further boosted by word truncation, achieving better faithfulness than competitive comparisons, as measured by both automatic metrics and human annotators.