Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models (2024)

Abstract: Abstractive summarization models often generate factually inconsistent content particularly when the parametric knowledge of the model conflicts with the knowledge in the input document. In this paper, we analyze the robustness of fine-tuning based summarization models to the knowledge conflict, which we call factual adaptiveness. We utilize pre-trained language models to construct evaluation sets and find that factual adaptiveness is not strongly correlated with factual consistency on original datasets. Furthermore, we introduce a controllable counterfactual data augmentation method where the degree of knowledge conflict within the augmented data can be adjustable. Our experimental results on two pre-trained language models (PEGASUS and BART) and two fine-tuning datasets (XSum and CNN/DailyMail) demonstrate that our method enhances factual adaptiveness while achieving factual consistency on original datasets on par with the contrastive learning baseline.

## introduction factual consistency is a crucial aspect, especially in abstractive summarization, ensuring that the facts presented in the generated summary align with those in the input document @xcite @xcite @xcite . recent summarization models using pre-training and/or fine-tuning of the language model have shown excellent performance in various aspects such as factual consistency @xcite @xcite @xcite . there are also studies on large language models @xcite @xcite for the summarization (zhang * corresponding author @xcite or the evaluation of summaries @xcite . previous works have also reported that (large) language models have parametric knowledge @xcite . the parametric knowledge of the language model is known to result in hallucinated contents, particularly when knowledge conflict occurs which refers to the mismatch between the knowledge in the document and the parametric knowledge of the model @xcite @xcite . because the hallucination problem degrades factual consistency of summarization models @xcite , it is important to study the robustness to knowledge conflict of summarization models. in abstractive summarization, most previous works have measured the factual consistency using the original document only, which is not sufficient to evaluate the robustness to knowledge conflict @xcite @xcite . there are studies on hallucination problems caused by knowledge conflict in abstractive summarization @xcite . however, the aforementioned document perturbation strategies do not control the degree of knowledge conflict, which offers valuable insight into the robustness of the summarization models to the knowledge conflict. in this paper, we define factual adaptiveness, the robustness to the knowledge conflict, of fine-tuning based abstractive summarization models. we focus on entity-level knowledge conflict and factual adaptiveness and use counterfactual samples obtained by replacing a single named entity (i.e., original entity) with another named entity (i.e., counterfactual entity). unlike previous works on knowledge conflict in question answering, there are two additional considerations in our work @xcite . first, we determine which named entity to replace in the reference summary by detecting parametric knowledge. second, we select the named entity to be replaced with to control knowledge conflict. to address those considerations, we utilize the parametric knowledge of the pre-trained language model (plm) during the knowledge conflict set construction. we first analyze the factual adaptiveness of various methods for improving factual consistency on original datasets such as data filtering @xcite , contrastive learning (cao and wang, 2021), and advanced decoding @xcite . our results demonstrate that methods for factual consistency on original datasets do not always effectively mitigate knowledge conflict problems, which indicates that factual consistency on original datasets can be orthogonal to factual adaptiveness. we next propose a controllable counterfactual data augmentation technique. specifically, the method constructs counterfactual samples based on a pre-defined degree of knowledge conflict. experimental results show that our method improves factual adaptiveness effectively and addresses the entity-level hallucination problem caused by knowledge conflict. our contributions can be summarized as follows: ## factual adaptiveness in this section, we define and analyze factual adaptiveness of various fine-tuning based summarization models which are known to improve factual consistency. we formulate factual adaptiveness in section 2.1, and explain the factual adaptiveness evaluation set construction method in section 2.2. in the remaining text, the term counterfactual indicates the presence of knowledge conflict caused by the entity replacement. we also denote a counterfactual sample as a pair of the counterfactual document and summary, assuming they are factually consistent. ## analysis on models for improving factual consistency in this section, we analyze summarization models using the evaluation set as described in section 2.2. specifically, we measure m cl and m f c of various models that are proposed to improve factual consistency to observe the relation between factual adaptiveness and factual consistency. ## analysis 6.1 entity replacement the proportion of summaries that contain the counterfactual entity without the original entity given the counterfactual document is shown in figure 2 . we can observe that our model exhibits a significantly high rate of generating counterfactual entities in both datasets. filtering exhibits relatively higher values among the baselines, which is consistent with the results in ## conclusion in this study, we analyze the factual adaptiveness of the fine-tuning based summarization models. we propose two complementary metrics of factual adaptiveness and elucidate the relationship between factual consistency and factual adaptiveness. we then propose a controllable counterfactual data augmentation method and observe that our method mitigates hallucination problems due to knowledge conflict. our experimental results show that our method effectively alleviates entitylevel hallucination problems, especially when a knowledge conflict occurs. we anticipate that our work will contribute to improving the faithfulness of summarization models that contain parametric knowledge.