Improving Consistency for Text Summarization with Energy Functions (2023)

Abstract: Current abstractive summarization models often generate inconsistent content, i.e. texts that are not directly inferable from the source document, are not consistent with respect to world knowledge, or are self-contradictory. These inconsistencies motivate a new consistency taxonomy that we define as faithfulness, factuality, and self-supportiveness. However, most recent work on reducing inconsistency in document summarization only focuses on faithfulness detection and correction while ignoring other inconsistency phenomena, which limits the model’s scalability. To improve the general consistency we introduce EnergySum, where we apply the Residual Energy-based Model by designing energy scorers that reflect each type of consistency. These energy scores are utilized in candidate re-ranking during the sampling process. Experiments on XSUM and CNN/DM datasets show that EnergySum mitigates the trade-off between accuracy and consistency.

## introduction while performing well in terms of overlapbased metrics like rouge @xcite and bertscore @xcite , current abstractive summarization methods often generate inconsistent content due to the inherently noisy dataset and the discrepancy between maximum likelihood estimation based training objectives and consistency measurements. inconsistency content in abstractive summarization has different interpretations, including text that is not directly inferable from the source document, is not factual with respect to world knowledge and commonsense, or is self-contradictory. we formalize the categorization of consistency into faithfulness, factuality, and self-supportiveness. most previous methods improve consistency in document summarization by filtering out noisy training samples @xcite , applying contrastive learning @xcite , post-editing @xcite , etc., with a limited scope of consistency to faithfulness. however, addressing inconsistency solely in terms of faithfulness is inadequate. unlike extractive methods, abstractive summarization introduces new content into the summary that is not directly copied from the source document and is not necessarily irrelevant. hence, detecting and alleviating inconsistency calls for the introduction of a larger reference corpus alongside the source document. factuality compares the generated content against world knowledge, while self-supportiveness verifies whether the generated sentence is consistent with its preceding one. in addition, consistency is measured on the entire prediction sequence while existing summarization objectives evaluate conditional distributions for individual tokens and lack global control over predictions. these motivate us to apply the residual energy-based model (rebm) @xcite framework to document summarization, which jointly trains a summarizer and a discriminator that learns to assign high scores to consistent summaries and low scores to inconsistent ones. the advantage of the energy-based methods @xcite is that they score the entire input simultaneously and avoid local normalization traps, offering a natural solution to address this issue. therefore, we introduce energysum that adapts the rebm framework for improving consistency. we design the energy functions that reflect each type of consistency and are agnostic to summarization model instances. we propose joint inference where energy scorers cooperate with decoding searching strategies in the candidate re-ranking step. in summary, our contributions include: correction-based methods are proposed for mitigating the trade-off between consistency improvement and rouge-based accuracy measurement decrease. @xcite propose a post-editing corrector module trained on synthetic examples, where heuristic transformations are inspired by an error analysis on reference summaries. span-fact @xcite is a factual correction model that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. @xcite propose a fact-aware summarization model to integrate factual relations into the summary generation process and a factual corrector model in the form of a finetuned denoising auto-encoder. automatic consistency evaluation models can be roughly classified into entailment-based and qabased methods. entailment-based metrics @xcite @xcite train classification models to predict if the summary is entailed by the source document. meanwhile, qa-based metrics @xcite @xcite ## method in the proposed energysum framework, we design energy scorers that correlate each type of consistency and integrate energy scores in candidate reranking during sampling. ## conclusion we propose to apply the residual ebm framework with energy scorers and joint inference to improve consistency in document summarization. experiments on xsum and cnn/dm datasets show that energysum mitigates the trade-off between accuracy and consistency. direct extensions of this work include proposing more fine-grained data augmentation strategies and investigating the relation between prediction certainty and energy scores.