Improving Factual Consistency of News Summarization by Contrastive Preference Optimization (2024)

Abstract: Despite the recent progress in news summarization made by large language models (LLMs), they often generate summaries that are factually inconsistent with original articles, known as “hallucinations” in text generation. Unlike previous small models (e.g., BART, T5), current LLMs make fewer silly mistakes but more sophisticated ones, such as imposing cause and effect, adding false details, overgeneralizing, etc. These hallucinations are challenging to detect through traditional methods, which poses great challenges for improving the factual consistency of text summarization. In this paper, we propose Contrastive Preference Optimization (CPO) to disentangle the LLMs’ propensities to generate faithful and fake content. Furthermore, we adopt a probing-based specific training method to improve their capacity of distinguishing two types of propensities. In this way, LLMs can execute the instructions more accurately and have enhanced perception of hallucinations. Experimental results show that CPO significantly improves the reliability of summarization based on LLMs.

## introduction although recent pre-trained language models have significantly boosted the performance of abstractive summarization @xcite @xcite , the hallucination problem -that models generate summaries that are factually inconsistent with the source text -remains difficult to resolve. as figure 1 shows, we expect the model to generate reliable summaries (understand the source text and only generate the faithful content). still, it often hallucinates and over-imagines, which means the model outputs tony's body was found by his son in coningswath road, on 9 january. jaydon, nance, marcus, and vincent were charged with murder. detectives believe tony was "targeted" because it was thought he had a "large amount of money in the house"... write a summary consistent with the above article. be faithful to the article add false details take speculation as fact four people have been charged with murdering a man. tony, a 58-year-old father-of-three, was found dead in coningswath road on 9 january. detectives believe tony was targeted due to having a large amount of money. figure 1 : the diagram of the llms' propensities to generate faithful and fake content. in abstractive summarization, the model is supposed to generate a factually consistent summary with the preference to be faithful to the context. however, it often hallucinates with the preference to over-imagine with internal knowledge. fake content without supporting evidence in the original article. worse still, the generation usually involves both types of content, making the summaries so full of half-truths that the hallucinations are much more hidden. early methods for improving factual consistency use post-processing models @xcite , which correct summaries with hallucinations, but they rely on external resources to obtain the error correction capability. @xcite introduces human revisions to achieve better performance, but data collection is still difficult and costly. besides, these two-stage methods have a complicated structure, consisting of summary generation and correction models. considering that, some studies try to solve hallucinations holistically during the pre-training stage @xcite . they design a new pre-training objective with sentence selection strategies, encouraging the model to generate a faithful summary. however, pre-training requires enormous computational resources, especially for large language models (llms). moreover, some methods adopt contrastive learn-ing @xcite in fine-tuning to teach the model to distinguish between true and false more clearly. to construct negative samples, they modify the references by entity swapping and masking-and-filling. unfortunately, these autogenerated negative samples are inconsistent with the distribution of errors made by llms in real scenarios. @xcite point that instructiontuned models have much stronger summarization abilities than previous fine-tuned ones. current llms make fewer silly mistakes (e.g., entity confusion, irrelevant information generation) but more sophisticated ones @xcite . for example, they fill in the details related to but not directly supported by the source text. sometimes, they rewrite original sentences by imposing cause and effect or taking speculation as fact. these mistakes are difficult to mimic by traditional perturbation-based approaches @xcite . with the rapid development of llms, designing prompts based on the chain of thoughts (cot) @xcite attracts scholarly attention. the models are posed with several questions about the critical content in the source text before final summarization, serving as contextual clues to guide models to generate factually consistent summaries. nevertheless, these methods are sensitive to the domain because they do not fundamentally improve the llms' reliability. inspired by preference optimization, many methods use reinforcement learning @xcite with entailment feedback (rlef) to ameliorate hallucination problems. as figure 2 shows, ppo-based methods @xcite (proximal policy optimization) train a natural language inference (nli) model for consistency detection and then regard it as the reward model in reinforcement learning. however, it is challenging for hallucinations generated by llms to be detected through traditional nli methods. therefore, the performance of these reward models constrains the training of summarization models. on the other hand, dpobased methods @xcite require paired data with preference annotation, which is difficult to construct. otherwise, reinforcement learning is usually unstable, and rewards are easily over-optimized @xcite . the problems mentioned above motivated us to propose contrastive preference optimization (cpo) which disentangles the llms' propensities 𝐴 1 𝐴 3 𝐴 2 𝑆 1 𝑆 3 𝑆 2 0.8 0.6 0.3 ppo 𝐴 1 𝐴 3 𝐴 2 𝑆 1 𝐶 𝑆 3 𝐶 𝑆 2 𝐼 𝑆 1 𝐼 𝑆 3 𝐼 𝑆 2 𝐶 𝐼 𝐼 𝐼 𝐼 𝐼 𝐼 𝑆 1 𝐶 𝑆 3 𝐶 𝑆 2 𝐼 𝑆 1 𝐼 𝑆 3 𝐼 𝑆 2 𝐶 > < > 𝐴 1 𝐴 3 𝐴 2 𝐼 𝐼 𝐼 𝑆 1 𝐶 𝑆 3 𝐶 𝑆 2 𝐼 𝐴 1 𝐴 3 𝐴 2 𝑆 1 𝐶 𝑆 3 𝐶 𝑆 2 𝐼 𝐼 𝐶 𝐼 𝐼 𝐼 𝐶 dpo cpo difficult to construct poor performance fine-tuned aligned aligned fine-tuned aligned fine-tuned 0.2 0.4 0.7 to generate faithful and fake content. furthermore, we dynamically probe for the model's distinguishing capacity for consistency and inconsistency and train the target layers in an sft mode, getting rid of the reliance on reinforcement learning (rl) frameworks. this work makes three main contributions: ## methodology we next describe our contrastive preference optimization (cpo) with probing-based specific training method. the whole methodology can be divided into three parts: (1) sentence-level data collection, obtained by collecting summaries from the most common llms and designing an appropriate prompt to get accurate automatic annotation based on chatgpt and gpt-4, (2) contrastive preference optimization, where we encourage llms to generate with different propensities according to different instructions and adopt adversarial training to enhance llms' perception of their capabilities, and (3) probing-based specific training, where we dynamically probe for the awareness of hallucinations and train the vulnerable modules to make up for the deficiency. ## experiments we conduct extensive experiments to verify the effectiveness of our proposed model decent and analyze it using ablation studies, case studies, and visualization results. in this section, we attempt to answer the following research questions: rq1: does cpo improve llm's summarization factual consistency? rq2: does cpo decouple llms' different propensities successfully? rq3: does probing-based specific training fill the gaps? rq4: is cpo better than other training strategies? ## conclusion this paper points out the problems of applying previous methods for summarization factual consistency to llms. we construct a summarization dataset (lesson) and propose contrastive preference optimization with probing-based specific training to improve factual consistency. the experimental results demonstrate the effectiveness of our method on the most common llms. we expect our work will direct more scholarly attention to constructing new datasets and enhancing factual consistency from the perspective of llms.