CLIFF : Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021)

Abstract: We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them. We further design four types of strategies for creating negative samples, to resemble errors made commonly by two state-of-the-art models, BART and PEGASUS, found in our new human annotations of summary errors. Experiments on XSum and CNN/Daily Mail show that our contrastive learning framework is robust across datasets and models. It consistently produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training, according to QA-based factuality evaluation. Human judges echo the observation and find that our model summaries correct more errors.

## introduction large pre-trained transformers have yielded remarkable performance on abstractive summarization @xcite @xcite with impeccable fluency, yet their summaries often contain factually inconsistent content @xcite @xcite , even for stateof-the-art models. three types of remedies have been proposed: running a separately learned error correction component @xcite , removing noisy training samples @xcite , and modifying the transformer architecture @xcite ). yet they either rely on heuristically created data for error handling, falling short of generalization, or require learning a large number of new parameters, and summary informativeness is often sacrificed. ## cliff: contrastive learning framework for summarization we design a contrastive learning (cl)-based training objective that drives the summarization model to learn a preference of faithful summaries over summaries with factual errors. it is then used for fine-tuning bart @xcite and pega-sus @xcite for training summarization models. formally, let an article x have a set of reference summaries p (henceforth positive samples) and another set of erroneous summaries n (negative samples). the contrastive learning objective is @xcite : to effectively employ cl in summarization, we need to address two challenges: (1) how to automatically construct both positive and negative samples, which are critical for cl efficacy @xcite , and (2) how to represent the summaries (i.e., h * ). below we describe positive sample generation and options for h * , leaving the strategies for negative samples to § 5. ## summary error annotation and model behavior analysis we first describe annotating unfaithfulness errors by state-of-the-arts, i.e., models fine-tuned from bart and pegasus on xsum and cnn/dm. we then probe into the model generation behavior that is indicative of errors, which guides the design of negative sample construction strategies. 600 (150 × 2 × 2) summaries are annotated to demonstrate how often do the models "hallucinate", i.e., generating content not grounded by the source. to characterize errors, we annotate text spans in summaries with (i) intrinsic errors caused by misconstructing phrases or clauses from the source; and (ii) extrinsic errors which include words not in the source that are either unverifiable or cannot be verified by wikipedia. content not covered by the article but can be validated by wikipedia is annotated as world knowledge, and the models' behavior pattern when generating them differs from when they generate errors. two fluent english speakers with extensive experience in summary evaluation and error labeling are hired. for each sample, they are shown the article and two system summaries, and instructed to annotate text spans with the aforementioned errors and world knowledge. after labeling every 50 samples, the annotators discuss and resolve any disagreement. the fleiss's kappas on xsum and cnn/dm are 0.35 and 0.45. error statistics are displayed in fig. 2 . extrinsic errors dominate both datasets, especially on xsum. 58.7% of summaries by bart (and 44.0% by pe-gasus) contain at least one extrinsic error. noticeably, pegasus is a newer model pre-trained with a larger amount of data, thus contains less errors than bart and other older models studied for error annotations by @xcite . this observation also highlights the usage of our annotations for future development and evaluation of summarization models. low confidence generation is indicative of extrinsic errors. inspired by recent work that studies model prediction confidence @xcite , we examine generation probabilities for tokens of different part-of-speech (pos) tags. fig. 3 shows salient results on the generation probabilities of the first token of a proper noun or a number (with additional analysis provided in appendix a). as observed, model confidence tends to be lower for the first tokens of proper nouns and numbers if they are part of spans with extrinsic errors. also note that world knowledge, which cannot be inferred from the source either, often has higher generation probability than extrinsic errors. take this snippet generated by a fine-tuned bart as an example: "manchester united captain wayne rooney's testimonial game against manchester city. . .". "manchester city" is an extrinsic error and "wayne" is produced as world knowledge. the model assigns a low probability of 0.10 to the first token of "manchester city" and a high probability of 0.92 to token "wayne". this implies that model confidence can be a useful indicator for negative sample collection.