CONFIT : Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2022)

Abstract: Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. Although significant progress has been achieved by using pre-trained neural language models, substantial amounts of hallucinated content are found during the human evaluation. In this work, we first devised a typology of factual errors to better understand the types of hallucinations generated by current models and conducted human evaluation on popular dialog summarization dataset. We further propose a training strategy that improves the factual consistency and overall quality of summaries via a novel contrastive fine-tuning, called CONFIT. To tackle top factual errors from our annotation, we introduce additional contrastive loss with carefully designed hard negative samples and self-supervised dialogue-specific loss to capture the key information between speakers. We show that our model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarization. On both datasets, we achieve significant improvements over state-of-the-art baselines using both automatic metrics, ROUGE and BARTScore, and human evaluation.

## introduction text summarization is used to generate a concise and accurate summary of a long text while focusing on the sections that convey the most useful information @xcite . in recent years, the resurgence of dialogue summarization has attracted significant research attentions @xcite @xcite @xcite @xcite @xcite @xcite . the goal of dialogue summarization is to condense the conversational input into brief sentences version but cover salient information @xcite . significant progress has been made recently on abstractive dialogue summarization with various pre-trained models. however, such pre-trained models are susceptible to generating hallucinate content that is not supported by the source documents @xcite @xcite . to tackle the issue of factual inconsistency in dialogue summarization, recent works correctly encode the names of speakers @xcite , explicitly incorporate coreference information @xcite , and order the personal named entities @xcite . but it is still challenging to improve the quality of summaries generated by different models and decrease the hallucination at the same time. to better understand the types of hallucinations generated by the pre-trained models, we devised a linguistically motivated taxonomy of factual errors for dialogue summarization, instead of simply classifying the summary as faithful or not. based on our typology, we defined an annotation protocol for factuality evaluation of dialogue summarization. we then conducted a human evaluation of several pre-trained abstractive summarizers, including bart @xcite , pegasus @xcite , and t5 @xcite , aiming at identifying the proportion of different types of factual errors and studying the weaknesses of the pre-trained models. our typology and annotation helps us gain deeper insights into the causes of factual inconsistency. unlike news summarization @xcite , we found that the challenges posed by dialogue summarization are more related to dialogue flow modeling, informal interactions between speakers, and complex coreference resolution. figure 1 shows a dialogue-summary pair with three specific errors. in order to tackle the top factual errors produced by existing models, we propose to replace the most commonly used fine-tuning with a linguisticallyinformed contrastive fine-tuning approach. for hey, do you have betty's number? amanda: lemme check sorry, can't find it. ask larry. he called her last time we were at the park together. i don't know him well. don't be shy, he's very nice. if you say so… i'd rather you texted him. okay. i just texted him. urgh.. alright. bye. hannah: amanda: hannah: amanda: hannah: amanda: hannah: hey, do you have betty's number? amanda: lemme check sorry, can't find it. ask larry. he called her last time we were at the park together. i don't know him well. don't be shy, he's very nice. if you say so… i'd rather you texted him. okay. i just texted him. urgh.. alright. bye. hannah: amanda: hannah: amanda: hannah: amanda: hannah: dialogue (copy 2) hey, do you have betty's number? amanda: lemme check sorry, can't find it. ask larry. he called her last time we were at the park together. i don't know him well. don't be shy, he's very nice. if you say so… i'd rather you texted him. okay. i just texted him. urgh.. alright. bye. hannah: amanda: hannah: amanda: hannah: amanda: hannah: dialogue (copy 1) dialogue (copy 3) hannah needs betty's number but amanda doesn't have it. amanda needs to contact larry. reference (c) missing information (b) modality & tense error (a) coreference error amanda can't find betty's number. larry called her last time they were at the park. amanda will text larry. ## confit model standard fine-tuning parameterizes the probability p α of the generator on a task-specific labeled dataset by maximizing cross-entropy loss. however, the cross-entropy loss has several shortcomings that can lead to factual inconsistency in dialogue summarization due to its sub-optimal generalization and instability. we propose a more efficient fine-tuning method confit for factual consistency driven by the intuition that good generalization requires capturing the similarity in one class and contrasting them in other classes. in confit, we introduce two additional losses: contrastive loss and self-supervised loss. we use two weights, actually which is coefficients, to adjust the ratio of l con and l self in the total loss of confit. the final training objective j (θ) of the proposed framework is as follows: our linguistically-informed typology and annotation help us gain deeper insights into the causes of different factual errors. to help our models generate more faithful summaries, the proposed confit learns to concentrate on the essential elements of dialogue and capture the dynamic role information as illustrated in figure 3 . ## results we observe that for all three pretrained models confit significantly beat baselines on rouge-1, rouge-l, and human faithfulness score for both datasets. for bartscore, we note that, while performance increases on samsum for all models, it decreases on ami. however, given the fact that human evaluators rated the outputs of all three confit models as more faithful than those of their corresponding baselines on both datasets, the decreases in bartscore on ami can likely be attributed to the imperfection of automated metrics at capturing faithfulness in text. ## conclusion we presented confit, a novel method to improve the faithfulness of abstractive dialogue summarization models via contrastive and self-supervised fine-tuning. by adapting the objective function during fine-tuning to incorporate a contrastive loss that learns to distinguish positives from examples with factual errors, and a self-supervised dialogue-specific loss that captures important dialogue information flow between multiple interlocutors, confit can significantly improve the faithfulness of the abstractive summaries generated by transformer-based sequence-to-sequence language models, and reduce multiple categories of factuality errors in the abstractive summaries by large margins. in our experiment on samsum and ami, we demonstrated that confit achieves better empirical performance compared to the baseline models fine-tuned with the traditional crossentropy loss, based on both automatic evaluation metrics and human evaluation. our work provides new insights into improving the faithfulness of abstractive summarization systems using carefully designed novel objective functions for fine-tuning that captures important structures and features of the text to summarize.