Entity-based De-noising Modeling for Controllable Dialogue Summarization (2022)

Abstract: Although fine-tuning pre-trained backbones produces fluent and grammatically-correct text in various language generation tasks, factual consistency in abstractive summarization remains challenging. This challenge is especially thorny for dialogue summarization, where neural models often make inaccurate associations between personal named entities and their respective actions. To tackle this type of hallucination, we present an entity-based de-noising model via text perturbation on reference summaries. We then apply this proposed approach in beam search validation, conditional training augmentation, and inference post-editing. Experimental results on the SAMSum corpus show that state-of-the-art models equipped with our proposed method achieve generation quality improvement in both automatic evaluation and human assessment.

## introduction abstractive dialogue summarization is an emerging research area @xcite . while the data size of available corpora is smaller than that for monological summarization @xcite , neural approaches have shown promising potential to generate fluent outputs via fine-tuning large-scale contextualized language backbones @xcite . in most corpus constructed for text summarization, only one reference summary is annotated, and models trained via supervised learning on such corpora provide summaries in a general-purpose manner. however, in practice, the generic text summarizers cannot meet the requirements of certain applications and use cases @xcite . for instance, when generating minutes for meeting transcripts, users have their preferences on different personal perspectives. in this case, controllable summarization provides a flexible solution @xcite since it allows users to obtain diverse generations. as the aim of dialogue summaries often focuses on "who did what" and their narrative flow usually starts with a subject (often persons), the generation process can be modulated by personal named entity planning or prompts @xcite . for example, as shown in figure 1 , a controllable system can produce different summaries based on the specific perspective prompts. 1 however, neural abstractive models often suffer from hallucinations, which lower the reliability of automatic summarization @xcite . in dialogue summarization, this issue commonly involves misaligned personal named entity associations @xcite . for instance, as shown in figure 1 , the model upon the prompt 'anna' generates the description "anna will call marcus and give him mark's number". while this sentence achieves a high score in word-overlapping metrics such as rouge @xcite , the semantic meaning it conveys is incorrect (according to the conversation, the personal named entities 'mark' and 'marcus' (colored in red) are misassigned). such factual inconsistency, the inability to adhere to facts from the source, is a prevalent and unsolved problem @xcite . this limitation is more substantial in controllable scenarios, as models are required to condense and paraphrase important contextual information from various personal perspectives. in this work, we focus on improving the accuracy of personal named entity assignment. given a source dialogue content, detecting and correcting the errors in a generated summary is similar to the de-noising process adopted in sequence-tosequence language modeling schemes @xcite . therefore, we build an entity-based denoising model for dialogue summarization via reference summary perturbation and recovery. we then leverage this de-noising model to improve controllable dialogue summarization: (1) at the training stage, we use the de-noising model as a discriminator, to validate beam search candidates under different prompts, and generate factually consistent summaries. then the validated summaries are added to the training set, which serves as conditional training augmentation. (2) at the inference stage, we use the de-noising model as a corrector, to amend the generated summaries via post-editing. this approach can also be applied to other generic and controllable dialogue summarizers. experiments are conducted on samsum @xcite , which consists of multi-turn dialogues and human-written summaries. empirical results show that our proposed method reduces personal named entity misassignment and achieves improved generation quality on both automatic measures and human evaluation. ## related work text summarization is studied in extractive and abstractive paradigms @xcite . in extractive studies, non-neural approaches utilize various linguistic and statistical features via lexical @xcite and graph-based modeling @xcite , and neural approaches bring about substantial improvements via featurerich distributional representation and hierarchical context modeling @xcite . in contrast, abstractive approaches are expected to generate more concise and fluent summaries, which brings about different technical challenges. to foster end-to-end data-driven methods, corpora in news domain (e.g., cnn/daily mail @xcite , nyt @xcite ) are constructed, and sophisticated neural architectures for abstractive summarization are proposed, such as lstm-based encoding-decoding @xcite , pointer-generator networks @xcite , hybrid extractive-abstractive summarizer @xcite , and fine-tuning large-scale pre-trained language models @xcite . recently, datasets for summarizing conversations are constructed from meetings @xcite or daily chats @xcite . based on the linguistic features of human conversations, many studies pay attention to utilizing conversational analysis for dialogue summarization, such as leveraging dialogue acts (goo and chen, 2018), multi-modal features @xcite , topic information @xcite , coreference @xcite , and fine-grained view segmentation with hierarchical modeling @xcite . controllable language generation introduces auxiliary signals to obtain diverse or task-specific outputs. such tasks include text style transfer @xcite and paraphrasing @xcite . there are various conditional signal formats, such as categorical labels @xcite , latent representations, semantic or syntactic exemplars @xcite , and keyword planning @xcite . for controllable text summarization, @xcite and @xcite proposed two generic frameworks in news domain with length constraint and question/entity indicators, and @xcite proposed personal named entity planning by leveraging the common narrative flow of dialogue summarization. tackling hallucinations in abstractive summarization is an essential research topic in making such summaries applicable to real-world scenarios @xcite . reinforcement approaches proposed using factual consistency as optimization reward @xcite and post-editing approaches @xcite focus on correcting summary of general news corpora or facts extracted from an external knowledge base @xcite . for dialogue summarization, @xcite proposed a binary classifier to detect personal named entity inconsistency. recently, @xcite proposed a post-correction model that can discriminate which type of speaker inconsistency, and revise the output accordingly. in this work, to the best of our knowledge, we are the first to exploit an entity-based de-noising model for abstractive dialogue summarization in both training and inference stages. ## entity-based de-noising modeling while existing abstractive neural models achieve state-of-the-art performance on quantitative evaluation, factual inconsistency remains a prevalent and unsolved problem @xcite . in both document and dialogue summarization, it has been demonstrated that a certain proportion of abstractive summaries contain hallucinated statements @xcite . such hallucinations raise concerns about the usefulness and reliability of automatic summarization, and are challenging to eradicate in neural approaches due to the implicit nature of learning representations. in dialogue summarization, the misassignment of personal named entities significantly affects generation quality @xcite . inspired by the de-noising sequence-tosequence pre-training schemes @xcite , here we propose an entitybased de-noising model to detect and recover the incorrect personal named entity tokens. compared with the binary classifier for factual inconsistency @xcite , the sequence-to-sequence framework supports revising the summaries via post-editing. ## leveraging de-noising modeling in this section, we then elaborate on how to leverage the entity-based de-noising model for control-lable dialogue summarization. ## conclusion in this paper, we focused on reducing incorrect assignments of personal named entities in dialogue summarization. we proposed an entity-based denoising model, and applied it to beam search validation, conditional training augmentation, and inference post-editing (which can be used for nonconditional and conditional summarization). experimental results demonstrated that our proposed method improves performance in both lexical and semantic evaluation metrics and is beneficial to entity-based factual consistency in both automatic and human evaluations. future work can be extending it to pronoun tokens and other entity types.