Language Model as an Annotator: Exploring D ialo GPT for Dialogue Summarization (2021)

Abstract: Current dialogue summarization systems usually encode the text with a number of general semantic features (e.g., keywords and topics) to gain more powerful dialogue modeling capabilities. However, these features are obtained via open-domain toolkits that are dialog-agnostic or heavily relied on human annotations. In this paper, we show how DialoGPT, a pre-trained model for conversational response generation, can be developed as an unsupervised dialogue annotator, which takes advantage of dialogue background knowledge encoded in DialoGPT. We apply DialoGPT to label three types of features on two dialogue summarization datasets, SAMSum and AMI, and employ pre-trained and non pre-trained models as our summarizers. Experimental results show that our proposed method can obtain remarkable improvements on both datasets and achieves new state-of-the-art performance on the SAMSum dataset.

## introduction dialogue summarization aims to generate a succinct summary while retaining essential information of the dialogue @xcite . theoretically, @xcite point out that a good summary is intuitively related to three aspects, including informativeness, redundancy and relevance. to this end, previous works have taken the above three aspects into account by incorporating auxiliary annotations into the dialogue. to improve informativeness, some works annotated linguistically specific words (e.g., nouns and verbs), domain terminologies and topic words in the dialogue @xcite @xcite . to reduce redundancy, some works used sentence similarity-based methods to annotate redundant utterances. @xcite . to improve relevance, some works annotated topics for the dialogue @xcite @xcite . however, these annotations are usually obtained via open-domain toolkits, which are not suitable for dialogues, or require manual annotations, which are labor-consuming. to alleviate the above problem, we explore the pre-trained language model as an unsupervised annotator to automatically provide annotations for the dialogue. recently, some works have investigated the use of pre-trained language models in an unsupervised manner. for example, @xcite exploited pre-trained models for assigning domain labels to wordnet synsets. the successful recipe is that a model is obtained extensive knowledge via pre-training on a huge volume of data. when it comes to the dialogue domain, @xcite is a sota conversational response generation model, which is pre-trained on the massive dialogue data. therefore, we draw support from dialogpt and present our dialogpt annotator, which can perform three dialogue annotation tasks, including keywords extraction, redundancy detection and topic segmentation, to measure informativeness, redundancy and relevance of the input dialogue, respectively. keywords extraction aims to automatically identify important words in the dialogue (shown in figure 1(a) ). our dialogpt annotator extracts unpredictable words as keywords. we assume that keywords contain high information, which are difficult to be predicted considering both background knowledge encoded in the dialogpt and contextual information of dialogue context. redundancy detection aims to detect redundant utterances that have no core contribution to the overall meaning of the dialogue (shown in figure 1(b) ). our dialogpt remember we are seeing the wedding planner after work sure, where are we meeting her? at nonna rita's i want to order seafood tagliatelle haha why not we remmber spaghetti pomodoro disaster from our last meeting omg it was over her white blouse i'll make time for it great! blair: chuck: blair: chuck: blair: chuck: blair: chuck: blair: [topic 3] [topic 2] [topic 1] dialogue remember we are seeing the wedding planner after work sure, where are we meeting her? at nonna rita's i want to order seafood tagliatelle haha why not we remmber spaghetti pomodoro disaster from our last meeting omg it was over her white blouse i'll make time for it great! annotator detects utterances that are useless for dialogue context representation as redundant. we assume that if adding a new utterance does not change the dialogue context representation, then this utterance has no effect on predicting the response, so it is redundant. topic segmentation aims to divide a dialogue into topically coherent segments (shown in figure 1(c )). our dialogpt annotator inserts a topic segmentation point before one utterance if it is unpredictable. we assume that if an utterance is difficult to be inferred from the dialogue context based on dialogpt, this utterance may belong to a new topic. we use our dialogpt annotator to annotate the samsum @xcite and ami @xcite datasets. each annotation is converted into a specific identifier and we insert them into the dialogue text. then, we employ pre-traind bart @xcite and non pre-trained pgn @xcite as our summarizers. extensive experimental results show that our method can obtain consistent and remarkable improvements over strong baselines on both datasets and achieves new stateof-the-art performance on the samsum dataset. ## preliminaries in this section, we will describe the task definition as well as the background of dialogpt. ## method in this section, we will first introduce our dialogpt annotator. the workflow consists of three steps (1) dialogue preprocessing; (2) dialogpt forward passing; (3) annotation. the overall framework is shown in figure 2 . then, we will describe our dialogue summarizer, including bart and pgn. ## conclusion we investigate to use dialogpt as unsupervised annotators for dialogue summarization, including keywords extraction, redundancy detection and topic segmentation. we conduct our dialogpt annotator on two datasets, samsum and ami. experimental results show that our method consistently obtains improvements upon pre-traind summarizer (bart) and non pre-trained summarizer (pgn) on both datasets. besides, combining all three annotations, our summarizer can achieve new state-of-the-art performance on the samsum dataset.