Multi-Stage Pre-training Enhanced by C hat GPT for Multi-Scenario Multi-Domain Dialogue Summarization (2023)

Abstract: Dialogue summarization involves a wide range of scenarios and domains. However, existing methods generally only apply to specific scenarios or domains. In this study, we propose a new pre-trained model specifically designed for multi-scenario multi-domain dialogue summarization. It adopts a multi-stage pre-training strategy to reduce the gap between the pre-training objective and fine-tuning objective. Specifically, we first conduct domain-aware pre-training using large-scale multi-scenario multi-domain dialogue data to enhance the adaptability of our pre-trained model. Then, we conduct task-oriented pre-training using large-scale multi-scenario multi-domain “dialogue-summary” parallel data annotated by ChatGPT to enhance the dialogue summarization ability of our pre-trained model. Experimental results on three dialogue summarization datasets from different scenarios and domains indicate that our pre-trained model significantly outperforms previous state-of-the-art models in full fine-tuning, zero-shot, and few-shot settings.

## introduction dialogue summarization is the task of generating a summary from a dialogue @xcite . specifically, open-domain dialogue summarization involves various scenarios (e.g., online-chat @xcite and daily-life @xcite ), while customer service dialogue summarization involves different domains (e.g., tweet @xcite and e-commerce @xcite ). recently, general-purpose pre-trained models have achieved significant success in dialogue summarization tasks @xcite @xcite . furthermore, several task-specific pre-trained models @xcite have further improved dialogue summarization. existing dialogue summarization pre-trained model @xcite achieves good performance on long dialogue summarization. however, it still has the following limitations: (1) it is only pre-trained on dialogue corpora that include two domains (i.e., interview and tv show), making it difficult to apply to dialogue summarization in a wide range of scenarios and domains. (2) it utilizes a window-based denoising task as the pre-training objective, which presents a significant gap with the fine-tuning objective. simultaneously, existing state-of-the-art (sota) models generally improve dialogue summarization by modeling dialogue interactions @xcite , incorporating extra information (e.g., topics and roles) @xcite , and rewriting dialogues @xcite . although these methods have some effect, they still have limited applicability to downstream datasets in different scenarios and domains and are often difficult to apply within the current pre-training paradigm due to complex model architectures. to address the limitations of previous works, in this study, our goal is to propose a task-specific pretrained model for dialogue summarization, which has extremely small gap between the pre-training objective and the fine-tuning objective, enabling it to excellently adapt to downstream datasets from a wide range of scenarios and domains in full finetuning, few-shot, and zero-shot settings. motivated by the above goal, we consider three key components in the implementation of our pretrained model: model architecture, pre-training corpus, and pre-training strategy. for model architecture, our pre-trained model is based on the standard transformer @xcite encoder-decoder architecture and is initialized with bart @xcite . to capture the underlying role interactions during the dialogue process, we incorporate additional speaker embeddings @xcite into token representations. for pre-training corpus, we collect 14 opendomain dialogue datasets across multiple scenarios and 6 multi-domain customer service dialogue datasets. furthermore, due to the development of large language models (llms) @xcite @xcite and their excellent generative ability, obtaining highquality "dialogue-summary" parallel pre-training data has become possible. therefore, we utilize chatgpt @xcite to annotate the collected multi-scenario multi-domain dialogues and obtain corresponding summaries. we refer to our pre-training corpus as lcm 3 ds (large-scale chatgpt-annotated multi-scenario multi-domain multi-turn dialogue summarization) (see figure 1 (a)). for pre-training strategy, we conduct multistage pre-training to reduce the gap between the pre-training objective and the fine-tuning objective. specifically, we first conduct domain-aware pretraining using the dialogue data from lcm 3 ds to enhance the adaptability of pre-trained model to dialogues in multiple scenarios and domains. then, we utilize the "dialogue-summary" parallel data from lcm 3 ds for task-oriented pre-training to enhance the ability of pre-trained model to summarize multi-scenario multi-domain dialogues. we refer to our pre-trained model as mp4 (multi-stage pretrained model for multi-scenario multi-domain dialogue summarization) (see figure 1 (b) ). we evaluate our pre-trained model on opendomain dialogue summarization datasets from two scenarios (i.e., online-chat @xcite and daily-life @xcite ), as well as a cus-tomer service dialogue summarization dataset from a specific domain (i.e., tweet @xcite ). the experimental results indicate that mp4 significantly outperforms previous sota models in full fine-tuning, zero-shot, and few-shot settings, demonstrating remarkable performance improvements. our contributions are summarized as follows: ## human evaluation we conduct human evaluation to further evaluate the performance of our pre-trained model and strong baselines under various paradigms, as well as the ground truth (i.e., mp4 (dap-top), chat-gpt, bart-large, ground truth). specifically, we randomly select 50 samples from the test set of samsum. then, we invite 3 participants to rank four candidate summaries according to four metrics: fluency (flu.), conciseness (conci.), informativeness (info.), and comprehensiveness (comp.). the top-ranking indicates the best performance on that metric. ptms for dialogue summarization. recently, general-purpose pre-trained models have achieved significant success in dialogue summarization tasks @xcite @xcite . furthermore, several task-specific pre-trained models @xcite have further improved dialogue summarization. moreover, existing stateof-the-art dialogue summarization models typically leverage pre-trained models and model the characteristics of dialogues to achieve better results, including modeling dialogue interactions @xcite , incorporating extra information @xcite , and dialogue rewriting @xcite . although these models are effective, they often have complex model structures that are difficult to apply within the current pre-training paradigm. large language models. more recently, llms have attracted widespread attention due to their remarkable performance in various knowledgeintensive nlp tasks @xcite @xcite . through large-scale pre-training on massive text corpora @xcite , llms possess powerful foundational capabilities. instruction tuning @xcite @xcite helps llms in understanding natural language task descriptions. while reinforcement learning with human feedback (rlhf) @xcite aligns generated text with human preferences. ## conclusion in this study, we propose mp4, a multi-stage pretrained model for multi-scenario multi-domain dialogue summarization. to conduct the pre-training, we construct a large-scale chatgpt-annotated multi-scenario multi-domain multi-turn dialogue summarization corpus called lcm 3 ds. extensive experimental results demonstrate that mp4 exhibits remarkable dialogue summarization capabilities.