DIONYSUS : A Pre-trained Model for Low-Resource Dialogue Summarization (2023)

Abstract: Dialogue summarization has recently garnered significant attention due to its wide range of applications. However, existing methods for summarizing dialogues have limitations because they do not take into account the inherent structure of dialogue and rely heavily on labeled data, which can lead to poor performance in new domains. In this work, we propose DIONYSUS (dynamic input optimization in pre-training for dialogue summarization), a pre-trained encoder-decoder model for summarizing dialogues in any new domain. To pre-train DIONYSUS, we create two pseudo summaries for each dialogue example: one from a fine-tuned summarization model and the other from important dialogue turns. We then choose one of these pseudo summaries based on information distribution differences in different types of dialogues. This selected pseudo summary serves as the objective for pre-training DIONYSUS using a self-supervised approach on a large dialogue corpus. Our experiments show that DIONYSUS outperforms existing methods on six datasets, as demonstrated by its ROUGE scores in zero-shot and few-shot settings

## introduction text summarization aims to produce concise and accurate summaries of long texts. recent research on pre-trained neural language models has shown success in summarizing monologues @xcite @xcite , such as news articles @xcite and scientific publications @xcite . however, dialogue summarization presents additional challenges due to the different information distribution in dialogues. self-supervised text summarization models @xcite ; phang et al., 2022) are typically pre-trained on free-form text data, with selected sentences as the pre-training objective. while this approach can be effective for monologues such as news articles, it is less successful at summarizing semistructured and multiparticipant dialogues. as illustrated in figure 1 , in daily chats, dialogue information is often dispersed across various dialogue turns, making it difficult to extract all relevant information through a few selected turns. while a golden summary needs to accurately captures vital information throughout the entire conversation. furthermore, real-world dialogue-summarization applications often have limited or even no labeled data, making it challenging to develop effective models. therefore, it is crucial to develop dialogue summarization models that can perform well in zero-shot and few-shot settings for their practical use. to address these challenges, we propose diony-sus, a pre-trained sequence-to-sequence model designed to summarize dialogues in any domain, even with a lack of labeled data. it uses pseudo summaries as its pre-training objective, which can be dynamically selected from two sources. first, for daily chats where multiple dialogue turns are not sufficient to summarize the dialogue, we train a summary helper using high-quality dialogue summarization datasets to generate pseudo summaries for these types of dialogues. on the other hand, for dialogues like meeting minutes, interviews, and debates, which can be summarized through a selection of essential turns, we use a method inspired by the gap sentence generation (gsg) technique in pegasus to select these turns as pseudo summaries for training. for instance, choosing the final few turns in a conversation can effectively summarize meeting minutes. we have improved upon the gsg method by using the generated summaries from the summary helper as references during gap sentence selection, as they tend to have less noise compared to the full dialogue context. we refer to this source of pseudo summaries as "principal" and refer to our improved method as gsg+. we find that our improved method outperforms previous methods in low-resource settings across different domains, such as daily chats, emails, and customer service dialogues. additionally, we study different objective strategies for selecting the pseudo summary as a pre-training objective from the generated summary and the "principal." we evaluate dionysus on six dialogue summarization datasets. our best model trained on 19 dialogue corpora surpasses pegasus large in a zero-shot setting across all domains. we also found that the best performance is achieved by selecting the source with the highest rouge score as the objective strategy. our main contributions are: ## approach figure 2 outlines the steps for constructing diony-sus: § 2.1 first, a summary helper is constructed using two high-quality dialogue summarization datasets. this helper generates a pseudo summary for each dialogue in our pre-training corpus. § 2.2 next, the "principal" is extracted using gsg+ as the other pseudo summary for the dialogue. § 2.3 finally, various strategies are employed to select the best pseudo summaries from the first and second steps to serve as the objective for pre-training. ## training corpus to train dionysus, we utilized 19 conversational corpora that do not come with pre-defined dialogue summaries. we employed a self-supervised approach by using pseudo-summaries as the pretraining objective. conversational corpora we collect 19 available conversational corpora consisting of 1.7m examples after truncating for pre-training. corpus information is listed in ## results and analysis we focus on low-resource dialogue summarization settings because it is difficult to collect enough training examples. we evaluate dionysus with "all g", "all p", and "better rouge" strategies in zero-shot and few-shot settings and compare it to the baselines. ## related work dialogue summarization is a rapidly growing area of research that focuses on automatically generating concise and informative summaries of conversations @xcite . unlike research on traditional documents like news articles @xcite or scientific papers @xcite , dialogue summarization is particularly relevant in multi-party interactions, such as emails @xcite , meetings @xcite , medical dialogues @xcite , and daily chats @xcite . however, many existing methods for dialogue summarization require a large training dataset with annotated summaries. this can be a major barrier to applying these methods in real-world scenarios, particularly in cases with limited or no annotated data available. our study examines the use of dialogue summarization in low-resource settings to make the process more practical and effortless in various contexts. pre-trained transformer-based @xcite language models @xcite @xcite have become increasingly popular in natural language processing tasks for tackling the data shortage problem. however, many of these models have limitations when it comes to dialogue summarization. @xcite propose pegasus, which masks multiple whole sentences and pre-trains sequence-tosequence models to reconstruct the original text. built on that, @xcite improve the sentence selection strategy and add modules for ensuring factuality during fine-tuning to address the problem of factuality in summarization. phang et al. (2022) extend pegasus with a modified architecture and long-sequence pre-training to tackle long-input summarization. @xcite propose zcode++, a pre-trained language model optimized for abstractive summarization with improved encoder. however, all these methods rely on the gap sentence selection method, which has limitations for dialogue summarization. in contrast, our approach uses pseudo-summary construction as the pre-training objective, making it possible for zeroshot dialogue summarization. another line of work focuses on pre-trained models for dialogues. dialogpt @xcite and plato @xcite , which are pretrained on large-scale conversation datasets such as reddit. for dialogue summarization, @xcite post-train pre-trained language models to rephrase dialogues into narratives and then finetunes them for summarization. in contrast, our approach follows the t5 model's unified text-to-text format for both pre-training and fine-tuning. @xcite train unilm @xcite with a window-based denoising framework for long dialogue understanding and summarization but do not focus on low-resource settings. @xcite propose a pre-training paradigm that pre-trains the encoder and decoder separately in a supervised manner. while our method uses a self-supervised pre-training approach that applies to any dialogue dataset, making it easier to extend to larger pretraining corpora for further improvement. ## conclusion and future work we present dionysus, a pre-trained encoderdecoder model for zero-shot dialogue summarization in any new domain. we pre-train using a self-supervised approach that generates pseudosummaries for large dialogue corpora as the pretraining objective. we investigate the impact of various pre-training objective strategies and model sizes on dialogue summarization performance. our experiments show that dionysus outperforms state-of-the-art models on six datasets in a zeroshot setting. furthermore, dionysus can be fine-tuned with only 10 examples to outperform vanilla t5 fine-tuning with 1,000 examples. this makes dialogue summarization more practical and easier to use in various contexts with minimal effort. we plan to extend this method to abstractive summarization tasks to develop a general zero-shot summarization model.