R ep S um: Unsupervised Dialogue Summarization based on Replacement Strategy (2021)

Abstract: In the field of dialogue summarization, due to the lack of training data, it is often difficult for supervised summary generation methods to learn vital information from dialogue context with limited data. Several attempts on unsupervised summarization for text by leveraging semantic information solely or auto-encoder strategy (i.e., sentence compression), it however cannot be adapted to the dialogue scene due to the limited words in utterances and huge gap between the dialogue and its summary. In this study, we propose a novel unsupervised strategy to address this challenge, which roots from the hypothetical foundation that a superior summary approximates a replacement of the original dialogue, and they are roughly equivalent for auxiliary (self-supervised) tasks, e.g., dialogue generation. The proposed strategy RepSum is applied to generate both extractive and abstractive summary with the guidance of the followed nˆth utterance generation and classification tasks. Extensive experiments on various datasets demonstrate the superiority of the proposed model compared with the state-of-the-art methods.

## introduction dialogue summarization distills key information from a dialogue context and synopsizes it into a concise summary. as a novel topic of critical importance, it offers powerful potentials for a number of scenarios, e.g, the court debate in civil trial, the customer service calls arisen from agent(s) and customer, the business meeting engaged with multimembers. it also assists users in quick access and consumes the essential content in the dialogue. major attempts on dialogue summarization are template-based @xcite in the primitive stage by extracting key information and filling it into the learned templates. however, these template-based techniques limit the figure 1 : a summary is generated from the input dialogue firstly, and then the original dialogue and its corresponding summary are exploited for n th utterance prediction, respectively. j is the ground truth, and j in different colors are the decoded utterances based on the original dialogue and the generated summary respectively. the difference between decoded j is employed for optimization of summary generation. the motivation is that a superior summary approximates a replacement of the original dialogue, and they are roughly equivalent for auxiliary tasks. while prior deep content generation methods rely on large amounts of annotated data, they are rarely available for dialogue summarization due to the prohibitive costs of labeled data. a straightforward way to alleviate the dependency of the annotated data is to apply the existing unsupervised methods designed for document summarization @xcite @xcite to the dialogue scene. however, we argue that these methods accompany weakness either in extractive or in abstractive dialogue summarization. in terms of extractive methods, they mainly rely on semantic information without any supervision signals. as a result, they are ragged in effects due to the limited words in dialogue utterances. as for abstractive approaches, they are commonly designed with an auto-encoder (ae) where the latent variable decodes to a summary which attempts to reconstruct the original input representation. hence, they are constrained to the small gap between the input text and the target summary (e.g., sentence compression) while failing to reconstruct long input text (e.g., dialogue). in this paper, we propose an innovative unsupervised strategy, dubbed repsum, which can be applied to both extractive and abstractive summarization. the key intuition is derived from the evaluation methods of extrinsic summarization @xcite , which testifies the impact of summarization based on how it affects the completion of some other tasks, such as information retrieval, relevance assessment, reading comprehension, etc. we claim that a superior summary can offer a semantic replacement of the original dialogue, which provides equivalent information for completing auxiliary tasks, e.g., dialogue generation, as shown in figure 1 . specifically, we propose two auxiliary tasks which are n th utterance generation and n th utterance selection from k candidates based on the previous contents. both the dialogue and the summary aim to achieve decent performances on the specific task respectively. besides, we introduce kl divergence to curtail the difference between results based on the dialogue and the summary. this strategy provides the summarization with essential self-supervised signals via auxiliary tasks. furthermore, it decouples the training from the reconstruction of ae, which enables to support longer text or dialogue to be effectively summarized. our main contributions are as follows: dialogue summarization extracts significant information from dialogues. most of the initial works adopted extractive-based methods. for instance, @xcite produced multiple short fragments from utterances and then selected the parse of the summary by svm combined with semanticsimilarity features. later, @xcite induced abstractive generation templates for constructing candidate summary sentences. moreover, to benefit from the existing technologies for document summarization, @xcite converted the conversation into a text document through discourse relations and lexical information and then created summaries via pointer-generator @xcite . however, given that dialogues are different from documents in terms of interactive patterns, most researchers explored to summarize the dialogue by leveraging auxiliary information hidden in the utterances. for example, goo and chen (2018) proposed to utilize dialogue act as an auxiliary supervised signal and design a sentence-gated mechanism for modeling the relationships between dialogue acts and the summary. in addition, @xcite predicted the keypoint sequence first and then use it to guide the summary prediction. in contrast to the supervision works, we focus on the unsupervised dialogue summarization considering the high cost and limitation of the labeled data in the dialogue scene. additionally, our proposed strategy is applicable to both extractive and abstractive models without using any outer infor-mation (e.g., template, dialogue acts, and keypoint) but leveraging its intrinsic self-supervised nature. unsupervised summarization historically, unsupervised summarization focused on extracting utterances directly. for example, lead chooses the first several utterances and textrank @xcite ranks utterances by running a graph-based algorithm, where each node represents an utterance and the weight between any two nodes is calculated by the semantic similarity. later, @xcite proposed a centroidbased method for text summarization that exploits the compositional capabilities of word embeddings. @xcite improved it by building graphs with directed edges considering the relative positions of any two sentences which contributes to their respective centrality. in recent works, the task of unsupervised summarization is framed as a self-supervised auto-encoder problem, namely sentence compression. @xcite blunsom (2016); baziotis et al. (2019); chu and liu (2019) applied the auto-encoder framework, where the expected abstract is set to the latent variables from which the input sentence is reconstructed. févry and phang (2018) added noise to extend sentences and trained a denoising auto-encoder to recover the input text. @xcite introduced a hierarchical variational auto-encoder to associate the individual reviews with stochastic latent codes for opinion summarization. recently, another line of works focused on edit-distance-based approaches. @xcite summarized by applying the information bottleneck principle to the objective of conditional language modeling. in addition, @xcite ; @xcite summarized by hill climbing with word-level extraction, which searches the text for a high-scoring summary by discrete optimization. compared to these works, to the best of our knowledge, our model is one of the pioneers attempting unsupervised dialogue summarization. to improve the effectiveness, we devise a generalized strategy repsum that incentivizes the summary to complete the auxiliary tasks as the original dialogue does, thus providing self-training signals and in turn enabling long texts to be summarized. ## conclusion this work investigates the problem of unsupervised dialogue summarization. we propose a novel unsupervised strategy repsum, which roots from the hypothetical foundation that a superior summary approximates a replacement of the original dialogue, and they are roughly equivalent for completing auxiliary tasks. repsum is employed on both extractive and abstractive-based models via a self-supervision from two auxiliary tasks. comprehensive experiments on various datasets show the effectiveness of the proposed mechanism compared to the other unsupervised baselines.