Low-Resource Dialogue Summarization with Domain-Agnostic Multi-Source Pretraining (2021)

Abstract: With the rapid increase in the volume of dialogue data from daily life, there is a growing demand for dialogue summarization. Unfortunately, training a large summarization model is generally infeasible due to the inadequacy of dialogue data with annotated summaries. Most existing works for low-resource dialogue summarization directly pretrain models in other domains, e.g., the news domain, but they generally neglect the huge difference between dialogues and conventional articles. To bridge the gap between out-of-domain pretraining and in-domain fine-tuning, in this work, we propose a multi-source pretraining paradigm to better leverage the external summary data. Specifically, we exploit large-scale in-domain non-summary data to separately pretrain the dialogue encoder and the summary decoder. The combined encoder-decoder model is then pretrained on the out-of-domain summary data using adversarial critics, aiming to facilitate domain-agnostic summarization. The experimental results on two public datasets show that with only limited training data, our approach achieves competitive performance and generalizes well in different dialogue scenarios.

## introduction with the explosion in the quantity of dialogue data from the internet and daily life, there is growing interest in automatic dialogue summarization for various scenarios and applications, such as email threads, meetings, customer service, and online chats @xcite @xcite . unfortunately, creating large-scale dialogue datasets with annotated summaries is costly and labor-intensive, which makes it difficult to build and train large summarization models using adequate supervision signals, especially in a new domain. hence, it is necessary to develop models for dialogue * corresponding authors. recently, domain adaptation approaches with large-scale pretraining have attracted much attention in low-resource summarization @xcite @xcite . a similar strategy is used in dialogues, whereby external summary data from other domains, e.g., the cnn/dailymail news dataset @xcite , are introduced for model pretraining prior to the final fine-tuning on low-resource dialogue summaries. recent works @xcite @xcite have also reported the effectiveness of pretrained summarizers for different kinds of dialogue scenarios, such as chat logs and medical conversations. however, dialogue summary data has several inherent and significant differences from conventional articles in terms of text styles and summary structures. (i) dialogues generally contain multiple participants who have distinct characteristics. (ii) rather than the formal expressions found in news documents, dialogues often comprise utterances with informal or ungrammatical phrases. (iii) the structure of a dialogue summary, including length and the level of abstraction, is quite different from that in other domains @xcite , e.g., cnn/dailymail. thus, considering the huge difference between dialogues and general documents, direct finetuning on dialogue summaries is not ideal when using a model pretrained from other domains. to better leverage summary data from domains such as news or scientific articles, in this work, we introduce a novel pretraining paradigm called domain-agnostic multi-source pretraining (dams) to summarize dialogues in a low-resource setting. we postulate that the pretraining of dialogue summarization could be decomposed into three procedures: the pretraining of encoder, decoder, and the combined encoder-decoder model. specifically, the dialogue encoder is pretrained on large-scale unannotated dialogues to learn the way of dialogue modeling and understanding. the summary decoder is pretrained on large-scale summarylike short texts to learn a language model in the style of the dialogue summaries. furthermore, the encoder and decoder are combined and pretrained on external summary data to go through an integral process of summarization. the above pretraining processes from the three sources are performed simultaneously. by this means, dams exploits large-scale non-summary data in the same domain to narrow the gap between pretraining and finetuning. additionally, adversarial critics are used to capture the features shared between dialogues and general documents, and to learn to perform domain-agnostic summarization. we conducted experiments on two public dialogue summary datasets, namely samsum @xcite and adsc @xcite . pretraining was conducted on datasets from multiple sources, including dialogue corpora, dailylife short text corpora, and text summarization datasets from the news domain. the experimental results show that with only limited training data of dialogue summaries, our approach achieved competitive performance and showed a promising ability for generalizing different dialogue scenarios. our codes and datasets are publicly available 1 . in summary, our contributions are three-fold: 1) we explore the task of dialogue summarization in a low-resource setting with the usage of external multi-source corpora. 2) a novel pretraining strategy is designed to bridge the gap between out-of-domain pretraining and in-domain finetuning for domain-agnostic summarization. 3) comprehensive studies on two datasets show the effectiveness of our method in various aspects. ## methodology in this section, we detail the low-resource dialogue summarization under the domain-agnostic multisource pretraining (dams). it consists of three pretraining objectives: two reconstruction losses with denoising auto-encoders that learn dialogue modeling and summary-like text generation; a sequence-to-sequence (seq2seq) training objective with the combined dialogue encoder and summary decoder that learns abstractive summarization. additionally, two adversarial critics are attached to the encoder's output representations and the decoder's input representations, learning to perform domainagnostic summarization. the overall framework is illustrated in figure 1 . ## results and analysis in this section, we show the main results of dams against other baselines for dialogue summarization, and probe the effectiveness of dams by explanatory experiments in various aspects. ## conclusion and future work in this paper, we propose a domain-agnostic multisource pretraining paradigm for low-resource dialogue summarization, which exploits external largescale corpora from multiple sources to facilitate dialogue modeling, summary language modeling, and abstractive summarization. the pretraining is conducted with adversarial signals to learn domainagnostic summarization. the experimental results verify the effectiveness and generalization of our method in low-resource settings. future directions are exploring how to keep the token-level cross attention in the multi-source pretraining strategy. in this way, we could adopt the strategy in the models with universal transformer architectures, e.g., bart, to benefit from large-scale pretraining language models.