A dapt S um: Towards Low-Resource Domain Adaptation for Abstractive Summarization (2021)

Abstract: State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstractive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three different settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task-adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model’s catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.

## introduction abstractive summarization models aim to extract essential information from long documents and to generate short, concise and readable text. recently, neural abstractive summarization models have achieved remarkable performance @xcite , and large-scale generative pre-training @xcite has shown itself to be surprisingly effective at generation tasks, including abstractive summarization. however, these models generally require large numbers of human-annotated summaries to achieve state-of-the-art performance, which makes them not scalable to low-resource domains where only a few labeled data are available. domain adaptation methods have naturally arisen to tackle the low-resource issue and enable models to quickly adapt to target domain tasks. yet, despite their practicality, very few studies have used domain adaptation methods on the lowresource scenario for the abstractive summarization task. to address this research gap, we present adaptsum, the first benchmark to simulate the low-resource domain adaptation setting for abstractive summarization systems with a combination of existing datasets across six diverse domains (dialog @xcite , email @xcite , movie review @xcite , debate @xcite , social media @xcite , and science @xcite ), and for each domain, we reduce the number of training samples to a small quantity so as to create a low-resource scenario. recently, conducting a second pre-training step on large-scale language models (e.g., bert @xcite , roberta @xcite ) has proven to be effective for domain adaptation tasks @xcite . however, the current methods incorporating such a step are mainly focused on classification or classification-based (e.g., named entity recognition) tasks, leaving a research gap in exploring their use for generation tasks. in this paper, we systematically investigate adding a second phase of pre-training on large-scale generative models under three settings: 1) source domain pre-training (sdpt) based on a labeled source domain summarization dataset; 2) domain-adaptive pre-training (dapt) based on an unlabeled substantial domain-related corpus; and 3) task-adaptive pre-training (tapt) based on an unlabeled smallscale task-related corpus. the second phase of pre-training could cause the catastrophic forgetting in the pre-trained model. thus, we propose to apply recadam @xcite into the pre-training process to alleviate this issue and further improve the adaptation performance. experimental results show that sdpt and tapt can generally improve on the performance of the fine-tuning method, while the effectiveness of dapt is correlated to the similarity between the pre-training data and the target domain task data. different from previous insights into adaptive pretraining on classification tasks @xcite , we find that in the summarization task, dapt could make the adaptation performance worse, even though the pre-training corpus is collected from domain-related sources. furthermore, we show that recadam can further boost the performance of the second pre-training step by effectively maintaining the pre-trained model's knowledge gained in the first phase of pre-training. our contributions are summarized as follows: ## adaptsum the goal of adaptsum is to provide an accessible benchmark for the evaluation of low-resource domain adaptation for abstractive summarization on a diverse set of domains. the vocabulary overlaps between domains are shown in figure 1 . adaptsum consists of six diverse target domains and the corresponding unlabeled domain-related corpora for dapt. we provide the data statistics of all domains in debate wang and ling (2016) introduced an abstractive debate summarization dataset which consists of arguments and the debate topic pairs. the unlabeled corpus is from @xcite . social media @xcite introduced an abstractive summarization dataset of reddit tifu posts, where the summary for each post come from its title. we collect the unlabeled corpus directly from reddit tifu. foot_2 science @xcite introduced a human-annotated abstractive summarization dataset on computational linguistics. we collect the unlabeled domain corpus from the acl anthology @xcite . ## methodology in this section, we will first introduce the three different settings that we investigate for a second pre-training step. then, we will discuss how we ## experimental setup training details we evaluate all of our models on adaptsum. for the dialog and email domains, we use the standard splits of @xcite , while for movie review, debate, social media and science domains, we split the whole dataset into training, validation and test sets by ourselves since the original works do not specify how to split these datasets or the published datasets do not contain the split training, validation and test sets. since the dataset sizes are limited for science, movie review and dialog domains, the maximum training samples for these domains are ## conclusion and future work in this paper, we present adaptsum, the first benchmark to simulate the low-resource setting for the abstractive summarization task with a combination of existing datasets across six diverse domains. we systematically study three different methods for a second phase of pre-training (i.e., sdpt, dapt and tapt), and propose to leverage recadam to alleviate the catastrophic forgetting issue caused by the continuing pre-training. experiments show that sdpt and tapt can generally improve on the performance of the fine-tuning method, while the effectiveness of dapt depends on the similarity between the pre-training data and the target domain task data, which is different from the insights into dapt for classification tasks. further analysis illustrates that recadam successfully alleviates the catastrophic forgetting issue for tapt and further boost its performance. finally, our work highlights several research challenges in low-resource domain adaptation for the abstractive summarization task: (1) how to construct an effective corpus for dapt; (2) how to better cope with the catastrophic forgetting issue for the second pre-training phase on a large corpus; (3) how to effectively integrate the task and domain knowledge (i.e., incorporate sdpt and dapt); (4) how to choose better source domain datasets for conducting sdpt; (5) how to build a more powerful domain adaptation models for the extremely low-resource summarization task. we hope that the proposed dataset and the highlighted research directions will accelerate the studies in this area.