Exploring Multitask Learning for Low-Resource Abstractive Summarization (2021)

Abstract: This paper explores the effect of using multitask learning for abstractive summarization in the context of small training corpora. In particular, we incorporate four different tasks (extractive summarization, language modeling, concept detection, and paraphrase detection) both individually and in combination, with the goal of enhancing the target task of abstractive summarization via multitask learning. We show that for many task combinations, a model trained in a multitask setting outperforms a model trained only for abstractive summarization, with no additional summarization data introduced. Additionally, we do a comprehensive search and find that certain tasks (e.g. paraphrase detection) consistently benefit abstractive summarization, not only when combined with other tasks but also when using different architectures and training corpora.

## introduction recent work has shown that training text encoders using data from multiple tasks helps to produce an encoder that can be used in numerous downstream tasks with minimal fine-tuning (e.g., t5 @xcite and bart @xcite ). however, in multitask learning for text summarization, it is still unclear what range of tasks can best support summarization, and most prior work has incorporated only one additional task during training @xcite @xcite . also, to our knowledge, no prior work has tried to tackle multitask summarization in low-resource domains. our work attempts to address these gaps by answering the following research questions: q1) can abstractive summarization performance be boosted via multitask learning when training from a small dataset? q2) are there some tasks that might be helpful and some that might be harmful for multitask abstractive summarization? q3) will the same findings emerge if a very different learning model is used or if pretraining is performed? q4) will the same findings emerge if a very different small training corpus is used? to answer q1, we use a pretrained bert model @xcite within a multitask framework, and train all tasks using a small-sized corpus of student reflections (around 400 samples). to answer q2, we explore the utility of training on four different tasks (both alone and in combination) in addition to abstractive summarization. to answer q3, instead of fine-tuning with the bert model, we perform experiments using the t5 transformer model @xcite . to answer q4, we replicate the student reflection experiments using two very different corpora (news and reviews). our results show that abstractive summarization in low resource domains can be improved via multitask training. we also find that certain auxiliary tasks such as paraphrase detection consistently improve abstractive summarization performance across different models and datasets, while other auxilary tasks like language modeling more often degrade model performance. low resource training data. while most abstractive summarization work takes advantage of large corpora such as cnn/dailymail, new york times, pubmed, etc. to train models from scratch @xcite @xcite , recent work has also targeted low resource domains. methods proposed to tackle little training data have included data synthesis @xcite , few shot learning @xcite , and pretraining @xcite . our approach is different in that we use the same data multiple times in a multitask setting to boost performance. ## summarization datasets coursemirror (cm) 1 is a student reflection dataset previously used to study both extractive @xcite and abstractive @xcite summarization. the dataset consists of documents (i.e., a set of student responses to a reflective instructor prompt regarding a course lecture) and summaries from four course instantiations: cs, engr, s2015, and s2016. cnn/dailymail (cnn-5%) is a widely used summarization dataset consisting of around 300k news-oriented documents @xcite . since the focus of our research is low resource data, we randomly select 5% (500 documents) from the 1 https://petal-cs-pitt.github.io/data.html cnn/dailymail test and validation sets. then, to keep the cnn-5% data distribution similar to cm (3 courses for training, 1 for testing), we randomly sample 1500 documents for the training set. amazon/yelp foot_0 is a dataset of opinions @xcite ) that is both small as well as similar to coursemirror in that documents consist of multiple human comments where order doesn't matter. this dataset contains customer reviews from amazon and yelp of 160 products/businesses. for each of these, 8 reviews to be summarized are selected from the full set of reviews. ## proposed models this section describes the different tasks used for multitask learning with the intuition behind them, followed by the two summarization models used. ## experiments, results and discussion q2: prior work showed the utility of extractive summarization @xcite and language models (magooda and marcjan, 2020) as auxiliary summarization tasks, and we too observe similar behavior for r1 and r2 in q3: ## conclusion and future work we explored the utility of multitask training for abstractive summarization, using three low resource datasets (cm, cnn-5%, amazon/yelp) and two fundamentally different models (bert, t5) with different preconditions (i.e. bert not pretrained with summarization dataset versus t5 pretrained with cnn dataset) to verify any observed behavior. we also integrated four different auxiliary tasks, in isolation and together. we conducted several experiments to find if training a multitask model, in general, is helpful, or if some tasks might in-troduce degradation in model performance. we showed that indeed some tasks might help improve rouge scores and some might not help, at least when trained in a low resource setting. we found that among all task combinations, (abstractive + paraphrase detection) improved almost all rouge scores across different datasets (cm , amazon/yelp, and cnn-5%) and different models (bert, and t5), with (abstractive + concept detection + paraphrase detection) as another good candidate. we also found that paraphrasing and concept detection, which had not been previously examined as auxiliary abstractive summarization tasks, can be helpful for low resource data. in the future, we plan to continue exploring the generality of our findings, by include new types of low resource data (e.g. discussions, emails), bart as one of the sota models, and new auxiliary tasks. we also plan to combine multitask learning with other low resource methods (e.g., data synthesis).