The Effect of Pretraining on Extractive Summarization for Scientific Documents (2021)

Abstract: Large pretrained models have seen enormous success in extractive summarization tasks. In this work, we investigate the influence of pretraining on a BERT-based extractive summarization system for scientific documents. We derive significant performance improvements using an intermediate pretraining step that leverages existing summarization datasets and report state-of-the-art results on a recently released scientific summarization dataset, SciTLDR. We systematically analyze the intermediate pretraining step by varying the size and domain of the pretraining corpus, changing the length of the input sequence in the target task and varying target tasks. We also investigate how intermediate pretraining interacts with contextualized word embeddings trained on different domains.

## introduction text summarization is a quintessential nlp task that involves generating a coherent and succinct summary of an article containing the most salient information from the original article. summarization systems are particularly useful for scientific articles that tend to be long and rich in technical content. summarization can arguably reduce information overload on researchers and facilitate the quick retrieval of relevant papers from vast amounts of scientific literature. broadly, summarization techniques can be categorized as extractive or abstractive. while abstractive systems treat the summarization problem as a natural language generation task and produce new phrases and sentences directly in the summary, extractive techniques select salient phrases or sentences verbatim from the original document to create a summary. @xcite , @xcite , @xcite report factual hallucinations in abstractive summarization. @xcite highlight the trade-off between faithfulness and abstractiveness. since for the scientific summarization task, it is critical to be factually-accurate and be faithful to the source document, we focus on extractive summarization of scientific articles. large pretrained language models (e.g. bert @xcite ) have been successfully used for many nlp tasks including summarization @xcite , using the following, now widely-adopted, two-step approach: pretraining. start with a pretrained model like bert and suitably adapt its architecture to fit the target task. finetuning. finetune the model using a labeled dataset for the target task. recent work shows the benefits of interspersing the pretraining and finetuning steps with an intermediate pretraining step @xcite , @xcite . this intermediate step often involves supervised pretraining using labeled datasets from different domains for a task that is related to or is the same as the target task. while the efficacy of such pretraining approaches have been studied in prior work for natural language understanding tasks (like entailment, question answering, etc. @xcite ), the effect of pretraining on summarization has been far less explored. in this work, we explore the benefits of intermediate pretraining using existing summarization datasets for a target task involving the summarization of scientific articles. we obtain improvements in performance over state-of-the-art extractive summarization baseline systems on a new sci-entific summarization benchmark, scitldr @xcite . we also make the following key observations: we also observe a trend of diminishing returns with the intermediate pretraining, as we increase the amount of pretraining data. ## related work transfer learning pretrained language models like bert @xcite are trained on selfsupervised training objectives over large amount of unlabelled text corpus. as shown in @xcite , @xcite , @xcite , the pretrained knowledge in these models can be leveraged by domain and task adaptive pretraining before finetuning the model to the desired target task. gururangan et al. (2020), chakrabarty et al. (2019), beltagy et al. (2019) finetune language models on the domains of interest and show improvements on the respective in-domain tasks. summarization recent works in summarization matchsum (zhong et al., 2020), bertsum (liu and lapata, 2019), stepwise etcsum (narayan et al., 2020) use pretrained language models. bart (lewis et al., 2020), pegasus (zhang et al., 2020) use variants of self-supervised training objectives on massive amounts of text corpora and compute to achieve stellar performance on summarization tasks. while most of the recent works focus on improving state-of-the-art results on news datasets like cnn/dailymail and xsum, improv-ing summarization on scientific documents is an overlooked area. intermediate pretraining howard and ruder (2018) first introduced the idea of intermediate pretraining in nlp and its benefits on the 6 tasks of classification. the benefits of this in summarization has been shown by yu et al. (2021) where in they finetune bart (lewis et al., 2020) on xsum (narayan et al., 2018) and show its results on low resource domain adaptation benchmark for summarization . we show the effects of intermediate pretraining in the context of scientific document summarization. scientific summarization cachola et al. (2020) introduce the scitldr task and benchmark a variety of summarization models such as matchsum and bertsum on the task. impressive results were reported by pilault et al. (2020), zaheer et al. (2020) on scientific datasets like pubmed, arxiv using compute-intensive transformer based models. we report results on pubmed and scitldr where our models use significantly less compute and achieve superior results on scitldr over bertsum and matchsum. cross task learning lebanoff et al. (2018), mao et al. (2020) use methods that adapt single document summarization task to multi document summarization setup, namely using the cnn/daily mail (cnn/dm) dataset. while it is similar to the idea of the intermediate finetuning used in this paper, the end task is different and are tested over a different set of metrics. zhong et al. (2019) conducts some experiments with supervised pretrained knowledge transfer, we do extensive experiments in the context of scientific summarization. ## base model in this paper, we base our experiments on the bert-sum @xcite architecture that uses bert embeddings and formulates extractive summarization as a sentence classification problem. the intermediate pretraining step uses data from a summarization task that is different from the target task and could also be from a different domain. we also experiment with replacing pretrained bert embeddings with scibert embeddings @xcite . ## results and discussion with pretraining and replacing bert with scib-ert, we observe a deterioration in performance indicated by the drop in rouge scores (especially with cnn/dm). the scibert initialization appears to be counterproductive when using cnn/dm during intermediate pretraining. it is more beneficial to start with bert, rather than scibert, and pretrain on cnn/dm before the final finetuning step. additionally, we undertake two ablation experiments. 1) we investigate the effect of varying amounts of pretraining data. we vary the size of cnn/dm to 83k, 176k and 286k articles and analyse the finetuning results on scitldr-aic with bert embeddings. as shown in ## conclusions and future work in this paper, we present a systematic investigation of the benefits of transfer learning via pretraining for extractive summarization of scientific articles. we show improvements in rouge scores for the scitldr benchmark using an intermediate pretraining that uses existing summarization datasets. we obtain additional benefits by filtering these existing datasets to construct a pretraining corpus that best matches the target task. this suggests the need for further explorations in future work on different criteria to be used for selective pretraining and how it could benefit both extractive and abstractive summarization.