Does Pretraining for Summarization Require Knowledge Transfer? (2021)

Abstract: Pretraining techniques leveraging enormous datasets have driven recent advances in text summarization. While folk explanations suggest that knowledge transfer accounts for pretraining’s benefits, little is known about why it works or what makes a pretraining task or dataset suitable. In this paper, we challenge the knowledge transfer story, showing that pretraining on documents consisting of character n-grams selected at random, we can nearly match the performance of models pretrained on real corpora. This work holds the promise of eliminating upstream corpora, which may alleviate some concerns over offensive language, bias, and copyright issues. To see whether the small residual benefit of using real data could be accounted for by the structure of the pretraining task, we design several tasks motivated by a qualitative study of summarization corpora. However, these tasks confer no appreciable benefit, leaving open the possibility of a small role for knowledge transfer.

## introduction despite the widespread success of pretrained models when fine-tuned on diverse downstream nlp tasks, such as summarization @xcite , question answering, sentiment analysis etc @xcite , scientific explanations for these benefits remain unknown. several works have claimed that pretrained models learn linguistic knowledge from the pretraining corpus @xcite @xcite , leading to a popular, but unproven hypothesis that credits knowledge transfer for the improvements seen on downstream tasks. however, several recent findings test the plausibility of this account. for example, benefits of pretraining have been observed even when the upstream text has no syntactic structure @xcite and others have shown benefits even when the upstream corpus is from a different domain entirely, such as music @xcite or amino acid sequences @xcite in this work, we show that, surprisingly, pretraining objectives previously demonstrated to be helpful for summarization @xcite , continue to deliver significant benefits even when applied on text consisting of randomly sampled nonsense words. because the text consists of nonsense words sampled independently and uniformly, it seems difficult to fathom a credible argument that the synthetic corpus encodes linguistic knowledge in any relevant sense. nevertheless, when pretraining transformer-based sequence-to-sequence models using this nonsense text, we achieve significant performance boosts on multiple downstream summarization benchmarks that nearly match the performance of pretrained transformers. remarkably, when pretraining with synthetic tasks, using real data offers no benefit over the nonsense data, on multiple summarization benchmarks. thus, we investigate whether a pretraining task better aligned with the demands of summarization might close this residual gap. we design a collection of pretraining tasks inspired by some of the basic primitive operations that appear to be common routines required in order to create real-world summaries. we carried out an extensive survey of public summarization datasets spanning different domains, and catalogued several elementary operations that were frequently invoked in producing summaries (e.g., extract sentences on a specific topic, or determine the most frequent among a set of relevant terms). in our proposed pretraining corpus, the summary is created by carrying out these elementary operations on the input. however, we find that our pretraining tasks deliver comparable performance gains to those proposed in @xcite leaving the small gap open. on cnn-dailymail and rotowire benchmarks, where median summary lengths are 73 and 456 tokens respectively, using our pretraining tasks with nonsense text results in achieving on average 95% of the performance gain in rouge-1 that standard t5 pretrained models enjoy relative to randomly initialized t5. by contrast, on xsum and rottentomatoes, where summaries are shorter (29 and 32 tokens respectively), we realize a relatively modest 37% of the benefit on average. the takeaways from our results are two-fold: first, these results challenge our understanding of why pretraining helps in summarization, suggesting that a large portion of the benefits seen may not be due to any knowledge transfer, but simply better initialization from an optimization perspective. second, the ability to realize the benefits of pretraining without using real-world data could alleviate concerns regarding bias, offensive speech, and intellectual property associated with using webscale pretraining corpora of unknown provenance @xcite . ## related work recently, multiple pretrained models have shown remarkable performance on text summarization. these models have been pretrained on real data with diverse denoising tasks, including masked language modeling @xcite ), text infilling @xcite , and sentence reordering @xcite , among others. while these pretraining objectives have shown benefits across multiple nlp tasks, @xcite proposed a set of three denoising pretraining tasks that are specifically motivated by summarization and deliver performance comparable to previous pretrained models. our paper shows that the pretraining tasks in @xcite improve summarization performance even if the pretraining corpus is artificial and does not encode any linguistic structure. our work extends a growing body of scientific literature that questions commonly-held beliefs about what properties of a pretraining corpus lead to improvements on different downstream tasks. recently, @xcite showed that word order in pretraining documents has negligible impact on downstream performance on the glue benchmark. even pretraining on sequences from different modalities such as java code and amino acid sequences @xcite have shown ben-efits on glue benchmark, similarly, for the task of language modeling, pretraining on musical scores, or even artificial sequences of nested parentheses has shown to achieve better perplexity on a human language (papadimitriou and jurafsky, 2020). our results go further-here the source documents contain no natural data at all, nor do they exhibit any non-trivial structure. recently, some machine learning theory literature has begun to question the mechanism by which transfer learning works. for example, neyshabur et al. (2020) attribute the benefits to low-level statistics of the data and optimization considerations rather than feature reuse. in other related work, @xcite show that networks pretrained on randomly labeled data sometimes enjoy considerable performance improvements on downstream tasks. ## generating the nonsense corpus for generating the nonsense pretraining corpus, we use an artificial vocabulary to create base documents that has little resemblance to any real language. our vocabulary simply consists of the first 5000 3-letter character combinations using the english alphabet in lexical order starting from the right (aaa, baa, caa, ..., aab, bab, ...). each sentence is generated by sampling each word in it independently from the uniform distribution over the entire vocabulary, and ending it with a period (see figure 1 for a sample nonsense document). the length of each sentence is selected uniformly from 5 to 15 words. the number of sentences per document is selected according to the pretraining task that it is used for. for the pretraining tasks proposed in @xcite , we sample sentences until the document reaches 512 tokens in length. for our pretraining tasks (introduced later), number of sentences in a document is decided by sampling uniformly from 7 to 13 sentences. ## step pretraining tasks step pretraining tasks are a collection of 3 tasks defined by @xcite . next sentence generation (nsg) provides the first half of a document as input and the target is to generate the latter half. sentence reordering (sr) presents a document with its sentences shuffled in random order, and requires generating the original document with correct sentence order. masked document generation (mdg) masks out a contiguous sequence of to-kens in the base document and requires generating the original document while correctly filling-in the masked tokens. more details and hyperparameters can be found in the original paper. ## our pretraining tasks to develop our pretraining tasks, we first undertook a qualitative analysis of existing summarization datasets. we surveyed all summarization papers published in the last 10 years with more than 25 citations, cataloguing a list of the summarization datasets that were used in them. we observed that datasets can be grouped together according to domain (e.g., news and conversations). we grouped the 28 retrieved datasets into 14 domains (see the appendix, from each selected dataset, we manually inspected ten randomly sampled input-summary pairs, looking for primitive subtasks that seem to express skills (informally) that are required in order to create the summaries demanded by this dataset for at least two of the ten instances. since we need to create artificial input-summary pairs for each subtask, we only chose subtasks for which it was possible to create large number of such artificial pairs. for example, in the samsum dataset @xcite which requires summarizing conversations between people, a frequently necessary subtask is to infer the unfolding social scenario (e.g. a fight, or a person helping another) but it is difficult to create a large number of varied artificial conversations that reflect the situation. on the other hand, subtasks such as extracting those sentences that address some specific topic, or (even simpler) extracting the first sentence of the input are simple enough to facilitate creating data points programatically. note that while copying the first sentences might seem like a trivial or uninteresting pretraining task, it can be very useful. for example, in news summarization datasets the lead-3 baseline (copying over first 3 sentences as the summary) works very well @xcite . based on this analysis, we developed 21 elementary tasks, including copying specific content, performing numerical operations, and more. see ## summarization benchmarks we fine-tune and evaluate our models on 4 downstream summarization benchmarks. cnn-dailymail-10k @xcite training (randomly sampled from the training set) so that the impact of pretraining is more visible. however, we still evaluate the fine-tuned model on the full test set. xsum-10k @xcite also a news summarization dataset. again, we train on a random subset of 10k instances from the training set. rottentomatoes @xcite this dataset concerns summarizing critical reviews of movies found on the website rottentomatoes.com. rotowire @xcite here, the task is to process the box-score of a basketball game (often requiring numerical reasoning) to create a post-game summary. ## experiments and results first, we pretrain the transformer-based sequenceto-sequence architecture used by the t5 model @xcite , on different corpora, each containing 100k input-summary pairs to get different pretrained models. we use the t5-small architecture in all experiments. next, we fine-tune each model on the downstream tasks and measure performance via rouge score ( pretraining with either our proposed pretraining tasks (ourtasks), or step tasks (steptasks) performs much better than random initialization, even when using nonsense data. for all summarization benchmarks except rottentomatoes, the performance remained comparable when we used real upstream data from wikipedia to create the pretraining datasets. this suggests that for some summarization benchmarks, there might be little or no additional benefit provided by using real world pretraining text. looking at individual steptasks, nsg has no training signal since the output is completely independent of the input, but surprisingly it leads to improvements in rotowire benchmark. sr and mdg performed much better than nsg on cnn-dm and xsum, likely because they involve copying sentences/unmasked tokens from the input. we created adjusted versions of these pretraining datasets, where there was no copying needed and it led to a drop in performance on both pretraining tasks, bringing it close to t5-ri for cnn-dm and xsum. in sr-adjusted, the task is to output only the numerical order in which sentences should be copied (versus actually generating the full output). in mdgadjusted, the task is to only output the masked-out tokens (versus outputting the entire document, including unmasked tokens). a randomly initialized pointer-generator model @xcite (pg-ri) performs far better than a randomly initialized t5 model. however, t5-architecture models pretrained on nonsense text were able to outperform pointer-generator on 3 out of 4 benchmarks, suggesting that transformer models pretrained on nonsense text can be a better choice than using non-pretrained lstm based models. interestingly, pretraining the pg model on either ourtasks or steptasks did not lead to any additional improvement. models pretrained separately on each task from ourtasks exhibit strong differences in their performance on cnn-dailymail-10k benchmark ( ## conclusion this paper demonstrated that transformer models pretrained on randomly generated nonsense data deliver remarkable performance gains across multiple summarization tasks, compared to their randomly initialized version. this suggests that a substantial part of the observed benefits of pretraining can not be attributed to knowledge transfer. to investigate whether the design of pretraining task itself plays a significant role and can lead to further performance gains, we explored summarization datasets to prepare a battery of tasks found useful in creating summaries. but these pretraining tasks performed comparably to more generic pretraining tasks used in literature. our work suggests that understanding pretraining may have more to do with poorlyunderstood aspects of how initialization influences optimization than with knowledge transfer.