A Thorough Evaluation of Task-Specific Pretraining for Summarization (2021)

Abstract: Task-agnostic pretraining objectives like masked language models or corrupted span prediction are applicable to a wide range of NLP downstream tasks (Raffel et al.,2019), but are outperformed by task-specific pretraining objectives like predicting extracted gap sentences on summarization (Zhang et al.,2020). We compare three summarization specific pretraining objectives with the task agnostic corrupted span prediction pretraining in controlled study. We also extend our study to a low resource and zero shot setup, to understand how many training examples are needed in order to ablate the task-specific pretraining without quality loss. Our results show that task-agnostic pretraining is sufficient for most cases which hopefully reduces the need for costly task-specific pretraining. We also report new state-of-the-art number for two summarization task using a T5 model with 11 billion parameters and an optimal beam search length penalty.

## introduction given these new results we want to answer the questions if task-specific pretraining objectives are still at an advantage. to avoid any influence of hyperparameters, pretraining datasets, tokenization or evaluation scripts we reimplement all experiments in the same framework, namely the pega-sus framework. 1 to our surprise, we found that in a controlled comparison the task-agnostic pretraining methods perform as good as task specific pretraining methods for large finetuning setups. we further extend our study to a low resource and zero shot setup, to understand how many training examples are needed in order to ablate the task specific pretraining without quality loss. and finally we want to see if our findings also translate to other text generation tasks. we therefore pretrain a model with corrupted text and evaluate it on grammatical error correction. ## pretraining models we use a transformer architecture @xcite with 12 hidden layers, a hidden size of 768, filter size 3072 and 12 attention heads, with a total of 223m parameters. all models are pretrained for 1.5 million steps on the c4 corpus @xcite with a batch size of 16, adafactor @xcite , a learning rate of 0.01, and maximum input-output lengths of 512 and 256, in the pegasus framework. if not mentioned otherwise we do not perform any hyperparameter tuning but use the best performing hyperparameters founded by @xcite . we do not explore a pretraining plus prefinetuning setup in this paper @xcite . we now briefly explain the task-agnostic objective of corrupted span prediction and two taskspecific objectives, salient sentence selection for summarization and text corruption for grammar error correction. additionally, we also experimented with the objectives of masking and predicting random and lead sentences. ## finetuning experiments all our experiments are done in the pegasus framework. we validate that the numbers are roughly identical with a comparable setup in t5. 2 for this, numbers in ## grammatical error correction we further investigate if our findings translate to other generation tasks. here, we focus on the task of grammatical error correction, but also other important aspects of text generation show benefit from task specific pretraining and are still underexplored; e.g., improving evaluations @xcite , factuality @xcite or planning for grounded generation @xcite . zero shot in contrast to summarization, no easy baseline exists for grammatical error correction. a simple copy baseline would give us a high word overlap like bleu or rouge, but on our main metric f 0.5 this only gets a score of 4.24. our pretrained textcor model achieves an f 0.5 score of 18.64, precision 40.94 and recall 5.87. the t5 model needs only 10 training examples to achieve the same f 0.5 score ( ## conclusion we evaluated several pretraining techniques on two different text generation tasks, summarization and grammatical error correction. our findings are that, while pretraining for summarization is very important, we found no evidence that task specific pretraining improved on common benchmarks for abstractive datasets, even in a low resource setting. on extractive datasets, task specific pretraining showed benefits but the results are below a sentence selection baseline, questioning the practical usefulness. given the trend to larger neural network models with significant costs to train them, we recommend to use a task agnostic pretraining regime. corrupted span prediction is currently our most successful candidate, with state-of-the-art results on two investigated summarization benchmarks. but we are curious if even more flexible pretraining technique will emerge. for grammar error correction, task specific pretraining was showing superior performance, especially in a low resource setting. we therefore believe that, task-specific pretraining or prefinetuning can still be useful for important aspects of text generation.