Adapting Pretrained Text-to-Text Models for Long Text Sequences (2023)

Abstract: We present an empirical study of adapting an existing pretrained text-to-text model for long-sequence inputs. Through a comprehensive study along three axes of the pretraining pipeline – model architecture, optimization objective, and pretraining corpus, we propose an effective recipe to build long-context models from existing short-context models. Specifically, we replace the full attention in transformers with pooling-augmented blockwise attention , and pretrain the model with a masked-span prediction task with spans of varying lengths. In terms of the pretraining corpus, we find that using randomly concatenated short-documents from a large open-domain corpus results in better performance than using existing long document corpora, which are typically limited in their domain coverage. With these findings, we build a long-context model that achieves competitive performance on long-text QA tasks and establishes the new state of the art on five long-text summarization datasets, often outperforming previous methods with larger model sizes.

## introduction nlp applications like summarization and question answering often require processing long text sequences. while there have been tremendous empirical breakthroughs @xcite from large pretrained language models (plms), most of these successes have been confined to short-context tasks @xcite . on long-context nlp benchmarks @xcite @xcite , where the input sequences are often longer than 10,000 tokens, there is still a significant gap between human performance and the state-of-the-art models. extending the success of plms to long texts is nontrivial for the following reasons. first, the quadratic complexity of self-attention makes it prohibitive to directly apply full-attention to long sequences. any long-range architecture needs to be computationally efficient and at the same time capture long-distance dependency. 1 second, the training objectives used by existing plms have largely focused on short text and have not been wellstudied for long-context scenarios. for instance, bart @xcite pretraining involves reconstructing the whole corrupted input sequence, which is impractical for long sequences given the computational overhead of decoder-side attention. additionally, while abundant short documents can be easily collected from web dumps to pretrain short-context models that work well across different domains, long documents are much scarcer and are often collected from specific domains as books or movie scripts @xcite . it is unknown whether the existing corpora are more effective for pretraining a versatile long-context model compared to using artificially constructed long texts. in this work, we conduct a thorough experimental study to find a recipe for building highperforming long-context models. in contrast to a recent work @xcite that pretrains a longcontext model from scratch, we choose to adapt an existing short-text model for long texts with further pretraining. our empirical results demonstrate the effectiveness of this strategy by achieving stronger performance on various downstream tasks, while saving on the high cost of pretraining from scratch. more specifically, we explore three axes of the pretraining pipeline, namely efficient long-range model architectures, long text corpora creation and the choice of pretraining objectives. our main findings are summarized as follows: using these findings, we build a strong longcontext text-to-text model that establishes new state-of-the-art on five long-text summarization tasks (@xmath0. ## conclusion through a comprehensive study on the effects of model architectures, training losses and pretraining dataset, we present an effective recipe to adapt existing pretrained text-to-text models for longsequence nlp tasks. the resulting model sets new state-of-the-art on five long-sequence summarization tasks and achieves consistent gains on qa over local-attention models that simply reuse bart's parameters. apart from presenting a stronger checkpoint for finetuning on downstream tasks, we hope our findings in the study can provide insights for future works that aim to develop stronger longsequence models for downstream tasks.