L ong T 5: E fficient Text-To-Text Transformer for Long Sequences (2022)

Abstract: Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present LongT5, a new model that explores the effects of scaling both the input length and model size at the same time. Specifically, we integrate attention ideas from long-input transformers (ETC), and adopt pre-training strategies from summarization pre-training (PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call Transient Global (TGlobal), which mimics ETC’s local/global attention mechanism, but without requiring additional side-inputs. We are able to achieve state-of-the-art results on several summarization and question answering tasks, as well as outperform the original T5 models on these tasks. We have open sourced our architecture and training code, as well as our pre-trained model checkpoints.

## introduction transformer models such as bert @xcite , and other variants @xcite @xcite have achieved state-of-the-art results on many challenging nlp tasks. moreover, recent work in longinput transformers @xcite @xcite has shown that increasing the input length a transformer is able to process results in further performance gains. additionally, it is also known that increasing model size also leads to performance gains in many tasks @xcite . in this paper, we present a new model, called longt5, with which we explore the effects of scaling both the input length and model size at the same time. to achieve this, we integrate long-input @xcite with different input length (x axis). baseline models: hat-bart @xcite , bigbird-pegasus @xcite , primer @xcite , led @xcite . the size of circle roughly indicates the # of parameters for each model. regarding attention, we design a new attention mechanism, which we call transient global (tglobal), that mimics etc's local/global mechanism @xcite . importantly, tglobal attention removes the need for the additional side inputs in etc, in order to fit within the t5 architecture. the main idea of etc's local/global mechanism is to introduce local sparsity in the attention mechanism to reduce the quadratic cost when scaling to long inputs. specifically, etc only allows tokens in the input (called the long input) to attend to a local neighborhood, and adds a secondary input called the global memory, through which tokens in the long input can attend to each other indirectly. one disadvantage of this mechanism is that it requires designing this secondary global input for each new problem. in order to adapt it to t5, our new tglobal mechanism synthesizes these global tokens on the fly (as aggregations of groups of tokens in the input), at each attention layer. our experiments show that this mechanism results in only a small degradation in performance with respect to full attention in the same input length but allows the model to scale to much larger input lengths, resulting in significant performance gains. regarding pre-training, we adopt the pretraining strategy in the pegasus @xcite model. this pre-training strategy was originally designed for abstractive summarization, but in our experiments, we found it also improves model performance for other tasks, such as question answering, and hence we adopted it in longt5. the key idea is to mask out key (principle) sentences from a document and ask the model to reproduce them as a single string, as if it was a summary. we evaluate longt5 on several summarization and question answering tasks (see sections 4.2.1 and 4.3.1 for detailed descriptions of these datasets). thanks to the scaling of both input length and model size, we achieve state-of-the-art results on many of them. the main contributions of this work are: for natural questions, we used a slightly different formulation than the original tasks, and hence we do not make state-of-the-art claims. ## conclusion this paper presented a new transformer-based neural model called longt5, with which we have explored the effects of scaling both input length and model size at the same time. specifically, the main differences of longt5 with respect to t5.1.1 are (1) a new scalable attention mechanism called tran-sient global attention, which is a drop-in replacement to the standard t5 attention mechanism, and hence can be used without needing additional sideinputs to the model or modifications to the model inputs; and (2) using a pegasus-style principle sentences generation pre-training objective. via experimentation in several challenging summarization and question answering datasets, we have explored the performance gains that can be achieved by scaling both input length and model size, resulting in state-of-the-art results on several datasets: arxiv, pubmed, bigpatent, mediasum, and triviaqa. as part of our future work, we would like to pursue several directions such as studying efficient attention mechanisms in the decoder and decoder-toencoder attention pieces of the model (both local attention and tglobal attention are only applied to the encoder in longt5 for now). additionally, we would like to incorporate additional long-input transformer ideas into the longt5 architecture, that could further improve model efficiency.