LongT5: Efﬁcient Text-To-Text Transformer for Long Sequences
Abstract
Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of
Transformer-based neural models.
In this paper, we present LongT5 , a new model that explores the effects of scaling both the input length and model size at the same time.
Speciﬁcally, we integrate attention ideas from long-input transformers (ETC), and adopt pretraining strategies from summarization pretraining (PEGASUS) into the scalable T5 architecture.
The result is a new attention mechanism we call Transient Global (TGlobal), which mimics ETC’s local/global attention mechanism, but without requiring additional side-inputs.
We are able to achieve state-ofthe-art results on several summarization and question answering tasks, as well as outperform the original T5 models on these tasks. We have open sourced our architecture and training code, as well as our pre-trained model checkpoints.
Introduction
Transformer models such as BERT (Devlin et al., 2019), and other variants (Liu et al., 2019; Radford et al., 2019; Raffel et al., 2019a; Lewis et al., 2020) have achieved state-of-the-art results on many challenging NLP tasks.
Moreover, recent work in longinput transformers (Ainslie et al., 2020; Zaheer et al., 2020b; Beltagy et al., 2020; Tay et al., 2021) has shown that increasing the input length a Transformer is able to process results in further performance gains.
Additionally, it is also known that increasing model size also leads to performance gains in many tasks (Kaplan et al., 2020).
In this paper, we present a new model, called
LongT5 , with which we explore the effects of scaling both the input length and model size at the same time. To achieve this, we integrate long-input 4k 8k 16k A v e r a g e R
O U G
E S c
o r e LongT5-XL HAT-BARTBigBird-Pegasus PRIMER
LED-large Arxiv 4k 8k [Figure] [Figure]
[Figure]
transformer attention and pre-training ideas into the scalable T5 (Raffel et al., 2019a) model architecture.
The resulting model, as shown in Figure 1, achieves state-of-the-art performance on several tasks which require handling long sequence inputs.
Regarding attention, we design a new attention mechanism, which we call Transient Global (TGlobal), that mimics ETC’s local/global mechanism (Ainslie et al., 2020).
Importantly, TGlobal attention removes the need for the additional side inputs in ETC, in order to ﬁt within the T5 architecture.
The main idea of ETC’s local/global mechanism is to introduce local sparsity in the attention mechanism to reduce the quadratic cost when scaling to long inputs.
Speciﬁcally, ETC only allows tokens in the input (called the long input ) to attend to a local neighborhood, and adds a secondary input called the global memory , through which tokens in the long input can attend to each other indirectly.
One disadvantage of this mechanism is that it requires designing this secondary global input for each new problem.
In order to adapt it to T5, our new TGlobal mechanism synthesizes these global tokens on the ﬂy (as aggregations of groups of tokens in the input), at each attention layer.
Our experiments show that this mechanism results in only a small degradation in performance with respect to full attention in the same input length but allows the model to scale to much larger input lengths, resulting in signiﬁcant performance gains.
Regarding pre-training, we adopt the pretraining strategy in the PEGASUS (Zhang et al., 2019a) model.
This pre-training strategy was originally designed for abstractive summarization, but in our experiments, we found it also improves model performance for other tasks, such as question answering, and hence we adopted it in LongT5.
The key idea is to mask out key (principle) sentences from a document and ask the model to reproduce them as a single string, as if it was a summary.
We evaluate LongT5 on several summarization and question answering tasks (see Sections 4.2.1 and 4.3.1 for detailed descriptions of these datasets).
Thanks to the scaling of both input length and model size, we achieve state-of-the-art results on many of them.
The main contributions of this work are:
• A new Transformer architecture, LongT5 , that allows for scaling both input length and model scale at the same time.
• A new attention mechanism (TGlobal), which mimics ETC’s local/global mechanism but is a drop-in replacement to regular attention for existing Transformer architectures like T5.
• An analysis of model performance when varying both input length and model size of vanilla T5 and LongT5 models (pushing both models up to the maximum lengths they can handle before encountering memory issues), to understand the trade-offs in both performance and computation cost.
• State-of-the-art results on the arXiv, PubMed, BigPatent, MediaSum, and TriviaQA datasets.
For Natural Questions, we used a slightly different formulation than the original tasks, and hence we do not make state-of-the-art claims.
• We open source our model architecture 1 and training code,
as well as pre-trained model checkpoints on GitHub 2 .
T
T5 (Raffel et al., 2019a) is a transformer based textto-text pre-trained language model that is gaining popularity for its uniﬁed framework that converts all text-based language problems into a text-to-text format, and its ease to scale up in number of parameters (from 60M to 11B parameters) with model parallelism.
With full attention transformer, T5 has been successfully applied to many NLP tasks, but the tasks only require shorter input sequences.
This is due to the limitation of quadratic computation growth with respect to input sequence length, resulting in larger memory consumption and longer training time.
Recently, Press et al.
(2021) explored scaling up T5 style models at inference time to longer sequences than seen during training, but how to scale up T5 style models in the input sequence length during training remains underexplored.
LongT 3. Architecture
We extend the original T5 encoder with globallocal attention sparsity patterns (Ainslie et al., 2020; Zaheer et al., 2020a) to handle long inputs.
For the work reported in this paper, we used a standard T5 decoder since all of the tasks we considered require relatively short output sequence lengths.
Architecturally, the main difference between T and LongT5 lies in the attention mechanism.
We experiment with two attention mechanism variations for LongT5, illustrated in Figure 2: (1) Local Attention and (2)
Transient Global Attention (TGlobal) .
Both variations preserve several properties of T5: relative position representations, support for example packing, and compatibility with
T5 checkpoints.
3.1. Local Attention
For Local Attention , we simply replace the encoder self-attention operation in T5 with a sparse slidingwindow local attention operation following the implementation in ETC (Ainslie et al., 2020).
Specifically, for a given local radius r , this formulation only allows each token to attend r tokens to the left and right of it (see Figure 2.a).
We found r =to be sufﬁcient in practice, where r is the number of neighboring tokens to the left and to the right.
Local Attention does not introduce any new parameters and easily accommodates the attention masking required for example packing 3 .
For a
Attention keys
A tt en t i on que r i e s
Attention keys
A tt e n ti on qu e r i e s
Attention keys
A tt e n ti on qu e r i e s x 1 … x k x k+ … x l g 1 … g m + + …
Input Tokens
Global Tokens
Each global token is the result of averaging k input tokens. Each
[Figure] attend to its neighborhood (like in local attention ), plus to all global tokens.
[Figure]
r r
[Figure]
given choice of r , complexity is linear in input sequence length l : O ( l × r ) . 3.1. Transient Global Attention (TGlobal)
To allow input tokens to interact with each other in each layer of the encoder at a longer range than Local Attention’s local radius, we introduce Transient
Global Attention as a modiﬁcation of ETC’s globallocal attention in a “ﬁxed blocks” pattern.
Namely, we divide the input sequence into blocks of k tokens, and for each block we compute a global token by summing (and then normalizing) the embeddings of every token in the block (see Figure 2.b).
Now when computing attention, we allow each input token to attend not only to nearby tokens like in Local Attention, but also to every global token.
We call these global tokens transient because in contrast to ETC-like global-local attention patterns, these tokens are dynamically constructed (and subsequently discarded) within each attention operation, removing any requirement for deciding which input tokens should be treated as “global”.
TGlobal attention only introduces a couple new parameters 4 : (1) T5-style relative position biases representing the distance from an input token’s block to the block of each global token it’s attending to, and (2) T5-style layer normalization parameters for normalizing each global token’s embedding.
The rest of the parameters are identical to T5, and we accommodate sequence packing by addition- ally masking attention from input tokens to global tokens of other examples.
We found block size k
= 16 to be sufﬁcient in practice.
Notice thus, that TGlobal attention introduces a block of l ∗ l/k additional attention key-value pairs to calculate on top of Local Attention ( l input tokens, attending to l/k global tokens; represented by the right most rectangle in Figure 2.b), hence for input sequence length l , complexity is O ( l ( r + l/k )) .
3. PEGASUS Principle Sentences Generation Pre-training
T5 is pre-trained with a span corruption objective, where spans of consecutive input tokens are replaced with a mask token and the model is trained to reconstruct the masked-out tokens.
While it is effective, recent work on masked language modeling (MLM) (Liu et al., 2019; Zhang et al., 2019b) shows that carefully selecting the prediction objective could lead to signiﬁcantly better performance.
One argument is that predicting more informative tokens from the text could force the model to learn better semantics of the text.
Motivated by that, we explore masking and generating the principle sentences from the text.
In particular, we adopt the Gap Sentences Generation with Principle IndUniq strategy from Zhang et al. (2019a), which was used for summarization pre-training.
Following Zhang et al.
(2019a), we select top- m scored ( Principle ) sentences based on
ROUGE-F1 score (Lin, 2004)
using s i =
rouge ( x i , D \ { x i } , ∀ i ) , where i is the sentence index, D is the collection of sentences in the document.
Each sentence is scored independently ( Ind ), and each n -gram is only counted once ( Uniq ).
Experiments 4. Conﬁgurations
LongT5 is implemented using JAX 5 and the Flaxformer 6 library.
Following the same setup as
T5.1.1 7 , we consider models of 3 sizes: base ( ∼ 220M), large ( ∼ 770M), and xl ( ∼ 3B), and use the same cased English SentencePiece vocab model used by T5.1.1, which contains 32000 sentence pieces.
We use batch size of 128 and Adafactor as the optimizer in all experiments.
We decide to use greedy decoding instead of beam search for all our experiments even with the test sets, therefore, our results reported below could potentially be improved further by using beam search, but we would like to make the setup consistent with our dev setup.
4.1. Pre-training
We pre-train LongT5 models for 1M steps on 4096 input sequence length and 910 output sequence length.
We use the same inverse squareroot learning rate schedule as T5, with learning rate set to 1 / (cid:112) max ( step, warm _ up steps ) , where warm_up steps is set to 10000.
The same as
T5.1.1, we pre-train LongT5 only on the C4 dataset (Raffel et al., 2019b), and we do not apply dropout during pre-training.
As described in section 3.2, we use the PEGASUS Principle Sentences Generation objective as our pre-training objective.
The conﬁguration is similar to what was described by
Zhang et al. (2019a) for their larger models, except for the masked sentence ratio in which we use a value of 0.2 instead of 0.45 8 .
In section 5.3, we will show our ablation study between Principle
Sentences Generation and Span Corruption.
4.1. Fine-tuning
For ﬁne-tuning, we use a constant learning rate of 0.001 and dropout rate of 0.1 for all tasks.
For summarization tasks, we experiment with values of 4096, 8192, and 16384 for input lengths andfor output lengths.
For QA tasks, we experiment with values starting at 512 and scale up tofor input lengths and 128 for output lengths.
4. Evaluation on Summarization Tasks
We choose to benchmark our models on summarization tasks that cover various context lengths, because of their long context understanding and generative nature.
4.2. Datasets
LongT5 was benchmarked on the following six datasets.
CNN / Daily Mail (Nallapati et al., 2016) News from CNN and Daily Mail are used as input and the article’s summary bullets are the target summary.
PubMed (Cohan et al., 2018)
Scientiﬁc documents were collected from PubMed, with a document’s content used as input and its corresponding abstract as the target summary.
arXiv (Cohan et al., 2018)
Similar to PubMed, but with documents taken from arXiv.
BigPatent (Sharma et al., 2019)
U.S. patent documents, with the patent’s details used as input and the patent’s abstract as the target summary.
MediaSum (Zhu et al., 2021)
Interview transcripts from CNN and NPR were used as input and their corresponding topic and overviews used as the target summary.
Multi-News (Fabbri et al., 2019)
The task involves summarizing multiple news documents about a topic into a human-written summary.
Table 1 provides statistics for the number of examples in train, validation, and test splits, and the average, median, max, and 90th percentile input sequence length.
As can be seen, these datasets are long in input length, and would beneﬁt from models that can model lengthier inputs.
We included the CNN / Daily Mail dataset to benchmark on a common task, especially to see how using TGlobal attention impacts the model, despite the length of the inputs being smaller than the other datasets.
4.2. Results
We compare LongT5 with various top approaches:
BigBird-PEGASUS (Zaheer et al., 2020b), HATBART (Rohde et al., 2021), DANCER PEGASUS (Gidiotis and Tsoumakas, 2020), PRIMER (Xiao et al., 2021), TG-MultiSum (Cui and Hu, 2021),
LED (Beltagy et al., 2020), and an application of
BART by Zhu et al.
For these comparisons, we use common evaluation metrics of ROUGE-1,
ROUGE-2, and ROUGE-L.
[Table]
[Table] As can be seen in Table [Table] LongT5 is able to achieve state-of-the-art rouge scores for arXiv,
PubMed, BigPatent, and MediaSum.
For arXiv and PubMed, which are composed of longer inputs, being able to scale up to 16k input length helps
LongT5 achieve strong results.
One dataset where LongT5 is not able to achieve state-of-the-art results is with Multi-News.
LongT is the 2nd best model, slightly worth than PRIMER.
This is understandable as the PRIMER model was pre-trained on a large corpus of documents related to news events, thus exposing the model to a similar corpus as that seen in Multi-News.
When looking at CNN / Daily Mail, we can see that LongT5 was comparable with HAT-BART, despite not having full attention.
LongT5 did at least get stronger scores in the ROUGE-2 metric.
4. Evaluation on QA Tasks
For the evaluation on QA tasks, we choose two popular benchmarks, Natural Questions and TriviaQA, that require long context understanding.
4.3. Datasets NaturalQuestions (NQ)
Questions are real queries issued by multiple users to Google search that retrieve a Wikipedia page in the top ﬁve search results.
Answer text is drawn from the search results (Kwiatkowski et al., 2019).
The original NQ dataset asks models to predict a short answer (including no-answer or yes/no) and a long answer.
We framed the task as a seq2seq task and ignored the long answer.
Hence, our results focus only on short answer.
Moreover, since our models predict answer texts instead of answer spans, our evaluation method differs slightly from the leader boards, and our results are not directly comparable to other existing approaches: (1) Since only the train and dev sets are publicly available, we use 90% of the ofﬁcial train set for training while using 10% as hold-out dev set to ﬁne-tune the hyperparameters and training epoch, and use [Table] [Table] the ofﬁcial dev set as our test set.
(2) We benchmark
LongT5 against the corresponding T5.1.1 models instead of directly comparing to the leader boards.
TriviaQA
Trivia enthusiasts authored questionanswer pairs.
Answers are drawn from Wikipedia and Bing web search results, excluding trivia websites (Joshi et al., 2017).
We use the ofﬁcial train/validation splits for training and ﬁne-tuning the hyperparameters and training epoch, then re-train that model combining both train and validation sets to evaluate on the
Wikipedia domain on the leader board 9 .
Table 3 shows the dataset statistics for the number of examples in train and validation splits, and the average, median, max, and 90th percentile input sequence length.
4.3. Results
Table 4 shows a summary of the results for the NQ and TriviaQA datasets (see Appendix B for full results).
For each dataset, we show two metrics:
EM (Exact Match) and F1 score (evaluating precision and recall of individual words in the answer compared to the ground truth, ignoring stop words).
For NQ, we compare T5.1.1, LongT5 with Local
Attention, and LongT5 with TGlobal attention.
We decided to run T5.1.1 (1) with the default 512 input sequence length 10 and (2) with the largest input sequence length that can ﬁt into device memory 11 , and use those as baselines.
Since we are comparing against T5.1.1, for LongT5 experiments we report results at 512 input length for base and large, and the largest input length allowed by each model before running out of memory on the same hardware conﬁguration used in our T5.1.1 experiments.
As the table shows, increasing input length generally results in signiﬁcant beneﬁts in NQ, with models with larger input lengths signiﬁcantly outperforming those with smaller input lengths in most cases.
Some times, models with the largest input [Figure] lengths underperform those with 4k length, but we believe those to be due to noise in the experiments, as results are the output of just one repetition of each experiment due to resource constraints.
Moreover, while LongT5 with Local Attention often underperforms T5.1.1, LongT5 with TGlobal attention signiﬁcantly outperforms T5.1.1.
For example, considering the large size models, T5.1. was able only to scale up to an input length of 3k tokens, while the TGlobal model was able to reach 6k tokens, outperforming T5.1.1 at 4k token length (there was a dip at 6k token length, but we hypothesize this is just due to variance, as we only did one run for each conﬁguration).
For TriviaQA, we compare LongT5 with various top approaches on the leader board: BigBird-ETC (Zaheer et al., 2020a), Fusion-in-Decoder (Izacard and Grave, 2021), and ReadTwice (Zemlyanskiy et al., 2021).
As shown in Table 3, TriviaQA inputs are quite long, therefore being able to scale up both in model size and to 16k input length helps LongT achieve state-of-the-art.
Analysis 5. Input Length vs Speed
In order to evaluate the training speed and memory consumption of LongT5, compared to T5.1.1, we performed a series of training runs in the NQ data set starting at input length 512, and increasing Sequences per second F 1 ( s ho r t a n s w e r ) T5.1.1 large
LongT
[Figure] Local LongT5 large TGlobal 1k 2k 4k 3k 10k 6k 2k 8k 1k 1k 4k  the input length steadily until models ran out of memory on a 4x8 TPUv3 slice. Results are shown in Figure 3, which compares 6 different model conﬁgurations: T5.1.1 base, T5.1.1 large, LongT (base Local), LongT5 (large Local), LongT5 (base
TGlobal), and LongT5 (large TGlobal).
For each model conﬁguration, we show a curve plotting the number of sequences per second processed during training (speed, in the vertical axis) for each input length (horizontal axis).
Both axes are shown in logarithmic scale.
We can see that at shorter lengths (512), T5.1.1,
LongT5 Local, LongT5 TGlobal have similar speeds, but as we increase the sequence length,
LongT5 becomes signiﬁcantly faster.
For example at sequence length 2048, T5.1.1 base can only process 479 sequences per second, while LongT (base TGlobal) can process 765 and LongT5 (base
Local) can process 860.
The differences grow even larger as sequence length increases.
Another important fact that Figure 3 shows is that T5.1.1 models reach their out of memory point much earlier.
For example, we could only scale up to 6k tokens for T5.1.1 base.
On the other hand, LongT5 (base Local) can go up to 36k tokens in length, and LongT5 (base TGlobal) up to 12k.
Large models show a similar picture with T5.1. large going only up to 3k, but the LongT5 variants going to 10k (large Local) and 6k (large TGlobal).
5. Input Length vs Performance
This section presents a similar analysis, but where we plotted model speed versus performance in NQ (F1 score).
Results are shown in Figure 4 for models with large size.
Each point in the curves is annotated with the corresponding sequence length.
As Figure 4 shows, performance increases signiﬁcantly as input length increases, highlighting the beneﬁts of LongT5.
Moreover, input length by itself is not enough to achieve good performance in all datasets, and in particular, in the NQ dataset (used in this ﬁgure), using Local Attention significantly hurts performance when compared with
TGlobal or with T5.1.1.
So, even at very long input lengths, LongT5 with Local Attention just matches T5.1.1 with input length of 3k in NQ.
However, LongT5 with TGlobal attention outperforms T5.1.1.
Moreover, note that although the plot shows a few irregularities (such as 8k length for LongT with Local Attention, or 6k length with TGlobal
Attention), that is because the plot shows only the results of a single run, and hence there is some noise.
However, trends can clearly be seen.
5. Principle Sentences Generation vs. Span Corruption
As mentioned in section 3.2, we use PEGASUS
Principle Sentences Generation instead of default
Span Corruption used in T5 as our pre-training objective.
Table 5 shows our ablation study for ﬁne-tuning on NQ and arXiv from a model pretrained using the default Span Corruption objective, a model pre-trained with Principle Sentences
Generation, and a model pre-trained with both objectives.
The comparison is done on the dev set of the tasks, and with TGlobal base models.
Both pretraining and ﬁne-tuning on the models mentioned above are done with input sequence length 4096.
The table shows, even though Principle Sentences Generation was developed by Zhang et al. (2019a) as a pre-training strategy for summarization, it beneﬁts both summarization and QA tasks, but using both objectives together perform worse than just using PSG. Table 6 shows an additional ablation study with arXiv and PubMed, where we compare using regular T5.1.1 with Span Corruption compared to T5.1.1 pretrained with Principle Sentences Generation while using the same pre-training input sequence length of 512 (as was done in the original
T5.1.1 pre-training task). As expected, Principle Sentences Generation helped the model achieve better results compared to Span Corruption when seeing the same amount of pre-training data.
We [Table] [Table]
also compare this with dev scores from LongT with TGlobal attention at 4k and 16k input lengths, such that we can see having full attention will allow for better results, but being able to scale to longer input sequence lengths allows LongT5 to achieve its stronger results.
Related Work
Language model pre-training followed by task speciﬁc ﬁne-tuning has proven to be a powerful tool for numerous NLP tasks (Devlin et al., 2019;
Liu et al., 2019; Zhang et al., 2019b; Radford et al., 2019; Raffel et al., 2019a; Lewis et al., 2020; Joshi et al., 2020).
BERT (Devlin et al., 2019) introduced Mask Language Model (MLM), where a model predicts masked tokens given a sequence of text input.
Fine-tuning a pre-trained BERT model has led to improved performance on various NLP tasks.
However, MLM predictions are not made auto-regressively, which limits the capability of the
BERT family for generation tasks.
Raffel et al. (2019a) introduced the span corruption task in T as the pre-training objective, where a model predicts the masked token span using an autoregressive model.
It can handle the generation tasks as the pretraining is done in a generative way.
BART (Lewis et al., 2020) is similar to T5 but used a slightly different pre-training objective, in which spans are masked from the input but the complete output is predicted.
However, none of these works tried to investigate pre-training for very long sequence inputs.
They often use a transformer (Vaswani et al., 2017) architecture as backbone, the complexity of which is quadratic to the input length, making them impractical to model very long sequence input.
Long text modeling
An extensive amount of work has also been done for modeling long text like documents.
The work from Roy et al.
; Chen (2017); Wu et al.
(2018) obtained document embeddings from word-level embeddings.
Another line of research tries to model long documents through hierarchical training.
The work from Yang et al. ; Miculicich et al.
(2018) employed Hierarchical Attention Networks for document classiﬁcation and neural machine translation, and Guo et al.
(2019) proposed using a hierarchy network to build document embeddings on top of sentence embeddings for parallel document mining.
More recent research has been focusing on improving the memory and computation efﬁciency of transformer models (Tay et al., 2020b, 2021) for handling long input.
One type of such approaches is using non-full attention patterns to restrict the attention ﬁeld range, so that it reduces the attention complexity from O ( n 2 ) to O ( nlogn ) or
O ( n ) , including Sinkhorn (Tay et al., 2020a), Longformer (Beltagy et al., 2020), ETC (Ainslie et al., 2020), and BigBird (Zaheer et al., 2020a).
Another type of approaches is leveraging the low-rank approximation of the attention matrix, such as Linformer (Wang et al., 2020), Performer (Choromanski et al., 2021), Random Feature Attention (Peng et al., 2021), and LUNA (Ma et al., 2021).
Conclusion
This paper presented a new Transformer-based neural model called LongT5 , with which we have explored the effects of scaling both input length and model size at the same time.
Speciﬁcally, the main differences of LongT5 with respect to T5.1.1 are (1) a new scalable attention mechanism called Transient Global attention, which is a drop-in replacement to the standard T5 attention mechanism, and hence can be used without needing additional sideinputs to the model or modiﬁcations to the model inputs; and (2) using a PEGASUS-style Principle
Sentences Generation pre-training objective.
Via experimentation in several challenging summarization and question answering datasets, we have explored the performance gains that can be achieved by scaling both input length and model size, resulting in state-of-the-art results on several datasets: arXiv, PubMed, BigPatent, MediaSum, and TriviaQA.
As part of our future work, we would like to pursue several directions such as studying efﬁcient attention mechanisms in the decoder and decoder-toencoder attention pieces of the model (both Local
Attention and TGlobal attention are only applied to the encoder in LongT5 for now).
Additionally, we would like to incorporate additional long-input transformer ideas into the LongT5 architecture, that could further improve model efﬁciency. open do main question answering
- sion
Jared Kaplan, Sam McCandlish, Tom Henighan,
Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei arXiv preprint arXiv:2001.08361 . Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redﬁeld, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey,
Jacob Devlin, Kenton Lee, Kristina N. Toutanova,
Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob
Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions benchmark Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer. nual
Barcelona
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta robustly optimized bert pretraining approach.
Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting
Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021.
In Thirty-Fifth Conference on Neural Information
Processing Systems .
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018.
Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2947–2954, Brussels, Belgium. Association for Computational Linguistics. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ça˘glar G ˙ ulçehre, and Bing Xiang. 2016.
Abstractive text summarization using sequence-to-sequence
RNNs and beyond.
In Proceedings of The 20th
SIGNLL Conference on Computational Natural Language Learning , pages 280–290, Berlin, Germany.
Association for Computational Linguistics.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy
Schwartz, Noah Smith, and Lingpeng Kong. 2021.
Random feature attention.
In International Conference on Learning Representations .
Oﬁr Press, Noah A. Smith, and Mike Lewis. 2021.
Train short, test long: Attention with linear biases enables input length extrapolation.
Alec Radford, Jeff Wu, Rewon Child, David Luan,
Dario Amodei, and Ilya Sutskever.
Language models unsupervised
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J. Liu
2019a Exploring of - former CoRR , abs/1910.10683.
Adam Roberts,
Dara
Abnar Yikang Shen
Dara Bahri
Pham Ashish Vaswani Noam Shazeer Niki Parmar
Jakob
Uszkoreit Llion Jones Aidan N Gomez ukasz
Kaiser and Illia Polosukhin
Attention is all you need
Pro cessing Systems volume 30.
Curran Associates, Inc
Sinong Wang, Belinda Z. Li
Madian Khabsa, Han
Fang and Hao Ma.
Linformer: Self-attention with linear complexity.
Lingfei Wu, Ian En-Hsu Yen, Kun Xu, Fangli
Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep
Ravikumar, and Michael J. Witbrock.
Word mover’s embedding word2vec document embedding
In Proceedings of the 2018 Conference on Empirical Methods in Natural Language
Processing , pages 4524–
Association for Computational Linguistics.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman
Cohan.
PRIMER: Pyramid-based masked sentence pre-training for multi-document summarization. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy.
Hierarchical attention document classiﬁcation In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1480–1489, San Diego, California.
Association for Computational Linguistics.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey,
Joshua Ainslie, Chris Alberti, Santiago Ontañón,
Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020a
Big
, abs/2007. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed
In Advances in
Neural Information Processing Systems , volume 33, pages 17283–17297.
Curran Associates, Inc.
Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong,
Philip Pham, Ilya Eckstein, and Fei Sha. 2021.
Readtwice: Reading very large documents with memories.
Jingqing Zhang, Yao Zhao, Mohammad
Maosong
In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 1441–1451, Florence, Italy.
Association for Computational Linguistics.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng.
MediaSum large scale media interview dataset for dialogue summarization
In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies , pages 5927–5934,
Online.
Association for Computational Linguistics.
A Summarization Results
Table 8 shows the full set of results on the summarization datasets used in this paper.
This includes both standard T5 model (using version T5.1.1), T with PEGASUS Principle Sentences Generation pre-training, and LongT5 model.
As can be seen, scaling up the input size for the models helps achieve better performance metrics.
T5 models though struggle when scaling up to 4k for input, as the ﬁne-tuning task can take many days even when using a large topology of TPUv3.
When comparing regular T5.1.1 model with a T5.1.1 model using PEGASUS Principle Sentences Generation pre-training, the latter was able to achieve better results, with the results also improving as the input size scaled up.
This helps show that both using the latter pre-training objective along with scaling up allows us to get the best results from these models.
LongT5, despite having a reduced attention from using TGlobal attention, is able to get strong performance results due to both scaling up to larger inputs and leveraging the Gap Sentences Generation pre-training strategy.
B QA Results
Table 7 shows the full set of results comparing
T5.1.1 and LongT5 models on the QA datasets used in this paper.
For both NQ and TriviaQA in this comparison study, we use 90% of the ofﬁcial training set for training while using 10% as holdout dev set to ﬁne-tune the hyperparameters and training epoch, and use the ofﬁcial dev set to report the numbers in this table.
We run each model to the largest input length allowed before running out of memory on speciﬁc hardware conﬁgurationbase/large models on 4x8 TPUv3 with no model partitioning, and xl models on 8x16 TPUv3 withpartitions.
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] LongT [Table]
[Table] [Table] LongT [Table]
[Table] [Table]
BigPatent
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
LongT
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]