Efficient Long-Text Understanding with Short-Text Models (2023)

Abstract: Transformer-based pretrained language models (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scientific articles, and long documents due to their quadratic complexity. While a myriad of efficient transformer variants have been proposed, they are typically based on custom implementations that require expensive pretraining from scratch. In this work, we propose SLED: SLiding-Encoder and Decoder, a simple approach for processing long sequences that re-uses and leverages battle-tested short-text pretrained LMs. Specifically, we partition the input into overlapping chunks, encode each with a short-text LM encoder and use the pretrained decoder to fuse information across chunks (fusion-in-decoder). We illustrate through controlled experiments that SLED offers a viable strategy for long text understanding and evaluate our approach on SCROLLS, a benchmark with seven datasets across a wide range of language understanding tasks. We find that SLED is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step.

## introduction transformer-based pretrained language models @xcite @xcite @xcite have been widely successful across all areas of natural language understanding. however, applying them over long texts (such as stories, scripts, or scientific articles) is prohibitive due to their quadratic complexity in the input length. to bridge this gap, recent work has developed more efficient transformer variants @xcite @xcite and applied them over long-range language understanding tasks @xcite . however, most efficient transformers use specialized architectures with custom implementations that are not guaranteed to scale as well as vanilla transformers @xcite . moreover, they require an expensive pretraining step and do not exploit off-the-shelf pretrained lms that were trained for short texts. to date, their performance on long texts has not matched the success of their short-range counterparts. in this work, we present sled: sliding-encoder and decoder, a simple yet powerful method for applying off-the-shelf pretrained encoder-decoder models on long text problems, with a linear time and space dependency. specifically (see figure 2 ), we partition long documents into overlapping chunks of tokens of constant length and encode each chunk independently with an already-pretrained encoder. then, a pretrained decoder attends to all contextualized input representations to generate the output. our main assumption is that input tokens can be contextualized through their local surrounding (using a short-text lm), and any global cross-chunk reasoning can be handled by the decoder, similar to fusion-in-decoder (fid) @xcite . our approach can be readily applied to any pretrained encoder-decoder lm such as t5 @xcite and bart @xcite (but is not applicable to decoder-only @xcite or encoder-only models @xcite ). we evaluate sled on a wide range of language understanding tasks. to substantiate sled's adequacy for text processing, we perform controlled experiments over modified versions of squad 1.1 @xcite and hotpotqa @xcite to show that sled can (a) find relevant information that is embedded within a long text sequence and (b) fuse information from chunks that were encoded separately. our main evaluation is over scrolls, a recently-released benchmark that includes 7 longrange tasks across question answering (qa), figure 1 : models' scrolls score @xcite as a function of parameter count. plugging existing pretrained lms into the sled framework dramatically improves their scrolls score (arrows from blue circles to orange stars). gray triangles indicate models with dedicated pretraining for capturing long-range dependencies. bart large -sled is competitive with longt5 base @xcite and ul2 @xcite (which has 50x more parameters), and slightly lags behind larger longt5 models. summarization, and natural language inference (nli). we show (figure 1 ) that taking a pre-trained encoder-decoder model, such as bart @xcite or t5 @xcite , and embedding it into sled's framework results in dramatic improvement in performance (6 points on average across models). moreover, bart large -sled's performance is comparable to longt5 base @xcite , a model that was specifically pretrained to handle long-range dependencies, and surpasses ul2 @xcite , which contains 50x more parameters. importantly, sled-based models can use any future pretrained lm out-of-the-box without requiring additional pretraining to further improve performance. due to its simplicity, sled can also be used as a diagnostic tool for analyzing long-range benchmarks. we analyze the seven datasets in scrolls through the lens of sled and show which datasets require the input to be contextualized with remote tokens. specifically, we find that in qa and nli tasks, relatively local contextualization is sufficient for high performance. while sled is similar to fid from a technical standpoint, past usage of fid has centered around open-domain question answering @xcite , where unrelated passages are naturally encoded independently. here, we test fusion-in-decoder on long documents, where local encoding of chunks is a modeling assumption that needs testing. in recent work, @xcite proposed a similar architecture to tackle long inputs from qmsum @xcite , but did not systematically analyze it. we standardize this methodology for the first time, and extensively analyze the effectiveness of fid for encoding long documents across multiple tasks. to summarize, our main contributions are: ## background recent advances in natural language processing have been by and large fueled by the transformer architecture @xcite . a core component of the transformer is the self-attention layer where every input token ''attends'' to every other token to produce its contextualized representation. this results in quadratic time and space dependency w.r.t. the length of the input, limiting the ability of transformers to process long sequences. this long-text limitation has sparked ample interest in developing efficient transformer variants. one prominent family of methods is based on sparse attention, where each token attends to a constant number of other tokens, overcoming the quadratic dependency. tokens typically attend either to their local surrounding @xcite @xcite or to tokens that are semantically similar @xcite . moreover, a constant number of global tokens that attend to and are attended by all input tokens are often added to each attention sub-layer. recent analyses @xcite have shown that sparse transformers with local attention are competitive with other variants on multiple language understanding tasks. our method, sled, falls into the family of local attention variants. however, unlike prior work, sled re-uses and extends existing short-range encoder-decoder models, and does not require specialized pretraining or dedicated cuda implementations. in most local attention variants, for example, led @xcite , attention is local per-layer, but the receptive field of tokens grows across layers. in sled, which we describe next, tokens have access to the same number of tokens, independent of a layer's depth, which enables better parallelization. for a survey on the families of efficient transformers, see @xcite . for an in-depth comparison of sled and led, we refer to appendix b. ## method in this work, we propose a simple approach for avoiding transformer's quadratic complexity, motivated by the locality of information assumption: in an encoder-decoder architecture, the encoder can effectively contextualize input tokens with local context only, leaving long-range dependencies to be handled by the decoder. sled relies on said modeling assumption to encode shorter chunks independently and perform fusion of information in the decoder @xcite . we now describe the sled model in detail. input sled uses a pretrained encoder-decoder model m as a backbone. sled receives a tokenized document of length n (blue squares in figure 2 ), and an optional short tokenized prefix of length m n, typically representing a question about the document, an instruction to perform some generation task, or a hypothesis (orange squares in figure 2 ). unlike static task-specific prefixes (e.g., ''summarize''), sled supports also sample-specific prefixes that are part of the input (e.g., the question in qa datasets). steps sled follows the following steps: sled requires handling a few edge cases, namely, dealing with the first and last chunk that do not have bidirectional context. we refer to appendix a for these details. sled's complexity sled divides an input of length n to c chunks of size c. since ρ ∈ [0, 0.5], it follows that @xmath0, 2n c . while the complexity of encoding each chunk is quadratic in c due to self-attention, c n is constant and thus the memory and compute dependency is linear in n. foot_1 in particular, the complexity to encode the input with a model of l attention layers is: decoding is done as proposed by @xcite , thus requiring o(nk + k 2 ) memory. assuming a constant output sequence length k n, this remains linear in n. ## efficacy of fusion in decoder as mentioned ( §3), sled relies on the assumption that chunks can be encoded independently and fusion across them can be delegated to the decoder (locality of information assumption). this is similar to the fusion-in-decoder approach, introduced by @xcite for open-domain question answering (odqa). however, there, the encoder-decoder receives a set of independent passages and needs to generate an answer that can typically be extracted from a single passage. here, we extend the scope of fid by applying it over a single, long, and coherent input that potentially requires global contextualization. to demonstrate the viability of fid for long text language tasks, we design two controlled experiments that quantify the extent to which fid can perform two operations at the heart of long-text processing. first, can fid find a ''needle-in-a-haystack'', that is, locate a piece of short information embedded in long text, disregarding irrelevant information. second, can fid ''piece the puzzle'' and fuse two pieces of information that are encoded independently when generating an output. ## experiments we evaluate sled on scrolls @xcite , a recently proposed benchmark for evaluating long text understanding. scrolls contains seven datasets that span three different language understanding tasks: for quality, we consider the four answer options part of the question. for each task, we use the official evaluation metrics defined in scrolls, which are based on the metrics from the original datasets. ## limitations we present sled as a simple and effective method to extend the capabilities of pretrained short-text models to long-text tasks. despite its impressive empirical performance on scrolls, sled suffers from two disadvantages which may limit its applicability to some long-range tasks. long output to obtain linear complexity, sled assumes the output length k is constant. this is because the decoder uses quadratic self-attention over the output, on top of o(nk) cross-attention between the output and input. while most current long-text tasks follow this assumption, future tasks, such as academic reports or script writing, may require long text generation. this limitation is not unique to sled and affects other long-range transformers including longt5 and led. aside from finetuning, this also affects pretraining models on long inputs with self-supervised losses such as span-corruption @xcite or denoising @xcite , which require the decoder to process an output that is linear in the length of the input.