The NLP Task Effectiveness of Long-Range Transformers (2023)

Abstract: Transformer models cannot easily scale to long sequences due to their O(Nˆ2) time and space complexity. This has led to Transformer variants seeking to lower computational complexity, such as Longformer and Performer. While such models have theoretically greater efficiency, their effectiveness on real NLP tasks has not been well studied. We benchmark 7 variants of Transformer models on 5 difficult NLP tasks and 7 datasets. We design experiments to isolate the effect of pretraining and hyperparameter settings, to focus on their capacity for long-range attention. Moreover, we present various methods to investigate attention behaviors to illuminate model details beyond metric scores. We find that the modified attention in long-range transformers has advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error.

## introduction transformer-based models @xcite have advanced the state of the art in natural language processing. however, their quadratic time and space complexity hinder their application on long texts. various proposals have been made to address these concerns @xcite , with mathematical guarantees on improved time or space. these models have been evaluated primarily via perplexity @xcite and non-nlp benchmarks @xcite . these metrics may not be ideal @xcite and may not reflect performance on complex nlp tasks @xcite . we argue these metrics have not been sufficient for the development of efficient transformers and their practical application on long texts, and that existing benchmarks are insufficient guides for architecture selection. it is not straightforward to have a fair and sideby-side comparison among those models due to the differences between their pretraining and hyperparameter settings @xcite , and the metrics alone cannot convey detailed information about the self-attention blocks @xcite . we wish to fairly validate the effectiveness of long-range attention techniques, and to uncover the underlying factors behind model behaviors. we critique the reliance on perplexity evaluations in previous work, experimenting with five difficult, long-text nlp tasks. these tasks cover typical nlp modeling scenarios: token or span-level prediction, sequencelevel classification, and sequence-to-sequence generation. to our knowledge, this is the first work to evaluate long-range transformers on such a wide spectrum of representative nlp tasks. to verify the key features of long-range transformers, we ablate distant attention to measure what they gain from long-range mechanisms. for models without pretrained checkpoints, we migrate parameters from their prototype models for fairness. we cover 3 main kinds of long-range transformers, including pattern-, recurrence-, and kernel-based methods. to our knowledge, we are the first to adopt all these methods to probe transformers. to investigate the relationship between performance and document lengths we break down the metric with a customized algorithm @xcite . also, we use entropy and attribution analysis @xcite to test the effectiveness of cached memories in recurrent transformers and the global tokens for query-based problems. we find that long-range context brings performance gains to transformers in some cases, which we attribute to more selective attention, especially for query-based tasks like qa. surprisingly we observe that some long-range models do not effectively utilize distant information, and the accumulated error of approximation is unacceptable. we hope this analysis helps practitioners better understand the current state of the art of long-range attentions and suggests paths for future research. 2 background ## discussion researchers have proposed many innovative methods for efficient self-attention over long sequences. the key ideas work as desired in certain cases, though we demonstrate several drawbacks. surprisingly, pattern-based methods, as the most popular approach, are not necessarily benefited from long-range attention in the general case. larger sliding windows are helpful, but the benefit would quickly saturate or become negative ( however, when a small portion of guidance text (e.g. query in qa) exists, setting it as global tokens can make it more attended and significantly boost the performance (section 5.3). when such text doesn't exist, setting all tokens as global would hurt the performance ( the memory of recurrence models generally improves performance, proving historical hidden states are beneficial for transformers in various tasks. however, xlnet does not fully exploit the history tokens, with distant information less attended (section 5.1). we speculate that it is because xlnet is pretrained to predict masked tokens, which does not frequently need the participation of long-range context @xcite . also, the stop-gradient trick may hinder the model from effectively attending to memories. the approximation of kernel-based methods works very well for shallow networks, but faces serious error accumulation problems when transformer layers are deeply stacked, which cannot be remedied by having high-dimensional random features (section 5.4). the resulting performance drop is not acceptable even for the "base" version of transformer encoders with 12 layers ( ## conclusion we conduct experiments with various long-range transformers on nlp tasks that involve long sequences, trying to fairly evaluate their long-range attention ability. while some methods are validated on certain tasks, we also find some previously unrecognized drawbacks. we further analyze the attention behaviors of these transformers with metric breakdown, attribution analysis, and entropy analysis, revealing the performance of those models might be correlated with the attribution of distant tokens, selectivity of attentions, or the approximation errors. we hope our work would shed light on the future development of long-range transformers. model selection takeaways based on our findings, we have the following suggestions. for common tasks, such as sequence classification or token-level prediction, it is still competitive to chunk the inputs and apply short-range transformers. when explicit guiding text, such as queries, exists, pattern-based models with global token mechanism is preferred. for seq2seq problems, long-range transformers with pretrained checkpoints deliver better performance. experimental bias due to the lack of pretrained checkpoints for general purposes, we focus on representative instead of each type of transformer variant. it is possible that these observations are particular to specific artifacts and implementations considered here. our goal is foremost to provide a roadmap for continued study on questions raised in this article, with new architectures being evaluated in the future by model developers themselves. for similar reasons, existing artifacts are biased towards english, as are many of the datasets employed in this study. we do not believe our findings are specific to english, but it remains for future work in long range transformer evaluation to extend our analysis into multilingual conditions. language bias for similar reasons, existing artifacts are biased towards english, as are many of the datasets employed in this study. we do not believe our findings are specific to english, but it remains for future work in long-range transformer evaluation to extend our analysis into multilingual conditions.