Efficient Classification of Long Documents Using Transformers (2022)

Abstract: Several methods have been proposed for classifying long textual documents using Transformers. However, there is a lack of consensus on a benchmark to enable a fair comparison among different approaches. In this paper, we provide a comprehensive evaluation of the relative efficacy measured against various baselines and diverse datasets — both in terms of accuracy as well as time and space overheads. Our datasets cover binary, multi-class, and multi-label classification tasks and represent various ways information is organized in a long text (e.g. information that is critical to making the classification decision is at the beginning or towards the end of the document). Our results show that more complex models often fail to outperform simple baselines and yield inconsistent performance across datasets. These findings emphasize the need for future studies to consider comprehensive baselines and datasets that better represent the task of long document classification to develop robust models.

## introduction transformer-based models @xcite have achieved much progress across many areas of nlp including text classification @xcite . however, such progress is often limited to short sequences because self-attention requires quadratic computational time and space with respect to the input sequence length. widelyused models like bert @xcite or roberta @xcite are typically pretrained to process up to 512 tokens. this is problematic because real-world data can be arbitrarily long. as such, different models and strategies have been proposed to process longer sequences. in particular, we can identify a few standard approaches for the task of long document classifi-cation. the simplest approach is to truncate long documents -using bert or roberta on the first 512 tokens is often used as a baseline. more efficient transformer models like longformer @xcite and big bird @xcite use sparse self-attention instead of full selfattention to process longer documents (e.g. up to 4,096 tokens). other approaches process long documents in their entirety by dividing them into smaller chunks (e.g. @xcite . an alternative idea proposed by recent work is to select sentences from the document that are salient to making the classification decision @xcite . however, the relative efficacy of these models is not very clear due to a lack of consensus on benchmark datasets and baselines. @xcite propose a benchmark for comparing transformers that can operate over long sequences, but this only includes a single, simulated foot_0 long document classification task. novel variants of efficient transformers are often compared to a bert/roberta baseline only, without much comparison to other transformer models designed for the task (e.g. @xcite . conversely, models designed for long document classification often focus exclusively on state-of-the-art models for particular datasets, and do not consider a bert/roberta baseline or any other transformer models (e.g. @xcite . this paper provides a much-needed comprehensive comparison among existing models for long document classification by evaluating them against unified datasets and baselines. we compare models that represent different approaches on various datasets and against transformer baselines. our datasets cover binary, multi-class, and multi-label classification. we also consider different ways information that is relevant to the classification is organized in texts (e.g. in the beginning or toward the end) and how this affects model performance. we also compare the models in terms of their training time, inference time, and gpu memory requirements to account for additional complexity that some of the models have relative to a bert baseline. this allows us to compare the practical efficacy of the models for real-world usage. our results show that more sophisticated models are often outperformed by simpler models (often including a bert baseline) and yield inconsistent performance across datasets. based on these findings, we highlight the importance of considering diverse datasets while developing models, especially those that represent different ways key information is presented in long texts. additionally, we recommend that future research should also always include simpler baseline models. to summarize, our contributions are: ## methods in this paper, we compare models representing different approaches to long document classification @xcite @xcite on unified datasets and baselines. ## discussion and recommendations our results show that complex models for long document classification do not consistently outperform simple baselines. the fact that the existing models were often outperformed by the simplest bert baseline suggests that the datasets tend to have key information accessible in the first 512 tokens. this is somewhat expected as the first two sections of eurlex are reported to carry the most information @xcite and 20newsgroups contains mostly short documents. including these datasets to evaluate models for long document classification is still reasonable given that a good model should work well across different settings. however, these datasets alone do not represent various ways information is presented in long texts. instead, future studies should evaluate their models across various datasets to create robust models. while it is often difficult to obtain datasets suited for long document classification, our modifications of existing datasets may provide ways to repurpose existing datasets for future studies. we invert the order of the sections of eurlex to create the inverted eurlex dataset, where key information is likely to appear toward the end of each document. our results in many of these findings were revealed only due to the choice of relevant baselines, and future work will benefit from including these as well. a bert/roberta baseline is essential to motivate the problem of long document classification using transformers and reveal how much information is retrievable in the first 512 tokens. bert+textrank and bert+random are stronger baselines that of-ten outperform more complex models that select key sentences. in fact, they outperformed cogltx on five of the six datasets. ## conclusion several approaches have been proposed to use transformers to classify long documents, yet their relative efficacy remains unknown. in this paper, we compare existing models and baselines on various datasets and in terms of their time and space requirements. our results show that existing models, while requiring more time and/or space, do not perform consistently well across datasets, and are often outperformed by baseline models. future studies should consider the baselines and datasets to establish robust performance.