Revisiting Transformer-based Models for Long Document Classification (2022)

Abstract: The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods.We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.

## introduction natural language processing has been revolutionised by the large scale self-supervised pretraining of language encoders @xcite , which are fine-tuned in order to solve a wide variety of downstream classification tasks. however, the recent literature in text classification mostly focuses on short sequences, such as sentences or paragraphs @xcite @xcite , which are sometimes misleadingly named as documents. 2 the transition from short-to-long document classification is non-trivial. one challenge is that bert and most of its variants are pre-trained on sequences containing up-to 512 tokens, which is not a long document. a common practice is to truncate actually long documents to the first 512 tokens, which allows the immediate application of these pre-trained models @xcite . we believe that this is an insufficient approach for long document classification because truncating the text may omit important information, leading to poor classification performance (figure 1 ). another challenge comes from the computational overhead of vanilla transformer: in the multi-head self-attention operation @xcite , each token in a sequence of n tokens attends to all other tokens. this results in a function that has o(n 2 ) time and memory complexity, which makes it challenging to efficiently process long documents. in response to the second challenge, longdocument transformers have emerged to deal with long sequences @xcite . however, they experiment and report results on non-ideal long document classification datasets, i.e., documents on the imdb dataset are not really long -fewer than 15% of examples are longer than 512 tokens; while the hyperpartisan dataset only has very few (645 in total) documents. on datasets with longer documents, such as the mimic-iii dataset @xcite with an average length of 2,000 words, it has been shown that multiple variants of bert perform worse than a cnn or rnn-based model @xcite @xcite @xcite . we believe there is a need to understand the performance of transformer-based models on classifying documents that are actually long. in this work, we aim to transfer the success of the pre-train-fine-tune paradigm to long document classification. our main contributions are: ## problem formulation and datasets we divide the document classification model into two components: (1) a document encoder, which builds a vector representation of a given document; and, (2) a classifier that predicts a single or multiple labels given the encoded vector. in this work, we mainly focus on the first component: we use transformer-based encoders to build a document representation, and then take the encoded document representation as the input to a classifier. for the second component, we use a tanh activated hidden layer, followed by the output layer. output probabilities are obtained by applying a sigmoid (multi-label) or softmax (multi-class) function to output logits. 3 we mainly conduct our experiments on the mimic-iii dataset @xcite , where researchers still fail to transfer "the magic of bert" to medical code assignment tasks @xcite . ## approaches in the era of transformer-based models, we identify two representative approaches of processing long documents in the literature that either acts as an inexpensive drop-in replacement for the vanilla self-attention (i.e., sparse attention) or builds a taskspecific architecture (i.e., hierarchical transformers). ## experimental setup backbone models we mainly consider two models in our experiments: longformer-base @xcite , and roberta-base @xcite which is used in hierarchical transformers. evaluation metrics for the mimic-iii (multilabel) dataset, we follow previous work @xcite and use micro-averaged auc (area under the receiver operating characteristic curve), macro-averaged auc, micro-averaged f 1 , macro-averaged f 1 and precision@5-the proportion of the ground truth labels in the top-5 predicted labels-as the metrics. we report micro and macro averaged f 1 for the ecthr (multilabel) dataset, and accuracy for both hyperpartisan (binary) and 20 news (multiclass) datasets. ## experiments we conduct a series of controlled experiments to understand the impact of design choices in different trldc models. bringing these optimal choices all together, we compare trldc against the state of the art, as well as baselines that only process upto 512 tokens. finally, based on our investigation, we derive practical advice of applying transformerbased models to long document classification regarding both effectiveness and efficiency. task-adaptive pre-training is a promising first step. domain-adaptive pre-training (dapt) -the continued pre-training a language model on a large corpus of domain-specific text -is known to improve downstream task performance @xcite . however, task-adaptive pre-training (tapt) -continues unsupervised pre-training on the task's datais comparatively less studied, mainly because most of the benchmarking corpora are small and thus the benefit of tapt seems less obvious than dapt. we believe document classification datasets, due to their relatively large size, can benefit from tapt. on both mimic-iii and ecthr, we continue to pre-train longformer and roberta using the masked language modelling pre-training objective (details about pre-training can be found at appendix 9.3). we find that task-adaptive pretrained models substantially improve performance on mimic-iii (figure 5 (a) and (b)), but there are smaller improvements on ecthr (figure 5 (c ) and (d)). we suspect this difference is because legal cases (i.e., ecthr) are publicly available and have been covered in pre-training data used for training longformer and roberta, whereas clinical notes (i.e., mimic-iii) are not @xcite . see appendix 9.5 for a short analysis on this matter. we also compare our tapt-roberta against publicly available domain-specific roberta, trained from scratch on biomedical articles and clinical notes. results (figure 8 in appendix) show that tapt-roberta outperforms domain-specific base model, but underperforms the larger model. ## practical advice we compile several questions that practitioners may ask regarding long document classification and provide answers based on our results: q1 when should i start to consider using long document classification models? a we suggest using trldc models if you work with datasets consisting of long documents (e.g., 2k tokens on average). we notice that on 20 news dataset, the gap between baselines that process 512 tokens and long document models is negligible. 9 q2 which model should i choose? longformer or hierarchical transformers? a we suggest longformer as the starting point if you do not plan on extensively tuning hyperparameters. we find the default config of longformer is robust, although it is possible to set a moderate size (64-128) of local attention window to improve 9 although hyperpartisan is a widely used benchmark for long document models, we do not recommend drawing practical conclusions based on our results because we observe high variance when we run experiments using different gpus or cuda versions. we attribute this may to the small size (65) of its test set and the subjectivity of the task. efficiency without sacrificing effectiveness, and a small number of additional global attention tokens to make the training more stable. on the other hand, hierarchical transformers may benefit from careful hyperparameter tuning (e.g., document splitting strategy, using lwan). we suggest splitting a document into small non-structure-derived segments (e.g., 128 tokens) which overlap as a starting point when employing hierarchical transformers. we also note that the publicly available longformer models can process sequences up-to 4096 tokens, whereas hierarchical transformers can be easily extended to process much longer sequence. ## related work long document classification document length was not a point of controversy in the pre-neural era of nlp, where documents are encoded with bagof-word representations, e.g., tf-idf scores. the issue arised with the introduction of deep neural networks. @xcite use cnn and bil-stm based hierarchical networks in a bottom-up fashion, i.e., first encode sentences into vectors, then combine those vectors in a single document vector. similarly, @xcite incorporate the attention mechanism when constructing the sentence and document representation. hierarchical variants of bert have also been explored for document classification @xcite , abstractive summarization @xcite , semantic matching @xcite . @xcite propose specialised pre-training tasks to explicitly capture sentence relations within a document. a very recent work by @xcite shows that trldc do not perform consistently well across datasets that consist of 700 tokens on average. methods of modifying transformer architecture for long documents can be categorised into two approaches: recurrent transformers and sparse attention transformers. the recurrent approach processes segments moving from left-to-right @xcite . to capture bidirectional context, @xcite propose a retrospective mechanism in which segments from a document are fed twice as input. sparse attention transformers have been explored to reduce the complexity of selfattention, via using dilated sliding window @xcite , and locality-sensitive hashing attention @xcite . recently, the combination of local (window) and global attention are proposed by @xcite and @xcite , which we have detailed in section 3. ## conclusions transformers have previously been criticised for being incapable of long document classification. in this paper, we carefully study the role of different components of transformer-based long document classification models. by conducting experiments on mimic-iii and other three datasets (i.e., ecthr, 20 news and hyperpartisan), we observe clear improvements in performance when a model is able to process more text. firstly, longformer, a sparse attention model, which can process up to 4096 tokens, achieves competitive results with cnn-based models on mimic-iii; its performance is relatively robust; a moderate size of local attention window (e.g., 128) and a small number (e.g., 16) of evenly chosen tokens with global attention can improve the efficiency and stability without sacrificing its effectiveness. secondly, hierarchical transformers outperform all cnn-based models by a large margin; the key design choice is how to split a document into segments which can be encoded by pre-trained models; although the best performing segment length is dataset dependent, we find splitting a document into small overlapping segments (e.g., 128 tokens) is an effective strategy. taken together, these experiments rebut the criticisms of transformers for long document classification.