Long Text Classification using Transformers with Paragraph Selection Strategies (2023)

Abstract: In the legal domain, we often perform classification tasks on very long documents, for example court judgements. These documents often contain thousands of words, so the length of these documents poses a challenge for this modelling task. In this research paper, we present a comprehensive evaluation of various strategies to perform long text classification using Transformers in conjunction with strategies to select document chunks using traditional NLP models. We conduct our experiments on 6 benchmark datasets comprising lengthy documents, 4 of which are publicly available. Each dataset has a median word count exceeding 1,000. Our evaluation encompasses state-of-the-art Transformer models, such as RoBERTa, Longformer, HAT, MEGA and LegalBERT and compares them with a traditional baseline TF-IDF + Neural Network (NN) model. We investigate the effectiveness of pre-training on large corpora, fine tuning strategies, and transfer learning techniques in the context of long text classification.

## introduction the performance of text classification methods has improved significantly over the last decade for text instances containing less than 512 characters. due to the high computational cost of processing longer text instances, this limitation is introduced in the most recent transformer models. to alleviate this problem and improve the classification of long texts, researchers tried to address the root causes of the computational cost and proposed the optimization of the attention mechanism @xcite , which is a key element of any transformer model. this research undertakes a comparative study of different ways of doing long text, multi-label classification on multiple legal datasets. we take existing approaches from literature and try an alternate approach of our own. the end goal is to strike a balance between model performance and computational efficiency. ## related work natural language processing (nlp) is a subfield of computer science and linguistics that focuses on processing language, including text. probably the most basic task within this field is text classification, which given text as an input the objective is to classify it among a set of categories. to perform this classification, the model needs to extract features from the text and preferably to understand correlations between its different parts, in a similar way that us humans do where we connect different entities in a sentence to extract meaning from them. some approaches treated text with a bag-of-words (bow) model, as introduced by harris @xcite , where grammar and word order are disregarded and we consider text as a set of words with their multiplicity. related to this concept, a popular idea to model text which became popular during the era of @xcite is tf-idf (term frequency -inverse document frequency). first described in 1972, tf-idf @xcite as its name indicates consists on vectorizing text by reflecting how important a term is to a document in a collection or corpus. its main intuition is to divide the occurrences of the term in the text by the occurrences of this term in the whole corpus, this way we get the relative importance of a given term. in the decade of the 2010s we dive into the era of the neural models for nlp, starting in 2013 with the word2vec embeddings @xcite used to train a recurrent neural network (rnn). this idea of training representation of words and terms quickly became very popular and since then there have appeared many methods to obtain these representations in an unsupervised way to then use them for downstream tasks, for example glove @xcite and fasttext @xcite . a big revolution in nlp started with the presentation of the transformer architecture by vaswani @xcite and its attention mechanism. this architecture later evolved into models that have also been pre-trained to obtain representations of the input tokens, as for example bert @xcite and roberta @xcite . legal-bert @xcite consists in bert heavily pre-trained with legal data, and this specialization has been proved to improve the performance of the model in downstream tasks that use this language. nonetheless, in this study we do not include models pre-trained with legal corpora because it would include a bias when comparing with other architectures that do not have a public version pre-trained on this type of language. the limitation of these mentioned models is that due to the quadratic computational cost the attention mechanism, they allowed up to 512 tokens in the input, while legal documents can easily be formed by thousands of them. this problem has led to two alternatives: truncation methods in order to process a part of the text, for example the first 512 tokens, and to derive the classification from that part and models that contain a more efficient attention mechanism rather than computing the pair-wise attention between all the tokens in the input. many sparse-attention models have been proposed, the most popular ones being longformer @xcite and most recently mega @xcite has proved to be the best in the benchmark for long sequence modeling "long range arena" @xcite . this latter one, despite being the best performing model for long sequences has not become very popular due to not being a public model that has been largely pre-trained. another popular approach to process long text are hierarchical attention transformers (hat) @xcite , which aim at processing the whole text by encoding chunks of typically 512 or less tokens at each step and then encoding these together in a hierarchical way, through cross-segment attention blocks @xcite . some research has been done in the direction of putting all these different methods with different intuitions in a fair comparison to solve a multi-label classification task. some of these studies include textguide @xcite which compared the performance of different truncation methods in selecting the text to train a language model, park @xcite studied and compared different efficient lms and chalkidis @xcite compared longformer to hats for long document classification. nonetheless, no work has obtained definitive and actionable conclusions and more importantly no other work has compared traditional approaches such as tf-idf vectorization followed by a neural network (nn) for classification with truncation methods to train popular language models such as bert and sparse-attention models designed to process long documents at once. ## research questions few works have focused on widely comparing different text classification approaches, so this method aims at setting the baseline for choosing an approach for multi-label / multi-class text classification given a set of constraints. so, the research questions that will be addressed are: ## datasets six datasets with long documents have been chosen to compare the performance of the different approaches: canadian abridgement dataset: this dataset contains 369,943 canadian court judgements and their classification as per the abridgement classification taxonomy as defined by the editorial team for checkpoint. the classification follows a hierarchy, starting with 55 broad classes (subject titles). these classes have further sub-categories going up to 6 levels. each unique classification up to the 6th level is treated as one class. the data belongs to the period 2000-2021. we have used only those cases whose classification belongs to the latest available version of the taxonomy. checkpoint tax type classification: thomson reuters monitors multiple sources (tax courts, rulings, official materials etc.) in order to incorporate the latest changes in law or guidance into checkpoint editorial content. documents from posture 50k: @xcite this data-set is publicly accessible and has been made available by thomson reuters. it is designed for the purpose of identifying the legal procedural postures involved in legal motions within a legal case. to illustrate, a plaintiff might request an appellate court to overturn a specific decision made by a lower court judge regarding a motion, which is referred to as an "on appeal" procedural posture. the dataset comprises 50,000 legal opinions, representing real-world legal cases in the united states, along with their corresponding postures. the majority of these cases span from the year 2013 to 2020, with just three cases occurring prior to 2013. scotus: @xcite this constitutes one of the six datasets within lexglue. it encompasses court opinions issued by the united states supreme court (scotus). the primary challenge within this dataset involves single-label multi-class classification, with the aim of forecasting the pertinent issue area for each court opinion. there are 14 distinct classes that align with specific issue areas, collectively encompassing 278 issues centered around the subject matter of the legal dispute. mimic-iii dataset: (johnson et al., 2016) contains approx. 50k discharge summaries from us hospitals. each summary is annotated with one or more codes (labels) from the icd-9 taxonomy. the input of the model is a discharge summary, and the output is the set of the relevant 1st level icd-9 (19 in total) codes. ecthr (task b): @xcite this is one of the six data-sets from lexglue. it com-prises approximately 11,000 cases sourced from the public database of the european court of human rights (ecthr). for each case, the dataset includes a collection of factual paragraphs extracted from the case description. each case is associated with specific articles of the european convention on human rights (echr) that were purportedly violated, as determined by the court. in terms of model input, it consists of the factual paragraph list for a given case, while the model's output comprises the set of articles that are alleged to have been violated. the first two datasets are internal from thomson reuters, while the other four are publicly available. mimic-iii contains medical documents and the rest contain legal text, which is our focus area. ## conclusions and future work we recommend the following model training guidelines: future work could be done in this topic by comparing the presented approaches to newer models that have come out after this study has been done. some of these models can be: