Processing Long Legal Documents with Pre-trained Transformers: Modding L egal BERT and Longformer (2022)

Abstract: Pre-trained Transformers currently dominate most NLP tasks. They impose, however, limits on the maximum input length (512 sub-words in BERT), which are too restrictive in the legal domain. Even sparse-attention models, such as Longformer and BigBird, which increase the maximum input length to 4,096 sub-words, severely truncate texts in three of the six datasets of LexGLUE. Simpler linear classifiers with TF-IDF features can handle texts of any length, require far less resources to train and deploy, but are usually outperformed by pre-trained Transformers. We explore two directions to cope with long legal texts: (i) modifying a Longformer warm-started from LegalBERT to handle even longer texts (up to 8,192 sub-words), and (ii) modifying LegalBERT to use TF-IDF representations. The first approach is the best in terms of performance, surpassing a hierarchical version of LegalBERT, which was the previous state of the art in LexGLUE. The second approach leads to computationally more efficient models at the expense of lower performance, but the resulting models still outperform overall a linear SVM with TF-IDF features in long legal document classification.

## introduction transformer-based models @xcite , like bert @xcite , roberta @xcite , and their numerous offspring, currently dominate most natural language processing (nlp) tasks. these models are pre-trained on very large corpora using generic tasks (e.g., masked token prediction) that do not require human annotations, and are then fine-tuned (further trained) on typically much smaller task-specific datasets with manually annotated ground truth. the quadratic complexity of their attention mechanisms, however, imposes limits on the maximum input length (512 sub-word tokens in bert, roberta), which are often too restrictive in the legal domain, where longer documents are common. the same restrictions apply to legalbert @xcite , a bert variant pre-trained on legal corpora. even the sparse-attention longformer @xcite , a well-known transformer that increases the maximum input to 4,096 sub-words, severely truncates texts in three of the six datasets (see fig. 2 ) of the lexglue legal nlp benchmark @xcite . on the other hand, simpler linear classifiers with tf-idf features @xcite , which were very common before deep learning, can handle texts of any length, at least in text classification tasks, require far less resources to train and deploy, but are nowadays usually outperformed by pre-trained transformers. motivated by these observations, we explore two directions to better cope with long legal texts: (i) we modify a longformer warm-started from legal-bert to handle even longer texts (up to 8,192 sub-words), a resource-intensive direction that further increases the parameters and processing time of large sparse-attention transformer models; and (ii) we modify legalbert to use tf-idf representations, which allows processing longer texts without increasing the model sizes. the first approach is the best overall in terms of performance, surpassing a hierarchical version of legalbert @xcite , which was the previous state of the art in lexglue. the second direction leads to computationally more efficient models at the expense of lower performance, but still outperforms overall a linear support vector machine (svm) @xcite with tf-idf features in long document classification. ## models considered we discuss models @xcite evaluated on lexglue as baselines, and models we introduce. the lexglue baselines also included roberta @xcite , deberta @xcite , bigbird @xcite , and caselaw-bert @xcite , which are not considered here. @xcite found roberta and deberta to be better than bert on lexglue, but worse than legalbert; no legally pre-trained variants of roberta and deberta are available. bigbird and caselaw-bert were found to be overall slightly worse than longformer and legalbert, respectively, on lexglue. ## conclusions and future work concluding, we presented bow variants of legal-bert, which remove duplicate words and consider tf-idf scores by reordering the remaining words and/or by employing a tf-idf embedding layer. these variants are more efficient than the original legalbert and still overall outperform a tf-idfbased svm in long legal document classification. we also modified longformer to handle even longer texts (up to 8,192 sub-words), use additional global tokens, and also showed the positive effect of warm-starting it from legalbert. unlike the bow models, this is a resource-intensive direction, with substantial improvements compared to the original longformer (up to 4,096 sub-words, a single global token, warm-started from roberta) in long legal document classification. the new legallongformer (and its variants) are the new state of the art in the long document tasks of lexglue. in future work, we would like to further pre-train the proposed bow variants of legalbert, legal-longformer, and variants on legal corpora, to help them better optimize the newly introduced modifications (e.g., tf-idf embeddings, additional posi-tional embeddings, updated attention scheme with additional global tokens). we would also like to experiment with long documents from other domains (e.g., long business documents).