Effectively Leveraging BERT for Legal Document Classification (2021)

Abstract: Bidirectional Encoder Representations from Transformers (BERT) has achieved state-of-the-art performances on several text classification tasks, such as GLUE and sentiment analysis. Recent work in the legal domain started to use BERT on tasks, such as legal judgement prediction and violation prediction. A common practise in using BERT is to fine-tune a pre-trained model on a target task and truncate the input texts to the size of the BERT input (e.g. at most 512 tokens). However, due to the unique characteristics of legal documents, it is not clear how to effectively adapt BERT in the legal domain. In this work, we investigate how to deal with long documents, and how is the importance of pre-training on documents from the same domain as the target task. We conduct experiments on the two recent datasets: ECHR Violation Dataset and the Overruling Task Dataset, which are multi-label and binary classification tasks, respectively. Importantly, on average the number of tokens in a document from the ECHR Violation Dataset is more than 1,600. While the documents in the Overruling Task Dataset are shorter (the maximum number of tokens is 204). We thoroughly compare several techniques for adapting BERT on long documents and compare different models pre-trained on the legal and other domains. Our experimental results show that we need to explicitly adapt BERT to handle long documents, as the truncation leads to less effective performance. We also found that pre-training on the documents that are similar to the target task would result in more effective performance on several scenario.

## introduction recent advances in deep learning contribute to effective performances of several natural language processing (nlp) tasks on legal text documents, such as, violation prediction @xcite , overruling prediction @xcite , legal judgement prediction @xcite , legal information extraction @xcite , and court opinion generation @xcite . bidirectional encoder representations from transformers (bert) @xcite has gained attentions from the nlp community due to its effectiveness on several nlp tasks @xcite @xcite . importantly, the effectiveness of bert is mainly due to the transfer learning ability that leverages semantic and syntactic knowledge from pre-training on a large non-labeled corpus @xcite @xcite . however, @xcite reported that bert could not effectively handle long documents in the european court of human rights (echr) dataset. in addition, pre-training bert is costly. we need access to a special type of machines to pre-train bert on a large corpora @xcite @xcite . in this work, we investigate how to effectively adapt bert to handle long documents, and how importance of pre-training on in-domain documents. specifically, we will focus on two legal document prediction tasks, including echr violation dataset @xcite and overruling task dataset @xcite . the echr violation dataset provides a multi-label classification task. given a list of facts described in free-texts, the task is to identify which articles of the european convention were violated. the overruling task is a binary classification task to predict whether a legal statement will be later overruled by the same or higher ranking court @xcite ). we will discuss more about the two tasks in section 4.2. the main contributions of this paper are threefold: the remainder of the paper is organised as follows. section 2 further discusses related work and positions our work in the literature. section 3 describes the two research questions we aim to answer in this paper and how we will find the answers. sections 4 and 5 discuss our experimental setup and results. section 6 provides more insight from the experimental results and answers the two research questions. finally, we provide concluding remarks in section 7. ## related work legal documents, such as eu & uk legislation, european court of human rights (echr) cases, case holdings on legal decisions (casehold) are normally written in a descriptive language in a non-structured text format and have unique characteristics that are different from those of other domains. in order to advance legal nlp research, several tasks and datasets have been developed, including violation prediction on the echr dataset @xcite , court overruling @xcite , legal docket classification @xcite and court view generation @xcite . in this work, we focus on text classification, which is a main research area of legal nlp. bidirectional encoder representations from transformers (bert) is a language representation model that is optimized during pre-training by selftraining using a masked language model prediction and a next sentence prediction as a joint training objective @xcite . as shown in figure 1 , bert model architecture is built upon a multilayer bidirectional transformer encoder of @xcite , where the number of input tokens is limited to 512. pre-training bert enables effective transfer learning from a large dataset before finetuning the model on a specific task @xcite . importantly, @xcite used this transfer learning method to achieve the state-of-the-art performance on several nlp datasets, such as glue @xcite , squad @xcite , concept nor-malisation @xcite and novel named entity recognition @xcite . in particular, when fine-tuning bert, we normally add a classification layer (either soft-max or sigmoid) on the c (or cls) representation in bert output layer, in order to compute the prediction probabilities as in figure 1 . in the legal domain, @xcite found that pre-training bert on legal documents before fine-tuning on particular tasks lead to a better performance than pre-training bert on general documents. however, @xcite found that bert did not perform well on the violation prediction task due to the length of the documents that are mostly longer than 512 tokens. they dealt with the long legal documents by using a hierarchical bert technique @xcite . difference from the previous work, we investigate the effectiveness of variances of pre-trained bert-based models and compare several methods to handle the long legal documents in legal text classification. several attempts @xcite @xcite have been made to enable bert-like models to work on documents with more than 512 tokens. for example, beltagy et al. (2020) and @xcite used several different attention-mechanism techniques, such as global attentions and sliding window attentions to enable learning on a longer number of tokens. @xcite investigated different approaches to apply bert on sequence chunks of texts in a document before aggregating the features using techniques, such as max pooling and mean pooling. in this work, we adapt these techniques to learn how to effectively use bert on long legal documents. ## research questions this section discusses research questions we aim to answer in this paper. ## experimental setup in section 3, we have discussed the two main research questions to be investigated in this paper. in this section, we discuss the hyper-parameters of our models in section 4.1. then, we provide the details of the two legal text classification datasets (section 4.2) and the variances of the bert models (section 4.3) used in the experiments. ## experimental results in this section, we compare the variances of bertbased models on the echr violation dataset (section 5.1) and overruling task dataset (section 5.2), respectively. ## discussions in this section, we provide further discussions on the experimental results from section 5, in order to answer the research questions posed in section 3. ## conclusions we have discussed the challenges of using bert for text classification in the legal domains, and posed two research questions regarding the pretrain documents and how to cope with long documents. to answer the research questions, we conducted the experiments on the echr violation dataset and the overruling task dataset. our experimental result showed that the models pretrained on the domain similar to the task enhanced the performance. in addition, the experiments on echr violation dataset supported that truncating or discarding parts of a document resulted in a poor performance. importantly, bigbird and longformer, which explicitly handled long documents using different attention techniques, achieved the best performance on long legal document classification. alternatively, applying bert on chunks of texts before aggregating the vector representation across all of the chunks using an appropriate function (e.g. max or mean) could achieve a reasonable result.