Classification of US S upreme C ourt Cases Using BERT -Based Techniques (2023)

Abstract: Models based on bidirectional encoder representations from transformers (BERT) produce state of the art (SOTA) results on many natural language processing (NLP) tasks such as named entity recognition (NER), part-of-speech (POS) tagging etc. An interesting phenomenon occurs when classifying long documents such as those from the US supreme court where BERT-based models can be considered difficult to use on a first-pass or out-of-the-box basis. In this paper, we experiment with several BERT-based classification techniques for US supreme court decisions or supreme court database (SCDB) and compare them with the previous SOTA results. We then compare our results specifically with SOTA models for long documents. We compare our results for two classification tasks: (1) a broad classification task with 15 categories and (2) a fine-grained classification task with 279 categories. Our best result produces an accuracy of 80% on the 15 broad categories and 60% on the fine-grained 279 categories which marks an improvement of 8% and 28% respectively from previously reported SOTA results.

## introduction every october, the us supreme court begins a new term on the first monday. each term includes a number of significant and complex cases that address a variety of issues, including environmental protection law, free speech, equal opportunity, tax law etc. legal court decisions are lawful statements acknowledged by a judge providing the justification and reasoning for a court ruling. the court's decisions not only have repercussions on the people involved in the case but also on the society as a whole. during a regularly scheduled court session, the justice who wrote the main decision sums it up from the bench. a copy of the opinion is then quickly posted on the concerned website. legal ex-perts must search through and classify these court decisions to support their research. this can require a lot of manual effort. artificial intelligence (ai) and machine learning (ml) can significantly reduce the burden of this type of manual work. washington university's collection of 8419 manually labeled supreme court documents (scdb) @xcite provides the basis for bench marking this type of ai task. there are multiple challenges involved when dealing with scdb or legal documents of similar kinds. first, these documents are exceptionally long which causes numerous difficulties. for example, feature vectors can be too large to fit in the memory or contextual variance can be too high for models to handle. next, the decision of a case requires identification of other relevant cases that can support the decision of the current case, which usually involve similar circumstances. therefore, it is of utmost importance to identify the similarities in terms of legal aspects while classifying the cases in the same category. finally, understanding of these legal documents requires supervision of legal expert. legal statements, like other specialized domains, have unique characteristics when compared to other generic corpora, such as unique vocabulary, exclusive syntax, idiosyncratic semantics etc. vocabulary, syntax, semantics and other linguistic characteristics can be specific to the legal domain or even the subdomain of court decisions, thus necessitating tools that are trained specifically for such domains. hence, an automatic system to categorize and process these documents is extremely useful. in this paper, we explore many novel techniques to counter the problem faced by models similar to bert @xcite when dealing with documents of length more than 512 tokens. some of the techniques include: analysing which 512 token chunks of documents make the best contribution to-wards classification, using summarized version of documents for classification, a voting-based ensemble approach, classification based on concatenation of 512 token chunks. we compare our results with previous non-bert like models as well as models which can take inputs of length more than 512 tokens. the rest of the paper is organized in the following way. section 2 talks about related work. section 3 talks about the scdb. we describe the working of bert-based techniques in section 4. section 5 concentrates on the experiments we conducted and the corresponding results we achieved. the final section discusses future work. ## related work recently, there has been a surge in research in the domain of nlp associated with the legal documents. @xcite talks about combining linguistic information from wordnet with a syntax-based retrieval of rules from legal text along with logic-based retrieval of dependencies from chunks of such texts leading to extraction of machine-readable rules from legal documents. @xcite illustrates several embeddingbased and symbol-based approaches for judgement prediction, legal question answering and similar case matching. @xcite discusses five areas of legal activity where nlp is playing an important role. these areas include legal research for finding information relevant to a legal decision, electronic discovery determining the relevance of documents in an information request, contract review to check that a contract is complete, document automation to generate routine legal documents and legal advice using question-answering dialogues. @xcite discusses different available approaches for summarization of legal texts and compares their performances on various datasets. @xcite showcases a scalable and flexible information extraction method, aimed at extraction of information from legal documents regardless of format, layout or structure, by considering the context. (yeung, 2019) comes up with a german legal bert model and evaluates its performance on downstream nlp tasks including classification, regression and similarity. bert-based models have shown some ground breaking performance in many nlp tasks. nowadays, they are being widely used in the legal domain as well. @xcite proposes bert-pli to capture the semantic relationships at the paragraph-level and then goes on to infer the relevance between two legal cases by aggregating paragraph-level interactions. @xcite releases legal-bert, a family of bert models for the legal domain intended to assist legal nlp research. @xcite studies a case in the context of legal professional search and presents how bert-based approach outperforms other traditional approaches. @xcite ) proposes an answer selection approach by finetuning bert on their vietnamese legal questionanswer pair corpus. they further pre-train bert on a vietnamese legal domain-specific corpus and show that this new bert performs better than the fine-tuned bert. classification of legal documents is an important nlp task which can automate alignment of legal documents with human-defined categories. @xcite classifies a proprietary corpus consisting of hundreds of thousands of legal agreements using bert. @xcite compares multi-label and binary classification of legal documents using variances of pre-trained bertbased models and other approaches to handle long documents. our work falls somewhat along similar lines but we use a different dataset of legal documents with considerably different properties. moreover, many of our bert-based techniques are significantly different from @xcite . @xcite presents baseline results for document type classification and theme assignment, a multi-label problem using their newly built brazil's supreme court digitalized legal documents dataset. @xcite proposes a method for learning chinese legal document classification using graph long short-term memory (lstm) combined with domain knowledge extraction. @xcite addresses multi-label classification of croatian legal documents using eurovoc thesaurus. @xcite experimented classification of singapore supreme court judgments using topic models, word embedding feature models and pretrained language models. @xcite presents results on predicting the rulings of the turkish constitutional court and courts of appeal using fact descriptions. @xcite investigates various text classification techniques to predict french supreme court decisions whereas @xcite does similar work for philippine supreme court. scdb has been used in various prominent nlp tasks. @xcite uses supreme court data and performs topic modelling using domainspecific embeddings. these embeddings are obtained from pre-trained legal-bert. @xcite constructs a model to predict the voting behavior of us supreme court and its justices in a generalized, out-of-sample context by using scdb along with some other derived features. @xcite talks about classification performance of different bert-based as well as non-neural architectures on many datasets of legal domain including scdb. our experiments differ from their work in multiple ways. first, they don't analyze the performance of these models across fine-grained 279 categories which is a harder classification task. second, they do not experiment different techniques to tackle the problem of restricted input sequence length of 512 tokens in bert-based models which is one of the key points of our work. the analysis of these techniques helps us in achieving sota across both broad as well as fine-grained classification tasks. finally, the version of scdb used by them differs from what we have used in our experiments. there is a significant overlap but the versions are not exactly the same. (undavia et al., 2018) presents classification of scdb across broad 15 categories and fine-grained 279 categories. our work exactly aligns with this work but the usage of bert-based techniques helps us in achieving better results. ## data our paper is primarily based on classification of us supreme court decisions dataset or supreme court database from washington university school of law @xcite . documents are classified by topic in a 2 level ontology, providing the basis for two different classification tasks: one using 15 broad category labels and another using 279 fine-grained category labels. the general statistics associated with this dataset can be found in ## proposed techniques we explore various bert-based techniques to classify scdb in this section. we apply these individual techniques to both the classification tasks i.e one with 15 categories and the other one with 279 categories. we use either bert or different versions of bert for our classification tasks. particularly, we use bert @xcite , roberta @xcite and legal-bert @xcite which have the restriction of maximum input sequence length of 512 tokens. we first try out our different techniques using bert, roberta and legal-bert. later, we compare the results of these techniques with other transformer-based models like longformer @xcite and legal-longformer foot_0 which accept longer sequences. ## experiments & results the code 3 related to all the experiments discussed below have been made public. we use weighted f1, accuracy and weighted precision as our evaluation metrics for both the classification tasks. the hyper-parameters used for all the above techniques except roberta include batch size to be 8, number of epochs as 5, learning rate to be 3e-5 and loss figure 2: concat-512, stride-64 general architecture to be categorical cross entropy. for roberta, we keep all the hyper-parameters to be the same except the learning rate which is changed to 1e-5. we split 8419 data points in 90:10 ratio of train and test sets. we run each experiment 5 times and take the average of the best epoch score to get the final score. we choose adam (kingma and ba, 2014) as our optimizer. we use bertbase-uncased version of bert, roberta-base version of roberta, legal-bert-base-uncased version of legal-bert, longformer-base-4096 version of longformer and legal-longformer-base-4096 version of legal-longformer from hugging face foot_1 . all the bert-based models accept a sequence of maximum of 512 tokens whereas longformer and legal-longformer accept a sequence of maximum of 4096 tokens. figure 1 shows the general architecture of best-512, summarization-512, ensemble and lsms with some differences in corresponding dimensions and input type. similarly, the general architecture of concat-512 and stride-64 with some differences in corresponding dimensions and input type can be seen in figure 2. each rectangular box in the image is divided into two parts. the left part of the box represents the name of the layer whereas the right part shows the output dimension of the layer. the architecture of both the types of models is mostly similar except for the form in which they accept their inputs. as we can see from ## conclusion & future work in this paper, we experimented with various bertbased techniques on scdb and presented the corresponding results comparing them with other state of the art models. we further did an analysis on how even with the given restriction of the input sequence length of 512 tokens for bert-based models, we can leverage these techniques to get some improvement. as a part of future work, we can leverage the knowledge embedded in references of a given scdb document. a reference in an scdb document basically refers to some other scdb document that has been cited to legally justify the decision taken on the former scdb document. the raw text of these references may not be very helpful in improving the classification tasks. we can use a graph structure to denote the relations between a given scdb document with other documents cited in it. the final classification result of a given scdb document can be calculated based on some form of aggregation incorporating classification results of its references weighted by the graph structure representing quantified relations with the scdb document at hand. another area of future work can be applying greedy approaches to the techniques discussed in this work. for example, we can have a greedy summarization technique where in the final 512 token summary, we can include more number of tokens from the best performing 512 token chunk as inferred from best-512 technique. similarly, we can have greedy ensemble technique where during the voting phase, we can give more weight to the best performing 512 token chunk as the results show from best-512 technique.