C lass A ction P rediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US (2022)

Abstract: The research field of Legal Natural Language Processing (NLP) has been very active recently, with Legal Judgment Prediction (LJP) becoming one of the most extensively studied tasks. To date, most publicly released LJP datasets originate from countries with civil law. In this work, we release, for the first time, a challenging LJP dataset focused on class action cases in the US. It is the first dataset in the common law system that focuses on the harder and more realistic task involving the complaints as input instead of the often used facts summary written by the court. Additionally, we study the difficulty of the task by collecting expert human predictions, showing that even human experts can only reach 53% accuracy on this dataset. Our Longformer model clearly outperforms the human baseline (63%), despite only considering the first 2,048 tokens. Furthermore, we perform a detailed error analysis and find that the Longformer model is significantly better calibrated than the human experts. Finally, we publicly release the dataset and the code used for the experiments.

## introduction recently, the literature in legal natural language processing (nlp) has grown at a fast pace, firmly establishing it as an important specialized domain in the broader nlp ecosystem. as part of this strong growth and as a first step establishing legal nlp in the field, many legal datasets have been released in the fields of legal judgment prediction (ljp) @xcite , law area prediction @xcite , legal information retrieval @xcite , argument mining @xcite , topic classification @xcite , named entity recognition (luz de @xcite @xcite figure 1 : calibration plot on the full text dataset. the human experts rated the confidence of their predictions on a score from 1 to 5. the confidence scores of the longformer models were binned into 5 buckets. 2019), natural language inference @xcite , question answering @xcite , and summarization @xcite . in particular, the field of ljp has been very active, with many datasets released recently. @xcite surveyed the field and divided the datasets into five subtasks. in this work, we release a dataset belonging to the category of the plea judgment prediction (pjp) task. most other pjp datasets use the facts summary, written by the court (clerks or judges) as input @xcite . the facts are written in such a way as to support the final decision @xcite ) and require extensive work by highly qualified legal experts @xcite . in contrast, in this work we consider the plaintiff's pleas (aka complaints) as input, making the task more realistic for use in real-world applications. most ljp datasets released so far are from countries with civil law. our dataset originates from the united states, the largest country employing the common law legal system. to the best of our knowledge, we are the first to release a dataset specifically targeting class action lawsuits. ## related work ljp is an important and well-studied task in legal nlp. @xcite subdivide ljp into five subtasks: (a) in the article recommendation task, systems predict relevant law articles for a given case @xcite @xcite . (b) the goal of the charge prediction task, mainly studied in china, is to predict the counts the defendant is charged for based on the facts of the case @xcite @xcite . (c) in the prison term prediction task, systems predict the prison time for the defendant as ruled by the judge @xcite . (d) in the court view generation task, systems generate court views (explanation written by judges to interpret the judgment decision) @xcite . (e) in the plea judgment prediction task, systems predict the case outcome based on the case's facts @xcite @xcite @xcite @xcite . since our work belongs to the pjp category, in the following, we elaborate more on the related work in this area. niklaus et al. (2021b ) released a trilingual (german, french, italian) swiss dataset from the federal supreme court of switzerland. they use the facts summary as input and predict a binary output: approval or dismissal of the plaintiff's pleas for approx. 85k decisions. @xcite released a dataset of approx. 127k french supreme court cases. as input, they used the entire case description and not only the facts summary, presumably making the task considerably easier and a possible explanation for their high performance on the dataset. as output, they consider up to 8 classes of decisions ruled by the court. lage-freitas et al. ## dataset description in this section, we describe the dataset origin and statistics in detail. additionally, we elaborate on the dataset construction process and the variants we produced. figures 2a and 2b show the distribution of cases across the most frequent states and courts in the dataset, respectively. note that the origin of these class action lawsuits is very diverse, both across states and across courts. not surprisingly, population-rich states like california, florida, and new york lead the list. however, while california is more than double in population (39.5m vs. 20.2m as of april 2021), the number of class action lawsuits has the inverse relationship (∼ 3k from new york and ∼ 1.8k from california). we assume that the complicated filing system in california could be a reason for this disparity foot_3 . ## error analysis neural networks (nns) and their latest incarnation, transformers @xcite , work very well across a wide range of tasks, especially if the tasks involve more "complicated" data like text or images. however, in contrast to traditional machine learning (ml) methods such as linear regression, they are not interpretable out-of-the-box. neural networks need additional methods to make them explain themselves better to humans. a rich body of literature investigates how to make nns and especially transformers more interpretable @xcite @xcite @xcite . interpretability is especially important in high-stakes domains such as law or medicine. in the following two sections, we analyze our models using the two interpretability methods calibration and ig to get a better understanding of their inner workings. ## conclusions and future work answers to the research questions rq1: to what extent is it possible to determine the winner of us class action cases using only the textual part of the complaints (without metadata)? it is possible, to some extent, to determine the winner of us class action cases using only the textual part of the complaints. our best model achieves an accuracy of 66.8% (legalroberta) on the datasets using only the allegations. however, as this number shows, there is still a lot of room for improvement. rq2: to what extent can we use temperature scaling (ts) to better calibrate our models? similar to natural language inference, paraphrase detection and commonsense reasoning tasks (desai and durrett, 2020), we also find that in the pjp task, ts helps in calibrating pretrained transformers. in our best model, ts led to a decrease in ece scores from 28 to 2. rq3: to what extent can expert human lawyers solve the proposed task? expert human lawyers perform better than chance on a randomly selected dataset of 200 samples and can increase their accuracy from 53% to 60% when they are confident in their decision. however, they are still outperformed by a longformer model having access to only the first 2,048 tokens in both scenarios.