Pretrained Language Models v. Court Ruling Predictions: A Case Study on a Small Dataset of F rench Court of Appeal Rulings (2023)

Abstract: NLP systems are increasingly used in the law domain, either by legal institutions or by the industry. As a result there is a pressing need to characterize their strengths and weaknesses and understand their inner workings. This article presents a case study on the task of judicial decision prediction, on a small dataset from French Courts of Appeal. Specifically, our dataset of around 1000 decisions is about the habitual place of residency of children from divorced parents. The task consists in predicting, from the facts and reasons of the documents, whether the court rules that children should live with their mother or their father. Instead of feeding the whole document to a classifier, we carefully construct the dataset to make sure that the input to the classifier does not contain any ‘spoilers’ (it is often the case in court rulings that information all along the document mentions the final decision). Our results are mostly negative: even classifiers based on French pretrained language models (Flaubert, JuriBERT) do not classify the decisions with a reasonable accuracy. However, they can extract the decision when it is part of the input. With regards to these results, we argue that there is a strong caveat when constructing legal NLP datasets automatically.

## introduction natural language processing (nlp) has now many applications in the legal domain, and is used in practice to provide tools for law practitioners, such as lawyers or judges (e.g. for information retrieval, document classification), to quantify judicial risks in specific cases. judicial decision prediction through nlp tools is also important for law researchers to identify the factors that most influence the final decision of a case, characterize judge's reasoning process, and help data annotation procedures. in this paper, we present a case study on the prediction of the verdict ruling on the habitual res-idency of children in appeal cases from french courts. these documents follow a typical structure (facts, judgement of the first court whose ruling was the object of an appeal, decision itself). however, the structure is only implicit (sections are not titled), and judges' writing is not standardised and differs a lot from one court of appeal to another. finally, it is often the case that information of each type (facts, reasons, decision) are scattered through the document. as a result, the document might contain 'spoilers' for the decision itself at various places in the document. in order to assess the ability of bert-style language models to infer a verdict from the legal content itself, rather than the full text of the document, we implemented an annotation campaign aiming at assigning a semantic type to every segment in each document as: (i) facts (ii) information about the first judgment (iii) reasons for the decision (iv) decision itself (containing explicitly the variable to predict) (v) none of these types. thanks to this annotation procedure, lead by law experts, we compare the usefulness of various types of inputs for several types of classifiers, including bag-of-ngrams classifiers and bert-based classifiers. in particular, we used 3 french pretrained language models: flaubert @xcite and camem-bert @xcite ) that were trained on generic-domain data, and juribert @xcite , that has been specifically trained on french legal texts. in summary, our contributions are as follows: predicting the outcome of a judicial decisions based on a textual input is rather recent and has first been applied to english decisions from the european court of human rights (ecthr aletras et al., 2016). in particular, aletras et al. (2016) constructed a dataset of ecthr decisions focusing on cases about a few articles of the european convention on human rights, and evaluated bagof-n-grams approaches. subsequently, several studies were made on the echr corpus, consisting of approximately 11 500 rulings from 1959 to our days. medvedeva et al. (2020) and liu and chen (2017) evaluated various linear classifiers. kaur and bozic (2019) used deep neural networks based on convolutional networks. o'sullivan and beel (2019) used word embeddings pretrained on documents from the ecthr data. chalkidis et al. (2019) introduced legal-bert, a bert-style model pretrained on legal data (legislative texts, ecthr cases and american contracts), and evaluated it on the ecthr dataset. other common dataset include decisions from the us supreme court, e.g. katz et al. ( 2014 ) use decision trees to predict the votes of judges. we refer the reader to @xcite for a general survey on judicial decision prediction. in contrast to these publications, we focus on french, and on a very specific type of litigation: the habitual residency of children of divorced parents. moreover we use both linear classifiers and deep nets based on fine-tuning pretrained language models. as regards legal nlp applied to french, @xcite and @xcite evaluated svm classifiers on the prediction of the area of a case, as well as its ruling, on decisions from the french supreme court (cour de cassation). @xcite used pretrained language models such as flaubert @xcite and camembert @xcite to predict the outcome of cases about conflicts between a tenant and a landlord in québec. they observed that pretrained language models outperformed linear classifiers. our dataset is on another type of litigation (habitual residency of children) and we focus on the manual construction of our dataset, instead of automatically constructing it. finally, @xcite introduced juribert, trained on légifrance, an official web-site publishing all french law and evaluated it on topic classification tasks for documents from the cour de cassation (highest court in france). ## data in france, when parents of minor children separate, they must decide on their children's place of residency. if the parents are unable to agree on a place of residency, the matter is referred to a court which, after hearing each parent, issues a ruling. we study the three principal outcomes: children live with their mother, their father, or in alternating residency. if the outcome of the lower court is not suitable for at least one parent, they can make an appeal to retry the case. new judges re-examine the facts and confirm or overturn the judgment handed down by the first court. these are the rulings we'll be using for the classification experiment. we collected 987 rulings from french courts of appeal from the jurica foot_0 databasis, that we accessed through a legal publisher (lexisnexis). the rulings have the following structure: although the structure of court rulings in sections (facts, judgment, reasons, decision) is conventional, judges' practices vary a lot in the length of these sections. for example, the facts section might be extremely short or very verbose, and its content may not be restricted to what lawyers define as facts. moreover information about facts or reasons may be scattered through the document and not only occur in their dedicated section. therefore, instead of relying on the structure of the documents to extract information (and let the model be exposed to potential 'spoilers' about the decision), we rely on the manual annotation of every segment of the document as either (i) fact (ii) judgement (iii) reason (iv) decision, or (v) not relevant. the manual annotations were carried out by two law professors (who are coauthors of the paper and designed the annotation guidelines), one law post-doc researcher and one law graduate students who were trained for the task. due to the time required to carry out annotations, each document was annotated by a single annotator, so we did not compute interannotator agreement scores. annotating the rulings was quite challenging, especially to differentiate the facts and the reasons, because depending on the judge's method, the parties' claims and arguments can be set out in the reasons to justify the verdict. therefore, in several rulings, the facts category is limited to the names and birth dates of parents and their children. prediction task our goal was to predict the outcome of appeal cases ruling on the residency of children among three possible outcomes: they live with their mother (label m, 50%), their father (f, 33%) or in alternate residency (b for both, 17%). 3 we trained and evaluated classifiers settings in several settings, depending on the information input to the model: (i) f+r: facts+reasons (ii) f+r+j: facts+reasons+judgment (iii) decision (iv) decision+judgment. the last two settings are meant to assess the model's ability to extract the information about the decision, 4 whereas the first two assess the model's ability to make inferences about the judges' reasoning. 3 an alternative way of framing the task with two labels would be to predict whether the first judgment is (i) confirmed (ii) reversed. however, these two labels do not provide the same information as the three-label task we chose, since labels have to be interpreted with regards to the first judgment, and the second label (reversed judgement) is not as informative (2 possible outcomes). ## models baseline we use various classical classifiers as baselines, namely: we vectorize the texts using bag-of-n-grams methods: either unweigthed, or weighted by tf-idf. we use both unigrams, trigrams or both (based on preliminary experiments). for each algorithm, we made a cross validation to determine the best hyperparameters of each classifier. all baselines were implemented with scikit-learn library @xcite . language models as regards pretrained language models, we used flaubert @xcite and camembert @xcite , two generic-purpose pretrained models for french, as well as juribert @xcite , a language model trained only on data from the legal domain. because of the 512 tokens limitation of language models, the classifiers could not read the entire data in input, and thus did not make the difference between f+r and f+r+j, or didn't have all the argumentation of the judge. therefore, we chunked each document based on line breaks, encode each chunk separately with the language model, and run a bi-lstm (2 layers of 256 units) on the chunk representations to obtain a fixed-size vector for the whole document. we use dropout before feeding the representation to a classification layer. finally, to tokenize the data, we used the tokenizers associated with each language models, provided by the platform hugging face, and the associated library transformers @xcite . ## experiments before starting data processing, we split the corpus into three stratified subsets: training (68%), testing (20%) and validation (12%). the bert-based classifiers were trained for 80 epochs, with a batch size of 1, on a gpu. we used the adamw optimizer @xcite to minimize the loss function (negative log likelihood), and a scheduler with warmup, to vary the learning rate for a few warm-up periods. we tuned the learning rate ({10 -6 , 2 • 10 -6 , 5 • 10 -6 } for flaubert, {10 -5 , 10 -6 , 2 • 10 -6 , 5 • 10 -6 }, {10 -5 , 2 • 10 -5 , 2 • 10 -6 , 5 • 10 -6 } for juribert) and the dropout value (0.15, 0.25) for each classifier, we selected the best models based on validation f1 score (macro-average over classes. we also report accuracy in results. ## conclusion our experiments have shown the limitations of language models on judicial documents, since classifiers based on these models fail to understand the textual input data. they perform well when the verdict is already given, but fail when the text is more important and requires a deeper analytical reading, during the court's argumentation for example. on the question of children's habitual residency, the subject is far too sensitive to entrust the task of deciding to an ai, and we can only imagine complementary tools, making it possible to reduce the duration of procedures, to support citizens in putting together their case or to help the parties' lawyers. by successfully processing data in such a way that a classifier can identify the most relevant terms and phrases, it would be possible to find the 'winning arguments,' those that convince the judge or that can tip a case over the edge. this could then help lawyers to build their arguments, by giving them access to these elements in cases similar to their own.