Exploiting Domain-Specific Knowledge for Judgment Prediction Is No Panacea (2021)

Abstract: Legal judgment prediction (LJP) usually consists in a text classification task aimed at predicting the verdict on the basis of the fact description. The literature shows that the use of articles as input features helps improve the classification performance. In this work, we designed a verdict prediction task based on landlord-tenant disputes and we applied BERT-based models to which we fed different article-based features. Although the results obtained are consistent with the literature, the improvements with the articles are mostly obtained with the most frequent labels, suggesting that pre-trained and fine-tuned transformer-based models are not scalable as is for legal reasoning in real life scenarios as they would only excel in accurately predicting the most recurrent verdicts to the detriment of other legal outcomes.

## introduction at the intersection of machine learning and law, legal judgment prediction (ljp) is a task that consists in predicting either the outcome of a lawsuit @xcite @xcite @xcite or some other case attributes such as legal areas @xcite or charges @xcite . one specificity of court rulings is that they are based on the application of legal articles to the factual description of the case. that is to say, a judge must determine whether some law articles are relevant to a case, and if applicable, whether the legal principles they embody are violated. therefore, articles as domain-specific knowledge can be used as leverage for improving ljp performance, as shown by @xcite and @xcite for charge prediction and divorce judgment prediction respectively. @xcite also went further by using articles for distinguishing confusing charges in a charge prediction task. meanwhile, transformers @xcite ) and bert models @xcite in particular have been widely used in nlp tasks with the assumption that such models, if first pre-trained on massive corpora and then fine-tuned on the dataset of a given task, could suffice for achieving significant improvements. on one hand, this turned out to be true with the cail2018 dataset (charge prediction task) as shown by @xcite . on the other hand, @xcite mentioned in a statutory reasoning entailment task that a transformer model does worse than a rule-based model, even after further pre-training on the domain corpus. furthermore, in an employment notice prediction task, @xcite emphasized that domain adaption of such models could even harm performance. these elements raise the question of how well a pre-trained transformer model can handle a legal nlp task and how well the input from domain-specific knowledge such as legislative text can improve the ljp performance. to the best of our knowledge, in the case of ljp tasks aimed at verdict prediction, no experiment has tested so far the application of pre-trained bert models on both tribunal decision text and cited law articles text combined altogether, which we intend to do in this work. we designed a multilabel classification task in which the model must predict the ruling outcomes on the basis of the facts description. one can imagine that such a predictive engine could be used for legal assistance for those who may not afford the services of a legal expert. unlike @xcite , we put the article prediction aside in order to focus solely on the verdict prediction and assess in which conditions input article-based features can improve classification. for our experiments, we use a landlord-tenant disputes corpus used by westermann et al. (2019) and salaün et al. (2020 ) from which we extracted fine-grained targets labels and article features in order to encompass as much as possible the variety of rulings, thus making the task more representative of real life cases. we present the preprocessing of the dataset along with the creation of article-based features in section 2. the architectures of the models are shown in section 3 along with three methods for integrating the information from the articles mentioned in the decisions. discussion of the results and concluding remarks are provided in sections 4 and 5 respectively. ## preparation of the dataset the administrative housing tribunal is a court of quebec in canada with an exclusive jurisdiction in landlord-tenant disputes. we got access to an exhaustive corpus of 667,305 decisions in french issued from 2001 to 2018 publicly accessible through soquij portal 1 . documents average and median lengths amount to 307 and 235 tokens respectively with a standard deviation of 371. each decision is split in two by applying heuristics based on the syntax of the documents: the pre-verdict section, used as text input (text before the italics in figure 1 ), and the verdict section containing the legal solution chosen by the judge in charge of the case (text in italics in figure 1 ). the pre-processing of both sections are described in subsections 2.1 and 2.2 respectively. ## models we aim at designing a multilabel classification task in which a model has to return the labels corresponding to the verdict on the basis of the preverdict section of each decision. our baseline is a one-versus-rest logistic regression, i.e. each label has its own classifier. the input text is vectorized through character-based tf-idf spanning bigrams to 8-grams (character-based features outperformed token-based). only the top 100k n-grams are retained. we also use camembert @xcite , a variant of bert @xcite that was pre-trained on generic french corpora. we further pre-trained the camembert-base default parameters (unsupervised masked language modelling task) from @xcite on the train set during 20 epochs 2 . we eventually fine-tuned them during the multilabel classification task. the batch size and the maximum number of fine-tuning epochs are set at 50 and 20 respectively. training is stopped when no further improvement is obtained in terms of exact match on validation set. we use the adam optimizer (kingma and ba, 2015) with a learning rate at 10 -5 (all other hyperparameters are left at default value). the optimization criterion is the binary cross entropy with logits loss for numerical stability during optimization. therefore the final output is that of a logit function, with scores ranging from -inf to +inf. a label is returned whenever its associated output value exceeds 0. we use a vanilla camembert model whose only input 2 exact match achieved by further pre-trained models is around one percent point greater than models with default pre-trained parameters. ## results and discussion the classification results are shown in ## concluding remarks we designed a ljp task as text multilabel classification for verdict prediction based on a collection of landlord-tenant disputes in french for which we used a further pre-trained camembert model and applied different types of features derived from the articles cited in the decisions (one-hot, node2vec, bert encoding of the text of articles). by doing so, we noticed that leveraging articles as input features (regardless of the representation used) made either marginal improvements for f1 scores of most frequent labels, either significant improvements for labels that are strongly correlated with certain articles. the use of article-based one-hot features achieves best exact match score (67.0%) while node2vec features achieve best f1 macro average score (63.2%). the model that encodes the text of the articles with bert does not outperform the two previous methods. as future work, we plan on comparing how models perform under both "realistic" setting (several rare target labels with no or few connections with the law available, as we did in this work) and "laboratory" setting (where low frequency targets and laws are aggressively filtered out). we also plan to assess whether the patterns observed in our work (performance improves when articles are strongly correlated with labels) also exist in other ljp datasets beyond housing law and canadian cases. furthermore, we plan to study further the attention weights and the mechanisms underlying the significant prediction improvement observed for certain labels when the input contains the text of articles that are highly correlated with the corresponding verdicts.