S wiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark (2021)

Abstract: In many jurisdictions, the excessive workload of courts leads to high delays. Suitable predictive AI models can assist legal professionals in their work, and thus enhance and speed up the process. So far, Legal Judgment Prediction (LJP) datasets have been released in English, French, and Chinese. We publicly release a multilingual (German, French, and Italian), diachronic (2000-2020) corpus of 85K cases from the Federal Supreme Court of Switzer- land (FSCS). We evaluate state-of-the-art BERT-based methods including two variants of BERT that overcome the BERT input (text) length limitation (up to 512 tokens). Hierarchical BERT has the best performance (approx. 68-70% Macro-F1-Score in German and French). Furthermore, we study how several factors (canton of origin, year of publication, text length, legal area) affect performance. We release both the benchmark dataset and our code to accelerate future research and ensure reproducibility.

## introduction frequently, legal information is available in textual form (e.g. court decisions, laws, legal articles or commentaries, contracts). with the abundance of legal texts comes the possibility of applying natural language processing (nlp) techniques to tackle challenging tasks @xcite @xcite . in this work, we study the task of legal judgment prediction (ljp) where the goal is to predict the outcome (verdict) of a decision given its facts @xcite @xcite @xcite . many relevant applications and tasks, such as court opinion generation @xcite and analysis @xcite have been also studied, while there is also work aiming to interpret (explain) the decisions of particular courts @xcite . models developed for ljp and relevant supportive tasks may assist both lawyers, e.g., help them prepare their arguments by identifying their strengths and weaknesses, and judges and clerks, e.g., review or prioritize cases, thus speeding up judicial processes and improving their quality. especially in areas with many pending cases such as indian foot_0 and brazilian foot_1 jurisdictions or us immigration cases 3 the deployment of such models may drastically shorten the backlog. such models can also help legal scholars to study case law @xcite and help sociologists and research ethicists to expose irresponsible use of ai in the justice system @xcite . so far, ljp datasets have been released for english @xcite @xcite , french @xcite and chinese @xcite . we introduce a new multilingual, diachronic ljp dataset of fscs cases, which spans 21 years (from 2000 to 2020) containing over 85k (50k german, 31k french and 4k italian) cases. to the best of our knowledge, it is the only publicly available multilingual ljp dataset to date. additionally, it is annotated with publication years, legal areas and cantons of origin; thus it can be used also as testbed for fairness and robustness in the critical application of nlp to law @xcite . @xcite argues that the nlp community is investing many more resources in the development of models rather than data. as a result, there are not enough challenging, high-quality and well curated benchmarks available. rogers assumes that the main reason for this imbalance is that the "data work" is considered less prestigious and top conferences are more likely to reject resource (dataset) papers. with our work (and the associated code and data) we hope to make a valuable contribution to the legal nlp field, where there are not many ready-to-use benchmarks available. ## experiments in this section, we describe the conducted experiments alongside the presentation of the results and an analysis of the results of the german dataset in terms of diachronicity (judgment year), legal area, input (text) length and canton of origin. ## conclusions & future work we introduced a new multilingual, diachronic dataset of 85k swiss federal supreme court (fscs) cases, including cases in german, french, and italian. we presented results considering three alternative bert-based methods, including methods that can process up to 2048 tokens and thus can read the entirety of the facts in most cases. we found that these methods outperform the standard bert models and have the best results in macro-f1, while the naive majority classifier has the best overall results in micro-f1 due to the high class imbalance of the dataset (more than 3 4 of the cases are dismissed). further on, we presented a bivariate analysis between performance and multiple factors (diachronicity, input (text) length, legal area, and canton of origin). the analysis showed that performance deteriorates as input (text) length increases, while the results in cases from different legal areas or cantons vary raising questions on models' robustness under different attributes. in future work, we would like to investigate the application of cross-lingual transfer learning techniques, for example the use of adapters @xcite . in this case, we could possibly improve the poor performance in the italian subset, where approx. 3k cases exists, by training a multilingual model across all languages, thus exploiting all available resources, ignoring the traditional language barrier. in the same direction, we could also exploit and transfer knowledge from other annotated datasets that aim at the ljp task (e.g., ecthr and scotus). more in depth analysis on robustness is also an interesting future avenue. in this direction, we would like to explore distributional robust optimization (dro) techniques @xcite that aim to mitigate disparities across groups of interest, i.e., labels, cantons and/or legal areas could be both considered in this framework. another interesting direction is a deeper analysis with models handling long textual input @xcite ) using alternative attention schemes (window-based, dilated, etc.). furthermore, none of the examined pre-trained models is legal-oriented, thus pre-training and evaluating such specialized models is also needed, similarly to the english legal-bert of @xcite .