LEXTREME : A Multi-Lingual and Multi-Task Benchmark for the Legal Domain (2023)

Abstract: Lately, propelled by phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well-curated and challenging benchmarks are crucial. Previous efforts have produced numerous benchmarks for general NLP models, typically based on news or Wikipedia. However, these may not fit specific domains such as law, with its unique lexicons and intricate sentence structures. Even though there is a rising need to build NLP systems for languages other than English, many benchmarks are available only in English and no multilingual benchmark exists in the legal NLP field. We survey the legal NLP literature and select 11 datasets covering 24 languages, creating LEXTREME. To fairly compare models, we propose two aggregate scores, i.e., dataset aggregate score and language aggregate score. Our results show that even the best baseline only achieves modest results, and also ChatGPT struggles with many tasks. This indicates that LEXTREME remains a challenging task with ample room for improvement. To facilitate easy use for researchers and practitioners, we release LEXTREME on huggingface along with a public leaderboard and the necessary code to evaluate models. We also provide a public Weights and Biases project containing all runs for transparency.

## introduction in the last decade, natural language processing (nlp) has gained relevance in legal artificial intelligence, transitioning from symbolic to subsymbolic techniques @xcite . such a shift is motivated partially by the nature of legal resources, which appear primarily in a textual format @xcite . following the advancements in nlp technologies, the legal nlp literature @xcite @xcite is flourishing with many new resources, such as large legal corpora @xcite , task-specific datasets @xcite @xcite , and pre-trained legal-oriented language models @xcite @xcite @xcite . greco and tagarelli (2023) offer a comprehensive survey on the topic. specifically, the emergence of pre-trained language models (plms) has led to significant performance boosts on popular benchmarks like glue @xcite or superglue @xcite , emphasizing the need for more challenging benchmarks to measure progress. legal benchmark suites have also been developed to systematically evaluate the performance of plms, showcasing the superiority of legal-oriented models over generic ones on downstream tasks such as legal document classification or question answering @xcite . even though these plms are shown to be effective for numerous downstream tasks, they are general-purpose models that are trained on broaddomain resources, such as wikipedia or news, and therefore, can be insufficient to address tasks specific to the legal domain @xcite @xcite . indeed, the legal domain is strongly characterized both by its lexicon and by specific knowledge typically not available outside of specialized domain resources. laypeople even sometimes call the language used in legal documents "legalese" or "legal jargon", emphasizing its complexity. moreover, the length of a legal document usually exceeds the length of a wikipedia or news article, and in some tasks the relationships between its entities may span across the entire document. therefore, it is necessary to develop specialized legal plms trained on extensive collections of legal documents and evaluate them on standardized legal benchmarks. while new plms capable of handling long documents have been developed in the last years, they are predominantly trained for the general domain and on english data only. the rising need to build nlp systems for languages other than english, the lack of textual resources for such languages, and the widespread use of code-switching in many cultures @xcite ) is pushing researchers to train models on massively multilingual data @xcite . nonetheless, to the best of our knowledge, no multilingual legal language model has been proposed so far. consequently, there is a need for standardized multilingual benchmarks that can be used to evaluate existing models and assess whether more research efforts should be directed toward the development of domain-specific models. this is particularly important for legal nlp where inherently multinational (european union, council of europe) or multilingual (canada, switzerland) legal systems are prevalent. in this work, we propose a challenging multilingual benchmark for the legal domain, named lextreme. we survey the literature from 2010 to 2022 and select 11 relevant nlu datasets covering 24 languages in 8 subdivisions (germanic, romance, slavic, baltic, greek, celtic, finnic, and hungarian) from two language families (indo-european and uralic). we evaluate five widely used multilingual encoder-based language models as shown in figure 1 and observe a correlation between the model size and performance on lex-treme. surprisingly, at the low end, distilbert @xcite strongly outperforms minilm @xcite (36.7 vs 19.0 lextreme score) while only having marginally more parameters (135m vs 118m). for easy evaluation of future models, we release the aggregate dataset on the huggingface hub foot_0 along with a public leaderboard and the necessary code to run experiments on github. 2 knowing that our work can not encompass "everything in the whole wide legal world" @xcite , we design lextreme as a living benchmark and provide detailed guidelines on our repository and encourage the community to contribute high-quality multilingual legal datasets. 3 finally, we integrated lextreme together with the popular english legal benchmark @xcite into helm @xcite (an effort to evaluate language models holistically using a large number of datasets from diverse tasks) to ease the adoption of curated legal benchmarks also for the evaluation of large language models such as @xcite , palm @xcite or llama @xcite . contributions of this paper are two-fold: ## models considered since our benchmark only contains nlu tasks, we consider encoder-only models for simplicity. due to resource constraints, we did not fine-tune models larger than 1b parameters. ## experimental setup multilingual models were fine-tuned on all languages of specific datasets. monolingual models used only the given model's language subset. some datasets are highly imbalanced, one of the best examples being bcd-u with a proportion of the minority class of about 2%. therefore, we applied random oversampling on all sltc datasets, except for glc, since all its subsets have too many labels, which would have led to a drastic increase in the data size and thus in the computational costs for fine-tuning. for each run, we used the same hyperparameters, as described in section a.3. as described in section 4.3, some tasks contain very long documents, requiring the usage of hierarchical variants to process sequence lenghts of 1024 to 4096 tokens. based on the distribution of the sequence length per example for each task (cf. appendix h), we decided on suitable sequence lengths for each task before fine-tuning. a list of suitable sequence lengths are in a.1. ## results larger models are better for both, we see a clear trend that larger models perform better. however, when looking at the individual datasets and languages, the scores are more erratic. note that xlm-r base underperforms on meu (especially on meu-3; see differing model variance across datasets we observe significant variations across datasets such as glc, ots or c19, with differences as large as 52 (in ots) between the worst-performing minilm and the best-performing xlm-r large. minilm seems to struggle greatly with these three datasets, while even achieving the best performance on gam. on other datasets, such as gam, sjp, and map the models are very close together (less than 6 points between best and worst model). even though xlm-r large takes the top spot on aggregate, it only has the best performance in six out of eleven datasets. less variability across languages in contrast to inconsistent results on the datasets, xlm-r large outperforms the other multilingual models on most languages (21 out of 24). additionally, we note that model variability within a language is similar to the variability within a dataset, however, we don't see extreme cases such as glc, ots, or c19. ## conclusions and future work conclusions we survey the literature and select 11 datasets out of 108 papers with rigorous criteria to compile the first multilingual benchmark for legal nlp. by open-sourcing both the dataset and the code, we invite researchers and practitioners to evaluate any future multilingual models on our benchmark. we provide baselines for five popular multilingual encoder-based language models of different sizes. we hope that this benchmark will foster the creation of novel multilingual legal models and therefore contribute to the progress of natural legal language processing. we imagine this work as a living benchmark and invite the community to extend it with new suitable datasets. future work in future work, we plan to extend this benchmark with other nlu tasks and also generation tasks such as summarization, simplification, or translation. additionally, a deeper analysis of the differences in the behavior of monolingual general-purpose models versus models trained on legal data could provide useful insights for the development of new models. another relevant aspect that deserves further studies is the impact of the jurisdiction and whether the jurisdiction information is predominantly learned as part of the llm or is instead learned during fine-tuning. finally, extending datasets in more languages and evaluating other models such as mt5 @xcite can be other promising directions.