M ulti L egal P ile: A 689 GB Multilingual Legal Corpus (2024)

Abstract: Large, high-quality datasets are crucial for training Large Language Models (LLMs). However, so far, few datasets are available for specialized critical domains such as law and the available ones are often small and only in English. To fill this gap, we curate and release MultiLegalPile, a 689GB corpus in 24 languages from 17 jurisdictions. MultiLegalPile includes diverse legal data sources and allows for pretraining NLP models under fair use, with most of the dataset licensed very permissively. We pretrain two RoBERTa models and one Longformer multilingually, and 24 monolingual models on each of the language-specific subsets and evaluate them on LEXTREME. Additionally, we evaluate the English and multilingual models on LexGLUE. Our multilingual models set a new SotA on LEXTREME and our English models on LexGLUE. We release the dataset, trained models, and all code under the most open licenses possible.

## introduction recent years have seen llms achieving remarkable progress, as demonstrated by their performance on various benchmarks such as superglue @xcite , mmlu @xcite , and several human exams (openai, 2023), including u.s. bar exams for admission to practice law @xcite . these models are typically trained on increasingly large corpora, such as the pile @xcite , c4 @xcite , and mc4 @xcite . however, public corpora available for training these models are predominantly in english, and often constitute web text with unclear licensing. this even led to lawsuits against llm producers foot_0 , highlighting this critical issue. furthermore, there is a scarcity of largescale, domain-specific pretraining corpora, which we find that only one in every thousand document in mc4 contains legal text. similarly, llms are predominantly english, especially considering domain-specific models, e.g., ones specialized in biomedical, legal, or financial texts. legal texts, often produced by public instruments (e.g., state governments, international organizations), are typically available under public licenses, offering a rich resource for domain-specific pretraining. given this context, we curate a massive, openly available, corpus of multilingual law text spanning across numerous jurisdictions (legal systems), predominantly under permissive licenses. further on, we continue pretraining xlm-r models @xcite on our corpus and evaluated these models on the recently introduced lextreme @xcite and lexglue @xcite benchmarks. given the often extensive nature of legal text, we also pretrained a longformer model @xcite for comparison with hierarchical models @xcite @xcite . our multilingual models set a new state-of-theart (sota) on lextreme overall. our legal longformer outperforms all other models in four lex-treme datasets and reaches the highest dataset aggregate score. our monolingual models outperform their base model xlm-r in 21 out of 24 languages, even reaching language specific sota in five. on lexglue our english models reach sota in five out of seven tasks with the large model achieving the highest aggregate score. in the spirit of open science, we provide the dataset under a cc by-nc-sa 4.0 license, with some subsets licensed more permissively. dataset creation scripts, models, and pretraining code are public under apache 2.0 licenses. this opensource approach encourages further research and advancements in the field of legal text analysis and understanding using llms. ## related work in this section, we briefly discuss prior general and domain-specific pretraining corpora. see appendix b for a more detailed discussion of related works. ## pretraining legal models as part of this study, we release 2 new multilingual legal-oriented plms, dubbed legal-xlm-rs, trained on the newly introduced multile-galpile corpus (section 3). for the newly released legal-xlm-rs we followed a series of bestpractices in the lm development literature: devlin et al. (2019) 6 , based on the findings of wettig et al. (2023). (d) for both training the tokenizer and our legal models, we use a sentence sampler with exponential smoothing of the sub-corpora sampling rate following conneau and lample (2019) and raffel et al. (2020), since there is a disparate proportion of tokens across sub-corpora and languages (figures 1 and 3) and we aim to preserve per-corpus and language capacity, i.e., avoid overfitting to the majority (approx. 50% of the total number of tokens) us-origin english texts. (e) we consider mixed cased models, i.e., both 6 devlin et al. -and many other follow-up work -used a 15% masking ratio, and a recipe of 80/10/10% of predictions made across masked/randomly-replaced/original tokens. to better account for long contexts often found in legal documents, we continue training the basesize multilingual model on long contexts (4096 tokens) with windowed attention (128 tokens window size) @xcite for 50k steps, dubbing it legal-xlm-lf-base. we use the standard 15% masking probability and increase the learning rate to 3e-5 before decaying but otherwise use the same settings as for training the small-context models. in addition to the multilingual models, we also train 24 monolingual models on each of the language-specific subsets of the corpus. except for choosing a smaller vocab size of 32k tokens, we use the same settings as for the multilingual models. due to resource constraints, we only train base-size models and stop training at 200k steps. due to limited data available in some low-resource languages, these models sometimes do multiple passes over the data. because of plenty of data and to achieve a better comparison on lexglue, we continued training the english model for 1m steps and also trained a large-size model for 500k steps. see we make all our models publicly available alongside all intermediate checkpoints (every 50k/10k training steps for roberta/longformer models) on the hugging face hub. foot_4 ## evaluating on legal benchmarks in the absence of established legal benchmarks for generative tasks and our focus on pretraining encoder-only models, we select two established legal benchmarks involving challenging text classification and named entity recognition tasks involving long documents: lextreme and lexglue. ## conclusions and future work conclusions due to a general lack of multilingual pretraining data especially in specialized domains such as law, we curate a large-scale highquality corpus in 24 languages from 17 jurisdictions. we continue pretraining xlm-r checkpoints on our data, achieving a new sota for base and large models on the lextreme benchmark and vastly outperforming previous methods in greek legal code. we turn our xlm-r base model into a longformer and continue pretraining on long documents. it reaches a new sota in four lextreme datasets and reaches the overall highest dataset aggregate score. monolingual models achieve huge gains over their base model xlm-r in some languages and even set language specific sota in five languages outperforming other models by as much as 11 macro f1. on lexglue our english models reach sota in five out of seven tasks with the large model achieving the highest aggregate score. to conclude, following best practices in continued pretraining on our comprehensive multilingual legal corpus establishes new state-of-the-art across multiple datasets and languages, significantly enhancing performance in legal text analysis. future work we focused on the 24 eu languages, but in the future, we would like to expand the corpus in terms of languages and jurisdictions covered. especially in china there exist many accessible sources suitable to extend the corpus. additionally, we would like to find out whether our findings on in-domain pretraining hold for multibillion generative models. finally, a detailed examination of the contents of the multilegalpile could provide valuable insights into its structure and efficacy in enhancing legal language models.