L e XF iles and L egal LAMA : Facilitating E nglish Multinational Legal Language Model Development (2023)

Abstract: In this work, we conduct a detailed analysis on the performance of legal-oriented pre-trained language models (PLMs). We examine the interplay between their original objective, acquired knowledge, and legal language understanding capacities which we define as the upstream, probing, and downstream performance, respectively. We consider not only the models’ size but also the pre-training corpora used as important dimensions in our study. To this end, we release a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented PLMs. We release two new legal PLMs trained on LeXFiles and evaluate them alongside others on LegalLAMA and LexGLUE. We find that probing performance strongly correlates with upstream performance in related legal topics. On the other hand, downstream performance is mainly driven by the model’s size and prior legal knowledge which can be estimated by upstream and probing performance. Based on these findings, we can conclude that both dimensions are important for those seeking the development of domain-specific PLMs.

## introduction following closely the advances in the development of nlp technologies, the legal nlp literature is flourishing with the release of many new resources, including large legal corpora @xcite , datasets @xcite @xcite @xcite , and pre-trained legal-oriented language models (plms) @xcite @xcite . benchmark suites @xcite @xcite to evaluate the performance of plms in a more systematic way * equal contribution. despite this impressive progress, there is still not a thorough study on (a) how plms trained under different settings (pre-training corpora, size of the model) perform across different legal @xcite what sort of knowledge such models have acquired from pre-training, and (c) how important is domain (legal) specificity vs general (cross-domain) legal knowledge. furthermore, often times, legal nlp relies on datasets without drawing clear lines and comparisons between the various legal systems they may reflect. a legal system may be defined as a set of rules adopted and enforced at a given governance level, which may be national, regional or international @xcite , e.g., uk, eu, us, coe, etc. we define the upstream evaluation as the task plms are explicitly designed to do: masked language modelling (mlm) @xcite . we then probe for specific legal concepts that are legalsystem specific, in a similar fashion as @xcite did using the "language models analysis" (lama) framework. finally, we assess the plms performance in lexglue @xcite downstream tasks. more importantly, we explore how the aforementioned factors (upstream, and probing performance) interplay and relate to downstream performance. our contributions are: (a) we release lexfiles, a new diverse english legal corpus including 11 sub-corpora that cover legislation and case law from 6 primarily english-speaking legal systems (eu, coe, canada, us, uk, india). the corpus comprises approx. 6 million documents which sum up to approx. 19 billion tokens. ## lexfiles corpus the lexfiles is a new diverse english multinational legal corpus that we created including 11 distinct sub-corpora ( the corpus contains approx. 19 billion tokens. in comparison, the pile of law corpus released by henderson* et al. (2022) comprises 32 billion in total, where the majority (26/30) of sub-corpora come from the united states of america (usa), hence the corpus as a whole is biased towards the us legal system in general, and the federal or state jurisdiction in particular, to a significant extent. the lexfiles's sub-corpora are: (a) eu legislation. we release 93.7k eu laws (regulations, decisions, directives) published in eur-lex, the website of the eu publication office. 1 (b) eu case law. we release 29.8k eu court decisions, mainly issued from the court of 1 https://eur-lex.europa.eu/ justice (cjeu), published in eur-lex. 1 (c) uk legislation. we release 52.5 uk laws published in uk.legislation.gov.uk, the official website of the uk national archives. 2 (d) uk case law. we release 47k uk court decisions published in the british and irish legal information institute (bailii) database. 3 (e) us legislation. we re-distribute 518 us state statutes (legislation) originally published by henderson* et al. (2022). (f) us case law. we release 4.6m us decisions (opinions) published by court listener, 4 a web database hosted by the free law project. 5 (g) us contracts. we release 622k us contracts (agreements) obtained from us securities and exchange commission (sec) filings, which are publicly available from the sec-edgar 6 database. (h) canadian legislation. we release 6k canadian laws (acts, regulations) published in the official legislation portal of canada. 7 (i) canadian case law. we re-distribute 13.5k canadian decisions (opinions) originally published by henderson* et al. (2022). (j) ecthr case law. we release 12.5k decisions ruled by the european court of human rights (ecthr) published in hudoc, foot_6 the database of ecthr. the lexfiles is pre-split into training and test subsets to provide a fair ground for comparing the performance of plms that have not been trained in the training set. we use the training subset of the lexfiles corpus to train 2 new transformer-based languages models, dubbed lexlms (section 4.1), and evaluate their mlm performance across many other already available plms (section 4.2). ## legallama benchmark language model analysis (lama) @xcite is a probing task that is designed to assess specific capabilities of plms. the general framework of lama is to let plms predict a target token behind a [mask] given its context, e.g., "paris is the capital of [mask]", where the answer is 'france'. legallama is a new probing benchmark suite inspired by this framework. it includes 8 sub-tasks that aim to assess the acquaintance of legal knowledge that plms acquired in the pretraining phase in a zero-shot fashion. such tasks cannot be resolved by laypersons or even law professionals that are not experts in the specific fields of law in many cases. 9 the acquaintance of legal knowledge can be interpreted as some form of primitive understanding of the law, for specific aspects in very controlled (limited) settings -limited legal concepts under a specific jurisdiction-. as @xcite mentioned: "rather than asking whether a language model understands or not, we should ask to what extent, and in which way, a model understands." we further extend the lama framework by allowing plms to predict multi-token targets. take for example the "drug trafficking" offence under the "drug-related" crimes of the us legislation. using the roberta tokenizer, this term is split into two tokens, that is "drug" and "trafficking". we replace thus the "drug trafficking" phrase with two [mask] tokens, and then ask the model to predict these tokens simultaneously. we evaluate the overall performance of plms using the macro-averaged mean reciprocal rank (mrr) @xcite over the set of labels (not the entire vocabulary). 10 in the case of multi-token targets, we average the mrr over the predicted tokens. 11 note that legallama examples come from the test subset of the related lexfiles sub-corpora in order to have a fair comparison between models trained or not on the lexfiles training sets. we provide a concrete example in figure 1 , and describe the tasks in detail: ## release of resources we release our code base to assure reproducibility and let others extend our study by experimenting with other plms, or develop new ones. 19 the new lexlm models (section 4.1), the lexfiles corpus foot_18 (section 2), and the legallama benchmark foot_19 (section 4.3) are available on hugging face hub @xcite . 22 ## conclusions and future work in this work, we introduced a multinational english legal corpus (lexfiles) and a legal knowledge probing benchmark (legallama) to facilitate training and detailed analysis of legal-oriented plms. we also released two new legal plms and evaluate them alongside others on legallama and lexglue. based on our analysis (section 4), we make the following general observations: in future work, we plan to further analyze the learning dynamics of legal language models by comparing their representations with representations derived from legal knowledge bases. given the availability of the new resources, the development of instruction-following @xcite fine-tuned legal-oriented gpt-like @xcite models is also an anticipated direction.