LEGAL - BERT : The Muppets straight out of Law School (2020)

Abstract: BERT has achieved impressive performance in several NLP tasks. However, there has been limited investigation on its adaptation guidelines in specialised domains. Here we focus on the legal domain, where we explore several approaches for applying BERT models to downstream legal tasks, evaluating on multiple datasets. Our findings indicate that the previous guidelines for pre-training and fine-tuning, often blindly followed, do not always generalize well in the legal domain. Thus we propose a systematic investigation of the available strategies when applying BERT in specialised domains. These are: (a) use the original BERT out of the box, (b) adapt BERT by additional pre-training on domain-specific corpora, and (c) pre-train BERT from scratch on domain-specific corpora. We also propose a broader hyper-parameter search space when fine-tuning for downstream tasks and we release LEGAL-BERT, a family of BERT models intended to assist legal NLP research, computational law, and legal technology applications.

## introduction pre-trained language models based on transformers @xcite , such as bert @xcite and its variants @xcite @xcite , have achieved state-of-the-art results in several downstream nlp tasks on generic benchmark datasets, such as glue @xcite , squad @xcite , and race @xcite . typically, transfer learning with language models requires a computationally heavy step where the language model is pre-trained on a large corpus and a less expensive step where the model is finetuned for downstream tasks. when using bert, the first step can be omitted as the pre-trained models are publicly available. being pre-trained on generic corpora (e.g., wikipedia, children's books, etc.) bert has been reported to under-perform in specialised domains, such as biomedical or scientific text @xcite . to overcome this limitation there are two possible strategies; either further pre-train (fp) bert on domain specific corpora, or pre-train bert from scratch (sc) on domain specific corpora. consequently, to employ bert in specialised domains one may consider three alternative strategies before fine-tuning for the downstream task (figure 1 ): (a) use bert out of the box, (b) further pre-train (fp) bert on domain-specific corpora, and (c) pre-train bert from scratch (sc) on domain specific corpora with a new vocabulary of sub-word units. in this paper, we systematically explore strategies (a)-(c) in the legal domain, where bert adaptation has yet to be explored. as with other specialised domains, legal text (e.g., laws, court pleadings, contracts) has distinct characteristics compared to generic corpora, such as specialised vocabulary, particularly formal syntax, semantics based on extensive domain-specific knowledge etc., to the extent that legal language is often classified as a 'sublanguage' @xcite @xcite . note, however, that our work contributes more broadly towards a better understanding of domain adaptation for specialised domains. our key findings are: (i) further pre-training (fp) or pre-training bert from scratch (sc) on domainone shortcoming is that all previous work does not investigate the effect of varying the number of pre-training steps, with the exception of @xcite . more importantly, when fine-tuning for the downstream task, all previous work blindly adopts the hyper-parameter selection guidelines of @xcite without further investigation. finally, no previous work considers the effectiveness and efficiency of smaller models (e.g., fewer layers) in specialised domains. the full capacity of larger and computationally more expensive models may be unnecessary in specialised domains, where syntax may be more standardized, the range of topics discussed may be narrower, terms may have fewer senses etc. we also note that although bert is the current state-of-the-art in many legal nlp tasks @xcite , no previous work has considered its adaptation for the legal domain. ## legal-bert: a new family of bert models for the legal domain training corpora: to pre-train the different variations of legal-bert, we collected 12 gb of diverse english legal text from several fields (e.g., legislation, court cases, contracts) scraped from publicly available resources (see ## experimental setup pre-training details: to be comparable with bert, we train legal-bert for 1m steps (approx. 40 epochs) over all corpora (section 3), in batches of 256 samples, including up to 512 sentencepiece tokens. we used adam with learning rate of 1e-4, as in the original implementation. we trained all models with the official bert code foot_3 using v3 tpus with 8 cores from google cloud compute services. legal nlp tasks: we evaluate our models on text classification and sequence tagging using three datasets. eurlex57k @xcite ) is a large-scale multi-label text classification dataset of eu laws, also suitable for few and zero-shot learning. echr-cases @xcite ) contains cases from the european court of human rights @xcite and can be used for binary and multi-label text classification. finally, contracts-ner @xcite is a dataset for named entity recognition on us contracts consisting of three subsets, contract header, dispute resolution, and lease details. we replicate the experiments of @xcite when fine-tuning bert for all datasets. 5 tune your muppets! as a rule of thumb to fine-tune bert for downstream tasks, @xcite suggested a minimal hyperparameter tuning strategy relying on a gridsearch on the following ranges: learning @xmath0, 3e-5, 4e-5, 5e-5}, number of training @xmath1, 4}, batch @xmath2, 32} and fixed dropout rate of 0.1. these not well justified suggestions are blindly followed in the literature @xcite @xcite . given the relatively small size of the datasets, we use batch @xmath3, 8, 16, 32}. interestingly, in preliminary experiments, we found that some models still underfit after 4 epochs, the maximum suggested, thus we use early stopping based on validation loss, without a fixed maximum number of training epochs. we also consider an additional lower learning rate (1e-5) to avoid overshooting local minima, and an additional higher drop-out rate (0.2) to improve regularization. figure 4 (top two bars) shows that our enriched grid-search (tuned) has a substantial impact in most of the end-tasks compared to the default hyper-parameter strategy of @xcite . 6 we adopt this strategy for legal-bert. ## experimental results pre-training results: figure 2 presents the training loss across pre-training steps for all versions of legal-bert. legal-bert-sc performs much better on the pre-training objectives than legal-bert-small, which was highly expected, given the different sizes of the two models. at the end of its pre-training, legal-bert-small has similar loss to that of bert-base pre-trained on generic corpora (arrow in figure 2 ). when we consider the additional pre-training of bert on legal corpora (legal-bert-fp), we observe that it adapts faster and better in specific sub-domains (esp. echr cases, us contracts), comparing to using the full collection of legal corpora, where the training loss does not reach that of legal-bert-sc. ## conclusions and future work we showed that the best strategy to port bert to a new domain may vary, and one may consider either further pre-training or pre-training from scratch. thus, we release legal-bert, a family of bert models for the legal domain achieving state-of-art results in three end-tasks. notably, the performance gains are stronger in the most challenging end-tasks (i.e., multi-label classification in echr-cases and contract header, lease details in contracts-ner) where in-domain knowledge is more important. we also release legal-bert-small, which is 3 times smaller but highly competitive to the other versions of legal-bert. thus, it can be adopted more easily in low-resource test-beds. finally, we show that an expanded grid search when fine-tuning bert for end-tasks has a drastic impact on performance and thus should always be adopted. in future work, we plan to explore the performance of legal-bert in more legal datasets and tasks. we also intend to explore the impact of further pre-training legal-bert-sc and legal-bert-small on specific legal sub-domains (e.g., eu legislation).