J uri BERT : A Masked-Language Model Adaptation for F rench Legal Text (2021)

Abstract: Language models have proven to be very useful when adapted to specific domains. Nonetheless, little research has been done on the adaptation of domain-specific BERT models in the French language. In this paper, we focus on creating a language model adapted to French legal text with the goal of helping law professionals. We conclude that some specific tasks do not benefit from generic language models pre-trained on large amounts of data. We explore the use of smaller architectures in domain-specific sub-languages and their benefits for French legal text. We prove that domain-specific pre-trained models can perform better than their equivalent generalised ones in the legal domain. Finally, we release JuriBERT, a new set of BERT models adapted to the French legal domain.

## introduction domain-specific language models have evolved the way we learn and use text representations in natural language processing. instead of using general purpose pre-trained models that are highly skewed towards generic language, we can now pre-train models that better meet our needs and are highly adapted to specific domains, like medicine and law. in order to achieve that, models are trained on large scale raw text data, which is a computationally expensive step, and then are used in many downstream evaluation tasks, achieving state-of-the-art results in multiple explored domains. the majority of domain-specific language models so far are applied to the english language. @xcite published french word vectors from large scale generic web content that surpassed previous non pre-trained word embeddings. furthermore, @xcite introduced camem-bert, a monolingual language model for french, that is used for generic everyday text, and proved its superiority in comparison with other multilingual models. in the meantime, domain-specific language models for french are in lack. there is an even greater shortage when it comes to the legal field. @xcite mentioned the importance of using state-of-the-art technologies to support law professionals and provide them with guidance and orientation. given this need, we introduce ju-ribert, a new set of bert models pre-trained on french legal text. we explore the use of smaller models architecturally when we are dealing with very specific sub-languages, like french legal text. thus, we publicly release juribert foot_0 in 4 different sizes online. ## related work previous work on domain-specific text data has indicated the importance of creating domain-specific language models. these models are either adaptations of existing generalised models, for example bert base by @xcite trained on general purpose english corpora, or pre-trained from scratch on new data. in both cases, domain-specific text corpora are used to adjust the model to the peculiarities of each domain. a remarkable example of adapting language models is the research done by @xcite who introduced biobert, a domain-specific language representation model pre-trained on large scale biomedical corpora. biobert outperformed bert and other previous models on many biomedical text mining tasks and showed that pre-training on specific biomedical corpora improves performance in the field. similar results were presented by @xcite that introduced scibert and showed that pre-training on scientific-related corpus improves performance in multiple domains, and by @xcite who showed that fin-bert, pre-trained on financial communication cor-pora, can outperform bert on three financial sentiment classification tasks. moving on to the legal domain, bambroo and awasthi (2021) worked on legaldb, a distilbert model @xcite pre-trained on english legal-domain specific corpora. legaldb outperformed bert at legal document classification. elwany et al. (2019) also proved that pre-training bert can improve classification tasks in the legal domain and showed that acquiring large scale english legal corpora can provide a major advantage in legal-related tasks such as contract classification. furthermore, @xcite introduced legalbert, a family of english bert models, that outperformed bert on a variety of datasets in text classification and sequence tagging. their work also showed that an architecturally large model may not be necessary when dealing with domainspecific sub-languages. a representative example is legal-bert-small that is highly competitive with larger versions of legalbert. we intent to further explore this theory with even smaller models. despite the increasing use of domain-specific models, we have mainly been limited to the english language. on the contrary, in the french language, little work has been done on the application of text classification methods to support law professionals, with the exception of @xcite that managed to achieve state-of-the-art results in three legal-domain classification tasks. it is also worth mentioning @xcite who introduced criminelbart, a fine-tuned version of barthez @xcite . criminelbart is specialised in criminal law by using french canadian legal judgments. all in all, no previous work has adapted a bert model in the legal domain using french legal text. ## downstream evaluation tasks in order to evaluate our models we will be using two legal text classification tasks provided by the court of cassation, the highest court of the french judicial order. the subject of the first task is assigning the court's claimant's pleadings, "mèmoires ampliatifs" in french, to a chamber and a section of the court. this leads to a multi-class classification task with 8 different imbalanced classes. in the second task is to classify the claiment's pleadings to a set of 151 subjects, "matières" as stated in french. figure 5 in appendix shows the support of the matières in the data. as we can see in figure 1 the 10 recessive matières have between 7 to 1 examples in our dataset. we decided to remove the last 3 matières as they have less than 3 examples and therefore it is not possible to split them in train, test and development sets. ## juribert we introduce a new set of bert models pre-trained from scratch in legal-domain specific corpora. we train our models on the masked language modeling (mlm) task. this means that given an input text sequence we mask tokens with 15% prob-ability and the model is then trained to predict these masked tokens. we follow the example of @xcite and choose to train significantly even smaller models, including bert-tiny and bert-mini. the architectural details of the models we pre-trained are presented in ## methods pre-training details all the models were pretrained for 1m steps. a learning rate of 1e -4 was used along with an adam optimizer (β 1 =0.9, β 2 =0.999) with weight decay of 0.1 and a linear scheduler with 10,000 warm-up steps. all the models were pre-trained with batch size of 8 expect for juribert base and juribert-fp that used batches of size 4. for the pre-training we used an nvidia gtx 1080ti gpu. fine-tuning details our models were fine-tuned on the downstream evaluation task using the same classification head as @xcite that consists of a dense layer with tanh function followed by a dense layer with softmax activation function and dropout layers with fixed dropout rate of 0.1. we applied grid-search to the learning rate on a range of {2e -5, 3e -5, 4e -5, 5e -5}. we used an adam optimizer along with a linear scheduler that provided the training with 100k warm-up steps. we train for a maximum of 30 epochs with patience of 2 epochs on the early stopping callback and checkpoints for the best model. for the classification we use only the paragraphs starting with 'alors que' from the pleadings dataset, as they include all the important information for the correct chamber and section. this was suggested by a lawyer from the court of cassation as the average size of a mèmoire ampliatif is extremely big, from 10 to 30 pages long. by using the 'alors que' paragraphs we have text sequences with average size of 800 tokens. for the chambers and sections classification task we split the data in 14% development and 16% test data. for the matières classification we split the data in 17% development and 14% test data and stratify in order to have all classes represented in each subset. both tasks use a fixed batch size of 4. for the fine-tuning we used an nvidia gtx 1080ti gpu. ## results the results on the downstream evaluation tasks are presented in tables 4 and 5 . we compare our models with two camembert versions, base and large, and with barthez, a sequence-to-sequence model dedicated to the french language. camem-bert has been pre-trained on 138gb of french raw text from the oscar corpus. despite the difference in pre-training corpora size, with our model using only 6.3gb of legal text, juribert small managed to outperform both camembert base and camembert large. this further proves the importance of domain-specific language models in natural language processing and transfer learning. despite our expectations, the performance of juribert base does not exceed the performance of its smaller equivalent models. we attribute this peculiarity in the usage of smaller batch sizes when pre-training juribert base and also the fact that larger models usually need more computational resources and more time and data in order to converge. juribert small also outperforms barthez on the chambers and sections evaluation task, which is pre-trained on 66gb of french raw text and usually used for generative tasks. on the matières classification task barthez is the dominant model with juribert small being second. we infer that the complexity of the second task benefits more from the robustness and size of barthez than from the specific-domain nature of juribert. comparing our models with the same architectures, it becomes apparent that all task-specific ju-ribert models perform better than their equivalent domain-specific juribert models besides using only 4gb of pre-training data. the results confirm that a bert model pre-trained from scratch only on the corpus that is then used for fine-tuning can perform better than a domain-specific one on the same task as we expected. juribert-fp outperforms juribert base and achieves similar results with camembert base on the chambers and sections classification task. this shows that further pre-training a general purpose language model can have better results than training from scratch. however, it did not manage to outperform juribert small in both tasks, which can be attributed to the smaller batch size used during pre-training and to the size of the model as mentioned before for juribert base. unfortunately, there are no smaller versions of camem-bert available to further test this theory. on the matières classification task, juribert-fp still outperforms juribert base. on the contrary, it performs worse than camembert base. along with the state-of-the-art results of barthez, this leads us to believe that in order to achieve better results in more complex tasks juribert models require more pre-training corpora. all in all, juribert small achieves equivalent results with previous larger generic language models with an accuracy of 83.95% on the first task a appendix