jur BERT : A R omanian BERT Model for Legal Judgement Prediction (2021)

Abstract: Transformer-based models have become the de facto standard in the field of Natural Language Processing (NLP). By leveraging large unlabeled text corpora, they enable efficient transfer learning leading to state-of-the-art results on numerous NLP tasks. Nevertheless, for low resource languages and highly specialized tasks, transformer models tend to lag behind more classical approaches (e.g. SVM, LSTM) due to the lack of aforementioned corpora. In this paper we focus on the legal domain and we introduce a Romanian BERT model pre-trained on a large specialized corpus. Our model outperforms several strong baselines for legal judgement prediction on two different corpora consisting of cases from trials involving banks in Romania.

## introduction in recent years, a paradigm shift stormed the entire nlp field. transformer @xcite blocks allowed the development of large models that efficiently exploit the power of transfer learning. pre-training transformers on large unlabeled text data, followed by a fast fine-tuning step has become the de facto approach across the field. moreover, transformer based architectures @xcite @xcite @xcite @xcite have achieved state-of-the-art results on several generic nlp tasks ranging from natural language understanding @xcite @xcite , question answering @xcite to textto-sql @xcite . nevertheless, for low resource languages or highly specialized domains, pre-trained language models tend to underperform in part due to the lack of pre-training data or due to the generic nature of these large corpora. for this reason, specific bert models have been trained and developed for numerous lan-guages such as french @xcite , @xcite , romanian @xcite , finish @xcite , spanish @xcite and for highly specialized domains such as science @xcite , legal @xcite or biomedical @xcite . in this work we set out to investigate the possibility of adapting and applying bert models for legal judgement prediction on a small, noisy dataset, in a low resource language (romanian). the corpus we use is a realistic representation of the kind of machine-readable data that is available to practitioners in this specialized field. the data, provided by a romanian bank, is composed of original lawsuit documents, and features the most frequent types of cases pertinent to the banking domain. our contributions can be summarized as follows: the legal domain provides a wide range of different tasks in which nlp techniques can and have been used. such tasks include detection of argumentative sentences @xcite , report summarization @xcite or identification of the law areas that are relevant to a case @xcite @xcite . @xcite propose the usage of an ensemble of support vector machines (svms) on word unigram and bigrams to solve three tasks related to french supreme court cases: predicting the law area of a case, predicting case ruling and estimating the time span of a given case or ruling. similarly, @xcite use svms on textual features (mainly word n-grams) to analyze 1 https://huggingface.co/readerbench dataset scope entries size rojur pre-training 11m -robanking downstream 108k 2,212 / 1,309 brdcases downstream 149 12,119 / 12,090 for small and noisy data, such as the real world brdcases dataset we use in our work, large models may underperform compared to simpler models (ezen-can, 2020; @xcite . lately, string kernels @xcite , an efficient character-level comparison technique, have been used with promising results in low resource settings such as native language identification @xcite , dialect identification @xcite , chat understanding @xcite or automated essay scoring @xcite . ## datasets the first dataset we employ, rojur, comprises all the final rulings, containing both civil and criminal cases, published by any romanian civil court between 2010 and 2018. each sample contains: a description of the involved parties, a summary of the critical arguments made by the plaintiffs and the defendants, the legal reasoning behind the verdict and the final verdict itself. the names of the entities involved, as well as other identification details are anonymized throughout the document. notably, the document is written by a human expert (i.e. the judge presiding over the case) who may have model mlm acc nsp acc jurbert-base 89.36 99.23 jurbert-large 90.05 99. 29 we devise a second dataset, robanking, from rulings encountered in rojur. specifically, we extract common types of cases pertinent to the banking domain (e.g. administration fee litigations, enforcement appeals). from each ruling we only keep the summary of the arguments provided by the plaintiffs and the defendants, and a boolean value denoting which party was favoured in the final verdict. finally, we use brdcases, representing a collection of cases in which a particular romanian bank (brd groupe société générale romania) was directly involved. each sample contains a section with the arguments provided by the plaintiff and a section for those provided by the defendant. the content of each section is extracted from the original lawsuit files. the plaintiff section is obtained through an ocr process and by employing heuristics to remove content that may be irrelevant to the case. consequently, the text is likely to contain typographical errors and other artifacts. moreover, there may be significant differences in writing style, stemming from the possible gap in juridical knowledge between the involved parties. however, this type of input is a realistic representation of the machine readable data that is available to the attorneys handling a specific case in a romanian bank. statistics pertaining to each dataset are presented in ## model -jurbert for all intents and purposes we stick to the same model architecture and training procedure proposed by @xcite . we opt to train two variants of jurbert, namely jurbert-base and jurbertlarge, with whole word masking (wwm), each with the same vocabulary of 33k tokens, for 40 epochs on a v3-8 tpu (kindly provided by tensorflow research cloud foot_0 ). for efficiency reasons we train with sequence lengths of 128 for 90% of the training steps, while for the last 10% of steps we use sequence lengths of 512. evaluation results on the pre-training corpus, rojur, are depicted in ## evaluation we evaluate our pre-trained model on the task of predicting whether the final verdict in a legal case is favourable to the plaintiff or the defendant. to this end, we leverage robanking and brdcases. despite the similarity in structure, there are significant differences between the two datasets, as presented in section 3. we extensively explore different fine-tuning strategies, to enable efficient transfer learning and to prevent catastrophic forgetting. inspired by previous approaches @xcite @xcite we investigate several strategies for dealing with long texts (e.g. using the first, middle or last part of the text), pooling type (i.e. <cls>, mean or max), layer unfreezing (e.g. optimize all weights all throughout the training process, gradual unfreezing of layers), learning rate (i.e. constant, discriminative or slanted triangular learning rate), dropout value, final fully connected layers (sizes and numbers) and different combinations of mentioned strategies. we note that the setup for finding the best training strategy is iterative: we test all aspects of a given strategy, select the best and only then moving to the next step while retaining previous strategies. henceforth, we refer to the best strategy foot_1 as the optimized strategy. both downstream tasks are framed as binary classification tasks (i.e. given the arguments, who wins the case). the results are reported using k-fold cross validation, with 5 folds for robanking and 10 folds for brdcases. cross-entropy loss is minimized using the adam optimizer (kingma and ba, 2015) as each model is trained 3 times. finally we report the mean auc and the standard deviation for each model. ## results the results on robanking, using only the plaintiff's plea, are presented in in while the two corpora differ in essence, fine-tuning on robanking greatly improves the downstream performance on brdcases. finally, we note the importance of the pre-training step (on rojur corpus) as jurbert consistently and significantly outperforms robert for all considered models and experiments, but especially for the real-word use case ( lastly, we investigate the effectiveness of simple handcrafted features for the legal judgement prediction task. handcrafted features include the county and the year for each case and, while already present in text, they were also added in the final decision layer in categorical form (one-hot encoding). experiments with said features are marked accordingly (+ handcrafted) in tables 3,4 and 5. in the case of robanking, the added features are not especially relevant, yielding mixed results: same or worse mean auc when using only the plaintiff's plea; see ## conclusions in this work, to the best of our knowledge, we employed the first study on applicability of stateof-the-art nlp methods for romanian legal judgement prediction. we pre-trained, released and evaluated our models with promising results on two highly practical datasets, robanking and brd-cases. on the first dataset, that contains a humangenerated summary of key arguments, our model, jurbert, outperforms the considered baselines. turning to the second dataset, that contains all the original arguments of the involved parties, jurbert is just slighty better than much simpler models, as it struggles to handle such long texts. especially in this case, the limitations of bert-like models with regards to the maximum input size are a significant factor that hampers their performance. our proposed methodology for legal judgment prediction on real world cases involves three steps. the first step is pre-training a bert model on a general purpose collection of cases (in our case rojur). the second step includes further training on a subset of the previous corpus (in our case robanking), in which the model learns to predict the verdict having access only to the summarized arguments of the involved parties. the final and the most important step in our work is training and evaluating on the industry-provided cases. one of our key findings is that the second step in this methodology is crucial for obtaining good results for legal judgment prediction. we emphasize that this methodology is language independent and can easily be applied to similar tasks. major improvement areas for our approach are the development and integration of more refined handcrafted features (e.g. the type of court or the identity of the judge) and tackling the problem of long texts that greatly exceed the maximum input size of our model. for the latter, lines of research include summarization of long texts or employing methods of increasing the inherent sequence length limit of transformer models @xcite . dragomir radev. 2019b. sparc: cross-domain semantic parsing in context. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4511-4523, florence, italy. association for computational linguistics. manzil zaheer, guru guruganesh, kumar avinava dubey, joshua ainslie, chris alberti, santiago ontanon, philip pham, anirudh ravula, qifan wang, li yang, et al. 2020. big bird: transformers for longer sequences. advances in neural information processing systems, 33. zhengyan zhang, xu han, zhiyuan liu, xin jiang, maosong sun, and qun liu. 2019. ernie: enhanced language representation with informative entities. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 1441-1451, florence, italy. association for computational linguistics.