A ra L egal- BERT : A pretrained language model for A rabic Legal text (2022)

Abstract: The effectiveness of the bidirectional encoder representations from transformers (BERT) model for multiple linguistic tasks is well documented. However, its potential for a narrow and specific domain, such as legal, has not been fully explored. In this study, we examine the use of BERT in the Arabic legal domain and customize this language model for several downstream tasks using different domain-relevant training and test datasets to train BERT from scratch. We introduce AraLegal-BERT, a bidirectional encoder transformer-based model that has been thoroughly tested and carefully optimized with the goal of amplifying the impact of natural language processing-driven solutions on jurisprudence, legal documents, and legal practice. We fine-tuned AraLegal-BERT and evaluated it against three BERT variants for the Arabic language in three natural language understanding tasks. The results showed that the base version of AraLegal-BERT achieved better accuracy than the typical and original BERT model concerning legal texts.

## introduction the impressive performance of bidirectional encoder representations from transformers (bert) @xcite inspired numerous authors to try and improve the original bert. such follow-up research progresses in several directions, including the development of specific solutions for various thematic domains. this is necessary because the vocabulary used in some fields significantly differs from the language used for everyday purposes and may contain the specific meanings of certain phrases or atypical relationships between contextual elements. this problem can be partially resolved through domain-specific adjustments to the training process. a good example of this approach is demonstrated in @xcite , who created legal-bert specifically for mining legal text in english, improving the output of the standard transformer algorithm in this domain. another form of the bert concept was successfully adapted by @xcite , who created models that were pretrained on a compilation of scientific and biomedical data from various fields, achieving significantly better performance on scientifically related natural language processing (nlp) tasks. these examples show that bert is currently far from a finished model and that its effectiveness could be further enhanced, at least for relatively narrowly defined tasks. for the arabic language, several bert-based models have consistently demonstrated superior performance on numerous linguistic tasks requiring semantic understanding, outperforming all benchmarks on public datasets such as the arabic ner corpus (anercorp) or the arabic reading comprehension dataset (arcd), such as the works presented by @xcite and mbert by google @xcite . this is largely a consequence of the efficient transfer learning inherent in this model, involving a high computational cost because this approach requires huge collections of training examples, followed by fine-tuning for specific downstream tasks. a significant advantage of bert is that the training phase can be skipped because a pretrained version of the model can be used and trained further. however, @xcite @xcite has shown that a generic approach to pretraining does not work well when bert must be used in a domain with highly specific terminology, such as legal, science, or medicine. there are two possible responses to this issue: continue specializing in a pretrained model or train a model from scratch with relevant materials from the domain. in this study, we built a language model from scratch based on the original bert @xcite , which is specific to arabic legal texts. the aim was to improve the performance on most state-of-the-art language understanding and processing tasks, especially related to arabic legal texts. we believe that the specific nature of legal documents and terminology needs to be considered because it affects the way sentences and paragraphs are constructed in this field. the extent of formal and semantic differences is such that some authors describe the linguistic content used for legal matters as almost a language of its own @xcite . by focusing on a single domain, the arabic legal text, this study attempts to reveal means of adapting an nlp model to fit any thematic domain. based on our experiments, we can confirm that pretraining bert with examples from the arabic legal domain from scratch provides a better foundation for working with documents containing arabic legal vocabulary than using the vanilla version of the algorithm. we introduce aralegal-bert, a transformer-based model that has been thoroughly tested and carefully optimized with the goal of amplifying the impact of nlp-driven solutions on jurisprudence, legal documents, and legal practice. we fine-tuned aralegal-bert and evaluated it against three bert variants for arabic in three natural language understanding (nlu) tasks. the results show that the base version of aralegal-bert achieved better accuracy than the general and original version of the bert model, in regard to legal text. aralegal-bert is a particularly efficient model that can keep up with the output of computationally intensive models while producing its findings faster and using far fewer resources. consequently, the base version of the model was observed to have the ability to achieve comparable accuracy to larger versions of the large general and original version of the bert model when they were trained with domain-relevant examples similar to those used to test our model. ## related work in a very short time, the transformer @xcite architecture has become the gold standard for machine learning methods in the field of linguistics @xcite . the unprecedented success of bert combined with its flexibility has led to a proliferation of tools based on it, built with a more narrowly defined vocabulary @xcite . @xcite is an example of such specialization and could have considerable practical value, given the number of arabic speakers worldwide. because the model is trained for some of the most common nlp tasks and has proven effective across regional variations in morphology and syntax, this language model has the potential to become a standard tool for analyzing arabic text. the pretraining and finetuning procedures described in this work may not be optimal; however, the output of the localized model clearly indicated that the initial approach was correct. with further refinement, the model can become sufficiently reliable for a wide range of real-world applications. however, these models are based on data, most of which were collected from modern standard arabic terms, and these language models may fail when the language switches to colloquial dialects @xcite . in addition, the performance of these language models can be affected when dealing with a language for a specific domain with special terms, such as scientific, medical, and legal terms @xcite . the majority of domain-specific bert models are related to scientific or medical texts, and legal texts; however, the texts are all in english @xcite @xcite . in a study by @xcite , the main area of interest was clinical practice; therefore, the authors developed two different variants by pretraining basic bert and biobert with examples from this domain, with positive results in both cases. another interesting related project was conducted by @xcite , resulting in the creation of scibert, a whole branch of variations optimized for use with scientific documents. in this case, two different optimization strategies, including additional training and training from scratch using documents comprising scientific terminology, were tested, with both approaches yielding measurable improvements. a study by @xcite involved pretraining transformer models for english legal text by comparing three possible approaches to adapt bert to thematic content niches: 1) using the vanilla model without any modifications, 2) introducing pretraining with datasets that contain examples from the target domain in addition to the standard training, and 3) using only domain-relevant samples to train bert from scratch. essentially, all the adaptations of the standard bert model that involve fine-tuning use the same approach to select hyperparameters as outlined in the original bert formulation, without even questioning it. another research gap is observed regarding the possibility of using shallow models to perform domain-specific tasks. the impressive generalizability of deep models with several layers could be argued as wasted when the model operates in a narrowly defined field where linguistic rules are more streamlined and vocabulary volume more limited. despite bert being the most successful deep learning model for various tasks related to the legal sphere, there have been no published attempts to develop a unique variation for this type of content, especially in arabic, inspiring this study. therefore, this is the first study to build a bertbased language model for legal texts in arabic. ## aralegal-bert: transformer model pretrained with arabic legal text to optimize bert to work with arabic legal documents, we followed the same procedures in the original bert model @xcite ; however, for the arabic language, we followed the same procedure in arabert (antoun et al., 2020). ## conclusions and future work our experimental results show that a bert model pretrained for a specific domain is better than the typical language models, for specific nlu tasks. therefore, we present aralegal-bert, which was trained exclusively for arabic legal texts and is capable of making highly accurate predictions on three main nlu tasks: legal text classification, named entity recognition, and keyword extraction. essentially, the level of difficulty of a task is correlated with the gains from choosing the right training strategy as the importance of domain-specific vocabulary and semantics becomes more pronounced. the tested version of aralegal-bert is the base, cost-efficient version suitable for a broad range of arabic legal text applications. our future work will focus on additional possibilities for pretraining other models, such as the electra, roberta and xlm-r models for several nlu tasks in the arabic legal domain with small, base, and large versions.