Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining (2023)

Abstract: Pretraining large language models has resulted in tremendous performance improvement for many natural language processing (NLP) tasks. While for non-domain specific tasks, such models can be used directly, a common strategy to achieve better performance for specific domains involves pretraining these language models over domain specific data using objectives like Masked Language Modelling (MLM), Autoregressive Language Modelling, etc. While such pretraining addresses the change in vocabulary and style of language for the domain, it is otherwise a domain agnostic approach. In this work, we investigate the effect of incorporating pretraining objectives that explicitly tries to exploit the domain specific language characteristics in addition to such MLM based pretraining. Particularly, we examine two distinct characteristics associated with the legal domain and propose pretraining objectives modelling these characteristics. The proposed objectives target improvement of token-level feature representation, as well as aim to incorporate sentence level semantics. We demonstrate superiority in the performance of the models pretrained using our objectives against those trained using domain-agnostic objectives over several legal downstream tasks.

## introduction pre-trained language models exhibit superior performance in several nlp tasks. most of the prominent language models optimized over masked language modelling with bert-like @xcite @xcite architecture using large unlabelled corpus to achieve state of the art results across many nlp tasks. while the sentence-level tasks like paraphrase @xcite and sentiment analysis @xcite benchmarks the capability of the model in effectively modeling the holistic representation of the sentence(s), the tokenlevel tasks like named entity recognition @xcite attempted to assess the quality of contextualized token embeddings furnished by the models. however, direct application of these models to domain-specific downstream tasks yields suboptimal performance @xcite , perhaps due to change in vocabulary and style of language. to overcome this limitation, most commonly used approach involves pre-training a language model over domain specific corpora. for instance, pubmedbert @xcite and legal-bert @xcite achieved state-ofthe-art results for the biomedical and legal domain specific tasks respectively by pre-training over domain specific corpus using a domain agnostic objective. in this paper, we argue that the performance of these models can be further improved by employing pre-training objectives that exploit the language characteristics of the domain. we examine two distinct language characteristics of the legal domain, propose pre-training objectives and finally demonstrate superior performance over domainspecific nlp tasks. legal domain departs from the generic corpora in terms of specialized vocabulary, particularly formal syntax, domain-specific knowledge semantics, etc. to the extent that it can be classified as a distinct "sub-language" @xcite , which may be addressed by mlm based pretraining. in this paper, we study the following additional domain characteristics and formulate closely aligned objectives in addition to domain agnostic objectives like mlm: the placeholders are substituted with appropriate replacements for specific documents. we include a pre-training objective for this characteristic by optimizing the model to distinguish the substitutions from the rest of the text. since, there is no labelled dataset that provides such information, we also outline the process to approximately label the data-points with placeholder spans. we leverage these two characteristics to design a pre-training strategy, and experimentally show that a language model pretrained using our strategy outperforms domain-specific language model which is trained only on domain-agnostic objectives, such as masked language modelling. the rest of the paper is organized as follows. in section 2, we discuss some prominent frameworks that provisions domain specific pre-trained models and survey important works in the legal ai. in section 3, we elucidate the details for the aforementioned legal domain characteristics and describe the objective formulation and dataset curation strategy. in section 4 discusses the training. in section 5, we briefly describe the baseline models used to compare the performance with our pretrained model for several legal domain tasks, and discuss the results. finally, in section 6, we conclude by explaining the implications of our work and discuss its natural extensions. ## domain specific objectives we now describe the legal domain characteristics which we will use for formulating the objectives. for each of the two objectives, we also describe the associated dataset used for training. we get different pretrained language model variants by incorporating various subsets of the following objectives while pretraining. the process of coming up with the right set of domain specific language characteristics requires significant exposure to the domain. the authors have been investigating several legal domain natural language processing tasks, and have been interviewing several practitioners for an extended period of time. the insights are a result of reading many legal domain documents and the interactions with domain experts. for one to extend our ideas in other domains, we expect them to require similar long exposure to the domain in question and opportunities to interact with domain experts. while we believe that the two characteristics identified in this work are not unique only to the legal domain, one will need to carefully evaluate whether the same characteristics apply to their chosen domain as well. ## training details we tune the parameters of our model using the algorithm employed for multi-task learning @xcite . this framework optimizes the language model over multiple tasks. the language model is shared across different tasks by employing same encoder with shared parameters for all the taskspecific overhead architectures. in each iteration of mini-batch gradient descent optimization, a task is randomly selected and corresponding task-specific mini-batch of data is sampled to apply single step of gradient descent using the task-specific objective. we curated the dataset for mlm pretraining by extracting text fragments from the sec corpus as curated by chalkidis et al., utilizing newline character (\n) as the delimiter. in our ablation studies to understand the impact of various terms in the pretraining objective on downstream performance, we utilized a randomly selected subset of 40, 000 text fragments to quickly assess the importance of each of the terms. thereafter, we also evaluate the performance of our model when a significantly larger corpora is provided for mlm. in this paper, the parameters of the shared language model are initialized using the weights of legal-bert (12-layer, 768-hidden, 12-heads, 110m parameters) foot_2 , a domain-specific language model pre-trained using mlm. thereafter, we investigate the performance of the model variants listed in ## results and discussion we begin this section by validating the choice of using pu learning for dynamic part detection instead of binary token classification objective. in the subsequent subsection, we describe various legal downstream tasks and their associated data to be used in comparing the performance of the models in ## conclusion in this paper, we demonstrated a novel approach to enhance the performance of domain-specific language model across several specialty downstream tasks by exploiting the language characteristics. the objectives presented in this paper may not be applicable to all domains, which is a limitation of our work, but the idea of formulating objectives for learning domain-specific characteristics can be applied to other specialty domains (biomedical, programming languages, etc.). future work might involve studying other characteristics of the legal domain and understanding their impact in downstream performance. we justified the positive impact of such pretraining across several downstream tasks by conducting extensive quantitative analysis. we conclude this section by enumerating the natural extensions of this work for future: