Mixed-domain Language Modeling for Processing Long Legal Documents (2023)

Abstract: The application of Natural Language Processing (NLP) to specialized domains, such as the law, has recently received a surge of interest. As many legal services rely on processing and analyzing large collections of documents, automating such tasks with NLP tools such as language models emerges as a key challenge since legal documents may contain specialized vocabulary from other domains, such as medical terminology in personal injury text. However, most language models are general-purpose models, which either have limited reasoning capabilities on highly specialized legal terminology and syntax, such as BERT or ROBERTA, or are expensive to run and tune, such as GPT-3.5 and Claude. Thus, in this paper, we propose a specialized language model for personal injury text, LEGALRELECTRA, which is trained on mixed-domain legal and medical corpora. We show that as a small language model, our model improves over general-domain and single-domain medical and legal language models when processing mixed-domain (personal injury) text. Our training architecture implements the ELECTRA framework but utilizes REFORMER instead of BERT for its generator and discriminator. We show that this improves the model’s performance on processing long passages and results in better long-range text comprehension.

## introduction following the striking success of large language models, the development of specialized language models that are adapted to domain-specific syntax and vocabulary has received increasing attention. in many application domains, such as medicine or the law, raw data is often given in text format, creating a need for nlp tools that aid in automating data processing. general-domain models typically lack expressivity on specialized domains. rare * work done while intern at claudius legal intelligence. in this work, we focus on pre-trained models for applications in the law with specialization in personal injury civil suits. personal injury legal cases arise when an individual suffers harm from an accident or injury, and someone else might be legally responsible for that harm. this can be due to negligence, reckless behavior, or intentional misconduct. personal injury law allows the injured person to go to civil court and get a legal remedy (damages) for all losses stemming from the accident or injury. this can include compensation for medical expenses, lost wages, and pain and suffering, among others. personal injury claims are quite common in lawsuits. personal injury claims constitute a substantial part of civil litigation. therefore, having a language model that concentrates on these cases is vital for natural language processing research in the legal field. the analysis of legal proceedings relies on access to case data, which is often given in the form of legal documents. processing such documents presents a challenge for general-purpose language models, due to the specialized terminology and syntax conventions in the law. a natural remedy is the development of a specialized legal-domain language model that is trained on legal text @xcite . there are three key challenges in developing legal-domain models: data on civil legal proceedings is often siloed due to privacy restrictions and there are few publicly available, curated data sets. hence, we focus on the development of a small, specialized language model, which can be trained, tuned and run on small data sets in a costeffective way. legalrelectra provides a pretrained model for personal injury text with a special focus on enabling long-range text comprehension. it lends itself to a plethora of applications that involve legal case documents, including summarization or extraction of key information for civil suits, identifying patterns and trends in legal proceedings and identifying precedent in past cases, among others. in addition, legal language models may aid in summarizing and analysing legal scholarship. we demonstrate the applicability of legalrelectra on a downstream task, for which we train a name entity recognition (ner) model. in practise, such an ner model may be applied to extract key legal information from case documents, such as the identities of plaintiff and defendant, medical injuries and civil case type. with that, ner enables a simple summarization of civil suits, which may serve as a basis for further case analytics. contributions. our main contributions are: the resulting model, legalrelectra, is well equipped to process long passages of mixeddomain text (here, personal injury cases, i.e., mixed legal and medical domain), as we demonstrate in a range of benchmark experiments against state of the art general and domain-specific models. recently, there has been growing interest in utilizing machine learning in the legal domain (legal artificial intelligence), including for judgment prediction @xcite , the analysis of fairness in legal proceedings @xcite @xcite , as well as legal document analysis @xcite @xcite . the development of specialized legal language models can aid in the latter. many variations of transformers that are adapted to specialized domains have been proposed in the literature (e.g., @xcite @xcite @xcite ) and demonstrated to be more efficient and accurate than bert on their specialized domains. however, most of these pre-trained domainspecific models do not adopt new frameworks but rely on the classical bert architecture. to the best of our knowledge, only lawformer @xcite , which is a chinese legal domain pre-trained model, utilizes longformer (beltagy et al., 2020), instead of bert. this architecture changes allows lawformer to process longer passages (up to 4,096 tokens). here, we adopt the electra @xcite framework, which has been shown to be more dataand parameter-efficient than bert. in addition, our model utilizes reformer as generator and discriminator, which significantly improves over the maximum text length of longformer (up to 8,092 tokens). in this paper, we show that the resulting model (legalrelectra) outperforms bert, as well as state-of-the-art single-domain adapted models on a downstream task (named entity recognition, sec. 5.3; legal case retrieval, sec. 5.4;). moreover, some current state-of-the-art domain-specific language models are pretrained using a general-domain tokenizer (e.g., the bert tokenizer in legal-bert) to preprocess the input data. here, we train a domain-specific tokenizer to pre-train legalrelectra. our results indicate that this improves the training process and downstream tasks (sec. 5.1 and 5.2). lastly, we note that all current domain-specific models focus on adaption to a single domain. in contrast, our model is trained on both legal and medical domains motivated by applications in processing personal injury text. our experimental results demonstrate that legalrelectra performs competitively against general-purpose and singledomain models (sec. 5.3). ## language modeling for legal text in this section, we outline the architecture of our model, describe its pretraining process, and detail the tasks on which it is assessed. ## experiment setup this section delineates the experimental configurations for both the pretraining phase and the finetuning for named entity recognition. since the legal case retrieval task is assessed without finetuning, we do not provide detailed setup specifics for it. ## conclusions in this paper, we introduced legalrelectra, a language model that is specialized to process mixed legal and medical domain (personal injury) text. this was achieved by pre-training with a corpus consisting of legal, medical and mixed domain (personal injury) text. we demonstrate in validation experiments that legalrelectra outperforms general-purpose language models (e.g., bert), as well as specialized legal-domain models (e.g., legal-bert) on legal and mixed-domain ner. as technical contributions, we proposed a novel model architecture that allows for improved performance on long-range text comprehension. legalrelectra provides a pretrained model for personal injury text with a special focus on enabling long-range text comprehension. it lends itself to a plethora of applications that involve legal case documents, including summarization or extraction of key information for civil suits, identifying patterns and trends in legal proceedings, identifying precedent in past cases, among others. in addition, legal language models may aid in summarizing and analysing legal scholarship. we demonstrate the applicability of legalr-electra in downstream tasks by training a named entity recognition (ner) model and utilizing the pretrained model directly as a legal case retriever. in practice, the ner model can extract crucial legal information from case documents, including plaintiff and defendant identities, medical injuries, and civil case types. this enables a basic summary of civil suits, serving as a foundation for further case analysis. the legal case retriever, on the other hand, can identify past cases similar to new ones, providing valuable reference points.