Parameter-Efficient Legal Domain Adaptation (2022)

Abstract: Seeking legal advice is often expensive. Recent advancements in machine learning for solving complex problems can be leveraged to help make legal services more accessible to the public. However, real-life applications encounter significant challenges. State-of-the-art language models are growing increasingly large, making parameter-efficient learning increasingly important. Unfortunately, parameter-efficient methods perform poorly with small amounts of data, which are common in the legal domain (where data labelling costs are high). To address these challenges, we propose parameter-efficient legal domain adaptation, which uses vast unsupervised legal data from public legal forums to perform legal pre-training. This method exceeds or matches the fewshot performance of existing models such as LEGAL-BERT on various legal tasks while tuning only approximately 0.1% of model parameters. Additionally, we show that our method can achieve calibration comparable to existing methods across several tasks. To the best of our knowledge, this work is among the first to explore parameter-efficient methods of tuning language models in the legal domain.

## introduction seeking legal advice from lawyers can be expensive. however, a machine learning system that can help answer legal questions could greatly aid laypersons in making informed legal decisions. existing legal forums, such as legal advice reddit and law stack exchange, are valuable data sources for various legal tasks. on one hand, they provide good sources of labelled data, such as mapping legal questions to their areas of law (for classification), as shown in figure 1 . on the other hand, they contain hundreds of thousands of legal questions that can be leveraged for domain adaptation. furthermore, questions on these forums can serve as a starting point for tasks that do not have labels found directly in the dataset, such as classifying the severity of a legal question. in this paper, we show that this vast unlabeled corpus can improve performance on question classification, opening up the possibility of studying other tasks on these public legal forums. in the past few years, large language models have shown effectiveness in legal tasks @xcite . a widespread method used to train these models is finetuning. although finetuning is very effective, it is prohibitively expensive; training all the parameters requires large amounts of memory and requires a full copy of the language model to be saved for each task. @xcite has shown great promise by tuning under 1% of the parameters and still achieving comparable performance to finetuning. unfortunately, prefix tuning performs poorly in low-data (i.e., fewshot) settings @xcite , which are common in the legal domain. conveniently, domain adaptation using large public datasets is an ideal setting for the legal domain with abundant unlabelled data (from public forums) and limited labelled data. to this end, we introduce prefix domain adaptation, which performs domain adaptation for prompt tuning to improve fewshot performance on various legal tasks. overall, our main contributions are as follows: ## related works forums-based datasets public forums have been used extensively as sources of data for machine learning. sites like stack overflow and quora have been used for duplicate question detection @xcite . additionally, many prior works have used posts from specific sub-communities (called a "subreddit") on reddit for nlp tasks, likely due to the diversity of communities and large amount of data provided. @xcite used a large number of internet memes from multiple meme-related subreddits to predict how likely a meme is to be popular. other works, such as @xcite , label posts from biomedical subreddits for biomedical entity linking. similar to the legal judgement prediction task, @xcite suggest using "crowdsourced data" from reddit to perform ethical judgement prediction; that is, they use votes from the "r/amitheasshole" subreddit to classify who is "in the wrong" for a given real-life anecdote. we explore using data from stack exchange and reddit, which has been vastly underexplored in previous works for the legal domain. ## methods here we outline our approach and other baselines for comparison. roberta to establish a baseline, we train roberta @xcite for downstream tasks using full model tuning (referred to as "full finetuning"). in addition to the state of the art performance that roberta achieves in many general nlp tasks, it has also shown very strong performance in legal tasks @xcite . unlike some transformer models, roberta has an encoder-only architecture, and is normally pre-trained on the masked language modelling task @xcite . we evaluate the model on both of its size variants, roberta-base (approximately 125m parameters) and roberta-large (approximately 335m parameters). legal-bert we evaluate the effectiveness of our approach against legal-bert, a fully domain-adapted version of bert for the legal domain @xcite . in our experiments, we further perform full finetuning for each downstream task. the number of parameters in legal-bert (109m) is comparable to roberta-base (125m), as used in our other experiments. ## datasets we evaluate each of the approaches listed above on three different datasets. ## experimental setup we test our approaches under a fewshot setting, where prompt tuning is known to perform poorly @xcite . we use roberta-base and roberta-large @xcite for our experiments. to simulate a fewshot learning scenario, we randomly undersample the train and validation sets for each dataset, ensuring that the distribution of train and validation data roughly matches. additionally, we vary the amount of data undersampled to study how fewshot size affects performance. in these tasks, we use a validation size of 256 (much smaller than the original) to represent true fewshot learning better @xcite . considering that fewshot learning is quite unstable, we ran all of our experiments five times, using the seeds { 10, 20, 30, 40, 50 }. we provide more training details in appendix a. there is often confusion around whether fewshot sizes represent the number of samples per class or the total number of samples @xcite . in our results, the fewshot sizes we show are the exact number of training samples used (i.e., total training samples). the exact number of samples is listed in ## results and discussion we make a few observations on our results, shown in in contrast to other datasets, the echr dataset's train and test split have different distributions. in fewshot scenarios with very little data (i.e., 4-16 examples), we find that prefix tuning based approaches perform better than full finetuning; this suggests that prefix tuning approaches are more robust to changes in distribution (and possibly noise). we also note that bert with truncation (maximum token length of 500) performs a lot better than initially reported in @xcite , who report an f1 worse than random guessing (macro f1 of 66.5 in ours, 17 in theirs). we believe this underperformance of finetuning bert could be caused by a mistake in their training process. in figure 4 , we show the trend of performance on reddit data as the number of samples increases. prefix domain adaptation is comparable to finetuning, consistently outperforming regular prefix tuning. as shown by the larger shaded area around the lines, the stability of finetuning is worse than prefix domain adaptation for this task. performance gradually converges increases as more data is given to each method. larger models typically provide better performance on various tasks. thus, we run experiments using roberta-large (over 2x larger than roberta-base) to see how our approach scales to larger models. as seen in ## conclusions in this paper, we propose a novel training framework, prefix domain adaptation, aiming to domain adapt a prompt using a large corpus of domainspecific text. we show that our approach matches or outperforms legal-bert or related techniques in performance while training fewer (0.1%) parameters. with our technique, we improve fewshot performance and convergence time compared to other parameter-efficient methods. we believe this will make fewshot data more usable (and thus reduce data labelling costs) while using parameterefficient methods to reduce computational and storage costs. additionally, we introduce two new datasets (legal advice reddit and law stack exchange) to lay foundations for future work in legal decisionmaking systems; as opposed to formal documents in echr, our two datasets are closer to legal questions asked by laypersons, helping to promote access to justice for all.