Privacy-Preserving Models for Legal Natural Language Processing (2022)

Abstract: Pre-training large transformer models with in-domain data improves domain adaptation and helps gain performance on the domain-specific downstream tasks. However, sharing models pre-trained on potentially sensitive data is prone to adversarial privacy attacks. In this paper, we asked to which extent we can guarantee privacy of pre-training data and, at the same time, achieve better downstream performance on legal tasks without the need of additional labeled data. We extensively experiment with scalable self-supervised learning of transformer models under the formal paradigm of differential privacy and show that under specific training configurations we can improve downstream performance without sacrifying privacy protection for the in-domain data. Our main contribution is utilizing differential privacy for large-scale pre-training of transformer language models in the legal NLP domain, which, to the best of our knowledge, has not been addressed before.

## introduction transformer-based models @xcite trained in a self-supervised fashion on a huge collection of freely accessible web texts belong to the currently most successful techniques for almost any downstream nlp task across languages or domains. their ability to 'learn' certain language properties @xcite and the need of having only a small amount of labeled data in the target domain for fine-tuning makes them superior to other approaches @xcite . moreover, additional pre-training with unlabeled target-domain data typically boosts their performance further @xcite . however, when it comes to preserving private information contained in the original large unlabeled text data, transformer models tend 'remember' way too much. @xcite show that it is possible to extract verbatim sensitive information from transformer models, such as names and addresses, even when such a piece of information had been 'seen' by the model during pre-training only once. current transformer models thus represent a threat to privacy protection, which can have harmful consequences if such models trained on very sensitive data are published, as is the current trend in sharing pre-trained models. in the legal domain, sensitive information, including names, addresses, dates of birth, are important part of many documents, such as court decisions. especially in countries with the case-law system, court decisions make the largest fraction of legal texts. however, transformer models pretrained on such corpora do not protect personal information by design, and ad-hoc solutions, e.g. whitening names in the original texts, are prone to errors and potential reconstruction attacks @xcite . existing approaches to privacy-preserving deep learning have adapted differential privacy (dp) @xcite , a rigorous mathematical treatment of privacy protection and loss. in particular, stochastic gradient descent with dp (dp-sgd) has been successfully applied to various nlp problems @xcite , including transformer pretraining @xcite . however, how well dp-regimes perform in the legal domain, pre-trained and fine-tuned across various downstream legal-nlp tasks, remains an open question. this paper addresses the following three research questions. first, what are the best strategies for pre-training transformer models to be applied in the legal domain? second, does dp-sgd training scale up to tens of gigabytes of pre-training data without ending up with an extremely big privacy budget? finally, can large-scale privacy-preserving transformers compete to their smallscale non-private alternatives? privacy-preserving nlp with differential privacy large machine learning models including transformer-based lms can be prone to privacy attacks such as membership inference attack @xcite @xcite , which means it is possible to predict whether or not a data record exists in the model's training dataset given only black-box query access to the model. it hinders the application of such models on numerous real-word tasks involving private user information. to mitigate this limitation, many recent studies devote to privacy-preserving algorithm for large nlp models. differential privacy (dp) @xcite has been taken as the goldstandard approach to ensure privacy for sensitive dataset. the main goal of privacy-preserving data analysis is to enable meaningful statistical analysis about the database while preventing leakage of individual information. the intuition behind dp is that an individual's data can't be revealed by a statistical release of the database regardless of whether or not the individual is present in the database, thus any individual shouldn't have significant influence on the statistical release. we formally introduce dp in section 3. unlike works focusing of privatization of individual texts @xcite @xcite , applying dp to training neural networks is typically done through differentiallyprivate stochastic gradient descent (dp-sgd) @xcite ; see also @xcite for a great explanation. although dp pre-training of bert has been shown to gain performance on a medical entity extraction task @xcite , how well it performs in the legal area still remains an open question. ## learning with differential privacy this section formally introduces differential privacy and can be skipped by readers familiar with that topic. ## experimental setup and data our experiments aim to find a strategy where bert can benefit from additional domain-specific dp pre-training. moreover, we explore the trade-off between the privacy budget and model utility under the best setup we obtain. privacy-protecting scenario in our scenario, we assume that we publish a pre-trained or finetuned model, to which an adversary has a full access @xcite . the model can be pretrained on (a) a public general dataset and (b) indomain, potentially sensitive legal documents, and fine-tuned on (c) a public down-stream task. our aim is to protect (b) from the adversary. ## downstream tasks and datasets we experiment with two downstream benchmark datasets, overrruling and casehold, on which 7 here we use bertwordpiecetokenizer from https://github.com/huggingface/ tokenizers , we set the vocabulary size to 30,522 which is the same as with bert-base. the overruling dataset @xcite corresponds to a binary classification task that predicts if a sentence has the meaning of voiding a legal decision made in a previous case, which is important to ensure the correctness and validity of legal agreements. the sentences in the dataset are sampled from the casetext law corpus, where positive overruling examples are manually annotated by lawyers, and negative examples are automatically generated by randomly sampling the casetext sentences because over 99% of them are non-overruling. the complete dataset contains 2400 items and the two classes are balanced. it is a relatively simple task that has already achieved state-of-the-art performance on bert-base model, since the positive examples explicitly contain 'overrule' or words with similar meaning such as disapprove, decline, reject, etc., which makes them highly distinguishable from the negative ones. the casehold (case holdings on legal decisions) is a multiple-choice qa task to select a correct holding statement among 5 potential answers that matches the given citing context from a judicial decision. @xcite construct the dataset by extracting the legal citations and the accompanying holding statements from the corpus of u.s. caselaw and using them as questions and answers respectively. here the cases contained in the casehold are removed from our legal pretraining corpus according to the case ids they provide. moreover, they search for propositions that are semantically similar to the corresponding answer from other extracted holding statements as the wrong answers according to the tf-idf similarity between them, which makes the casehold a multiple-choice qa task. the labels of the correct answers are uniformly distributed within the 5 indices 0-4. excluding some samples containing invalid labels, the full dataset we use has a total of 52,978 items. it is a challenging task and yields only a macro f1 of around 0.613 using the general bert-base @xcite . we use it to investigate whether a sufficiently difficult legal task benefits from additional domain pre-training in the private preserving scenario. for the in-domain pre-training with differential privacy, we prepare a legal corpus consisting of 14gb legal texts that are collected from three different resources (see ## results and analysis baselines our baseline results ( small-scale pre-training with dp we experimented with further pre-training of two baseline models on a small-scale 2.3gb legal sub-corpus. the goal was to efficiently explore the effect of several key hyper-parameters on dp training with a small amount of data. we trained for 29k steps at batch size 256. ## discussion here we clarify some questions and comments raised by the reviewers. is the 2% improvement worth the effort? we believe so. let's put our result into a broader context by having a closer look at results achieved by legal-bert @xcite . on three downstream tasks, they gained similar improvements. first, "in echr-cases, we [...] observe small differences [...] in the performance on the binary classification task (0.8% improvement)." second, on ner they observed an "increase in f1 on the contract header (1.8%) and dispute resolution (1.6%) subsets. in the lease details subset, we also observe an improvement (1.1%)." finally, on eurlex57k, they observed "a more substantial improvement in the more difficult multi-label task (2.5%) indicating that the legal-bert variations benefit from in-domain knowledge." moreover, our approach achieves similar gains under differential privacy guarantees. how expensive is dp training? we experimentally evaluate the running performance of different frameworks on a binary classification task (overruling) in both private and non-private cases. figure 3 show the runtimes per epoch taken from the median over 20 epochs of training. in our experiments, opacus is unable to support bert's embedding layer, although we use its official tutorial for training. this also prevents us to use it for the dp pretraining. we freeze its embedding layer for the finetuning, which reduces nearly 22% training parameters compared to other methods. by doubling the batch size each time, 64 is the maximum batch size that the current gpu can support for jax framework. opacus uses a batchmemorymanager to eliminate the limit of batch size similar to gradient accumulation, but the physical batch size it can achieve is actually much smaller than 64. this indi-cates that jax has higher memory-efficiency than opacus. the runtime of all the methods decreases significantly as the batch size grows except for py-vacy. in summary, due to the performance of jax in the dp training, the 'extra costs' are negligible and allows us to upscale dp pre-training. the title is misleading, the authors do not propose a privacy-preserving legal nlp model. if we take the definition of privacy through the lenses of differential privacy, then our pre-trained model is privacy-preserving; see, e.g., @xcite for a terminology clarification, or parallel works with the t5 language model @xcite . why even do this? the scenario in which we want to protect privacy is the following. say a company has huge amounts of in-house sensitive legal texts (e.g., contracts) which are valuable for pre-training a lm. this model is likely to be better performing on similar domains, so the company wants to offer an api or provide the model to other parties for further fine-tuning. without dp, privacy of the pre-training data can be compromised @xcite @xcite . ## conclusion this paper shows that we can combine large-scale in-domain pretraining for a better downstream performance while protecting privacy of the entire pre-training corpus using formal guarantees of differential privacy. in particular, we implemented highly-scalable training of the bert model with differentially-private stochastic gradient descent and pre-trained the model on ≈ 13 gb legal texts, using a @xmath2. the downstream results on the casehold benchmark show up to 2% improvements over baseline models with tuned hyper-parameters and models trained from scratch with a custom legal vocabulary. our main contribution is utilizing differentially-private largescale pre-training in the legal nlp domain. we believe that adapting formal privacy guarantees for training models might help overcome the difficulties of using large but potentially sensitive datasets in the legal domain.