Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocation (2024)

Abstract: Language models are capable of memorizing detailed patterns and information, leading to a double-edged effect: they achieve impressive modeling performance on downstream tasks with the stored knowledge but also raise significant privacy concerns. Traditional differential privacy based training approaches offer robust safeguards by employing a uniform noise distribution across all parameters. However, this overlooks the distinct sensitivities and contributions of individual parameters in privacy protection and often results in suboptimal models. To address these limitations, we propose ANADP, a novel algorithm that adaptively allocates additive noise based on the importance of model parameters. We demonstrate that ANADP narrows the performance gap between regular fine-tuning and traditional DP fine-tuning on a series of datasets while maintaining the required privacy constraints.

## introduction language models have achieved remarkable success and shown impressive abilities in a wide range of tasks @xcite @xcite . their advanced capabilities in memorizing detailed information and patterns in data as well as making connections among them have not only helped language models to achieve impressive modeling performance on downstream tasks, but also raised significant and ubiquitous privacy concerns if it is not properly handled @xcite @xcite . differential privacy (dp) is a principled framework for mitigating privacy risks, providing theoretical guarantees that prevent inferring the presence or absence of an individual's data in a model's output @xcite . conventional dp-enhanced fine-tuning offers robust safeguards by assuming a uniform noise distribution across all parameters to protect privacy @xcite @xcite . recent work has begun exploring dp in parameter efficient fine tuning (peft) @xcite , which is built on the same assumption on the additional tunable parameters. unfortunately, such an assumption overlooks the distinct sensitivities and contributions of individual parameters in privacy protection and often results in suboptimal models. in this paper, we introduce anadp, a novel dp method that adaptively distributes the noise and privacy budget among a language model's parameters during fine-tuning, based on their importance to the model at a given training step. our work was inspired by @xcite , who utilized the sensitivity and uncertainty of parameters for model pruning. importantly, our approach not only respects the inherent heterogeneity of parameter significance but also maintains strong privacy protection. the proposed integration addresses the key challenges in effectively measuring the contributions of parameters and ensures that models are trained stably. we demonstrate that anadp consistently improves performance over the traditional dp fine-tuning under the same privacy budget and bridges the gap between traditional dp and non-dp fine-tuning (no privacy guarantee). the contributions of our work are summarized below: in machine learning, @xcite introduced the widely used differentially private stochastic gradient descent (dp-sgd). adaptive differential privacy is a recent development. research @xcite have been developed to preserve adaptive dp in deep neural networks. however, these methods fall short in capturing the complex parameter interactions within transformers, potentially leading to suboptimal models and trade-offs between privacy and utility. fine-tuning and peft. full fine-tuning used to be a prominent approach but can be resourceintensive and less efficient @xcite . peft has emerged as another option for effectively training llms. many peft techniques, such as lora @xcite , adapters @xcite , and prefix tuning @xcite have been proposed to tune small, additional modules instead of the whole model. @xcite provided a unified view revealing the connections among various parameter-efficient transfer learning methods. recent work by @xcite introduced adalora to dynamically adjust the amount of parameter tuning based on the task and model requirements. despite their efficiency, integrating these methods with privacypreserving techniques remains an area that requires further exploration. ## the anadp model the integration of differential privacy for language model fine-tuning is crucial for deploying llms in privacy-sensitive applications. in this work, we introduce anadp, an adaptive noise allocation dp training method based on the importance score of models' parameters, which provides a generic solution that can be applied to a wide range of llms. the fundamental idea is that adding less noise to the parameters that are more important and more to the less important parameters would help improve the model's utility given the same privacy we follow @xcite 's definition of dp. specifically, we achieve dp through a randomized algorithm a over an output space s. given a privacy budget ϵ and error probability δ, we say a is (ϵ, δ)-differentially private ((ϵ, δ)-dp) if for any neighboring datasets d and d ′ , which differ in exactly one data record, the following inequality holds: parameter importance. in order to gauge parameters' importance, our work is inspired by @xcite , which calculates importance based on the sensitivity and uncertainty of the parameter for model pruning. we use the moving averages of the sensitivity and uncertainty of the model parameters at training step t: this formulation ensures that parameters with moderate sensitivity but high uncertainty are still considered important, which prevents prematurely discarding parameters that could become important as training progresses. importance normalization. using i t as defined in eq. 5 may lead to zero-gradients. therefore, we smooth the importance scores this means that the overall noise added to the model will follow the same distribution as that of @xcite who uses a uniform noise across all parameters. as the overall noise added is the same, anadp satisfies the (ϵ, δ)-dp guarantees following the proof of @xcite . the smoothed importance score is utilized to adaptively add noise to the gradient. replicating eq. 2, our new gradient is: ## conclusion this paper introduces anadp, a novel approach to integrating dp with language model fine-tuning in both the full fine-tuning and peft settings and dynamically adjusting the noise added to the gradients, based on measuring model parameters' importance. we demonstrated that under the same privacy budget, anadp consistently outperforms the standard dp-sgd training on different benchmark datasets. while performance degradation re-mains between our method and non-dp training, we achieved consistent reduction of the gap, in both the fine-tuning and peft settings. our additional exposure risk analysis shows that anadp provides privacy protection comparable to the standard dp-sgd training. we hope this work enables better deployment of privacy-preserving language models and encourages future research on adaptive dp for language model training.