Differentially Private Learning Needs Better Model Initialization and Self-Distillation (2025)

Abstract: Differentially private SGD (DPSGD) enables privacy-preserving training of language models, but often reduces utility, diversity, and linguistic quality. We introduce DPRefine, a three-phase method that initializes a model using data synthesis from a small pre-trained LM with rigorous filtering, applies DP finetuning on private data, and performs self-distillation to refine outputs. This approach significantly outperforms vanilla DPSGD, with AlpacaEval preferring DPRefine’s generations in 78.38% of cases across all datasets and metrics, while also demonstrating substantial improvements in lexical diversity, achieving 85.31% in MSTTR and 86.82% in Jaccard similarity. Our fine-grained analysis reveals that DPRefine reduces linguistic errors in generated text by 84%, mitigating grammar errors, spelling mistakes, and missing punctuation commonly associated with DPSGD. It also reduces inconsistencies present in non-private models, such as fabricated details and misattributed quotes. We find that small models like GPT-2 and T5 are effective for initialization and distillation, highlighting their potential in enabling scalable and efficient deployment of high-performing, privacy-preserving language models with improved linguistic quality and consistency.

## introduction training machine learning models on private data offers significant potential for enhancing performance on domain-specific tasks, particularly in natural language processing @xcite @xcite . however, the use of sensitive information raises critical privacy concerns. differentially private stochastic gradient descent (dpsgd) has emerged as a prominent technique to bound information leakage during model training by clipping gradients and adding calibrated noise during optimization @xcite . while dpsgd provides strong privacy guarantees, the introduced noise and gradient modifications lead to significant challenges: decreased model utility @xcite , less diverse text generation due to distribution smoothing @xcite , and notably, as our analysis reveals, increased linguistic errors in generated text ( a common scenario in practice involves domainspecific tasks with limited private labeled data, where organizations aim to leverage existing pre-trained language models while preserving privacy. simply apply-ing dpsgd to fine-tune these models on private data often yields poor results, particularly when the private dataset is small @xcite . recent work has shown that leveraging better hand-crafted features @xcite or features from large pre-trained language models @xcite can improve the privacy-utility tradeoff in differentially private learning. however, these approaches have limitations: smaller pre-trained models offer limited benefits, and fine-tuning larger models on private data may be infeasible due to proprietary concerns or infrastructure limitations. this raises a critical question: can we develop small, domain-specific language models that achieve high performance without requiring large private datasets or large, pre-trained models? in this paper we introduce dprefine, building on the intuition that model performance under differential privacy heavily depends on model initialization and feature representation @xcite . additionally, recent advances in nlp have demonstrated the effectiveness of data synthesis @xcite , filtering and self-distillation in improving the performance of smaller models @xcite . our method has three phases, depicted in figure 1 : in the first phase we create a strong initialization by generating high-quality synthetic data using a small pre-trained language model (e.g., gpt-2) with rigorous filtering -importantly, this phase operates entirely independently of any private data. in the second phase we apply dpsgd to fine-tune this initialized model on the private labeled data, representing the only stage where private data is accessed. finally, we apply self-distillation where the dp model generates new training data for further refinement, again without accessing the original private data. we evaluate dprefine on three datasets across two domain-specific tasks: summarization, xsum @xcite and pubmed @xcite , and paraphrasing, mrpc @xcite . our experiments demonstrate that dprefine significantly outperforms vanilla dpsgd, with alpacaeval @xcite preferring dprefine's generations in 78.38% of cases across all datasets and metrics. beyond standard metrics, we conduct a fine-grained manual error analysis of the generated text, constructing a taxonomy of both linguistic errors (e.g., grammar and spelling mistakes) and non-linguistic errors, which we refer to throughout the paper as inconsistencies and hallucinations (e.g., misattributed quotes or ungrounded contextual details), as shown in ## proposed method we present dprefine, depicted in figure 1 , a threephase method designed to enhance the linguistic quality of differentially private large language models for domain-specific tasks like summarization and paraphrasing. our method integrates data synthesis, differentially private fine-tuning, and self-distillation to produce 1 https://github.com/uvm-plaid/private_llm_ generation high-quality outputs while maintaining strong privacy guarantees. our method integrates three key phases: data synthesis, differentially private fine-tuning, and self-distillation. in the first phase, data synthesis provides better initialization and richer feature representations, allowing the model to learn key patterns without privacy concerns. in the second phase, differentially private fine-tuning preserves these learned features while ensuring privacy, adding noise selectively to maintain model robustness. finally, in the third phase, self-distillation combined with careful filtering further refines the model's outputs, correcting privacy-induced errors and boosting linguistic quality, all while maintaining strong privacy guarantees. ## experimental results in this section we evaluate the efficacy of dprefine. we conduct extensive quantitative and qualitative analysis: (1) overall evaluation using llm-as-a-judge, (2) reference-based, targeted assessments and (4) diversity evaluations. then, we provide a fine-grained, qualitative error-analysis which categorizes the types of errors different models make, and exemplifies the improvements that dprefine provides. finally, we perform ablation studies. ## conclusion we introduced dprefine, a three-phase method to improve the utility and linguistic quality of differen-tially private language models. dprefine addresses dpsgd's limitations by combining data synthesis, privacy-preserving fine-tuning, and self-distillation, resulting in more accurate, coherent, and diverse outputs. our experiments show dprefine outperforms dpsgd, reducing language errors and inconsistencies while preserving privacy. by using synthetic data for initialization and self-distillation for refinement, dprefine mitigates privacy noise and balances quality with privacy guarantees. fine-grained analysis confirms that dprefine significantly reduces inconsistencies and errors compared to dpsgd, making it a promising approach for developing high-performing, privacy-preserving models for sensitive tasks. future work could explore its application to additional nlp tasks and larger models.