K nowledge SG : Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server (2024)

Abstract: The success of large language models (LLMs) facilitate many parties to fine-tune LLMs on their own private data. However, this practice raises privacy concerns due to the memorization of LLMs. Existing solutions, such as utilizing synthetic data for substitution, struggle to simultaneously improve performance and preserve privacy.They either rely on a local model for generation, resulting in a performance decline, or take advantage of APIs, directly exposing the data to API servers. To address this issue, we propose KnowledgeSG , a novel client-server framework which enhances synthetic data quality and improves model performance while ensuring privacy. We achieve this by learning local knowledge from the private data with differential privacy (DP) and distilling professional knowledge from the server. Additionally, inspired by federated learning, we transmit models rather than data between the client and server to prevent privacy leakage.Extensive experiments in medical and financial domains demonstrate the effectiveness of *KnowledgeSG*. Our code is now publicly available at https://github.com/wwh0411/KnowledgeSG.

## introduction the world has witnessed the tremendous success of large language models (llms) across a variety of tasks @xcite . such success has attracted numerous parties to fine-tune their customized llms by leveraging their local private data @xcite @xcite . nonetheless, training such llms on private data could cause significant privacy concerns, since llms are shown to memorize sensitive information from the training data @xcite . to address this privacy issue, a series of methods have been proposed to circumvent the direct usage of private data by using synthetic data for substitution @xcite @xcite . specifically, some methods use application programming interface (apis) to generate diverse instructions, directly exposing private data to the api server @xcite . while others rely solely on a local base model, which leads to a quality degradation in synthetic data and eventually lower model performance @xcite . therefore, existing methods suffer from the tradeoff between privacy risk and model performance. in this work, we aim to efficiently enhance synthetic data quality while maintaining strict privacy protection. to achieve this goal, we propose knowledgesg (knowledge-based synthetic data generation), a novel client-server framework which leverages a professional server to assist the local client in data generation under theoretical privacy guarantee. our framework compensates the quality gap between synthetic and original data observed in previous works @xcite by efficiently distilling knowledge from the professional model deployed on the server, rather than relying merely on the local model. additionally, unlike api-based methods, we draw inspiration from federated learning @xcite by transmitting model weights instead of data for knowledge exchange, thereby improving privacy protection. specifically, on the client side, we fine-tune the local model with differentially privacy (dp) to learn local knowledge from private data within a privacy budget. for convenient and secure communication between the client and server, we transmit only the lora @xcite adapter of the dp-finetuned model instead of directly transmitting private data. on the server side, raw synthetic instructions are first generated using the uploaded local model. these instructions are then judged by the professional model for quality filtration in an efficient manner @xcite . once filtered, the top instructions are fed directly into the professional model to generate accurate responses, bypassing the need to generate potentially incorrect responses from the local model. finally, the dp-finetuned local model is further optimized by fine-tuning it with the top instructions and corresponding responses to boost its performance. upon completion, the optimized model is transmitted back to the client, concluding the entire process. we conduct a series of experiments on two privacy-sensitive domains: medicine and finance. the results prove the effectiveness of our proposed framework on both privacy and performance benchmarks. it is worth mentioning that our method gains a relative improvement of 120.39% than non-private training measured by medical free-form evaluation, even surpassing alpacare @xcite , the professional model we deploy. to conclude, our main contributions are: ## conclusions this paper addresses the challenge of preserving privacy while fine-tuning large language models on sensitive data. to improve the quality of synthetic data, an aspect often overlooked in previous works, we introduce a novel client-server framework called knowledgesg. specifically, knowl-edgesg leverages knowledge distillation from a professional server, by prompting it to provide judgments and corrections for raw synthetic data generated by the dp-finetuned base model. inspired by federated learning, knowledgesg transmits models rather than data through a specially designed transmitting unit to ensure privacy. we conduct extensive experiments, and the results validate the effectiveness of knowledgesg. the framework achieves a relative improvement of 120.39% compared to the non-private training, as measured by medical free-form evaluation. additionally, knowledgesg significantly reduces the reconstruction rate from 97.13 to 0.87, demonstrating its strong privacypreserving capabilities.