Public Data Assisted Differentially Private In-Context Learning (2025)

Abstract: In-context learning (ICL) in Large Language Models (LLMs) has shown remarkable performance across various tasks without requiring fine-tuning. However, recent studies have highlighted the risk of private data leakage through the prompt in ICL, especially when LLMs are exposed to malicious attacks. While differential privacy (DP) provides strong privacy guarantees, it often significantly reduces the utility of in-context learning (ICL). To address this challenge, we incorporate task-related public data into the ICL framework while maintaining the DP guarantee. Based on this approach, we propose a private in-context learning algorithm that effectively balances privacy protection and model utility. Through experiments, we demonstrate that our approach significantly improves the utility of private ICL with the assistance of public data. Additionally, we show that our method is robust against membership inference attacks, demonstrating empirical privacy protection.

## introduction with the emergence of large language models (llms), in-context learning (icl) has demonstrated remarkable performance across various tasks by enabling models to infer from provided examples without modifying internal parameters @xcite @xcite . this flexibility allows llms to adapt to diverse domains without explicit training, leading to its widespread adoption @xcite . meanwhile, concerns about privacy leakage have been raised regarding the deployment of icl in llms @xcite @xcite ). for a concrete example, consider the icl scenario depicted in figure 1 , where private patient treatment records are used as demonstration examples. a malicious attacker aiming to identify † corresponding author. private clinical record data may try inference attack targeting llm. the llm with an icl algorithm, without privacy protection, could potentially expose sensitive clinical data to attackers. such exposure of personal health information may contravene data protection regulations like the general data protection regulation (gdpr), which mandates appropriate technical and organizational measures to ensure data security and protect individual privacy rights @xcite . differential privacy (dp) has emerged as the gold standard for rigorous privacy protection across multiple domains, including computer vision, rec-ommendation systems, natural language processing, and census data @xcite . the essence of differential privacy is to ensure that the output of an algorithm is minimally influenced by the inclusion or exclusion of any individual's data, thereby reducing the risk of privacy leakage. additionally, the level of privacy protection can be adjusted by tuning the privacy protection parameter ε. differential privacy is often employed in conjunction with sample and aggregate techniques, where multiple noisy outputs are combined to enhance privacy. however, when dp is integrated into icl algorithms, two main challenges arise. first, the performance of the icl algorithm deteriorates significantly under strong privacy protection, where output perturbation negatively impacts the model's utility. to mitigate this degradation, we incorporate public data into our icl framework to minimize utility loss. the second challenge is to design a privacy-preserving aggregation method for language generation outputs. given the high dimensionality inherent in the output space of llm responses, it is essential to transform them into a lower-dimensional representation while ensuring that they remain reconstructable in the original response space. to achieve this, we project llmgenerated responses into semantic embeddings and form semantic groups for private aggregation. we evaluate our private icl framework with dp guarantees on question-answering tasks (chatdoctor, @xcite ) and a document summarization task (samsum, @xcite ). the experimental results show that our private icl framework performs comparably to non-private baselines at a strong privacy protection level (ε = 1) and outperforms the private data-only counterpart at the same privacy protection level. we further demonstrate that using out-of-distribution (ood) public data, as well as in-distribution (id) public data, is beneficial for minimizing utility degradation. moreover, we show that our private icl framework is robust against empirical privacy attacks by simulating membership inference attacks on the model. in summary, our main contributions are the following: in prediction methods, many works build upon the pate framework @xcite , where each teacher model is trained on a subset of the private dataset, and the aggregate of the teacher ensemble is used for prediction. apart from methods with differential privacy guarantees, text sanitization methods @xcite and unlearning methods @xcite have also been proposed to protect individual privacy. @xcite propose promptpate, where unlabeled public data is privately annotated by a teacher ensemble trained on the private dataset, and the annotated public data is then used as demonstration examples in the prompt for icl. @xcite propose a token-level differentially private icl framework that generates synthetic texts as demonstration examples for subsequent queries. however, the output space of this icl framework is limited to the label space and does not cover long-text responses. @xcite develop an icl framework with dp guarantees that can handle long-text responses. to address the highdimensionality issue of the output text space, they propose embedding space aggregation and keyword space aggregation. however, the performance with strong privacy protection is not satisfactory compared to the non-private counterpart. ## private icl with public data in this section, we present our private icl framework. we first explain each stage of the private icl framework and then present the dp privacy analysis of the framework. the schematic diagram of our private-icl framework is presented in figure 2 and the algorithm is described in algorithm 1. ## experiments and results in this section, we present the privacy-utility tradeoff of our private icl models in question answering and summarization tasks. we begin by evaluating model performance using in-distribution (id) public data and out-of-distribution (ood) public data. finally, we assess the empirical effectiveness of private icl through a membership inference experiment. in addition to ood public data, we also evaluate the performance of private icl with noisy public data in the appendix 5.5. ## enhancing utility and efficiency in private icl in many domains, high-quality public datasets may be unavailable or inferior to the private datasets at hand, making it challenging to apply our framework. to address this issue, we allocate a small privacy budget to augment the public dataset. our augmentation follows a simple approach: first, we collect public examples that are closest to privatized centroids with a budget of ε = 1. then, we instruct the gpt-turbo model to generate additional data based on these filtered public examples. the performance of the proposed augmentation technique on the summarization task is reported in figure 5 . the results show that even with a small privacy budget, the augmentation method is effective, showing about 1 ∼ 2 rouge-l score difference compared to id public setting and outperforming the baseline that relies solely on ood public datasets. finally, when no public data is available, a private generator can be an effective solution by synthesizing samples for augmentation. ## conclusion we propose a private in-context learning (icl) framework that leverages public data. to address high dimensionality, we project the llm output into a semantic group space. public data is used in ensemble aggregation and selection to mitigate utility degradation from differential privacy. experiments show an effective privacy-utility tradeoff, with notable improvements from public data. additionally, empirical tests demonstrate strong defense against privacy and membership inference attacks across all privacy levels. finally, we present techniques for public data enhancement and inference acceleration, offering a practical solution for realworld applications.