DP - GTR : Differentially Private Prompt Protection via Group Text Rewriting (2025)

Abstract: Prompt privacy is crucial, especially when using online large language models (LLMs), due to the sensitive information often contained within prompts. While LLMs can enhance prompt privacy through text rewriting, existing methods primarily focus on document-level rewriting, neglecting the rich, multi-granular representations of text. This limitation restricts LLM utilization to specific tasks, overlooking their generalization and in-context learning capabilities, thus hindering practical application. To address this gap, we introduce DP-GTR, a novel three-stage framework that leverages local differential privacy (DP) and the composition theorem via group text rewriting. DP-GTR is the first framework to integrate both document-level and word-level information while exploiting in-context learning to simultaneously improve privacy and utility, effectively bridging local and global DP mechanisms at the individual data point level. Experiments on CommonSense QA and DocVQA demonstrate that DP-GTR outperforms existing approaches, achieving a superior privacy-utility trade-off. Furthermore, our framework is compatible with existing rewriting techniques, serving as a plug-in to enhance privacy protection. Our code is publicly available at anonymous.4open.science for reproducibility.

## introduction the rise of llms in natural language processing has catalyzed an urgent research focus on their security and privacy @xcite @xcite , including investigations into differential privacy (dp) techniques to mitigate the risk of sensitive information leakage @xcite @xcite . while dp, the gold standard for computational privacy, has seen broad adoption in machine learning, existing text-based dp methods face significant challenges. these methods generally fall into four categories: training-based optimizations (e.g., dp-sgd @xcite ), embedding perturbations @xcite , document-level paraphrasing @xcite , and in-context learning (icl) enhancements @xcite . however, training-based approaches are computationally expensive, embedding perturbations can compromise semantic coherence, and icl often neglects client-side prompt privacy. document paraphrasing offers a promising balance between privacy and utility. state-of-theart methods achieve differentially private nexttoken generation using the exponential mechanism (em) @xcite , replacing the standard softmax. initial work employed decoder-only models like finetuned @xcite , progressing to encoder-decoder @xcite and encoder-only architectures (e.g., bart, roberta) @xcite . dp-prompt @xcite leverages prompt learning for zero-shot paraphrasing, and recent advancements combine dp post-processing with adversarial fine-tuning @xcite . a critical limitation, however, persists: lack of finegrained control over the privacy-utility trade-off. current em-based dp methods provide only coarse-grained control via the privacy budget, hindering practical deployment. most approaches (except dp-prompt) also necessitate resourceintensive fine-tuning. while document-level paraphrasing preserves more contextual information than embedding perturbations, it often overlooks word-level privacy vulnerabilities. these limitations highlight the need for a training-free, finegrained privacy solution that fully leverages textual information, a capability well-suited to the icl paradigm of llms. prior work on dp in icl has predominantly focused on server-side, global dp implementations, often using a "sample-and-aggregate" ap-proach @xcite to privately partition and aggregate context databases @xcite . client-side prompt privatization, in contrast, requires the stronger guarantees of local dp (ldp), protecting individual data points rather than entire datasets. this distinction creates a significant gap between global and local dp in icl, motivating the need for approaches that bridge it. addressing these gaps, we propose dp-gtr, a three-stage, differentially private prompt protection framework built upon a novel group text rewriting (gtr) mechanism (see figure 1 ). dp-gtr is designed to provide fine-grained control over the privacy-utility trade-off while remaining compatible with existing paraphrasing techniques. in stage-1, gtr generates multiple client-side paraphrases of an input prompt, forming a "rewriting group" that preserves rich contextual information and enables bag-of-words-like count analysis. notably, gtr connects local and global dp principles on the client-side. stage-2 uses these counts for fine-grained privacy-utility control, identifying potentially sensitive private consensus keywordswords appearing frequently across paraphrases despite dp-driven variations. we mitigate this risk by releasing a fixed number of these keywords or using a differentially private aggregator, and select the lowest-perplexity paraphrase to maximize output quality. stage-3 suppresses the identified keywords, limiting privacy leakage, and uses the selected paraphrase as an icl example to improve utility. in addition, we evaluate dp-gtr in a realistic question-answering (qa) scenario, simulating real-world llm usage. our key contributions are: ## related work global vs. local dp: differential privacy text sanitization methods are classified into global differential privacy (global-dp) and local differential privacy (local-dp) based on where the privacy mechanism is applied. in global-dp, data is aggregated centrally before applying the privacy mechanism, while methods like dp-sgd use differentially private optimization techniques for training text models @xcite @xcite . dp-icl operates within a "sample-and-aggregate" framework by perturbing the embedding and vocabulary selected for release @xcite . in contrast, local-dp incorporates the differential privacy mechanism before data reaches the centralized processor, typically affording stronger privacy protection @xcite . local-dp: private document release methods are categorized into three tiers based on where noise is added: word-level, sentence-level, and documentlevel. at the word level, noise is added to word embeddings, and the perturbed vectors are then mapped to the nearest vocabulary word @xcite @xcite . carvalho et al. (2023) employ the exponential mechanism for token selection, while chen et al. (2023a) propose customized token mappings for individual words. moreover, meisenbacher et al.'s (2024a) study generates multiple candidate perturbations using various word embedding models, and mattern et al. (2022) highlights that word-level approaches inherently lack contextual information. similarly, sentence-level methods inject noise into sentence embeddings (reimers, 2019; meehan et al., 2022). document-level: at the document level, paraphrasing technologies are grouped into three categories based on model architecture. mattern et al. (2022) uses a decoder-only fine-tuned gpt-2 model. later work adopts encoder-decoder models, such as a bart-based approach @xcite with sensitivity clipping via thresholding and pruning @xcite . encoderonly methods, like dp-mlm @xcite , use a roberta-based masked language model for fine-tuning. these methods require finetuning. in contrast, dp-prompt @xcite introduces a zero-shot prompt learning paradigm using black-box llms, and meisenbacher and matthes (2024) employs post-processing and adversarial fine-tuning to enhance rewriting. ## preliminaries threat model. we assume an adversary attacker on the side of the cloud-based model vendor, whose objective is to extract private information (such as personal identifiable information) from the confidential transmission content. the adversary's access is limited to a customized prompt supplied by the client, but they are free to facilitate their attack. ## dp-gtr existing document-level prompt sanitization methods often employ the em for privacy-preserving rewriting. however, these coarse-grained approaches, relying on a single ϵ for the entire document (prompt) rewriting, struggle to balance privacy and utility. critically, noise introduced during rewriting irreversibly alters textual elements, hindering utility recovery. maintaining acceptable utility thus necessitates low initial noise levels, requiring a high privacy budget and consequently reducing actual privacy protection. furthermore, a high privacy budget under the em can even lead to complete data exposure. to address these limitations, we propose dp-gtr, a word-and document-level hybrid prompt privacy adopted framework that leverages group text rewriting and post-processing to enhance privacy while maintaining high utility under dp guarantees. dp-gtr enables low-noise paraphrasing to identify and suppress the generation of privacy-sensitive terms with high exposure. ## ablation study the ablation experiments are conducted on two additional configurations: one with references but without keywords, and another with keywords but without references. the configuration is gpt-3.5 and docvqa. the results are shown in tab. 2. experimental results indicate that using either isolated privacy keywords or the reference alone yields significantly lower performance than our approach. removing the reference substantially impairs the model's ability to generate valid questions, where utility's rouge-1 score is no more than 1%. omitting private keywords disrupts the trade-off, resulting in slight privacy leakage that is disproportionate to the utility loss. optimal performance is achieved only when both elements are combined. we attribute this to the fact that, although privacy keywords serve as suppression targets, llms can leverage these keywords to produce effective paraphrases, thereby enhancing utility. ## adversarial attack based on the threat model in section 3, we design two adversarial experiments: static and adaptive attacks. in the static setting, the attacker is unaware of dp-gtr and directly attempts to recover the original question from cloud-received paraphrases. in the adaptive setting, the attacker fully understands dp-gtr, replicates its framework, and leverages privacy consensus keywords to infer the original question. in the results of adversarial attacks demonstrate that dp-gtr offers strong robustness against both static and adaptive threats, with no observed increase in question rouge-1 similarity, indicating no privacy leakage. these findings underscore effectiveness in safeguarding against adversarial risks. ## conclusion this paper proposes dp-gtr, a novel three-stage local differential privacy (ldp) framework that leverages differentially private paraphrasing and the composition theorem through group text rewriting to enhance the privacy-utility trade-off. dp-gtr is the first approach, to our knowledge, to apply in-context learning for ldp prompt privatization and to connect global and local dp mecha-nisms via grouped paraphrased text. furthermore, our framework is generalizable and compatible with any existing paraphrasing technique. evaluations on open-and closed-answer qa datasets (docvqa and commonsense qa), simulating realworld llm application scenarios, demonstrate that dp-gtr achieves a significantly superior privacyutility trade-off compared to existing state-of-theart methods. with the rapidly increasing adoption of llms, dp-gtr provides a practical, robust and readily deployable solution for mitigating the risk of user prompt privacy leakage.