PAPILLON : Privacy Preservation from I nternet-based and Local Language Model Ensembles (2025)

Abstract: Users can divulge sensitive information to proprietary LLM providers, raising significant privacy concerns. While open-source models, hosted locally on the user’s machine, alleviate some concerns, models that users can host locally are often less capable than proprietary frontier models. Toward preserving user privacy while retaining the best quality, we propose Privacy-Conscious Delegation, a novel task for chaining API-based and local models. We utilize recent public collections of user-LLM interactions to construct a natural benchmark called PUPA, which contains personally identifiable information (PII). To study potential approaches, we devise PAPILLON, a multi-stage LLM pipeline that uses prompt optimization to address a simpler version of our task. Our best pipeline maintains high response quality for 85.5% of user queries while restricting privacy leakage to only 7.5%. We still leave a large margin to the generation quality of proprietary LLMs for future work.

## introduction large language models (llms) such as chatgpt have started to power applications in increasingly sensitive domains, including therapy @xcite , education @xcite , and healthcare @xcite , causing growing concern about data privacy. existing research has studied privacy in llm training data memorization @xcite . unfortunately, sanitizing training data does not prevent users from disclosing personally identifiable information (pii) at inference time @xcite @xcite within downstream applications. this poses significant risk to users, both as a result of llm fine-tuning on user interactions or from llm 1 butterfly in french. 2 our data and code is available at https://github.com/ columbia-nlp-lab/papillon server-side data breaches. as llms are increasingly deployed at scale, it has become crucial to address these inference-time privacy concerns. to improve privacy at inference time, we could simply redact sensitive information before submitting user requests to llms. indeed, @xcite has demonstrated that text anonymization helps prevent llms from inferring personal information. unfortunately, simple redaction may lower llm response utility. for example, if a user asks chatgpt to draft a job application email based on their resume, redaction hinders chatgpt from leveraging its knowledge about the redacted entities. such concerns are fueling community interest in hosting open-source models on users' local gpus or through gpu services that a user trusts, but models available and efficient enough for this purpose often fall behind the best api-based llms in terms of generation quality @xcite . in this work, we study the tradeoffs that emerge between trusted but weaker models, typically open-source or in-house models, and untrusted but powerful models, typically proprietary and over the network. first, to combine the best of both types of models, we propose the task of privacy-conscious delegation. privacy-conscious delegation promotes a paradigm where a locally hosted open-source model serves as a privacy-conscious proxy of the user and queries the api-based llm on the user's behalf. in other words, privacy-conscious delegation requires trusted, weaker models to use untrusted, powerful models as tools to fulfill private user requests. instead of a differential privacy perspective on inference-time privacy @xcite @xcite , we seek to prevent untrusted llms from accessing private information to begin with. our task is also positioned to support the paradigm shift toward deploying smaller llms on edge devices and can have many variants like small llms solving highly complex tasks efficiently by delegating a few, selected steps to larger llms. second, to evaluate the ability of llm systems to conduct privacy-conscious delegation, we construct a new dataset, namely, private user prompt annotations (pupa). pupa consists of 901 instances of real-world user-agent interactions sampled from the publicly-available wild-chat dataset @xcite . to do this, we introduce a framework for mining data from userassistant dialogues for our task. we further define metrics for privacy-conscious delegation, measuring (i) how well systems preserve response quality in comparison to using the best-in-class api models and (ii) how much private information systems leak to the api model. lastly, to tackle privacy-conscious delegation for the first time, we design the privacy preservation from internet-based and local language model ensembles (papillon) system. we utilize prompt optimization with dspy @xcite to identify the best prompts for pa-pillon's multi-stage pipeline and evaluate our instantiations of papillon, backed by common open-source models, on the pupa dataset. the optimized papillon pipeline with llama-3.1-8b-instruct as the local model and gpt-4o-mini as the api-based model performs well in terms of preserving the generation quality of the final responses (85.5% of the time) while leaking little private information (7.5% of the time). overall, our new task, dataset, and system design demonstrate the potential of privacy-conscious delegation systems. such systems can enhance user privacy during inference while giving users access to the capabilities of the best llms. through this task, we outline a vision in which small local llms drive interactions with users while carefully delegating a few selected steps to larger llms, which may be more accurate at the cost of reduced privacy or, in principle, higher cost or latency. ## related work most past llm privacy research has focused on training-time data memorization @xcite @xcite and differential privacy @xcite . however, mitigating llm privacy risks should not only occur during training @xcite , especially when the most popular proprietary llms are costly to train foot_0 and complete training data sanitization may be challenging @xcite . therefore, we must strengthen inference-time privacy of llms. privacy preservation approaches during llm inference have centered around defending against attackers attempting to recover user queries and preserving differential privacy guarantees @xcite . @xcite uses a deep language network @xcite and prompt-tuning @xcite on the client-side to preserve client-side data privacy in sentiment classification tasks. text sanitization approaches such as feyisetan et al. (2020) and xu et al. ( 2020 ) allow text-to-text privatization, often assuming access to lm word embeddings, which is not practical for our use case. here, we do not focus on guarding against an attacker but instead emphasize the user. our approach of chaining a local proxy model with llm calls is also sufficiently lightweight, without additional training requirements, and we aim to adapt to various tasks. existing benchmarks in the inference-time privacy space often focus on evaluating leakage of either memorized training data @xcite @xcite or private information disclosed during the interactions @xcite . @xcite proposes privacylens, the first benchmark to quantify an llm's privacy norm awareness. while this benchmark includes user-agent interactions, they are synthetic and might not accu-rately reflect natural user-assistant exchanges. @xcite discusses different contexts of user disclosures in the wildchat corpus @xcite ; they discover a surprisingly high disclosure rate in unexpected contexts, and find that private information may be revealed implicitly. the authors call for designs to encourage users to moderate their interactions. our work serves as a first step in this direction. instead of preventing llms from leaking information, we are interested in helping users avoid revealing pii to these models in the first place, while still allowing users to benefit from their utility. ## private user prompt annotations (pupa) benchmark we need a dataset containing naturally occurring instances of people leaking pii to large language models. for data privacy, it would be ideal that the queries in our benchmark have already been exposed to common api-based proprietary models. @xcite introduces a set of annotations containing user-assistant dialogues with personally identifiable information from a subset of the wildchat corpus @xcite , which is a collection of one million dialogues between users and chatgpt or gpt-4. as these categories only comprise a small percentage of the original annotations, we expand the coverage of conversations from wildchat using the original prompts. to maximize annotation quality, we compare the accuracies of gpt-4-turbo (the same as the original paper), gpt-4o-mini, and gpt-3.5-turbo on the existing annotations. we present our results in ## papillon as a first step in exploring methods for the privacy-conscious delegation task, we propose papil-lon (figure 1 ), a pipeline structure that addresses the simplest task setting: both m local and m remote are used for output production. ## conclusion we introduce privacy-conscious delegation, a task that provides new insights into preserving inference-time user privacy when interacting with large language models. we construct pupa, a dataset containing real-life examples of user queries with personal information. we additionally define robust metrics for our task to measure generation quality. as an initial concrete approach, we design the papillon pipeline, an ensemblebased approach with prompt optimization, and experiment with different local and cloud-based models. our evaluations show that the papillon pipeline with llama-3.1-8b-instruct as the local model and gpt-4o-mini as the proprietary model produces the highest-quality responses with a low leakage rate. despite the promising performance of this pipeline, the performance gap between pa-pillon and the proprietary models is still not closed. for future work, we aim to explore training approaches and pipeline structures to improve pa-pillon performance. specifically, we would like to develop small, privacy-conscious local models. another future direction we are interested in is offering theoretical guarantees for our approach, such as from a differential privacy perspective, which is crucial for any large-scale deployment. in conclusion, our work presents a new perspective for preserving user privacy in the era of large language models by preventing users' sensitive data leakage to proprietary models in the first place. the data extraction pipeline for pupa allows the incorporation of new user-assistant conversations. our proposed task, benchmark, and approach lay the foundation for further privacy research in nlp.