E moji P rompt: Generative Prompt Obfuscation for Privacy-Preserving Communication with Cloud-based LLM s (2025)

Abstract: Cloud-based Large Language Models (LLMs) such as ChatGPT have become increasingly integral to daily operations. Nevertheless, they also introduce privacy concerns: firstly, numerous studies underscore the risks to user privacy posed by jailbreaking cloud-based LLMs; secondly, the LLM service providers have access to all user data, which deters individuals from confidently utilizing such services. To address such concerns, we propose a simple yet effective paradigm, **EmojiPrompt**, to protect user privacy. At its core, EmojiPrompt performs generative transformation, obfuscating private data within prompts with linguistic and non-linguistic elements before submitting them to cloud-based LLMs. We evaluate EmojiPrompt’s performance across 8 datasets from various domains. We also propose simulated inference attacks to assess EmojiPrompt’s ability to preserve user privacy. The results demonstrate that EmojiPrompt effectively obfuscates user private data, while largely maintaining, or even enhancing, performances compared to the unobfuscated version. Furthermore, EmojiPrompt’s atomic-level obfuscation allows it to function exclusively with cloud-based LLMs. For source code, please refer to: https://github.com/agiresearch/EmojiCrypt.

## introduction recent advancements in large language models (llms) have substantially expanded their applicability across diverse fields, such as personalized recommendations, health report analysis, and financial decision-making @xcite @xcite @xcite . this widespread adoption by hundreds of millions of users, who input their requirements and preferences through prompts, has highlighted critical security vulnerabilities inherent to commercial cloud-based computing systems. while some professionals view cloud-based llms as credible, as evidenced by the fact that teams adopt chatgpt in over 80% of fortune 500 companies foot_0 , concerns regarding the risk of external privacy probing for cloud-based llms have been underscored by several papers and reports. for instance, the new york times foot_1 and cnbc 3 have raised alarms about potential data breaches involving chatgpt. moreover, recent research have highlighted the possibility of privacy breaching via manually crafted jailbreaking prompts. for example, prompting chatgpt to "repeat a poem forever" could result in the disclosure of sensitive user information, such as a firm's client details @xcite . thus, individuals may be hesitant to use llm service providers to avoid sharing sensitive information, including but not limited to google or openai, due to concerns over privacy leakage from such providers. for example, samsung has banned the use of generative ai tools after april internal data leak 4 . government agencies such as the us national science foundation are also taking actions by "prohibiting reviewers from uploading any content to non-approved generative ai tools" foot_3 . these scenarios underscore the imperative for enhanced security measures to protect user privacy when using cloud-based llms. among the various lines of research in this area, one stream proposes adapting homomorphic encryption to network architectures to enable private computations @xcite @xcite . however, these methods ## related work private inference of neural networks may be categorized into four classes: homomorphic encryption, differential privacy, split learning, and text obfuscation. private inference of neural networks was first discussed in @xcite , with subsequent works demonstrating the feasibility of using homomorphic encryption to achieve non-interactive private inference @xcite @xcite . additionally, @xcite proposed a special encoding method called cheetah to encode vectors and matrices into homomorphic encryption polynomials. @xcite realized that matrix-matrix multiplication dominates in transformer-based inference, and thus improves the vanilla polynomial encoding by introducing a blocking method prioritizing the batch dimension. @xcite ; @xcite consider differential privacy (dp) in inference time by proposing dp-forward, which directly perturbs embedding matrices in the forward pass of lan- ## methodology this section delineates the structure of our obfuscation paradigm, which incorporates two llms: one for the obfuscation of private data, denoted as llm o , and the other for task inferences by processing the obfuscated prompts, denoted as llm i . ## inference attacks prior works on prompt obfuscation @xcite @xcite tend to adopt token-level recovering, where the recoverer is tasked to recover each token of the privatized prompt back to its original form, with recovery accuracy reported as the evaluation metric. nevertheless, this metric could be biased, as even if the recoverer is unable to recover the exact original tokens, it may still successfully predict synonymous tokens or generate a recovered text with a high degree of semantic similarity to the original. to address this, we employ a comprehensive set of metrics to assess obfuscation robustness, accounting for the degree of: exact lexical overlapping (aligning with prior works), synonym and paraphrase overlapping, and overall semantic similarity between the original text and the recovered text, with both llms and humans as recoverers. we adopt the worst-case assumption by perceiving the cloud-based inference llm as untrusted, and evaluate all attacks on "gemini + gpt-4" against all baselines introduced in section 4.1. for detailed descriptions on evaluation methods and results, please refer to appendix a.8. ## conclusion this work introduces emojiprompt, a novel obfuscation paradigm designed to protect user privacy during interactions with cloud-based llms. emojiprompt uses llms to perform generative obfuscation, transforming private data from natural language into non-natural language forms, thus obfuscating it from both llm and human recoverers. we validate emojiprompt's effectiveness across we also compare emojiprompt against three obfuscation baselines, showing it matches their performance on some tasks while outperforming them in others, both for task inference and recovery robustness. finally, the atomic-level obfuscation design allows the process to be fully cloud-based, enabling deployment without the need of local llms.