Q uery S hield: A Platform to Mitigate Enterprise Data Leakage in Queries to External LLM s (2025)

Abstract: Unrestricted access to external Large Language Models (LLM) based services like ChatGPT and Gemini can lead to potential data leakages, especially for large enterprises providing products and services to clients that require legal confidentiality guarantees. However, a blanket restriction on such services is not ideal as these LLMs boost employee productivity. Our goal is to build a solution that enables enterprise employees to query such external LLMs, without leaking confidential internal and client information. In this paper, we propose QueryShield - a platform that enterprises can use to interact with external LLMs without leaking data through queries. It detects if a query leaks data, and rephrases it to minimize data leakage while limiting the impact to its semantics. We construct a dataset of 1500 queries and manually annotate them for their sensitivity labels and their low sensitivity rephrased versions. We fine-tune a set of lightweight model candidates using this dataset and evaluate them using multiple metrics including one we propose specific to this problem.

## introduction the rapid advancement of generative ai (gen-ai), especially large language models (llms), has significantly improved productivity across various industries. these models, capable of understanding and generating human-like text, save considerable time in tasks that traditionally required extensive human effort @xcite . this efficiency allows businesses to enhance throughput without sacrificing output quality. ai is emerging as a tool that augments human capabilities, and by integrating ai, businesses can maintain a competitive edge @xcite . companies that adopted ai experienced substantial productivity gains over those who did not @xcite . this disparity has further expanded with the introduction of gen-ai. however, the privacy, security and safety implications of gen-ai demands special investigation. we have seen sensitive details inadvertently surfacing in model outputs since they are trained on gargantuan datasets @xcite . the accurate and coherent performance of llms emerge from their ability to memorize rare training samples, and this poses significant privacy threats when the datasets used to train them contain sensitive data @xcite . the above works, among others discuss the inevitable leakage of private data from an llm. in contrast, there is potential for data leakage to an llm through user queries (or prompts) as humans are the weakest link in security and privacy @xcite . llm service providers may use this interaction data for further model training and this may consequently spill the same sensitive data, that was once sent as a query, when attacked @xcite . this risk is further exacerbated when employees of companies, in attempts to gain competitive edge, leak confidential company data through their prompts to an external llm service such as chat-gpt or google gemini. despite the confidentiality guarantees provided by the llm service providers, there have been unintentional instances where chat data was leaked @xcite . this concern has led some companies to enforce an organizational ban on chat models @xcite . such restrictions severely impact the competitive edge of a company, especially if competent in-house alternatives are not provided. there is an increasing need for a privacy preserving prompting solution that not only safeguards against data leakage, but also ensures that the utility provided by powerful external llms like gpt-4o is not impacted. this is an instance of private inferencing (pi) problem of neural networks @xcite , where inferencing is done on encrypted data. cryptographic methods like fully homomorphic encryption (fhe) @xcite and secure multiparty computation (mpc) @xcite also are employed to solve this problem. however, the communication and computation complexities of the above methods make it unrealistic to perform inference on large language models. moreover, cryptographic methods require implementation in the server-side and the client (prompter) side. execution of server-side code is not entertained by external llm providers like open-ai (chatgpt), rendering such solutions impractical. we propose that client-side input guardrails, that do not impact the usefulness of an external llm, are a necessity to prevent data leakage through queries. a direct solution is data sanitization, where we detect the parts of the text that leak sensitive information @xcite . this approach is limited by the fact that even generic words may leak private information depending on the context in which they are used @xcite . so, we need a method that analyzes the potential for data leakage from a query as a whole. additionally, this analysis should be used to rephrase the query such that data leakage, if any, is minimized, without impacting the semantic integrity of the message that the query aims to convey. this requires a system that can semantically understand the query, while simultaneously understanding the concept of data leakage. in this paper, we propose queryshield, a platform that lies between the enterprise environment and any external llm (figure 1 ). it detects outgoing queries that leak sensitive data and rephrases them to remove the sensitive contents. queries that do not leak sensitive data are allowed to pass through to the external llm. on the other hand, the rephrased versions of high sensitive queries (along with the identified types of leakage ( ## related work private inferencing (pi) refers to the process of drawing predictions from a neural network while keeping the input to the neural network private @xcite . this is traditionally realized using cryptographic methods like mpc @xcite , fhe @xcite , and differential privacy (dp) @xcite . since mpc and fhe have high communication overheads, hybrid approaches that aim to optimize the solution from both an ml and fhe/mpc perspectives were used to advance pi offerings @xcite @xcite . the sheer scale of llms made even such optimizations insufficient to achieve pi in real-time. this shifted the focus to other natural language processing methods. the first of such attempts included the usage of parts of speech tagging @xcite , named entity recognition @xcite and personally identifiable information (pii) detection @xcite . dp based methods add noise into private data to guarantee plausible deniability @xcite . dp is used in llm queries at the word, sentence, and document levels @xcite introduce noise in sentence embeddings @xcite . this captures context based data leakages where words leak data depending on the context in which they are used. @xcite performs word based replacement of the queries and then rectifies the replaced words in the response. most recently, @xcite propose prosan which targets individual words using the context from the entire prompt. our work, in contrast, does a semantic rephrasing of the entire query instead of targeting individual words. ## conclusion and future work to balance between access to external llms and the potential risk of enterprise data leakage, we proposed the queryshield platform. it lies between any external llm and the enterprise environment and detects sensitive data leakage in the queries as well as rephrases the original queries to remove any potential data leakage. we explored multiple lightweight language models as part of queryshield so that they can be hosted in-house with limited hardware resources. we evaluated these models for the tasks of detecting sensitive data leakage, rephrasing sensitive queries, and identifying data leakage types, using a manually annotated dataset of 1500 queries. in future, we would incorporate human feedback once the model is deployed, so that the deployed models can be further fine-tuned periodically. we are also extending the platform to handle data leakage from the context of sequential queries.