Balancing Privacy and Utility in Personal LLM Writing Tasks: An Automated Pipeline for Evaluating Anonymizations (2025)

Abstract: Large language models (LLMs) are widely used for personalized tasks involving sensitive information, raising privacy concerns. While anonymization techniques exist, their impact on response quality remains underexplored. This paper introduces a fully automated evaluation framework to assess anonymization strategies in LLM-generated responses. We generate synthetic prompts for three personal tasks—personal introductions, cover letters, and email writing—and apply anonymization techniques that preserve fluency while enabling entity backmapping. We test three anonymization strategies: simple masking, adding context to masked entities, and pseudonymization. Results show minimal response quality loss (roughly 1 point on a 10-point scale) while achieving 97%-99% entity masking. Responses generated with Llama 3.3:70b perform best with simple entity masking, while GPT-4o benefits from contextual cues. This study provides a framework and empirical insights into balancing privacy protection and response quality in LLM applications.

## introduction the intersection of ai governance and data protection has garnered significant attention from academia @xcite , industry, @xcite and regulatory bodies (european data protection supervisor, 2025). as large language models (llms) become widely adopted, concerns regarding privacy risks in user interactions have increased. particularly, the substantial costs of hosting llms, along with restricted access to certain proprietary models, pose significant challenges for individuals and small enterprises seeking to deploy llms locally. as a result, many rely on external llm services, increasing privacy risks @xcite . moreover, llms are frequently used in tasks that involve sensitive personal or corporate information, such as their names, company information, or location information. this raises critical questions about how anonymization strategies impact both privacy protection and response quality in these real-world use cases. existing research has primarily focused on privacy protection from adversarial attacks, such as attribute inference and re-identification risks @xcite . approaches like differential privacy @xcite and prompt obfuscation @xcite have been explored to mitigate these risks. however, these methods often concentrate on preventing external inference attacks rather than evaluating the direct trade-offs between anonymization and response quality in personal tasks. while some studies have examined the utility of anonymized text, they often primarily focus on traditional nlp benchmarks like text classification or summarization @xcite . however, the impact of anonymization on personalized, user-driven tasks, where coherence and contextual relevance are crucial, remains underexplored. moreover, existing anonymization methods can degrade response quality, limiting real-world usability. many privacy-enhancing techniques also rewrite entire user inputs, making it harder to retain original context and provide users with responses that align with their initial prompts. ## methodology our approach presents a fully automated end-toend workflow for evaluating anonymization strategies in llm-based interactions, as depicted in figure 1 . moreover, figure 2 illustrates the different anonymization strategies. the pipeline spans synthetic prompt generation, guardrail-based anonymization, response generation, deanonymization, and evaluation, ensuring a systematic assessment of privacy protection and response quality. to achieve this, we leverage two main categories of models: main llm models (response generation): these models are responsible for generating responses to user prompts. they reflect how proprietary ai systems process user inputs in real-world applications. we experiment with two state-of-the-art llms to evaluate the effects of anonymization on response quality: (i) chatgpt 4o, and (ii) llama 3.3:70b. while llama can be locally deployed, we use it primarily to mimic proprietary ai systems, given its state-of-the-art performance, ensuring a controlled yet representative evaluation of anonymization effects. ## discussion our results demonstrate that anonymization can effectively protect sensitive information while maintaining response quality in personalized llm tasks. across different anonymization strategies, we observe a minimal reduction in response quality (roughly 1 point on a 10-point scale), while achieving 97%-99% entity masking, indicating a strong privacy gain. interestingly, simple anonymization and deanonymization methods (e.g., direct entity masking and backmapping) yield the best results for llamagenerated responses, suggesting that additional context can introduce unnecessary variability. although prompts clarify that contextual information is provided solely as background information, we found that models often overintegrate these details into responses, such as mentioning that the user lives in an east coast city in a cover letter. in contrast, gpt-4o benefits from contextualized anonymization, where entity replacements include descriptive labels. this indicates that some models may better leverage contextual cues to compensate for missing specific entity references. our findings highlight the importance of tailoring anonymization strategies to specific llm architectures and task types, as different models interpret masked entities and contextual information differently. additionally, we show that effective anonymization does not necessarily require complex transformations, as simpler techniques achieve comparable privacy protection with minimal response degradation. this study underscores the feasibility of deploying automated anonymization workflows for real-world, privacy-sensitive llm applications. future work could explore adaptive anonymization techniques, where models dynamically adjust anonymization levels based on task sensitivity and model behavior. ## limitations while our study provides valuable insights into the effects of anonymization on llm-generated responses, several limitations should be considered when interpreting our findings. first, our analysis is limited to chatgpt-4o and llama models, meaning the results may not generalize to other large language models, such as claude, gemini, or mistral, which may process anonymized prompts differently. different llm architectures may exhibit varying sensitivity to entity masking, contextualization, or pseudonymization, potentially leading to different response quality and privacy trade-offs. future work could expand the analysis to a broader range of models to assess generalizability across llm ecosystems. second, while we employ the llm-as-a-judge method to automate response quality evaluation, our study does not incorporate human raters. although recent work suggests that ratings with gpt-4o align well with human preferences, llm-based scoring may not fully capture nuances such as subtle coherence issues, tone, or factual correctness. similarly, our evaluation does not explicitly assess truthfulness or detect hallucinations in de-anonymized responses. for example, a de-anonymized cover letter could introduce fabricated details not present in the original prompt. future research could incorporate human evaluations and factual consistency checks to ensure that anonymization does not introduce unintended distortions or hallucinated content that may not be detected by ai-based scoring. third, our dataset consists of synthetically generated prompts rather than real user queries. while this allows for an automated workflow, realworld user prompts may introduce greater variation, ambiguity, or complexity that could affect both anonymization performance and response generation. in particular, one challenge is anonymizing lesser-known entities, such as small businesses or less prominent organizations, which llm-based techniques may struggle to recognize. since our synthetic prompts are llm-generated, they may overrepresent well-known entities, whereas real-world inputs may include more unique or less widely recognized names that could be more challenging to identify and anonymize effectively. future research could explore realworld anonymization cases to assess how different anonymization strategies perform in practical applications. moreover, while our privacy evaluation effectively quantifies entity masking and assesses re-identification risks using llm inference, it does not fully capture the severity of a single entity leakage. the current approach assumes that privacy loss is proportional to the number of entities disclosed, but in real-world applications, even a single leaked entity (such as a person's name) could constitute a significant privacy risk. this is particularly critical in tasks like cover letters and business emails, where context may allow an adversary to infer personal details even if only one entity is revealed. finally, our study employs a single anonymization approach, using a bert-based ner model for entity recognition. while this approach is effective for structured anonymization, other anonymization techniques exist, including llm-based ner. in addition, recent privacypreserving prompt sanitization techniques, such as casper @xcite , extend beyond ner by incorporating topic-based anonymization and rule-based filters. future research could explore how different anonymization methods interact with various llms, assessing trade-offs between privacy effectiveness and response degradation.