Privacy- and Utility-Preserving NLP with Anonymized data: A case study of Pseudonymization (2023)

Abstract: This work investigates the effectiveness of different pseudonymization techniques, ranging from rule-based substitutions to using pre-trained Large Language Models (LLMs), on a variety of datasets and models used for two widely used NLP tasks: text classification and summarization. Our work provides crucial insights into the gaps between original and anonymized data (focusing on the pseudonymization technique) and model quality and fosters future research into higher-quality anonymization techniques better to balance the trade-offs between data protection and utility preservation. We make our code, pseudonymized datasets, and downstream models publicly available.

## introduction with the advances in artificial intelligence and data-hungry machine learning systems, privacy and compliant data governance have become increasingly important. text documents, such as emails, court rulings, customer service chats, interview transcripts, and patient records, frequently contain personally identifiable information (pii), such as mentions of persons, locations, organizations, etc. while the collection and use of text data is necessary for improving products or services, conducting research, or providing personalized recommendations, it has to be done in a safe, responsible and compliant way. however, access to text data becomes a challenge where data containing personally identifiable mentions is involved. although it is a widely accepted notion that no data is truly anonymous and is said to be an unattainable target @xcite , pseudonymization, on the other hand, is recognized by the gdpr as one of the ## related work pseudonymization has predominantly been researched in clinical nlp up until recently, focusing on nlp techniques on how to replace pii such as named entities in medical texts, across different languages. for english medical texts, @xcite was one of the first pseudonymization systems, followed by numerous works such as @xcite the techniques proposed in related works range from simply replacing the detected text spans by a placeholders, pseudonyms or synthetic surrogates using lists, lexical substitution such as synonyms or hypernyms, or knowledge bases @xcite . relatedly, techniques such as c-sanitize @xcite , t-plausibility @xcite , and more recently, @xcite have proposed frameworks for privacyaware and -preserving document sanitization and pseudonymization. while numerous recent works such as the aforementioned ones have investigated the topic of pseudonymization, our work comes closest to lampoltshammer et al. (2019); obeid et al. (2019); berg et al. (2020); vakili et al. (2022) and liu et al. (2023) , which focus on analyzing different techniques of data anonymization or pseudonymization, and their effect on downstream tasks. however, our work differs from those since they focus on different domains, different tasks, and different techniques. ## pseudonymization systems the general architecture of a pseudonymization system consists of two steps, where they first recognize entities bearing pii (detection), and the second sub-system their replacement by realistic substitutes (replacement). for this work, we restrict our analysis to three predominant categories of named entities: person (per), location (loc), and organization (org). using this general framework, we describe the three types of systems that are used in our experiments: ## experiments in this section, we experimentally evaluate the considered pseudonymization methods. first, we evaluate the negative impact of pseudonymization on the downstream tasks' quality. next, we compare the privacy preservation quality of different pseudonymization methods. finally, we evaluate the consistency and privacy-preservation characteristics of pseudonymized texts through a text syntheticity detection experiment. ## conclusions we investigate the effectiveness of pseudonymization for nlp research with privacy-sensitive data. we develop three different approaches for this task, and evaluate them from three aspects: downstream task performance (on two downstream tasks: text summarization and text classification), privacy preservation, and text syntheticity detection. we find that the proposed approaches have pros and cons for pseudonymization, so one must chose what task and objective (privacy vs. utility) is the most important for them. ner-based systems with flair perform the best for privacy preservation and downstream task performance, whereas the llm-based system shows the best results for preserving the integrity of the text.