Differential Privacy in Natural Language Processing: The Story So Far (2022)

Abstract: As the tide of Big Data continues to influence the landscape of Natural Language Processing (NLP), the utilization of modern NLP methods has grounded itself in this data, in order to tackle a variety of text-based tasks. These methods without a doubt can include private or otherwise personally identifiable information. As such, the question of privacy in NLP has gained fervor in recent years, coinciding with the development of new Privacy- Enhancing Technologies (PETs). Among these PETs, Differential Privacy boasts several desirable qualities in the conversation surrounding data privacy. Naturally, the question becomes whether Differential Privacy is applicable in the largely unstructured realm of NLP. This topic has sparked novel research, which is unified in one basic goal how can one adapt Differential Privacy to NLP methods? This paper aims to summarize the vulnerabilities addressed by Differential Privacy, the current thinking, and above all, the crucial next steps that must be considered.

## introduction in an age where a vast amount of data is being produced daily, the opportunities created by this proliferation increase concurrently. the availability of big data enables countless downstream tasks whose accuracy and utility seem to increase with the amount of data used. specifically, the fields of machine learning (ml) and deep learning (dl) have profited from such data. particularly in the case of natural language processing (nlp), the tasks at hand more often than not concern the handling of unstructured data, meaning data that is not neatly organized into a traditional row-like database structure, and furthermore, data that is not necessarily static. in fact, it is estimated that data on the order of zettabytes (zb) is being produced every day @xcite , and within this amount, roughly 80% is unstructured, e.g. textual data @xcite . at the same time as this profound boom in popularity of big data tasks, there has been an increase in the attention paid to the way in which data is used, specifically to the issue of privacy. the problem is exacerbated when sensitive parts of the data relate to a specific task (e.g. with medical data). the threat becomes more serious when the models themselves used with the learning tasks are vulnerable to attacks. although many useful privacy-enhancing technologies have emerged, one in particular seems to be a good fit when faced with the scale of these big data learning tasks: differential privacy @xcite . the key feature of differential privacy is its mathematically grounded notion of privacy, which can be intuitively explained using the privacy parameter, most often called ϵ. this idea was originally intended for data stored in structured databases, i.e. a relational schema. as a result, differential privacy upon its inception became an excellent way to start to reason about privacy in ml and dl models that were trained on these types of databases. alas, in the field of nlp, where the core unit of data is unstructured, fuzzy text rather than a structured data point, an initial attempt to apply differential privacy poses some challenges. chief among these is the challenge of how to transfer the core concepts of differential privacy, namely the "individual" and adjacency, to the textual domain where these concepts are not easily perceivable. thus, it becomes the goal to find new ways of reasoning about differential privacy in order to adapt it to the unstructured data domain of nlp. through the course of this paper, the foundations of differential privacy in the lens of nlp will be investigated, motivated by some privacy vulnerabilities that surface from nlp techniques. afterwards, the limitations and open questions of differential privacy with nlp will be analyzed with an in-depth discussion. ## methodology to accomplish the goals of this paper, the following research questions have been defined: ## privacy vulnerabilities in nlp techniques by first analyzing some privacy vulnerabilities in nlp techniques (rq1), we hope to motivate the thinking behind the incorporation of differential privacy in nlp, presented in sections 5 and 6. here, we differentiate between two overarching categories of vulnerabilties: (1) information leakage @xcite ) and (2) unintended memorization @xcite . the focus is placed on the former, as this is more relevant to nlp, while the latter pertains more generally to the dl applications. ## differential privacy in nlp with the privacy issues that can arise when performing nlp tasks in mind, it is a logical step to consider the application of differential privacy to mitigate these privacy issues. before one can consider how to do this, it may be be useful to understand what exactly differential privacy can protect against in the context of nlp. ## metric differential privacy for nlp the idea of d x -privacy @xcite , also d-privacy or metric differential privacy, was first introduced in 2013 as a generalization of differential privacy, with the goal of extending the concept beyond structured databases to arbitrary domains (e.g. location data). the key for achieving this comes with the reasoning about adjacency between two databases. in domains without an immediate notion of adjacency between individuals, it becomes necessary to find an alternate expression. the answer comes with the utilization of a (distance) metric existing within some metric space, whose members are often referred to as points. a relaxed sense of differential privacy thus enables its application to arbitrary domains endowed with a metric, and naturally this fits well with text. ## discussion with the application of differential privacy to the area of nlp also come several challenges. ultimately, these limitations serve as a basis for future work and motivation for further improvements. ## conclusion the investigation into differential privacy's place within the nlp sphere results in many interesting findings and discussions. understanding that there does indeed exist privacy vulnerabilities to nlp techniques, looking to differential privacy for a so-