Protecting Privacy in Classifiers by Token Manipulation (2024)

Abstract: Using language models as a remote service entails sending private information to an untrusted provider. In addition, potential eavesdroppers can intercept the messages, thereby exposing the information. In this work, we explore the prospects of avoiding such data exposure at the level of text manipulation. We focus on text classification models, examining various token mapping and contextualized manipulation functions in order to see whether classifier accuracy may be maintained while keeping the original text unrecoverable. We find that although some token mapping functions are easy and straightforward to implement, they heavily influence performance on the downstream task, and via a sophisticated attacker can be reconstructed. In comparison, the contextualized manipulation provides an improvement in performance.

## introduction large language models (llms) have greatly advanced the field of nlp in recent years, exhibiting exceptional proficiency across a wide spectrum of tasks, including dependency parsing @xcite , natural language understanding @xcite , automatic question-answering @xcite , machine translation @xcite ), text classification @xcite , and many more @xcite . however, this success comes with potential privacy risks, as the models process vast amounts of data that might contain personal or sensitive information and may abuse or leak it. for instance, information can be leaked by model inversion @xcite , re-identification techniques @xcite , exploitation of feature memorization within the llm @xcite , and more. offering llms as cloud services, such as chatgpt @xcite , might also impose potential threats to privacy if the server exhibits a semi-honest stance, actively seeking to glean more insights from the input than is appropriate or by a possible eavesdropper intercepting the input sent to the server. in order to safeguard privacy, many privacypreserving techniques have been proposed, based on the local differential privacy framework (ldp; @xcite . in this framework, the user applies a differential privacy mechanism, which can be hosted on a local server, and then sends the privatized data to the remote server. this approach doesn't require trust from the remote server, and protects the data against potential eavesdroppers. in general, any privacy mechanism can be applied at one or several components of the llm pipeline. figure 1 depicts these components: at the text level (text privatization), after the tokenization process (token privatization), after the initial embedding lookup (token embedding privatization), or after applying several layers of the encoder (sequence embedding privatization). currently, most privacy-preserving strategies focus on incorporating noise into sequence embedding vectors. the rationale behind this strategy is to minimize the privacy-preserving technique's impact on the downstream task. specifically, most systems first obtain a sequence embedding repre-sentation, either by assuming partial access to the remote model @xcite @xcite or by using a dedicated model to create these embeddings @xcite @xcite @xcite . afterwards, random noise is incorporated into the embeddings, thus concealing the original input. however, this approach relies on partial access to the remote model, on the ability to provide input to the remote model in vector form, or on sufficient computational and memory resources on the user's end. these are often not the case. in addition, @xcite showed that publishing a model's encoder along with the contextualized embeddings allows an adversary to generate data to train a decoder with a high level of reconstruction accuracy, making these approaches highly susceptible to violation of privacy. we propose a secure way to use llms without assuming access to their parameters. in our framework, both input and output for the privacyproviding mechanism must be given in a token sequence format, eliminating the need to intervene with the llm's pre-training procedure or text processing. we focus on applying privacy preservation techniques at the token level, corresponding to layer (b) in figure 1 . specifically, we propose two privacy-preserving techniques based on manipulating the input token sequence. the first set of techniques relies on naïve rules of token substitution. the second is based on leveraging contextual information to strategically replace tokens, aiming to retain as much actionable information as possible for the classifier to minimize the impact on the performance of the downstream task. we test these techniques both for their impact on the downstream task accuracy and for their resilience against reconstruction attacks. we find that replacing tokens based on simple rules is easy for a knowledgeable attacker to reverse, while manipulating tokens based on contextual information can enhance privacy without sacrificing much of the performance. foot_0 ## lossy mapping in order to protect against potential eavesdropping by a middle party, under the assumption that the layers of llms are inaccessible to the local device, we start by employing several mapping functions on the tokens of the input text available at the local device. our initial, naïve mapping functions introduce a random noise component that follows a specific rule: the vocabulary is partitioned into pairs of tokens (u, v), or triplets (u, v, z), and when encountered in an input text to be manipulated, all tokens are mapped to a single representative token of their tuple, without loss of generality u. this strategy produces outputs that are inherently ambiguous, blocking any potential eavesdroppers from recovering the original input text deterministically, given that a many-to-one mapping is not invertible. the only available recourse for an attacker is a statistical strategy, which imposes assumptions on the properties of the input, for example that it was grammatical english text written by a speaker with high proficiency. indeed, even if an eavesdropper obtains full information of the privacy system, i.e. the partition into token tuples and each tuple's representative token, each mapped sequence of length m still generates a candidate set of 2 m or 3 m possible permutations (depending on tuple size) through which the attacker must search. we will examine the practical implications of this large search space later in the section. for our stated use case of manipulating text being input into a sequence classifier operating atop an llm, there are two distinct scenarios depending on when we may apply our manipulation. the first scenario involves applying the manipulation process only during the inference phase of a model trained on regular, unmanipulated text, which we will refer to as the test case. this operation mode simulates a query sent by a user to an alreadytrained model, such as a user interacting with chat-gpt or another model allowing only inference text interaction via user interface or an api. in the second scenario, which we call all, we also apply the manipulation during the training phase, protecting sensitive information in the training data, hoping that the inference phase will now leverage the model's ability to handle manipulated input as expected and produce better results. in this scenario the model does not inadvertently learn or memorize the sensitive data during the training process, nor does it spend learning resources on tokens never to be seen during inference, but since it is not always possible to assume its availability, we perform our experiments in both settings. when protecting the original input data, it is essential for the mapper to have minimal impact on the performance of the downstream task, defining the fundamental trade-off in our study. therefore, the selection process for grouping tokens and selecting each tuple's representative token is crucial, as it aims to both minimize the mapping's effect on the downstream task and hinder the attacker's ability to uncover the original text. we consider the following mapping functions: purely random mapping the selection of the token pairs tuples from the vocabulary and of each tuple's representative is uniformly random. high-frequency mapping token pairs are selected based on their frequency of occurrence in a tokenized corpus, such as wikipedia @xcite . this involves pairing a higher-frequency token with a lower-frequency token, with the higherfrequency token being designated as the representative. in our mapper, given a vocabulary of even size v , sorted by descending frequency, each token with low-frequency mapping the process is similar to that of the higher-frequency mapper, except that the lower-frequency token is chosen as the representative. opting for less-frequent tokens as representatives can aid in preserving privacy, but it will likely harm the downstream task. due to the simplicity of these mapping strategies, we consider them baselines for further research and developing better, potentially language-aware strategies. in addition, these mapping functions can easily be generalized to larger tuples, expanding the search space even further, but greatly harming downstream task performance as a result of a much more restricted active vocabulary. ## stencil privacy preservation in the context of protecting privacy within nlp practices, a widely adopted approach for implementing local differential privacy involves introducing a controlled level of noise into different components of the model, effectively concealing the original input. these components may include sequence embeddings, token embeddings, or the tokens themselves @xcite @xcite @xcite . however, in essence, the success of models in most nlp tasks is primarily attributed to their effective utilization of contextual information. moreover, our study focuses on token-level privacy preservation, i.e., we assume that the parameters of the llms are inaccessible, making the importance of contextual information more pronounced. therefore, a fundamental limitation associated with incorporating noise is the exclusion of contextual information when defining the noise. this omission may hinder the potential benefits contextual details can offer for maintaining the performance of the downstream tasks. given this limitation, we propose a new privacy preservation technique, which we call stencil. 3 with this technique, a mapped token in a sequence "absorbs" information from adjacent tokens to form a new context-aware token, effectively concealing the original token while retaining information beneficial for maintaining task performance. in order to generate the new contextualized token t @xmath0, we first retrieve an embedding vector representation of the neighborhood, of size n + 1, containing the tokens t i , ∀@xmath1. . . k + n/2} using some embedding lookup table @xmath2, which can be trained independently in a preliminary step or obtained from an available model such as the target model itself. we then subject the n + 1 embedding vector representations to a weighted transformation and incorporate them to form a new "quasi-embedding" vector finally, we return the token t ′ k that is closest to the quasi-embedding vector in the embedding space, based on cosine-similarity or euclidean distance computation, as an output. to further enhance privacy, we ensure that the new token is different from the original one. formally, the process can be defined as follows: the level of privacy enhancement and its impact on the downstream task by employing the stencil method can be managed by adjusting the window size and the properties of the weighted function f . in our study, we use the gaussian smoothing function as the weighted function. consequently, the standard deviation, σ, plays a crucial role in the performance and amount of privacy achieved. as a baseline for our proposed technique, we adopt @xcite 's proposed privacypreserving technique. in contrast to our proposed technique, this approach does not consider context but rather incorporates random noise into token embeddings to enhance privacy. the random noise is obtained by multiplying a sample from a gamma distribution γ(d, 1/η) and a uniform sample from a unit hypersphere, where η corresponds to the amount of noise introduced to the original token and d is the dimension of the embedding space. we note that the most time-intensive operation in both stencil and noise-based techniques involves searching for the closest token to the perturbed quasi-embedding vector, while all other operations are negligible in comparison. overall, the average computational cost per token is 0.005 seconds on two 16-core 3.2 ghz amd epyc 7343 milan processors. ## conclusion in this paper, we propose several token manipulation methods to preserve privacy under the assumption that the model parameters are inaccessible. we first introduce four mappers that offer advantages compared to existing privacy-preserving techniques. these mappers operate independently of the llm and the specific downstream task, resulting in a high degree of versatility. additionally, their computational complexity is relatively low, making them efficient choices for privacy preservation on local, low-resource devices. however, these mappers harm the performance of the downstream tasks and can be easily reconstructed by a knowledgeable attacker. the second mapper class we propose is based on utilizing contextualized information to maintain performance while obfuscating the original input text. this technique achieves higher privacy measures and has less impact on the downstream task, which makes it more applicable for cases where the downstream task is important. nevertheless, opting for different weighted functions, such as ones based on a trained model, can further help improve both accuracy and privacy. an inherent problem with existing privacypreserving techniques is their inability to maintain linguistic properties such as grammar and readability (as seen in lastly, our experiments were limited to classification tasks in the english language. in future research, we intend to explore the effectiveness of these methods in generative tasks, across languages, and in multilingual settings.