An Inversion Attack Against Obfuscated Embedding Matrix in Language Model Inference (2024)

Abstract: With the rapidly-growing deployment of large language model (LLM) inference services, privacy concerns have arisen regarding to the user input data. Recent studies are exploring transforming user inputs to obfuscated embedded vectors, so that the data will not be eavesdropped by service provides. However, in this paper we show that again, without a solid and deliberate security design and analysis, such embedded vector obfuscation failed to protect users’ privacy. We demonstrate the conclusion via conducting a novel inversion attack called Element-wise Differential Nearest Neighbor (EDNN) on the glide-reflection proposed in (CITATION), and the result showed that the original user input text can be 100% recovered from the obfuscated embedded vectors. We further analyze security requirements on embedding obfuscation and present several remedies to our proposed attack.

## introduction inference services of language models are now gaining popularity, with a considerable number of language models having been deployed on the cloud server. however, users might concern about the privacy of their data when requesting inference service, that is, their data would be eavesdropped by malicious service providers.to address this problem, recent research has turned to adopting obfuscation techniques on the embedding matrix, ensuring that user inputs cannot be recovered from the obfuscated embeddings by service providers. embedding obfuscation becomes appealing since the obfuscated embeddings can be directly forwarded to inference process as efficient as plaintext embeddings, leading to practical potential for real applications compared with secure multi-party computation (mpc) and homomorphic encryption (he). for example, the state-of-the-art work in @xcite leverages glide-reflection for embed-ding obfuscation combined with the user-side keybased hashing, to claim a private and secure inference solution. nonetheless, recent studies show that a malicious server can indeed reconstruct user data through embedding inversion attacks (eia) @xcite . consequently, without formal security analysis, concerns persist regarding the potential existence of novel eias capable of extracting user information from these embedding obfuscation methods. in this paper, we analyze the security of the glide-reflection methodology used in @xcite , ultimately uncovering its vulnerability. we innovatively design an element-wise differential nearest neighbor (ednn) attack to effectively break the security of the glide-reflection. our experimental outcomes conclusively demonstrate that the ednn entirely recovers 100% of the user data tokens which ostensibly secured by the glide-reflection. subsequently, we present an insight on why the naive linear-transformation based obfuscation, like glidereflection, fails to safeguard user data. we further discuss the security requirements of embedding obfuscation and demonstrate that the deliberate security design is necessary. we also introduce several possible defenses against eia base on our analysis. ## obfuscation schema based on glide-reflection in this section, we describe the system and threat model of obfuscation schema in @xcite . then we give the formal description of the schema and explain its vulnerability. ## proposed attack: ednn the authors of @xcite have evaluated the security of glide-reflection against the nearest neighbor (nn) attack, and the accuracy of token recovery for which turns out to be negligible. by extending nn, we propose an efficient inversion attack called element-wise differential nearest neighbor (ednn) to break the glidereflection. the ednn selects the closest token from pretrained embeddings as the real token by utilizing the difference of vector elements for neighbor retrieval. therefore, it is effect on the glidereflection which does not change the element differences within embedding vector. algorithm 1 ednn input: a obfuscated and fine-tuned embedding matrix and output vocabulary v @xmath0. 2: for i, j ← 1 to m do 3: we present the details of ednn in algorithm 1, where e d×m is the obfuscated embedding matrix after fine-tuning. the algorithm will output a vocabulary v r which stores the recovered tokens corresponded to the obfuscated embeddings e d×m . to compute the element difference inside embedding vector, the algorithm use lshf t(•) function to cyclically shift the vector to the left by one position and calculate element-wise subtraction. the algorithm will evaluate the distance between every pairs of plaintext and encrypted tokens. then for each encrypted token, the algorithm is able to output the plaintext token with the minimum distance in the embedding space as its substitute. to explain the attack effect of ednn, we fit the embeddings of 100 tokens into 2d plot by t-distributed stochastic neighbor embedding (t-sne) (van der @xcite and scale them into (-1, 1) as shown in figure 1 . by comparing the results of figure 2 and 1b, we can observe that the element-wise differences inner each embedding vector from the original model and the transformed model are the same. ## experiment experimental details. we encrypted the model according to @xcite and fine-tune the model with specific task. then we evaluate ednn on the fine-tuned model to recover its encrypted tokens. datasets and models. we use the same setting as @xcite and conduct experiments on datasets including the general language understanding evaluation (glue) benchmark dataset @xcite , the conll2003 named entity recognition dataset @xcite and the xnli dataset @xcite . we use bert, roberta, and mbert models from huggingface foot_1 . element-wise differential comparison. for each embedding obfuscated by glide-reflection, we first evaluate the distance of element-wise differential to its corresponding original embedding and other nearest embedding. the results in fig. 2 shows that after fine-tuning, the element-wise differentials between each embedding and irrelevant embeddings exhibit a three-order-of-magnitude discrepancy compared to its original embedding, facilitating the ednn to capture the correspondence between obfuscated embeddings and their original counterparts. ## analysis and possible defenses in this section, we analyze the security requirements for the embedding obfuscation and propose security requirements for embedding obfuscation. ## conclusion in this paper, we investigate the vulnerability of the glide-reflection used for embedding obfuscation. we devise an innovate embedding inversion attack to break the security of the glide-reflection. furthermore, we conduct a comprehensive analysis and introduce two essential security requirements for embedding obfuscation. we explore various techniques that can be leveraged to enhance the security of embedding obfuscation.