Transferable Embedding Inversion Attack: Uncovering Privacy Risks in Text Embeddings without Model Queries (2024)

Abstract: This study investigates the privacy risks associated with text embeddings, focusing on the scenario where attackers cannot access the original embedding model. Contrary to previous research requiring direct model access, we explore a more realistic threat model by developing a transfer attack method. This approach uses a surrogate model to mimic the victim model’s behavior, allowing the attacker to infer sensitive information from text embeddings without direct access. Our experiments across various embedding models and a clinical dataset demonstrate that our transfer attack significantly outperforms traditional methods, revealing the potential privacy vulnerabilities in embedding technologies and emphasizing the need for enhanced security measures.

## introduction text embeddings serve as universal representations of textual data, which can be utilized as features for various downstream tasks. recent developments in text embedding models @xcite have significantly streamlined the process of generating embeddings. additionally, systems that employ large language models (llms) often incorporate a vector database of text embeddings to store and infuse domain specific knowledge or auxiliary data. retrieval-augmented generation (rag) @xcite is a typical example that enhances llms' knowledge by incorporating retrieved documents into the model's prompt. this has led to a growing adoption of vector database services like chroma 1 and faiss @xcite , known for their efficient and scalable embedding searches. in these databases, only the text embeddings are shared with third-party services, not the actual text, leading these platforms to claim that storing embeddings is safe and encouraging the upload of private data. although existing works @xcite @xcite have studied the privacy risks of text embeddings, the observed threats essentially rely on a strong assumption, which is that the adversary has query access to the embedding model. by querying the embedding model extensively, the adversary can either iteratively edit the input text such that the text is as close as possible to a given embedding or obtain a large amount of paired data to reverseengineer the embedding model accordingly. here, we argue that such privileged knowledge might not always be available in real-world scenarios. for instance, consider the data leakage of an online vector database where only a small number of documents and their associated text embeddings were exposed to the adversary. in that case, the adversary was passively offered a small number of query pairs, while querying the embedding model is not allowed. inspired by this, this work particularly focuses on the privacy risk without assuming the accessibility of the original embedding model for querying; instead, only a small portion of paired document-embedding data is available. we consider the problem of a black-box attack, where the target victim model is entirely hidden from the attacker. in this setting, standard whitebox attacks @xcite or even querybased black-box attacks @xcite become ineffective. as the victim model becomes invisible to the adversary, we present an alternative solution to attack the victim model through a transfer attack. the transferability property of an attack is satisfied when an attack developed for a particular machine learning model (i.e., a surrogate model) is also effective against the target model. specifically, our transfer attack aims to achieve two goals: 1) encoder stealing attempts to learn a surrogate model to steal the victim model only through their returned representations. if the surrogate model successfully replicates the victim model, the adversary gains query access to some extent. 2) threat model transferability enables the adversary to build a threat model by attacking the surrogate model and hopes the threat model can also successfully fool the victim black-box model. 2 to achieve the first goal, an off-the-shelf text embedding model (e.g., gtr-t5 @xcite ) followed by a mlp-based adapter is used as the surrogate model. the surrogate model is then optimized with our proposed consistency regularization loss to mimic the behavior of the victim model. to achieve the second goal, we use the adversarial training to mitigate the embedding discrepancy between the surrogate and victim models and thus improve the attack transferability. to validate the effectiveness of our attack, we perform extensive experiments on 3 popular embedding models, including sentence-bert @xcite , sentence-t5 @xcite , and openai text embedding. experimental results show that the transfer attack can be 40%-50% more effective than the standard attack approach. the key factors for stealing the victim model are discussed in sec. 6. to study the privacy risk on a specific threat domain, we conduct a case study on the mimic-iii clinical note dataset. results demonstrated in sec. 7 show that our transfer attack can identify sensitive attributes (e.g., age, sex, disease) with 80%-99% accuracy. ## methodology as illustrated in figure 1 , our transfer attack pipeline consists of three major components. first, the encoder stealing aims to learn a surrogate model to mimic the behavior of the victim model ϕ. this goal is achieved by optimizing the surrogate model with our intra-and inter-consistency regularization losses. second, we adopt adversarial training to make the surrogate embedding indistinguishable from the private embedding and improve attack transferability. finally, embedding-to-text leverages a gpt-based decoder to invert embeddings to their original text sequence. ## experiment setup victim embedding models. to assess the embedding inversion attack, we utilize three victim models acting as our blackbox encoders: text-embeddings-ada-002 from openai, sbert @xcite , and st5 @xcite . these encoder models remain frozen, with their preevaluation metrics. we use the following four metrics to evaluate the text reconstruction attack performance. rougel @xcite ) is used to measure the accuracy and overlap between ground truth and reconstructed text based on n-grams. perplexity @xcite is used to evaluate the performance of language models by measuring how well they predict a given sequence of words. embedding similarity (cos): to evaluate the semantic similarity in latent space, we utilize sentence-bert @xcite to compute the cosine similarity between the ground truth sentences' embedding and the embedding of the generated sentences. llm-eval (lin and chen, 2023): we use chatgpt to provide a score ranging from 0 to 1 to evaluate the relevance between prediction and ground truth. more details can be found in appendix f. ## conclusion in this work, we study the privacy risks associated with text embeddings, especially under constraints where attackers lack direct query access to the embedding models. through the development of a transfer attack method, we demonstrated the feasibility of inferring sensitive information from embeddings without needing to interact with the origi-nal model. our extensive experiments across various embedding models and a detailed case study on a clinical dataset underline the effectiveness of our approach. as the use of text embeddings continues to grow in a wide range of applications, our work serves as a crucial step toward understanding and mitigating potential privacy threats.