Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence (2023)

Abstract: Sentence-level representations are beneficial for various natural language processing tasks. It is commonly believed that vector representations can capture rich linguistic properties. Currently, large language models (LMs) achieve state-of-the-art performance on sentence embedding. However, some recent works suggest that vector representations from LMs can cause information leakage. In this work, we further investigate the information leakage issue and propose a generative embedding inversion attack (GEIA) that aims to reconstruct input sequences based only on their sentence embeddings. Given the black-box access to a language model, we treat sentence embeddings as initial tokens’ representations and train or fine-tune a powerful decoder model to decode the whole sequences directly. We conduct extensive experiments to demonstrate that our generative inversion attack outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as the original inputs.

## introduction sentence embeddings serve as "universal embeddings" that have been widely used for numerous natural language processing tasks, e.g., text classification, question-answering, semantic retrieval, and other semantic similarity tasks @xcite . recently, embedding models exploit large pre-trained language models to achieve revolutionary performance @xcite . the notable improvement allows sentence embeddings to be directly used as inputs for downstream tasks. however, when applying sentence embeddings in downstream tasks, unintended data disclosure may violate the legislation, result in fines and hinder individuals from contributing their data to service models. for example, in some cases, such as legal document search, when submitting a query embedding to a service based on neural models, we may not want to leak the sensitive information in the query embedding. language models have already been proven to memorize training data and some private training data can be extracted @xcite @xcite . besides the memorization issue, representations learned from language models also inherently leak sensitive information and suffer from attribute inference attacks @xcite . notably, for sentence embeddings, some of the words in the original sentence can be recovered, which is called embedding inversion attacks @xcite . as shown in figure 1 , both attribute inference attacks and embedding inversion attacks take sentence representations from language models as inputs. for attribute inference attacks, the adversary builds a multi-layer perceptron (mlp) to infer the input sentences' private attributes (e.g., gender, race, and other identifiable personal information). for embedding inversion attacks, existing approaches are viewed as classification tasks that aim to recover partial keywords or unordered sets of original words from input sequences. however, attribute inference attacks and existing embedding inversion attacks are not enough to explore the information leakage of sentence embeddings. if the malicious adversary can recover original sentences from their embeddings, attribute inference attacks and previous embedding inversion attacks can be conducted subsequently. attribute inference attacks can be performed over the recovered sentences, which gives more flexibility for attack classifiers. moreover, existing embedding inversion attacks can be done by directly converting the recovered sentences to sets (bag-of-words). lastly, recovering the input sentences reveals more semantic information beyond the two attacks. in this paper, we propose a generative embedding inversion attack (geia) to reconstruct input sentences given their embeddings from various language models. unlike previous embedding inversion attacks, our attack is able to generate ordered sequences that share high contextual similarities with actual input sentences. our proposed attack advances preceding embedding inversion attacks from classification to generation. moreover, the generated sequences are mostly coherent and some are even verbatim text sequences from inputs. finally, our attack is adaptive and effective to various lm-based sentence embedding models regardless of the models' architectures and training methods. we perform extensive experiments to demonstrate the effectiveness of our attacks on sentence-bert @xcite , simcse-bert/simcse-roberta @xcite , sentence-t5 @xcite and mpnet @xcite . we also conduct experiments to show that our geia can even outperform previous attacks on classification metrics. our contributions can be summarized as follows: foot_0 (1) to the best of our knowledge, we are the first to treat the embedding inversion attack as a generation task rather than a classification task. this allows our attack to reconstruct ordered sequences. privacy leakage on language models. even though language models bring dramatic improvements to sentence representations, there are rising privacy concerns on language models. @xcite showed that language models tended to memorize training data and performed training data extraction attacks to recover private training data. @xcite studied deep gradient leakage @xcite in language models and extracted training texts from aggregated gradients. besides the memorization issue, sentence embeddings from language models also encode private information that can easily be inferred by the adversary. pan et al. (2020) recovered partial fixed patterns and keywords from language models' sentence representations by querying language models with external annotated datasets. similarly, song and raghunathan (2020) performed attribute inference attacks input representation any sentence embedding model: simcse-bert/sroberta/st5 … … … … … … what are you in ? sentence tokens grade [eos] … … what are you in ? sentence tokens grade [cls] [eos] sentence embedding what are you in ? grade [eos] [eos] attacker model: gpt-2 input representation projection align & concatenate and embedding inversion attacks to predict unordered sets of words from sentence embeddings. though their attack setups are similar to our work, they simply view embedding inversion attacks as classification problems and cannot reconstruct input sequences. in this work, we reformulate the embedding inversion attack as a generation task and aim to invert ordered and informative sentences. moreover, our proposed attack outperforms previous attacks even on classification metrics. ## conclusion in this paper, we show the limitations of existing inversion attacks and propose a generative embedding inversion attack to better recover original text sequences given their sentence embeddings. then we show that lm-based sentence embedding models are potentially vulnerable to our proposed attack. we conduct extensive experiments to demonstrate the inability of previous embedding inversion attacks and disclose the privacy risks from our attack. the defenses against the embedding inversion attack are not well studied yet, even though it is much more malicious than the attribute inference attack. for future work, we call for more attention to effective defenses to address the embedding inversion attack with minor costs.