Invernet: An Inversion Attack Framework to Infer Fine-Tuning Datasets through Word Embeddings (2022)

Abstract: Word embedding aims to learn the dense representation of words and has become a regular input preparation in many NLP tasks. Due to the data and computation intensive nature of learning embeddings from scratch, a more affordable way is to borrow the pretrained embedding available in public and fine-tune the embedding through a domain specific downstream dataset. A privacy concern can arise if a malicious owner of the pretrained embedding gets access to the fine-tuned embedding and tries to infer the critical information from the downstream datasets. In this study, we propose a novel embedding inversion framework called Invernet that materializes the privacy concern by inferring the context distribution in the downstream dataset, which can lead to key information breach. With extensive experimental studies on two real-world news datasets: Antonio Gulli’s News and New York Times, we validate the feasibility of proposed privacy attack and demonstrate the effectiveness of Invernet on inferring downstream datasets based on multiple word embedding methods.

## introduction discriminative representation learning is a critical factor in determining the success of machine learning models. in natural language processing (nlp), many deep neural networks need the input words or documents to be represented in a form of numerical vectors which indicate the semantic distinction among information. different methods of converting vocabulary tokens to representations have been discussed in literature @xcite . for example, an intuitive way of converting categorical vocabulary to numerical vectors is to one-hot encode the tokens into a bitstring or commonly known as a bag of words representation harris @xcite . however, one-hot encoding can easily result in high-dimensional representations and incapability of accommodating unseen information. another way is using distributed representation called word embedding @xcite , which is low-dimensional and has become increasingly popular by capturing complex and subtle semantic difference of input words or documents @xcite . in practice, well trained word embeddings demand large-scale training corpus, sufficient training iterations, and high computational capacity. for example, @xcite shows that training a bert-base model emits around 1,500 lbs of carbon dioxide and costs around @xmath012,000. ensuring such environmentally and financially costly training factors are not economical for most small organizations and individuals. as a result, large organizations (e.g., google cloud, hugging face) with computational resource would train word embeddings on large and generic datasets, and offer the pretrained embeddings to public through platforms like tensorflow model hub or hugging face model repository @xcite . with the pretrained embedding in hand, public users can further refine the embeddings by training on their own datasets. as a result, domain adapted downstream embeddings can be generated with much smaller corpus and cost. such refinement process is known as fine-tuning and can be considered as the de facto standard practice of how word representations are commonly learned in today's natural language processing tasks @xcite , such as the biomedical domain @xcite and the financial domain @xcite , etc. often times, users of pretrained word embeddings think that the embeddings are simply dense vector representations that nothing much can be inferred about the training corpus from a collection of numbers. so, most of the times, they do not exercise caution in keeping their downstream word embeddings secure. for this widespread case, we attempt to address a question: is it safe to release fine-tuned embeddings if the downstream dataset is sensitive? specifically, by assuming that the fine-tune embeddings inherit certain optimization pattern from pretrained embeddings, the inference of a downstream dataset can be made inversely from the fine-tuning embedding if we know the examples of how the pretrained embeddings are derived at the beginning. therefore, in this work, we consider a particular attack model: when the malicious party knows the pretrained datasets (e.g., pretrained embedding publishers), and gets the access to fine-tuned embeddings, is it feasible to infer the semantic information of downstream corpora such as word co-occurrence statistics? attacks on machine learning models have been studied extensively @xcite . in @xcite , the authors train adversarial shadow models to carry out membership inference attacks against different commercial models. feature-based analyses of information leakage in machine learning models are discussed in @xcite , and @xcite . in @xcite , the authors consider the owner of the downstream dataset as the attacker and perform a feature alignment based attack on the source dataset in a deep transfer learning setting. in @xcite , the authors try to infer the constituent words of a sentence given that they have access to sentence embedding. however, our proposed research problem is unique in that we consider the owner of the source dataset to start the attack on the downstream dataset. with the advantage of having inference datasets similarly distributed as source datasets, the source dataset owner can simulate fine-tune training multiple times and study the fine-tuning results based on a particular input. therefore, the attackers can inversely infer how the downstream dataset looks like by comparing the resulting embedding with pre-trained embedding. furthermore, the attack strategy makes the inference model-agnostic, thereby rendering our proposed problem and method generic across various embedding models, such as word2vec and glove. specifically, we develop an inversion framework invernet for attacking downstream dataset based on leaked fine-tuning embeddings. the primary components of invernet architecture are a focused document sampling scheme, and a deep-learningbased inference model. the goal of the focused sampler is to ensure that the selected samples for inference will have reduced variance. then the inference model is designed as an ensemble classification problem where multi-layer perception models try to learn the inverse relationship between the fine-tuning embeddings and the context information in the downstream dataset from different inference samples. in the experimental study, we validate the performance of the proposed framework in terms of how much context information of the downstream dataset is predicted, and we illustrate the utility of context prediction through the experiments on membership attack. based on the results on two real-world datasets and three embedding models, our work validates the privacy concern of publishing fine-tuned embeddings, and demonstrates the effectiveness of proposed inference framework. the contributions of our work are as follows: ## attack model and problem statement as shown in fig. 1 , alice is a malicious user (attacker) who has the computational power to train complex embedding models from a large dataset d alice and publishes her pre-trained word embeddings φ alice . bob, the victim, downloads and finetunes φ alice on his smaller dataset d bob . v alice and v bob are sets of all the unique words from d alice and d bob respectively, excluding the stopwords. the formal assumptions are as follows: alice's attack objective is to obtain the private semantic information in d bob . given the fine-tuned embeddings from bob, we formulate the task as predicting the word-to-word co-occurrence statistics among bob's vocabulary, considering that cooccurrence statistics can truly reflect the semantic information of a dataset. therefore, the question we are asking is: "can alice learn the binary word-to-word co-occurrence c bob from bob's finetuned word embeddings φ bob with the knowledge of d alice , c alice , and φ alice ?" formally, we define our question as a learning problem where alice learns a set of mapping functions f, such that ## conclusion we focus on the situation when a malicious participant has access to a pretraining dataset, pretrained word embeddings and also downstream word embeddings that are fine-tuned on a private downstream dataset. access to the pretraining components can be through ownership rights or from the public domain. access to the downstream word embeddings can be through public domain or malicious means. we have carried out extensive experiments and demonstrated that a significant amount of context distribution from the downstream dataset can be inferred using invernet framework. in light of our findings, we wish to raise awareness about taking care when storing or sharing fine-tuned word embedding models.