Extracted BERT Model Leaks More Information than You Think! (2022)

Abstract: The collection and availability of big data, combined with advances in pre-trained models (e.g. BERT), have revolutionized the predictive performance of natural language processing tasks. This allows corporations to provide machine learning as a service (MLaaS) by encapsulating fine-tuned BERT-based models as APIs. Due to significant commercial interest, there has been a surge of attempts to steal remote services via model extraction. Although previous works have made progress in defending against model extraction attacks, there has been little discussion on their performance in preventing privacy leakage. This work bridges this gap by launching an attribute inference attack against the extracted BERT model. Our extensive experiments reveal that model extraction can cause severe privacy leakage even when victim models are facilitated with state-of-the-art defensive strategies.

## introduction the emergence of pre-trained language models (plms) has revolutionized the natural language processing (nlp) research, leading to state-of-the-art (sota) performance on a wide range of tasks @xcite . this breakthrough has enabled commercial companies to deploy machine learning models as black-box apis on their cloud platforms to serve millions of users, such as google prediction api 1 , microsoft azure machine learning foot_0 , and amazon machine learning foot_1 . however, recent works have shown that existing nlp apis are vulnerable to model extraction attack (mea), which can reconstruct a copy of the remote nlp model based on the carefully-designed queries and outputs of the target api @xcite , causing the financial losses of the target api. prior to our work, researchers have investigated the hazards of model extraction under various settings, including stealing commercial apis @xcite , ensemble model extraction @xcite , and adversarial examples transfer @xcite . previous works have indicated that an adversary can leverage the extracted model to conduct adversarial example transfer, such that these examples can corrupt the predictions of the victim model @xcite . given the success of mea and adversarial example transfer, we conjecture that the predictions from a victim model could reveal its private information unconsciously, as victim models can memorize side information in addition to the task-related message @xcite @xcite . thus, we are interested in examining whether the victim model can leak the private information of its data to the extracted model, which has received little attention in previous research. in addition, a list of defenses against mea has been devised @xcite @xcite . although these technologies can alleviate the effects of mea, it is unknown whether such defenses can prevent private information leakage, e.g., gender, age, identity. to study the privacy leakage from mea, we first leverage mea to obtain a white-box extracted model. then, we demonstrate that from the extracted model, it is possible to infer sensitive attributes of the data used by the victim model. to the best of our knowledge, this is the first attempt that investigates privacy leakage from the extracted model. moreover, we demonstrate that the privacy leakage is resilient to advanced defense strategies even though the task utility of the extracted model is significantly diminished, which could motivate further investigation on defense technology in mea. 4 2 related work mea aims to steal an intellectual model from cloud services @xcite @xcite . it has been studied both empirically and theoretically, on simple classification tasks @xcite , vision tasks @xcite , and nlp tasks @xcite . mea targets at imitating the functionality of a black-box victim model @xcite , i.e., a model replicating the performance of the victim model. furthermore, the extracted model could be used as a reconnaissance step to facilitate later attacks @xcite . for instance, the adversary could construct transferrable adversarial examples over the extracted model to corrupt the predictions of the victim model @xcite . prior works @xcite have shown malicious users can infer confidential attributes based on the interaction with a trained model. however, to the best of our knowledge, none of the previous works investigate whether the extracted model can facilitate privacy leakage of the data used by the black-box victim model. in conjunction with mea, a list of avenues has been proposed to defend against mea. these approaches focus on the perturbation of the posterior prediction. @xcite suggested revealing the top-k posterior probabilities only. @xcite demonstrated that api owners could increase the difficulty of mea by softening the posterior probabilities and imposing a random noise on the non-argmax probabilities. @xcite introduced an adversarial training process to discourage the knowledge distillation from the victim model to the extracted model. however, these approaches are specific to model extraction, which are not effective to defend against attribute inference attack, as shown in section 5. ## attacking bert-based api we first describe the process of mea. then we detail the proposed attack: attribute inference attack (aia). throughout this paper, we mainly focus on the bert-based api as the victim model, which is widely used in commercial black-box apis. ## defense although we primarily focus on the privacy vulnerability of bert-based apis in this work, we also test four representative defenses: i): softening predictions: using τ on softmax layer to scale probability vector @xcite . ii): prediction perturbation: adding gaussian noises with a variance of σ to the probability vector @xcite . iii). reverse sigmoid: softening the posterior probabilities and injecting a random noise on the non-argmax probabilities @xcite . iv). nasty teacher: using an adversarial loss to discourage the knowledge distillation from the victim model to the extracted model @xcite . we also propose a new defense called most least, in which we set the predicted probabilities of the most and least likely categories to 0.5+ and 0.5-, and zero out others. could be set as small as possible. for defense experiment, we set to 10 -5 . according to regarding aia, although mostleast manages to defend against mea, it still falls short of preventing privacy leakage from the extracted model (c.f., ## conclusions this work reveals that the hazards of the extracted model have been underestimated. in addition to the violation of ip, it can vastly exacerbate the privacy leakage even under challenging scenarios (e.g., limited query budget; queries from distributions that are different from that of the training data used by the victim apis). such a vulnerability cannot be alleviated by the strong defensive strategies against model extraction. we hope our work can raise the alarm for more investigations to the vulnerability of model extraction attack.