M eae Q : Mount Model Extraction Attacks with Efficient Queries (2023)

Abstract: We study model extraction attacks in natural language processing (NLP) where attackers aim to steal victim models by repeatedly querying the open Application Programming Interfaces (APIs). Recent works focus on limited-query budget settings and adopt random sampling or active learning-based sampling strategies on publicly available, unannotated data sources. However, these methods often result in selected queries that lack task relevance and data diversity, leading to limited success in achieving satisfactory results with low query costs. In this paper, we propose MeaeQ (Model extraction attack with efficient Queries), a straightforward yet effective method to address these issues. Specifically, we initially utilize a zero-shot sequence inference classifier, combined with API service information, to filter task-relevant data from a public text corpus instead of a problem domain-specific dataset. Furthermore, we employ a clustering-based data reduction technique to obtain representative data as queries for the attack. Extensive experiments conducted on four benchmark datasets demonstrate that MeaeQ achieves higher functional similarity to the victim model than baselines while requiring fewer queries.

## introduction the adoption of machine learning as a service (mlaas) via apis has introduced a new security challenge known as model extraction attacks @xcite . attackers repeatedly access the api to acquire outputs by utilizing meticulously crafted inputs (queries) and subsequently train a local model based on the gathered input-output pairs. this attack aims to obtain an extracted model that closely approximates the performance of the victim model, thereby posing a substantial threat to the intellectual property of the model owner. in black-box scenarios, wherein the architecture and training data of the victim model remain unknown, the primary concern for attackers is to consider how to design high-quality queries with limited query budgets. this is crucial since frequent api calls not only result in considerable costs @xcite but also carries the potential of triggering the victim model's defense mechanisms @xcite @xcite , thereby diminishing the effectiveness of the attack. some studies @xcite @xcite sample queries from annotated data associated with the victim model training data, e.g., using bbc news data to extract models trained on agnews @xcite , which deviates from the assumption of black-box model extraction, where the training data distribution of the victim model is unknown. therefore, recent research has focused on leveraging publicly available unannotated data sources for model extraction attacks. the pioneering work by @xcite explores the utilization of these data sources and proposes an active learning-based sampling strategy that dynamically adjusts query selection based on self-feedback. likewise, @xcite presents two query construction methods, one involving sentences combined with random words and the other involving actual sentences randomly sampled from wikipedia. despite the demonstrated efficacy of these methods in their individual studies, we experimentally find that the queries sampled by these approaches often suffer from category imbalance. for instance, when applying the random strategy to an online hate speech detection api, the ratio of positive to negative samples becomes skewed as high as 30 : 1. such a notable imbalance in sample distribution presents challenges for model training, especially when considering low query costs. we attribute this issue to task-irrelevant text content and insufficient data diversity within the queries. in this paper, we propose meaeq (model extraction attack with efficient queries), a straightforward yet effective method to address these issues. meaeq comprises two modules: task relevance filter (trf) and data reduction based on clustering (drc). the trf module aims to select data that is highly relevant to the task. to achieve this, we utilize a pre-trained zero-shot sequence inference classifier, in combination with api service information, to infer the entailment relations between the pre-designed prompt and all actual sentences extracted from a publicly available text corpus. the second module, drc, is designed to mitigate information redundancy within the query pool filtered by the trf module. to accomplish this, drc initially extracts embeddings from all texts in the query pool and then employs a clustering method to create multiple clusters. it subsequently selects the data nearest to the centroid of each cluster as the ultimate query. finally, we send these queries to the victim model and then use the outputs as labels to fine-tune our local model. extensive experiments on simulated victim models demonstrate that the model extracted by meaeq exhibits higher functional similarity to the victim model than the baseline methods on four benchmark datasets. furthermore, we validate the generalizability of meaeq across diverse model architectures. in-depth analyses reveal the significant contributions of both the trf and drc in the model extraction attack. the primary contributions of this paper can be summarized as follows: ## related work we introduce related work from four perspectives of model extraction attacks, including the type of victim model, the type of api feedback, the query source, and the strategy of query sampling. ## methodology in this section, we present our method for nlp model extraction attacks. we first formalize the problem and then elucidate how to sample queries with high task relevance and low information redundancy. finally, the attacker samples the queries from q r , sends them to the api, and then uses the outputs as labels to fine-tune their own model such as bert @xcite . ## experiments in this section, we conduct extensive experiments to evaluate the effectiveness of our method. ## analysis in this section, we provide further analysis of the trf module and drc module in our proposed method for model extraction attacks. ## conclusion in this paper, we propose a straightforward yet effective model extraction attack method meaeq. in particular, we initially utilize a zero-shot sequence inference classifier in combination with the api service information to filter data with high task relevance from a public unannotated text corpus. subsequently, we employ a clustering-based data reduction technique to obtain representative data as queries for the attack. extensive experiments demonstrate that our method achieves superior performance than baselines with limited query budgets on four benchmark datasets.