Paper: Beyond Labeling Oracles: What does it mean to steal ML models? (ICLR-2024)

Reviewer comment (Experiments):
The rationale behind the mixed use of original and surrogate datasets requires further clarification.

Author response:
We investigate how essential it is for an attacker to have prior knowledge over the true data distribution. For this goal, we model prior knowledge as having actual access to real training samples. We mix the real training samples with the surrogate data samples, to model how the attack performance is affected by the availability of such prior knowledge.

Reviewer comment (Evaluation):
The authors utilize linear classification as an illustrative example to convey the idea that in model extraction (ME) attacks, adversaries require either prior knowledge about the data distribution or the ability to make a substantial number of queries to the target model. However, in my interpretation, this example primarily underscores that these conditions are crucial for preserving high fidelity rather than accuracy. The focus of the adversary in this case centers on replicating the decision boundary. While accuracy and fidelity share similarities in certain contexts, distinct decision boundaries may result in comparable performance but differ in fidelity. Consequently, it would be more appropriate for the authors to place greater emphasis on the concept of fidelity in their paper.

Author response:
We note in the introduction that in our paper we focus on accuracy rather than fidelity. The reasoning is twofold: first, fidelity-based extraction is not realistic since they assume very small ReLU-based networks; second, they are extremely inefficient even in such restricted settings. 
While fidelity is indeed an interesting objective for exploration, it has been shown by previous works [1,2,3] to underperform and therefore is out of scope for our work. 

In our work we focus on investigating the effect of having prior knowledge over the in-distribution region, and note that we make no assumptions about the input distribution itself. In our linear example, we provide intuition of the effect of an adversary that has no knowledge of what regions are of relevance and importance, and therefore require high accuracy, and what regions are of lesser importance and it is a waste of queries and effort to optimise the performance in these regions. The same argument can be sketched in a multi-dimensional setting. However, while in 2D the accuracy and fidelity arguments are inherently closely attached, this does not necessarily hold for higher dimensions.

Reviewer comment (Experiments):
In Section 4.3, the authors discuss the ME performance gains from the use of surrogate datasets. However, they don't just use surrogate data alone; instead, they mix part of the original dataset with the surrogate data. It would be beneficial to understand the rationale behind this mixed approach.

Author response:
We investigate how essential it is for an attacker to have prior knowledge over the true data distribution. For this goal, we model prior knowledge as having actual access to real training samples. We mix the real training samples with the surrogate data samples, to model how the attack performance is affected by the availability of such prior knowledge.

Reviewer comment (Presentation):
Presentation can be improved.

Author response:
We thank the reviewer for this comment and will improve the clarity of the manuscript. We edited the last paragraph of Section 3 to clarify its connection to the evaluations that follow. We additionally replaced the term “model stealing” with “model extraction” to avoid any confusion.

Reviewer comment (Experiments):
Based on my understanding, it is a measurement paper. Therefore, to achieve the conclusion, more experimental settings should be considered. For instance, if the architecture of the target models will influence the attack performance, or whether the target model that is trained under unsupervised loss may exhibit different results. I recommend the authors consider more settings for the comprehensive measurement conclusion.

Author response:
In our paper we investigate the effect of data access assumptions over the success of query-based model extraction attacks. Although there are numerous attack papers, many of them share common methodology. Therefore, we carefully chose our test cases to best capture the similarities. This includes investigating both vision and NLP tasks, as well as investigating different types of surrogate datasets and query selection methods. As for the effect of different architectures - we performed preliminary analysis of the effect of the size of the model in Appendix I, and observed that it has no significant impact. We show that the victim model mostly serves as a labelling oracle, it is its accuracy (and correspondingly capacity) that matters for the attack’s performance, and not the exact architecture. This is also established in the knowledge distillation literature that shows how a model of different architecture and size can distil the knowledge of another model [1, 2]. This conclusion also applies for model’s trained under different loss functions - the victim model is labelling the queries according to its learned decision boundaries and is therefore leaking this information when queried with relevant samples.

Reviewer comment (Writing):
The writing needs to be improved. It seems that the warm up part is not necessarily connected to the following evaluation part. Also, in this paper, model stealing attacks and model extraction attacks are used confusingly. In the experimental results part, the main conclusion is also not that obvious.

Author response:
We thank the reviewer for this comment and will improve the clarity of the manuscript. We edited the last paragraph of Section 3 to clarify its connection to the evaluations that follow. We additionally replaced the term “model stealing” with “model extraction” to avoid any confusion.

Reviewer comment (Theory):
Besides the experimental results, a certain degree of theoretical proof is necessary. Note that the proof is not the illustration presented in the warm up part. I would suggest the authors explain the commonality of ME attacks more rigorously.

Author response:
Could the reviewer please clarify what kind of theoretical proof is missing? In addition to linear intuition, we discuss in Section 5.1-5.2 the hardness of sampling in-distribution in the absence of prior knowledge of it and connect it to the fundamental classification hardness.  We discuss how in the absence of the ability to distinguish OOD queries, the attacker inevitably faces a higher query complexity by being forced to sample from both regions. On the other hand, following the results of [4] we show that if the attacker can distinguish between the regions, it can already perform classification, given labels provided by the victim model or any other labelling oracle. 

As for our OOD component, we discuss the theoretical reasoning behind it in Section F.4. Moreover, a connection between ME and active learning has been established by [3], therefore posing a theoretical limit to what can be proved in this setting. In other words, anything general we can prove here will inevitably improve classical generalisation bounds, hence we chose to convey intuition using a toy linear setting instead.

Reviewer comment (Evaluation):
While the authors draw conclusions from a diverse set of experiments, there appears to be a need for a principled approach to assess model extraction attacks. It would be valuable if the authors could provide clear and concise definitions that offer a unified perspective on all the attacks and defenses discussed in their paper. Currently, the experimental findings, while intuitive, seem somewhat fragmented and lack a well-organized presentation.

Author response:
Model extraction attacks in the current literature practically perform knowledge distillation. That in turn means that classical learning generalisation bounds apply and no general theory can be developed. 

In the same direction, [1, USENIX] argued that ``the process of ME is very similar to active learning’’ and therefore suggested that any improvement in active learning translates to improvement in ME attacks. In our work we address the common assumption that ME poses a significant threat (see [2]) and identify the reason why existing attacks “work”. We discover this success is mostly the result of having some prior knowledge, and with the absence of such prior the attack reduces to providing a labelling oracle for the attacker’s unlabeled data. As such we call for the community to re-evaluate the threat and perform more appropriate threat modelling. 

We are certain that stronger attacks will appear and it is paramount that we are capable of telling when they are discovered and why such attacks work. Our paper does the first step in that direction and separates away the impact of data prior and the attack policy.

Reviewer comment (Theory):
Similar challenges are evident in the comparison between OOD and IND. If the authors could provide a clear and concise mathematical definition aligned with the objectives of model extraction attacks, it could offer deeper insights.

Author response:
We do model the OOD adversaries and connect with the existing literature in our Section 5.2, which allows us in many cases to formally state that extracting the model is as hard as solving the classification task at hand. At the same time, we do want to also note that the IND/OOD community also has no clear definitions of what the distinctions are here.

We hope this clarifies the reviewer’s concerns and ask the reviewer to please let us know if there are any other concerns we can address.

Reviewer comment (Theory):
In summary, while the paper presents intriguing empirical findings, there is a need for the authors to establish straightforward and principled concepts to enhance the clarity of their arguments and conclusions.

Author response:
Model extraction attacks in the current literature practically perform knowledge distillation. That in turn means that classical learning generalisation bounds apply and no general theory can be developed. 

In the same direction, [1, USENIX] argued that ``the process of ME is very similar to active learning’’ and therefore suggested that any improvement in active learning translates to improvement in ME attacks. In our work we address the common assumption that ME poses a significant threat (see [2]) and identify the reason why existing attacks “work”. We discover this success is mostly the result of having some prior knowledge, and with the absence of such prior the attack reduces to providing a labelling oracle for the attacker’s unlabeled data. As such we call for the community to re-evaluate the threat and perform more appropriate threat modelling. 

We are certain that stronger attacks will appear and it is paramount that we are capable of telling when they are discovered and why such attacks work. Our paper does the first step in that direction and separates away the impact of data prior and the attack policy.