Paper: SEEKER: Semi-Supervised Knowledge Transfer for Query-Efficient Model Extraction (ICLR-2024)

Reviewer comment (Theory):
In the methodology part, there is no explicit discussion or intuition why the proposed method can improve the query efficiency, i.e., reduce the number of queries. If so, what is the query efficiency just a side effect? We need more justification than only experimental results.

Author response:
We thank the reviewer for acknowledging the technical merits of our manuscript and pointing out the problems in the presentation. We would like to make the following clarification on our contribution and make corresponding modifications to the manuscript.

> 1. Give explicit discussion or intuition why the proposed method can improve the query efficiency.

We appreciate the suggestion of the reviewer.
Our work improves the query-efficiency of model extraction attacks by proposing a *query-free self-supervised* scheme and a *query-efficient supervised* approach.
Both methods are aimed at extracting the knowledge from the public dataset effectively to reduce the query cost.
We would like to discuss the motivation of each method as follows.

**Query-free Self-supervised Scheme.**
We note that the conventional public dataset based model extraction attacks only use public data for crafting queries to the victim model.
In contrast, we develop a self-supervised training scheme to optimize the substitute model exclusively based on the public dataset, which does not require any queries to the victim model.
Our approach builds on the assumption that a well-trained victim model
maintains semantic consistency, i.e., outputs similar representations for different images featuring the same object.
Using this semantic consistency as an additional prior, our method optimizes the substitute model to learn similar encoding patterns to the victim model.
We also design different variants of the scheme according to the characteristics of offline and online stages that reduce the query cost of both stages.

**Query-efficient Supervised Scheme.**
To further improve the query-efficiency of the supervised learning stage of conventional attacks, we focus on synthesizing queries with richer information.
We point out that most existing model extraction methods based on public data feature a one-to-one correspondence between the public data and the generated query, which limits the information density in the query.
To increase the information diversity in the query generation process, we propose to parallelly process different public data when synthesizing a single query.
We design a multi-encoder aggregation design and reconstruction loss to merge the information from different public data, and develop inconsistency and diversity loss to ensure such information is fully leveraged for minimizing the gap between the substitute and the victim models.

In our revised manuscript, we also made major revisions to the overview of our method (Section 3.2), the semantic alignment scheme (Section 3.3), and the aggregated generation method (Section 3.4), including a more detailed discussion of how our attack reduces the query cost of the model extraction process.

Reviewer comment (Writing):
The paper presents a complex methodology without providing much intuition. I can get a main idea of the whole methodology, but the motivation of each part is not clear. For example, in section 3.2, why a good framework should be designed this way.

Author response:
> 2. Explain the motivation behind each part of the framework.

We appreciate the suggestion of the reviewer.
Here, we would like to introduce how each component of the method contributes to the overall performance and how they are incorporated into the framework.

Our framework comprises four main procedures: offline semantic alignment, online semantic alignment, query generation, and supervised training.
The offline semantic alignment belongs to an independent offline pre-training stage, and the adversary iterates through the online semantic alignment, query generation, and supervised training procedures during the online stage.
Among these procedures, conventional model extraction attacks only include query generation and supervised training procedures in the online stage, i.e., generate query inputs, query the victim model, and use the victim feedback to optimize the query generation process.
Our framework improves the existing pipeline by 1) introducing an offline stage (offline semantic alignment), 2) designing the online semantic alignment to assist the conventional supervised training scheme, and 3) proposing a novel aggregation generator for the query generation process.


**Offline semantic alignment.**
In contrast to existing online attacks, we design an offline stage to pre-train the substitute model.
During this stage, the adversary can not directly interact with the victim, but have access to a public dataset.
Under such circumstances, we design an offline semantic alignment strategy that leverages the public dataset to optimize the encoding layers of the substitute model in a self-supervised manner.
Notably, we identify semantic consistency as a common property in neural networks and utilize the prior to train the substitute model.
We also design a symmetric network architecture and projection layers that are suitable for the offline stage.

**Online semantic alignment.**
In addition to the offline semantic alignment, we further develop a self-supervised learning scheme to augment the online stage.
Different from the offline stage, the substitute model in the online stage is trained with the victim output, and has similar outputs to the victim model.
Therefore, the online semantic alignment scheme is aimed at further improving the performance of the substitute.
Online semantic alignment uses weak augmentations for the substitute to produce pseudo labels, and aligns the substitute output of strongly augmented data with the pseudo labels.
This approach gains extra accuracy increase for the substitute model over the original supervised training scheme.

**Informative query generation.**
We also make an important contribution to the conventional model extraction framework by designing a query generation procedure with richer information.
Our proposed query generation process achieves three main goals to increase the information density of synthesized queries:
1) *Aggregating*: the generator can effectively merge information from multiple public data, 
2) *Informative*: the generator can produce information-extracting queries that minimize the gap between the substitute and the victim,
3) *Stealthy*: the synthesized queries maintain the structure of a natural image instead of collapsing into indistinguishable patterns.
We propose
an aggregation architecture and three loss functions for the query generator to achieve all three goals at the same time.

We have also modified our manuscript to provide an overall discussion on the design of the framework (Section 3.2), and provide more motivation for each main component of our framework (Section 3.3, Section 3.4).

Reviewer comment (Presentation):
Is it possible to provide a simpler graph than Figure 1 for illustrating the main idea?

Author response:
> 3. Is it possible to provide a simpler graph than Figure 1 for illustrating the main idea?

We thank the reviewer for the suggestion.
We have modified Figure 1 to provide a clearer illustration of our two main contributions.
Specifically, we have simplified the illustration of our proposed aggregated query generation to highlight the novel aggregation design.

Reviewer comment (Novelty):
Semantic consistency is presented as one of the two methodological improvements proposed by the authors. The concept, however, doesn’t strike me as particularly new. Offline semantic alignment is simply self-supervised pre-training on a different dataset (very reminiscent of [b], in fact); it’s not particularly surprising that this is helpful, as that’s more or less the whole point of self-supervised learning. The similarities in Grad-CAM visualizations doesn’t necessarily have much to do with model extraction: it may be more due to having stronger features, which presumably the victim model also has, resulting in strong correlation. Online semantic consistency is more or less just knowledge distillation [d] with augmentations.

Author response:
We thank the reviewers for the questions. We would like to make the following clarifications. 


First, we point out that no existing work in the field of model extraction has explored self-supervised learning.
Although the Siamese design is common in self-supervised learning, we are the first to find that the design can be effective in model extraction, and adapt it with specific modules that are suitable for both the offline and online stages of the model extraction process.
Moreover, we are the first to design an offline pre-training stage for model extraction.
When designing the offline stage, we note that simply using *supervised* pre-training method for extracting *supervised* models has two major disadvantages: 1) supervised pre-training makes a strong and impractical assumption that the public dataset is annotated, and 2) such scheme performs worse that the proposed self-supervised scheme.
In contrast, we demonstrate the *self-supervised* learning approach not only eliminates the need for public annotation, but also demonstrates superior performance in extracting *supervised* models, as shown in Tab. R1.
Additionally, we would like to clarify that knowledge distillation is not applicable for online self-supervised training.
Knowledge distillation minimizes the distance between the outputs of the teacher and student model, but the two models are the same in a self-supervised setting.
Therefore, we design weak and strong augmentations to introduce semantic consistency as an extra prior that optimizes the substitute model in a self-supervised manner.

Table R1. Comparisons between supervised pre-training and our proposed self-supervised pre-training

|Pre-training strategy|Acc (\%)|Fid (\%)|ASR (\%)|
|---|---|---|---|
|Baseline|74.30|75.87|46.92|
|Baseline+supervised pre-training|81.56|82.02|78.31|
|Baseline+self-supervised pre-training (ours)|85.11|86.26|84.74|

Finally, we kindly point out that we carefully design the self-supervised approach according to the characteristics of the offline and online extraction stages.
Specifically, we would like to highlight the following contributions in designing self-supervised methods for two stages.

* Network architecture:
In the offline stage, the adversary does not have any knowledge of the victim model, so we design a symmetric architecture for this stage.
In the online stage, as the substitute model already has similar prediction scores to the victim, we design a weakly augmented branch without backpropagation to produce pseudo labels and a strongly augmented branch to learn richer semantic information.
We also design different levels of augmentations accordingly for each stage.

* Feature space for semantic alignment:
In the offline stage, we perform semantic alignment in a high-dimensional feature space to transfer representational information from the public dataset.
In the online stage, we directly optimize semantic consistency loss on the classification scores, as the substitute predictions are relatively accurate in this stage.

Reviewer comment (Experiments):
It is unclear why the query perturbation/residual is added to the first input sample $x_{pub,1}$ (as opposed to, say, $x_{pub,2}$). This seems like an arbitrary choice, and there could be a smarter way to select which of the $m$ samples to choose as the base sample.

Author response:
We thank the reviewer for the valuable suggestion.
We would like to point out that selecting the most suitable public data as the base image requires comparing the $m$ images based on the return of $V$, which causes extra queries and does not align with our goal of query efficiency.
In such a case, there is an exploration-utilization tradeoff between high query-efficiency and finding the theoretically optimal base image.
Therefore, we choose to achieve higher query-efficiency by randomly selecting a base image.

Reviewer comment (Theory):
It is unclear why there is a separate encoder per query index. Since the relative ordering of queries within a parallel batch is not meaningful, there doesn’t seem to be added value in having separate encoders compared to having a single shared encoder or another permutationally invariant design (e.g. a transformer).

Author response:
This is a very insightful comment, and we have also considered the shared encoder design when designing the aggregated query generator.
As shown in Tab. R2, we have found that using a shared encoder performs slightly worse than using different encoders for different indexes, and finally, we chose the latter as the current design.

Table R2. Comparisons between Encoder Architecture

|Encoder Architecture|Acc (\%)|Fid (\%)|ASR (\%)|
|---|---|---|---|
|shared encoder|86.72|88.26|94.82|
|different encoders (ours)|88.01|88.94|96.43|

A potential explanation for this observation is that the encoders for the base image (i.e., $x_{pub,1}$) and the merged images  (i.e., $x_{pub,2}, ..., x_{pub,m}$) learn different parameters for encoding different inputs.

Reviewer comment (Theory):
The design of the reconstruction loss is not fully clear. Why do we care about reconstructing the input data when we already have it? Instead of generating new images through an encoder and decoder, simpler methods like MixUp or CutMix could be considered. Additionally, the formulation in Equation 3 involves many hyperparameters $\alpha_j$ to tune, and it’s unclear why they should differ meaningfully; as in the previous point, the order of samples seems to have no inherent meaning, so for example why should $\alpha_2 \neq \alpha_3$?

Author response:
We design the reconstruction loss that optimizes the query generator to effectively merge the information from different images, and we introduce $\alpha_j$ to ensure the generated image largely resembles the base image instead of collapsing into unnatural patterns.
Therefore, we set the $\alpha_1$ to be larger for the base image and $\alpha_2,...,\alpha_m$ to be smaller for the other images.
Empirically, we find setting $\alpha_2=...=\alpha_m$ achieves a good performance, and did not carefully tune $\alpha$ for each image.

As for simple aggregations,  
we have considered a few basic aggregation operations when choosing the aggregation design.
We found that such aggregations have two shortcomings: 1) such methods produce unnatural query images that can be easily detected by the victim, and 2) such methods provide no significant improvements or even worse results when compared to the baseline.
As shown in Tab. R3, we have found our proposed aggregated generator performs significantly better than simple linear combinations, such as average, MixUp (weighted average with a random probability), or CutMix.

Table R3. Comparisons among different aggregation strategies
|Aggregation Strategy|Acc (\%)|Fid (\%)|ASR (\%)|
|---|---|---|---|
|Baseline|74.30|75.87|46.92|
|Average|72.77|73.65|56.32|
|MixUp|73.79|74.37|62.83|
|CutMix|75.41|76.49|64.47|
|Aggregated generator (ours)|84.75|86.23|94.33|

Reviewer comment (Novelty):
The ablation study in Section 4.4 shows that offline semantic alignment accounts for most of the improvements. While this is reasonable, as stated earlier it essentially amounts to starting with a stronger substitute model via standard self-supervised pre-training, which is well known to be generally effective for many downstream tasks. The form of self-supervised learning used here is fairly standard, so it is unclear if this can be considered the authors’ contribution.

Author response:
We thank the question of the reviewer regarding the novelty of self-supervised model extraction. 
We would like to clarify that it is necessary and novel to design self-supervised learning approach for model extraction.
As most existing model extraction attacks only focus on information-extracting **data synthesis**, we point out a different approach regarding the **training procedure** of the substitute model.
Our self-supervised approach is orthogonal to existing studies on supervised model extraction, and our proposed offline and online self-supervised methods can be combined with existing supervised learning attacks for significantly improved accuracy, ASR, and query-efficiency without extra query cost.
We believe this finding is impactful on the topic of model extraction, uncovering a different research direction for this field.
Furthermore, by proposing the self-supervised model extraction, we reveal an important finding that the utilization of public data can pose serious security threats to private training data and confidential models, raising concerns about the protection of AI privacy and robustness.

Additionally, we kindly point out that the modifications we made for the original self-supervised framework are novel and effective for model extraction.
Particularly, we have designed the stage-specific architecture, the feature space for alignment, and different semantically equivalent branches, all of which contribute to the strong overall performance.

Reviewer comment (Writing):
On page 2, “Mosafi et al.(Mosafi et al., 2019)” should use in-text citation.

Author response:
Section 2.1, used in-time citation: "Mosafi et al. (2019)"

Reviewer comment (Writing):
On page 3, “with [a] generative adversarial network” has a grammatical issue.

Author response:
Fixed the typo in Section 2.2: "with [a] generative adversarial network"

Reviewer comment (Novelty):
Section 2.2 omits transfer-based blackbox attacks, which generate attacks on the attacker’s own model and then transfer them to the target model. Such approaches can be combined with query-based attacks with low query requirements, e.g., [a].

Author response:
We have added the following description regarding transfer-based attacks in Section 2.2: "Generally speaking, we can classify black-box adversarial attacks into two categories:
substitute-based, transfer-based, and query-based. 
First,
a number of substitute-based attacks have already demonstrated the effectiveness
of using the extracted substitute model as a base for launching black-box 
adversarial attacks. 
Additionally, transfer-based attacks assume the adversary can obtain a substitute model trained on the same dataset as the victim model, and focus on improving the transferability of the adversarial samples synthesized based on the substitute model.
Therefore, substitute-based and transfer-based attacks are generally complementary to each other."

Reviewer comment (Writing):
On page 4, “a[n] aggregated” has a grammatical error.

Author response:
Fixed the typo in Section 3.2: "a[n] aggregated"

Reviewer comment (Writing):
On page 4, consider denoting $x_{pub}$ as $x_p$ to match $D_P$.

Author response:
We denote the public dataset as $\mathcal{D}\_{pub}$ to match $x_{pub}$, which is clearer to read. We also denote the secret dataset as $\mathcal{D}\_{secret}$ in our revised manuscript.

Reviewer comment (Writing):
On page 4, “$i$-th query dataset” may actually refer to a sample and label, not a dataset.

Author response:
We use a new index $j$ for a more clear description in Section 3.2: "Here, we take the $i$-th iteration with the query number of $n_i$ as 
an example...the adversary obtains the $i$-th query dataset $\mathcal{Q}\_i=\{(x\_{query}^{i,j}, y\_{query}^{i,j} = V(x\_{query}^{i,j}) | j = 1, ..., n_i\}$ by querying the victim.""

Reviewer comment (Novelty):
On page 4, the approach described is a standard Siamese network for self-supervised learning; more such methods should be cited (e.g. [b,c]).

Author response:
We have cited more papers for Siamese network architecture in Section 3.3.1.

Reviewer comment (Writing):
On page 5, “we [a] propose aggregated query generator” contains a grammatical issue.

Author response:
Fixed the typo in Section 3.4: "we propose [an] aggregated query generator"

Reviewer comment (Theory):
Regarding Eq. 5 — it may actually correspond to the $i+1$th iteration rather than the $i$th. Also, as it is written, it only ensures that a new query sample differs from the immediately previous one; it does not prevent a query from alternating between two queries.

Author response:
We have changed the notation $x_{query}^i$ to $x_{query}$. In the diversity loss, we are aimed at reducing the distance between the queries in the $i$-th iteration and all past queries. We thank the reviewer for pointing out the typo in the index.

Reviewer comment (Experiments):
Figure 5’s caption should specify the dataset used, which is also not mentioned in the text.

Author response:
Denoted the experimental setting for Fig.5, Tab.2, Tab.3, and Tab.4 in Section 4.1:  "In Figure 5, Table 2,
Table 3 and Table 4, we use CIFAR-10 as $\mathcal{D}\_{secret}$ and CIFAR-100 as $\mathcal{D}\_{pub}$ "

Reviewer comment (Experiments):
Table 2 does not specify the dataset used, which is also unspecified in the text.

Author response:
Denoted the experimental setting for Fig.5, Tab.2, Tab.3, and Tab.4 in Section 4.1:  "In Figure 5, Table 2,
Table 3 and Table 4, we use CIFAR-10 as $\mathcal{D}\_{secret}$ and CIFAR-100 as $\mathcal{D}\_{pub}$ "

Reviewer comment (Experiments):
Table 3 does not specify the dataset used, which is also unspecified in the text.

Author response:
Denoted the experimental setting for Fig.5, Tab.2, Tab.3, and Tab.4 in Section 4.1:  "In Figure 5, Table 2,
Table 3 and Table 4, we use CIFAR-10 as $\mathcal{D}\_{secret}$ and CIFAR-100 as $\mathcal{D}\_{pub}$ "

Reviewer comment (Experiments):
Table 4 does not specify the dataset used, which is also unspecified in the text.

Author response:
Denoted the experimental setting for Fig.5, Tab.2, Tab.3, and Tab.4 in Section 4.1:  "In Figure 5, Table 2,
Table 3 and Table 4, we use CIFAR-10 as $\mathcal{D}\_{secret}$ and CIFAR-100 as $\mathcal{D}\_{pub}$ "

Reviewer comment (Experiments):
How does the similarity of the public dataset to the target model’s dataset affect results? The main results in Table 1 show significant overlap: CIFAR-10 is used as the public dataset when CIFAR-100 is the hidden dataset, and vice versa, and even Tiny-ImageNet has strong similarities compared to CIFAR 10/100.

Author response:
We thank the reviewer for the question. We would like to point out that our attacks can still achieve state-of-the-art performance when the two datasets are very dissimilar in distribution.
As shown in Tab. 1 of the main manuscript, we have tested the performance of different attacks when the secret dataset is CIFAR-100, and the public dataset is ImageNet.
The CIFAR-100 and ImageNet datasets are collected independently, have distinctive classes without overlap, and also have different resolutions.
With such different distributions between the two datasets, our attack is able to demonstrate significantly stronger performance than the existing attacks.

Reviewer comment (Experiments):
Is the substitute model the same architecture as the victim model? Is that a reasonable assumption? What if they are different?

Author response:
We kindly point out that our attack obtains better accuracy and ASR than the best-performing attack across different substitute model architectures, as shown in Tab. 3 of the main manuscript.
Moreover, we have compared the performance of our attack across more substitute model architectures in Tab. A2 of our appendix.

Reviewer comment (Theory):
The method itself, although works great, seems a bit ad-hoc.

Author response:
We really appreciate the positive comments and valuable suggestions of the reviewer.
We would like to provide the following theoretical justifications for our proposed semantic alignment in this response.
We have also included this content in Section C of our revised Appendix.

The key insight of our semantic alignment approach is that the adversary can optimize a substitute model $S$ on a public dataset $\mathcal{D}\_{pub}$ to learn features that are similar to the victim model $V$, which is trained on a secret dataset $\mathcal{D}\_{secret}$.
Without loss of generality, we assume the substitute model $S$ is composed of an encoding function $f$ and a fully connected layer $W$.
In such a case, our observation can be reformulated as follows: if $f$ has a low semantic consistency loss and $W$ is trained on  $\mathcal{D}\_{secret}$ to evaluate the classification performance of $f$, $S$ has a low average classification loss on $\mathcal{D}\_{secret}$.
To prove this observation, we first formally define the notations.

The encoding function of $S$ belongs to $\mathcal{F}$, a class of representation functions $f: \mathcal{X} \rightarrow \mathbb{R}^d$, such that $||f(\cdot)||\leq R$ for some $R > 0$.
We denote the set of all classes in $\mathcal{D}\_{pub}$ as $\mathcal{C}$, and each $c\in\mathcal{C}$ follows a probability distribution $\mathcal{D}\_c$.
The supervised classification loss of $S$ on $\mathcal{D}\_{secret}$ can be defined as 
$$\mathcal{L}\_{\text{sup}}(S):= \mathbb{E}\_{(x,c)\in\mathcal{D}\_{secret}} \left[ l\left(S(x)\_c - S(x)\_{c'\neq c} \right) \right],$$
where $l$ is a standard hinge loss or logistic loss, and $S=Wf$.
When evaluating $S$, the best $W$ can be found by fixing $f$ and finetuning $W$. Therefore, we only denote the supervised loss of $f$ on $\mathcal{D}\_{secret}$ as:
$$\mathcal{L}\_{\text{sup}}(f) =\inf_W \mathcal{L}\_{\text{sup}}(Wf).$$
Also, our offline semantic consistency loss can be formalized as
$$
        \mathcal{L}\_C = -\mathbb{E}\_{x\in\mathcal{D}\_{pub}}\left[{\rm log}
        \frac{{\rm sim}(S({\rm aug_{M}^1}(x)),S({\rm aug_{M}^2}(x)))}
        {\sum_{x'\in\mathcal{D}\_{pub}, x' \neq x} {\rm sim}(S(x),S(x'))} \right],
$$
The loss term can be simplified as 
$$
\mathcal{L}\_C  = \frac{1}{M} \sum_{i=1}^{M} l\left(f(x_j)^{T}(f(x_j^+) - f(x_j'))\right),
$$
Here, $x_j$ and $x_j^+$ are semantically equivalent data constructed by augmentations.
With the notations above, we formalize a proposition as follows.

**Proposition 1.**  *For a substitute model $S$ composed of an encoding function $f$ and a fully connected classification layer $W$, $S$ has a low average linear classification loss on $\mathcal{D}\_{secret}$ if $f$ has a low offline semantic consistency loss on $\mathcal{D}\_{pub}$.*


We use a theorem proposed in [1] (Theorem 4.1 of the original paper) to prove this proposition.
Let $\mathcal{S}=\lbrace x_j, x_j^+, x_j' \rbrace_{j=1}^M$ be the triplets sampled from $\mathcal{D}\_{pub}$ to optimize semantic consistency loss, 
$f\_{|\mathcal{S}} = \left(f_t(x_j), f_t(x_j^+), f_t(x_j')\right)\_{j\in[M],t\in[d]}\in \mathbb{R}^{3dM}$ be the restriction for $\mathcal{S}$ for any $f\in \mathcal{F}$, and we have a complexity measure with the following Rademacher average
$$
\mathcal{R}(\mathcal{F}) = \mathbb{E}\_{\sigma\in\lbrace\pm 1\rbrace^{3dM}}\left[\sup_{f\in \mathcal{F}}<\sigma, f\_{|\mathcal{S}}>\right].
$$
Let $\tau=\mathbb{E}\_{c,c'\sim\rho^2}\lbrace c=c'\rbrace$, and we have the following theorem[1]:

**Theorem 1.** *With probability at least $1-\tau$, for all $f\in F$*
$$\mathcal{L}\_{\text{sup}} (\widehat{f}) \leqslant \frac{1}{(1-\tau)} \mathcal{L}\_C(f) - \frac{\tau}{(1-\tau)} + \frac{1}{(1-\tau)} Gen_{M}$$
*where*
$$\ Gen_{M} = O\left(R \frac{R_{s}(F)}{M} + R^{2} \sqrt{\frac{\log \frac{1}{d}}{M}}\right)$$
Here, we have $Gen_{M} \rightarrow 0$ as $M\rightarrow \infty$, and when $\rho$ is uniform and the number of classes $|C|\rightarrow \infty$, we have that $\frac{1}{(1-\tau)}\rightarrow 0, -\frac{\tau}{(1-\tau)}\rightarrow 0$.
Therefore, when the number of sampled training triplets is large and $f$ has a low offline semantic consistency loss on $\mathcal{D}\_{pub}$, then $S$ has a low average linear classification loss on $\mathcal{D}\_{secret}$.

[1] Saunshi, Nikunj, et al. "A theoretical analysis of contrastive unsupervised representation learning." International Conference on Machine Learning. PMLR, 2019.

Reviewer comment (Theory):
It will be great to see some theoretical justifications behind the framework.

Author response:
We really appreciate the positive comments and valuable suggestions of the reviewer.
We would like to provide the following theoretical justifications for our proposed semantic alignment in this response.
We have also included this content in Section C of our revised Appendix.

The key insight of our semantic alignment approach is that the adversary can optimize a substitute model $S$ on a public dataset $\mathcal{D}\_{pub}$ to learn features that are similar to the victim model $V$, which is trained on a secret dataset $\mathcal{D}\_{secret}$.
Without loss of generality, we assume the substitute model $S$ is composed of an encoding function $f$ and a fully connected layer $W$.
In such a case, our observation can be reformulated as follows: if $f$ has a low semantic consistency loss and $W$ is trained on  $\mathcal{D}\_{secret}$ to evaluate the classification performance of $f$, $S$ has a low average classification loss on $\mathcal{D}\_{secret}$.
To prove this observation, we first formally define the notations.

The encoding function of $S$ belongs to $\mathcal{F}$, a class of representation functions $f: \mathcal{X} \rightarrow \mathbb{R}^d$, such that $||f(\cdot)||\leq R$ for some $R > 0$.
We denote the set of all classes in $\mathcal{D}\_{pub}$ as $\mathcal{C}$, and each $c\in\mathcal{C}$ follows a probability distribution $\mathcal{D}\_c$.
The supervised classification loss of $S$ on $\mathcal{D}\_{secret}$ can be defined as 
$$\mathcal{L}\_{\text{sup}}(S):= \mathbb{E}\_{(x,c)\in\mathcal{D}\_{secret}} \left[ l\left(S(x)\_c - S(x)\_{c'\neq c} \right) \right],$$
where $l$ is a standard hinge loss or logistic loss, and $S=Wf$.
When evaluating $S$, the best $W$ can be found by fixing $f$ and finetuning $W$. Therefore, we only denote the supervised loss of $f$ on $\mathcal{D}\_{secret}$ as:
$$\mathcal{L}\_{\text{sup}}(f) =\inf_W \mathcal{L}\_{\text{sup}}(Wf).$$
Also, our offline semantic consistency loss can be formalized as
$$
        \mathcal{L}\_C = -\mathbb{E}\_{x\in\mathcal{D}\_{pub}}\left[{\rm log}
        \frac{{\rm sim}(S({\rm aug_{M}^1}(x)),S({\rm aug_{M}^2}(x)))}
        {\sum_{x'\in\mathcal{D}\_{pub}, x' \neq x} {\rm sim}(S(x),S(x'))} \right],
$$
The loss term can be simplified as 
$$
\mathcal{L}\_C  = \frac{1}{M} \sum_{i=1}^{M} l\left(f(x_j)^{T}(f(x_j^+) - f(x_j'))\right),
$$
Here, $x_j$ and $x_j^+$ are semantically equivalent data constructed by augmentations.
With the notations above, we formalize a proposition as follows.

**Proposition 1.**  *For a substitute model $S$ composed of an encoding function $f$ and a fully connected classification layer $W$, $S$ has a low average linear classification loss on $\mathcal{D}\_{secret}$ if $f$ has a low offline semantic consistency loss on $\mathcal{D}\_{pub}$.*


We use a theorem proposed in [1] (Theorem 4.1 of the original paper) to prove this proposition.
Let $\mathcal{S}=\lbrace x_j, x_j^+, x_j' \rbrace_{j=1}^M$ be the triplets sampled from $\mathcal{D}\_{pub}$ to optimize semantic consistency loss, 
$f\_{|\mathcal{S}} = \left(f_t(x_j), f_t(x_j^+), f_t(x_j')\right)\_{j\in[M],t\in[d]}\in \mathbb{R}^{3dM}$ be the restriction for $\mathcal{S}$ for any $f\in \mathcal{F}$, and we have a complexity measure with the following Rademacher average
$$
\mathcal{R}(\mathcal{F}) = \mathbb{E}\_{\sigma\in\lbrace\pm 1\rbrace^{3dM}}\left[\sup_{f\in \mathcal{F}}<\sigma, f\_{|\mathcal{S}}>\right].
$$
Let $\tau=\mathbb{E}\_{c,c'\sim\rho^2}\lbrace c=c'\rbrace$, and we have the following theorem[1]:

**Theorem 1.** *With probability at least $1-\tau$, for all $f\in F$*
$$\mathcal{L}\_{\text{sup}} (\widehat{f}) \leqslant \frac{1}{(1-\tau)} \mathcal{L}\_C(f) - \frac{\tau}{(1-\tau)} + \frac{1}{(1-\tau)} Gen_{M}$$
*where*
$$\ Gen_{M} = O\left(R \frac{R_{s}(F)}{M} + R^{2} \sqrt{\frac{\log \frac{1}{d}}{M}}\right)$$
Here, we have $Gen_{M} \rightarrow 0$ as $M\rightarrow \infty$, and when $\rho$ is uniform and the number of classes $|C|\rightarrow \infty$, we have that $\frac{1}{(1-\tau)}\rightarrow 0, -\frac{\tau}{(1-\tau)}\rightarrow 0$.
Therefore, when the number of sampled training triplets is large and $f$ has a low offline semantic consistency loss on $\mathcal{D}\_{pub}$, then $S$ has a low average linear classification loss on $\mathcal{D}\_{secret}$.

[1] Saunshi, Nikunj, et al. "A theoretical analysis of contrastive unsupervised representation learning." International Conference on Machine Learning. PMLR, 2019.