Original: P ROTECTING DNN S FROM T HEFT USING AN E NSEMBLE OF D IVERSE M ODELS
Revised: P ROTECTING DNN S FROM T HEFT USING AN E NSEMBLE OF D IVERSE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Several recent works have demonstrated highly effective model stealing (MS) attacks on Deep Neural Networks (DNNs) in black-box settings, even when the training data is unavailable.
Revised: Several recent works have demonstrated highly effective model stealing (MS) attacks on Deep Neural Networks (DNNs) in black-box settings, even when the training data is unavailable.

Original: These attacks typically use some form of Out of Distribution (OOD) data to query the target model and use the predictions obtained to train a clone model.
Revised: These attacks typically use some form of Out of Distribution (OOD) data to query the target model and use the predictions obtained to train a clone model.

Original: Such a clone model learns to approximate the decision boundary of the target model, achieving high accuracy on in-distribution examples.
Revised: Such a clone model learns to approximate the decision boundary of the target model, achieving high accuracy on in-distribution examples.

Original: We propose Ensemble of Diverse Models (EDM) to defend against such MS attacks.
Revised: We propose Ensemble of Diverse Models (EDM) to defend against such MS attacks.

Original: EDM is made up of models that are trained to produce dissimilar predictions for
Revised: EDM is made up of models that are trained to produce dissimilar predictions for

Original: OOD inputs.
Revised: OOD inputs.

Original: By using a different member of the ensemble to service different queries, our defense produces predictions that are highly discontinuous in the input space for the adversary’s
Revised: By using a different member of the ensemble to service different queries, our defense produces predictions that are highly discontinuous in the input space for the adversary’s

Original: OOD queries.
Revised: OOD queries.

Original: Such discontinuities cause the clone model trained on these predictions to have poor generalization on in-distribution examples.
Revised: Such discontinuities cause the clone model trained on these predictions to have poor generalization on in-distribution examples.

Original: Our evaluations on several image classiﬁcation tasks demonstrate that
Revised: Our evaluations on several image classiﬁcation tasks demonstrate that

Original: EDM defense can severely degrade the accuracy of clone models (up to 39 .
Revised: EDM defense can severely degrade the accuracy of clone models (up to 39 .

Original: Our defense has minimal impact on the target accuracy, negligible computational costs during inference, and is compatible with existing defenses for MS attacks.
Revised: Our defense has minimal impact on the target accuracy, negligible computational costs during inference, and is compatible with existing defenses for MS attacks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: MS attacks allow an adversary with black-box access to the predictions of the target model to copy its functionality and create a high-accuracy clone model, posing a threat to the conﬁdentiality of proprietary DNNs.
Revised: MS attacks allow an adversary with black-box access to the predictions of the target model to copy its functionality and create a high-accuracy clone model, posing a threat to the conﬁdentiality of proprietary DNNs.

Original: Such attacks also open the door to a wide range of security vulnerabilities including adversarial attacks (Goodfellow et al., 2014) that cause misclassiﬁcation, membership-inference attacks (Shokri et al., 2017) that leak membership, and model-inversion attacks (Fredrikson et al., 2015) that reveal the data used to train the model.
Revised: Such attacks also open the door to a wide range of security vulnerabilities including adversarial attacks (Goodfellow et al., 2014) that cause misclassiﬁcation, membership-inference attacks (Shokri et al., 2017) that leak membership, and model-inversion attacks (Fredrikson et al., 2015) that reveal the data used to train the model.

Original: MS is carried out using the principle of Knowledge distillation (KD), wherein the adversary uses a dataset D to query the target model.
Revised: MS is carried out using the principle of Knowledge distillation (KD), wherein the adversary uses a dataset D to query the target model.

Original: The predictions of the target on D are then used to train a clone model that replicates the target model’s functionality.
Revised: The predictions of the target on D are then used to train a clone model that replicates the target model’s functionality.

Original: Since access to training data is limited in most real-world settings, attacks typically use some form of OOD data to perform KD.
Revised: Since access to training data is limited in most real-world settings, attacks typically use some form of OOD data to perform KD.

Original: Clone models trained in this way closely approximate the decision boundaries of the target model, achieving high-accuracy on in-distribution examples.
Revised: Clone models trained in this way closely approximate the decision boundaries of the target model, achieving high-accuracy on in-distribution examples.

Original: The goal of this paper is to defend against MS attacks by creating a target model that is inherently hard to steal using Knowledge Distillation with OOD data.
Revised: The goal of this paper is to defend against MS attacks by creating a target model that is inherently hard to steal using Knowledge Distillation with OOD data.

Original: Our key observation is the existing MS attacks (Orekondy et al., 2019a; Papernot et al., 2017; Juuti et al., 2019) implicitly assume that the target model produces continuous predictions in the input space.
Revised: Our key observation is the existing MS attacks (Orekondy et al., 2019a; Papernot et al., 2017; Juuti et al., 2019) implicitly assume that the target model produces continuous predictions in the input space.

Original: We hypothesise that making the predictions of the target model discontinuous makes MS attacks harder to carry out.
Revised: We hypothesise that making the predictions of the target model discontinuous makes MS attacks harder to carry out.

Original: To this end, we propose Ensemble of Diverse Models (EDM) to defend against MS attacks.
Revised: To this end, we propose Ensemble of Diverse Models (EDM) to defend against MS attacks.

Original: The models in EDM are trained using a novel diversity loss to produce dissimilar predictions on OOD data.
Revised: The models in EDM are trained using a novel diversity loss to produce dissimilar predictions on OOD data.

Original: Each input query to EDM is serviced by a single model that is selected from the ensemble using an input-based hashing function.
Revised: Each input query to EDM is serviced by a single model that is selected from the ensemble using an input-based hashing function.

Original: We develop a DNN-based perceptual hashing algorithm for this purpose, which is invariant to simple transformations of the input and prevents adaptive attacks.
Revised: We develop a DNN-based perceptual hashing algorithm for this purpose, which is invariant to simple transformations of the input and prevents adaptive attacks.

Original: Since different models in the ensemble are used to service different queries and the models are trained to produce dissimilar predictions, the adversary obtains predictions that are highly discontinuous in the input space.
Revised: Since different models in the ensemble are used to service different queries and the models are trained to produce dissimilar predictions, the adversary obtains predictions that are highly discontinuous in the input space.

Original: The clone model, when trained on these predictions, tries to approximate the complex discontinuous decision boundary of the EDM.
Revised: The clone model, when trained on these predictions, tries to approximate the complex discontinuous decision boundary of the EDM.

Original: Our empirical evaluations shows that the resulting clone model generalizes poorly on in-distribution data, degrading clone accuracy and reducing the efﬁcacy of MS attacks.
Revised: Our empirical evaluations show that the resulting clone model generalizes poorly on in-distribution data, degrading clone accuracy and reducing the efﬁcacy of MS attacks.

Original: In contrast to existing defenses that rely on perturbing output predictions, our defense
Revised: In contrast to existing defenses that rely on perturbing output predictions, our defense

Original: [Figure]
Revised: [Figure]

Original: ( i ) U n d e f e n d e d
Revised: ( i ) U n d e f e n d e d

Original: B a s e li n e
Revised: B a s e li n e

Original: Model-A
Revised: Model-A

Original: Model-B ( ii ) E
Revised: Model-B ( ii ) E

Original: D M + does not require modifying the output and uses a diverse ensemble to produce discontinuous predictions are inherently hard
Revised: D M + does not require modifying the output and uses a diverse ensemble to produce discontinuous predictions are inherently hard

Original: to
Revised: to

Original: We illustrate the working of our defense by comparing the results of MS attacks on two target models– (i) Undefended baseline (ii) EDM – trained on a toy binary-classiﬁcation problem shown in Fig.
Revised: We illustrate the working of our defense by comparing the results of MS attacks on two target models– (i) Undefended baseline (ii) EDM – trained on a toy binary-classiﬁcation problem shown in Fig.

Original: The training data and predictions of the target model under attack are shown in Fig.
Revised: The training data and predictions of the target model under attack are shown in Fig.

Original: 1a.
Revised: 1a.

Original: We use a set of OOD points as shown in Fig.
Revised: We use a set of OOD points as shown in Fig.

Original: 1b to query the target model and obtain its predictions.
Revised: 1b to query the target model and obtain its predictions.

Original: The predictions of the target model on the OOD data is ﬁnally used to train the clone model to replicate the functionality of the target as shown in Fig. 1c.
Revised: The predictions of the target model on the OOD data is ﬁnally used to train the clone model to replicate the functionality of the target as shown in Fig. 1c.

Original: For the undefended baseline target, which uses a simple linear model to produce continuous predictions, the clone model obtained by the attack closely approximates the decision boundary of the target model, achieving good classiﬁcation accuracy on in-distribution data.
Revised: For the undefended baseline target, which uses a simple linear model to produce continuous predictions, the clone model obtained by the attack closely approximates the decision boundary of the target model, achieving good classiﬁcation accuracy on in-distribution data.

Original: The EDM target consists of two diverse models (Model-A and Model-B) that produce dissimilar predictions on the OOD data.
Revised: The EDM target consists of two diverse models (Model-A and Model-B) that produce dissimilar predictions on the OOD data.

Original: EDM uses an input-based hash function to select the model that services the input query.
Revised: EDM uses an input-based hash function to select the model that services the input query.

Original: As a result, the target model produces highly discontinuous decisions on OOD data (Fig.
Revised: As a result, the target model produces highly discontinuous decisions on OOD data (Fig.

Original: 1a,b for EDM).
Revised: 1a,b for EDM).

Original: The clone model trained on this data fails to capture any meaningful decision boundary (Fig. 1c for
Revised: The clone model trained on this data fails to capture any meaningful decision boundary (Fig. 1c for

Original: EDM) and produces poor accuracy on the classiﬁcation task, making MS attacks harder to carry out.
Revised: EDM) and produces poor accuracy on the classiﬁcation task, making MS attacks harder to carry out.

Original: In summary, our paper makes the following key contributions:
Revised: In summary, our paper makes the following key contributions:

Original: We propose a novel Diversity Loss function to train an Ensemble of Diverse Models (EDM) that produce dissimilar predictions on OOD data.
Revised: We propose a novel Diversity Loss function to train an Ensemble of Diverse Models (EDM) that produce dissimilar predictions on OOD data.

Original: We propose using EDM to defend against model stealing attacks.
Revised: We propose using EDM to defend against model stealing attacks.

Original: Our defense creates discontinuous predictions for the adversary’s
Revised: Our defense creates discontinuous predictions for the adversary’s

Original: OOD queries, making MS attacks harder to carry out, without causing degradation to the model’s accuracy on benign queries.
Revised: OOD queries, making MS attacks harder to carry out, without causing degradation to the model’s accuracy on benign queries.

Original: We develop a DNN-based perceptual hash function, which produces the same hash value even with large changes to the input, making adaptive attacks harder to carry out.
Revised: We develop a DNN-based perceptual hash function, which produces the same hash value even with large changes to the input, making adaptive attacks harder to carry out.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Model Stealing Attacks: MS Attacks fall into one of three categories: (i) parameter stealing, (ii) hyper-parameter stealing and (iii) functionality stealing attacks.
Revised: Model Stealing Attacks: MS Attacks fall into one of three categories: (i) parameter stealing, (ii) hyper-parameter stealing and (iii) functionality stealing attacks.

Original: Parameter stealing (Tram`er et al., 2016; Lowd & Meek, 2005) and hyperparameter stealing attacks (Wang & Gong, 2018;
Revised: Parameter stealing (Tram`er et al., 2016; Lowd & Meek, 2005) and hyperparameter stealing attacks (Wang & Gong, 2018;

Original: Oh et al., 2019) focus on inferring the exact model parameters or hyperparameters used in the architecture/training of the target model respectively.
Revised: Oh et al., 2019) focus on inferring the exact model parameters or hyperparameters used in the architecture/training of the target model respectively.

Original: Our work focuses on defending against functionality stealing attacks, where the adversary’s goal is to train a clone model that achieves high accuracy on a given task.
Revised: Our work focuses on defending against functionality stealing attacks, where the adversary’s goal is to train a clone model that achieves high accuracy on a given task.

Original: Since the training data of the target is usually unavailable, attacks leverage alternate forms of data to query the target.
Revised: Since the training data of the target is usually unavailable, attacks leverage alternate forms of data to query the target.

Original: Orekondy et al. (2019a); Correia-Silva et al.
Revised: Orekondy et al. (2019a); Correia-Silva et al.

Original: (2018) use datasets from related problem domains, while other attacks (Papernot et al., 2017; Juuti et al., 2019; Roberts et al., 2019; Kariyappa et al., 2020) have proposed crafting synthetic examples to query the target to perform model stealing.
Revised: (2018) use datasets from related problem domains, while other attacks (Papernot et al., 2017; Juuti et al., 2019; Roberts et al., 2019; Kariyappa et al., 2020) have proposed crafting synthetic examples to query the target to perform model stealing.

Original: Model Stealing Defenses: Existing defenses that have been proposed to defend against model functionality stealing involve perturbing the output predictions to prevent the adversary from having reliable access to the predictions of the target model.
Revised: Model Stealing Defenses: Existing defenses that have been proposed to defend against model functionality stealing involve perturbing the output predictions to prevent the adversary from having reliable access to the predictions of the target model.

Original: Perturbation-based defenses modify the output of the model y =
Revised: Perturbation-based defenses modify the output of the model y =

Original: f ( x ) to produce a perturbed output y (cid:48)
Revised: f ( x ) to produce a perturbed output y (cid:48)

Original: =
Revised: =

Original: y + δ .
Revised: y + δ .

Original: These defenses can be further sub-categorized into accuracy-preserving and accuracy-constrained defenses.
Revised: These defenses can be further sub-categorized into accuracy-preserving and accuracy-constrained defenses.

Original: Accuracy-Preserving Defenses (Lee et al., 2018) ensure that the class-labels of the perturbed predictions y (cid:48) are the same as the original prediction i.e. argmax ( y (cid:48) )
Revised: Accuracy-Preserving Defenses (Lee et al., 2018) ensure that the class-labels of the perturbed predictions y (cid:48) are the same as the original prediction i.e. argmax ( y (cid:48) )

Original: =
Revised: =

Original: argmax ( y ) .
Revised: argmax ( y ) .

Original: Since the class-labels are unchanged, the benign accuracy of the model is preserved.
Revised: Since the class-labels are unchanged, the benign accuracy of the model is preserved.

Original: In contrast, accuracyconstrained defenses (Kariyappa & Qureshi, 2020; Orekondy et al., 2019b) allow for the usage of a higher level of perturbation, trading off benign accuracy for increased security against MS attacks.
Revised: In contrast, accuracyconstrained defenses (Kariyappa & Qureshi, 2020; Orekondy et al., 2019b) allow for the usage of a higher level of perturbation, trading off benign accuracy for increased security against MS attacks.

Original: This work presents the ﬁrst defense that is not reliant on adding perturbations to the predictions of the model.
Revised: This work presents the ﬁrst defense that is not reliant on adding perturbations to the predictions of the model.

Original: Instead, we aim to create an entirely new class of ensemble-based models that is inherently hard to steal with a data-limited attack, due to the discontinuous nature of the predictions produced by the model.
Revised: Instead, we aim to create an entirely new class of ensemble-based models that is inherently hard to steal with a data-limited attack, due to the discontinuous nature of the predictions produced by the model.

Original: Our method has minimal impact on the benign accuracy and offers a much higher security against MS attacks compared to a single model.
Revised: Our method has minimal impact on the benign accuracy and offers a much higher security against MS attacks compared to a single model.

Original: Additionally, our model can be used in conjunction with perturbation-based defenses to further improve robustness against MS attacks.
Revised: Additionally, our model can be used in conjunction with perturbation-based defenses to further improve robustness against MS attacks.

Original:  P RELIMINARIES 3. 
Revised:  P RELIMINARIES 3.

Original: A
Revised: A

Original: TTACK
Revised: TTACK

Original: : O
Revised: : O

Original: BJECTIVE
Revised: BJECTIVE

Original: AND C ONSTRAINTS
Revised: AND C ONSTRAINTS

Original: The goal of the attacker is to maximize the accuracy of a clone model C on a test set D test i.e. max θ C E x,y ∼D test [ Acc ( C ( x ; θ C ) , y )] .
Revised: The goal of the attacker is to maximize the accuracy of a clone model C on a test set D test i.e. max θ C E x,y ∼D test [ Acc ( C ( x ; θ C ) , y )] .

Original: MS attacks use the principle of knowledge distillation to train a clone model.
Revised: MS attacks use the principle of knowledge distillation to train a clone model.

Original: First, the attacker queries the target model using inputs { x i } ni =1 to obtain the target’s predictions { y i
Revised: First, the attacker queries the target model using inputs { x i } ni =1 to obtain the target’s predictions { y i

Original: =
Revised: =

Original: T ( x i ) } .
Revised: T ( x i ) } .

Original: The labeled dataset { x i , y i } created this way can then be used to train a clone model C , completing the attack.
Revised: The labeled dataset { x i , y i } created this way can then be used to train a clone model C , completing the attack.

Original: The predictions returned from T can either be soft-label prediction , where T returns a probability distribution over the output classes, or hardlabel predictions , where only the argmax class-label is returned.
Revised: The predictions returned from T can either be soft-label prediction , where T returns a probability distribution over the output classes, or hardlabel predictions , where only the argmax class-label is returned.

Original: We consider attacks under both prediction types in this paper.
Revised: We consider attacks under both prediction types in this paper.

Original: Since there is a ﬁxed cost associated with querying T , we assume that the number of queries is upper bounded by a query budget Q .
Revised: Since there is a ﬁxed cost associated with querying T , we assume that the number of queries is upper bounded by a query budget Q .

Original: Data Constraints :
Revised: Data Constraints :

Original: In most real-world settings, the adversary does not have access to the dataset used to train the target model.
Revised: In most real-world settings, the adversary does not have access to the dataset used to train the target model.

Original: A data-limited attacker can use alternate forms of data to carry out model stealing attacks.
Revised: A data-limited attacker can use alternate forms of data to carry out model stealing attacks.

Original: KnockoffNets
Revised: KnockoffNets

Original: (Orekondy et al., 2019a) uses data from an alternate surrogate dataset, which has semantic similarities with the target model and Jacobian-Based Dataset
Revised: (Orekondy et al., 2019a) uses data from an alternate surrogate dataset, which has semantic similarities with the target model and Jacobian-Based Dataset

Original: Augmentation (Papernot et al., 2017) uses synthetic datasets to query the target model.
Revised: Augmentation (Papernot et al., 2017) uses synthetic datasets to query the target model.

Original: 3. D EFENSE : O BJECTIVE AND C ONSTRAINTS
Revised: 3. D EFENSE : O BJECTIVE AND C ONSTRAINTS

Original: The goal of the defense is to minimize the accuracy of the clone model i.e. min E x,y ∼D test
Revised: The goal of the defense is to minimize the accuracy of the clone model i.e. min E x,y ∼D test

Original: [ Acc ( C ( x ; θ C ) , y )] .
Revised: [ Acc ( C ( x ; θ C ) , y )] .

Original: The defense is allowed to make changes to the target model or the model predictions.
Revised: The defense is allowed to make changes to the target model or the model predictions.

Original: However, we require the defense to have minimal impact on the accuracy of the model for benign inputs to preserve the utility of the model.
Revised: However, we require the defense to have minimal impact on the accuracy of the model for benign inputs to preserve the utility of the model.

Original: The defender is unaware of the kind of attack being used, or the dataset used by the adversary to query the target model.
Revised: The defender is unaware of the kind of attack being used, or the dataset used by the adversary to query the target model.

Original: In addition to existing attacks, the defense also needs to be effective against adaptive attacks that are tailored for the defense.
Revised: In addition to existing attacks, the defense also needs to be effective against adaptive attacks that are tailored for the defense.

Original:  O UR P ROPOSAL : E NSEMBLE OF D IVERSE M ODELS
Revised:  O UR P ROPOSAL : E NSEMBLE OF D IVERSE M ODELS

Original: This paper proposes a novel type of model ensemble called Ensemble of Diverse Models (EDM), which is signiﬁcantly more robust to MS attacks compared to a single DNN model.
Revised: This paper proposes a novel type of model ensemble called Ensemble of Diverse Models (EDM), which is signiﬁcantly more robust to MS attacks compared to a single DNN model.

Original: EDM consists
Revised: EDM consists

Original: [Figure]
Revised: [Figure]

Original: of a set of diverse models, which produce dissimilar predictions for OOD data.
Revised: of a set of diverse models, which produce dissimilar predictions for OOD data.

Original: By using an
Revised: By using an

Original: - based hash function
Revised: - based hash function

Original: , EDM selects a single model from the ensemble to service each query.
Revised: , EDM selects a single model from the ensemble to service each query.

Original: Since the models are trained to produce dissimilar predictions, EDM outputs predictions that are highly discontinuous in the input space for OOD data.
Revised: Since the models are trained to produce dissimilar predictions, EDM outputs predictions that are highly discontinuous in the input space for OOD data.

Original: The complexity of the discontinuous predictions cause the clone model trained on these predictions to generalize poorly on in-distribution examples, making MS attacks less effective.
Revised: The complexity of the discontinuous predictions cause the clone model trained on these predictions to generalize poorly on in-distribution examples, making MS attacks less effective.

Original: We explain how EDM models can be trained and deployed for inference in this section.
Revised: We explain how EDM models can be trained and deployed for inference in this section.

Original: 4. T RAINING A D IVERSE E NSEMBLE
Revised: 4. T RAINING A D IVERSE E NSEMBLE

Original: [Figure]
Revised: [Figure]

Original: EDM leverages an ensemble of diverse models to produce discontinuous predictions.
Revised: EDM leverages an ensemble of diverse models to produce discontinuous predictions.

Original: Let { f i } i = N i
Revised: Let { f i } i = N i

Original: =
Revised: =

Original: 1 denote the ensemble of models used in EDM with model parameters { θ i } i
Revised: 1 denote the ensemble of models used in EDM with model parameters { θ i } i

Original: = N
Revised: = N

Original: i =1 .
Revised: i =1 .

Original: These models are jointly trained with a two-fold training objective using an in-distribution dataset of labeled examples D in and an out-ofdistribution dataset of unlabeled examples D out as explained below:
Revised: These models are jointly trained with a two-fold training objective using an in-distribution dataset of labeled examples D in and an out-ofdistribution dataset of unlabeled examples D out as explained below:

Original: Accuracy objective: Models in the ensemble should be trained to produce high accuracy for the examples in the training dataset of in-distribution examples D in as shown in Fig.
Revised: Accuracy objective: Models in the ensemble should be trained to produce high accuracy for the examples in the training dataset of in-distribution examples D in as shown in Fig.

Original: 2a.
Revised: 2a.

Original: For a multi-class classiﬁcation problem, let { ˆ y i
Revised: For a multi-class classiﬁcation problem, let { ˆ y i

Original: =
Revised: =

Original: f i ( x ; θ i ) } Ni =1 be the prediction probabilities of the models where ( x, y ) ∈ D in is the input, label pair.
Revised: f i ( x ; θ i ) } Ni =1 be the prediction probabilities of the models where ( x, y ) ∈ D in is the input, label pair.

Original: We use the cross-entropy loss averaged across all models:
Revised: We use the cross-entropy loss averaged across all models:

Original: N (cid:80) Ni
Revised: N (cid:80) Ni

Original: =1 L CE (ˆ y i , y ) to train the models in the ensemble to achieve high accuracy on the in-distribution training data.
Revised: =1 L CE (ˆ y i , y ) to train the models in the ensemble to achieve high accuracy on the in-distribution training data.

Original: Diversity objective: The diversity objective requires the models in the ensemble to produce dissimilar predictions for examples in the out of distribution dataset D out as depicted in Fig.
Revised: Diversity objective: The diversity objective requires the models in the ensemble to produce dissimilar predictions for examples in the out of distribution dataset D out as depicted in Fig.

Original: 2b.
Revised: 2b.

Original: If { ˜ y i = f i (˜ x ; θ i ) } Ni =1 are the output probabilities produced by the ensemble for an OOD input ˜ x ∈
Revised: If { ˜ y i = f i (˜ x ; θ i ) } Ni =1 are the output probabilities produced by the ensemble for an OOD input ˜ x ∈

Original: D out , then the diversity objective requires the set of output vectors { ˜ y i } Ni =1 to be misaligned.
Revised: D out , then the diversity objective requires the set of output vectors { ˜ y i } Ni =1 to be misaligned.

Original: We use the coherence metric (Tropp, 2006) to measure the alignment of the output probability vectors.
Revised: We use the coherence metric (Tropp, 2006) to measure the alignment of the output probability vectors.

Original: Coherence of a set of vectors measures the maximum cosine similarity (i.e. cosine of the smallest angle) between all pairs of probability vectors in the set as shown in Fig.
Revised: Coherence of a set of vectors measures the maximum cosine similarity (CS) i.e. cosine of the smallest angle, between all pairs of probability vectors in the set as shown in Fig.

Original: Coherence can be computed as follows:
Revised: Coherence can be computed as follows:

Original: [Equation]
Revised: [Equation]

Original: In order to increase the misalignment of probability vectors, we need to reduce the value of coherence.
Revised: In order to increase the misalignment of probability vectors, we need to reduce the value of coherence.

Original: Unfortunately, the max function in Eqn 1 makes coherence non-smooth, which precludes the use of ﬁrst-order methods to optimize the value of coherence.
Revised: Unfortunately, the max function in Eqn 1 makes coherence non-smooth, which precludes the use of ﬁrst-order methods to optimize the value of coherence.

Original: To overcome this limitation, we use the LogSumExp function to get a smooth approximation of the max function that allows it to be used in ﬁrst-order optimization algorithms.
Revised: To overcome this limitation, we use the LogSumExp function to get a smooth approximation of the max function that allows it to be used in ﬁrst-order optimization algorithms.

Original: We term the resulting loss function as the diversity loss :
Revised: We term the resulting loss function as the diversity loss :

Original: [Equation]
Revised: [Equation]

Original:  exp( CS (˜ y a , ˜ y b )) (cid:33) .
Revised: N exp( CS (˜ y a , ˜ y b )) (cid:33) .

Original: [Figure]
Revised: [Figure]

Original: Models in the ensemble are jointly trained on the combination of accuracy and diversity objectives:
Revised: Models in the ensemble are jointly trained on the combination of accuracy and diversity objectives:

Original: [Equation]
Revised: [Equation]

Original: where ˆ y i =
Revised: where ˆ y i =

Original: f i ( x ) , ˜ y i =
Revised: f i ( x ) , ˜ y i =

Original: f i (˜ x ) .
Revised: f i (˜ x ) .

Original: Here λ  is a hyperparameter that dictates the ratio of importance between the diversity and accuracy objectives.
Revised: Here λ D is a hyperparameter that dictates the ratio of importance between the diversity and accuracy objectives.

Original: Note that the loss function described above requires access a dataset of labeled in-distribution examples D in as well as a dataset of out-of-distribution examples D out .
Revised: Note that the loss function described above requires access a dataset of labeled in-distribution examples D in as well as a dataset of out-of-distribution examples D out .

Original: Since we do not know the OOD samples that is used by the adversary a-priori, we make use of an auxiliary OOD dataset to train EDM.
Revised: Since we do not know the OOD samples that is used by the adversary a-priori, we make use of an auxiliary OOD dataset to train EDM.

Original: Similar use of auxiliary OOD datasets have been explored in anomaly detection literature (Hendrycks et al., 2018) to detect unseen anomalies.
Revised: Similar use of auxiliary OOD datasets have been explored in anomaly detection literature (Hendrycks et al., 2018) to detect unseen anomalies.

Original: Our experiments in Section 5.
Revised: Our experiments in Section 5.

Original: suggest that the diversity objective generalizes to the unseen OOD dataset used by the adversary.
Revised: suggest that the diversity objective generalizes to the unseen OOD dataset used by the adversary.

Original: 4. EDM I NFERENCE
Revised: 4. EDM I NFERENCE

Original: The objective of EDM is to produce predictions that are discontinuous in the input space.
Revised: The objective of EDM is to produce predictions that are discontinuous in the input space.

Original: EDM produces discontinuous predictions by selecting a model in the ensemble based on the input to service each request as shown in Fig. 4.
Revised: EDM produces discontinuous predictions by selecting a model in the ensemble based on the input to service each request as shown in Fig. 4.

Original: Given an input query x , EDM uses a hashing function H to compute a hash of the input, from which the index of the model that is used to service the query can be computed: index = 1 + ( H ( x )% N ) .
Revised: Given an input query x , EDM uses a hashing function H to compute a hash of the input, from which the index of the model that is used to service the query can be computed: index = 1 + ( H ( x )% N ) .

Original: The output prediction is given by y out = f index ( x ) .
Revised: The output prediction is given by y out = f index ( x ) .

Original: 4.2. C HOOSING THE H ASH F UNCTION
Revised: 4.2. C HOOSING THE H ASH F UNCTION

Original: To ensure that an adversary is unable to launch an adaptive attack to game the hash function, we select a hash function that satisﬁes the following properties:
Revised: To ensure that an adversary is unable to launch an adaptive attack to game the hash function, we select a hash function that satisﬁes the following properties:

Original: Security: The hash function needs to be kept secret from the adversary to avoid the adversary from computing the hash and ﬁguring out which model was used to service a given input.
Revised: Security: The hash function needs to be kept secret from the adversary to avoid the adversary from computing the hash and ﬁguring out which model was used to service a given input.

Original: Transformation invariance: While the hash function should map different inputs to different hash values, we want the hash value to be invariant to small transformations in the input images such as scaling, rotation, translation, warping, contrast adjustment, etc.
Revised: Transformation invariance: While the hash function should map different inputs to different hash values, we want the hash value to be invariant to small transformations in the input images such as scaling, rotation, translation, warping, contrast adjustment, etc.

Original: This invariance property is necessary to prevent an adversary from using slightly modiﬁed versions of the original image to change the hash value and query a different model.
Revised: This invariance property is necessary to prevent an adversary from using slightly modiﬁed versions of the original image to change the hash value and query a different model.

Original: Without the invariance property, the adversary can query all the models in the ensemble and get the average prediction for a given input, which is continuous in nature and allows for more effective model stealing.
Revised: Without the invariance property, the adversary can query all the models in the ensemble and get the average prediction for a given input, which is continuous in nature and allows for more effective model stealing.

Original: Cryptographic hash function would be unsuitable for our application as they lack transformation invariance.
Revised: Cryptographic hash function would be unsuitable for our application as they lack transformation invariance.

Original: Perceptual hashing (pHash) (Zauner, 2010) are a class of hash functions that ensure that perceptually similar inputs map to the same/similar hash values by using transformation invariant features (like discrete cosine transform), making them better suited for our application.
Revised: Perceptual hashing (pHash) (Zauner, 2010) are a class of hash functions that ensure that perceptually similar inputs map to the same/similar hash values by using transformation invariant features (like discrete cosine transform), making them better suited for our application.

Original: Empirically, we found existing pHash schemes to be less robust to various transformations of the inputs (discussed in Appendix A.2).
Revised: Empirically, we found existing pHash schemes to be less robust to various transformations of the inputs (discussed in Appendix A.2).

Original: To solve this problem, we propose a novel DNN-based perceptual hashing function that outputs the same hash value even with large changes to the input.
Revised: To solve this problem, we propose a novel DNN-based perceptual hashing function that outputs the same hash value even with large changes to the input.

Original: Our insight is that for any DNN that is trained on a classiﬁcation problem involving natural images, the output prediction of the DNN is typically invariant to transformations of the input that retain perceptual similarity.
Revised: Our insight is that for any DNN that is trained on a classiﬁcation problem involving natural images, the output prediction of the DNN is typically invariant to transformations of the input that retain perceptual similarity.

Original: Given a DNN f H that is trained on a secret M -class classiﬁcation problem, we can use the argmax predictions to compute a hash of the input: H ( x ) = argmax ( f H ( x )) .
Revised: Given a DNN f H that is trained on a secret M -class classiﬁcation problem, we can use the argmax predictions to compute a hash of the input: H ( x ) = argmax ( f H ( x )) .

Original: To ensure that the hash function uniformly maps inputs to different indices, we increase the output space of the hash function by subdividing each class into m sub-classes (achieving a total of M ∗ m possible outputs) using the following function: H ( x )
Revised: To ensure that the hash function uniformly maps inputs to different indices, we increase the output space of the hash function by subdividing each class into m sub-classes (achieving a total of M ∗ m possible outputs) using the following function: H ( x )

Original: = m · argmax ( f H ( x ))
Revised: = m · argmax ( f H ( x ))

Original: + (cid:106)
Revised: + (cid:106)

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: · m (cid:107) .
Revised: · m (cid:107) .

Original: We use a small value for m ( m = 2 ) in our experiments to ensure transformation invariance is preserved.
Revised: We use a small value for m ( m = 2 ) in our experiments to ensure transformation invariance is preserved.

Original: Additionally, a relatively small model can be used for f H to reduce computational overheads.
Revised: Additionally, a relatively small model can be used for f H to reduce computational overheads.

Original: The model architecture and the dataset used to train H are unknown to the adversary, making the output of the hash function hard to estimate.
Revised: The model architecture and the dataset used to train H are unknown to the adversary, making the output of the hash function hard to estimate.

Original: The security of the hash function can further be improved by training it on a custom dataset that is not available in the public domain.
Revised: The security of the hash function can further be improved by training it on a custom dataset that is not available in the public domain.

Original:  E XPERIMENTAL R ESULTS 5. S ETUP
Revised:  E XPERIMENTAL R ESULTS 5. S ETUP

Original: [Table]
Revised: [Table]

Original: Victim models and datasets: We evaluate our defense by comparing the clone accuracy achieved by various attacks on EDM and an undefended baseline model.
Revised: Victim models and datasets: We evaluate our defense by comparing the clone accuracy achieved by various attacks on EDM and an undefended baseline model.

Original: A lower clone accuracy implies better security.
Revised: A lower clone accuracy implies better security.

Original: We use DNNs trained on various image-classiﬁcation datasets listed in
Revised: We use DNNs trained on various image-classiﬁcation datasets listed in

Original: Table 3 as the target models for performing MS attacks.
Revised: Table 3 as the target models for performing MS attacks.

Original: For each dataset, we train two target models 1.
Revised: For each dataset, we train two target models 1.

Original: undefended baseline: A single model trained on D in using cross entropy loss.
Revised: undefended baseline: A single model trained on D in using cross entropy loss.

Original: EDM : An ensemble of models trained on the EDM loss function (Eqn. 3).
Revised: EDM : An ensemble of models trained on the EDM loss function (Eqn. 3).

Original: The dataset, model architecture and test accuracies obtained by the undefended target and EDM targets are shown in Table 1.
Revised: The dataset, model architecture and test accuracies obtained by the undefended target and EDM targets are shown in Table 1.

Original: Note that, despite the additional diversity objective , the accuracy of EDM is comparable to the undefended baseline model.
Revised: Note that, despite the additional diversity objective , the accuracy of EDM is comparable to the undefended baseline model.

Original: Attacks: We consider an adversary who is interested in stealing model functionality by training a high-accuracy clone-model on the predictions of the target model.
Revised: Attacks: We consider an adversary who is interested in stealing model functionality by training a high-accuracy clone-model on the predictions of the target model.

Original: We assume that the adversary does not have access to the entirety of the training dataset that was used to train the target model.
Revised: We assume that the adversary does not have access to the entirety of the training dataset that was used to train the target model.

Original: The number of queries that the adversary can make to the target model is limited by the query budget
Revised: The number of queries that the adversary can make to the target model is limited by the query budget

Original: Q (We use Q = 50000 for our evaluations).
Revised: Q (We use Q = 50000 for our evaluations).

Original: Additionally, we assume that the adversary is aware of the target model’s architecture and use a clone model with the same architecture as the target.
Revised: Additionally, we assume that the adversary is aware of the target model’s architecture and use a clone model with the same architecture as the target.

Original: We consider three recent MS attacks to evaluate our defense:
Revised: We consider three recent MS attacks to evaluate our defense:

Original: KnockoffNets
Revised: KnockoffNets

Original: (Orekondy et al., 2019a) performs model stealing by using a surrogate dataset to query the target model and training a clone model C on the predictions of the target model.
Revised: (Orekondy et al., 2019a) performs model stealing by using a surrogate dataset to query the target model and training a clone model C on the predictions of the target model.

Original: We use FashionMNIST, MNIST, CIFAR100, CIFAR10, and Indoor67 (Quattoni & Torralba, 2009) as the surrogate datasets for MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and Flowers-17 (Nilsback & Zisserman, 2006) models respectively.
Revised: We use FashionMNIST, MNIST, CIFAR100, CIFAR10, and Indoor67 (Quattoni & Torralba, 2009) as the surrogate datasets for MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and Flowers-17 (Nilsback & Zisserman, 2006) models respectively.

Original: The clone model is trained using an SGD optimizer with an initial learning rate of 0.1 for 50 epochs with cosine-annealing learning-rate schedule.2.
Revised: The clone model is trained using an SGD optimizer with an initial learning rate of 0.1 for 50 epochs with cosine-annealing learning-rate schedule.2.

Original: Jacobian Based Dataset Augmentation (JBDA) (Papernot et al., 2017) uses synthetic data to query the target model.
Revised: Jacobian Based Dataset Augmentation (JBDA) (Papernot et al., 2017) uses synthetic data to query the target model.

Original: JBDA starts with a small set of in-distribution “seed” examples { x i } .
Revised: JBDA starts with a small set of in-distribution “seed” examples { x i } .

Original: The attack iteratively trains the clone model by performing the following steps: (i) Obtain a labeled dataset { x i , y i } by querying the target model (ii) Train the clone model on the labeled dataset (iii) Augment the dataset with synthetic examples x (cid:48)
Revised: The attack iteratively trains the clone model by performing the following steps: (i) Obtain a labeled dataset { x i , y i } by querying the target model (ii) Train the clone model on the labeled dataset (iii) Augment the dataset with synthetic examples x (cid:48)

Original: i by perturbing the original input x i to change the prediction of the clone model by using the Jacobian
Revised: i by perturbing the original input x i to change the prediction of the clone model by using the Jacobian

Original: of the loss function:
Revised: of the loss function:

Original: x (cid:48)
Revised: x (cid:48)

Original: i
Revised: i

Original: = x i + βsign ( ∇ x L ( C ( x i ; θ c ) , y i )) .
Revised: = x i + βsign ( ∇ x L ( C ( x i ; θ c ) , y i )) .

Original: We use 6 rounds of data augmentation with the value of β set to 0.2 for the JBDA attack.
Revised: We use 6 rounds of data augmentation with the value of β set to 0.2 for the JBDA attack.

Original: The clone model is trained for 10 epochs in each round with the Adam optimizer with a learning rate of 0 .
Revised: The clone model is trained for 10 epochs in each round with the Adam optimizer with a learning rate of 0 .

Original: JBDA-TR is a variation of JBDA proposed by Juuti et al.
Revised: JBDA-TR is a variation of JBDA proposed by Juuti et al.

Revised: (2019) that produces targeted misclas-

Original: (2019) that produces targeted misclassiﬁcation from the clone model by using the gradient of the loss function to move towards a random target class ( y T ): x (cid:48)
Revised: siﬁcation from the clone model by using the gradient of the loss function to move towards a random target class ( y T ): x (cid:48)

Original: i =
Revised: i =

Original: x i − λsign ( ∇ x L ( C ( x i ; θ c ) , y T ))
Revised: x i − λsign ( ∇ x L ( C ( x i ; θ c ) , y T ))

Original: EDM Defense: For the EDM defense, we use an ensemble of 5 models that are jointly trained on the accuracy and diversity objectives by using the loss function in Eqn. 3.
Revised: EDM Defense: For the EDM defense, we use an ensemble of 5 models that are jointly trained on the accuracy and diversity objectives by using the loss function in Eqn. 3.

Original: The value of λ  was selected with the constraint of having the degradation in benign accuracy of the target modelbe
Revised: The value of λ D was selected with the constraint of having the degradation in benign accuracy of the target model

Original: [Table]
Revised: [Table]

Original:  less than 0 .
Revised: be less than 0 .

Original: EDM also requires an auxiliary OOD dataset for the diversity objective.
Revised: EDM also requires an auxiliary OOD dataset for the diversity objective.

Original: We use KMNIST (Clanuwat et al., 2018) for MNIST and FashionMNIST, Tiny-Images (Torralba et al., 2008) for CIFAR10 and CIFAR100, and ImageNet1k for Flowers-17 as the auxiliary OOD datasets.
Revised: We use KMNIST (Clanuwat et al., 2018) for MNIST and FashionMNIST, Tiny-Images (Torralba et al., 2008) for CIFAR10 and CIFAR100, and ImageNet1k for Flowers-17 as the auxiliary OOD datasets.

Original: For the DNN-based perceptual hashing we use M = 10 and m = 2 .
Revised: For the DNN-based perceptual hashing we use M = 10 and m = 2 .

Original: KMNIST is used as the secret classiﬁcation task for MNIST and FashionMNIST.
Revised: KMNIST is used as the secret classiﬁcation task for MNIST and FashionMNIST.

Original: SVHN is used for the rest of the datasets.
Revised: SVHN is used for the rest of the datasets.

Original: We use a simple 3-layer convolutional neural network trained on the secret classiﬁcation task as the hash function f H to keep the computational overheads low.
Revised: We use a simple 3-layer convolutional neural network trained on the secret classiﬁcation task as the hash function f H to keep the computational overheads low.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Our results comparing the efﬁcacy of various MS attacks targeting an undefended baseline and EDM targets for models trained on different classiﬁcation tasks are shown in Table 3.
Revised: Our results comparing the efﬁcacy of various MS attacks targeting an undefended baseline and EDM targets for models trained on different classiﬁcation tasks are shown in Table 3.

Original: In each case, we report the accuracy of the clone model along with the normalized clone accuracy computed with respect to the accuracy of the target model.
Revised: In each case, we report the accuracy of the clone model along with the normalized clone accuracy computed with respect to the accuracy of the target model.

Original: Our results show that the clone accuracies produced by the KnockoffNets attack are much lower for EDM compared to the baseline model.
Revised: Our results show that the clone accuracies produced by the KnockoffNets attack are much lower for EDM compared to the baseline model.

Original: With the JBDA and JBDA-TR atacks, the beneﬁts of EDM are less pronounced as these attacks use perturbed versions of in-distribution examples to query the target, which may not produce large discontinuities in the predictions.
Revised: With the JBDA and JBDA-TR attacks, the beneﬁts of EDM are less pronounced as these attacks use perturbed versions of in-distribution examples to query the target, which may not produce large discontinuities in the predictions.

Original: However, JBDA and JBDA-TR only work well on simpler datasets (MNIST,
Revised: However, JBDA and JBDA-TR only work well on simpler datasets (MNIST,

Original: FashionMNIST) and do not produce good accuracies on higher dimensional datasets.
Revised: FashionMNIST) and do not produce good accuracies on higher dimensional datasets.

Original: Overall, we ﬁnd that the discontinuous predictions produced by EDM can reduce the effectiveness of model stealing attacks for most attacks across multiple target models, offering up to a 39 .
Revised: Overall, we ﬁnd that the discontinuous predictions produced by EDM can reduce the effectiveness of model stealing attacks for most attacks across multiple target models, offering up to a 39 .

Original: 7% reduction in clone accuracy compared to an undefended baseline model for existing attacks.
Revised: 7% reduction in clone accuracy compared to an undefended baseline model for existing attacks.

Original: Furthermore, our defense is also effective against adaptive attacks that are tailored for our defense (see Appendix A.1).
Revised: Furthermore, our defense is also effective against adaptive attacks that are tailored for our defense (see Appendix A.1).

Original: [Table]
Revised: [Table]

Original: Comparison with prior work: We compare the clone accuracy obtained from our method with the adaptive misinformation (AM) defense (Kariyappa & Qureshi, 2020).
Revised: Comparison with prior work: We compare the clone accuracy obtained from our method with the adaptive misinformation (AM) defense (Kariyappa & Qureshi, 2020).

Original: AM aims to output incorrect predictions for the adversary’s queries by adaptively perturbing the predictions of the model for OOD inputs.
Revised: AM aims to output incorrect predictions for the adversary’s queries by adaptively perturbing the predictions of the model for OOD inputs.

Original: Unlike our work, AM trades off the benign accuracy for improved security.
Revised: Unlike our work, AM trades off the benign accuracy for improved security.

Original: We compare the clone accuracy obtained from KnockoffNets attack between
Revised: We compare the clone accuracy obtained from KnockoffNets attack between

Original: EDM and AM, with equivalent benign accuracy in Table 2.
Revised: EDM and AM, with equivalent benign accuracy in Table 2.

Original: Our results show that EDM is able to achieve up to 32 .
Revised: Our results show that EDM is able to outperform AM in most datasets with up to 32 .

Original: 16% lower clone accuracy compared to the AM defense.
Revised: 16% lower clone accuracy compared to the AM defense.

Revised: We see comparable performance with AM in case of Flowers-17 dataset.

Original: 5. A NALYSIS
Revised: 5. A NALYSIS

Original: Impact of Diversity Loss on Coherence: To quantify the efﬁcacy of the diversity loss in producing dissimilar predictions, we compare the CDF of coherence values obtained for the predictions of an
Revised: Impact of Diversity Loss on Coherence: To quantify the efﬁcacy of the diversity loss in producing dissimilar predictions, we compare the CDF of coherence values obtained for the predictions of an

Original: EDM model trained with diversity loss and a regular ensemble (with the same number of models) trained without diversity loss for the adversary’s queries used in the KnockoffNets attack.
Revised: EDM model trained with diversity loss and a regular ensemble (with the same number of models) trained without diversity loss for the adversary’s queries used in the KnockoffNets attack.

Original: From our results in Fig.
Revised: From our results in Fig.

Original: 5, we ﬁnd that the models in EDM produces much lower coherence values compared to a regular ensemble, which shows that  models in EDM produce highly dissimilar predictions on the adversary’s OOD queries. 1.0 Coherence 0.
Revised: 5, we ﬁnd that the models in EDM produce much lower coherence values compared to a regular ensemble. Additionally, these plots show that the models in EDM, which are trained to produce low coherence values on an auxiliary OOD dataset, can generalize to unseen OOD data used by the adversary to carry out model stealing

Original: F r a c t i o n o f D a t a MNIST 0.
Revised: [Figure]

Original: 1.0 Coherence 0. F r a c t i o n o f D a t a FashionMNIST 0. 1.0 Coherence 0.
Revised: [Figure]

Original: F r a c t i o n o f D a t a CIFAR0. 1.0 Coherence 0. F r a c t i o n o f D a t a
Revised: [Table]

Original: CIFAR0.
Revised: [Figure]

Original: 1.0 Coherence 0.
Revised: 15(0.31x)

Original: F r a c t i o n o f D a t a Flowers-17 Diverse Ens. Regular Ens.
Revised: Impact of Diversity Loss on

Original: [Table]
Revised: Clone

Original: Impact of Diversity Loss on Clone Accuracy: Models trained without the diversity objective can also be used in our proposal to produce discontinuous predictions.
Revised: Accuracy: Models trained without the diversity objective can also be used in our proposal to produce discontinuous predictions.

Original: However, such models produce correlated predictions, making them easier to steal.
Revised: However, such models produce correlated predictions, making them easier to steal.

Original: We show the importance of using the diversity objective by comparing the clone accuracies obtained by attacking two ensembles– 1.
Revised: We show the importance of using the diversity objective by comparing the clone accuracies obtained by attacking two ensembles– 1.

Original: Regular Ensemble (trained without diversity loss) 2.
Revised: Regular Ensemble (trained without diversity loss) 2.

Original: EDM (trained with diversity loss)– using the KnockoffNets attack.
Revised: EDM (trained with diversity loss)– using the KnockoffNets attack.

Original: Our results in Table 4 show that regular ensembles produce higher clone accuracies compared to the diverse ensemble, emphasising the importance of training with diversity loss.
Revised: Our results in Table 4 show that regular ensembles produce higher clone accuracies compared to the diverse ensemble, emphasising the importance of training with diversity loss.

Original:  D ISCUSSION ON A DAPTIVE A TTACKS
Revised:  D ISCUSSION ON A DAPTIVE A TTACKS

Original: An adversary who is aware of our defense can potentially modify the attack to improve the efﬁcacy of MS.
Revised: An adversary who is aware of our defense can potentially modify the attack to improve the efﬁcacy of MS.

Original: Fundamentally, an adversary can overcome our defense if the discontinuous predictions of
Revised: Fundamentally, an adversary can overcome our defense if the discontinuous predictions of

Original: EDM can be converted to a continuous function.
Revised: EDM can be converted to a continuous function.

Original: We discuss two adaptive attack strategies and explain why such adaptive attacks are hard to carry out.
Revised: We discuss two adaptive attack strategies and explain why such adaptive attacks are hard to carry out.

Original: Attacking a single model: Since the predictions of any individual model in EDM are continuous, the attacker can try to obtain the predictions of a single model and use these predictions to train the clone model.
Revised: Attacking a single model: Since the predictions of any individual model in EDM are continuous, the attacker can try to obtain the predictions of a single model and use these predictions to train the clone model.

Original: However, since the hash function is kept secret, it is not possible for the adversary to know which model was used to service a given request, making such attacks difﬁcult to execute.
Revised: However, since the hash function is kept secret, it is not possible for the adversary to know which model was used to service a given request, making such attacks difﬁcult to execute.

Original: Attacking the Ensemble: A second strategy is to somehow obtain the average predictions of all the models in the ensemble for a given input.
Revised: Attacking the Ensemble: A second strategy is to somehow obtain the average predictions of all the models in the ensemble for a given input.

Original: Since the average prediction is continuous, this can potentially be used to train a high-accuracy clone model.
Revised: Since the average prediction is continuous, this can potentially be used to train a high-accuracy clone model.

Original: To obtain the average predictions, the adversary needs to make small perturbations to the input that cause the input to get hashed to a different model.
Revised: To obtain the average predictions, the adversary needs to make small perturbations to the input that cause the input to get hashed to a different model.

Original: However, the use of perceptual hashing in our defense makes such perturbationbased averaging harder to perform.
Revised: However, the use of perceptual hashing in our defense makes such perturbationbased averaging harder to perform.

Original: Perceptual hashing ensures that small perturbations to the input result in the input mapping to the same hash.
Revised: Perceptual hashing ensures that small perturbations to the input result in the input mapping to the same hash.

Original: For large perturbations that do map to a different hash, the input no longer resembles the original input, making the averaging process meaningless.
Revised: For large perturbations that do map to a different hash, the input no longer resembles the original input, making the averaging process meaningless.

Original: Furthermore, we note that even if the hash function was known to the adversary, such an attack would be costly, requiring an O ( N ) increase in the query budget to carry out.
Revised: Furthermore, we note that even if the hash function was known to the adversary, such an attack would be costly, requiring an O ( N ) increase in the query budget to carry out.

Original: We refer the reader to 
Revised: Additionally, querying the target model with a set of perturbed inputs opens up the attack to detection using methods such as PRADA (Juuti et al., 2019), which can limit the applicability of such adaptive attacks.

Original:  Appendix A.1 for a detailed analysis of this adaptive attack strategy.
Revised: We refer the reader to Appendix A.1 for a detailed analysis of this adaptive attack strategy.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes Ensemble of Diverse Models , a novel type of model ensemble that is inherently robust to MS attacks due to the discontinuous nature of the model’s predictions.
Revised: This paper proposes Ensemble of Diverse Models , a novel type of model ensemble that is inherently robust to MS attacks due to the discontinuous nature of the model’s predictions.

Original: The models in
Revised: The models in

Original: EDM are trained to produce dissimilar predictions on OOD data by using the diversity loss function.
Revised: EDM are trained to produce dissimilar predictions on OOD data by using the diversity loss function.

Original: Our defense selects a different model from the ensemble to service different inputs using an inputbased hash function, producing predictions that are highly discontinuous in the input space for the adversary’s OOD inputs.
Revised: Our defense selects a different model from the ensemble to service different inputs using an inputbased hash function, producing predictions that are highly discontinuous in the input space for the adversary’s OOD inputs.

Original: Attacks that use these discontinuous predictions to train a clone model end up with a bad quality clone model, reducing the efﬁcacy of model stealing attacks.
Revised: Attacks that use these discontinuous predictions to train a clone model end up with a bad quality clone model, reducing the efﬁcacy of model stealing attacks.

Original: Through our empirical evaluations, we show that EDM can reduce the clone accuracy by up to 39 .
Revised: Through our empirical evaluations, we show that EDM can reduce the clone accuracy by up to 39 .

Original: 7% , with minimal degradation ( < 0 .
Revised: 7% , with minimal degradation ( < 0 .

Original: 5% ) in the accuracy of the target model on benign inputs.
Revised: 5% ) in the accuracy of the target model on benign inputs.

Original: Additionally,
Revised: Additionally,

Original: EDM has low computational overheads, making it a practical solution to defend against MS attacks.
Revised: EDM has low computational overheads, making it a practical solution to defend against MS attacks.

Original: A A PPENDIX A. E VALUATING ADAPTIVE ATTACKS AGAINST EDM
Revised: A A PPENDIX A. E VALUATING ADAPTIVE ATTACKS AGAINST EDM

Original: As discussed in section  one possible strategy to attack EDM is to obtain the average predictions of all the models in the ensemble and train the clone model on this average prediction.
Revised: As discussed in section 6, one possible strategy to attack EDM is to obtain the average predictions of all the models in the ensemble and train the clone model on this average prediction.

Original: To explain this adaptive attack, consider an input x for which we would like to compute the average prediction of all the models in the ensemble: ¯ y = 1 N (cid:80) Ni
Revised: To explain this adaptive attack, consider an input x for which we would like to compute the average prediction of all the models in the ensemble: ¯ y = 1 N (cid:80) Ni

Original: =

Revised: =

Original: f i ( x ) , where { f i } are the set of all the models in EDM.
Revised: f i ( x ) , where { f i } are the set of all the models in EDM.

Original: EDM uses an input-based hash: index ( x ) = 

Revised: EDM uses an input-based hash: index ( x ) =

Original: + H ( x )% N to select the model f index that is used to service x .
Revised: + H ( x )% N to select the model f index that is used to service x .

Original: If the hash function H was known, the adversary could modify the original input x to obtain a perturbed input x (cid:48)
Revised: If the hash function H was known, the adversary could modify the original input x to obtain a perturbed input x (cid:48)

Original: =
Revised: =

Original: x + δ that maps to a different hash value such that index ( x (cid:48) )
Revised: x + δ that maps to a different hash value such that index ( x (cid:48) )

Original: (cid:54) = index ( x ) .
Revised: (cid:54) = index ( x ) .

Original: By querying EDM with multiple perturbed inputs { x (cid:48) j } that produce different values for index ( x j ) , the adversary can obtain an approximation of the average predictions ¯ y = 1 K (cid:80) Kj =1 f EDM ( x (cid:48) j ) .
Revised: By querying EDM with multiple perturbed inputs { x (cid:48) j } that produce different values for index ( x j ) , the adversary can obtain an approximation of the average predictions ¯ y = 1 K (cid:80) K j =1 f EDM ( x (cid:48) j ) .

Original: This approximation estimates the true average prediction well as long as the size of the perturbation is small enough to not change the predictions of the model signiﬁcantly i.e. f i ( x (cid:48) j )
Revised: This approximation estimates the true average prediction well as long as the size of the perturbation is small enough to not change the predictions of the model signiﬁcantly i.e. f i ( x (cid:48) j )

Original: ≈
Revised: ≈

Original: f i ( x ) .
Revised: f i ( x ) .

Original: Once the average predictions are obtained, a labeled dataset { x, ¯ y } can be constructed using these average predictions to train the clone model.
Revised: Once the average predictions are obtained, a labeled dataset { x, ¯ y } can be constructed using these average predictions to train the clone model.

Original: [Table]
Revised: [Table]

Original: However, the hash function is unknown to the adversary, which gives rise to two challenges.
Revised: However, the hash function is unknown to the adversary, which gives rise to two challenges.

Original: First, evaluating the value of the index ( x ) is not possible and second, ﬁnding small perturbations that would change the value of index ( x (cid:48) j ) is hard.
Revised: First, evaluating the value of the index ( x ) is not possible and second, ﬁnding small perturbations that would change the value of index ( x (cid:48) j ) is hard.

Original: One approach to solving this problem is by averaging the predictions of the EDM over multiple inputs with large random perturbations 
Revised: One approach to solving this problem is by averaging the predictions of the EDM over multiple inputs with large random perturbations { x (cid:48)

Original: { x (cid:48) j } kj =1 such that ¯ y = 1 K (cid:80) Kj =

Revised: j } kj =1 such that ¯ y = 1 K (cid:80) Kj =

Original: f EDM ( x (cid:48) j ) .
Revised: f EDM ( x (cid:48) j ) .

Original: Large perturbations increases the likelihood that inputs map to different indices.
Revised: Large perturbations increases the likelihood that inputs map to different indices.

Original: Averaging over a a large set of such perturbed inputs improves the estimation of the average prediction.
Revised: Averaging over a a large set of such perturbed inputs improves the estimation of the average prediction.

Original: We evaluate the efﬁcacy of such an adaptive attack empirically.
Revised: We evaluate the efﬁcacy of such an adaptive attack empirically.

Original: We perform an adaptive version of KnockoffNets attack by using the average predictions over K = 5 perturbed inputs to train the clone model.
Revised: We perform an adaptive version of KnockoffNets attack by using the average predictions over K = 5 perturbed inputs to train the clone model.

Original: Note  that such an averaging attack requires a 5-fold increase in the query budget to carry out.
Revised: Note [Figure] that such an averaging attack requires a 5-fold increase in the query budget to carry out.

Original: We use a combination of random afﬁne transforms (rotation, translation, scaling, shear) and random changes to brightness, contrast, color and hue of an image to obtain the perturbed inputs { x (cid:48)
Revised: We use a combination of random afﬁne transforms (rotation, translation, scaling, shear) and random changes to brightness, contrast, color and hue of an image to obtain the perturbed inputs { x (cid:48)

Original: j } kj =1 from the original input x .
Revised: j } kj =1 from the original input x .

Original: Our results comparing the clone accuracy obtained from this adaptive attack compared to the non-adaptive attack on EDM and the single undefended model are shown in Table 5.
Revised: Our results comparing the clone accuracy obtained from this adaptive attack compared to the non-adaptive attack on EDM and the single undefended model are shown in Table 5.

Original: [Figure]

Original: Our results show that, while the clone accuracy improves with the adaptive attack compared to the non-adaptive attack on EDM, it still falls short of the clone accuracy obtained by attacking a single model.
Revised: Our results show that, while the clone accuracy improves with the adaptive attack compared to the non-adaptive attack on EDM, it still falls short of the clone accuracy obtained by attacking a single model.

Original: The improvement in clone accuracy can be attributed to the averaging of the decisions from multiple models.
Revised: The improvement in clone accuracy can be attributed to the averaging of the decisions from multiple models.

Original: There are two reasons why the adaptive attack still falls short of the accuracy with a single model: 1.
Revised: There are two reasons why the adaptive attack still falls short of the accuracy with a single model: 1.

Original: The predictions of the model could be different for the perturbed input i.e f i ( x (cid:48)
Revised: The predictions of the model could be different for the perturbed input i.e f i ( x (cid:48)

Original: j ) (cid:54)≈ f i ( x ) , which gives rise to an approximation error.
Revised: j ) (cid:54)≈ f i ( x ) , which gives rise to an approximation error.

Original: The DNN-based perceptual hashing algorithm ensures that most of the perturbed inputs map to the original index, preventing proper averaging across the predictions of all models.
Revised: The DNN-based perceptual hashing algorithm ensures that most of the perturbed inputs map to the original index, preventing proper averaging across the predictions of all models.

Original: A. E FFICACY OF DNN- BASED P H ASH
Revised: A. E FFICACY OF DNN- BASED P H ASH

Original: An ideal perceptual hash function would generate the same hash value for the original ( x ) and perturbed inputs ( x (cid:48) ).
Revised: An ideal perceptual hash function would generate the same hash value for the original ( x ) and perturbed inputs ( x (cid:48) ).

Original: To understand the efﬁcacy of our proposed pHash algorithm, we plot the sorted histogram of index values: index ( x ) = 1 + H ( x )% N , generated from our hashing function for 10000 randomly perturbed images.
Revised: To understand the efﬁcacy of our proposed pHash algorithm, we plot the sorted histogram of index values: index ( x ) = 1 + H ( x )% N , generated from our hashing function for 10000 randomly perturbed images.

Original: We aggregate the histogram of sorted indices for 1000 random examples in the training set of the CIFAR-10 dataset.
Revised: We aggregate the histogram of sorted indices for 1000 random examples in the training set of the CIFAR-10 dataset.

Original: We compare the performance of our proposed hash function against a Discrete Cosine Transform (DCT) based perceptual hash Zauner (2010) and an SHA-1 cryptographic hash.
Revised: We compare the performance of our proposed hash function against a Discrete Cosine Transform (DCT) based perceptual hash Zauner (2010) and an SHA-1 cryptographic hash.

Original: From our results in Fig.
Revised: From our results in Fig.

Original: 6, we can see that our DNN-based pHash produces a histogram that more skewed compared to the other hashing schemes.
Revised: 6, we can see that our DNN-based pHash produces a histogram that more skewed compared to the other hashing schemes.

Original: This implies that our hashing function has better transformation invariance property as it maps most of the perturbed inputs to the same index, making it suitable for perceptual hashing.
Revised: This implies that our hashing function has better transformation invariance property as it maps most of the perturbed inputs to the same index, making it suitable for perceptual hashing.

Original: A. C OMPATIBILITY OF EDM WITH OTHER D EFENSES
Revised: A. C OMPATIBILITY OF EDM WITH OTHER D EFENSES

Original: Our paper proposes to use the discontinuous predictions produced by an ensemble of diverse models to defend against MS attacks.
Revised: Our paper proposes to use the discontinuous predictions produced by an ensemble of diverse models to defend against MS attacks.

Original: Prior works on MS defense have mainly focused on perturbing the output predictions to prevent the adversary from reliably accessing the predictions of the model.
Revised: Prior works on MS defense have mainly focused on perturbing the output predictions to prevent the adversary from reliably accessing the predictions of the model.

Original: Since EDM does not modify the output predictions of the model, our defense can be combined with existing perturbation based defenses (Orekondy et al., 2019b; Lee et al., 2018; Kariyappa & Qureshi, 2020) to further improve the security of the model against MS attacks.
Revised: Since EDM does not modify the output predictions of the model, our defense can be combined with existing perturbation based defenses (Orekondy et al., 2019b; Lee et al., 2018; Kariyappa & Qureshi, 2020) to further improve the security of the model against MS attacks.

Original: A. C OMPUTATIONAL C OST
Revised: A. C OMPUTATIONAL C OST

Original: Compared to an undefended baseline model, EDM requires training multiple models, increasing the training cost by a factor of N .
Revised: Compared to an undefended baseline model, EDM requires training multiple models, increasing the training cost by a factor of N .

Original: However, during inference, only a single model from the ensemble is selected and evaluated, making the inference cost comparable with the baseline.
Revised: However, during inference, only a single model from the ensemble is selected and evaluated, making the inference cost comparable with the baseline.

Original: The additional compute associated with evaluating the hash function is negligible compared to the cost of evaluating the DNN used for the main classiﬁcation task.
Revised: The additional compute associated with evaluating the hash function is negligible compared to the cost of evaluating the DNN used for the main classiﬁcation task.

Original: For instance, the Conv3 model used in the hash function for CIFAR-10 only requires a ∼ 2 .
Revised: For instance, the Conv3 model used in the hash function for CIFAR-10 only requires a ∼ 2 .

Original: 5% increase in the number of ﬂops compared to evaluating a single Wres16-4 network.
Revised: 5% increase in the number of ﬂops compared to evaluating a single Wres16-4 network.

Original: EDM also requires a factor of N increase in memory costs  to store the ensemble.
Revised: EDM also requires a factor of N increase in memory costs [Table] to store the ensemble.

Original: However, in a datacenter setting with multiple machines running the same inference task, the memory cost can be avoided by spreading the models of the ensemble across multiple machines, with each input being routed to the selected model for inference.
Revised: However, in a datacenter setting with multiple machines running the same inference task, the memory cost can be avoided by spreading the models of the ensemble across multiple machines, with each input being routed to the selected model for inference.

Original: The negligible impact on benign accuracy and minimal computational overheads make EDM a practical solution to defend against MS attacks.
Revised: The negligible impact on benign accuracy and minimal computational overheads make EDM a practical solution to defend against MS attacks.

Revised: A. K NOCKOFF N ETS WITH A L ARGER C LONE M ODEL

Revised: In the main paper, we evaluated our defense with attacks that used clone models with the same architecture as the target model.

Revised: In this section, we report results for the KnockoffNets attack by using a larger model as the clone.

Revised: We compare the clone accuracy obtained by using the original and the larger clone models in Table 6.

Revised: Our results show that clone accuracy obtained with a larger model architecture is similar to the clone accuracy obtained with the original model architecture.

Revised: A. E VALUATIONS OF A TTACKS WITH A L

Revised: We used an attack query budget of 50K to evaluate our defense in the main paper.

Revised: In this section, we report the results with an increased query budget of 100K queries ( 2 × ).

Revised: Note that, for the

Revised: KnockoffNets attack, in addition to the query budget, the attacker is bound by the availability of surrogate data used for the attack.

Revised: Thus, for our evaluations, we stop querying the target model when we either run out of query budget or the surrogate data samples available to query the target model.

Revised: For the JBDA and JBDA-TR attacks, to utilize the larger query budget, we increase the number of augmentation rounds from 6 to 10.

Revised: Our results are reported in Table 7.

Revised: We observe only a slight improvement in clone accuracy for KnockoffNets as the attack is limited by the availability of surrogate data (70K was the size of the largest surrogate dataset used).

Revised: Even in the case of

Revised: JBDA and JBDA-TR attacks, the clone accuracy only improves marginally as the improvement in clone accuracy plateaus after 6 augmentation rounds.

Revised: For all the attacks, EDM continues to have a signiﬁcantly lower clone accuracy compared to the baseline, even with a larger query budget.

Revised: [Table]

Revised: A. E VALUATIONS WITH A DAPTIVE M ISINFORMATION D EFENSE

Revised: We compare the trade-off between defender accuracy (for benign examples) and clone accuracy of the Adaptive-Misinformation (AM) defense (Kariyappa & Qureshi, 2020) with our proposal.

Revised: AM uses an out of distribution (OOD) detector to identify the adversary’s

Revised: OOD data and services such queries with incorrect predictions.

Revised: By changing the value of the threshold used in the OOD detector,

Revised: [Figure]

Revised: this defense offers a trade-off between benign accuracy and clone accuracy.

Revised: In the case of EDM, one could expect a similar trade-off to exist by changing the value of the hyper-parameter λ D in Eqn

Revised: However, empirically we found that there is only a negligible loss in defender accuracy even with a large value of λ D .

Revised: This is because, the Diversity Loss term associated with λ D has a lower bound of 0.

Revised: Empirically, we found that by increasing the value of λ D , we can minimize the Diversity Loss term to 0 with a negligible loss in defender accuracy.

Revised: Thus, instead of a trade-off curve, we denote

Revised: EDM with a single trade-off point in our plot to compare against AM.

Revised: Our results in Fig.

Revised: 7 show that for the same defender accuracy, EDM yields a lower clone accuracy compared to AM.