Original: M ULTI - WAY E NCODING FOR R OBUSTNESS TO A DVERSARIAL A TTACKS
Revised: M ULTI - WAY E NCODING FOR R OBUSTNESS TO A DVERSARIAL A TTACKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep models are state-of-the-art for many computer vision tasks including image classiﬁcation and object detection.
Revised: Deep models are state-of-the-art for many computer vision tasks including image classiﬁcation and object detection.

Original: However, it has been shown that deep models are vulnerable to adversarial examples.
Revised: However, it has been shown that deep models are vulnerable to adversarial examples.

Original: We highlight how one-hot encoding directly contributes to this vulnerability and propose breaking away from this widely-used, but highly-vulnerable mapping.
Revised: We highlight how one-hot encoding directly contributes to this vulnerability and propose breaking away from this widely-used, but highly-vulnerable mapping.

Original: We demonstrate that by leveraging a different output encoding, multi-way encoding, we can make models more robust.
Revised: We demonstrate that by leveraging a different output encoding, multi-way encoding, we can make models more robust.

Original: Our approach makes it more difﬁcult for adversaries to ﬁnd useful gradients for generating adversarial attacks.
Revised: Our approach makes it more difﬁcult for adversaries to ﬁnd useful gradients for generating adversarial attacks.

Original: We present state-of-the-art robustness results for black-box, white-box attacks and achieve higher clean accuracy on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN when combined with adversarial training.
Revised: We present state-of-the-art robustness results for black-box, white-box attacks and achieve higher clean accuracy on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN when combined with adversarial training.

Original: The strength of our approach is also presented in the form of an attack for model watermarking, raising challenges in detecting stolen models
Revised: The strength of our approach is also presented in the form of an attack for model watermarking, raising challenges in detecting stolen models

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning models are vulnerable to adversarial examples [Szegedy et al.
Revised: Deep learning models are vulnerable to adversarial examples [Szegedy et al.

Original: Evidence shows that adversarial examples are transferable [Papernot et al.
Revised: Evidence shows that adversarial examples are transferable [Papernot et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: This weakness can be exploited even if the adversary does not know the target model under attack, posing severe concerns about the security of the models.
Revised: This weakness can be exploited even if the adversary does not know the target model under attack, posing severe concerns about the security of the models.

Original: This is because an adversary can use a substitute model for generating adversarial examples for the target model, also known as black-box attacks.
Revised: This is because an adversary can use a substitute model for generating adversarial examples for the target model, also known as black-box attacks.

Original: Black-box attacks such as Goodfellow et al.
Revised: Black-box attacks such as Goodfellow et al.

Original: (2014) rely on perturbing input by adding an amount dependent upon the gradient of the loss function with respect to the input of a substitute model.
Revised: (2014) rely on perturbing input by adding an amount dependent upon the gradient of the loss function with respect to the input of a substitute model.

Original: An example adversarial attack is x adv = x
Revised: An example adversarial attack is x adv = x

Original: + (cid:15)sign ( ∇ x Loss ( f ( x )) , where f ( x ) is the model used to generate the attack.
Revised: + (cid:15)sign ( ∇ x Loss ( f ( x )) , where f ( x ) is the model used to generate the attack.

Original: This added “noise” can fool a model although it may not be visually evident to a human.
Revised: This added “noise” can fool a model although it may not be visually evident to a human.

Original: The assumption of such gradient-based approaches is that the gradients with respect to the input, of the substitute and target models, are correlated.
Revised: The assumption of such gradient-based approaches is that the gradients with respect to the input, of the substitute and target models, are correlated.

Original: Our key observation is that the setup of conventional deep classiﬁcation frameworks aids in the correlation of such gradients.
Revised: Our key observation is that the setup of conventional deep classiﬁcation frameworks aids in the correlation of such gradients.

Original: Typically, a cross-entropy loss, a soft-max layer, and a one-hot vector encoding for a target label are used when training deep models.
Revised: Typically, a cross-entropy loss, a soft-max layer, and a one-hot vector encoding for a target label are used when training deep models.

Original: These conventions make a model more vulnerable to black-box attacks.
Revised: These conventions make a model more vulnerable to black-box attacks.

Original: This setting constrains the encoding length, and the number of possible non-zero gradient directions at the encoding layer.
Revised: This setting constrains the encoding length, and the number of possible non-zero gradient directions at the encoding layer.

Original: This makes it easier for an adversary to pick a harmful gradient direction and perform an attack.
Revised: This makes it easier for an adversary to pick a harmful gradient direction and perform an attack.

Original: We aim to increase the adversarial robustness of deep models.
Revised: We aim to increase the adversarial robustness of deep models.

Original: Our multi-way encoding representation relaxes the one-hot encoding to a real number encoding, and embeds the encoding in a space that has dimension higher than the number of classes.
Revised: Our multi-way encoding representation relaxes the one-hot encoding to a real number encoding, and embeds the encoding in a space that has dimension higher than the number of classes.

Original: These encoding methods lead to an increased number of possible gradient directions, as illustrated in Figure 1.
Revised: These encoding methods lead to an increased number of possible gradient directions, as illustrated in Figure 1.

Original: This makes it more difﬁcult for an adversary to pick a harmful direction that would cause a misclassiﬁcation of a correctly classiﬁed point, generating a targeted or untargeted attack.
Revised: This makes it more difﬁcult for an adversary to pick a harmful direction that would cause a misclassiﬁcation of a correctly classiﬁed point, generating a targeted or untargeted attack.

Original: Untargeted attacks aim to misclassify a point, while targeted attacks aim to misclassify a point to a speciﬁc target class.
Revised: Untargeted attacks aim to misclassify a point, while targeted attacks aim to misclassify a point to a speciﬁc target class.

Original: Multi-way encoding also helps improve a model’s robustness in cases where the adversary has full knowledge of the target model under attack: a white-box attack.
Revised: Multi-way encoding also helps improve a model’s robustness in cases where the adversary has full knowledge of the target model under attack: a white-box attack.

Original: The beneﬁts of multi-way encoding are demonstrated in experiments with four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN.
Revised: The beneﬁts of multi-way encoding are demonstrated in experiments with four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN.

Original: We also demonstrate the strength of our approach by introducing an attack for the recent model watermarking algorithm of Zhang et al.
Revised: We also demonstrate the strength of our approach by introducing an attack for the recent model watermarking algorithm of Zhang et al.

Original: (2018), which deliberately trains a model to misclassify
Revised: (2018), which deliberately trains a model to misclassify

Original: [Figure]
Revised: [Figure]

Original: certain watermarked images.
Revised: certain watermarked images.

Original: We interpret such watermarked images as adversarial examples.
Revised: We interpret such watermarked images as adversarial examples.

Original: We demonstrate that the multi-way encoding reduces the transferability of the watermarked images making it more challenging to detect stolen models.
Revised: We demonstrate that the multi-way encoding reduces the transferability of the watermarked images making it more challenging to detect stolen models.

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: We show that the traditional 1 ofK mapping is a source of vulnerability to adversarial gradients.
Revised: We show that the traditional 1 ofK mapping is a source of vulnerability to adversarial gradients.

Original: We propose a novel solution using multi-way encoding to alleviate the vulnerability caused by the 1 ofK mapping.
Revised: We propose a novel solution using multi-way encoding to alleviate the vulnerability caused by the 1 ofK mapping.

Original: We empirically show that the proposed approach improves model robustness against both black-box and white-box attacks.
Revised: We empirically show that the proposed approach improves model robustness against both black-box and white-box attacks.

Original: We also show how to apply our encoding framework in attacking the recently proposed model watermarking scheme of Zhang et al.
Revised: We also show how to apply our encoding framework in attacking the recently proposed model watermarking scheme of Zhang et al.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A wide range of work on adversarial attacks and defenses is presented in Akhtar & Mian (2018).
Revised: A wide range of work on adversarial attacks and defenses is presented in Akhtar & Mian (2018).

Original: We review recent attacks and defenses that are closely related to our work and present how alternate output encoding schemes have been utilized in deep classiﬁcation models.
Revised: We review recent attacks and defenses that are closely related to our work and present how alternate output encoding schemes have been utilized in deep classiﬁcation models.

Original: Attacks.
Revised: Attacks.

Original: Adversarial examples are crafted images for fooling a classiﬁer with small perturbations.
Revised: Adversarial examples are crafted images for fooling a classiﬁer with small perturbations.

Original: Recently, many different types of attacks have been proposed to craft adversarial examples.
Revised: Recently, many different types of attacks have been proposed to craft adversarial examples.

Original: We focus on gradient-based attacks such as [Goodfellow et al.
Revised: We focus on gradient-based attacks such as [Goodfellow et al.

Original: (2014); Kurakin et al.
Revised: (2014); Kurakin et al.

Original: ; Athalye et al.
Revised: ; Athalye et al.

Original: (2018)] which deploy the gradient of the loss with respect to the input.
Revised: (2018)] which deploy the gradient of the loss with respect to the input.

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2014) propose the Fast Gradient Sign Method (FGSM) which generates adversarial images by adding the sign of the input gradients scaled by (cid:15) , where the (cid:15) restricts (cid:96) ∞ of the perturbation.
Revised: (2014) propose the Fast Gradient Sign Method (FGSM) which generates adversarial images by adding the sign of the input gradients scaled by (cid:15) , where the (cid:15) restricts (cid:96) ∞ of the perturbation.

Original: Kurakin et al.
Revised: Kurakin et al.

Original: (2016) propose the Basic Iterative Method (BIM), which is an iterative version of FGSM and is also called Projected Gradient Descent (PGD).
Revised: (2016) propose the Basic Iterative Method (BIM), which is an iterative version of FGSM and is also called Projected Gradient Descent (PGD).

Original: Madry et al.
Revised: Madry et al.

Original: (2017) show that PGD with randomly chosen starting points within allowed perturbation can make an attack stronger.
Revised: (2017) show that PGD with randomly chosen starting points within allowed perturbation can make an attack stronger.

Original: Defenses.
Revised: Defenses.

Original: Most of the state-of-the-art adversarial defenses rely on gradient masking [Papernot et al.
Revised: Most of the state-of-the-art adversarial defenses rely on gradient masking [Papernot et al.

Original: (2017)] by designing a defense that makes it more difﬁcult for an adversary to ﬁnd useful gradients to generate adversarial examples.
Revised: (2017)] by designing a defense that makes it more difﬁcult for an adversary to ﬁnd useful gradients to generate adversarial examples.

Original: However, Athalye et al.
Revised: However, Athalye et al.

Original: (2018) show that works including Buckman et al.
Revised: (2018) show that works including Buckman et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: ; Dhillon et al.
Revised: ; Dhillon et al.

Original: ; Xie et al.
Revised: ; Xie et al.

Original: ; Song et al.
Revised: ; Song et al.

Original: (2017) which use obfuscated gradients, a special case of gradient masking, are vulnerable to the Backward Pass
Revised: (2017) which use obfuscated gradients, a special case of gradient masking, are vulnerable to the Backward Pass

Original: Differentiable Approximation Attack (BPDA).
Revised: Differentiable Approximation Attack (BPDA).

Original: Defenses that do not use obfuscated gradients, but rely on adversarial training instead [Madry et al.
Revised: Defenses that do not use obfuscated gradients, but rely on adversarial training instead [Madry et al.

Original: ; Kannan et al.
Revised: ; Kannan et al.

Original: (2018)], are robust to BPDA attack.
Revised: (2018)], are robust to BPDA attack.

Original: These methods are most similar to our approach because they do not rely on obfuscated gradients.
Revised: These methods are most similar to our approach because they do not rely on obfuscated gradients.

Original: However, Madry et al.
Revised: However, Madry et al.

Original: (2017) and Kannan et al.
Revised: (2017) and Kannan et al.

Original: (2018) use the conventional one-hot ( 1 ofK ) encoding for both source and target models, while we propose a higher dimensional multiway encoding that obstructs the adversarial gradient search.
Revised: (2018) use the conventional one-hot ( 1 ofK ) encoding for both source and target models, while we propose a higher dimensional multiway encoding that obstructs the adversarial gradient search.

Original: output encoding.
Revised: Output encoding.

Original: There have been attempts to use alternate output encodings, also known as targetencodings, for  classiﬁcation in deep models.
Revised: There have been attempts to use alternate output encodings, also known as target encodings, for image classiﬁcation in deep models.

Original: For example, Yang et al. (2015) and Rodr´ıguez et al.
Revised: For example, Yang et al. (2015) and Rodr´ıguez et al.

Original: (2018) use an output encoding that is based on Error-Correcting Output Codes (ECOC), for increased performance and faster convergence, but not for adversarial defense.
Revised: (2018) use an output encoding that is based on Error-Correcting Output Codes (ECOC), for increased performance and faster convergence, but not for adversarial defense.

Original: In contrast, we use an alternate output encoding scheme, multi-way encoding, to make models more robust to adversarial attacks.
Revised: In contrast, we use an alternate output encoding scheme, multi-way encoding, to make models more robust to adversarial attacks.

Original:  O UR A PPROACH
Revised:  O UR A PPROACH

Original: In this section we will explain our approach using the following notation: g ( x ) is the target model to be attacked, and f ( x ) is the substitute model used to generate a black-box attack for g ( x ) .
Revised: In this section we will explain our approach using the following notation: g ( x ) is the target model to be attacked, and f ( x ) is the substitute model used to generate a black-box attack for g ( x ) .

Original: In the case of a white-box attack, f ( x ) is g ( x ) .
Revised: In the case of a white-box attack, f ( x ) is g ( x ) .

Original: Canonical state-of-the-art attacks like FGSM and PGD are gradient-based methods.
Revised: Canonical state-of-the-art attacks like FGSM and PGD are gradient-based methods.

Original: Such approaches perturb an input x by an amount dependent upon sign ( ∇ x Loss ( f ( x ))) .
Revised: Such approaches perturb an input x by an amount dependent upon sign ( ∇ x Loss ( f ( x ))) .

Original: An adversarial example x adv is generated as follows:
Revised: An adversarial example x adv is generated as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) is the strength of the attack.
Revised: where (cid:15) is the strength of the attack.

Original: Therefore x adv would be a translated version of x , in a vicinity further away from that of the ground-truth class, and thus becomes more likely to be misclassiﬁed, resulting in a successful adversarial attack.
Revised: Therefore x adv would be a translated version of x , in a vicinity further away from that of the ground-truth class, and thus becomes more likely to be misclassiﬁed, resulting in a successful adversarial attack.

Original: If the attack is a targeted one, x could be deliberately moved towards some other speciﬁc target class.
Revised: If the attack is a targeted one, x could be deliberately moved towards some other speciﬁc target class.

Original: This is conventionally accomplished by using the adversarial class as the ground truth when back-propagating the loss, and subtracting the perturbation from the original input.
Revised: This is conventionally accomplished by using the adversarial class as the ground truth when back-propagating the loss, and subtracting the perturbation from the original input.

Original: The assumption being made in such approaches is:
Revised: The assumption being made in such approaches is:

Original: [Equation]
Revised: [Equation]

Original: We now present the most widely used setup for state-of-the-art deep classiﬁcation networks.
Revised: We now present the most widely used setup for state-of-the-art deep classiﬁcation networks.

Original: Let the output activation of neuron i in the ﬁnal encoding (fully-connected) layer be s i , where i = 1 , 2 , . . .
Revised: Let the output activation of neuron i in the ﬁnal encoding (fully-connected) layer be s i , where i = 1 , 2 , . . .

Original: , l and l is the encoding length.
Revised: , l and l is the encoding length.

Original: Then, the softmax prediction y i of s i , and the cross-entropy loss are:
Revised: Then, the softmax prediction y i of s i , and the cross-entropy loss are:

Original: [Equation]
Revised: [Equation]

Original: respectively, where k is the number of classes.
Revised: respectively, where k is the number of classes.

Original: The partial derivative of the loss with respect to the pre-softmax logit output is:
Revised: The partial derivative of the loss with respect to the pre-softmax logit output is:

Original: [Equation]
Revised: [Equation]

Original: Combined with the most widely used one-hot ( 1 ofK ) encoding scheme, the derivative in Eqn.
Revised: Combined with the most widely used one-hot ( 1 ofK ) encoding scheme, the derivative in Eqn.

Original: 4 makes the gradients of substitute and target models more correlated.
Revised: 4 makes the gradients of substitute and target models more correlated.

Original: We demonstrate this as follows:
Revised: We demonstrate this as follows:

Original: Given a ground-truth example belonging to class [1,0,...,0], non-zero gradients of neuron 1 of the encoding layer will always be negative, while all other neurons will always be positive since 0 < y i < 1 .
Revised: Given a ground-truth example belonging to class [1,0,...,0], non-zero gradients of neuron 1 of the encoding layer will always be negative, while all other neurons will always be positive since 0 < y i < 1 .

Original: So, regardless of the model architecture, the signs of the partial derivatives are determined by the category, and thus the gradients for that category only lie in a hyperoctant (see Fig.
Revised: So, regardless of the model architecture, the signs of the partial derivatives are determined by the category, and thus the gradients for that category only lie in a hyperoctant (see Fig.

Original: 1 for the 2D case).
Revised: 1 for the 2D case).

Original: This constraint causes strong correlation in gradients in the ﬁnal layer for different models using the 1 ofK encoding.
Revised: This constraint causes strong correlation in gradients in the ﬁnal layer for different models using the 1 ofK encoding.

Original: Our experiments suggest that this correlation can be carried all the way back to the input perturbations, making these models more vulnerable to attacks.
Revised: Our experiments suggest that this correlation can be carried all the way back to the input perturbations, making these models more vulnerable to attacks.

Original: In this work, we aim to make ∇ x Loss ( f ( x )) and ∇ x Loss ( g ( x )) less correlated.
Revised: In this work, we aim to make ∇ x Loss ( f ( x )) and ∇ x Loss ( g ( x )) less correlated.

Original: We do this by proposing multi-way encoding instead of the conventional 1 ofK encoding used by deep models for classiﬁcation.
Revised: We do this by proposing multi-way encoding instead of the conventional 1 ofK encoding used by deep models for classiﬁcation.

Original: Multi-way encoding signiﬁcantly reduces the correlation between the gradients of the substitute and target models, making it more challenging for an adversary to create an attack that is able to fool the classiﬁcation model.
Revised: Multi-way encoding signiﬁcantly reduces the correlation between the gradients of the substitute and target models, making it more challenging for an adversary to create an attack that is able to fool the classiﬁcation model.

Original: The multi-way encoding we propose in this work is Random Orthogonal ( RO ) output vector encoding generated via Gram-Schmidt orthogonalization.
Revised: The multi-way encoding we propose in this work is Random Orthogonal ( RO ) output vector encoding generated via Gram-Schmidt orthogonalization.

Original: Starting with a random matrix A
Revised: Starting with a random matrix A

Original: = [ a 1 | a 2 | . . .
Revised: = [ a 1 | a 2 | . . .

Original: | a n ] ∈ R k × l , the ﬁrst, second, and k th orthogonal vectors are computed as follows:
Revised: | a n ] ∈ R k × l , the ﬁrst, second, and k th orthogonal vectors are computed as follows:

Original: [Equation]
Revised: [Equation]

Original: For a classiﬁcation problem of k classes, we create a codebook C RO ∈ R k × l , where C i =
Revised: For a classiﬁcation problem of k classes, we create a codebook C RO ∈ R k × l , where C i =

Original: βe
Revised: βe

Original: i is a length l encoding for class i , and i ∈ 1 , . . .
Revised: i is a length l encoding for class i , and i ∈ 1 , . . .

Original: , k , and β is a scaling hyper-parameter dependent upon l .
Revised: , k , and β is a scaling hyper-parameter dependent upon l .

Original: A study on the selection of the length l is presented in the experiments section.
Revised: A study on the selection of the length l is presented in the experiments section.

Original: By breaking-away from the 1 ofK encoding, softmax and cross-entropy become ill-suited for the model architecture and training.
Revised: By breaking-away from the 1 ofK encoding, softmax and cross-entropy become ill-suited for the model architecture and training.

Original: Instead, we use the loss between the output of the encoding-layer and the RO ground-truth vector, Loss ( f ( x ) , t RO ) , where f ( x ) ∈ R l .
Revised: Instead, we use the loss between the output of the encoding-layer and the RO ground-truth vector, Loss ( f ( x ) , t RO ) , where f ( x ) ∈ R l .

Original: In our multi-way encoding setup, s and f ( x ) become equivalent.
Revised: In our multi-way encoding setup, s and f ( x ) become equivalent.

Original: Classiﬁcation is performed using arg min i Loss ( f ( x ) , t iRO ) .
Revised: Classiﬁcation is performed using arg min i Loss ( f ( x ) , t iRO ) .

Original: We use Mean Squared Error (MSE) Loss.
Revised: We use Mean Squared Error (MSE) Loss.

Original: Figure 1 illustrates how using a multi-way and longer encoding results in an increased number of possible gradient directions, reducing the probability of an adversary selecting a harmful direction that would cause misclassiﬁcation.
Revised: Figure 1 illustrates how using a multi-way and longer encoding results in an increased number of possible gradient directions, reducing the probability of an adversary selecting a harmful direction that would cause misclassiﬁcation.

Original: For simplicity we consider a binary classiﬁer.
Revised: For simplicity we consider a binary classiﬁer.

Original: Axis s i in each graph represents the output activation of neuron i in the output encoding layer, where i = 1 , . .
Revised: Axis s i in each graph represents the output activation of neuron i in the output encoding layer, where i = 1 , . .

Original: , l .
Revised: , l .

Original: The depicted points are correctly classiﬁed points for the green and blue classes.
Revised: The depicted points are correctly classiﬁed points for the green and blue classes.

Original: The arrows depict the sign of non-zero gradients
Revised: The arrows depict the sign of non-zero gradients

Original: ∂Loss∂s i .
Revised: ∂Loss∂s i .

Original: (a) Using a 1 ofK encoding and a softmax-cross entropy classiﬁer, there are only two directions for a point to move, a direct consequence of 1 ofK encoding together with Eqn. 4.
Revised: (a) Using a 1 ofK encoding and a softmax-cross entropy classiﬁer, there are only two directions for a point to move, a direct consequence of 1 ofK encoding together with Eqn. 4.

Original: Of these two directions, only one
Revised: Of these two directions, only one

Original: (b) Using 2-dimensional multi-way encoding, we get four possible non-zero gradient directions.
Revised: (b) Using 2-dimensional multi-way encoding, we get four possible non-zero gradient directions.

Original: The fraction of directions that now move a correctly classiﬁed point to the adversarial class is reduced.
Revised: The fraction of directions that now move a correctly classiﬁed point to the adversarial class is reduced.

Original: (c) Using a higher dimension multi-way encoding results in a less constrained gradient space compared to that of 1 ofK encoding.
Revised: (c) Using a higher dimension multi-way encoding results in a less constrained gradient space compared to that of 1 ofK encoding.

Original: In the case of attacks formulated following Eqn. 1, this results in 2 l possible gradient directions, rather than l in the case of 1 ofK encoding.
Revised: In the case of attacks formulated following Eqn. 1, this results in 2 l possible gradient directions, rather than l in the case of 1 ofK encoding.

Original: The fraction of gradients whose direction would move an input from the green ground-truth class to the blue class, or vice versa, decreases signiﬁcantly.
Revised: The fraction of gradients whose direction would move an input from the green ground-truth class to the blue class, or vice versa, decreases signiﬁcantly.

Original: In addition, multi-way encoding provides additional robustness by increasing the gradients’ dimensionality.
Revised: In addition, multi-way encoding provides additional robustness by increasing the gradients’ dimensionality.

Original: We also combine multi-way encoding with adversarial training for added robustness.
Revised: We also combine multi-way encoding with adversarial training for added robustness.

Original: We use the following formulation to solve the canonical min-max problem [Madry et al.
Revised: We use the following formulation to solve the canonical min-max problem [Madry et al.

Original: (2017), Kannan et al.
Revised: (2017), Kannan et al.

Original: (2018)] against PGD attacks:
Revised: (2018)] against PGD attacks:

Original: [Equation]
Revised: [Equation]

Original: where ˆ p data is the underlying training data distribution, ( x, y ) are the training points, and λ determines a weight of the loss on clean data together with the adversarial examples at train time.
Revised: where ˆ p data is the underlying training data distribution, ( x, y ) are the training points, and λ determines a weight of the loss on clean data together with the adversarial examples at train time.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct experiments on four commonly-used benchmark datasets: MNIST, CIFAR-10, CIFAR100, and SVHN.
Revised: We conduct experiments on four commonly-used benchmark datasets: MNIST, CIFAR-10, CIFAR100, and SVHN.

Original: MNIST [LeCun et al.
Revised: MNIST [LeCun et al.

Original: (1998)] is a dataset of handwritten digits.
Revised: (1998)] is a dataset of handwritten digits.

Original: It has a training set of 60K examples, and a test set of 10K examples.
Revised: It has a training set of 60K examples, and a test set of 10K examples.

Original: CIFAR-10 [Krizhevsky & Hinton (2009)] is a canonical benchmark for image classiﬁcation and retrieval, with 60K images from 10 classes.
Revised: CIFAR-10 [Krizhevsky & Hinton (2009)] is a canonical benchmark for image classiﬁcation and retrieval, with 60K images from 10 classes.

Original: The training set consists of 50K images, and the test set consists of 10K images.
Revised: The training set consists of 50K images, and the test set consists of 10K images.

Original: CIFAR- [Krizhevsky & Hinton (2009)] is similar to CIFAR-10 in format, but has 100 classes containingimages each.
Revised: CIFAR- [Krizhevsky & Hinton (2009)] is similar to CIFAR-10 in format, but has 100 classes containingimages each.

Original: Each class has 500 training images and 100 testing images.
Revised: Each class has 500 training images and 100 testing images.

Original: SVHN [Netzer et al.
Revised: SVHN [Netzer et al.

Original: (2011)] is an image dataset for recognizing street view house numbers obtained from Google Street
Revised: (2011)] is an image dataset for recognizing street view house numbers obtained from Google Street

Original: View images.
Revised: View images.

Original: The training set consists of 73K images, and the test set consists of 26K images.
Revised: The training set consists of 73K images, and the test set consists of 26K images.

Original: In this work we deﬁne a black-box attack as one where the adversary knows the architecture but not the weights, and not the output encoding used.
Revised: In this work we deﬁne a black-box attack as one where the adversary knows the architecture but not the weights, and not the output encoding used.

Original: This allows us to test the efﬁcacy of our proposed encoding when the adversary assumes the conventional 1 ofK encoding.
Revised: This allows us to test the efﬁcacy of our proposed encoding when the adversary assumes the conventional 1 ofK encoding.

Original: We deﬁne a white-box attack as one where the adversary knows full information about our model, including the encoding.
Revised: We deﬁne a white-box attack as one where the adversary knows full information about our model, including the encoding.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: is more vulnerable to attacks when f ( x ) uses the same encoding, hence the lower reported accuracy.
Revised: is more vulnerable to attacks when f ( x ) uses the same encoding, hence the lower reported accuracy.

Original: b) Even when the source and target models are the same and use the same encoding (*), i.e. white-box attacks,
Revised: b) Even when the source and target models are the same and use the same encoding (*), i.e. white-box attacks,

Original: RO encoding leads to better accuracy compared to 1 ofK .
Revised: RO encoding leads to better accuracy compared to 1 ofK .

Original: c)
Revised: c)

Original: In brackets is the Pearson correlation coefﬁcient of the gradients of g ( x ) and f ( x ) with respect to the input x .
Revised: In brackets is the Pearson correlation coefﬁcient of the gradients of g ( x ) and f ( x ) with respect to the input x .

Original: Gradients are less correlated when the source and target models use different encodings.
Revised: Gradients are less correlated when the source and target models use different encodings.

Original: In addition, if the same encoding is used in the source and target models, RO results in a lower correlation compared to 1 ofK .
Revised: In addition, if the same encoding is used in the source and target models, RO results in a lower correlation compared to 1 ofK .

Original: 4. D EFENSES WITHOUT A DVERSARIAL T RAINING
Revised: 4. D EFENSES WITHOUT A DVERSARIAL T RAINING

Original: In this section we analyze the case where neither the target nor substitute model undergoes adversarial training.
Revised: In this section we analyze the case where neither the target nor substitute model undergoes adversarial training.

Original: In all experiments we use RO encoding as the multi-way encoding with dimension 2000 determined by Table 1 and β = 1000 .
Revised: In all experiments we use RO encoding as the multi-way encoding with dimension 2000 determined by Table 1 and β = 1000 .

Original: We ﬁrst analyze using our multi-way encoding scheme in-depth using the MNIST dataset (4.1.1).
Revised: We ﬁrst analyze using our multi-way encoding scheme in-depth using the MNIST dataset (4.1.1).

Original: We then present results of comprehensive experiments on white-box and black-box attacks, targeted and untargeted, on the four benchmark datasets (4.1.2).
Revised: We then present results of comprehensive experiments on white-box and black-box attacks, targeted and untargeted, on the four benchmark datasets (4.1.2).

Original: 4.1. M ULTI - WAY E NCODING
Revised: 4.1. M ULTI - WAY E NCODING

Original: We conduct experiments to examine how multi-way output encodings can increase adversarial robustness.
Revised: We conduct experiments to examine how multi-way output encodings can increase adversarial robustness.

Original: We compare models trained on 1 ofK encodings ( A 1 ofK and C 1 ofK ) with models having the same architecture but trained on Random Orthogonal output encodings ( A RO and C RO ).
Revised: We compare models trained on 1 ofK encodings ( A 1 ofK and C 1 ofK ) with models having the same architecture but trained on Random Orthogonal output encodings ( A RO and C RO ).

Original: Models
Revised: Models

Original: A and C are LeNet-like CNNs and inherit their names from Tram`er et al.
Revised: A and C are LeNet-like CNNs and inherit their names from Tram`er et al.

Original: We use their architecture with dropout before fully-connected layers.
Revised: We use their architecture with dropout before fully-connected layers.

Original: We trained models A and C on MNIST with the momentum optimizer and an initial learning rate of 0.01, momentum = 0 .9 for 1 ofK models and momentum = 0 .
Revised: We trained models A and C on MNIST with the momentum optimizer and an initial learning rate of 0.01, momentum = 0 .

Original: 5 for RO models.

Original: We generated adversarial examples using FGSM with an attack strength (cid:15) = 0 .
Revised: We generated adversarial examples using FGSM with an attack strength (cid:15) = 0 .

Original: All models achieve ∼ 99% on the clean test set.
Revised: All models achieve ∼ 99% on the clean test set.

Original: It should be noted that substitute and target models are trained on clean data and do not undergo any form of adversarial training.
Revised: It should be noted that substitute and target models are trained on clean data and do not undergo any form of adversarial training.

Original: Table 2 presents the classiﬁcation accuracy (%) of target models under attack from various substitute models.
Revised: Table 2 presents the classiﬁcation accuracy (%) of target models under attack from various substitute models.

Original: Columns represent the substitute models used to generate adversarial examples and rows represent the target models to be tested on the adversarial examples.
Revised: Columns represent the substitute models used to generate adversarial examples and rows represent the target models to be tested on the adversarial examples.

Original: The diagonal represents whitebox attacks, i.e. generating attacks from the target model, and others represent black-box attacks.
Revised: The diagonal represents whitebox attacks, i.e. generating attacks from the target model, and others represent black-box attacks.

Original: Every cell in this table generates attacks from a substitute model f ( x ) for a target model g ( x ) .
Revised: Every cell in this table generates attacks from a substitute model f ( x ) for a target model g ( x ) .

Original: It is evident from the results of Table 2 that g ( x ) is more vulnerable to attacks when f ( x ) uses the same encoding, hence the lower reported accuracy.
Revised: It is evident from the results of Table 2 that g ( x ) is more vulnerable to attacks when f ( x ) uses the same encoding, hence the lower reported accuracy.

Original: This suggests that a model can be far more robust if the output encoding is hidden from an adversary.
Revised: This suggests that a model can be far more robust if the output encoding is hidden from an adversary.

Original: left the substitute model is C 1 ofK , therefore the attacks generated by this model will have a stronger negative effect on a model trained using 1 ofK , and a less negative effect on a that uses a different analogous argument goes

Original: It is also evident from the results of this experiment in Table 2 that even when the source and target models are the same, denoted by (*), i.e. white-box attacks, and use the same encoding, RO encoding leads to better accuracy, and therefore robustness to attack, compared to 1 ofK encoding. 
Revised: It is also evident from the results of this experiment in Table 2 that even when the source and target models are the same, denoted by (*), i.e. white-box attacks, and use the same encoding, RO encoding leads to better accuracy, and therefore robustness to attack, compared to 1 ofK encoding. We present further ablation studies in Appendix A.

Original: Finally, Table 2 also reports the Pearson correlation coefﬁcient of sign ( ∇ x Loss ( f ( x ))) and sign ( ∇ x Loss ( g ( x ))) used to perturb an input image x to create an adversarial example x adv as shown in Eqn.
Revised: Finally, Table 2 also reports the Pearson correlation coefﬁcient of sign ( ∇ x Loss ( f ( x ))) and sign ( ∇ x Loss ( g ( x ))) used to perturb an input image x to create an adversarial example x adv as shown in Eqn.

Original: These gradients are signiﬁcantly less correlated when the source and target models use different encodings.
Revised: These gradients are signiﬁcantly less correlated when the source and target models use different encodings.

Original: In addition, if the same encoding is used in the source and target models, RO results in a lower correlation compared to 1 ofK .
Revised: In addition, if the same encoding is used in the source and target models, RO results in a lower correlation compared to 1 ofK .

Revised: We report correlation coefﬁcients for all convolutional layers in Appendix B.

Original: Figure 2 presents black-box FGSM attacks of varying strengths for 1 ofK and RO encodings.
Revised: Figure 2 presents black-box FGSM attacks of varying strengths for 1 ofK and RO encodings.

Original: On the left is a 1 ofK substitute model used to generate attacks for a model originally trained using a 1 ofK encoding (green), and a model originally trained using a RO encoding (blue). On the right is a RO substitute model used to generate attacks for a model originally trained using a 1 ofK encoding (green), and a model originally trained using a RO encoding (blue).
Revised: On the left is a 1 ofK substitute model used to generate attacks for a model originally trained using a 1 ofK encoding (green), and a model originally trained using a RO encoding (blue). On the right is a RO substitute model used to generate attacks for a model originally trained using a 1 ofK encoding (green), and a model originally trained using a RO encoding (blue).

Original: This conﬁrms that using a different encoding for the source and target models makes the target model more robust to adversarial attacks; Maintaining a higher accuracy even as the strength of the attack increases.
Revised: This conﬁrms that using a different encoding for the source and target models makes the target model more robust to adversarial attacks; Maintaining a higher accuracy even as the strength of the attack increases.

Original: 4.1. B ENCHMARK R ESULTS
Revised: 4.1. B ENCHMARK R ESULTS

Original: We now demonstrate how using multi-way encoding helps increase robustness  compared to 1 ofK encoding for both targeted and untargeted attacks on the four benchmark datasets.
Revised: We now demonstrate how using multi-way encoding helps increase robustness in black-box attacks compared to 1 ofK encoding for both targeted and untargeted attacks on the four benchmark datasets.

Original: Targeted attacks are attacks where an adversary would like to misclassify an example to a speciﬁc incorrect class.
Revised: Targeted attacks are attacks where an adversary would like to misclassify an example to a speciﬁc incorrect class.

Original: Targeted attacks use the sign of the gradients of the loss on the target class and subtract the perturbation from the original input.
Revised: Targeted attacks use the sign of the gradients of the loss on the target class and subtract the perturbation from the original input.

Original: We use PGD attacks with a random start, and follow the PGD parameter conﬁguration of Madry et al.
Revised: We use PGD attacks with a random start, and follow the PGD parameter conﬁguration of Madry et al.

Original: (2017), Kannan et al.
Revised: (2017), Kannan et al.

Original: (2018), and Buckman et al.
Revised: (2018), and Buckman et al.

Original: Black-box attacks are generated from a substitute model independently trained using a 1 ofK encoding.
Revised: Black-box attacks are generated from a substitute model independently trained using a 1 ofK encoding.

Original: For MNIST and Cifar-10, we follow the experimental settings in Madry et al.
Revised: For MNIST and Cifar-10, we follow the experimental settings in Madry et al.

Original: ; for MNIST we use LeNet, for CIFAR-10 we use a ResNet [He et al.
Revised: ; for MNIST we use LeNet, for CIFAR-10 we use a ResNet [He et al.

Original: (2016)] of Madry et al.
Revised: (2016)] of Madry et al.

Original: For
Revised: For

Original: Cifar-100 and SVHN we use a WideResNet [Zagoruyko & Komodakis (2016)] of depth 28 and 16, respectively, with a width factor 4 and a dropout of 0.3 following [Buckman et al.
Revised: Cifar-100 and SVHN we use a WideResNet [Zagoruyko & Komodakis (2016)] of depth 28 and 16, respectively, with a width factor 4 and a dropout of 0.3 following [Buckman et al.

Original: We use the optimizer used by Madry et al.
Revised: We use the optimizer used by Madry et al.

Original: (2017) and Buckman et al.
Revised: (2017) and Buckman et al.

Original: The result of this experiment is presented in Table 3.
Revised: The result of this experiment is presented in Table 3.

Original: In the ﬁrst column we present the average classiﬁcation accuracy over all classes, and ﬁnd that models using RO encoding are consistently more resilient to black-box attacks compared to models using 1 ofK encoding.
Revised: In the ﬁrst column we present the average classiﬁcation accuracy over all classes for untargeted attacks, and ﬁnd that models using RO encoding are consistently more resilient to black-box attacks compared to models using 1 ofK encoding.

Original: In the second column we present the average targeted attack success rate over all classes for a white-box attack, a blackbox attack for a 1 ofK target model, and a black-box attack for a RO target model.
Revised: In the second column we present the average targeted attack success rate over all classes.

Original: RO consistently results in a signiﬁcantly lower attack success rate compared to 1 ofK for all four benchmark datasets.
Revised: RO consistently results in a signiﬁcantly lower attack success rate compared to 1 ofK for all four benchmark datasets.

Original: [Table]
Revised: [Table]

Original: 4. D EFENSES WITH A DVERSARIAL T RAINING
Revised: 4. D EFENSES WITH A DVERSARIAL T RAINING

Original: In this section we analyze the case where target models undergo adversarial training.
Revised: In this section we analyze the case where target models undergo adversarial training.

Original: This is when adversarial examples are injected in the training data of the target model, making it more difﬁcult for a substitute model to attack.
Revised: This is when adversarial examples are injected in the training data of the target model [Table] making it more difﬁcult for a substitute model to attack.

Original: We compare against state-of-the-art methods, which also use adversarial training.
Revised: We compare against state-of-the-art methods, which also use adversarial training.

Original: All black-box attacks in this section are generated from an independently trained copy of Madry et al.
Revised: All black-box attacks in this section are generated from an independently trained copy of Madry et al.

Original: (2017) (substitute model).
Revised: (2017) (substitute model).

Original: For adversarial training, we use a mix of clean and adversarial examples for MNIST, CIFAR-10, and CIFAR-100, and adversarial examples only for
Revised: For adversarial training, we use a mix of clean and adversarial examples for MNIST, CIFAR- [Table] and CIFAR- [Table]

Revised: and adversarial examples only for

Original: SVHN following the experimental setup used by Madry et al.
Revised: SVHN following the experimental setup used by Madry et al.

Original: (2017) and Buckman et al.
Revised: (2017) and Buckman et al.

Original: We compare against state-of-the-art defense methods Madry et al.
Revised: We compare against state-of-the-art defense methods Madry et al.

Original: (2017) and Kannan et al.
Revised: (2017) and Kannan et al.

Original: Both approaches use a LeNet for MNIST.
Revised: Both approaches use a LeNet for MNIST.

Original: Madry et al.
Revised: Madry et al.

Original: (2017) presents results for Cifar-10 on a
Revised: (2017) presents results for Cifar-10 on a

Original: WideResNet (He et al.
Revised: WideResNet (He et al.

Original: , we implement the approach of Kannan et al.
Revised: , we implement the approach of Kannan et al.

Original: (2018) on the same architecture and compare both against our approach.
Revised: (2018) on the same architecture and compare both against our approach.

Original: We implement Madry et al.
Revised: We implement Madry et al.

Original: (2017) and Kannan et al.
Revised: (2017) and Kannan et al.

Original: (2018) on WideResNet [Zagoruyko & Komodakis (2016)] following Buckman et al.
Revised: (2018) on WideResNet [Zagoruyko & Komodakis (2016)] following Buckman et al.

Original: and compare against our approach for CIFAR-100 and SVHN.
Revised: and compare against our approach for CIFAR-100 and SVHN.

Original: Table 4 presents the results of combining our multi-way encoding formulation with adversarial training.
Revised: Table 4 presents the results of combining our multi-way encoding formulation with adversarial training.

Original: We obtain state-of-the-art robustness for white-box and black-box attacks, while at the same time increasing the accuracy on the clean dataset for all four benchmark datasets.
Revised: We obtain state-of-the-art robustness for white-box and black-box attacks, while at the same time increasing the accuracy on the clean dataset for all four benchmark datasets.

Original: (*) indicates our replication of Kannan et al.
Revised: (*) indicates our replication of Kannan et al.

Original: (2018) using the experimental setting of Madry et al.
Revised: (2018) using the experimental setting of Madry et al.

Original: (2017) on MNIST, also used by Ours, that uses only 90% of the training set.
Revised: (2017) on MNIST, also used by ours, that uses only 90% of the training set.

Original:  A PPLICATION : A TTACKING M ODEL W ATERMARKING
Revised:  A PPLICATION : A TTACKING M ODEL W ATERMARKING

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) introduced an algorithm to detect whether a model is stolen or not.
Revised: (2018) introduced an algorithm to detect whether a model is stolen or not.

Original: They do so by adding a watermark to sample images of speciﬁc classes and deliberately training the model to misclassify these examples to other speciﬁc classes.
Revised: They do so by adding a watermark to sample images of speciﬁc classes and deliberately training the model to misclassify these examples to other speciﬁc classes.

Original: This work has demonstrated to be robust even when the model is ﬁne-tuned on a different training set.
Revised: This work has demonstrated to be robust even when the model is ﬁne-tuned on a different training set.

Original: We introduce an attack for this algorithm using our multi-way encoding, making it more challenging to detect whether a model is stolen or not.
Revised: We introduce an attack for this algorithm using our multi-way encoding, making it more challenging to detect whether a model is stolen or not.

Original: We do this by ﬁne-tuning the stolen model using multiway encoding, rather than the encoding used in pre-training the model.
Revised: We do this by ﬁne-tuning the stolen model using multiway encoding, rather than the encoding used in pre-training the model.

Original: We interpret the watermarked image used to deliberately cause a misclassiﬁcation as an adversarial example.
Revised: We interpret the watermarked image used to deliberately cause a misclassiﬁcation as an adversarial example.

Original: When the encoding of the substitute and target models is different, adversarial examples become less transferable.
Revised: When the encoding of the substitute and target models is different, adversarial examples become less transferable.

Original: [Table]

Original: [Table]

Original: a stolen model using RO encoding remarkably reduces the watermarking detection accuracy, and makes it comparable to the accuracy of models trained from scratch and do not use the stolen model.

Original: The accuracy of ﬁne-tuned models beneﬁts signiﬁcantly from the pre-trained weights of the stolen model.

Original: We follow the same CIFAR-10 experimental setup for detecting a stolen model as Zhang et al.
Revised: We follow the same CIFAR-10 experimental setup for detecting a stolen model as Zhang et al.

Original: (2018): We split the test set into two halves.
Revised: (2018): We split the test set into two halves.

Original: The ﬁrst half is used to ﬁne-tune pre-trained networks, and the second half is used to evaluate new models.
Revised: The ﬁrst half is used to ﬁne-tune pre-trained networks, and the second half is used to evaluate new models.

Original: When we ﬁne-tune the 1 ofK model, we reinitialize the last layer.
Revised: When we ﬁne-tune the 1 ofK model, we reinitialize the last layer.

Original: When we ﬁne-tune the RO model we replace the output encoding layer with our 2000-dimension fully-connected layer, drop the softmax, and freeze convolutional weights. 
Revised: When we ﬁne-tune the RO model we replace the output encoding layer with our 2000-dimension fully-connected layer, drop the softmax, and freeze convolutional weights. [Table] [Table]

Revised: remarkably reduces the watermarking detection accuracy, and makes it comparable to the accuracy of models trained from scratch and do not use the stolen model.

Revised: The accuracy of ﬁne-tuned models beneﬁts signiﬁcantly from the pre-trained weights of the stolen model.

Original: We present results on the CIFAR-10 dataset in Table 5.
Revised: We present results on the CIFAR-10 dataset in Table 5.

Original: When the ﬁne-tuning was performed using the 1 ofK encoding (also used in pre-training the model), watermarking detection is 87.8%, and when the ﬁne-tuning was performed using the multi-way RO encoding the watermarking detection is only 12.9%.
Revised: When the ﬁne-tuning was performed using the 1 ofK encoding (also used in pre-training the model), watermarking detection is 87.8%, and when the ﬁne-tuning was performed using the multi-way RO encoding the watermarking detection is only 12.9%.

Original: The watermark detection rate of the model ﬁne-tuned using RO is signiﬁcantly lower than that ﬁne-tuned using 1 ofK encoding, and is more comparable to models that are trained from scratch and do not use the stolen model (6.1% and 10.0%).
Revised: The watermark detection rate of the model ﬁne-tuned using RO is signiﬁcantly lower than that ﬁne-tuned using 1 ofK encoding, and is more comparable to models that are trained from scratch and do not use the stolen model (6.1% and 10.0%).

Original: The accuracy of the ﬁne-tuned models beneﬁts signiﬁcantly from the pre-trained weights of the stolen model.
Revised: The accuracy of the ﬁne-tuned models beneﬁts signiﬁcantly from the pre-trained weights of the stolen model.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: By relaxing the 1 ofK encoding to a real number encoding, together with increasing the encoding dimensionality, our multi-way encoding confounds an attacker by making it more difﬁcult to perturb an input in gradient direction(s) that would result in misclassiﬁcation of a correctly classiﬁed example, for a targeted or untargeted attack.
Revised: By relaxing the 1 ofK encoding to a real number encoding, together with increasing the encoding dimensionality, our multi-way encoding confounds an attacker by making it more difﬁcult to perturb an input in gradient direction(s) that would result in misclassiﬁcation of a correctly classiﬁed example, for a targeted or untargeted attack.

Original: We present state-of-the-art results on four benchmark datasets for both black and white-box attacks and achieve higher classiﬁcation accuracy on clean data.
Revised: We present state-of-the-art results on four benchmark datasets for both black and white-box attacks and achieve higher classiﬁcation accuracy on clean data.

Original: We also demonstrate the strength of our approach by introducing an attack for model watermarking, making it more difﬁcult to detect stolen models.
Revised: We also demonstrate the strength of our approach by introducing an attack for model watermarking, making it more difﬁcult to detect stolen models.

Revised: Appendix A A BLATION S TUDY ON ENCODINGS

Revised: We perform ablation studies to further investigate the effectiveness of our RO encoding.

Revised: We train the model used in Table 2 with two different combinations of encodings and loss functions.

Revised: A. A LTERNATIVE APPROACH A.1. RO softmax

Revised: We evaluate a network that uses RO encoding, a softmax layer, and cross-entropy loss.

Revised: We compute the probability of i th class as follows:

Revised: [Equation]

Revised: where s is the normalized ﬁnal layer representation, e i is the RO encoding vector (ground-truth vector) from the codebook, and n is the number of classes.

Revised: A.1. 1 ofK MSE

Revised: We also evaluate a network that uses mean-squared error (MSE) loss with the 1 ofK encoding.

Revised: A. E VALUATION

Revised: We generate FGSM attacks with (cid:15) = 0 .

Revised: 2 from substitute models A 1 ofK and C 1 ofK on MNIST to evaluate the models of Section A.1.1 and Section A.1.2.

Revised: We also measure a correlation coefﬁcient of the sign of the input gradients between target and substitute models as explained in Section 4.1.1.

Revised: Tables 6 and 7 demonstrate that RO , among the different target models, achieves the highest accuracy and the lowest input gradient correlation with the substitute model.

Revised: [Table]

Revised: [Table]

Revised: B C ORRELATION OF C ONVOLUTIONAL LAYERS

Revised: In response to the reviewer, we measure the correlation of gradients between all convolutional layers of the different models.

Revised: We ﬁrst compute the gradients of the loss with respect to intermediate features of Conv1 and Conv2.

Revised: Then, we compute the Pearson correlation coefﬁcient of the sign of the gradients with respect to such intermediate features between models.

Revised: For further comparison, we train models A (cid:48) 1 ofK and A (cid:48) RO , that are independently initialized from A 1 ofK and A RO .

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: In order to measure proper correlations, we average gradients of convolutional layers over channels similar to the way used to generate a gradient-based saliency map Selvaraju et al.

Revised: Otherwise, the order of convolutional ﬁlters affects the correlations and makes it hard to measure proper correlations between models.

Revised: In this sense, the correlations at FC1 (before the last layer) may not give meaningful information since neurons in the FC layer do not have a strict ordering.

Revised: In Table 8 and 9, we ﬁnd that the correlations of Conv1 and Conv2 between 1ofK models are much higher than those of RO models.

Revised: In addition, even though RO models used the same output encoding, they are not highly correlated.

Revised: Table 10 shows that the correlations between RO and 1ofK are also low.