Original: A N A LGORITHM FOR O UT -O F -D ISTRIBUTION A TTACK TO N EURAL N ETWORK E NCODER
Revised: A N A LGORITHM FOR O UT -O F -D ISTRIBUTION A TTACK TO N EURAL N ETWORK E NCODER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks (DNNs), especially convolutional neural networks, have achieved superior performance on image classiﬁcation tasks.
Revised: Deep neural networks (DNNs), especially convolutional neural networks, have achieved superior performance on image classiﬁcation tasks.

Original: However, such performance is only guaranteed if the input to a trained model is similar to the training samples, i.e., the input follows the probability distribution of the training set.
Revised: However, such performance is only guaranteed if the input to a trained model is similar to the training samples, i.e., the input follows the probability distribution of the training set.

Original: Out-Of-Distribution (OOD) samples do not follow the distribution of training set, and therefore the predicted class labels on OOD samples become meaningless.
Revised: Out-Of-Distribution (OOD) samples do not follow the distribution of training set, and therefore the predicted class labels on OOD samples become meaningless.

Original: Classiﬁcation-based methods have been proposed for OOD detection; however, in this study we show that this type of method has no theoretical guarantee and is practically breakable by our OOD Attack algorithm because of dimensionality reduction in the DNN models.
Revised: Classiﬁcation-based methods have been proposed for OOD detection; however, in this study we show that this type of method has no theoretical guarantee and is practically breakable by our OOD Attack algorithm because of dimensionality reduction in the DNN models.

Original: We also show that Glow likelihood-based OOD detection is breakable as well.
Revised: We also show that Glow likelihood-based OOD detection is breakable as well.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs), especially convolutional neural networks (CNNs), have become the method of choice for image classiﬁcation.
Revised: Deep neural networks (DNNs), especially convolutional neural networks (CNNs), have become the method of choice for image classiﬁcation.

Original: Under the i.i.d.
Revised: Under the i.i.d.

Original: (independent and identically distributed) assumption, a high-performance DNN model can correctly-classify an input sample as long as the sample is “generated” from the distribution of training data.
Revised: (independent and identically distributed) assumption, a high-performance DNN model can correctly-classify an input sample as long as the sample is “generated” from the distribution of training data.

Original: If an input sample is not from this distribution, which is called Out-Of-Distribution (OOD), then the predicted class label from the model is meaningless.
Revised: If an input sample is not from this distribution, which is called Out-Of-Distribution (OOD), then the predicted class label from the model is meaningless.

Original: It would be great if the model has the ability to distinguish OOD samples from in-distribution samples.
Revised: It would be great if the model has the ability to distinguish OOD samples from in-distribution samples.

Original: OOD detection is needed especially when applying DNN models in life-critical applications, e.g., vision-based self-driving or image-based medical diagnosis.
Revised: OOD detection is needed especially when applying DNN models in life-critical applications, e.g., vision-based self-driving or image-based medical diagnosis.

Original: It was shown by Nguyen et al.
Revised: It was shown by Nguyen et al.

Original: (Nguyen et al., 2015) that DNN classiﬁers can be easily fooled by OOD data, and an evolutionarily algorithm was used to generate OOD samples such that
Revised: (Nguyen et al., 2015) that DNN classiﬁers can be easily fooled by OOD data, and an evolutionarily algorithm was used to generate OOD samples such that

Original: DNN classiﬁers had high output conﬁdence on these samples.
Revised: DNN classiﬁers had high output conﬁdence on these samples.

Original: Since then, many methods have been proposed for OOD detection using classiﬁers or encoders (Hendrycks & Gimpel, 2017; Hendrycks et al., 2019; Liang et al., 2018; Lee et al., 2018b;a; Alemi et al., 2018; Hendrycks & Gimpel, 2017).
Revised: Since then, many methods have been proposed for OOD detection using classiﬁers or encoders (Hendrycks & Gimpel, 2017; Hendrycks et al., 2019; Liang et al., 2018; Lee et al., 2018b;a; Alemi et al., 2018; Hendrycks & Gimpel, 2017).

Original: For instance, Hendrycks et al.
Revised: For instance, Hendrycks et al.

Original: (Hendrycks & Gimpel, 2017) show that a classiﬁer’s prediction probabilities of OOD examples tend to be more uniform, and therefore the maximum predicted class probability from the softmax layer was used for OOD detection.
Revised: (Hendrycks & Gimpel, 2017) show that a classiﬁer’s prediction probabilities of OOD examples tend to be more uniform, and therefore the maximum predicted class probability from the softmax layer was used for OOD detection.

Original: Regardless of the details of these methods, every method needs a classiﬁer or an encoder, which takes an image x as input and compresses it into a vector z in the laten space; after some further transform, z is converted to an
Revised: Regardless of the details of these methods, every method needs a classiﬁer or an encoder, which takes an image x as input and compresses it into a vector z in the laten space; after some further transform, z is converted to an

Original: OOD detection score τ .
Revised: OOD detection score τ .

Original: This computing process can be expressed as: z
Revised: This computing process can be expressed as: z

Original: = f ( x ) and τ = d ( z ) .
Revised: = f ( x ) and τ = d ( z ) .

Original: To perform OOD detection, a detection threshold needs to be speciﬁed, and then x is OOD if τ is smaller/larger than the threshold.
Revised: To perform OOD detection, a detection threshold needs to be speciﬁed, and then x is OOD if τ is smaller/larger than the threshold.

Original: For the evaluation of OOD detection methods, (Hendrycks & Gimpel, 2017), an OOD detector is usually trained on a dataset (e.g. Fashion-MNIST as indistribution) and then it is tested on another dataset (e.g. MNIST as OOD).
Revised: For the evaluation of OOD detection methods, (Hendrycks & Gimpel, 2017), an OOD detector is usually trained on a dataset (e.g. Fashion-MNIST as indistribution) and then it is tested on another dataset (e.g. MNIST as OOD).

Original: As will be shown in this study, the above mentioned classiﬁcation-based OOD detection methods are practically breakable.
Revised: As will be shown in this study, the above mentioned classiﬁcation-based OOD detection methods are practically breakable.

Original: As an example (more details in Section 3), we used the Resnet-18 model (He et al., 2016)
Revised: As an example (more details in Section 3), we used the Resnet-18 model (He et al., 2016)

Original: pre-trained on the ImageNet dataset.
Revised: pre-trained on the ImageNet dataset.

Original: Let x in denote a 224×224×3 image (indistribution sample) in ImageNet and x out denote an OOD sample which could be any kinds of images (even random noises) not belonging to any category in ImageNet.
Revised: Let x in denote a 224×224×3 image (indistribution sample) in ImageNet and x out denote an OOD sample which could be any kinds of images (even random noises) not belonging to any category in ImageNet.

Original: Let z denote the 512dimensional feature vector in Resnet-18, which is the input to the last fully-connected linear layer before the softmax operation.
Revised: Let z denote the 512dimensional feature vector in Resnet-18, which is the input to the last fully-connected linear layer before the softmax operation.

Original: Thus, we have z in = f
Revised: Thus, we have z in = f

Original: ( x in ) and z out = f
Revised: ( x in ) and z out = f

Original: ( x out ) .
Revised: ( x out ) .

Original: In Fig.
Revised: In Fig.

Original: 1, x in
Revised: 1, x in

Original: [Figure]
Revised: [Figure]

Original: is the image of Santa Claus, and x out could be a chest x-ray image or a random-noise image, and “surprisingly”, z out ∼ =
Revised: is the image of Santa Claus, and x out could be a chest x-ray image or a random-noise image, and “surprisingly”, z out ∼ =

Original: z in which renders OOD detection score to be useless: d ( z out ) ∼ = d
Revised: z in which renders OOD detection score to be useless: d ( z out ) ∼ = d

Original: ( z in ) .
Revised: ( z in ) .

Original: In Section 2, we will introduce an algorithm to generate OOD samples such that z out ∼ =
Revised: In Section 2, we will introduce an algorithm to generate OOD samples such that z out ∼ =

Original: z in .
Revised: z in .

Original: In
Revised: In

Original: Section 3, we will show the evaluation results on publicly available datasets, including ImageNet subset, GTSRB, OCT, and COVID-19 CT.
Revised: Section 3, we will show the evaluation results on publicly available datasets, including ImageNet subset, GTSRB, OCT, and COVID-19 CT.

Original: Since some generative models (e.g. Glow (Kingma &
Revised: Since some generative models (e.g. Glow (Kingma &

Original: Dhariwal, 2018)) can approximate the distribution of training samples (i.e. p ( x in ) ), likelihood-based generative models were utilized for OOD detection (Nalisnick et al., 2019).
Revised: Dhariwal, 2018)) can approximate the distribution of training samples (i.e. p ( x in ) ), likelihood-based generative models were utilized for OOD detection (Nalisnick et al., 2019).

Original: It has been shown that likelihoods derived from generative models may not distinguish between OOD and training samples (Nalisnick et al., 2019;
Revised: It has been shown that likelihoods derived from generative models may not distinguish between OOD and training samples (Nalisnick et al., 2019;

Original: Ren et al., 2019; Choi et al., 2018), and a ﬁx to the problem could be using likelihood ratio instead of raw likelihood score (Serr ` a et al., 2019).
Revised: Ren et al., 2019; Choi et al., 2018), and a ﬁx to the problem could be using likelihood ratio instead of raw likelihood score (Serr ` a et al., 2019).

Original: Although not the main focus of this study, we will show that the OOD sample’s likelihood score from the Glow model (Kingma & Dhariwal, 2018; Serr`a et al., 2019) can be arbitrarily manipulated by our algorithm (Section 2.1)
Revised: Although not the main focus of this study, we will show that the OOD sample’s likelihood score from the Glow model (Kingma & Dhariwal, 2018; Serr`a et al., 2019) can be arbitrarily manipulated by our algorithm (Section 2.1)

Original: such that the output probability p ( x in ) ∼ = p ( x out ) , which further diminishes the effectiveness of any
Revised: such that the output probability p ( x in ) ∼ = p ( x out ) , which further diminishes the effectiveness of any

Original: Glow likelihood-based detection methods.
Revised: Glow likelihood-based detection methods.

Original:  M ETHODOLOGY 2. OOD ATTACK ON DNN E NCODER
Revised:  M ETHODOLOGY 2. OOD ATTACK ON DNN E NCODER

Original: We introduce an algorithm to perform OOD attack on a DNN encoder z
Revised: We introduce an algorithm to perform OOD attack on a DNN encoder z

Original: = f ( x ) which takes an image x as input and transforms it into a feature vector z in a latent space.
Revised: = f ( x ) which takes an image x as input and transforms it into a feature vector z in a latent space.

Original: Preprocessing on x can be considered as the very ﬁrst layer inside of the model f ( x ) .
Revised: Preprocessing on x can be considered as the very ﬁrst layer inside of the model f ( x ) .

Original: The algorithm needs a weak assumption that f ( x ) is sub-differentiable.
Revised: The algorithm needs a weak assumption that f ( x ) is sub-differentiable.

Original: A CNN classiﬁer can be considered a composition of a feature encoder z
Revised: A CNN classiﬁer can be considered a composition of a feature encoder z

Original: = f ( x ) and a feature classiﬁer p = g ( z ) where p is the softmax probability distribution over multiple classes.
Revised: = f ( x ) and a feature classiﬁer p = g ( z ) where p is the softmax probability distribution over multiple classes.

Original: Let’s consider an in-distribution sample x in and an OOD sample x (cid:48) out , and apply the model:
Revised: Let’s consider an in-distribution sample x in and an OOD sample x (cid:48) out , and apply the model:

Original: z in = f
Revised: z in = f

Original: ( x in ) and z (cid:48) out = f
Revised: ( x in ) and z (cid:48) out = f

Original: ( x (cid:48) out ) .
Revised: ( x (cid:48) out ) .

Original: Usually, z (cid:48) out (cid:54) =
Revised: Usually, z (cid:48) out (cid:54) =

Original: z in .
Revised: z in .

Original: However, if we add a relatively small amount of noise δ to x (cid:48) out , then it could be possible that f ( x (cid:48) out + δ ) =
Revised: However, if we add a relatively small amount of noise δ to x (cid:48) out , then it could be possible that f ( x (cid:48) out + δ ) =

Original: z in and x (cid:48) out + δ is still OOD.
Revised: z in and x (cid:48) out + δ is still OOD.

Original: This idea is realized in Algorithm 1, OOD Attack on DNN Encoder.
Revised: This idea is realized in Algorithm 1, OOD Attack on DNN Encoder.

Original: The clip operation in Algorithm 1 is very important: it will limit the difference between x out and x (cid:48) out so that x out may be OOD.
Revised: The clip operation in Algorithm 1 is very important: it will limit the difference between x out and x (cid:48) out so that x out may be OOD.

Original: The algorithm is inspired by the method called projected gradient descent (PGD) (Kurakin et al., 2016; Madry et al., 2018) which is used for adversarial attacks.
Revised: The algorithm is inspired by the method called projected gradient descent (PGD) (Kurakin et al., 2016; Madry et al., 2018) which is used for adversarial attacks.

Original: We note that the term “adversarial attack” usually refers to adding a small perturbation to a clean sample x in a dataset such that a classiﬁer will incorrectly-classify the noisy sample while being
Revised: We note that the term “adversarial attack” usually refers to adding a small perturbation to a clean sample x in a dataset such that a classiﬁer will incorrectly-classify the noisy sample while being

Original: OOD in the dataset neural network feature encoder perturbation measured by
Revised: OOD in the dataset neural network feature encoder perturbation measured by

Original: Lp of learning optimizer
Revised: Lp of learning optimizer

Original: the
Revised: the

Original: .g
Revised: .g

Original: 0 to 1)
Revised: 0 to 1)

Original: L inf is
Revised: L inf is

Original: ( J the
Revised: ( J the

Original: L2 norm is
Revised: L2 norm is

Original: ( J (cid:48) a normalizes J by its L2 norm implementation able to correctly-classify the original clean sample x .
Revised: ( J (cid:48) a normalizes J by its L2 norm implementation able to correctly-classify the original clean sample x .

Original: Thus, OOD attack and adversarial attack are completely different things.
Revised: Thus, OOD attack and adversarial attack are completely different things.

Original: In practice, the Algorithm 1 can be repeated many times to ﬁnd the best solution.
Revised: In practice, the Algorithm 1 can be repeated many times to ﬁnd the best solution.

Original: Random initialization is performed in line-1 and line-2 of the algorithm process.
Revised: Random initialization is performed in line-1 and line-2 of the algorithm process.

Original: By adding initial random noise ξ to x (cid:48) out , the algorithm will have a better chance to avoid local minima caused by a bad initialization.
Revised: By adding initial random noise ξ to x (cid:48) out , the algorithm will have a better chance to avoid local minima caused by a bad initialization.

Original: 2. D IMENSIONALITY R EDUCTION AND OOD A TTACK
Revised: 2. D IMENSIONALITY R EDUCTION AND OOD A TTACK

Original: Recall that in a classiﬁcation-based OOD Detection approach, a DNN encoder transforms the input to a feature vector, i.e., z = f ( x ) , and an OOD detection score is computed by another transform on z , i.e., and τ = d ( z ) .
Revised: Recall that in a classiﬁcation-based OOD Detection approach, a DNN encoder transforms the input to a feature vector, i.e., z = f ( x ) , and an OOD detection score is computed by another transform on z , i.e., and τ = d ( z ) .

Original: If z out ∼ =
Revised: If z out ∼ =

Original: z in , then d ( z out ) ∼ =
Revised: z in , then d ( z out ) ∼ =

Original: d ( z in ) which breaks the OOD detector regardless of the transform d .
Revised: d ( z in ) which breaks the OOD detector regardless of the transform d .

Original: Usually, a DNN encoder makes dimensionality reduction: the dimension of z is signiﬁcantly smaller than the dimension of x .
Revised: Usually, a DNN encoder makes dimensionality reduction: the dimension of z is signiﬁcantly smaller than the dimension of x .

Original: In the example shown in Fig.
Revised: In the example shown in Fig.

Original: 1, z is a 512-dimensional feature vector ( dim ( z ) = 512) in Resnet-18, and the dimension of x is( 224 × 224 × 3 ).
Revised: 1, z is a 512-dimensional feature vector ( dim ( z ) = 512) in Resnet-18, and the dimension of x is( 224 × 224 × 3 ).

Original: Dimensionality reduction in an encoder provides the opportunity for the existence of the mapping of OOD and in-distribution samples to the same locations in the latent space.
Revised: Dimensionality reduction in an encoder provides the opportunity for the existence of the mapping of OOD and in-distribution samples to the same locations in the latent space.

Original: This is simply because the vectors in a lower-dimensional space cannot represent all of the vectors/objects in a higher-dimensional space, which is the Pigeonhole Principle.
Revised: This is simply because the vectors in a lower-dimensional space cannot represent all of the vectors/objects in a higher-dimensional space, which is the Pigeonhole Principle.

Original: Let’s do an analysis on the Resnetexample in Fig.
Revised: Let’s do an analysis on the Resnetexample in Fig.

Original: A pixel of the color image x has 8-bits.
Revised: A pixel of the color image x has 8-bits.

Original: In the 150528-dimension discrete input space, there are 256 224 × 224 × 3 different images/vectors, which deﬁnes the size of the input space.
Revised: In the 150528-dimension discrete input space, there are 256 224 × 224 × 3 different images/vectors, which deﬁnes the size of the input space.

Original: ﬂoat32 data type is usually used in computation, a ﬂoat32 variable can roughly represent 2 32 unique real numbers.
Revised: ﬂoat32 data type is usually used in computation, a ﬂoat32 variable can roughly represent 2 32 unique real numbers.

Original: Thus, in the 512-dimensional latent space, there are 2 32 × 512 unique vectors/objects, which deﬁnes the size of the latent space.
Revised: Thus, in the 512-dimensional latent space, there are 2 32 × 512 unique vectors/objects, which deﬁnes the size of the latent space.

Original: The ratio is (cid:16)
Revised: The ratio is (cid:16)

Original: [Table]
Revised: [Table]

Original: (cid:17) (cid:28) 1 , and it shows that the latent space is signiﬁcantly smaller than the input space.
Revised: (cid:17) (cid:28) 1 , and it shows that the latent space is signiﬁcantly smaller than the input space.

Original: Thus, for some sample x in the dataset, we can ﬁnd another sample x (cid:48) such that f ( x (cid:48) )
Revised: Thus, for some sample x in the dataset, we can ﬁnd another sample x (cid:48) such that f ( x (cid:48) )

Original: =
Revised: =

Original: f ( x ) as long as dim ( z ) < dim ( x ) .
Revised: f ( x ) as long as dim ( z ) < dim ( x ) .

Original: A question arises: will the x (cid:48) be in-distribution or OOD?
Revised: A question arises: will the x (cid:48) be in-distribution or OOD?

Original: To answer this question, let’s partition the input discrete space Ω into two disjoint regions ( Ω = Ω in ∪ Ω out ), Ω in of in-distribution samples and Ω out of OOD samples.
Revised: To answer this question, let’s partition the input discrete space Ω into two disjoint regions ( Ω = Ω in ∪ Ω out ), Ω in of in-distribution samples and Ω out of OOD samples.

Original: | Ω | denotes the size of Ω .
Revised: | Ω | denotes the size of Ω .

Original: Usually, the training set is only a subset of Ω in , and the size of Ω out is signiﬁcantly larger than the size of Ω in .
Revised: Usually, the training set is only a subset of Ω in , and the size of Ω out is signiﬁcantly larger than the size of Ω in .

Original: For example, if Ω in is ImageNet, then Ω out contains medical images, noise images, and other weird images.
Revised: For example, if Ω in is ImageNet, then Ω out contains medical images, noise images, and other weird images.

Original: If Ω in contains human face images, then Ω out contains non-face images and then | Ω in | (cid:28) | Ω out | .
Revised: If Ω in contains human face images, then Ω out contains non-face images and then | Ω in | (cid:28) | Ω out | .

Original: The latent space (z-space) is denoted by F and partitioned into two subspaces: F = F in ∪ F out .
Revised: The latent space (z-space) is denoted by F and partitioned into two subspaces: F = F in ∪ F out .

Original: An encoder is applied such that Ω in → F in and Ω out → F out .
Revised: An encoder is applied such that Ω in → F in and Ω out → F out .

Original: If there is overlap F in ∩F out (cid:54) = ∅ , then the encoder is vulnerable to
Revised: If there is overlap F in ∩F out (cid:54) = ∅ , then the encoder is vulnerable to

Original: OOD attack.
Revised: OOD attack.

Original: Usually, the encoder is a part of a classiﬁer trained to classify in-distribution samples into different classes, and therefore the encoder cannot guarantee that there is no overlap between
Revised: Usually, the encoder is a part of a classiﬁer trained to classify in-distribution samples into different classes, and therefore the encoder cannot guarantee that there is no overlap between

Original: F in and F out .
Revised: F in and F out .

Original: What is the size of F in ∩F out
Revised: What is the size of F in ∩F out

Original: or what is the probability P ( |F in ∩ F out | ≥ a ) ?
Revised: or what is the probability P ( |F in ∩ F out | ≥ a ) ?

Original: While it is hard to calculate it for an arbitrary encoder and dataset, we can do a worst-case-scenario analy- sis.
Revised: While it is hard to calculate it for an arbitrary encoder and dataset, we can do a worst-case-scenario analy- sis.

Original: Assuming that every OOD sample is i.i.d.
Revised: Assuming that every OOD sample is i.i.d.

Original: mapped to the latent space with a uniform distribution over a number of |F| spots, then the probability of OOD samples covering the entire latent space is
Revised: mapped to the latent space with a uniform distribution over a number of |F| spots, then the probability of OOD samples covering the entire latent space is

Original: P ( F out = F ) = |F| !
Revised: P ( F out = F ) = |F| !

Original: ×
Revised: ×

Original: Stirling ( | Ω out | , |F| ) / |F| | Ω out | → 1 as |F| / | Ω out | → 0 , where Stirling is the Stirling number of the second kind.
Revised: Stirling ( | Ω out | , |F| ) / |F| | Ω out | → 1 as |F| / | Ω out | → 0 , where Stirling is the Stirling number of the second kind.

Original: Noting that |F| / | Ω out | =
Revised: Noting that |F| / | Ω out | =

Original: [Table]
Revised: [Table]

Original: ≈ 0 and 1 . 4 × 10 7 being the number of samples in ImageNet, then it could be true that almost (with probability close to 1) the entire latent space of Resnet-18 is covered by the z vectors of OOD samples.
Revised: ≈ 0 and 1 . 4 × 10 7 being the number of samples in ImageNet, then it could be true that almost (with probability close to 1) the entire latent space of Resnet-18 is covered by the z vectors of OOD samples.

Original: Next, we discuss how to construct OOD samples to fool neural networks.
Revised: Next, we discuss how to construct OOD samples to fool neural networks.

Original: First, let’s take a look at one-layer linear network:
Revised: First, let’s take a look at one-layer linear network:

Original: z =
Revised: z =

Original: Wx , and make notations: an in-distribution input x ∈ R M , latent code z ∈ R K and K (cid:28) M .
Revised: Wx , and make notations: an in-distribution input x ∈ R M , latent code z ∈ R K and K (cid:28) M .

Original: W is a K × M matrix, and rank ( W ) ≤ K .
Revised: W is a K × M matrix, and rank ( W ) ≤ K .

Original: The null space of W is Ω null =
Revised: The null space of W is Ω null =

Original: { η ; Wη = 0 } .
Revised: { η ; Wη = 0 } .

Original: Now, let’s take out the basis vectors of this space, η 1 , η 2 , . . .
Revised: Now, let’s take out the basis vectors of this space, η 1 , η 2 , . . .

Original: , η M − K , and compute x (cid:48) =
Revised: , η M − K , and compute x (cid:48) =

Original: (cid:80) i λ i
Revised: (cid:80) i λ i

Original: η i
Revised: η i

Original: + x where λ i is a non-zero scalar.
Revised: + x where λ i is a non-zero scalar.

Original: Obviously, z (cid:48) =
Revised: Obviously, z (cid:48) =

Original: Wx (cid:48)
Revised: Wx (cid:48)

Original: =
Revised: =

Original: z .
Revised: z .

Original: We can set the magnitude of the “noise” (cid:80) i λ i η i to be arbitrarily large such that x (cid:48) will look like garbage and become OOD, which is another explanation of the existence of OOD samples.
Revised: We can set the magnitude of the “noise” (cid:80) i λ i η i to be arbitrarily large such that x (cid:48) will look like garbage and become OOD, which is another explanation of the existence of OOD samples.

Original: Then, we can try to apply this attack method to multi-layer neural network.
Revised: Then, we can try to apply this attack method to multi-layer neural network.

Original: If the neural network only uses ReLU activation, then the input-output relationship can be exactly expressed as a piecewise-linear mapping (Ding et al., 2020), a similar approach can be applied layer by layer.
Revised: If the neural network only uses ReLU activation, then the input-output relationship can be exactly expressed as a piecewise-linear mapping (Ding et al., 2020), a similar approach can be applied layer by layer.

Original: If ReLU is not used, a new method is needed.
Revised: If ReLU is not used, a new method is needed.

Original: We note that the ﬁlter bank of a convolution layer can be converted to a weight matrix.
Revised: We note that the ﬁlter bank of a convolution layer can be converted to a weight matrix.

Original: We have examined the state-of-the-art CNN models that are pre-trained on ImageNet and available in Pytorch, and dimensionality reduction is performed in most of the layers (except 1 orlayers near the input), i.e. |F| ≤ | Ω in | (cid:28) | Ω out | .
Revised: We have examined the state-of-the-art CNN models that are pre-trained on ImageNet and available in Pytorch, and dimensionality reduction is performed in most of the layers (except 1 orlayers near the input), i.e. |F| ≤ | Ω in | (cid:28) | Ω out | .

Original: Instead of constructing an OOD sample by adding perturbations to an in-distribution sample, in Algorithm-1, we construct an OOD sample paired with an in-distribution sample by starting from an initial sample that is OOD.
Revised: Instead of constructing an OOD sample by adding perturbations to an in-distribution sample, in Algorithm-1, we construct an OOD sample paired with an in-distribution sample by starting from an initial sample that is OOD.

Original: Could an encoder be made robust to the OOD attack by including OOD samples in training set for supervised binary classiﬁcation: in vs out?
Revised: Could an encoder be made robust to the OOD attack by including OOD samples in training set for supervised binary classiﬁcation: in vs out?

Original: Usually | Ω in | (cid:28) | Ω out | and we will have to collect and label “enough” samples in Ω out , which is infeasible considering the large size of Ω out ≈ Ω .
Revised: Usually | Ω in | (cid:28) | Ω out | and we will have to collect and label “enough” samples in Ω out , which is infeasible considering the large size of Ω out ≈ Ω .

Original: As a comparison, to enhance DNN classiﬁer robustness against adversarial noises, it is very effective to include noisy samples in the training set, i.e. Ω in = Ω in clean ∪ Ω in noisy .
Revised: As a comparison, to enhance DNN classiﬁer robustness against adversarial noises, it is very effective to include noisy samples in the training set, i.e. Ω in = Ω in clean ∪ Ω in noisy .

Original: It is known as adversarial training (Goodfellow et al., 2018) and computationally feasible as | Ω in noisy | (cid:28) | Ω out | .
Revised: It is known as adversarial training (Goodfellow et al., 2018) and computationally feasible as | Ω in noisy | (cid:28) | Ω out | .

Original: 2. P ROBLEM OF G LOW LIKELIHOOD - BASED OOD D ETECTION
Revised: 2. P ROBLEM OF G LOW LIKELIHOOD - BASED OOD D ETECTION

Original: Generative models have been developed to approximate the training data distribution.
Revised: Generative models have been developed to approximate the training data distribution.

Original: Glow (Kingma & Dhariwal, 2018) is one of these models, and it has a very special property: it is bijective and the latent space dimension is the same as the input space dimension, i.e., no dimensionality reduction, which is the reason that we studied this model.
Revised: Glow (Kingma & Dhariwal, 2018) is one of these models, and it has a very special property: it is bijective and the latent space dimension is the same as the input space dimension, i.e., no dimensionality reduction, which is the reason that we studied this model.

Original: Several studies have found the problem of Glow-based OOD detection: likelihoods derived from
Revised: Several studies have found the problem of Glow-based OOD detection: likelihoods derived from

Original: Glow may not distinguish between OOD and training samples (Ren et al., 2019; Choi et al., 2018), and a possible ﬁx to the issue could be using likelihood ratio (Serr ` a et al., 2019).
Revised: Glow may not distinguish between OOD and training samples (Ren et al., 2019; Choi et al., 2018), and a possible ﬁx to the issue could be using likelihood ratio (Serr ` a et al., 2019).

Original: In this study, we further show that negative log-likelihood (NLL) from the Glow model can be arbitrarily manipulated by our algorithm in which f ( x ) denotes NLL.
Revised: In this study, we further show that negative log-likelihood (NLL) from the Glow model can be arbitrarily manipulated by our algorithm in which f ( x ) denotes NLL.

Original: The results on CelebA face image dataset are in
Revised: The results on CelebA face image dataset are in

Original: Section 3.
Revised: Section 3.

Original: We think the major reason causing Glow’s vulnerability to OOD attack is that we do not have enough training data in high dimensional space.
Revised: We think the major reason causing Glow’s vulnerability to OOD attack is that we do not have enough training data in high dimensional space.

Original: Glow is a mapping: x in → z in → p ( z in ) → p ( x in ) , the probability of x in .
Revised: Glow is a mapping: x in → z in → p ( z in ) → p ( x in ) , the probability of x in .

Original: For an OOD sample x out , the mapping is x out → z out → p ( z out ) → p ( x out ) .
Revised: For an OOD sample x out , the mapping is x out → z out → p ( z out ) → p ( x out ) .

Original: Since the number of training samples is signiﬁcantly smaller than the size of the space, there are a huge number of “holes” in the latent space (i.e., regions that no training samples are mapped to), and it is easy to put z out in one of these “holes” close to z in such that p ( z out ) ∼ = p ( z in ) .
Revised: Since the number of training samples is signiﬁcantly smaller than the size of the space, there are a huge number of “holes” in the latent space (i.e., regions that no training samples are mapped to), and it is easy to put z out in one of these “holes” close to z in such that p ( z out ) ∼ = p ( z in ) .

Original: 2. R ECONSTRUCTION - BASED OOD D ETECTION
Revised: 2. R ECONSTRUCTION - BASED OOD D ETECTION

Original: Auto-encoder style OOD detection has been developed for anomaly detection (Chalapathy &
Revised: Auto-encoder style OOD detection has been developed for anomaly detection (Chalapathy &

Original: Chawla, 2019; Cohen et al., 2019) based on reconstruction error.
Revised: Chawla, 2019; Cohen et al., 2019) based on reconstruction error.

Original: The data ﬂow of an auto-encoder is x → z → ˆ x where ˆ x is the reconstruction of x .
Revised: The data ﬂow of an auto-encoder is x → z → ˆ x where ˆ x is the reconstruction of x .

Original: The OOD detection score can be the difference between x and ˆ x , e.g., the Lp distance || x − ˆ x || p or Mahalanobis Distance.
Revised: The OOD detection score can be the difference between x and ˆ x , e.g., the Lp distance || x − ˆ x || p or Mahalanobis Distance.

Original: This type of method has two known issues.
Revised: This type of method has two known issues.

Original: The ﬁrst issue is that auto-encoder may well reconstruct OOD samples,
Revised: The ﬁrst issue is that auto-encoder may well reconstruct OOD samples,

Original: i.e., x out ≈
Revised: i.e., x out ≈

Original: ˆ x out .
Revised: ˆ x out .

Original: Thus, one needs to make sure it has large reconstruction errors on OOD samples, which can be done by limiting the capacity of auto-encoder or saturating it with in-distribution samples.
Revised: Thus, one needs to make sure it has large reconstruction errors on OOD samples, which can be done by limiting the capacity of auto-encoder or saturating it with in-distribution samples.

Original: The second issue is that pixel-to-pixel distance is not a good measurement of image dissimilarity, especially for medical images.
Revised: The second issue is that pixel-to-pixel distance is not a good measurement of image dissimilarity, especially for medical images.

Original: For example, x could be a CT image of a heart and ˆ x could be the image of the same heart that deforms a little bit, but the pixel-to-pixel distance between x and ˆ x can be very large.
Revised: For example, x could be a CT image of a heart and ˆ x could be the image of the same heart that deforms a little bit, but the pixel-to-pixel distance between x and ˆ x can be very large.

Original: Thus, a robust image similarity measurement is needed.
Revised: Thus, a robust image similarity measurement is needed.

Original: Interestingly, the proposed OOD attack algorithm has no effect on this type of method.
Revised: Interestingly, the proposed OOD attack algorithm has no effect on this type of method.

Original: Let’s consider the data ﬂow: x in → z in →
Revised: Let’s consider the data ﬂow: x in → z in →

Original: ˆ x in and x out → z out → ˆ x out .
Revised: ˆ x in and x out → z out → ˆ x out .

Original: If z out = z in , then ˆ x out = ˆ x in .
Revised: If z out = z in , then ˆ x out = ˆ x in .

Original: Then, it is easy to ﬁnd out that x out is OOD because || x out − ˆ x out || p
Revised: Then, it is easy to ﬁnd out that x out is OOD because || x out − ˆ x out || p

Original: =
Revised: =

Original: || x out − ˆ x in || p which is very large.
Revised: || x out − ˆ x in || p which is very large.

Original: Ironically, in this case, the attack algorithm helps to identify the OOD sample.
Revised: Ironically, in this case, the attack algorithm helps to identify the OOD sample.

Original: In future work, we will evaluate the effectiveness of combining the proposed algorithm and auto-encoder for
Revised: In future work, we will evaluate the effectiveness of combining the proposed algorithm and auto-encoder for

Original: OOD detection.
Revised: OOD detection.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We applied the proposed algorithm to attack state-of-the-art DNN models on image datasets.
Revised: We applied the proposed algorithm to attack state-of-the-art DNN models on image datasets.

Original: For each in-distribution sample x in , an OOD sample x out is generated by the algorithm.
Revised: For each in-distribution sample x in , an OOD sample x out is generated by the algorithm.

Original: To measure attack strength, mean absolute percentage error is calculated by MAPE ( z out )
Revised: To measure attack strength, mean absolute percentage error is calculated by MAPE ( z out )

Original: = mean ( | z out − z in | ) /max
Revised: = mean ( | z out − z in | ) /max

Original: ( | z in | ) .
Revised: ( | z in | ) .

Original: Here, z out = f ( x out ) and z in = f ( x in ) .
Revised: Here, z out = f ( x out ) and z in = f ( x in ) .

Original: | z out − z in | is an error vector, and mean ( | z out − z in | ) is the average error.
Revised: | z out − z in | is an error vector, and mean ( | z out − z in | ) is the average error.

Original: max ( | z in | ) is the maximum absolute value in the vector z in .
Revised: max ( | z in | ) is the maximum absolute value in the vector z in .

Original: We also applied the algorithm to attack the Glow model on CelebA dataset.
Revised: We also applied the algorithm to attack the Glow model on CelebA dataset.

Original: In all of the evaluations,
Revised: In all of the evaluations,

Original: L2 norm was used in the proposed algorithm.
Revised: L2 norm was used in the proposed algorithm.

Original: Pytorch was used to implement the algorithm.
Revised: Pytorch was used to implement the algorithm.

Original: Nvidia
Revised: Nvidia

Original: Titan V GPUs were used for model training and testing.
Revised: Titan V GPUs were used for model training and testing.

Original: 3. E VALUATION ON A S UBSET OF I MAGE N ET
Revised: 3. E VALUATION ON A S UBSET OF I MAGE N ET

Original: [Table]
Revised: [Table]

Original: has over 1 million images in 1000 classes.
Revised: has over 1 million images in 1000 classes.

Original: Given the limited computing power, it is impractical to test the algorithm on the whole dataset.
Revised: Given the limited computing power, it is impractical to test the algorithm on the whole dataset.

Original: Instead, we used a subset ofimages in 200 categories.
Revised: Instead, we used a subset ofimages in 200 categories.

Original: The size of each image is 224×224×3.
Revised: The size of each image is 224×224×3.

Original: Two CNN models pretrained on the ImageNet were evaluated, which are Resnet-18 and Densenet-121 available in Pytorch.
Revised: Two CNN models pretrained on the ImageNet were evaluated, which are Resnet-18 and Densenet-121 available in Pytorch.

Original: Resnet-18 latent space has 512 dimensions.
Revised: Resnet-18 latent space has 512 dimensions.

Original: Since ImageNet covers a variety of natural and artiﬁcial objects, we choose medical images and random-noise images to make sure that x (cid:48) out is indeed OOD.
Revised: Since ImageNet covers a variety of natural and artiﬁcial objects, we choose medical images and random-noise images to make sure that x (cid:48) out is indeed OOD.

Original: Using each of the three initial OOD samples (chest x-ray, lung-CT, and random noise to be x (cid:48) out ), we generated 1000 OOD samples paired with the 1000 (in-distribution) samples in the dataset and calculated MAPE values.
Revised: Using each of the three initial OOD samples (chest x-ray, lung-CT, and random noise to be x (cid:48) out ), we generated 1000 OOD samples paired with the 1000 (in-distribution) samples in the dataset and calculated MAPE values.

Original: The three MAPE histograms are shown in Fig.
Revised: The three MAPE histograms are shown in Fig.

Original: Most of the MAPE values are less than 0 .
Revised: Most of the MAPE values are less than 0 .

Original: We also evaluated another CNN, named Densenet-121, and obtained similar results.
Revised: We also evaluated another CNN, named Densenet-121, and obtained similar results.

Original: The latent space has 1024 dimensions.
Revised: The latent space has 1024 dimensions.

Original: Again, using each of the three initial OOD samples, 1000 OOD samples are generated for the samples in the dataset, and then MAPE values are calculated.
Revised: Again, using each of the three initial OOD samples, 1000 OOD samples are generated for the samples in the dataset, and then MAPE values are calculated.

Original: The three MAPE histograms are shown in Fig.
Revised: The three MAPE histograms are shown in Fig.

Original: Most of the MAPE values are less than 0 .
Revised: Most of the MAPE values are less than 0 .

Original: 1% , indicating strong
Revised: 1% , indicating strong

Original: OOD attack.
Revised: OOD attack.

Original: From the results in Fig.2 to Fig.
Revised: From the results in Fig.2 to Fig.

Original: 4, it can be seen that each of the two CNN models mapped signiﬁcantly different OOD samples and in-distribution samples to almost the same locations in the latent space.
Revised: 4, it can be seen that each of the two CNN models mapped signiﬁcantly different OOD samples and in-distribution samples to almost the same locations in the latent space.

Original: Dimensionality reduction leads to the existence of such mapping, and our algorithm can ﬁnd such OOD samples out.
Revised: Dimensionality reduction leads to the existence of such mapping, and our algorithm can ﬁnd such OOD samples out.

Original: In other words, the mapping from input space to the latent space is many-to-one, not bijective.
Revised: In other words, the mapping from input space to the latent space is many-to-one, not bijective.

Original: And therefore, it is almost guaranteed that such OOD samples exist and they can break any OOD detector d that computes a detection score
Revised: And therefore, it is almost guaranteed that such OOD samples exist and they can break any OOD detector d that computes a detection score

Original: d ( z ) only from the latent space (z-space).
Revised: d ( z ) only from the latent space (z-space).

Original: We tested a classical OOD detection method using the maximum of softmax output as detection score (Hendrycks & Gimpel, 2017).
Revised: We tested a classical OOD detection method using the maximum of softmax output as detection score (Hendrycks & Gimpel, 2017).

Original: The results are shown in Table-1, and the AUROC scores are close to 0.5, showing that the method is unable to tell the difference between the
Revised: The results are shown in Table-1, and the AUROC scores are close to 0.5, showing that the method is unable to tell the difference between the

Original: OOD samples and 1000 in-distribution samples.
Revised: OOD samples and 1000 in-distribution samples.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 3. E VALUATION ON C ELEB A D ATASET
Revised: 3. E VALUATION ON C ELEB A D ATASET

Original: We tested the algorithm and the Glow model (Kingma & Dhariwal, 2018) on the CelebA dataset (human face images).
Revised: We tested the algorithm and the Glow model (Kingma & Dhariwal, 2018) on the CelebA dataset (human face images).

Original: The size of each image is 64×64×3.
Revised: The size of each image is 64×64×3.

Original: After training, the model was able to generate realistic face images.
Revised: After training, the model was able to generate realistic face images.

Original: The model also outputs the negative log-likelihood (NLL) of the input sample, i.e., NNL ( x ) = − log ( p ( x )) .
Revised: The model also outputs the negative log-likelihood (NLL) of the input sample, i.e., NNL ( x ) = − log ( p ( x )) .

Original: By setting f ( x ) =
Revised: By setting f ( x ) =

Original: NNL ( x ) , our algorithm can make f ( x out ) to be close to 0 or very large to match any f ( x in ) , which renders NLL score useless for OOD detection.
Revised: NNL ( x ) , our algorithm can make f ( x out ) to be close to 0 or very large to match any f ( x in ) , which renders NLL score useless for OOD detection.

Original: To demonstrate the effectiveness of our algorithm, we randomly selected(in-distribution) samples in the dataset.
Revised: To demonstrate the effectiveness of our algorithm, we randomly selected(in-distribution) samples in the dataset.

Original: We used a color spiral image as the initial OOD sample x (cid:48) out , and NNL ( x (cid:48) out )
Revised: We used a color spiral image as the initial OOD sample x (cid:48) out , and NNL ( x (cid:48) out )

Original: = 3 . 5268 .
Revised: = 3 . 5268 .

Original: The distributions of NLL ( x in ) from 160 in-distribution samples and NLL ( x out ) from 160 corresponding OOD samples, as well as OOD sample images are shown in Fig.
Revised: The distributions of NLL ( x in ) from 160 in-distribution samples and NLL ( x out ) from 160 corresponding OOD samples, as well as OOD sample images are shown in Fig.

Original: The two distributions are almost identical.
Revised: The two distributions are almost identical.

Original: More examples of OOD samples are shown in
Revised: More examples of OOD samples are shown in

Original: Fig.
Revised: Fig.

Original: In each row of Fig.
Revised: In each row of Fig.

Original: 6, although the images have different NLL scores, they look like each other.
Revised: 6, although the images have different NLL scores, they look like each other.

Original: We have done more evaluations of our algorithm and OOD detection methods, please read the appendices.
Revised: We have done more evaluations of our algorithm and OOD detection methods, please read the appendices.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We hypothesized that dimensionality reduction in an encoder provides the opportunity for the existence of the mapping of OOD and in-distribution samples to the same locations in the latent space.
Revised: We hypothesized that dimensionality reduction in an encoder provides the opportunity for the existence of the mapping of OOD and in-distribution samples to the same locations in the latent space.

Original: We applied the OOD Attack algorithm to DNN classiﬁers on various datasets (see Appendices A and B), and the results (i.e. low MAPE values) conﬁrmed our hypothesis.
Revised: We applied the OOD Attack algorithm to DNN classiﬁers on various datasets (see Appendices A and B), and the results (i.e. low MAPE values) conﬁrmed our hypothesis.

Original: The results imply that classiﬁer/encoder -based OOD detection methods may be vulnerable to the OOD attack.
Revised: The results imply that classiﬁer/encoder -based OOD detection methods may be vulnerable to the OOD attack.

Original: By using our OOD Attack algorithm, we evaluated nine OOD detection methods (see Appendices C to J).
Revised: By using our OOD Attack algorithm, we evaluated nine OOD detection methods (see Appendices C to J).

Original: The AUROC scores of these methods are close to 0.5 in our experiments, which means these methods could not distinguish between the in-distribution samples (e.g. CIFAR10) and the OOD samples generated by our algorithms.
Revised: The AUROC scores of these methods are close to 0.5 in our experiments, which means these methods could not distinguish between the in-distribution samples (e.g. CIFAR10) and the OOD samples generated by our algorithms.

Original: Our algorithm was unable to break a recent method named
Revised: Our algorithm was unable to break a recent method named

Original: Certiﬁed Certain Uncertainty (Meinke & Hein, 2020), because this method utilizes Gaussian mixture models (GMMs) in the input space (note: no dimensionality reduction in GMMs).
Revised: Certiﬁed Certain Uncertainty (Meinke & Hein, 2020), because this method utilizes Gaussian mixture models (GMMs) in the input space (note: no dimensionality reduction in GMMs).

Original: However, it is well known that GMMs have convergence issues for high dimensional data (e.g. medical images).
Revised: However, it is well known that GMMs have convergence issues for high dimensional data (e.g. medical images).

Original: Compared to adversarial attacks and defenses, it is much more difﬁcult to defense against OOD attacks.
Revised: Compared to adversarial attacks and defenses, it is much more difﬁcult to defense against OOD attacks.

Original: Adversarial attacks and OOD attacks are doing completely different things to neural networks, although the attack algorithms may use similar optimization techniques.
Revised: Adversarial attacks and OOD attacks are doing completely different things to neural networks, although the attack algorithms may use similar optimization techniques.

Original: For image classiﬁcation applications, an adversarial attack will add a small amount of noise to the input (clean) image, and the resulting noisy image is still human-recognizable.
Revised: For image classiﬁcation applications, an adversarial attack will add a small amount of noise to the input (clean) image, and the resulting noisy image is still human-recognizable.

Original: Therefore, the magnitudes of adversarial noises are constrained.
Revised: Therefore, the magnitudes of adversarial noises are constrained.

Original: For example, a noisy image of a panda is still an image of the panda.
Revised: For example, a noisy image of a panda is still an image of the panda.

Original: By the judgment of humans, the noisy image and the clean image are the images of the same object, and the two images should be classiﬁed into the same class.
Revised: By the judgment of humans, the noisy image and the clean image are the images of the same object, and the two images should be classiﬁed into the same class.

Original: Compared to adversarial samples, OOD samples, which can be generated by our OOD Attack algorithm, have much more freedom (e.g. they can be random noises), as long as they do not look like in-distribution samples.
Revised: Compared to adversarial samples, OOD samples, which can be generated by our OOD Attack algorithm, have much more freedom (e.g. they can be random noises), as long as they do not look like in-distribution samples.

Original: Thus, OOD detection is very challenging.
Revised: Thus, OOD detection is very challenging.

Original: We would like to point out that it is difﬁcult to evaluate an OOD detector to “prove” that it can detect, say 90% of the OOD samples by experimentally testing it on Ω out because Ω out is too large to be tested on: | Ω in | (cid:28) | Ω out | ≈ | Ω | .
Revised: We would like to point out that it is difﬁcult to evaluate an OOD detector to “prove” that it can detect, say 90% of the OOD samples by experimentally testing it on Ω out because Ω out is too large to be tested on: | Ω in | (cid:28) | Ω out | ≈ | Ω | .

Original: For example, if Fashion-MNIST is used as in-distribution, then MINST and Omniglot are usually as OOD, which is the “standard” approach in the literature.
Revised: For example, if Fashion-MNIST is used as in-distribution, then MINST and Omniglot are usually as OOD, which is the “standard” approach in the literature.

Original: Clearly, MINST and Omniglot cannot cover Ω out the space of OOD samples.
Revised: Clearly, MINST and Omniglot cannot cover Ω out the space of OOD samples.

Original: If the image size is larger, then | Ω out | becomes much larger.
Revised: If the image size is larger, then | Ω out | becomes much larger.

Original: Could we design an evaluation method (experimental or analytical) that does not rely on OOD samples?
Revised: Could we design an evaluation method (experimental or analytical) that does not rely on OOD samples?

Original: Before the OOD detection issue is fully resolved, for life-critical applications, any machine learning system that uses DNN classiﬁers should not make decisions independently and can only serve as assistants to humans.
Revised: Before the OOD detection issue is fully resolved, for life-critical applications, any machine learning system that uses DNN classiﬁers should not make decisions independently and can only serve as assistants to humans.

Original: The OOD Attack algorithm and the experimental results can serve as a reference for the evaluation of new OOD detection methods.
Revised: The OOD Attack algorithm and the experimental results can serve as a reference for the evaluation of new OOD detection methods.

Original: We will release the code on GitHub when the paper is accepted.
Revised: We will release the code on GitHub when the paper is accepted.

Original: All ﬁgures are in high-resolution, please zoom in.
Revised: All ﬁgures are in high-resolution, please zoom in.

Original: A PPENDIX
Revised: A PPENDIX

Original: The parameters in each evaluation are listed.
Revised: The parameters in each evaluation are listed.

Original: Parameters for evaluation on ImageNet subset Attacking resnet-18: ε = 5 , N = 1 e 4 , α = ε/with x-ray and CT; ε = 20 , N = 1 e 4 , α = ε/ 100 with random noise.
Revised: Parameters for evaluation on ImageNet subset Attacking resnet-18: ε = 5 , N = 1 e 4 , α = ε/with x-ray and CT; ε = 20 , N = 1 e 4 , α = ε/ 100 with random noise.

Original: Attacking densnet-121: ε = 5 , N = 1 e 4 , α = ε/ 100 with x-ray and CT; ε = 30 , N = 1 e 4 , α = ε/ 100 with random noise.
Revised: Attacking densnet-121: ε = 5 , N = 1 e 4 , α = ε/ 100 with x-ray and CT; ε = 30 , N = 1 e 4 , α = ε/ 100 with random noise.

Original: We inspected the OOD images from random noise: they are not recognizable to human vision
Revised: We inspected the OOD images from random noise: they are not recognizable to human vision

Original: Parameters for evaluation on OCT dataset ε = 10 , N = 1 e 4 , α = ε/ 100 with retinal fundus photography image ε = 20 , N = 1 e 4 , α = ε/ 100 with random noise.
Revised: Parameters for evaluation on OCT dataset ε = 10 , N = 1 e 4 , α = ε/ 100 with retinal fundus photography image ε = 20 , N = 1 e 4 , α = ε/ 100 with random noise.

Original: Parameters for evaluation on COVID-

Revised: Parameters for evaluation on COVID-

Original: CT dataset ε = 20 , N = 1 e 4 , α = ε/4.
Revised: CT dataset ε = 20 , N = 1 e 4 , α = ε/4.

Original: Parameters for evaluation on GTSRB dataset ε = 10 , N = 1 e 4 , α = ε/5.
Revised: Parameters for evaluation on GTSRB dataset ε = 10 , N = 1 e 4 , α = ε/5.

Original: Parameters for Evaluation on CelebA Dataset ε = 10 , N = 1 e 4 , α = ε/
Revised: Parameters for Evaluation on CelebA Dataset ε = 10 , N = 1 e 4 , α = ε/

Original: B A PPENDIX B. E VALUATION ON OCT DATASET
Revised: B A PPENDIX B. E VALUATION ON OCT DATASET

Original: We tested our algorithm and Resnet-18 on a retinal optical coherence tomography (OCT) dataset (Kermany et al., 2018), which has four classes.
Revised: We tested our algorithm and Resnet-18 on a retinal optical coherence tomography (OCT) dataset (Kermany et al., 2018), which has four classes.

Original: Each image is resized to 224×224.
Revised: Each image is resized to 224×224.

Original: 1000 samples per class were randomly selected to obtain a training set of 4000 samples.
Revised: 1000 samples per class were randomly selected to obtain a training set of 4000 samples.

Original: The test set has 968 images.
Revised: The test set has 968 images.

Original: We modiﬁed Resnet-18 for this four-class classiﬁcation task.
Revised: We modiﬁed Resnet-18 for this four-class classiﬁcation task.

Original: The latent space has 512 dimensions.
Revised: The latent space has 512 dimensions.

Original: After training, the Resnet-18 model achieved a classiﬁcation accuracy > 95% on the test set.
Revised: After training, the Resnet-18 model achieved a classiﬁcation accuracy > 95% on the test set.

Original: We used two references images as the initial OOD sample x (cid:48) out .
Revised: We used two references images as the initial OOD sample x (cid:48) out .

Original: The ﬁrst reference image is a grayscale retinal image converted from an RGB color retinal fundus photography image.
Revised: The ﬁrst reference image is a grayscale retinal image converted from an RGB color retinal fundus photography image.

Original: Compared to this retinal fundus photography image, the OCT images have unique patterns of horizontal “white
Revised: Compared to this retinal fundus photography image, the OCT images have unique patterns of horizontal “white

Original: [Figure]
Revised: [Figure]

Original: bands”.
Revised: bands”.

Original: We selected this OOD image by purpose: there may be a chance that both types of images are needed for retinal diagnosis.
Revised: We selected this OOD image by purpose: there may be a chance that both types of images are needed for retinal diagnosis.

Original: The second reference image is generated from random noises.
Revised: The second reference image is generated from random noises.

Original: Examples are shown in Fig.
Revised: Examples are shown in Fig.

Original: 7, and the two MAPE histograms are shown in Fig.
Revised: 7, and the two MAPE histograms are shown in Fig.

Original: The results conﬁrm that the algorithm can generate OOD samples (968) which are mapped by the DNN model to the locations of the in-distribution samples (968) in the latent space, i.e., z out ∼ =
Revised: The results conﬁrm that the algorithm can generate OOD samples (968) which are mapped by the DNN model to the locations of the in-distribution samples (968) in the latent space, i.e., z out ∼ =

Original: z in .
Revised: z in .

Original: B. E VALUATION ON COVID-19 CT D ATASET
Revised: B. E VALUATION ON COVID-19 CT D ATASET

Original: We also tested our algorithm and Resnet-18 on a public COVID-19 lung CT (2D) image dataset (Soares et al., 2020).
Revised: We also tested our algorithm and Resnet-18 on a public COVID-19 lung CT (2D) image dataset (Soares et al., 2020).

Original: It contains 1252 CT scans (2D images) that are positive for COVID-19 infection and 1230 CT scans (2D images) for patients non-infected by COVID-19, 2482 CT scans in total.
Revised: It contains 1252 CT scans (2D images) that are positive for COVID-19 infection and 1230 CT scans (2D images) for patients non-infected by COVID-19, 2482 CT scans in total.

Original: From infected cases, we randomly selected 200 samples for testing, 30 for validation, and 1022 for training.
Revised: From infected cases, we randomly selected 200 samples for testing, 30 for validation, and 1022 for training.

Original: From the uninfected cases, we randomly selected 200 for testing, 30 for validation and 1000 for training.
Revised: From the uninfected cases, we randomly selected 200 for testing, 30 for validation and 1000 for training.

Original: Each image is resized to 224×224.
Revised: Each image is resized to 224×224.

Original: We modiﬁed the last layer of Resnet-18 for this binary classiﬁcation task, infected vs uninfected.
Revised: We modiﬁed the last layer of Resnet-18 for this binary classiﬁcation task, infected vs uninfected.

Original: We also replaced batch normalization with instance normalization because it is known that batch normalization is not stable for small batch-size (Wu & He, 2018).
Revised: We also replaced batch normalization with instance normalization because it is known that batch normalization is not stable for small batch-size (Wu & He, 2018).

Original: The latent space still hasdimensions.
Revised: The latent space still hasdimensions.

Original: We set batch-size to 32, the number of training epochs to 100, and used AdamW optimizer with the default parameters.
Revised: We set batch-size to 32, the number of training epochs to 100, and used AdamW optimizer with the default parameters.

Original: After training, the model achieved a classiﬁcation accuracy > 95% on test set.
Revised: After training, the model achieved a classiﬁcation accuracy > 95% on test set.

Original: We used two reference images as the initial OOD sample x (cid:48) out , a chest x-ray image, and a randomnoise image.
Revised: We used two reference images as the initial OOD sample x (cid:48) out , a chest x-ray image, and a randomnoise image.

Original: The two MAPE histograms are shown in Fig.
Revised: The two MAPE histograms are shown in Fig.

Original: 9 that most of the MAPE values are less than 0 .
Revised: 9 that most of the MAPE values are less than 0 .

Original: The results also conﬁrm that the algorithm can generate OOD samples (400) which are mapped by the DNN model to the locations of the in-distribution samples (400) in the latent space, i.e., z out ∼ =
Revised: The results also conﬁrm that the algorithm can generate OOD samples (400) which are mapped by the DNN model to the locations of the in-distribution samples (400) in the latent space, i.e., z out ∼ =

Original: z in .
Revised: z in .

Original: Examples are shown in Fig.
Revised: Examples are shown in Fig.

Original: As reported in the previous studies (Shi et al., 2020), infected regions in the images have a unique pattern called ground-glass opacity.
Revised: As reported in the previous studies (Shi et al., 2020), infected regions in the images have a unique pattern called ground-glass opacity.

Original: The CT images in the 1st and 3rd rows show COVID-19 infections with ground-glass opacity on the upper-left area.
Revised: The CT images in the 1st and 3rd rows show COVID-19 infections with ground-glass opacity on the upper-left area.

Original: The CT image in the 5th row does not show any signs of infection.
Revised: The CT image in the 5th row does not show any signs of infection.

Original: It can be seen that the random-noise images and the COVID-19 CT images have the same feature vectors in the latent space, which is astonishing.
Revised: It can be seen that the random-noise images and the COVID-19 CT images have the same feature vectors in the latent space, which is astonishing.

Original: [Figure]
Revised: [Figure]

Original: MAPE values are embedded in these scatter-plots.
Revised: MAPE values are embedded in these scatter-plots.

Original: Please zoom-in for better visualization.
Revised: Please zoom-in for better visualization.

Original: B. E VALUATION ON GTSRB T RAFFIC S IGN D ATASET
Revised: B. E VALUATION ON GTSRB T RAFFIC S IGN D ATASET

Original: We tested our algorithm and a start-of-the-art trafﬁc sign classiﬁer on the GTSRB dataset.
Revised: We tested our algorithm and a state-of-the-art trafﬁc sign classiﬁer on the GTSRB dataset.

Original: The classiﬁer is similar to the one in (Arcos-Garcia et al., 2018), which has a spatial-transformer network.
Revised: The classiﬁer is similar to the one in (Arcos-Garcia et al., 2018), which has a spatial-transformer network.

Original: The size of each image is 32×32×3.
Revised: The size of each image is 32×32×3.

Original: The latent space has 128 dimensions.
Revised: The latent space has 128 dimensions.

Original: After training, the classiﬁer achieved over 99% accuracy on the test set.
Revised: After training, the classiﬁer achieved over 99% accuracy on the test set.

Original: We used a random-noise image as the initial
Revised: We used a random-noise image as the initial

Original: OOD sample x (cid:48) out to generate 12630 OOD samples paired with the 12630 in-distribution samples in the test set.
Revised: OOD sample x (cid:48) out to generate 12630 OOD samples paired with the 12630 in-distribution samples in the test set.

Original: The MAPE histogram is shown in Fig.
Revised: The MAPE histogram is shown in Fig.

Original: 11, in which most of the MAPE values are less than 0 .
Revised: 11, in which most of the MAPE values are less than 0 .

Original: Examples are shown in Fig.
Revised: Examples are shown in Fig.

Original: It can be seen that z out of random-noise images are almost the same as z in of the stop sign, the speed limit sign, and the turning signs.
Revised: It can be seen that z out of random-noise images are almost the same as z in of the stop sign, the speed limit sign, and the turning signs.

Original: Not only the classiﬁer cannot tell the difference between a real trafﬁc sign and a generated noise image, but also any detectors based on z in for OOD detection will fail.
Revised: Not only the classiﬁer cannot tell the difference between a real trafﬁc sign and a generated noise image, but also any detectors based on z in for OOD detection will fail.

Original: We note that adversarial robustness of trafﬁc sign classiﬁers has been studied (Eykholt et al., 2018), and after adding adversarial noises to the trafﬁc sign images, the noisy images are still recognizable.
Revised: We note that adversarial robustness of trafﬁc sign classiﬁers has been studied (Eykholt et al., 2018), and after adding adversarial noises to the trafﬁc sign images, the noisy images are still recognizable.

Original: OOD noises and adversarial noises are very different (discussed in Sections 2.1 and 2.2).
Revised: OOD noises and adversarial noises are very different (discussed in Sections 2.1 and 2.2).

Original: Thus, it would be wise to disable any vision-based auto-pilot in your self-driving cars today until this issue is resolved.
Revised: Thus, it would be wise to disable any vision-based auto-pilot in your self-driving cars today until this issue is resolved.

Original: [Figure]
Revised: [Figure]

Original: C A PPENDIX
Revised: C A PPENDIX

Original: We applied our OOD Attack algorithm to test the OOD detection method named ODIN (Liang et al., 2018).
Revised: We applied our OOD Attack algorithm to test the OOD detection method named ODIN (Liang et al., 2018).

Original: C. S UMMARY OF THE ODIN METHOD
Revised: C. S UMMARY OF THE ODIN METHOD

Original: The method does temperature scaling on the logits (input to softmax) by logits/ T , which is the Eq.(1) in the ODIN paper.
Revised: The method does temperature scaling on the logits (input to softmax) by logits/ T , which is the Eq.(1) in the ODIN paper.

Original: The temperature T could be in the range of 1 to 1000.
Revised: The temperature T could be in the range of 1 to 1000.

Original: The ODIN method also does input preprocessing, which is the Eq.(2) in the ODIN paper.
Revised: The ODIN method also does input preprocessing, which is the Eq.(2) in the ODIN paper.

Original: For preprocessing, the perturbation magnitude ( PM ) could be in the range of 0 to 0.004.
Revised: For preprocessing, the perturbation magnitude ( PM ) could be in the range of 0 to 0.004.

Original: The OOD score is deﬁned to be the maximum of the softmax outputs from a neural network, given the preprocessed input.
Revised: The OOD score is deﬁned to be the maximum of the softmax outputs from a neural network, given the preprocessed input.

Original: An OOD sample is expected to have a low OOD score.
Revised: An OOD sample is expected to have a low OOD score.

Original: C. E VALUATION ON CIFAR 
Revised: C. E VALUATION ON CIFAR

Original: Wide residual network with depth 28 and widen factor 10 is used in the ODIN paper.
Revised: Wide residual network with depth 28 and widen factor 10 is used in the ODIN paper.

Original: After training for 200 epochs, the model achieved the classiﬁcation accuracy of 94.71 on CIFAR10 test set.
Revised: After training for 200 epochs, the model achieved the classiﬁcation accuracy of 94.71 on CIFAR10 test set.

Original: In our algorithm, we set f ( x ) to be the logits output from the model, given the preprocessed input.
Revised: In our algorithm, we set f ( x ) to be the logits output from the model, given the preprocessed input.

Original: For the CIFAR10 dataset, the logits output contains 10 elements, which is signiﬁcant dimensionality reduction compared to the size of an input color image: 32 × 32 × 3 .
Revised: For the CIFAR10 dataset, the logits output contains 10 elements, which is signiﬁcant dimensionality reduction compared to the size of an input color image: 32 × 32 × 3 .

Original: In our algorithm, the parameters are ε = 10 , α = ε/ 100 , N = 100 .
Revised: In our algorithm, the parameters are ε = 10 , α = ε/ 100 , N = 100 .

Original: The initial OOD sample is a random noise image.
Revised: The initial OOD sample is a random noise image.

Original: For every sample in the CIFAR10 test set, the algorithm generated an OOD sample to match the logits output.
Revised: For every sample in the CIFAR10 test set, the algorithm generated an OOD sample to match the logits output.

Original: The generated OOD samples look like random noises.
Revised: The generated OOD samples look like random noises.

Original: The OOD scores of these samples were calculated by the ODIN method.
Revised: The OOD scores of these samples were calculated by the ODIN method.

Original: The results are reported in Table 2.
Revised: The results are reported in Table 2.

Original: Fig.
Revised: Fig.

Original: 13 shows the OOD score histograms of the in-distribution and OOD samples
Revised: 13 shows the OOD score histograms of the in-distribution and OOD samples

Original: when T =1000 and PM =0.001.
Revised: when T =1000 and PM =0.001.

Original: When T = 1 and PM = 0 , ODIN becomes the
Revised: When T = 1 and PM = 0 , ODIN becomes the

Original: Baseline method (Hendrycks & Gimpel, 2017).
Revised: Baseline method (Hendrycks & Gimpel, 2017).

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D A PPENDIX
Revised: D A PPENDIX

Original: We applied our OOD Attack algorithm to test the OOD detection method named Mahalanobis (Lee et al., 2018b).
Revised: We applied our OOD Attack algorithm to test the OOD detection method named Mahalanobis (Lee et al., 2018b).

Original: D. S UMMARY OF THE M AHALANOBIS METHOD
Revised: D. S UMMARY OF THE M AHALANOBIS METHOD

Original: The method extracts the feature maps from multiple layers of a neural network and applies average pooling per channel to reduce each feature map into a 1D feature vector.
Revised: The method extracts the feature maps from multiple layers of a neural network and applies average pooling per channel to reduce each feature map into a 1D feature vector.

Original: Then, Mahalanobis distance is calculated between a feature vector and the corresponding mean vector.
Revised: Then, Mahalanobis distance is calculated between a feature vector and the corresponding mean vector.

Original: The distance values from all of the feature vectors are linearly combined together to produce a single distance, i.e., the
Revised: The distance values from all of the feature vectors are linearly combined together to produce a single distance, i.e., the

Original: OOD score.
Revised: OOD score.

Original: The OOD score of an OOD sample is expected to be large.
Revised: The OOD score of an OOD sample is expected to be large.

Original: To further improve the performance, the method does input preprocessing with a given perturbation magnitude (PM), and then OOD score of the preprocessed input is obtained.
Revised: To further improve the performance, the method does input preprocessing with a given perturbation magnitude (PM), and then OOD score of the preprocessed input is obtained.

Original: The weights to combine the Mahalanobis distances from multiple layers could be determined on a validation set of OOD samples.
Revised: The weights to combine the Mahalanobis distances from multiple layers could be determined on a validation set of OOD samples.

Original: In practice, it is impossible to obtain such a validation set.
Revised: In practice, it is impossible to obtain such a validation set.

Original: In our evaluation, we simply take the average of the distance values, which gives the OOD score.
Revised: In our evaluation, we simply take the average of the distance values, which gives the OOD score.

Original: Although the feature maps from multiple layers are utilized, the method still does dimensionality reduction to those feature maps (e.g. averaging).
Revised: Although the feature maps from multiple layers are utilized, the method still does dimensionality reduction to those feature maps (e.g. averaging).

Original: Therefore, the method is breakable by our OOD
Revised: Therefore, the method is breakable by our OOD

Original: Attack algorithm.
Revised: Attack algorithm.

Original: D. E VALUATION ON CIFAR10 AND CIFAR 
Revised: D. E VALUATION ON CIFAR10 AND CIFAR

Original: The neural network model is a residual network named Resnet34 in the Mahalanobis paper, and by changing the number of outputs, it can be used for CIFAR10 and CIFAR100.
Revised: The neural network model is a residual network named Resnet34 in the Mahalanobis paper, and by changing the number of outputs, it can be used for CIFAR10 and CIFAR100.

Original: We used the pre-trained models that are available online at https://github.com/pokaxpoka/deep
Revised: We used the pre-trained models that are available online at https://github.com/pokaxpoka/deep

Original: Mahalanobis detector/.
Revised: Mahalanobis detector/.

Original: The layers used for feature extraction, are exactly the same as those in the source code of the method.
Revised: The layers used for feature extraction, are exactly the same as those in the source code of the method.

Original: In our algorithm, we have two different settings for f ( x ) : (1) it can be the OOD score, and (2) it can be the concatenation of the feature vectors, given the original (not preprocessed) input.
Revised: In our algorithm, we have two different settings for f ( x ) : (1) it can be the OOD score, and (2) it can be the concatenation of the feature vectors, given the original (not preprocessed) input.

Original: We did experiments with the two settings.
Revised: We did experiments with the two settings.

Original: In our algorithm, the parameters are ε = 10 , α = ε/ 100 , N = 1000 for all experiments.
Revised: In our algorithm, the parameters are ε = 10 , α = ε/ 100 , N = 1000 for all experiments.

Original: The initial OOD sample is a random noise image.
Revised: The initial OOD sample is a random noise image.

Original: For every sample in the test set, the algorithm generated an OOD sample to match the corresponding output.
Revised: For every sample in the test set, the algorithm generated an OOD sample to match the corresponding output.

Original: The generated
Revised: The generated

Original: OOD samples look like random noises.
Revised: OOD samples look like random noises.

Original: The OOD scores of these samples were calculated by the
Revised: The OOD scores of these samples were calculated by the

Original: Mahalanobis method.
Revised: Mahalanobis method.

Original: The results on the two datasets are reported in Table 3 and Table 4.
Revised: The results on the two datasets are reported in Table 3 and Table 4.

Original: Fig.
Revised: Fig.

Original: 14 shows the OOD score histograms of the in-distribution and OOD samples, when the in-distribution dataset is CIFAR10,
Revised: 14 shows the OOD score histograms of the in-distribution and OOD samples, when the in-distribution dataset is CIFAR10,

Original: PM=0.01, and f ( x ) =
Revised: PM=0.01, and f ( x ) =

Original: OOD score.
Revised: OOD score.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Fig.
Revised: Fig.

Original: 15 shows the OOD score histograms of the in-distribution and OOD samples, when the indistribution dataset is CIFAR10, PM=0.01, and f ( x ) = feature concatenation.
Revised: 15 shows the OOD score histograms of the in-distribution and OOD samples, when the indistribution dataset is CIFAR10, PM=0.01, and f ( x ) = feature concatenation.

Original: It can be seen that the
Revised: It can be seen that the

Original: OOD samples have smaller distances, which is caused by input preprocessing.
Revised: OOD samples have smaller distances, which is caused by input preprocessing.

Original: E A PPENDIX
Revised: E A PPENDIX

Original: We applied our OOD Attack algorithm to test the OOD detection method named Outlier Exposure (Hendrycks et al., 2019).
Revised: We applied our OOD Attack algorithm to test the OOD detection method named Outlier Exposure (Hendrycks et al., 2019).

Original: E. S UMMARY OF THE O UTLIER E XPOSURE METHOD
Revised: E. S UMMARY OF THE O UTLIER E XPOSURE METHOD

Original: The method trains a neural network on not only the standard training set (i.e. in distribution) but also an auxiliary dataset of outliers (i.e. OOD samples).
Revised: The method trains a neural network on not only the standard training set (i.e. in distribution) but also an auxiliary dataset of outliers (i.e. OOD samples).

Original: In the paper, it states that the OOD score is deﬁned to be the cross-entropy between a uniform distribution and the softmax-output distribution.
Revised: In the paper, it states that the OOD score is deﬁned to be the cross-entropy between a uniform distribution and the softmax-output distribution.

Original: In the actual implementation (i.e. source code of the method), the OOD score is deﬁned to be the average of the logits minus the logsumexp of the logits.
Revised: In the actual implementation (i.e. source code of the method), the OOD score is deﬁned to be the average of the logits minus the logsumexp of the logits.

Original: In the evaluation, we used the actual implementation in the source code.
Revised: In the evaluation, we used the actual implementation in the source code.

Original: E. E VALUATION ON SVHN, CIFAR10 AND CIFAR 
Revised: E. E VALUATION ON SVHN, CIFAR10 AND CIFAR

Original: Wide residual networks are used in the Outlier Exposure paper.
Revised: Wide residual networks are used in the Outlier Exposure paper.

Original: We downloaded the source code and pre-trained weights from https://github.com/hendrycks/outlier-exposure.
Revised: We downloaded the source code and pre-trained weights from https://github.com/hendrycks/outlier-exposure.

Original: The models were trained from scratch using Outlier Exposure, and they were named “oe scratch” by the authors.
Revised: The models were trained from scratch using Outlier Exposure, and they were named “oe scratch” by the authors.

Original: In our algorithm, we set f ( x ) to be the logits (input to softmax) from each model.
Revised: In our algorithm, we set f ( x ) to be the logits (input to softmax) from each model.

Original: The parameters are ε =10, α = ε/ 100 , N =1e4 for all experiments.
Revised: The parameters are ε =10, α = ε/ 100 , N =1e4 for all experiments.

Original: The initial OOD sample is a random noise image.
Revised: The initial OOD sample is a random noise image.

Original: For every sample in the test set, the algorithm generated an OOD sample to match the logits output.
Revised: For every sample in the test set, the algorithm generated an OOD sample to match the logits output.

Original: The generated OOD samples look like random noises.
Revised: The generated OOD samples look like random noises.

Original: The OOD scores of these samples were calculated by the Outlier Exposure method.
Revised: The OOD scores of these samples were calculated by the Outlier Exposure method.

Original: The results are reported in Table 5.
Revised: The results are reported in Table 5.

Original: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.
Revised: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.

Original: 16, where the in-distribution dataset is CIFAR10.
Revised: 16, where the in-distribution dataset is CIFAR10.

Original: [Figure]
Revised: [Figure]

Original: F A PPENDIX
Revised: F A PPENDIX

Original: We applied our OOD Attack algorithm to test the OOD detection method named Deep Ensemble (Lakshminarayanan et al., 2017).
Revised: We applied our OOD Attack algorithm to test the OOD detection method named Deep Ensemble (Lakshminarayanan et al., 2017).

Original: F. S UMMARY OF THE D EEP E NSEMBLE METHOD
Revised: F. S UMMARY OF THE D EEP E NSEMBLE METHOD

Original: Deep Ensemble is a collection of neural network models working together for a classiﬁcation task.
Revised: Deep Ensemble is a collection of neural network models working together for a classiﬁcation task.

Original: The output of a Deep Ensemble is a probability distribution across the classes, which is the average of the probability/softmax outputs of individual models.
Revised: The output of a Deep Ensemble is a probability distribution across the classes, which is the average of the probability/softmax outputs of individual models.

Original: In the experiments, the number of models is 5 in a Deep Ensemble.
Revised: In the experiments, the number of models is 5 in a Deep Ensemble.

Original: To further improve performance, adversarial training is applied to the models.
Revised: To further improve performance, adversarial training is applied to the models.

Original: The OOD score is deﬁned to be the entropy of the probability distribution from the Deep
Revised: The OOD score is deﬁned to be the entropy of the probability distribution from the Deep

Original: Ensemble.
Revised: Ensemble.

Original: The entropy is expected to be large for an OOD sample.
Revised: The entropy is expected to be large for an OOD sample.

Original: F. E VALUATE ON CIFAR 
Revised: F. E VALUATE ON CIFAR

Original: The authors of the Deep Ensemble method did not provide source code and trained models.
Revised: The authors of the Deep Ensemble method did not provide source code and trained models.

Original: Therefore, we used pre-trained models from a recent work on adversarial robustness (Ding et al., 2020), which presented a state-of-the-art adversarial training method.
Revised: Therefore, we used pre-trained models from a recent work on adversarial robustness (Ding et al., 2020), which presented a state-of-the-art adversarial training method.

Original: Six pre-trained models were downloaded from https://github.com/BorealisAI/mma training/tree/master/trained models.
Revised: Six pre-trained models were downloaded from https://github.com/BorealisAI/mma training/tree/master/trained models.

Original: The names of the models are cifar10-L2-MMA-1.0-sd0, cifar10-L2-MMA-2.0-sd0, cifar10-L2-OMMA-1.0-sd0, cifar10-L2-OMMA-2.0-sd0, cifar10-Linf-MMA-12-sd0, cifar10-Linf-OMMA-12-sd0.
Revised: The names of the models are cifar10-L2-MMA-1.0-sd0, cifar10-L2-MMA-2.0-sd0, cifar10-L2-OMMA-1.0-sd0, cifar10-L2-OMMA-2.0-sd0, cifar10-Linf-MMA-12-sd0, cifar10-Linf-OMMA-12-sd0.

Original: The models were trained on CIFAR10 to be robust against adversarial noises in a large range.
Revised: The models were trained on CIFAR10 to be robust against adversarial noises in a large range.

Original: Classiﬁcation accuracy of the ensemble on test set is 89.85%.
Revised: Classiﬁcation accuracy of the ensemble on test set is 89.85%.

Original: In our algorithm, we set f ( x ) to be the concatenation of the logits from each of the six models.
Revised: In our algorithm, we set f ( x ) to be the concatenation of the logits from each of the six models.

Original: The parameters are ε =10, α = ε /100, N =1e4 for all experiments.
Revised: The parameters are ε =10, α = ε /100, N =1e4 for all experiments.

Original: The initial OOD sample is a random noise image.
Revised: The initial OOD sample is a random noise image.

Original: For every sample in the test set, the algorithm generated an OOD sample to match the logits output.
Revised: For every sample in the test set, the algorithm generated an OOD sample to match the logits output.

Original: The generated OOD samples look like random noises.
Revised: The generated OOD samples look like random noises.

Original: The OOD scores of these samples were calculated by the Deep Ensemble method.
Revised: The OOD scores of these samples were calculated by the Deep Ensemble method.

Original: The AUROC of the Deep Ensemble method is 0.500 on CIFAR10 vs OOD.
Revised: The AUROC of the Deep Ensemble method is 0.500 on CIFAR10 vs OOD.

Original: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.
Revised: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.

Original: G A PPENDIX
Revised: G A PPENDIX

Original: We applied our OOD Attack algorithm to test the OOD detection method that builds ConﬁdenceCalibrated Classiﬁers (Lee et al., 2018a).
Revised: We applied our OOD Attack algorithm to test the OOD detection method that builds ConﬁdenceCalibrated Classiﬁers (Lee et al., 2018a).

Original: G. S UMMARY OF THE OOD DETECTION METHOD
Revised: G. S UMMARY OF THE OOD DETECTION METHOD

Original: The method jointly trains a classiﬁcation network and a generative neural network (i.e. GAN) that generates OOD samples for training the classiﬁcation network.
Revised: The method jointly trains a classiﬁcation network and a generative neural network (i.e. GAN) that generates OOD samples for training the classiﬁcation network.

Original: Given an input, the OOD score is deﬁned to be the maximum of the softmax outputs from the classiﬁcation network.
Revised: Given an input, the OOD score is deﬁned to be the maximum of the softmax outputs from the classiﬁcation network.

Original: The OOD score is expected to be low for an OOD sample.
Revised: The OOD score is expected to be low for an OOD sample.

Original: G. E VALUATION ON SVHN AND CIFAR 
Revised: G. E VALUATION ON SVHN AND CIFAR

Original: The neural network model is VGG13, and the source code of the method is provided by the authors at https://github.com/alinlab/Conﬁdent classiﬁer.
Revised: The neural network model is VGG13, and the source code of the method is provided by the authors at https://github.com/alinlab/Conﬁdent classiﬁer.

Original: We downloaded the code and trained a VGG model with a GAN on SVHN and another VGG13 model with a GAN on CIFAR10 by using the parameters in the source code.
Revised: We downloaded the code and trained a VGG model with a GAN on SVHN and another VGG13 model with a GAN on CIFAR10 by using the parameters in the source code.

Original: VGG13 has a feature module and a classiﬁer module.
Revised: VGG13 has a feature module and a classiﬁer module.

Original: In our algorithm, we set f ( x ) to be the vector input to the classiﬁer module of VGG13.
Revised: In our algorithm, we set f ( x ) to be the vector input to the classiﬁer module of VGG13.

Original: The parameters are ε =10, α = ε /100, N =1e4 for all experiments.
Revised: The parameters are ε =10, α = ε /100, N =1e4 for all experiments.

Original: The initial OOD sample is a random noise image.
Revised: The initial OOD sample is a random noise image.

Original: For every sample in the test set, the algorithm generated an OOD sample to match the vector input to the classiﬁer module.
Revised: For every sample in the test set, the algorithm generated an OOD sample to match the vector input to the classiﬁer module.

Original: The generated OOD samples look like random noises.
Revised: The generated OOD samples look like random noises.

Original: The OOD scores of these samples were calculated by the OOD detection method.
Revised: The OOD scores of these samples were calculated by the OOD detection method.

Original: The results are reported in Table 

Revised: The results are reported in Table

Original: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.
Revised: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.

Original: 18, where the in-distribution dataset is CIFAR10.
Revised: 18, where the in-distribution dataset is CIFAR10.

Original: [Figure]
Revised: [Figure]

Original: H A PPENDIX
Revised: H A PPENDIX

Original: We applied our OOD Attack algorithm to test the OOD detection method named Gram (Sastry &
Revised: We applied our OOD Attack algorithm to test the OOD detection method named Gram (Sastry &

Original: Oore, 2019).
Revised: Oore, 2019).

Original: H. S UMMARY OF THE G RAM METHOD
Revised: H. S UMMARY OF THE G RAM METHOD

Original: The method extracts the feature map F l from every layer of a network and then computes the p-th order Gram matrix
Revised: The method extracts the feature map F l from every layer of a network and then computes the p-th order Gram matrix

Original: G pl =
Revised: G pl =

Original: ( F pl F plT ) 1 /p .
Revised: ( F pl F plT ) 1 /p .

Original: Gram matrices with different p values from different layers are then used to compute the OOD score, which is named Total Deviation of the input sample.
Revised: Gram matrices with different p values from different layers are then used to compute the OOD score, which is named Total Deviation of the input sample.

Original: An
Revised: An

Original: OOD sample is expected to have a high OOD score.
Revised: OOD sample is expected to have a high OOD score.

Original: H. R ESOLVING A NUMERICAL PROBLEM
Revised: H. R ESOLVING A NUMERICAL PROBLEM

Original: The formula of the p-th order Gram matrix can be written as A =
Revised: The formula of the p-th order Gram matrix can be written as A =

Original: ( B ) 1 /p .
Revised: ( B ) 1 /p .

Original: The Gram matrices caused gradients to be inf or nan during back-propagation in the OOD attack algorithm.
Revised: The Gram matrices caused gradients to be inf or nan during back-propagation in the OOD attack algorithm.

Original: To resolve this problem, we tried three tricks:
Revised: To resolve this problem, we tried three tricks:

Original: (a) use double precision (ﬂoat64) (b) rewrite A = exp ( 1 p log ( B + eps )) where eps =1e- (c) use the equation in (b) to generate images during OOD attack and use the original equation A
Revised: (a) use double precision (ﬂoat64) (b) rewrite A = exp ( 1 p log ( B + eps )) where eps =1e- (c) use the equation in (b) to generate images during OOD attack and use the original equation A

Original: =
Revised: =

Original: ( B ) 1 /p to compute OOD scores.
Revised: ( B ) 1 /p to compute OOD scores.

Original: The above tricks work for p in the range of 1 to 5.
Revised: The above tricks work for p in the range of 1 to 5.

Original: For larger p , we still get numerical problems (inf or nan).
Revised: For larger p , we still get numerical problems (inf or nan).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2 of the Gram paper, the method has already achieved better performance compared to the Mahalanobis method when the max value of p is 5.
Revised: 2 of the Gram paper, the method has already achieved better performance compared to the Mahalanobis method when the max value of p is 5.

Original: Thus, we set the max value of p to 5 in our experiments.
Revised: Thus, we set the max value of p to 5 in our experiments.

Original: H. E VALUATION ON CIFAR10 AND CIFAR 
Revised: H. E VALUATION ON CIFAR10 AND CIFAR

Original: The source code and pre-trained Resnet models are provided by the authors at https://github.com/VectorInstitute/gram-ood-detection
Revised: The source code and pre-trained Resnet models are provided by the authors at https://github.com/VectorInstitute/gram-ood-detection

Original: Due to the unique process of the method, it is very difﬁcult to do parallel computing with minibatches, and we have to set batch size =1.
Revised: Due to the unique process of the method, it is very difﬁcult to do parallel computing with minibatches, and we have to set batch size =1.

Original: The computing process is very time-consuming, and therefore we selected the ﬁrst 500 samples in CIFAR10 test set and the ﬁrst 500 samples in CIFAR100 test set in our experiments.
Revised: The computing process is very time-consuming, and therefore we selected the ﬁrst 500 samples in CIFAR10 test set and the ﬁrst 500 samples in CIFAR100 test set in our experiments.

Original: In our algorithm, we set f ( x ) to be the OOD score of x , and the parameters are ε =10, α = ε /100,
Revised: In our algorithm, we set f ( x ) to be the OOD score of x , and the parameters are ε =10, α = ε /100,

Original: N =100.
Revised: N =100.

Original: The initial OOD sample is a random noise image.
Revised: The initial OOD sample is a random noise image.

Original: For every in-distribution sample, the algorithm generated an OOD sample to match the OOD score.
Revised: For every in-distribution sample, the algorithm generated an OOD sample to match the OOD score.

Original: The generated OOD samples look like random noises.
Revised: The generated OOD samples look like random noises.

Original: The OOD scores of these samples were calculated by the Gram method.
Revised: The OOD scores of these samples were calculated by the Gram method.

Original: The results are reported in Table 7.
Revised: The results are reported in Table 7.

Original: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.
Revised: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.

Original: 19, where the in-distribution dataset is CIFAR
Revised: 19, where the in-distribution dataset is CIFAR

Original: The results show that the OOD score from the Gram method can be arbitrarily manipulated by our algorithm
Revised: The results show that the OOD score from the Gram method can be arbitrarily manipulated by our algorithm

Original: [Table] [Table]
Revised: [Table] [Table]

Original: I A PPENDIX
Revised: I A PPENDIX

Original: We applied our OOD Attack algorithm to test the OOD detection method based on Glow (Serr ` a et al., 2019).
Revised: We applied our OOD Attack algorithm to test the OOD detection method based on Glow (Serr ` a et al., 2019).

Original: I. S UMMARY OF THE OOD DETECTION METHOD
Revised: I. S UMMARY OF THE OOD DETECTION METHOD

Original: The method combines Glow negative log-likelihood (NLL) and input-complexity.
Revised: The method combines Glow negative log-likelihood (NLL) and input-complexity.

Original: PNG compression is used to compress the input image.
Revised: PNG compression is used to compress the input image.

Original: The input-complexity, L, is measured by bits per dimension, where the “bits” refers to the number of bits of the compressed image, and the dimension is the total number of pixels per image.
Revised: The input-complexity, L, is measured by bits per dimension, where the “bits” refers to the number of bits of the compressed image, and the dimension is the total number of pixels per image.

Original: The OOD score is NLL – L.
Revised: The OOD score is NLL – L.

Original: I. E VALUATE ON C ELEB A
Revised: I. E VALUATE ON C ELEB A

Original: The source code of the Glow model is downloaded from https://github.com/rosinality/glow-pytorch, and we trained it from scratch on CelbaA dataset, in which the size of each face image is 64643 .
Revised: The source code of the Glow model is downloaded from https://github.com/rosinality/glow-pytorch, and we trained it from scratch on CelbaA dataset, in which the size of each face image is 64643 .

Original: After training, the model was able to generate realistic face images.
Revised: After training, the model was able to generate realistic face images.

Original: For method evaluation, we randomly selected 160 samples in the dataset because the computation cost is very high.
Revised: For method evaluation, we randomly selected 160 samples in the dataset because the computation cost is very high.

Original: In our algorithm, we set f ( x ) to be the OOD score of x , and the parameters are ε =10, α = ε /100,
Revised: In our algorithm, we set f ( x ) to be the OOD score of x , and the parameters are ε =10, α = ε /100,

Original: N =1e4.
Revised: N =1e4.

Original: The initial OOD sample is a color spiral image.
Revised: The initial OOD sample is a color spiral image.

Original: For every in-distribution sample, the algorithm generated an OOD sample to match the OOD score.
Revised: For every in-distribution sample, the algorithm generated an OOD sample to match the OOD score.

Original: The generated OOD samples look like color spirals, i.e., not face images.
Revised: The generated OOD samples look like color spirals, i.e., not face images.

Original: The OOD scores of these samples were calculated by the
Revised: The OOD scores of these samples were calculated by the

Original: OOD detection method.
Revised: OOD detection method.

Original: AUROC of the method is 0.500 on CelbaA vs OOD.
Revised: AUROC of the method is 0.500 on CelbaA vs OOD.

Original: The OOD score histograms of the indistribution and OOD samples are shown in Fig.
Revised: The OOD score histograms of the indistribution and OOD samples are shown in Fig.

Original: 20, The result indicates that NLL combined with input complexity can still be arbitrarily manipulated.
Revised: 20, The result indicates that NLL combined with input complexity can still be arbitrarily manipulated.

Original: J A PPENDIX
Revised: J A PPENDIX

Original: We applied our OOD Attack algorithm to test the OOD detection method using an energy-based model named JEM (Grathwohl et al., 2020).
Revised: We applied our OOD Attack algorithm to test the OOD detection method using an energy-based model named JEM (Grathwohl et al., 2020).

Original: J. S UMMARY OF THE OOD DETECTION METHOD USING JEM
Revised: J. S UMMARY OF THE OOD DETECTION METHOD USING JEM

Original: In the JEM paper, it was shown that a standard classiﬁer can be trained to be an energy-based model (EBM).
Revised: In the JEM paper, it was shown that a standard classiﬁer can be trained to be an energy-based model (EBM).

Original: Using the EBM, three types of OOD scores can be obtained for an input sample, which are: (1) Log Likelihood logp ( x ) , (2) the maximum of softmax classiﬁcation output, i.e. max y p ( y | x ) , and (3) −|| ∂logp ( x ) ∂x || 2 .
Revised: Using the EBM, three types of OOD scores can be obtained for an input sample, which are: (1) Log Likelihood logp ( x ) , (2) the maximum of softmax classiﬁcation output, i.e. max y p ( y | x ) , and (3) −|| ∂logp ( x ) ∂x || 2 .

Original: An OOD sample is expected to have a low OOD score.
Revised: An OOD sample is expected to have a low OOD score.

Original: J. E VALUATION ON CIFAR 
Revised: J. E VALUATION ON CIFAR

Original: A wide residual network pretrained on CIFAR10 is available at https://github.com/wgrathwohl/JEM.
Revised: A wide residual network pretrained on CIFAR10 is available at https://github.com/wgrathwohl/JEM.

Original: In our algorithm, we set f ( x ) to be the logits (i.e. input to softmax for classiﬁcation), and the parameters are ε =10, α = ε /100, N =1e3.
Revised: In our algorithm, we set f ( x ) to be the logits (i.e. input to softmax for classiﬁcation), and the parameters are ε =10, α = ε /100, N =1e3.

Original: The initial OOD sample is a color spiral image.
Revised: The initial OOD sample is a color spiral image.

Original: For every in-distribution sample, the algorithm generated an OOD sample to match the logits output.
Revised: For every in-distribution sample, the algorithm generated an OOD sample to match the logits output.

Original: The generated OOD samples look very weird, not in any of the 10 classes of CIFAR10.
Revised: The generated OOD samples look very weird, not in any of the 10 classes of CIFAR10.

Original: The OOD scores of these samples were calculated by the OOD detection method.
Revised: The OOD scores of these samples were calculated by the OOD detection method.

Original: We note that we tried to use random noises as initial OOD samples, but many generated images look like images in CIFAR10 dataset, and therefore, we used a color spiral image as the initial OOD sample.
Revised: We note that we tried to use random noises as initial OOD samples, but many generated images look like images in CIFAR10 dataset, and therefore, we used a color spiral image as the initial OOD sample.

Original: The results are reported in Table 8.
Revised: The results are reported in Table 8.

Original: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.
Revised: The OOD score histograms of the in-distribution and OOD samples are shown in Fig.

Original: 21, Fig. 22, and Fig.
Revised: 21, Fig. 22, and Fig.

Original: 23, We note that when using −|| ∂logp ( x ) ∂x || 2 as the
Revised: 23, We note that when using −|| ∂logp ( x ) ∂x || 2 as the

Original: OOD score, the AUROC is 0.203.
Revised: OOD score, the AUROC is 0.203.

Original: One may think if we ﬂip the sign of the OOD score, then AUROC will increase to 0.797.
Revised: One may think if we ﬂip the sign of the OOD score, then AUROC will increase to 0.797.

Original: If we do so, then AUROC scores in the last row of Table 3 in the JEM paper will be close to 0 for the OOD detection experiments done by the authors.
Revised: If we do so, then AUROC scores in the last row of Table 3 in the JEM paper will be close to 0 for the OOD detection experiments done by the authors.

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 24 shows an example of the loss curve over 1000 iteration in the OOD attack algorithm.
Revised: 24 shows an example of the loss curve over 1000 iteration in the OOD attack algorithm.

Original: Fig.
Revised: Fig.

Original: 25 shows some of the generated images, which look like the images of Frankenstein’s monsters: randomly put some parts of objects together, twist/deform them, and then pour some paint on them.
Revised: 25 shows some of the generated images, which look like the images of Frankenstein’s monsters: randomly put some parts of objects together, twist/deform them, and then pour some paint on them.

Original: It may be difﬁcult for neural networks to learn what is an object (e.g. airplane) just from images and class labels.
Revised: It may be difﬁcult for neural networks to learn what is an object (e.g. airplane) just from images and class labels.

Original: Energy-based models (EBMs), such as JEM, can generate OOD samples during training, which may explain why the OOD attack failed when the initial OOD sample was random noise.
Revised: Energy-based models (EBMs), such as JEM, can generate OOD samples during training, which may explain why the OOD attack failed when the initial OOD sample was random noise.

Original: If we take a closer look at the sampling procedure (e.g. Langevin dynamics) and the objective function, it is easy to ﬁnd out EBM training algorithm is trying to pull down the energy scores of positive (indistribution) samples and pull up the energy scores of negative (OOD) samples (Du & Mordatch, 2019), which is similar to the basic idea of adversarial training.
Revised: If we take a closer look at the sampling procedure (e.g. Langevin dynamics) and the objective function, it is easy to ﬁnd out EBM training algorithm is trying to pull down the energy scores of positive (indistribution) samples and pull up the energy scores of negative (OOD) samples (Du & Mordatch, 2019), which is similar to the basic idea of adversarial training.

Original: From this perspective, OOD detection using EBMs could be a promising direction if the computation cost is acceptable, and the challenge is how to train a neural network to learn what is an object?
Revised: From this perspective, OOD detection using EBMs could be a promising direction if the computation cost is acceptable, and the challenge is how to train a neural network to learn what is an object?