Original: E FFICIENT O UT - OF -D ISTRIBUTION D ETECTION VIA CVAE DATA G ENERATION
Revised: E FFICIENT O UT - OF -D ISTRIBUTION D ETECTION VIA CVAE DATA G ENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, contrastive loss with data augmentation and pseudo class creation has been shown to produce markedly better results for out-of-distribution (OOD) detection than previous methods.
Revised: Recently, contrastive loss with data augmentation and pseudo class creation has been shown to produce markedly better results for out-of-distribution (OOD) detection than previous methods.

Original: However, a major shortcoming of this approach is that it is extremely slow due to signiﬁcant increase in the data size and the number of classes and the quadratic complexity of pairwise similarity computation.
Revised: However, a major shortcoming of this approach is that it is extremely slow due to signiﬁcant increase in the data size and the number of classes and the quadratic complexity of pairwise similarity computation.

Original: This paper proposes a novel and simple method that can build an effective data generator using Conditional Variational Auto-Encoder (CVAE) to generate pseudo OOD samples.
Revised: This paper proposes a novel and simple method that can build an effective data generator using Conditional Variational Auto-Encoder (CVAE) to generate pseudo OOD samples.

Original: Based on the generated pseudo OOD data, a ﬂexible and efﬁcient OOD detection method is proposed through ﬁne-tuning, which achieves results comparable to the state-of-the-art OOD detection techniques, but the execution speed is at least 10 times faster.
Revised: Based on the generated pseudo OOD data, a ﬂexible and efﬁcient OOD detection method is proposed through ﬁne-tuning, which achieves results comparable to the state-of-the-art OOD detection techniques, but the execution speed is at least 10 times faster.

Original: Also importantly, the proposed approach is in fact a general framework that can be applied to many existing OOD methods and improve them via the proposed ﬁne-tuning.
Revised: Also importantly, the proposed approach is in fact a general framework that can be applied to many existing OOD methods and improve them via the proposed ﬁne-tuning.

Original: We have combined it with the best baseline
Revised: We have combined it with the best baseline

Original: OOD models in our experiments to produce new state-of-the-art results.
Revised: OOD models in our experiments to produce new state-of-the-art results.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the development of deep learning, a well-trained neural network model is able to obtain very high accuracy on its testing data.
Revised: With the development of deep learning, a well-trained neural network model is able to obtain very high accuracy on its testing data.

Original: However, when exposed to samples or data instances drawn from a distribution that is far from the training distribution (called In-distribution (IND)), the model may make arbitrary predictions under the known framework (Nguyen et al., 2015; Recht et al., 2019).
Revised: However, when exposed to samples or data instances drawn from a distribution that is far from the training distribution (called In-distribution (IND)), the model may make arbitrary predictions under the known framework (Nguyen et al., 2015; Recht et al., 2019).

Original: This limits the model’s application in a broad range of applications, including secure authentication (Sharif et al., 2016), autonomous driving (Nitsch et al., 2020) and medical diagnosis (Caruana et al., 2015) as in these applications novel or out-of distribution (OOD) data instances occur frequently.
Revised: This limits the model’s application in a broad range of applications, including secure authentication (Sharif et al., 2016), autonomous driving (Nitsch et al., 2020) and medical diagnosis (Caruana et al., 2015) as in these applications novel or out-of distribution (OOD) data instances occur frequently.

Original: Therefore, OOD detection (Hodge & Austin, 2004), which aims to detect abnormal or novel data that are very different from the training data, is an important research task.
Revised: Therefore, OOD detection (Hodge & Austin, 2004), which aims to detect abnormal or novel data that are very different from the training data, is an important research task.

Original: A variety of approaches have been proposed to solve this problem, from distance-based methods (Bendale & Boult, 2015; 2016;
Revised: Many approaches have been proposed to solve this problem, from distance-based methods (Bendale & Boult, 2015; 2016;

Original: Gunther et al., 2017; J ´ unior et al., 2017),
Revised: Gunther et al., 2017; J ´ unior et al., 2017),

Original: to generative models (Ge et al., 2017; Neal et al., 2018; Oza & Patel, 2019; Nalisnick et al., 2018) and self-supervised learning (Bergman & Hoshen, 2020; Golan & El-Yaniv, 2018; Hendrycks et al., 2019).
Revised: to generative models (Ge et al., 2017; Neal et al., 2018; Oza & Patel, 2019; Nalisnick et al., 2018) and self-supervised learning (Bergman & Hoshen, 2020; Golan & El-Yaniv, 2018; Hendrycks et al., 2019).

Original: Recently, contrastive learning has been shown highly effective in many applications (Hjelm et al., 2018;
Revised: Recently, contrastive learning has been shown highly effective in many applications (Hjelm et al., 2018;

Original: Oord et al., 2018; Chen et al., 2020a).
Revised: Oord et al., 2018; Chen et al., 2020b;a; Falcon & Cho, 2020).

Original: When combined with data augmentation, it improves the quality of representation learning greatly (Chen et al., 2020b;a; Falcon & Cho, 2020).

Original: Applying supervised contrastive learning and data augmentation, the recent CSI system has produced state-of-the-art (SOTA)
Revised: Applying supervised contrastive learning and data augmentation, the recent CSI system has produced state-of-the-art (SOTA)

Original: OOD detection results (Tack et al., 2020).
Revised: OOD detection results (Tack et al., 2020).

Original: However, data augmentation-based contrastive learning also has some drawbacks.
Revised: However, data augmentation-based contrastive learning also has some drawbacks.

Original: First, designing data augmentation functions and deciding how to use various types of augmented data in contrastive learning involve a great deal of trial and error and manual work.
Revised: First, designing data augmentation functions and deciding how to use various types of augmented data in contrastive learning involve a great deal of trial and error and manual work.

Original: That is, there is a large number of transformations (or augmentations) such as cropping, rotation and gray-scaling that can be exerted on images but not all of them may beneﬁt the end tasks.
Revised: That is, there is a large number of transformations (or augmentations) such as cropping, rotation and gray-scaling that can be exerted on images but not all of them may beneﬁt the end tasks.

Original: In SimCLR (Chen et al., 2020a), systematic experiments have been reported to explore the augmentations’ inﬂuence on classiﬁcation tasks.
Revised: In SimCLR (Chen et al., 2020a), systematic experiments have been reported to explore the augmentations’ inﬂuence on classiﬁcation tasks.

Original: Based on SimCLR’s results, CSI (Tack et al., 2020) chooses several operations for OOD detection.
Revised: Based on SimCLR’s results, CSI (Tack et al., 2020) chooses several operations for OOD detection.

Original: Second, contrastive learning with data augmentation is very time-consuming to run and resourcehungry due to a large amount of augmented data and quadratic pairwise similarity computation during training.
Revised: Second, contrastive learning with data augmentation is very time-consuming to run and resourcehungry due to a large amount of augmented data and quadratic pairwise similarity computation during training.

Original: For example, CSI creates 3 shifted instances for each original image sample and the 4 images are then subjected to an additional augmentation operation.
Revised: For example, CSI creates 3 shifted instances for each original image sample and the 4 images are then subjected to an additional augmentation operation.

Original: Each image is ﬁnally expanded to 8 images or samples.
Revised: Each image is ﬁnally expanded to 8 images or samples.

Original: Furthermore, every 2 samples in the augmented batch is treated as a pair to calculate contrastive loss.
Revised: Furthermore, every 2 samples in the augmented batch is treated as a pair to calculate contrastive loss.

Original: The performance is negatively affected if the batch size is not large enough.
Revised: The performance is negatively affected if the batch size is not large enough.

Original: Therefore, it is necessary to use a large batch size, which demands a huge amount of memory and takes a very long time to run.
Revised: Therefore, it is necessary to use a large batch size, which demands a huge amount of memory and takes a very long time to run.

Original: It is thus not suitable for applications on hardware devices that do not have the required resources such as edge devices.
Revised: It is thus not suitable for applications on hardware devices that do not have the required resources such as edge devices.

Original: In this paper, we propose a general and much more efﬁcient solution, called CGA ( CVAE-based
Revised: In this paper, we propose a general and much more efﬁcient solution, called CGA ( CVAE-based

Original: Generative data Augmentation for OOD detection ).
Revised: Generative data Augmentation for OOD detection ).

Original: CGA consists of two parts.
Revised: CGA consists of two parts.

Original: The ﬁrst part generates pseudo OOD data and the second part makes use of the pseudo OOD data to train an OOD detection model.
Revised: The ﬁrst part generates pseudo OOD data and the second part makes use of the pseudo OOD data to train an OOD detection model.

Original: We discuss the ﬁrst part ﬁrst.
Revised: We discuss the ﬁrst part ﬁrst.

Original: OOD detection is basically a classiﬁcation problem but the challenge is that there is no OOD data to use in training.
Revised: OOD detection is basically a classiﬁcation problem but the challenge is that there is no OOD data to use in training.

Original: This paper proposes a novel method to generate pseudo OOD data.
Revised: This paper proposes a novel method to generate pseudo OOD data.

Original: This method works in the latent space of a Conditional Variational
Revised: This method works in the latent space of a Conditional Variational

Original: Auto-Encoder (CVAE) and uses CVAE’s decoder to generate pseudo OOD data.
Revised: Auto-Encoder (CVAE) and uses CVAE’s decoder to generate pseudo OOD data.

Original: CVAE is able to generate instances from the training distribution on the basis of latent representations consisting of conditional information and variables sampled from a prior distribution of CVAE, normally the
Revised: CVAE is able to generate instances from the training distribution on the basis of latent representations consisting of conditional information and variables sampled from a prior distribution of CVAE, normally the

Original: Gaussian distribution.
Revised: Gaussian distribution.

Original: If the latent space features or representations are created with some abnormal conditional information , the CVAE will generate “bad” instances but such instances can serve as effective pseudo OOD data.
Revised: If the latent space features or representations are created with some abnormal conditional information , the CVAE will generate “bad” instances but such instances can serve as effective pseudo OOD data.

Original: The second part of CGA is a ﬁne-tuning framework that can make use of the generated pseudo OOD data to ﬁne-tune any classiﬁcation model built using only the IND data.
Revised: The second part of CGA is a ﬁne-tuning framework that can make use of the generated pseudo OOD data to ﬁne-tune any classiﬁcation model built using only the IND data.

Original: Applying the framework to a simple IND classiﬁcation model, we can already produce comparable results to existing SOTA contrastive learning models but much more efﬁcient.
Revised: Applying the framework to a simple IND classiﬁcation model, we can already produce comparable results to existing SOTA contrastive learning models but much more efﬁcient.

Original: Applying it to the existing SOTA methods, we can improve their results too.
Revised: Applying it to the existing SOTA methods, we can improve their results too.

Original: Our contributions can be summarized as follows: (1) We propose to employ a CVAE structure to generate pseudo OOD samples by providing some synthetic conditional information, which, to our knowledge, has not been done before.
Revised: Our contributions can be summarized as follows: (1) We propose to employ a CVAE structure to generate pseudo OOD samples by providing some synthetic conditional information, which, to our knowledge, has not been done before.

Original: (2) We design a two-stage framework to train an OOD detection model by leveraging the generated pseudo OOD data.
Revised: (2) We design a two-stage framework to train an OOD detection model by leveraging the generated pseudo OOD data.

Original: The ﬁrst stage simply builds a normal classiﬁcation model using only the IND data.
Revised: The ﬁrst stage simply builds a normal classiﬁcation model using only the IND data.

Original: The second stage ﬁne-tunes the model using the generated OOD data to produce an effective OOD detection model.
Revised: The second stage ﬁne-tunes the model using the generated OOD data to produce an effective OOD detection model.

Original: Extensive experiments show that this approach achieves comparable performance to the state-of-the-art (SOTA) contrastive learning methods in OOD detection, while consuming only one-tenth of the execution time.
Revised: Extensive experiments show that this approach achieves comparable performance to the state-of-the-art (SOTA) contrastive learning methods in OOD detection, while consuming only one-tenth of the execution time.

Original: (3) Equally importantly, the proposed framework can be applied to existing SOTA OOD detection models to improve them to produce new SOTA results.
Revised: (3) Equally importantly, the proposed framework can be applied to existing SOTA OOD detection models to improve them to produce new SOTA results.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Out-of-distribution Detection.
Revised: Out-of-distribution Detection.

Original: It is well-known that the discriminative neural networks can produce overconﬁdent predictions on out-of-distribution (OOD) inputs (Hendrycks & Gimpel, 2016;
Revised: It is well-known that the discriminative neural networks can produce overconﬁdent predictions on out-of-distribution (OOD) inputs (Hendrycks & Gimpel, 2016;

Original: Lakshminarayanan et al., 2016).
Revised: Lakshminarayanan et al., 2016).

Original: The early idea to solve this problem focuses on modifying softmax scores to obtain calibrated conﬁdence for OOD detection (Bendale & Boult, 2016;
Revised: The early idea to solve this problem focuses on modifying softmax scores to obtain calibrated conﬁdence for OOD detection (Bendale & Boult, 2016;

Original: Guo et al., 2017).
Revised: Guo et al., 2017).

Original: In addition, many other score functions have been proposed, such as likelihood ratio (Ren et al., 2019), input complexity (Serr`a et al., 2019) and typicality (Nalisnick et al., 2019).
Revised: In addition, many other score functions have been proposed, such as likelihood ratio (Ren et al., 2019), input complexity (Serr`a et al., 2019) and typicality (Nalisnick et al., 2019).

Revised: A recent work utilizes Gram matrices to characterize activity patterns and identify anomalies in Gram matrix values to do OOD detection (Sastry & Oore, 2020).

Original: Some methods found that auxiliary anomalous data signiﬁcantly improve detection performance (Hendrycks et al., 2018; Mohseni et al., 2020) and thus generative models are adopted to anticipate the distribution of novel samples.
Revised: Some methods found that auxiliary anomalous data signiﬁcantly improve detection performance (Hendrycks et al., 2018; Mohseni et al., 2020) and thus generative models are adopted to anticipate the distribution of novel samples.

Original: In some of these methods, generated data are treated as OOD samples to optimize the decision boundary and calibrate the conﬁdence (Ge et al., 2017; Vernekar et al., 2019).
Revised: In some of these methods, generated data are treated as OOD samples to optimize the decision boundary and calibrate the conﬁdence (Ge et al., 2017; Vernekar et al., 2019).

Original: In some other works, generative models such as auto-encoders (Zong et al., 2018; Pidhorskyi et al., 2018) and generative adversarial networks (GAN) (Deecke et al., 2018; Perera et al., 2019) are used to reconstruct the training data.
Revised: In some other works, generative models such as auto-encoders (Zong et al., 2018; Pidhorskyi et al., 2018) and generative adversarial networks (GAN) (Deecke et al., 2018; Perera et al., 2019) are used to reconstruct the training data.

Revised: During the training of a GAN model, low quality samples acquired by the generator can also work as OOD data directly (Pourreza et al., 2021).

Original: Owing to the fact that the model can hardly be generalized to unknown data, the reconstruction loss can help detect OOD data.
Revised: Owing to the fact that the model can hardly be generalized to unknown data, the reconstruction loss can help detect OOD data.

Original: There are also works using auxiliary OOD data to ﬁne-tune the model (Liu et al., 2020). 
Revised: There are also works using auxiliary OOD data to ﬁne-tune the model (Liu et al., 2020). Fort et al.

Revised: (2021) showed that using pre-trained representations and taking few-shot outlier exposure can improve the results.

Original: Recently, self-supervised techniques have been applied to OOD detection.
Revised: Recently, self-supervised techniques have been applied to OOD detection.

Original: It focuses on acquiring rich representations through training with some pre-deﬁned tasks (Gidaris et al., 2018; Kolesnikov et al., 2019).
Revised: It focuses on acquiring rich representations through training with some pre-deﬁned tasks (Gidaris et al., 2018; Kolesnikov et al., 2019).

Original: Self-supervised models show outstanding performance on OOD detection tasks (Kolesnikov et al., 2019; Bergman & Hoshen, 2020).
Revised: Self-supervised models show outstanding performance on OOD detection tasks (Kolesnikov et al., 2019; Bergman & Hoshen, 2020).

Original: CSI (Tack et al., 2020) is a representative method (see more below), which uses contrastive learning and data augmentation to improve the features of all labeled IND data and produce state-of-the-art (SOTA) results.
Revised: CSI (Tack et al., 2020) is a representative method (see more below), which uses contrastive learning and data augmentation to improve the features of all labeled IND data and produce state-of-the-art (SOTA) results.

Original: Some researchers also tried to improve contrastive learning based methods (Sehwag et al., 2021), and some also explore distance-based methods (Miller et al., 2021).
Revised: Some researchers also tried to improve contrastive learning based methods (Sehwag et al., 2021) and proposed distance-based methods (Miller et al., 2021).

Original: Although these new methods get outstanding results on some datasets, based on our experiments, CSIstill outperforms them on average.
Revised: However, based on our experiments, CSI outperforms them.

Original: Our method falls into the generative approach and makes use of generated data to assist training OOD models.
Revised: Our method falls into the generative approach.

Original: Unlike existing methods that use perturbations to anticipate OOD data, our method uses synthetic conditions and CVAE to obtain effective and diverse pseudo OOD data.
Revised: Unlike existing methods that use perturbations to anticipate OOD data, our method uses synthetic conditions and CVAE to obtain effective and diverse pseudo OOD data.

Original: Contrastive Learning .
Revised: Contrastive Learning .

Original: Contrastive learning learns representations by contrasting positive pairs against negative pairs (Hadsell et al., 2006).
Revised: Contrastive learning learns representations by contrasting positive pairs against negative pairs (Hadsell et al., 2006).

Original: It has been applied to various domains (Oord et al., 2018).
Revised: It has been applied to various domains (Oord et al., 2018).

Original: Recently, a new method called SimCLR (Chen et al., 2020a) was proposed to create sample pairs via data augmentation.
Revised: Recently, a new method called SimCLR (Chen et al., 2020a) was proposed to create sample pairs via data augmentation.

Original: It is effective but also very time and resource consuming.
Revised: It is effective but also very time and resource consuming.

Original: SimCLR also shows that contrastive learning beneﬁts more from larger batch sizes and longer training time.
Revised: SimCLR also shows that contrastive learning beneﬁts more from larger batch sizes and longer training time.

Original: CSI (Tack et al., 2020) proposes that augmentation can not only help construct positive pairs but also negative pairs and makes use of them to detect OOD samples with supervised contrastive learning.
Revised: CSI (Tack et al., 2020) proposes that augmentation can not only help construct positive pairs but also negative pairs and makes use of them to detect OOD samples with supervised contrastive learning.

Original: It obtains the SOTA OOD detection results with labeled IND data.
Revised: It obtains the SOTA OOD detection results with labeled IND data.

Original: However, due to contrastive learning, it is extremely slow and memory demanding.
Revised: However, due to contrastive learning, it is extremely slow and memory demanding.

Original: Our proposed method generates pseudo
Revised: Our proposed method generates pseudo

Original: OOD using CVAE, avoiding the use of contrastive loss, and is much more efﬁcient than CSI.
Revised: OOD using CVAE, avoiding the use of contrastive loss, and is much more efﬁcient than CSI.

Original: Auto-Encoder Auto-Encoder (AE) is a family of unsupervised neural networksand has a long history (Rumelhart et al., 1986; Baldi & Hornik, 1989).
Revised: Auto-Encoder Auto-Encoder (AE) is a family of unsupervised neural networks (Rumelhart et al., 1986; Baldi & Hornik, 1989).

Original: A basic AE consists of an encoder and a decoder.
Revised: A basic AE consists of an encoder and a decoder.

Original: The encoder encodes the input data into a low-dimensional hidden representation and the decoder transforms the representation back to the reconstructed input data (Vincent et al., 2008; Chen et al., 2012; Hinton et al., 2006).
Revised: The encoder encodes the input data into a low-dimensional hidden representation and the decoder transforms the representation back to the reconstructed input data (Vincent et al., 2008; Chen et al., 2012; Hinton et al., 2006).

Original: AE has been applied to perform data generation.

Original: Variational auto-encoder is a special kind of AE (Kingma & Welling, 2013).
Revised: Variational auto-encoder is a special kind of AE (Kingma & Welling, 2013).

Original: It encodes the input as a given probability distribution (usually Gaussian) and the decoder reconstructs data instances according to variables sampled from that distribution.
Revised: It encodes the input as a given probability distribution (usually Gaussian) and the decoder reconstructs data instances according to variables sampled from that distribution.

Original: CVAE is an extension of VAE (Kingma et al., 2014).
Revised: CVAE is an extension of VAE (Kingma et al., 2014).

Original: It encodes the label or conditional information into the latent representation so that a CVAE can generate new samples from speciﬁed class labels or categories.
Revised: It encodes the label or conditional information into the latent representation so that a CVAE can generate new samples from speciﬁed class labels.

Original: CVAE makes it possible to control the generating process, i.e., to generate samples with features of speciﬁed classes.
Revised: CVAE makes it possible to control the generating process, i.e., to generate samples with features of speciﬁed classes.

Original: We make use of this property of CVAE to generatepseudo OOD data of high quality by specifying novel conditions.
Revised: We make use of this property of CVAE to generate high quality pseudo OOD data.

Original:  P ROPOSED CGA M ETHOD
Revised:  P ROPOSED CGA M ETHOD

Original: In tasks related to out-of-distribution (OOD) detection, the problem of recognition is commonly formulated as a classiﬁcation problem.
Revised: In tasks related to out-of-distribution (OOD) detection, the problem of recognition is commonly formulated as a classiﬁcation problem.

Original: The main challenge is that an important class, OOD data, is not available.
Revised: The main challenge is that an important class, OOD data, is not available.

Original: Therefore, to effectively train an OOD detection model, an intuitive idea is to generate pseudo OOD data and use them together with the IND data to train the model.
Revised: Therefore, to effectively train an OOD detection model, an intuitive idea is to generate pseudo OOD data and use them together with the IND data to train the model.

Original: As we mentioned earlier, data augmentation and contrastive learning have been shown especially effective for this purpose.
Revised: As we mentioned earlier, data augmentation and contrastive learning have been shown especially effective for this purpose.

Original: However, this approach is extremely inefﬁcient.
Revised: However, this approach is extremely inefﬁcient.

Original: We propose to use Conditional Variational
Revised: We propose to use Conditional Variational

Original: Auto-Encoder (CVAE) to generate pseudo OOD data and present a new ﬁne-tuning framework to leverage the generated pseudo OOD data to train an OOD detection model.
Revised: Auto-Encoder (CVAE) to generate pseudo OOD data and present a new ﬁne-tuning framework to leverage the generated pseudo OOD data to train an OOD detection model.

Original: 3. C ONDITIONAL V ARIATIONAL A UTO - ENCODER
Revised: 3. C ONDITIONAL V ARIATIONAL A UTO - ENCODER

Original: Conditional Variational Auto-Encoder (CVAE) is derived from Variational auto-encoder (VAE).
Revised: Conditional Variational Auto-Encoder (CVAE) is derived from Variational auto-encoder (VAE).

Original: We ﬁrst introduce VAE which is a conditional directed graphical model consisting of three main parts, an encoder q φ ( · ) with parameters φ , a decoder p θ ( · ) with parameters θ and a loss function L ( x ; θ, φ ) , where x represents an input sample.
Revised: We ﬁrst introduce VAE which is a conditional directed graphical model consisting of three main parts, an encoder q φ ( · ) with parameters φ , a decoder p θ ( · ) with parameters θ and a loss function L ( x ; θ, φ ) , where x represents an input sample.

Original: The loss function is as follows:
Revised: The loss function is as follows:

Original: [Equation]
Revised: [Equation]

Original: where q φ ( z | x ) is a proposal distribution to approximate the prior distribution p θ ( z ) , p θ ( x | z ) is the likelihood of the input x with a given latent representation z , and KL ( · ) is the function to calculate Kullback-Leibler divergence.
Revised: where q φ ( z | x ) is a proposal distribution to approximate the prior distribution p θ ( z ) , p θ ( x | z ) is the likelihood of the input x with a given latent representation z , and KL ( · ) is the function to calculate Kullback-Leibler divergence.

Original: q φ ( z | x ) is the encoder and p θ ( x | z ) is the decoder.
Revised: q φ ( z | x ) is the encoder and p θ ( x | z ) is the decoder.

Original: In Eq.(1), the expected negative log-likelihood term encourages the decoder to learn to reconstruct the data with samples from the latent distribution.
Revised: In Eq.(1), the expected negative log-likelihood term encourages the decoder to learn to reconstruct the data with samples from the latent distribution.

Original: The KL-divergence term forces the latent distribution to conform to a speciﬁc prior distribution such as the Gaussian distribution.
Revised: The KL-divergence term forces the latent distribution to conform to a speciﬁc prior distribution such as the Gaussian distribution.

Original: After training, a VAE can generate data using the decoder p θ ( x | z ) with a set of latent variables z sampled from the prior distribution p θ ( z ) .
Revised: After training, a VAE can generate data using the decoder p θ ( x | z ) with a set of latent variables z sampled from the prior distribution p θ ( z ) .

Original: Commonly, the prior distribution is the centered isotropic multivariate Gaussian p θ ( z ) =
Revised: Commonly, the prior distribution is the centered isotropic multivariate Gaussian p θ ( z ) =

Original: N ( z ; 0 , I ) .
Revised: N ( z ; 0 , I ) .

Original: However, VAE does not consider the class label information which is available in classiﬁcation datasets and thus has difﬁculty generating data of a particular class.
Revised: However, VAE does not consider the class label information which is available in classiﬁcation datasets and thus has difﬁculty generating data of a particular class.

Original: Conditional variational Auto-
Revised: Conditional variational Auto-

Original: Encoder (CVAE) was introduced to extend VAE to address this problem.
Revised: Encoder (CVAE) was introduced to extend VAE to address this problem.

Original: It improves the generative process by adding a conditional input information into latent variables so that a CVAE can generate samples with some speciﬁc characteristics or from certain classes.
Revised: It improves the generative process by adding a conditional input information into latent variables so that a CVAE can generate samples with some speciﬁc characteristics or from certain classes.

Original: We use c to denote the prior class information.
Revised: We use c to denote the prior class information.

Original: The loss function for CVAE can be written as follows:
Revised: The loss function for CVAE can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: One implementation of CVAE uses a one-hot vector to represent a class label y c , and a weight matrix is multiplied to it to turn the one-hot vector to a class embedding y c .
Revised: One implementation of CVAE uses a one-hot vector to represent a class label y c , and a weight matrix is multiplied to it to turn the one-hot vector to a class embedding y c .

Original: Then a variable z , generated from the prior distribution p θ ( z ) , is concatenated with y c to construct the whole latent variable.
Revised: Then a variable z , generated from the prior distribution p θ ( z ) , is concatenated with y c to construct the whole latent variable.

Original: Finally, the generated instance p θ ( x | z , c ) of class c is produced.
Revised: Finally, the generated instance p θ ( x | z , c ) of class c is produced.

Original: We can formulate the process as:
Revised: We can formulate the process as:

Original: [Equation]
Revised: [Equation]

Original: 3. G ENERATING P SEUDO OOD DATA
Revised: 3. G ENERATING P SEUDO OOD DATA

Original: The ability of CVAE to control the generating process using the conditional information (e.g. class label in our case) inspired us to design a method to generate possible OOD samples.
Revised: The ability of CVAE to control the generating process using the conditional information (e.g. class label in our case) inspired us to design a method to generate possible OOD samples.

Original: This is done by the conditional decoder using atypical prior information c in p θ ( x | z , c ) .
Revised: This is done by the conditional decoder using atypical prior information c in p θ ( x | z , c ) .

Original: As introduced before, OOD data need to be different from in-distribution data but also resemble them.
Revised: As introduced before, OOD data need to be different from in-distribution data but also resemble them.

Revised: The continuity property of CVAE, which means that two close points in the latent space should not give two completely different contents once decoded (Cemgil et al., 2019), ensures that we can manipulate CVAE latent space features to generate high quality pseudo OOD data.

Original: Since we have no information of the future OOD data, we have to make use of the existing training data (i.e., indistribution data) to construct pseudo OOD data.
Revised: Since we have no information of the future OOD data, we have to make use of the existing training data (i.e., in-distribution data) to construct pseudo OOD data.

Original: Based on the property of CVAE, we can provide it with pseudo label information to generate pseudo OOD data.
Revised: We can provide it with pseudo label information to generate pseudo OOD data.

Original: Speciﬁcally, we propose to construct pseudo class embedding by combining the embeddings of two existing classes in the in-distribution training data.
Revised: Speciﬁcally, we propose to construct pseudo class embedding by combining the embeddings of two existing classes in the in-distribution training data.

Original: The formulation is as follows:
Revised: The formulation is as follows:

Original: [Equation]
Revised: [Equation]

Original: where k is a vector generated from Bernoulli distribution B (0 . 5) with the same length as the class label embedding.
Revised: where k is a vector generated from Bernoulli distribution B (0 . 5) with the same length as the class label embedding.

Original: k is basically for the system to randomly select the vector components of the two class embeddings with equal probability.
Revised: k is basically for the system to randomly select the vector components of the two class embeddings with equal probability.

Original: Such a generated sample p θ ( x | z , k , c i , c j ) will not likely to be an instance of either class c i or c j but still keep some of their characteristics, which meets the need for pseudo OOD data.
Revised: Such a generated sample p θ ( x | z , k , c i , c j ) will not likely to be an instance of either class c i or c j but still keep some of their characteristics, which meets the need for pseudo OOD data.

Original: Furthermore, the pseudo class embedding has a great variety, owing to the diverse choices of classes and the vector k .
Revised: Furthermore, the pseudo class embedding has a great variety, owing to the diverse choices of classes and the vector k .

Original: To generate pseudo OOD samples, we also need to sample z from the encoder.
Revised: To generate pseudo OOD samples, we also need to sample z from the encoder.

Original: In the training of CVAE, we ensure that z ﬁts the Gaussian distribution
Revised: In the training of CVAE, we ensure that z ﬁts the Gaussian distribution

Original: N ( 0 , I ) .
Revised: N ( 0 , I ) .

Original: To sample z , we use another ﬂatter Gaussian distribution N ( 0 , σ 2 ∗ I ) , where σ > 1 ∈ Z , to make the generated samples highly diverse.
Revised: To sample z , we use another ﬂatter Gaussian distribution N ( 0 , σ 2 ∗ I ) , where σ > 1 ∈ Z , to make the generated samples highly diverse.

Original: 3. T RAINING P ROCESS FOR OOD D ETECTION
Revised: 3. T RAINING P ROCESS FOR OOD D ETECTION

Original: With the generated pseudo OOD samples and the original in-distribution (IND) training data, training an OOD detection model consists of two stages.
Revised: With the generated pseudo OOD samples and the original in-distribution (IND) training data, training an OOD detection model consists of two stages.

Original: Stage 1 (IND classiﬁer building and CVAE training): Only the original IND data is used to train a classiﬁcation model C .
Revised: Stage 1 (IND classiﬁer building and CVAE training): Only the original IND data is used to train a classiﬁcation model C .

Original: The classiﬁcation model can be decomposed into two functions f
Revised: The classiﬁcation model can be decomposed into two functions f

Original: and h , where f is the ﬁnal linear classiﬁer and h is the feature extractor .
Revised: and h , where f is the ﬁnal linear classiﬁer and h is the feature extractor .

Original: f ( h ( x )) is the classiﬁcation output.
Revised: f ( h ( x )) is the classiﬁcation output.

Original: A separate CVAE model is also trained for generating pseudo OOD data.
Revised: A separate CVAE model is also trained for generating pseudo OOD data.

Original: Stage 2 (ﬁne-tuning with pseudo OOD data): We keep the trained feature extractor h ﬁxed (or frozen) and ﬁne-tune only the classiﬁcation layer f for OOD detection.
Revised: Stage 2 (ﬁne-tuning with pseudo OOD data): We keep the trained feature extractor h ﬁxed (or frozen) and ﬁne-tune only the classiﬁcation layer f for OOD detection.

Original: The proposed CGA approach is in fact a framework, which is illustrated in Figure 1 together with the two-stage training process.
Revised: The proposed CGA approach is in fact a framework, which is illustrated in Figure 1 together with the two-stage training process.

Original: The framework is very ﬂexible as the classiﬁer in the ﬁrst stage can use any model.
Revised: The framework is very ﬂexible as the classiﬁer in the ﬁrst stage can use any model.

Original: Stage 2 is also ﬂexible and can use many approaches.
Revised: Stage 2 is also ﬂexible and can use many approaches.

Original: Here we introduce two speciﬁc methods.
Revised: Here we introduce two speciﬁc methods.

Original: They produce results comparable to the state-of-the-art OOD detection models, but are very simple and very efﬁcient.
Revised: They produce results comparable to the state-of-the-art OOD detection models, but are very simple and very efﬁcient.

Original: In fact, as we will see in the experiments, ﬁne-tuning the existing state-of-the-art (SOTA)
Revised: In fact, as we will see in the experiments, ﬁne-tuning the existing state-of-the-art (SOTA)

Original: OOD detection models can also improve them to produce new SOTA results.
Revised: OOD detection models can also improve them to produce new SOTA results.

Original: CGA-softmax.
Revised: CGA-softmax.

Original: In the ﬁne-tuning stage, we simply add additional class (let us call it the OOD class) in the classiﬁcation layer to accept the pseudo OOD data.
Revised: In the ﬁne-tuning stage, we simply add additional class (let us call it the OOD class) in the classiﬁcation layer to accept the pseudo OOD data.

Original: If the IND data has N classes, we add parameters to the classiﬁer to make it output N + 1 logits.
Revised: If the IND data has N classes, we add parameters to the classiﬁer to make it output N + 1 logits.

Original: These added parameters related to ( N + 1) th OOD class is randomly initialized.
Revised: These added parameters related to ( N + 1) th OOD class is randomly initialized.

Original: We then train the model by only ﬁne-tuning the classiﬁcation layer using the cross entropy loss with feature extractor trained in stage 1 ﬁxed.
Revised: We then train the model by only ﬁne-tuning the classiﬁcation layer using the cross entropy loss with feature extractor trained in stage 1 ﬁxed.

Original: Finally, we use the softmax score of the ( N + 1) th class as the OOD score.
Revised: Finally, we use the softmax score of the ( N + 1) th class as the OOD score

Original: CGA-energy.
Revised: CGA-energy.

Original: In this approach, we add an energy loss to the cross entropy loss ( L ent + λ L energy ) to ﬁne-tune the classiﬁcation layer using the IND data and the pseudo OOD data to produce an OOD score for each test instance.
Revised: In this approach, we add an energy loss to the cross entropy loss ( L ent + λ L energy ) to ﬁne-tune the classiﬁcation layer using the IND data and the pseudo OOD data to produce an OOD score for each test instance.

Original: In this case, no OOD class is added.
Revised: In this case, no OOD class is added.

Original: The energy loss is as follows,
Revised: The energy loss is as follows,

Original: [Equation]
Revised: [Equation]

Original: where D in denotes the IND training data, D out denotes generated pseudo OOD data, and m in and m out are margin hyper-parameters.
Revised: where D in denotes the IND training data, D out denotes generated pseudo OOD data, and m in and m out are margin hyper-parameters.

Original: The idea of this loss is to make the OOD data get similar values for all N logits so that they will not be favored by any N IND data classes.
Revised: The idea of this loss is to make the OOD data get similar values for all N logits so that they will not be favored by any N IND data classes.

Original: Here N is the number of classes of the IND data.
Revised: Here N is the number of classes of the IND data.

Original: As the loss function shows, the OOD data is necessary.
Revised: As the loss function shows, the OOD data is necessary.

Original: This loss was used in (Liu et al., 2020), which has to employ OOD data but such OOD data is often not available in practice.
Revised: This loss was used in (Liu et al., 2020), which has to employ OOD data but such OOD data is often not available in practice.

Original: This loss cannot be used by other OOD methods since they have no OOD data available (Bendale & Boult, 2016;
Revised: This loss cannot be used by other OOD methods since they have no OOD data available (Bendale & Boult, 2016;

Original: Khosla et al., 2020; Tack et al., 2020).
Revised: Khosla et al., 2020; Tack et al., 2020).

Original: However, this is not an issue for us as we have pseudo data to replace real OOD training data.
Revised: However, this is not an issue for us as we have pseudo data to replace real OOD training data.

Original: Stage 2 produces an energy score calculated from a classiﬁcation model for OOD detection:
Revised: Stage 2 produces an energy score calculated from a classiﬁcation model for OOD detection:

Original: [Equation]
Revised: [Equation]

Original:  where E ( x ; f ( h ( · ))) denotes the energy of instance x with the classiﬁcation model f
Revised: (6) where E ( x ; f ( h ( · ))) denotes the energy of instance x with the classiﬁcation model f

Original: ( h ( · )) , which maps x to N logits, where N is the number of classes in the IND data, f i ( h ( x )) is the i -th logit and
Revised: ( h ( · )) , which maps x to N logits, where N is the number of classes in the IND data, f i ( h ( x )) is the i -th logit and

Original: T is the temperature parameter.
Revised: T is the temperature parameter.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We construct OOD detection tasks using benchmark datasets and compare our proposed technique
Revised: We construct OOD detection tasks using benchmark datasets and compare our proposed technique

Original: CGA with the state-of-the-art existing methods.
Revised: CGA with the state-of-the-art existing methods.

Original: The code of CGA has been submitted .
Revised: The code of CGA has been submitted .

Original: 4. E XPERIMENT S ETTINGS AND D ATA P REPARATION
Revised: 4. E XPERIMENT S ETTINGS AND D ATA P REPARATION

Original: We use two experimental setups to evaluate our system.
Revised: We use two experimental setups to evaluate our system.

Original: Setting 1 - OOD Detection on the Same Dataset: In this setting, IND (in-distribution) and OOD instances are from different classes of the same dataset.
Revised: Setting 1 - OOD Detection on the Same Dataset: In this setting, IND (in-distribution) and OOD instances are from different classes of the same dataset.

Original: This setting is often called open-set detection .
Revised: This setting is often called open-set detection .

Original: We use the following 4  datasets for our experiments in this setting, which are widely used in the open-set detection literature.
Revised: We use the following 4 popular datasets for our experiments in this setting.

Original: (1) MNIST (LeCun et al., 2010):
Revised: (1) MNIST (LeCun et al., 2010):

Original: A handwritten digit classiﬁcation dataset of 10 classes.
Revised: A handwritten digit classiﬁcation dataset of 10 classes.

Original: The dataset has 70,000 examples/instances, with the splitting of 60,000 for training and 10,000 for testing.
Revised: The dataset has 70,000 examples/instances, with the splitting of 60,000 for training and 10,000 for testing.

Original: (2) CIFAR-10 (Krizhevsky & Hinton, 2010):
Revised: (2) CIFAR-10 (Krizhevsky & Hinton, 2010):

Original: An image classiﬁcation dataset consisting of 60, 32x32 color images of 10 classes with the splitting of 50,000 for training and 10,000 for testing.
Revised: An image classiﬁcation dataset consisting of 60, 32x32 color images of 10 classes with the splitting of 50,000 for training and 10,000 for testing.

Original: (3) SVHN (Netzer et al., 2011):
Revised: (3) SVHN (Netzer et al., 2011):

Original: A colorful street view house number classiﬁcation dataset ofclasses.
Revised: A colorful street view house number classiﬁcation dataset ofclasses.

Original: It contains 99289 instances with the splitting of 73257 for training and 26032 for testing.
Revised: It contains 99289 instances with the splitting of 73257 for training and 26032 for testing.

Original: (4) TinyImageNet
Revised: (4) TinyImageNet

Original: (Le & Yang, 2015):
Revised: (Le & Yang, 2015):

Original: A classiﬁcation dataset of 200 classes.
Revised: A classiﬁcation dataset of 200 classes.

Original: Each class contains 500 training samples and 50 testing samples.
Revised: Each class contains 500 training samples and 50 testing samples.

Original: The resolution of the images is 64x64.
Revised: The resolution of the images is 64x64.

Original: We follow the data processing method in (Sun et al., 2020; Zhou et al., 2021) to split known and unknown classes.
Revised: We follow the data processing method in (Sun et al., 2020; Zhou et al., 2021) to split known and unknown classes.

Original: For each dataset, we conduct 5 experiments using different splits of known (IND) and unknown (OOD) classes.
Revised: For each dataset, we conduct 5 experiments using different splits of known (IND) and unknown (OOD) classes.

Original: These same 5 splits are used by all baselines and our system.
Revised: These same 5 splits are used by all baselines and our system.

Original: Following (Sun et al., 2020), for MNIST, CIFAR-10 and SVHN, 6 classes are chosen as IND classes, and the other 4 classes are regarded as OOD classes.
Revised: Following (Sun et al., 2020), for MNIST, CIFAR-10 and SVHN, 6 classes are chosen as IND classes, and the other 4 classes are regarded as OOD classes.

Original: The following 5 ﬁxed sets of IND classes, 0-5, 1-6, 2-7, 3-8, and 4-9, are used and they are called partition 1, 2, 3, 4 , and 5 , respectively.
Revised: The following 5 ﬁxed sets of IND classes, 0-5, 1-6, 2-7, 3-8, and 4-9, are used and they are called partition 1, 2, 3, 4 , and 5 , respectively.

Original: The rest 4 classes in each case serve as the OOD classes.
Revised: The rest 4 classes in each case serve as the OOD classes.

Original: For TinyImageNet, each set of IND data contains 20 classes and the sets of IND classes in the 5 experiments are 0-19, 40-59, 80-99, 120-139, and 160-189 respectively.
Revised: For TinyImageNet, each set of IND data contains 20 classes and the sets of IND classes in the 5 experiments are 0-19, 40-59, 80-99, 120-139, and 160-189 respectively.

Original: The rest 180 classes are regarded as the OOD classes.
Revised: The rest 180 classes are regarded as the OOD classes.

Original: Setting 2 - OOD Detection on Different Datasets: The IND data and OOD data come from different datasets.
Revised: Setting 2 - OOD Detection on Different Datasets: The IND data and OOD data come from different datasets.

Original: Following (Tack et al., 2020), we use CIFAR-10 as the IND dataset and each of the following datasets as the OOD dataset.
Revised: Following (Tack et al., 2020), we use CIFAR-10 as the IND dataset and each of the following datasets as the OOD dataset.

Original: (1) SVHN (Netzer et al., 2011): See above.
Revised: (1) SVHN (Netzer et al., 2011): See above.

Original: All 26032 testing samples are used as OOD data.
Revised: All 26032 testing samples are used as OOD data.

Original: (2) LSUN (Yu et al., 2015):
Revised: (2) LSUN (Yu et al., 2015):

Original: This is a large-scale scene understanding dataset with a testing set of 10,000 images from 10 different scenes.
Revised: This is a large-scale scene understanding dataset with a testing set of 10,000 images from 10 different scenes.

Original: Images are resized to 32x32 in our experiment.
Revised: Images are resized to 32x32 in our experiment.

Original: (3) LSUN-FIX (Tack et al., 2020): To avoid artiﬁcial noises brought by general resizing operation, this dataset is generated by using a ﬁxed resizing operation on LSUN to change the images to 32x32.
Revised: (3) LSUN-FIX (Tack et al., 2020): To avoid artiﬁcial noises brought by general resizing operation, this dataset is generated by using a ﬁxed resizing operation on LSUN to change the images to 32x32.

Original: (4) TinyImageNet
Revised: (4) TinyImageNet

Original: (Le & Yang, 2015): See above.
Revised: (Le & Yang, 2015): See above.

Original: All 10000 testing samples are used as OOD data.
Revised: All 10000 testing samples are used as OOD data.

Original: (5) ImageNet-FIX (Le & Yang, 2015): 10,000 images are randomly selected from the training set of ImageNet-30, excluding “airliner”, “ambulance”, “parkingmeter”, and “schooner” classes to avoid overlapping with CIFAR-10.
Revised: (5) ImageNet-FIX (Le & Yang, 2015): 10,000 images are randomly selected from the training set of ImageNet-30, excluding “airliner”, “ambulance”, “parkingmeter”, and “schooner” classes to avoid overlapping with CIFAR-10.

Original: A ﬁxed resizing operation is applied to transform the images to 32x32.
Revised: A ﬁxed resizing operation is applied to transform the images to 32x32.

Original: (6) CIFAR100 (Krizhevsky et al., 2009):
Revised: (6) CIFAR100 (Krizhevsky et al., 2009):

Original: An image classiﬁcation dataset consisting of 60,000 32x color images of 100 classes.
Revised: An image classiﬁcation dataset consisting of 60,000 32x color images of 100 classes.

Original: Its 10,000 test samples are used as the OOD data.
Revised: Its 10,000 test samples are used as the OOD data.

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We compare with the following state-of-the-art baselines using the ofﬁcial code from their authors.
Revised: We compare with the following state-of-the-art baselines using the ofﬁcial code from their authors.

Original: (1) Softmax : This is the popular classiﬁcation score model.
Revised: (1) Softmax : This is the popular classiﬁcation score model.

Original: The highest softmax probability is used as the conﬁdence score for OOD detection.
Revised: The highest softmax probability is used as the conﬁdence score for OOD detection.

Original: (2) OpenMax (Bendale & Boult, 2016):
Revised: (2) OpenMax (Bendale & Boult, 2016):

Original: This method combines the softmax score with the distance between the test sample and IND class centers to detect OOD data.
Revised: This method combines the softmax score with the distance between the test sample and IND class centers to detect OOD data.

Original: (3) ODIN
Revised: (3) ODIN

Original: (Liang et al., 2017): This method improves the OOD detection performance of a pretrained neural network by using temperature scaling and adding small perturbations to the input.
Revised: (Liang et al., 2017): This method improves the OOD detection performance of a pretrained neural network by using temperature scaling and adding small perturbations to the input.

Original: (4) Maha (Lee et al., 2018):
Revised: (4) Maha (Lee et al., 2018):

Original: This method uses Mahalanobis distance to evaluate the probability that an instance belongs to OOD.
Revised: This method uses Mahalanobis distance to evaluate the probability that an instance belongs to OOD.

Original: (5) CCC (Lee et al., 2017):
Revised: (5) CCC (Lee et al., 2017):

Original: This is a GAN-based method, jointly training the classiﬁcation and pseudo OOD generator for OOD detection.
Revised: This is a GAN-based method, jointly training the classiﬁcation and pseudo OOD generator for OOD detection.

Original: (6) OSRCI (Neal et al., 2018):
Revised: (6) OSRCI (Neal et al., 2018):

Original: This method also uses GAN to generate pseudo instances and further improves the model to predict novelty (OOD) examples.
Revised: This method also uses GAN to generate pseudo instances and further improves the model to predict novelty (OOD) examples.

Original: (7) CAC (Miller et al., 2021):
Revised: (7) CAC (Miller et al., 2021):

Original: This is a distance-based method, using the Class Anchor Clustering loss to cluster IND samples tightly around the anchored centres for OOD detection.
Revised: This is a distance-based method, using the Class Anchor Clustering loss to cluster IND samples tightly around the anchored centres for OOD detection.

Original: (8) SupCLR (Khosla et al., 2020):
Revised: (8) SupCLR (Khosla et al., 2020):

Original: This is a contrastive learning based method.
Revised: This is a contrastive learning based method.

Original: It extends contrastive learning to fully-supervised setting to improve the quality of features for classiﬁcation.
Revised: It extends contrastive learning to fully-supervised setting to improve the quality of features for classiﬁcation.

Original: (9) CSI (Tack et al., 2020):
Revised: (9) CSI (Tack et al., 2020):

Original: This is also a supervised contrastive learning method.
Revised: This is also a supervised contrastive learning method.

Original: It uses extensive data augmentations to generate shifted data instances.
Revised: It uses extensive data augmentations to generate shifted data instances.

Original: It also has a score function that beneﬁts from the augmented instances for OOD detection.
Revised: It also has a score function that beneﬁts from the augmented instances for OOD detection.

Original: For Softmax, OpenMax and OSRCI, we use OSRCI’s implementation 1 .
Revised: For Softmax, OpenMax and OSRCI, we use OSRCI’s implementation 1 .

Original: For SupCLR and CSI, we use CSI’s code 2 .
Revised: For SupCLR and CSI, we use CSI’s code 2 .

Original: For ODIN, Maha, CCC and CAC, we use their original code 3456 .
Revised: For ODIN, Maha, CCC and CAC, we use their original code 3456 .

Original: We also use their default hyper-parameters.
Revised: We also use their default hyper-parameters.

Original: 4. I MPLEMENTATION DETAILS
Revised: 4. I MPLEMENTATION DETAILS

Original: For MNIST, we use a 9-layer CNN as the encoder (feature extractor) and a 2-layer MLP as the projection head.
Revised: For MNIST, we use a 9-layer CNN as the encoder (feature extractor) and a 2-layer MLP as the projection head.

Original: CVAE includes a 2-layer CNN as the encoder and a 2-layer deconvolution network (Zeiler et al., 2011) as the decoder, as well as two 1-layer MLPs to turn features into means and variations.
Revised: CVAE includes a 2-layer CNN as the encoder and a 2-layer deconvolution network (Zeiler et al., 2011) as the decoder, as well as two 1-layer MLPs to turn features into means and variations.

Original: For the other datasets, the encoder is a ResNet18 (He et al., 2016) and the projection head is a 2-layer MLP.
Revised: For the other datasets, the encoder is a ResNet18 (He et al., 2016) and the projection head is a 2-layer MLP.

Original: CVAE also uses ResNet18 as the encoder, and 2 residual blocks and a 3-layer deconvolution network as the decoder.
Revised: CVAE also uses ResNet18 as the encoder, and 2 residual blocks and a 3-layer deconvolution network as the decoder.

Original: The mean and variation projection are completed by two 1-layer MLPs.
Revised: The mean and variation projection are completed by two 1-layer MLPs.

Original: During the ﬁrst stage of training, we use Adam optimizer (Kingma & Ba, 2014) with β 1 = 0 .
Revised: During the ﬁrst stage of training, we use Adam optimizer (Kingma & Ba, 2014) with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 and learning rate of 0.001.
Revised: 999 and learning rate of 0.001.

Original: We train both the classiﬁcation model and CVAE model for 200 epochs with batch size 512.
Revised: We train both the classiﬁcation model and CVAE model for 200 epochs with batch size 512.

Original: In the second stage, the learning rate is set to 0.
Revised: In the second stage, the learning rate is set to 0.

Original: and the ﬁne-tuning process with the generated pseudo data are run for 10 epochs.
Revised: and the ﬁne-tuning process with the generated pseudo data are run for 10 epochs.

Original: The number of generated pseudo OOD data is the same as the IND data (we will study this further shortly).
Revised: The number of generated pseudo OOD data is the same as the IND data (we will study this further shortly).

Original: Each batch has 128 IND samples and 128 generated OOD samples.
Revised: Each batch has 128 IND samples and 128 generated OOD samples.

Original: There is no special hyper-parameter for CGA-softmax in stage 2.
Revised: There is no special hyper-parameter for CGA-softmax in stage 2.

Original: For CGA-energy, two special hyper-parameters of the energy loss m in and m out are decided at the beginning of stage 2 by IND and pseudo data.
Revised: For CGA-energy, two special hyper-parameters of the energy loss m in and m out are decided at the beginning of stage 2 by IND and pseudo data.

Original: We calculate the energy of all training IND data and generated pseudo data.
Revised: We calculate the energy of all training IND data and generated pseudo data.

Original: Then m in and m out are chosen to make 80 % of
Revised: Then m in and m out are chosen to make 80 % of

Original: IND data’s energy larger than m in and 80 % of pseudo data’s energy smaller than m out .
Revised: IND data’s energy larger than m in and 80 % of pseudo data’s energy smaller than m out .

Original: This ensures that 80 % of data get non-zero loss.
Revised: This ensures that 80 % of data get non-zero loss.

Original: We use the same 1 NVIDIA-GeForce-RTX-2080Ti GPU for the experiments of evaluating the running speed of different methods.
Revised: We use the same 1 NVIDIA-GeForce-RTX-2080Ti GPU for the experiments of evaluating the running speed of different methods.

Original: 4. R ESULTS AND D ISCUSSIONS
Revised: 4. R ESULTS AND D ISCUSSIONS

Original: Table 1 shows the results of the two OOD detection settings on different datasets.
Revised: Table 1 shows the results of the two OOD detection settings on different datasets.

Original: Due to the large image size, a large number of IND classes and a large batch size requirement, we were unable to run SupCLR and CSI using TinyImageNet on our hardware and thus do not have their results in
Revised: Due to the large image size, a large number of IND classes and a large batch size requirement, we were unable to run SupCLR and CSI using TinyImageNet on our hardware and thus do not have their results in

Original: Setting 1.
Revised: Setting 1.

Original: On average, our CGA model can achieve the best results in Setting 1.
Revised: On average, our CGA model can achieve the best results in Setting 1.

Original: In Setting 2, beneﬁting from strong features learned using contrastive loss, CSI performs the best on average and our CGA-e (CGA-energy) is slightly weaker than CSI.
Revised: In Setting 2, beneﬁting from strong features learned using contrastive loss, CSI performs the best on average and our CGA-e (CGA-energy) is slightly weaker than CSI.

Original: But from Table 3 we learn that CGA is much more efﬁcient than the contrastive learning methods.
Revised: But from Table 3 we learn that CGA is much more efﬁcient than the contrastive learning methods.

Original: With comparable overall performances on
Revised: With comparable overall performances on

Original: OOD detection, CGA spends only about 10% of CSI’s training time.
Revised: OOD detection, CGA spends only about 10% of CSI’s training time.

Original: We also notice that our CGA-s (CGA-softmax) is slightly weaker than CGA-e, which shows that the energy function is effective.
Revised: We also notice that our CGA-s (CGA-softmax) is slightly weaker than CGA-e, which shows that the energy function is effective.

Original: Equally importantly, Table 2 demonstrates that CGA’s ﬁne-tuning (stage 2) can improve the 3 best performing baselines in Table 1, i.e., GAN-based OSRCI and contrastive learning based SupCLR and CSI.
Revised: Equally importantly, Table 2 demonstrates that CGA’s ﬁne-tuning (stage 2) can improve the 3 best performing baselines in Table 1, i.e., GAN-based OSRCI and contrastive learning based SupCLR and CSI.

Original: Here after each baseline ﬁnishes its training, we apply ﬁne-tuning of CGA’s stage 2 to ﬁnetune the trained model using CGA-energy.
Revised: Here after each baseline ﬁnishes its training, we apply ﬁne-tuning of CGA’s stage 2 to ﬁnetune the trained model using CGA-energy.

Original: Although CSI already produces the best OOD detection result, it is improved further by our CGA framework to produce a new SOTA result.
Revised: Although CSI already produces the best OOD detection result, it is improved further by our CGA framework to produce a new SOTA result.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: We now perform the ablation study with various options of our system and report the AUC scores on 5 partitions of the CIFAR10 dataset in Setting 1 - OOD detection on the same dataset
Revised: We now perform the ablation study with various options of our system and report the AUC scores on 5 partitions of the CIFAR10 dataset in Setting 1 - OOD detection on the same dataset

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: CGA Stage 2 .
Revised: CGA Stage 2 .

Original: To verify the effect of different options of stage 2, we compare the results of CGA-e model with (1) without stage 2 ,
Revised: To verify the effect of different options of stage 2, we compare the results of CGA-e model with (1) without stage 2 ,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e., we directly use energy score of stage 1 to compute AUC, (2) stage 2 without using pseudo OOD data
Revised: e., we directly use energy score of stage 1 to compute AUC, (2) stage 2 without using pseudo OOD data

Original: [Table]
Revised: [Table]

Original: and (3) full stage 2 .
Revised: and (3) full stage 2 .

Original: Figure 2(a) shows that without stage 2, stage 1 produces poor results.
Revised: Figure 2(a) shows that without stage 2, stage 1 produces poor results.

Original: Stage 2 without the generated pseudo OOD data only improves the performance slightly.
Revised: Stage 2 without the generated pseudo OOD data only improves the performance slightly.

Original: The full stage 2 with the generated pseudo data greatly improves the performance of OOD detection.
Revised: The full stage 2 with the generated pseudo data greatly improves the performance of OOD detection.

Original: The experiments prove the necessity of stage 2 and the effectiveness of the generated pseudo OOD data.
Revised: The experiments prove the necessity of stage 2 and the effectiveness of the generated pseudo OOD data.

Original: We do not vary stage 1 as our contribution is in stage 2.
Revised: We do not vary stage 1 as our contribution is in stage 2.

Original: Amount of Pseudo OOD Data .
Revised: Amount of Pseudo OOD Data .

Original: We run experiments of stage 2 with different numbers of generated pseudo OOD samples to further analyze their effectiveness.
Revised: We run experiments of stage 2 with different numbers of generated pseudo OOD samples to further analyze their effectiveness.

Original: Figure 2(b) demonstrates that the model can beneﬁt from only a few pseudo samples signiﬁcantly.
Revised: Figure 2(b) demonstrates that the model can beneﬁt from only a few pseudo samples signiﬁcantly.

Original: Though the amount is only 10 % of the
Revised: Though the amount is only 10 % of the

Original: IND data, the pseudo data can improve the results markedly, which indicates that pseudo samples are highly effective.
Revised: IND data, the pseudo data can improve the results markedly, which indicates that pseudo samples are highly effective.

Original: The results are similar when pseudo samples are more than half of the IND training samples.
Revised: The results are similar when pseudo samples are more than half of the IND training samples.

Original: We use the same number of pseudo samples as the IND samples in all our experiments.
Revised: We use the same number of pseudo samples as the IND samples in all our experiments.

Original: Pseudo OOD Data Distribution.
Revised: Pseudo OOD Data Distribution.

Original: The CVAE generator is trained to make the latent variables or features conform to the Gaussian distribution N ( 0 , I ) (see Section 3.2).
Revised: The CVAE generator is trained to make the latent variables or features conform to the Gaussian distribution N ( 0 , I ) (see Section 3.2).

Original: To make pseudo data diverse and different from the training data, we sample the latent variables z from a pseudo data sampling distribution N ( 0 , σ 2 ∗ I ) .
Revised: To make pseudo data diverse and different from the training data, we sample the latent variables z from a pseudo data sampling distribution N ( 0 , σ 2 ∗ I ) .

Original: We conduct experiments to study the effect of the distribution.
Revised: We conduct experiments to study the effect of the distribution.

Original: First, we
Revised: First, we

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: study the inﬂuence of σ .
Revised: study the inﬂuence of σ .

Original: Note that σ is 1 in training.
Revised: Note that σ is 1 in training.

Original: With larger σ values, the sampled values will be more likely to be far from 0 (which is the mean) to make the latent features different from those seen in training.
Revised: With larger σ values, the sampled values will be more likely to be far from 0 (which is the mean) to make the latent features different from those seen in training.

Original: Figure 3(a) shows the results, which indicate the necessity of using σ > 1 .
Revised: Figure 3(a) shows the results, which indicate the necessity of using σ > 1 .

Original: Results are similar when σ ≥ 5 and we use 5 in our experiments.
Revised: Results are similar when σ ≥ 5 and we use 5 in our experiments.

Original: Intuitively, we may only keep latent features z that are far from the Gaussian distribution mean by ﬁltering out values that are close to 0 (or the mean).
Revised: Intuitively, we may only keep latent features z that are far from the Gaussian distribution mean by ﬁltering out values that are close to 0 (or the mean).

Original: We use a ﬁltering threshold t to ﬁlter out the sampled z whose component values are within the range [ − t, t ] .
Revised: We use a ﬁltering threshold t to ﬁlter out the sampled z whose component values are within the range [ − t, t ] .

Original: Experimental results in
Revised: Experimental results in

Original: Figure 3(b) allow us to make the following observations.
Revised: Figure 3(b) allow us to make the following observations.

Original: When σ = 1 , as t grows, the performance improves slightly.
Revised: When σ = 1 , as t grows, the performance improves slightly.

Original: But comparing with the results in Figure 3(a), we see that a larger σ improves the performance more.
Revised: But comparing with the results in Figure 3(a), we see that a larger σ improves the performance more.

Original: Figure 3(c) tells us that when σ = 5 , the effect of ﬁltering diminishes.
Revised: Figure 3(c) tells us that when σ = 5 , the effect of ﬁltering diminishes.

Original: For simplicity and efﬁciency, all our experiments employed σ = 5 without ﬁltration.
Revised: For simplicity and efﬁciency, all our experiments employed σ = 5 without ﬁltration.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: It has been shown recently that contrastive learning with extensive data augmentations can produce the state-of-the-art out-of-distribution detection results.
Revised: It has been shown recently that contrastive learning with extensive data augmentations can produce the state-of-the-art out-of-distribution detection results.

Original: Our experiments also conﬁrmed that.
Revised: Our experiments also conﬁrmed that.

Original: However, such algorithms are extremely inefﬁcient and resource hungry due to a large amount of augmented data and quadratic pairwise similarity computation, which makes such algorithms unsuitable for applications that do not have the required computing resources, e.g., edge devices.
Revised: However, such algorithms are extremely inefﬁcient and resource hungry due to a large amount of augmented data and quadratic pairwise similarity computation, which makes such algorithms unsuitable for applications that do not have the required computing resources, e.g., edge devices.

Original: In this paper, we proposed a simple alternative based on data generation using CVAE which delivers similar accuracy results but is much more efﬁcient.
Revised: In this paper, we proposed a simple alternative based on data generation using CVAE which delivers similar accuracy results but is much more efﬁcient.

Original: What is also interesting is that our proposed approach can improve the results of those state-of-the-art contrastive learning based methods too.
Revised: What is also interesting is that our proposed approach can improve the results of those state-of-the-art contrastive learning based methods too.

Original: Thus, our future work will focus on improving the detection accuracy while maintaining or further reducing the computing resources requirements.
Revised: Thus, our future work will focus on improving the detection accuracy while maintaining or further reducing the computing resources requirements.

Revised: We also plan to explore pre-trained feature extractors and more advanced variants of CVAE to improve our method.

Original: On broader impacts, we believe that the proposed approach has a great potential and may also be applicable to one-class learning, positive and unlabeled (PU) learning, and continual learning.
Revised: On broader impacts, we believe that the proposed approach has a great potential and may also be applicable to one-class learning, positive and unlabeled (PU) learning, and continual learning.

Original: for these problems, the ability to generate pseudo OOD data is useful.
Revised: for these problems, the ability to generate pseudo OOD data is useful.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: We believe that our work has no ethic issues as we propose a general algorithm for out-of-distribution detection, which is not for any speciﬁc application.
Revised: We believe that our work has no ethic issues as we propose a general algorithm for out-of-distribution detection, which is not for any speciﬁc application.

Original: Our experimental datasets are all public domain datasets.
Revised: Our experimental datasets are all public domain datasets.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We have submitted our code in the supplementary material.
Revised: We have submitted our code in the supplementary material.

Original: A I MPLEMENTATION D ETAILS OF B ASELINES
Revised: A I MPLEMENTATION D ETAILS OF B ASELINES

Original: For OOD detection on the same dataset, since the partitions of in-distribution (IND) and out-ofdistribution (OOD) classes from a dataset affect the results signiﬁcantly we show the detailed results of individual partitions in the next section.
Revised: For OOD detection on the same dataset, since the partitions of in-distribution (IND) and out-ofdistribution (OOD) classes from a dataset affect the results signiﬁcantly, we show the detailed results of individual partitions in the next section.

Original: As for OOD detection on different datasets, we copy
Revised: As for OOD detection on different datasets, we copy

Original: SupCLR and CSI’s results from the CSI paper and produce our own results for the other baselines
Revised: Results of the 5 partitions of each dataset are listed in the table [Table]

Original: B D ETAILED E XPERIMENT R ESULTS
Revised: B D ETAILED E XPERIMENT R ESULTS

Original: We found that when conducting OOD detection experiments using the same dataset (i e., some classes of the dataset are used as the IND classes and the rest as the OOD classes), the choices of IND and OOD classes can affect the results greatly.
Revised: We found that when conducting OOD detection experiments using the same dataset (i.e., some classes of the dataset are used as the IND classes and the rest as the OOD classes), the choices of IND and OOD classes can affect the results greatly.

Original: Some partitions of IND and OOD classes are very hard while some are easy.
Revised: Some partitions of IND and OOD classes are very hard while some are easy.

Original: Therefore, we ﬁx 5 sets of IND and OOD classes to make the comparison between different methods fair.
Revised: Therefore, we ﬁx 5 sets of IND and OOD classes to make the comparison between different methods fair.

Original: Table 4 shows the average AUC score for each of the 5 partitions of each dataset.
Revised: Table 4 shows the average AUC score for each of the 5 partitions of each dataset.

Original: From Table 4, we learn that the ranges (or differences) of the results for different IND and OOD partitions of the datasets are very large.
Revised: From Table 4, we learn that the ranges (or differences) of the results for different IND and OOD partitions of the datasets are very large.

Original: On the CIFAR-10 dataset, the difference is as large as 17.2%.
Revised: On the CIFAR-10 dataset, the difference is as large as 17.2%.

Original: Results of the 5 partitions of each dataset are listed in the table

Original: [Table]

Original: C V ISUALIZATION OF D ATA D ISTRIBUTION
Revised: C V ISUALIZATION OF D ATA D ISTRIBUTION

Original: We visualize the latent representations of different data to learn their distributional property.
Revised: We visualize the latent representations of different data to learn their distributional property.

Original: The visualization is done on partition 1 of MNIST dataset.
Revised: The visualization is done on partition 1 of MNIST dataset.

Original: Figure 4 shows the distribution of the generated data after dimensionality reduction by t-distributed Stochastic Neighbor Embedding (t-SNE) method.
Revised: Figure 4 shows the distribution of the generated data after dimensionality reduction by t-distributed Stochastic Neighbor Embedding (t-SNE) method.

Original: The left ﬁgure shows 6 clusters of IND data of 6 classes.
Revised: The left ﬁgure shows 6 clusters of IND data of 6 classes.

Original: Red points in the middle ﬁgure represent our generated pseudo data.
Revised: Red points in the middle ﬁgure represent our generated pseudo data.

Original: This ﬁgure demonstrates that a large number of pseudo samples are around the boundaries of the clusters (representing the known classes).
Revised: This ﬁgure demonstrates that a large number of pseudo samples are around the boundaries of the clusters (representing the known classes).

Original: These are the most effective OOD-like samples.
Revised: These are the most effective OOD-like samples.

Original: At the same time, our generated data distribute widely, which shows that the pseudo data is of great diversity.
Revised: At the same time, our generated data distribute widely, which shows that the pseudo data is of great diversity.

Original: Pink points in the right ﬁgure represent real OOD data during testing.
Revised: Pink points in the right ﬁgure represent real OOD data during testing.

Original: Comparing red points and pink points, we can learn that the diverse pseudo data can cover nearly all areas where there are true OOD instances.
Revised: Comparing red points and pink points, we can learn that the diverse pseudo data can cover nearly all areas where there are true OOD instances.

Original: D V ISUALIZATION OF P SEUDO OOD I MAGES
Revised: D V ISUALIZATION OF P SEUDO OOD I MAGES

Original: Figure 5 shows the generated pseudo OOD images from our CVAE.
Revised: Figure 5 shows the generated pseudo OOD images from our CVAE.

Original: For each group, we ﬁrst employ the standard class embedding and sample latent variables to generate 2 IND images.
Revised: For each group, we ﬁrst employ the standard class embedding and sample latent variables to generate 2 IND images.

Original: Then we use these 2 class embeddings to compute our pseudo class embedding (see the paper) and then generate pseudo OOD images with variables or features from different sampling distributions N ( 0 , σ 2 ∗ I ) .
Revised: Then we use these 2 class embeddings to compute our pseudo class embedding (see the paper) and then generate pseudo OOD images with variables or features from different sampling distributions N ( 0 , σ 2 ∗ I ) .

Original: It is easy to see some characteristics of IND images from pseudo images.
Revised: It is easy to see some characteristics of IND images from pseudo images.

Original: When σ = 1 , pseudo OOD images still resemble the IND images a great deal.
Revised: When σ = 1 , pseudo OOD images still resemble the IND images a great deal.

Original: But as σ grows, the difference between the IND images and pseudo OOD images gets greater and the diversity also increases.
Revised: But as σ grows, the difference between the IND images and pseudo OOD images gets greater and the diversity also increases.

Original: That also explains why changing σ can improve the ﬁnal performance.
Revised: That also explains why changing σ can improve the ﬁnal performance.