Original: H ARNESSING O UT -O F -D ISTRIBUTION E XAMPLES VIA A UGMENTING C ONTENT AND S TYLE
Revised: H ARNESSING O UT -O F -D ISTRIBUTION E XAMPLES VIA A UGMENTING C ONTENT AND S TYLE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Machine learning models are vulnerable to Out-Of-Distribution (OOD) examples, and such a problem has drawn much attention.
Revised: Machine learning models are vulnerable to Out-Of-Distribution (OOD) examples, and such a problem has drawn much attention.

Original: However, current methods lack a full understanding of different types of OOD data: there are benign OOD data that can be properly adapted to enhance the learning performance, while other malign
Revised: However, current methods lack a full understanding of different types of OOD data: there are benign OOD data that can be properly adapted to enhance the learning performance, while other malign

Original: OOD data would severely degenerate the classification result.
Revised: OOD data would severely degenerate the classification result.

Original: To Harness OOD data, this paper proposes a HOOD method that can leverage the content and style from each image instance to identify benign and malign OOD data.
Revised: To Harness OOD data, this paper proposes a HOOD method that can leverage the content and style from each image instance to identify benign and malign OOD data.

Original: Particularly, we design a variational inference framework to causally disentangle content and style features by constructing a structural causal model.
Revised: Particularly, we design a variational inference framework to causally disentangle content and style features by constructing a structural causal model.

Original: Subsequently, we augment the content and style through an intervention process to produce malign and benign
Revised: Subsequently, we augment the content and style through an intervention process to produce malign and benign

Original: OOD data, respectively.
Revised: OOD data, respectively.

Original: The benign OOD data contain novel styles but hold our interested contents, and they can be leveraged to help train a style-invariant model.
Revised: The benign OOD data contain novel styles but hold our interested contents, and they can be leveraged to help train a style-invariant model.

Original: In contrast, the malign OOD data inherit unknown contents but carry familiar styles, by detecting them can improve model robustness against deceiving anomalies.
Revised: In contrast, the malign OOD data inherit unknown contents but carry familiar styles, by detecting them can improve model robustness against deceiving anomalies.

Original: Thanks to the proposed novel disentanglement and data augmentation techniques, HOOD can effectively deal with OOD examples in unknown and open environments, whose effectiveness is empirically validated in three typical OOD applications including OOD detection, open-set semi-supervised learning, and open-set domain adaptation.
Revised: Thanks to the proposed novel disentanglement and data augmentation techniques, HOOD can effectively deal with OOD examples in unknown and open environments, whose effectiveness is empirically validated in three typical OOD applications including OOD detection, open-set semi-supervised learning, and open-set domain adaptation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning in the presence of Out-Of-Distribution (OOD) data has been a challenging task in machine learning, as the deployed classifier tends to fail if the unseen data drawn from unknown distributions are not properly handled (Hendrycks & Gimpel, 2017; Pan & Yang, 2009).
Revised: Learning in the presence of Out-Of-Distribution (OOD) data has been a challenging task in machine learning, as the deployed classifier tends to fail if the unseen data drawn from unknown distributions are not properly handled (Hendrycks & Gimpel, 2017; Pan & Yang, 2009).

Original: Such a critical problem ubiquitously exists when deep models(LeCun et al., 2015) meet domain shift (Ganin et al., 2016;
Revised: Such a critical problem ubiquitously exists when deep models meet domain shift (Ganin et al., 2016;

Original: Tzeng et al., 2017) and unseen-class data (Hendrycks & Gimpel, 2017; Scheirer et al., 2012), which has drawn a lot of attention in some important fields such as OOD detection (Hein et al., 2019; Hendrycks & Gimpel, 2017; Lee et al., 2018; Liang et al., 2018; Liu et al., 2020; Wang et al., 2022), Open-Set Domain Adaptation (DA) (Liu et al., 2019;
Revised: Tzeng et al., 2017) and unseen-class data (Hendrycks & Gimpel, 2017; Scheirer et al., 2012), which has drawn a lot of attention in some important fields such as OOD detection (Hein et al., 2019; Hendrycks & Gimpel, 2017; Lee et al., 2018; Liang et al., 2018; Liu et al., 2020; Wang et al., 2022a; 2023; 2022b), OpenSet Domain Adaptation (DA) (Liu et al., 2019;

Original: Saito et al., 2018), and Open-Set 
Revised: Saito et al., 2018), and Open-Set Semi-Supervised

Original: Semi-Supervised Learning (SSL) (Oliver et al., 2018;
Revised: Learning (SSL) (Huang et al., 2021b; 2022b;a; Oliver et al., 2018;

Original: Saito et al., 2021; Yu et al., 2020).
Revised: Saito et al., 2021; Yu et al., 2020).

Original: In the above fields, OOD data can be divided into two types, namely benign OOD data  and malign
Revised: In the above fields, OOD data can be divided into two types, namely benign OOD data 1 and malign

Original: OOD data.
Revised: OOD data.

Original: The benign OOD data can boost the learning performance on the target distribution through DA techniques (Ganin & Lempitsky, 2015; Tzeng et al., 2017), but they can be misleading if not being properly exploited.
Revised: The benign OOD data can boost the learning performance on the target distribution through DA techniques (Ganin & Lempitsky, 2015; Tzeng et al., 2017), but they can be misleading if not being properly exploited.

Original: To improve model generalization, many positive data augmentation techniques (Cubuk et al., 2018; Xie et al., 2020) have been proposed.
Revised: To improve model generalization, many positive data augmentation techniques (Cubuk et al., 2018; Xie et al., 2020) have been proposed.

Original: For instance, the performance of SSL (Berthelot et al., 2019; Sohn et al., 2020) has been greatly improved thanks to the augmented benign OOD data 1 .
Revised: For instance, the performance of SSL (Berthelot et al., 2019; Sohn et al., 2020) has been greatly improved thanks to the augmented benign OOD data.

Original: On the contrary, malign OOD data with unknown classes can damage the classification results, but they are deceiving and hard to detect (Hendrycks & Gimpel, 2017; Liang et al., 2018).
Revised: On the contrary, malign OOD data with unknown classes can damage the classification results, but they are deceiving and hard to detect (Hendrycks & Gimpel, 2017; Liang [Equation] (c) [Equation]

Original: To train a robust model against malign OOD data, some works (Kong & Ramanan, 2021; Sinha et al., 2020) conduct negative data augmentation to generate “hard” malign data which resemble in-distribution (ID) data. By separating such “hard” data from ID data, the OOD detection performance can be improved.
Revised: C ത S ෡ C ഥ X intervention intervention D [Equation]

Original: When presented with both malign and benign OOD data, it is more challenging to decide which to separate and which to exploit.
Revised: b

Original: As a consequence, the performance of existing open-set methods could be sub-optimal due to two major problems: 1) radically exploiting too much malign OOD data, and 2) conservatively denying too much benign OOD data.[Equation]
Revised: [Equation] et al., 2018; Wei et al., 2022b;a). To train a robust model against malign OOD data, some works (Kong & Ramanan, 2021; Sinha et al., 2020) conduct negative data augmentation to generate “hard” malign data which resemble in-distribution (ID) data.

Original: D [Equation] [Figure]
Revised: By separating such “hard” data from ID data, the

Original: (c) D S
Revised: OOD detection performance can be improved.

Original: X Y C ത S ෡ C ഥ X intervention intervention D
Revised: When presented with both malign and benign OOD data, it is more challenging to decide which to separate and which to exploit.

Original: X [Figure] ) ෨ S Y
Revised: As a consequence, the performance of existing open-set methods could be sub-optimal due to two drawbacks: 1) radically exploiting too much malign OOD data, and 2) conservatively denying too much benign OOD data.

Original: In this paper, we propose a HOOD framework (see Fig. 2) to properly harness OOD data in several
Revised: In this paper, we propose a HOOD framework (see Fig. 2) to properly harness OOD data in several

Original: OOD problems.
Revised: OOD problems.

Original: To distinguish benign and malign OOD data, we model the data generating process by following the structural causal model (SCM)
Revised: To distinguish benign and malign OOD data, we model the data generating process by following the structural causal model (SCM)

Original: (Glymour et al., 2016; Pearl, 2009) in Fig.
Revised: (Glymour et al., 2016; Pearl, 2009) in Fig.

Original: 1 (a).
Revised: 1 (a).

Original: Particularly, we decompose an image instance X into two latent components: 1) content variable C which denotes the interested object, and 2) style variable S which contains other influential factors such as brightness, orientation, and color.
Revised: Particularly, we decompose an image instance X into two latent components: 1) content variable C which denotes the interested object, and 2) style variable S which contains other influential factors such as brightness, orientation, and color.

Original: The content C can indicate its true class Y , and the style S is decisive for the environmental condition, which is termed as domain D .
Revised: The content C can indicate its true class Y , and the style S is decisive for the environmental condition, which is termed as domain D .

Original: Intuitively, malign OOD data cannot be incorporated into network training, because they contain unseen contents, thus their true classes are different from any known class; and benign OOD data can be adapted because they only have novel styles but contain the same contents as ID data.
Revised: Intuitively, malign OOD data cannot be incorporated into network training, because they contain unseen contents, thus their true classes are different from any known class; and benign OOD data can be adapted because they only have novel styles but contain the same contents as ID data.

Original: Therefore, we can distinguish the benign and malign OOD data based on the extracted the content and style features.
Revised: Therefore, we can distinguish the benign and malign OOD data based on the extracted the content and style features.

Original: In addition, we conduct causal disentanglement through maximizing an approximated evidence lowerbound (ELBO) (Blei et al., 2017) of joint distribution P ( X, Y, D ) .
Revised: In addition, we conduct causal disentanglement through maximizing an approximated evidence lower-bound (ELBO) (Blei et al., 2017; Yao et al., 2021; Xia et al., 2022b) of joint distribution P ( X, Y, D ) .

Original: As a result, we can effectively break the spurious correlation (Pearl, 2009;
Revised: As a result, we can effectively break the spurious correlation (Pearl, 2009;

Original: Glymour et al., 2016; Hermann et al., 2020; Li et al., 2021b) between content and style which commonly occurs during network training (Arjovsky et al., 2019), as shown by the dashed lines in Fig.
Revised: Glymour et al., 2016; Hermann et al., 2020; Li et al., 2021b; Zhang et al., 2022) between content and style which commonly occurs during network training (Arjovsky et al., 2019), as shown by the dashed lines in Fig.

Original: 1 (b).
Revised: 1 (b).

Original: In the ablation study, we find that HOOD can correctly disentangle content and style, which can correspondingly benefit generalization tasks (such as open-set DA and open-set SSL) and detection task (such as OOD detection).
Revised: In the ablation study, we find that HOOD can correctly disentangle content and style, which can correspondingly benefit generalization tasks (such as open-set DA and open-set SSL) and detection task (such as OOD detection).

Original: To further improve the learning performance, we conduct both positive and negative data augmentation by solely intervening the style and content, respectively, as shown by the blue and red lines in Fig.(c).
Revised: To further improve the learning performance, we conduct both positive and negative data augmentation by solely intervening the style and content, respectively, as shown by the blue and red lines in Fig.(c).

Original: Such process is achieved through backpropagating the gradient computed from an intervention objective.
Revised: Such process is achieved through backpropagating the gradient computed from an intervention objective.

Original: As a result, style-changed data ¯ X must be identified as benign OOD data, and contentchanged data ˆ X should be recognized as malign OOD data.
Revised: As a result, style-changed data ¯ X must be identified as benign OOD data, and contentchanged data ˆ X should be recognized as malign OOD data.

Original: Without including any bias, the benign
Revised: Without including any bias, the benign

Original: OOD data can be easily harnessed to improve model generalization, and the malign OOD data can be directly recognized as harmful ones which benefits the detection of unknown anomalies.
Revised: OOD data can be easily harnessed to improve model generalization, and the malign OOD data can be directly recognized as harmful ones which benefits the detection of unknown anomalies.

Original: By conducting extensive experiments on several OOD applications, including OOD detection, open-set
Revised: By conducting extensive experiments on several OOD applications, including OOD detection, open-set

Original: SSL, and open-set DA, we validate the effectiveness of our method on typical benchmark datasets.
Revised: SSL, and open-set DA, we validate the effectiveness of our method on typical benchmark datasets.

Original: To sum up, our contributions are three-fold:
Revised: To sum up, our contributions are three-fold:

Original: • We propose a unified framework dubbed HOOD which can effectively disentangle the content and style features to break the spurious correlation.
Revised: • We propose a unified framework dubbed HOOD which can effectively disentangle the content and style features to break the spurious correlation.

Original: As a result, benign OOD data and malign OOD data can be correctly identified based on the disentangled features.
Revised: As a result, benign OOD data and malign OOD data can be correctly identified based on the disentangled features.

Original: • We design a novel data augmentation method which correspondingly augments the content and style features to produce benign and malign OOD data, and further leverage them to enhance the learning performance.
Revised: • We design a novel data augmentation method which correspondingly augments the content and style features to produce benign and malign OOD data, and further leverage them to enhance the learning performance.

Original: • We experimentally validate the effectiveness of HOOD on various OOD applications, including OOD detection, open-set SSL, and open-set DA.
Revised: • We experimentally validate the effectiveness of HOOD on various OOD applications, including OOD detection, open-set SSL, and open-set DA.

Original: R ELATED W ORK

Original: OOD applications contains three typical problems, namely OOD detection, open-set SSL, and openset DA.

Original: OOD detection aims to train a robust model which can accurately identify the newly-emerged malign OOD data during the test phase.

Original: Open-set SSL deals with the problem when labeled data are scarce and the unlabeled data are contaminated by malign OOD data.

Original: As for open-set DA, it tries to transfer the knowledge from source ID data to the benign OOD data in target domain, meanwhile detecting the malign OOD data that are encountered during transferring.

Original: In both three applications, the predictive confidence has been frequently leveraged to separate malign OOD data (Hendrycks

Original: [-Gimpel, 
-]
Revised: [-Gimpel, 
-]

Original: ; Liang et al., 2018; Ren et al., 2019).

Original: Moreover, ID data and OOD data can be distinguished via using a discriminator (Kong & Ramanan, 2021; Neal et al., 2018; Yu et al., 2020).

Original: Further, various open classifiers are designed to predict OOD dataset as unknown (Ge et al., 2017;

Original: Padhy et al., 2020; Saito et al., 2018).

Original: Thanks to the advances in unsupervised learning, many approaches employ self-supervised learning to make ID data and OOD data separable (Cao et al., 2022; Li et al., 2021a;

Original: Saito et al., 2020) from each other.

Original: Causality in OOD problems mainly focuses on learning invariant representations that stay constant when other causal factors are changing, thus achieving better performance when facing non-stationary data distribution.

Original: To accomplish this goal, it is common to learn causal factors and non-causal factors through the variational auto-encoder framework (Blei et al., 2017; Kingma & Welling, 2013).

Original: Thanks to which, domain adaptation (Gong et al., 2016; Schölkopf et al., 2011; Zhang et al., 2013) and domain generalization (Li et al., 2018; Shankar et al., 2018) can be tackled through extracting the domain invariant features.

Original: Moreover, based on causal effects, the biased feature can be eliminated through re-weighting (Bahadori et al., 2017; Shen et al., 2018). Additionally, the spurious correlation which is harmful for inference could be alleviated through do-calculus (Lee et al., 2021; Nam et al., 2020; Pearl, 2009). Recent methods (Ilse et al., 2021; Mitrovic et al., 2020; Von Kügelgen et al., 2021) conduct data augmentations with self-supervised learning to train a robust model that can handle distribution shifts and corruptions.
Revised: [Figure]

Original: In general, HOOD has two major differences from existing methods in OOD applications and causality.

Original: On one hand, instead of treating an image instance as a whole as commonly done in many approaches, HOOD can properly leverage OOD examples through their disentangled contents and styles.

Original: Moreover, augmenting content and style can help improve generalization and robustness simultaneously.

Original: On the other hand, current causal approaches are incapable of dealing with malign

Original: OOD data, but HOOD is able to learn style-invariant features from benign OOD data, meanwhile avoiding the damage brought by malign OOD data.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we propose our HOOD framework as shown in Fig.
Revised: In this section, we propose our HOOD framework as shown in Fig.

Original: Specifically, we utilize the class labels of labeled data and the pseudo labels (Lee, 2013) of unlabeled data as the class supervision to capture the content feature.
Revised: Specifically, we utilize the class labels of labeled data and the pseudo labels (Lee, 2013) of unlabeled data as the class supervision to capture the content feature.

Original: Moreover, we perform different types of data augmentation and regard the augmentation types as the domain supervision for each style.
Revised: Moreover, we perform different types of data augmentation and regard the augmentation types as the domain supervision for each style.

Original: Thereby, each instance x is paired with a class label y and a domain label d .
Revised: Thereby, each instance x is paired with a class label y and a domain label d .

Original: Then, we apply two separate encoders g c and g s parameterized by θ c and θ s to model the posterior distributions q θ c ( C | X ) and q θ s ( S | X ) , respectively.
Revised: Then, we apply two separate encoders g c and g s parameterized by θ c and θ s to model the posterior distributions q θ c ( C | X ) and q θ s ( S | X ) , respectively.

Original: Subsequently, the generated C and S are correspondingly fed into two fully-connected classifiers f c and f s parameterized by ϕ c and ϕ s , which would produce the label predictions q ϕ c ( Y |
Revised: Subsequently, the generated C and S are correspondingly fed into two fully-connected classifiers f c and f s parameterized by ϕ c and ϕ s , which would produce the label predictions q ϕ c ( Y |

Original: C ) and q ϕ s ( D | S ) , respectively.
Revised: C ) and q ϕ s ( D | S ) , respectively.

Original: To further enhance the identifiability of C and S , a decoder h with parameter ψ is employed to reconstruct the input instance x based on its content and style.
Revised: To further enhance the identifiability of C and S , a decoder h with parameter ψ is employed to reconstruct the input instance x based on its content and style.

Original: Below, we describe the detailed procedures and components during modeling HOOD.
Revised: Below, we describe the detailed procedures and components during modeling HOOD.

Original: We first introduce the proposed variational inference framework for disentangling the content and style based on the constructed SCM.
Revised: We first introduce the proposed variational inference framework for disentangling the content and style based on the constructed SCM.

Original: Subsequently, we conduct intervention to produce benign OOD data and malign OOD data.
Revised: Subsequently, we conduct intervention to produce benign OOD data and malign OOD data.

Original: Further, we appropriately leverage the benign and malign OOD data to boost the learning performance.
Revised: Further, we appropriately leverage the benign and malign OOD data to boost the learning performance.

Original: Finally, we formulate the deployment of HOOD in three OOD applications.
Revised: Finally, we formulate the deployment of HOOD in three OOD applications.

Original: 3. V ARIATIONAL I NFERENCE FOR C ONTENT AND S TYLE D ISENTANGLEMENT
Revised: 2. V ARIATIONAL I NFERENCE FOR C ONTENT AND S TYLE D ISENTANGLEMENT

Original: First, we assume that the data generating process can be captured by certain probability distributions.
Revised: First, we assume that the data generating process can be captured by certain probability distributions.

Original: Therefore, according to the constructed SCM in Fig. 1 (a), the joint distribution P ( X, Y, D, C, S ) of the interested variables can be factorized as follows:
Revised: Therefore, according to the constructed SCM in Fig. 1 (a), the joint distribution P ( X, Y, D, C, S ) of the interested variables can be factorized as follows:

Original: [Equation]
Revised: [Equation]

Original: Based on the SCM in Fig.
Revised: Based on the SCM in Fig.

Original: 1 (a), Y and D are conditionally independent to each other
Revised: 1 (a), Y and D are conditionally independent to each other

Original: , i.e. , Y ⊥ ⊥ D | ( C, S ) , so we have P ( Y, D | C, S )
Revised: , i.e. , Y ⊥ ⊥ D | ( C, S ) , so we have P ( Y, D | C, S )

Original: =
Revised: =

Original: P ( Y | C, S ) P ( D | C, S ) .
Revised: P ( Y | C, S ) P ( D | C, S ) .

Original: Similarly, we have
Revised: Similarly, we have

Original: P ( C, S ) =
Revised: P ( C, S ) =

Original: P ( C ) P ( S ) .
Revised: P ( C ) P ( S ) .

Original: Moreover, we can also know that Y is not conditioned on S , and D is not conditioned on C .
Revised: Moreover, we can also know that Y is not conditioned on S , and D is not conditioned on C .

Original: Hence, we can further derive P ( Y, D | C, S )
Revised: Hence, we can further derive P ( Y, D | C, S )

Original: =
Revised: =

Original: P ( Y | C ) P ( D | S ) .
Revised: P ( Y | C ) P ( D | S ) .

Original: However, the aforementioned spurious correlation frequently appears when facing OOD examples (Arjovsky et al., 2019).
Revised: However, the aforementioned spurious correlation frequently appears when facing OOD examples (Arjovsky et al., 2019).

Original: As a consequence, when variational inference is based on the factorization in Eq. 1, the approximated content ˜ C and style ˜ S could both directly influence Y and D , i.e. ,
Revised: As a consequence, when variational inference is based on the factorization in Eq. 1, the approximated content ˜ C and style ˜ S could both directly influence Y and D , i.e. ,

Original: Y ← ˜ C → D and Y ← ˜ S
Revised: Y ← ˜ C → D and Y ← ˜ S

Original: → D , thus leading to inaccurate approximations.
Revised: → D , thus leading to inaccurate approximations.

Original: However, the desired[Figure] condition is Y ← C ↛ D and Y ↚ S
Revised: However, the desired condition is Y ← C ↛ D and Y ↚ S

Original: → D .
Revised: → D .

Original: We can see that the unwanted correlations ˜ C
Revised: We can see that the unwanted correlations ˜ C

Original: → D and ˜
Revised: → D and ˜

Original: S → Y in Fig.
Revised: S → Y in Fig.

Original: 1 (b) is caused by erroneous posteriors P ( D | ˜ C ) and P ( Y | ˜ S ) .
Revised: 1 (b) is caused by erroneous posteriors P ( D | ˜ C ) and P ( Y | ˜ S ) .

Original: Therefore, to break the correlations, the posteriors q ϕ s ( D | C ) and q ϕ c ( Y | S ) which are correspondingly approximated by the decoders ϕ s and ϕ c can be used as denominators to q ϕ c ( Y | C ) and q ϕ s ( D | S ) , respectively.
Revised: Therefore, to break the correlations, the posteriors q ϕ s ( D | C ) and q ϕ c ( Y | S ) which are correspondingly approximated by the decoders ϕ s and ϕ c can be used as denominators to q ϕ c ( Y | C ) and q ϕ s ( D | S ) , respectively.

Original: In this way, we can successfully disentangle content C and style S and ensure the decoding process of Y and D would not be influenced by spurious features from S and C , respectively.
Revised: In this way, we can successfully disentangle content C and style S and ensure the decoding process of Y and D would not be influenced by spurious features from S and C , respectively.

Original: To this end, our factorization in Eq.
Revised: To this end, our factorization in Eq.

Original: 1 can be approximated as:
Revised: 1 can be approximated as:

Original: [Equation]
Revised: [Equation]

Original: Then, we maximize the log-likelihood of the joint distribution p ( x , y, d ) of each data point ( x , y, d ) :
Revised: Then, we maximize the log-likelihood of the joint distribution p ( x , y, d ) of each data point ( x , y, d ) :

Original: [Equation]
Revised: [Equation]

Original: in which we use lower case to denote the values of corresponding variables.
Revised: in which we use lower case to denote the values of corresponding variables.

Original: Due to the integration of latents C and S is intractable, we follow variational inference (Blei et al., 2017) to obtain an approximated evidence lower-bound 
Revised: Due to the integration of latents C and S is intractable, we follow variational inference (Blei et al., 2017) to obtain an approximated evidence lower-bound ˜ ELBO

Original: [Equation]
Revised: [Equation]

Original: , d ) of the log-likelihood in Eq.
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Recall the modified joint distribution factorization in Eq. 2, we can have: ˜ ELBO
Revised: y

Original: [Equation]
Revised: [Equation]

Original: x , y, d ) =
Revised: of the log-likelihood in Eq.

Original: [Equation]
Revised: [Equation]

Original: c,s
Revised: Recall the modified joint distribution factorization in Eq. 2, we can have:

Original: [Equation]
Revised: [Equation]

Original: In Eq.
Revised: In Eq.

Original: 5a, the first two terms indicate the Kullback-Leibler divergence between the latent variables
Revised: 5a, the first two terms indicate the Kullback-Leibler divergence between the latent variables

Original: C and S and their prior distributions.
Revised: C and S and their prior distributions.

Original: In practice, we assume that the priors p ( C ) and p ( S ) follow standard multivariate Gaussian distributions.
Revised: In practice, we assume that the priors p ( C ) and p ( S ) follow standard multivariate Gaussian distributions.

Original: The third and fourth terms contain the approximated log-likelihoods of label predictions and the disentanglement of the content and style.
Revised: The third and fourth terms contain the approximated log-likelihoods of label predictions and the disentanglement of the content and style.

Original: The last term stands for estimated distribution of x .
Revised: The last term stands for estimated distribution of x .

Original: Note that in Eq.
Revised: Note that in Eq.

Original: 5b, our approximated ˜ ELBO is composed of two parts: the original ELBO which could be obtained from the factorization in Eq. 1, and two regularization terms that aims to disentangle C and S through maximizing the log-likelihoods log q ϕ s ( d | c ) and log q ϕ c ( y | s ) , which is shown by the dashed lines in Fig.
Revised: 5b, our approximated ˜ ELBO is composed of two parts: the original ELBO which could be obtained from the factorization in Eq. 1, and two regularization terms that aims to disentangle C and S through maximizing the log-likelihoods log q ϕ s ( d | c ) and log q ϕ c ( y | s ) , which is shown by the dashed lines in Fig.

Original: By maximizing ˜ ELBO , we can train an accurate class predictor which is invariant to different styles.
Revised: By maximizing ˜ ELBO , we can train an accurate class predictor which is invariant to different styles.

Original: The detailed derivation is provided in supplementary material.
Revised: The detailed derivation is provided in supplementary material.

Original: Next, we introduce our data augmentation to assist in harnessing OOD examples.
Revised: Next, we introduce our data augmentation to assist in harnessing OOD examples.

Original: 3. D ATA A UGMENTATION WITH C ONTENT AND S TYLE I NTERVENTION
Revised: 2. D ATA A UGMENTATION WITH C ONTENT AND S TYLE I NTERVENTION

Original: After disentangling content and style, we try to harness OOD examples via two opposite augmentation procedures, namely positive data augmentation and negative data augmentation which aim to produce benign OOD data ¯ x and malign OOD data ˆ x , respectively, so as to further enhance model generalization and improve robustness against anomalies.
Revised: After disentangling content and style, we try to harness OOD examples via two opposite augmentation procedures, namely positive data augmentation and negative data augmentation which aim to produce benign OOD data ¯ x and malign OOD data ˆ x , respectively, so as to further enhance model generalization and improve robustness against anomalies.

Original: Specifically, to achieve this, positive data augmentation only conducts intervention on the style feature meanwhile keeping the content information the same; and the negative data augmentation attempts to affect the content feature while leaving the style unchanged, so as to produce malign OOD data, as shown in Fig.
Revised: Specifically, to achieve this, positive data augmentation only conducts intervention on the style feature meanwhile keeping the content information the same; and the negative data augmentation attempts to affect the content feature while leaving the style unchanged, so as to produce malign OOD data, as shown in Fig.

Original: 1 (b).
Revised: 1 (b).

Original: To achieve this goal, we employ adversarial data augmentation (Goodfellow et al., 2014; Miyato et al., 2018; Volpi et al., 2018) which can directly conduct intervention on the latent variables without influencing each other, thus it is perfect for our intuition of augmenting content and style.
Revised: To achieve this goal, we employ adversarial data augmentation (Goodfellow et al., 2014; Miyato et al., 2018; Volpi et al., 2018) which can directly conduct intervention on the latent variables without influencing each other, thus it is perfect for our intuition of augmenting content and style.

Original: Particularly, by adding a learnable perturbation e to each instance x , we can obtain malign OOD data ˆ x and benign OOD data ¯ x with augmented content and style, respectively.
Revised: Particularly, by adding a learnable perturbation e to each instance x , we can obtain malign OOD data ˆ x and benign OOD data ¯ x with augmented content and style, respectively.

Original: For each data point ( x , y, d ) , the perturbation e can be obtained through minimizing the intervention objective L ( · ) :
Revised: For each data point ( x , y, d ) , the perturbation e can be obtained through minimizing the intervention objective L ( · ) :

Original: [Equation]
Revised: [Equation]

Original: where ϵ denotes the magnitude of the perturbation e with ℓ p -norm.
Revised: where ϵ denotes the magnitude of the perturbation e with ℓ p -norm.

Original: Since our goal of positive and negative data augmentation is completely different, here the intervention objective is designed differently for producing ¯ x and ˆ x .
Revised: Since our goal of positive and negative data augmentation is completely different, here the intervention objective is designed differently for producing ¯ x and ˆ x .

Original: For positive data augmentation, the intervention objective is:
Revised: For positive data augmentation, the intervention objective is:

Original: [Equation]
Revised: [Equation]

Original: where the first term L d ( · ) indicates the distance measured between the contents extracted from the original instance and its perturbed version, and the second term L ce ( · ) denotes the cross-entropy loss.
Revised: where the first term L d ( · ) indicates the distance measured between the contents extracted from the original instance and its perturbed version, and the second term L ce ( · ) denotes the cross-entropy loss.

Original: By minimizing L pos , the perturbation e would not significantly affect the content feature, meanwhile introducing a novel style that is distinct from its original domain d .
Revised: By minimizing L pos , the perturbation e would not significantly affect the content feature, meanwhile introducing a novel style that is distinct from its original domain d .

Original: Consequently, the augmented benign data with novel styles can be utilized to train a style-invariant model that is resistant to domain shift.
Revised: Consequently, the augmented benign data with novel styles can be utilized to train a style-invariant model that is resistant to domain shift.

Original: Moreover, a specific style with domain label d ′ can be injected via modifying L pos as:
Revised: Moreover, a specific style with domain label d ′ can be injected via modifying L pos as:

Original: [Equation]
Revised: [Equation]

Original: Different from Eq.
Revised: Different from Eq.

Original: 7, we hope to minimize the cross-entropy loss such that the perturbed instance can contain the style information from a target domain d ′ .
Revised: 7, we hope to minimize the cross-entropy loss such that the perturbed instance can contain the style information from a target domain d ′ .

Original: As a result, the augmented benign data can successfully bridge the gap between source domain and target domain, and further improve the test performance in the target distribution.
Revised: As a result, the augmented benign data can successfully bridge the gap between source domain and target domain, and further improve the test performance in the target distribution.

Original: As for negative data augmentation, the intervention objective is defined as:
Revised: As for negative data augmentation, the intervention objective is defined as:

Original: [Equation]
Revised: [Equation]

Original: By minimizing L neg , the perturbation would not greatly change the style information but would deviate the content from its original one with class label y .
Revised: By minimizing L neg , the perturbation would not greatly change the style information but would deviate the content from its original one with class label y .

Original: Subsequently, by recognizing the augmented malign data as unknown, the trained model would be robust to deceiving anomalies with familiar styles, thus boosting the OOD detection performance.
Revised: Subsequently, by recognizing the augmented malign data as unknown, the trained model would be robust to deceiving anomalies with familiar styles, thus boosting the OOD detection performance.

Original: To accomplish the adversarial data augmentation process, here we perform multi-step projected gradient descent (Madry et al., 2018).
Revised: To accomplish the adversarial data augmentation process, here we perform multi-step projected gradient descent (Madry et al., 2018; Wang et al., 2021).

Original: Formally, the optimal ¯ x and ˆ x can be iteratively found through:
Revised: Formally, the optimal ¯ x and ˆ x can be iteratively found through:

Original: [Equation]
Revised: [Equation]

Original: where the final iteration t is set to 15 in practice.
Revised: where the final iteration t is set to 15 in practice.

Original: Further, the optimal augmented data will be incorporated into model training, which is described in the next section.
Revised: Further, the optimal augmented data will be incorporated into model training, which is described in the next section.

Original: 3. M ODEL T RAINING WITH B ENIGN AND M ALIGN OOD DATA

Original: Finally, based on the aforementioned disentanglement and data augmentation in Sections 3.1 and 3.2, we can obtain a benign OOD data ¯ x and a malign OOD data ˆ x from each data point ( x , y, d ) , which will be appended to the benign dataset ¯ D and malign dataset ˆ D , respectively.

Original: For utilization of benign OOD data ¯ x , we assign it with the original class label y and perform supervised training.

Original: For separation of malign OOD data ˆ x , we employ a one-vs-all classifier (Padhy et al., 2020) to recognize them as unknown data that is distinct from its original class label y .

Original: The proposed HOOD method is summarized in Algorithm 1.

Original: Below, we specify the proposed HOOD algorothm to three typical applications with OOD data, namely OOD detection, open-set SSL, and open-set DA.

Original: 3. D EPLOYMENT TO OOD APPLICATIONS

Original: Generally, in all three investigated applications, we are given a labeled set D l = { ( x i , y i ) } li = containing l labeled examples drawn from data distribution P l , and an unlabeled set D u = { x i } ui = composed of u unlabeled examples sampled from data distribution P u .
Revised: [Equation]

Original: Moreover, the label space of

Original: D l and D u are defined as Y l and Y u , respectively.

Original: 1: Labeled set D l
Revised: 1: Labeled set D l

Original: = { ( x i , y i ) } li =1 , unlabeled set D u
Revised: = { ( x i , y i ) } li =1 , unlabeled set D u

Original: = { ( x i ) } ui =1 .
Revised: = { ( x i ) } ui =1 .

Original: 2: for i = 1 to Max _ Iter do 3:
Revised: 2: for i = 1 to Max _ Iter do 3:

Original: Pre-train the variational inference framework through maximizing ˜ ELBO in Eq.
Revised: Pre-train the variational inference framework through maximizing ˜ ELBO in Eq.

Original: 5a; 4:
Revised: 5a; 4:

Original: Assigning pseudo labels y ps for unlabeled data
Revised: Assigning pseudo labels y ps for unlabeled data

Original: D u := { ( x i ; y psi ) } ui =1 ; 5: if i ==
Revised: D u := { ( x i ; y psi ) } ui =1 ; 5: if i ==

Original: Augmentation _
Revised: Augmentation _

Original: Iter then 6:
Revised: Iter then 6:

Original: Conduct Adversarial Data Augmentation to obtain ¯ x and ˆ x through Eq. 10; 7:
Revised: Conduct Adversarial Data Augmentation to obtain ¯ x and ˆ x through Eq. 10; 7:

Original: Add ¯ x and ˆ x into ¯ D and ˆ D , respectively; 8: end if 9:
Revised: Add ¯ x and ˆ x into ¯ D and ˆ D , respectively; 8: end if 9:

Original: Enumerate ¯ D and conduct supervised training for each ¯ x ; 10:
Revised: Enumerate ¯ D and conduct supervised training for each ¯ x ; 10:

Original: Enumerate ˆ D and recognize each ˆ x as unknown; 11: end for
Revised: Enumerate ˆ D and recognize each ˆ x as unknown; 11: end for

Revised: Finally, based on the aforementioned disentanglement and data augmentation in Sections 2.

Revised: and 2.2, we can obtain a benign OOD data ¯ x and a malign OOD data

Revised: ˆ x from each data point ( x , y, d ) , which will be appended to the benign dataset ¯ D and malign dataset ˆ D , respectively.

Revised: For utilization of benign OOD data ¯ x , we assign it with the original class label y and perform supervised training.

Revised: For separation of malign

Original: OOD detection.
Revised: OOD data ˆ x , we employ a one-vs-all classifier (Padhy et al., 2020) to recognize them as unknown data that is distinct from its original class label y . The proposed HOOD method is summarized in Algorithm 1. Below, we specify the proposed HOOD algorothm to three typical applications with OOD data, namely OOD detection, open-set SSL, and open-set DA. 2. D EPLOYMENT TO OOD APPLICATIONS

Revised: Generally, in all three investigated applications, we are given a labeled set D l = { ( x i , y i ) } li

Original: The labeled set is used for training, and the unlabeled set is used as a test set which contains both ID data and malign
Revised: = containing l labeled examples drawn from data distribution P l , and an unlabeled set D u = { x i } ui = composed of u unlabeled examples sampled from data distribution P u . Moreover, the label space of

Revised: D l and D u are defined as Y l and Y u , respectively.

Original: OOD data.
Revised: OOD detection.

Revised: The labeled set is used for training, and the unlabeled set is used as a test set which contains both ID data and malign OOD data.

Original: Particularly, the data distribution of unlabeled ID data Q id is the same as distribution P , but the distribution of OOD data P uood is different from P , i.e. , P uid 
Revised: Particularly, the data distribution of unlabeled ID data Q id is the same as distribution P , but the distribution of OOD data P uood is different from P , i.e. , P uid =

Original: = P l ̸ = P uood .
Revised: P l ̸ = P uood .

Original: The goal is to correctly distinguish OOD data from ID data in the test phase.
Revised: The goal is to correctly distinguish OOD data from ID data in the test phase.

Original: During training, we conduct data augmentation to obtain domain label d , and then follow the workflow described by Algorithm 1 to obtain the model parameters.
Revised: During training, we conduct data augmentation to obtain domain label d , and then follow the workflow described by Algorithm 1 to obtain the model parameters.

Original: During test, we only use the  content branch to predict the OOD score which is produced by the one-vs-all classifier.
Revised: During test, we only use the [Table] content branch to predict the OOD score which is produced by the one-vs-all classifier.

Original: An instance is considered as an ID datum if the OOD score is smaller than 0 . 5 , and an OOD datum otherwise.
Revised: An instance is considered as an ID datum if the OOD score is smaller than 0 . 5 , and an OOD datum otherwise.

Original: Open-set SSL.
Revised: Open-set SSL.

Original: The labeled set D l and unlabeled set D u are both used for training, and they are sampled from the same data distribution with different label spaces.
Revised: The labeled set D l and unlabeled set D u are both used for training, and they are sampled from the same data distribution with different label spaces.

Original: Specifically, the unlabeled data contain some ID data that have the same classes as D l , and the rest unlabeled OOD data are from some unknown classes that do not exist in D l , formally, Y l ⊂ Y u , Y u \ Y l ̸ = ∅ and
Revised: Specifically, the unlabeled data contain some ID data that have the same classes as D l , and the rest unlabeled OOD data are from some unknown classes that do not exist in D l , formally, Y l ⊂ Y u , Y u \ Y l ̸ = ∅ and

Original: P l ( x | y )
Revised: P l ( x | y )

Original: =
Revised: =

Original: P u ( x | y ) , y ∈ Y l .
Revised: P u ( x | y ) , y ∈ Y l .

Original: The goal is to properly leverage the labeled data and unlabeled ID data without being misled by malign OOD data, and correctly classify test data with labels in Y l .
Revised: The goal is to properly leverage the labeled data and unlabeled ID data without being misled by malign OOD data, and correctly classify test data with labels in Y l .

Original: The training process is similar to OOD detection, except that HOOD would produce an OOD score for each unlabeled data.
Revised: The training process is similar to OOD detection, except that HOOD would produce an OOD score for each unlabeled data.

Original: If an unlabeled instance is recognized as OOD data, it would be left out.
Revised: If an unlabeled instance is recognized as OOD data, it would be left out.

Original: Open-set DA.
Revised: Open-set DA.

Original: The labeled set is drawn from source distribution P l which is different from the target distribution P u of unlabeled set.
Revised: The labeled set is drawn from source distribution P l which is different from the target distribution P u of unlabeled set.

Original: In addition, the label space Y l is also a subset of Y u .
Revised: In addition, the label space Y l is also a subset of Y u .

Original: Therefore, the unlabeled data consist of benign OOD data which have the same class labels as labeled data, and malign OOD data which have distinct data distribution as well as class labels from labeled data, formally, P l ̸
Revised: Therefore, the unlabeled data consist of benign OOD data which have the same class labels as labeled data, and malign OOD data which have distinct data distribution as well as class labels from labeled data, formally, P l ̸

Original: = P u , Y l ⊂ Y u , Y u \ Y l ̸ = ∅ .
Revised: = P u , Y l ⊂ Y u , Y u \ Y l ̸ = ∅ .

Original: The goal is to transfer the knowledge of labeled data to the benign OOD data, meanwhile identify the malign OOD data as unknown.
Revised: The goal is to transfer the knowledge of labeled data to the benign OOD data, meanwhile identify the malign OOD data as unknown.

Original: In this application, we assign each target instance with a domain label to distinguish them from other augmented data.
Revised: In this application, we assign each target instance with a domain label to distinguish them from other augmented data.

Original: Then we alter the positive data augmentation objective from Eq.
Revised: Then we alter the positive data augmentation objective from Eq.

Original: 7 to Eq.
Revised: 7 to Eq.

Original: 8 and train the framework through Algorithm 1.
Revised: 8 and train the framework through Algorithm 1.

Original: During test, HOOD would predict each target instance as some class if it is benign OOD data, and as unknown otherwise.
Revised: During test, HOOD would predict each target instance as some class if it is benign OOD data, and as unknown otherwise.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we first describe the implementation details.
Revised: In this section, we first describe the implementation details.

Original: Then, we experimentally validate our method on three applications, namely OOD detection, open-set SSL, and open-set DA.
Revised: Then, we experimentally validate our method on three applications, namely OOD detection, open-set SSL, and open-set DA.

Original: Finally, we present extensive performance analysis on our disentanglement and intervention modules.
Revised: Finally, we present extensive performance analysis on our disentanglement and intervention modules.

Original: Additional details and quantitative findings can be found in the supplementary material.
Revised: Additional details and quantitative findings can be found in the supplementary material.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 3. I MPLEMENTATION D ETAILS

Original: In experiments, we choose Wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) for OOD detection and Open-set SSL tasks, and follow (You et al., 2020; Cao et al., 2019) to utilize ResNet50 pre-trained on Imagenet (Russakovsky et al., 2015) for Open-set DA.
Revised: In experiments, we choose Wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) for OOD detection and Open-set SSL tasks, and follow (You et al., 2020; Cao et al., 2019) to utilize ResNet50 pre-trained on Imagenet (Russakovsky et al., 2015) for Open-set DA.

Original: For implementing HOOD, we randomly choose 4 augmentation methods from the transformation pool in RandAugment (Cubuk et al., 2020), to simulate different styles.
Revised: For implementing HOOD, we randomly choose 4 augmentation methods from the transformation pool in RandAugment (Cubuk et al., 2020), to simulate different styles.

Original: The pre-training iteration Augmentation _ Iter is set to 100,000, and the perturbation magnitude ϵ = 0 .
Revised: The pre-training iteration Augmentation _ Iter is set to 100,000, and the perturbation magnitude ϵ = 0 .

Original: 03 , following (Volpi et al., 2018) in all experiments.
Revised: 03 , following (Volpi et al., 2018) in all experiments.

Original: Next, we will experimentally validate HOOD in three applications.
Revised: Next, we will experimentally validate HOOD in three applications.

Original: 4. OOD DETECTION
Revised: 3. OOD DETECTION

Original: In OOD detection task, we use SVHN (Netzer et al., 2011) and CIFAR10 (Krizhevsky et al., 2009) as the ID datasets, and use LSUN (Yu et al., 2015), DTD (Cimpoi et al., 2014), CUB (Wah et al., 2011), Flowers (Nilsback & Zisserman, 2006), Caltech (Griffin et al., 2007), and Dogs (Khosla et al., 2011) datasets as the OOD datasets that occur during test phase.
Revised: In OOD detection task, we use SVHN (Netzer et al., 2011) and CIFAR10 (Krizhevsky et al., 2009) as the ID datasets, and use LSUN (Yu et al., 2015), DTD (Cimpoi et al., 2014), CUB (Wah et al., 2011), Flowers (Nilsback & Zisserman, 2006), Caltech (Griffin et al., 2007), and Dogs (Khosla et al., 2011) datasets as the OOD datasets that occur during test phase.

Original: Particularly,  we  sample 100 labeled data from each class andthen conduct supervised training, then we test the trained model on the  OOD dataset.
Revised: Particularly, to explore the model generalization ability, we only sample 100 labeled data from each class and conduct supervised training, then we test the trained model on the unlabeled OOD dataset.

Original: To evaluate the performance,we utilize AUROC (Hendrycks & Gimpel, 2017) which is an essential metric for OOD detection, and a higher AUROC value indicates a better performance.
Revised: To evaluate the performance,

Original: [Table]
Revised: [Table]

Original:  For comparison, we choose some typical OOD detection methods including Likelihood (Hendrycks & Gimpel, 2017) which simply utilizes softmax score as the detection criterion, ODIN (Liang et al., 2018) which enhances the performance of Likelihood through adding adversarial attack, Likelihood
Revised: we utilize AUROC (Hendrycks & Gimpel, 2017) which is an essential metric for OOD detection, and a higher AUROC value indicates a better performance. For comparison, we choose some typical OOD detection methods including Likelihood (Hendrycks & Gimpel, 2017) which simply utilizes softmax score as the detection criterion, ODIN (Liang et al., 2018) which enhances the performance of Likelihood through adding adversarial attack, Likelihood

Original: Ratio (Ren et al., 2019) which modifies the softmax score through focusing on the semantic feature, and OpenGAN (Kong & Ramanan, 2021) which can further improve the performance via separating the generated “hard” examples that are deceivingly close to ID data.
Revised: Ratio (Ren et al., 2019) which modifies the softmax score through focusing on the semantic feature, and OpenGAN (Kong & Ramanan, 2021) which can further improve the performance via separating the generated “hard” examples that are deceivingly close to ID data.

Original: The experimental results are shown in Table 1, we can see that HOOD can greatly surpass Likelihood,
Revised: The experimental results are shown in Table 1, we can see that HOOD can greatly surpass Likelihood,

Original: ODIN, and Likelihood Ratio, and can outperform OpenGAN in most scenarios.
Revised: ODIN, and Likelihood Ratio, and can outperform OpenGAN in most scenarios.

Original: When compared with softmax-prediction-based methods such as Likelihood and ODIN, HOOD surpasses them in a large margin, as HOOD can correctly separate some overconfident OOD examples from ID data.
Revised: When compared with softmax-prediction-based methods such as Likelihood and ODIN, HOOD surpasses them in a large margin, as HOOD can correctly separate some overconfident OOD examples from ID data.

Original: As for
Revised: As for

Original: Likelihood Ratio, our method can achieve better performance through producing “hard” malign OOD data, thus successfully avoiding deceiving examples that are extremely close to ID data.
Revised: Likelihood Ratio, our method can achieve better performance through producing “hard” malign OOD data, thus successfully avoiding deceiving examples that are extremely close to ID data.

Original: Although both OpenGAN and HOOD generate “hard” malign data to train an open classifier, HOOD can successfully distinguish content and style thanks to the aforementioned disentanglement, thus avoid rejecting too much benign OOD data and further yield better detection performance than OpenGAN.
Revised: Although both OpenGAN and HOOD generate “hard” malign data to train an open classifier, HOOD can successfully distinguish content and style thanks to the aforementioned disentanglement, thus avoid rejecting too much benign OOD data and further yield better detection performance than OpenGAN.

Original: 4. O PEN -S ET SSL
Revised: 3. O PEN -S ET SSL

Original: In open-set SSL task, we follow (Guo et al., 2020) to construct our training dataset using two benchmark datasets CIFAR10 and CIFAR100 (Krizhevsky et al., 2009), which contains 10 andclasses, respectively.
Revised: In open-set SSL task, we follow (Guo et al., 2020) to construct our training dataset using two benchmark datasets CIFAR10 and CIFAR100 (Krizhevsky et al., 2009), which contains 10 andclasses, respectively.

Original: The constructed dataset has 20,000 randomly sampled unlabeled data and a varied number of labeled data.
Revised: The constructed dataset has 20,000 randomly sampled unlabeled data and a varied number of labeled data.

Original: Here the number of labeled data is set to 50, 100, and 400 per class in both CIFAR10 and CIFAR100.
Revised: Here the number of labeled data is set to 50, 100, and 400 per class in both CIFAR10 and CIFAR100.

Original: Moreover, to create the open-set problem in CIFAR10, the unlabeled data is sampled from all 10 classes and the labeled data is sampled from the 6 animal classes.
Revised: Moreover, to create the open-set problem in CIFAR10, the unlabeled data is sampled from all 10 classes and the labeled data is sampled from the 6 animal classes.

Original: As for CIFAR100, the unlabeled data are sampled from all 100 classes and the labeled data is sampled from the first 60 classes.
Revised: As for CIFAR100, the unlabeled data are sampled from all 100 classes and the labeled data is sampled from the first 60 classes.

Original: For evaluation, we first use the test dataset from the original CIFAR and CIFAR100 and denote the test accuracy as “Clean Acc.”.
Revised: For evaluation, we first use the test dataset from the original CIFAR and CIFAR100 and denote the test accuracy as “Clean Acc.”.

Original: Further, to evaluate the capability of handling OOD examples, we test on CIFAR10-C and CIFAR100-C (Hendrycks & Dietterich, 2019) which add different types of corruptions to CIFAR10 and CIFAR100, respectively.
Revised: Further, to evaluate the capability of handling OOD examples, we test on CIFAR10-C and CIFAR100-C (Hendrycks & Dietterich, 2019) which add different types of corruptions to CIFAR10 and CIFAR100, respectively.

Original: The test accuracy from the corrupted datasets can reveal the robustness of neural networks against corruptions and perturbations, and it is denoted as “Corrupted Acc.”.
Revised: The test accuracy from the corrupted datasets can reveal the robustness of neural networks against corruptions and perturbations, and it is denoted as “Corrupted Acc.”.

Original: For comparison, we choose some typical open-set SSL methods including Uncertainty-Aware SelfDistillation method UASD (Chen et al., 2020) and T2T (Huang et al., 2021) which filters out the
Revised: For comparison, we choose some typical open-set SSL methods including Uncertainty-Aware SelfDistillation method UASD (Chen et al., 2020) and T2T (Huang et al., 2021a) which filters out the

Original: OOD data via using OOD detection, Safe Deep Semi-Supervised Learning DS3L (Guo et al., 2020) which employs meta-learning to down-weight the OOD data,
Revised: OOD data via using OOD detection, Safe Deep Semi-Supervised Learning DS3L (Guo et al., 2020) which employs meta-learning to down-weight the OOD data,

Original: Multi-Task Curriculum Framework
Revised: Multi-Task Curriculum Framework

Original: MTCF (Yu et al., 2020) which recognizes the OOD data as different domain, and OpenMatch (Saito et al., 2021) which utilizes open-set consistency training on OOD data.
Revised: MTCF (Yu et al., 2020) which recognizes the OOD data as different domain, and OpenMatch (Saito et al., 2021) which utilizes open-set consistency training on OOD data.

Original: The experimental results are shown in Table 2.
Revised: The experimental results are shown in Table 2.

Original: Compared to the strongest baseline method OpenMatch, which randomly samples eleven different transformations from a transformation pool, our method has transformations that are limited to only four types.
Revised: Compared to the strongest baseline method OpenMatch, which randomly samples eleven different transformations from a transformation pool, our method has transformations that are limited to only four types.

Original: In CIFAR10 and CIFAR100 regarding the Clean Acc., the proposed HOOD is slightly outperformed by OpenMatch.
Revised: In CIFAR10 and CIFAR100 regarding the Clean Acc., the proposed HOOD is slightly outperformed by OpenMatch.

Original: However, thanks to  the disentanglement, HOOD can be invariant to different styles and focus on the content feature.
Revised: However, thanks to [Table] the disentanglement, HOOD can be invariant to different styles and focus on the content feature.

Original: Therefore, when facing corruption, HOOD can be more robust than all baseline methods.
Revised: Therefore, when facing corruption [Table] HOOD can be more robust than all baseline methods.

Original: As shown by the Corrupted Acc. results, our method surpasses OpenMatch for more than 3% .
Revised: As shown by the Corrupted Acc. results, our method surpasses OpenMatch for more than 3% .

Original: 4. O PEN -S ET DA
Revised: 3. O PEN -S ET DA

Original: In open-set DA task, we follow (Saito et al., 2018) to validate on two DA benchmark datasets
Revised: In open-set DA task, we follow (Saito et al., 2018) to validate on two DA benchmark datasets

Original: Office (Saenko et al., 2010) and VisDA (Peng et al., 2018).
Revised: Office (Saenko et al., 2010) and VisDA (Peng et al., 2018).

Original: Office dataset contains three domains
Revised: Office dataset contains three domains

Original: [Table]

Original: [Table]

Original: Amazon (A), Webcam (W)
Revised: Amazon (A), Webcam (W)

Original: [Table]
Revised: [Table]

Original: and DSLR (D), and each domain is composed of 31 classes.
Revised: and DSLR (D) [Table] and each domain is composed of 31 classes.

Original: VisDA dataset contains two domains Sythetic and Real, and each domain consists of 12 classes.
Revised: VisDA dataset contains two domains Sythetic and Real, and each domain consists of 12 classes.

Original: To create an open-set situation in Office, we follow (Saito et al., 2018;
Revised: To create an open-set situation in Office, we follow (Saito et al., 2018;

Original: Liu et al., 2019) to construct the source dataset by sampling from the first 21 classes in alphabetical order.
Revised: Liu et al., 2019) to construct the source dataset by sampling from the first 21 classes in alphabetical order.

Original: Then, the target dataset is sampled from all 31 classes.
Revised: Then, the target dataset is sampled from all 31 classes.

Original: As for VisDA, we choose the first 6 classes for source domain, and use all theclasses for target domain.
Revised: As for VisDA, we choose the first 6 classes for source domain, and use all theclasses for target domain.

Original: We use “A → W” to indicate the transfer from “A” domain to “W” domain.
Revised: We use “A → W” to indicate the transfer from “A” domain to “W” domain.

Original: For comparison, we choose three typical open-set DA approaches including Open-Set DA by BackPropagation OSBP (Saito et al., 2018) which employs an OpenMax classifier to recognize unknown classes and perform gradient flipping for open-set DA, Universal Adaptation Network UAN (You et al., 2020) which utilize entropy and domain similarity to down-weight malign OOD data, and
Revised: For comparison, we choose three typical open-set DA approaches including Open-Set DA by BackPropagation OSBP (Saito et al., 2018) which employs an OpenMax classifier to recognize unknown classes and perform gradient flipping for open-set DA, Universal Adaptation Network UAN (You et al., 2020) which utilize entropy and domain similarity to down-weight malign OOD data, and

Original: Separate To Adapt STA (Liu et al., 2019) which utilizes SVM to separate the malign OOD data.
Revised: Separate To Adapt STA (Liu et al., 2019) which utilizes SVM to separate the malign OOD data.

Original: The experimental results are shown in Table 3.
Revised: The experimental results are shown in Table 3.

Original: Compared to the baseline methods, the proposed
Revised: Compared to the baseline methods, the proposed

Original: HOOD is largely benefited from the generated benign OOD data, which have two major strengths: 1) they resemble target domain data by having common styles, and 2) their labels are accessible as they share the same content as their corresponding source data.
Revised: HOOD is largely benefited from the generated benign OOD data, which have two major strengths: 1) they resemble target domain data by having common styles, and 2) their labels are accessible as they share the same content as their corresponding source data.

Original: Therefore, through conducting supervised training such benign OOD data, the domain gap can be further mitigated, thus achieving better performance than baseline methods.
Revised: Therefore, through conducting supervised training such benign OOD data, the domain gap can be further mitigated, thus achieving better performance than baseline methods.

Original: Quantitative results show that HOOD can surpass other methods in most scenarios.
Revised: Quantitative results show that HOOD can surpass other methods in most scenarios.

Original: Especially in VisDA, HOOD can outperform the second-best method with 6 % improvement, which proves the effectiveness of HOOD in dealing with open-set DA.
Revised: Especially in VisDA, HOOD can outperform the second-best method with 6 % improvement, which proves the effectiveness of HOOD in dealing with open-set DA.

Original: 4. P ERFORMANCE A NALYSIS
Revised: 3. P ERFORMANCE A NALYSIS

Original: [Table]
Revised: [Table]

Original: Ablation Study: To verify the effectiveness of each module, we conduct an ablation study on three OOD applications by eliminating one component at a time.
Revised: Ablation Study: To verify the effectiveness of each module, we conduct an ablation study on three OOD applications by eliminating one component at a time.

Original: Specifically, our HOOD can be ablated into: “w/o disentanglement” which indicates removing the disentanglement loss in Eq.
Revised: Specifically, our HOOD can be ablated into: “w/o disentanglement” which indicates removing the disentanglement loss in Eq.

Original: 5a, “w/o benign OOD data” which denotes training without benign
Revised: 5a, “w/o benign OOD data” which denotes training without benign

Original: OOD data, “w/o malign OOD data” which stands for discarding malign OOD data, and “w/o both augmentations” indicates training without both benign and malign OOD data.
Revised: OOD data, “w/o malign OOD data” which stands for discarding malign OOD data, and “w/o both augmentations” indicates training without both benign and malign OOD data.

Original: Here in OOD detection, we use CIFAR10 as the ID dataset, and use LSUN as the OOD dataset.
Revised: Here in OOD detection, we use CIFAR10 as the ID dataset, and use LSUN as the OOD dataset.

Original: In open-set SSL, we choose
Revised: In open-set SSL, we choose

Original: CIFAR10 with 400 labels for each class.
Revised: CIFAR10 with 400 labels for each class.

Original: As for open-set DA, we use VisDA dataset.
Revised: As for open-set DA, we use VisDA dataset.

Original: The experimental results are shown in Table 4.
Revised: The experimental results are shown in Table 4.

Original: We can see each module influences the performance differently in three applications.
Revised: We can see each module influences the performance differently in three applications.

Original: First, we can see that the malign OOD data is essential for OOD detection, as it can act as unknown anomalies and reduce the overconfidence in unseen data.
Revised: First, we can see that the malign OOD data is essential for OOD detection, as it can act as unknown anomalies and reduce the overconfidence in unseen data.

Original: Then, benign OOD data can largely improve the learning performance in open-set SSL and open-set DA, as they can enforce the model to focus on the content feature for classification.
Revised: Then, benign OOD data can largely improve the learning performance in open-set SSL and open-set DA, as they can enforce the model to focus on the content feature for classification.

Original: Additionally, we can see that discarding both benign and malign OOD data shows performance degradation compared to both 
Revised: Additionally, we can see that discarding both benign and malign OOD data shows performance degradation compared to both “w/o benign OOD data” and “w/o malign OOD data”. Therefore, our HOOD can correctly change the style and content, which can correspondingly benefit generalization tasks (such as open-set DA

Original: O r i g i n a l
Revised: O r i g i n a l

Original: N e g a ti v e
Revised: N e g a ti v e

Original: P o s iti v e ship ship airplane airplane airplane bird deer deer deer bird bird frog dog dog deer cat cat deer cat cat dog car car frog“w/o benign OOD data” and “w/o malign OOD data”. Therefore, our HOOD can correctly change the style and content, which can correspondingly benefit generalization tasks (such as open-set DA and open-set SSL) and detection tasks (such as OOD detection).
Revised: P o s iti v e ship ship airplane airplane airplane bird deer deer deer bird bird frog dog dog deer cat cat deer cat cat dog car car frog and open-set SSL) and detection tasks (such as OOD detection).

Original: Moreover, open-set DA relies more on the disentanglement than the rest two modules, owing to the disentanglement can exclude the style changing across different domains.
Revised: Moreover, open-set DA relies more on the disentanglement than the rest two modules, owing to the disentanglement can exclude the style changing across different domains.

Original: Hence, our disentanglement can effectively eliminate the distribution change from different domains and help learn invariant features.
Revised: Hence, our disentanglement can effectively eliminate the distribution change from different domains and help learn invariant features.

Original: Analysis on Augmentation Number:
Revised: Analysis on Augmentation Number:

Original: Since
Revised: Since

Original: HOOD does not introduce any hyper-parameter, the most influential setting is the number of data augmentation.
Revised: HOOD does not introduce any hyper-parameter, the most influential setting is the number of data augmentation.

Original: To analyze its influence on the learning results, we vary the number of augmentations that are sampled from the RandAugment
Revised: To analyze its influence on the learning results, we vary the number of augmentations that are sampled from the RandAugment

Original: Pool (Cubuk et al., 2020) from 2 to 6.
Revised: Pool (Cubuk et al., 2020) from 2 to 6.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: We can see that both too less and too many augmentations would hurt the results.
Revised: We can see that both too less and too many augmentations would hurt the results.

Original: This is because a small augmentation number would undermine the generalization to various styles; and a large augmentation number would increase the classification difficulty of the style branch, further making the disentanglement hard to achieve.
Revised: This is because a small augmentation number would undermine the generalization to various styles; and a large augmentation number would increase the classification difficulty of the style branch, further making the disentanglement hard to achieve.

Original: Therefore, setting the augmentation number to 4 is reasonable.
Revised: Therefore, setting the augmentation number to 4 is reasonable.

Original: Visualization: Furthermore, to show the effect of our data augmentations, we visualize the augmented images by applying large perturbation magnitude ( 4 . 7 )
Revised: Visualization: Furthermore, to show the effect of our data augmentations, we visualize the augmented images by applying large perturbation magnitude ( 4 . 7 )

Original: Tsipras et al.
Revised: Tsipras et al.

Original: (2018) in Fig.
Revised: (2018) in Fig.

Original: The model prediction is shown below each image.
Revised: The model prediction is shown below each image.

Original: We can see that the negative data augmentation significantly changes the content which is almost unidentifiable.
Revised: We can see that the negative data augmentation significantly changes the content which is almost unidentifiable.

Original: However, positive data augmentation can still preserve most of the content information and only change the style of images.
Revised: However, positive data augmentation can still preserve most of the content information and only change the style of images.

Original: Therefore, the augmented data can be correctly leveraged to help train a robust classifier.
Revised: Therefore, the augmented data can be correctly leveraged to help train a robust classifier.

Original: Disentanglement of Content and Style: To further testify that our disentanglement between content and style is effective, we select the latent variables from different content and style categories, and use the learned class and domain classifiers for cross-prediction.
Revised: Disentanglement of Content and Style: To further testify that our disentanglement between content and style is effective, we select the latent variables from different content and style categories, and use the learned class and domain classifiers for cross-prediction.

Original: Specifically, there are four kinds of input-prediction types: content-class, content-domain, style-class, and style-domain.
Revised: Specifically, there are four kinds of input-prediction types: content-class, content-domain, style-class, and style-domain.

Original: As we can see in Fig.
Revised: As we can see in Fig.

Original: 5, only the content features are meaningful for class prediction, and the same phenomenon goes for style input and domain prediction.
Revised: 3, only the content features are meaningful for class prediction, and the same phenomenon goes for style input and domain prediction.

Original: However, neither of the style and content features can be identified by the class predictor and domain predictor, respectively.
Revised: However, neither of the style and content features can be identified by the class predictor and domain predictor, respectively.

Original: Therefore, we can reasonably conclude that our disentanglement between content and style is effectively achieved.
Revised: Therefore, we can reasonably conclude that our disentanglement between content and style is effectively achieved.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose HOOD to effectively harness OOD examples.
Revised: In this paper, we propose HOOD to effectively harness OOD examples.

Original: Specifically, we construct a
Revised: Specifically, we construct a

Original: SCM to disentangle content and style, which can be leveraged to identify benign and malign OOD data.
Revised: SCM to disentangle content and style, which can be leveraged to identify benign and malign OOD data.

Original: Subsequently, by maximizing ELBO, we can successfully disentangle the content and style feature and break the spurious correlation between class and domain.
Revised: Subsequently, by maximizing ELBO, we can successfully disentangle the content and style feature and break the spurious correlation between class and domain.

Original: As a result, HOOD can be more robust when facing distribution shifts and unseen OOD data.
Revised: As a result, HOOD can be more robust when facing distribution shifts and unseen OOD data.

Original: Furthermore, we augment the content and style through a novel intervention process to produce benign and malign OOD data, which can be leveraged to improve classification and OOD detection performance.
Revised: Furthermore, we augment the content and style through a novel intervention process to produce benign and malign OOD data, which can be leveraged to improve classification and OOD detection performance.

Original: Extensive experiments are conducted to empirically validate the effectiveness of HOOD on three typical OOD applications.
Revised: Extensive experiments are conducted to empirically validate the effectiveness of HOOD on three typical OOD applications.

Original:  A A PPENDIX
Revised:  A CKNOWLEDGEMENT

Revised: ZH was supported by JD Technology Scholarship for Postgraduate Research in Artificial Intelligence

Revised: No. SC4103.

Revised: XX was supported by Australian Research Council Projects DE-190101473 and Google

Revised: PhD Fellowship.

Revised: LS was partially supported by the Major Science and Technology Innovation 2030 “New Generation Artificial Intelligence” key project No. 2021ZD0111700.

Revised: BH was supported by NSFC Young Scientists Fund No. 62006202, Guangdong Basic and Applied Basic Research

Revised: Foundation No. 2022A1515011652 and RGC Early Career Scheme No. 22200720.

Revised: MG was supported by ARC DE210101624.

Revised: CG was supported by NSF of China No. 61973162, NSF of

Revised: Jiangsu Province No. BZ2021013, NSF for Distinguished Young Scholar of Jiangsu Province No.

Revised: BK20220080, and CAAI-Huawei MindSpore Open Fund.

Original:  In this supplementary material, we first complement the implementation details of HOOD in Section4.1.
Revised: TL was partially supported by Australian Research Council Projects IC-190100031, LP-220100527, DP-220102121, and FT-220100318. A A PPENDIX In this supplementary material, we first discuss some related work in Section

Revised: B.

Original: Then, we  present the detailed derivation of ELBO in Section C. Moreover, to further understand the effectiveness of HOOD, we provide additional analysis in
Revised: Then, we complement the implementation details of HOOD in Section C. Further, we present the detailed derivation of ELBO in Section D. Additionally, we discuss two notable causal graph assumptions in Section E. Moreover, to further understand the effectiveness of HOOD, we provide additional analysis in

Original: Section D. Finally, we will discuss the limitations and social impact of our method in Section E.
Revised: Section F. Finally, we will discuss the limitations and social impact of our method in Section G. B R ELATED W ORK

Revised: OOD applications contains three typical problems, namely OOD detection, open-set SSL, and openset DA.

Revised: OOD detection aims to train a robust model which can accurately identify the newly-emerged malign OOD data during the test phase.

Revised: Open-set SSL deals with the problem when labeled data are scarce and the unlabeled data are contaminated by malign OOD data.

Original: B C OMPLEMENTARY D ETAILS
Revised: As for open-set DA, it tries to transfer the knowledge from source ID data to the benign OOD data in target domain, meanwhile detecting the malign OOD data that are encountered during transferring. In both three applications, the predictive confidence has been frequently leveraged to separate malign OOD data (Hendrycks & Gimpel, 2017; Liang et al., 2018; Ren et al., 2019; Xia et al., 2022a). Moreover, ID data and OOD data can be distinguished via using a discriminator (Kong & Ramanan, 2021;

Revised: Neal et al., 2018;

Revised: Yu et al., 2020; Xia et al., 2021).

Revised: Further, various open classifiers are designed to predict OOD dataset as unknown (Ge et al., 2017; Padhy et al., 2020;

Revised: Saito et al., 2018).

Revised: Thanks to the advances in unsupervised learning, many approaches employ self-supervised learning to make ID data and OOD data separable (Cao et al., 2022; Li et al., 2021a;

Revised: Saito et al., 2020) from each other.

Revised: Causality in OOD problems mainly focuses on learning invariant representations that stay constant when other causal factors are changing, thus achieving better performance when facing non-stationary data distribution.

Revised: To accomplish this goal, it is common to learn causal factors and non-causal factors through the variational auto-encoder framework (Blei et al., 2017; Kingma & Welling, 2013).

Revised: Thanks to which, domain adaptation (Gong et al., 2016; Schölkopf et al., 2011; Zhang et al., 2013) and domain generalization (Li et al., 2018; Shankar et al., 2018) can be tackled through extracting the domain invariant features.

Revised: Moreover, based on causal effects, the biased feature can be eliminated through re-weighting (Bahadori et al., 2017; Shen et al., 2018).

Revised: Additionally, the spurious correlation which is harmful for inference could be alleviated through do-calculus (Lee et al., 2021;

Revised: Nam et al., 2020; Pearl, 2009).

Revised: Recent methods (Ilse et al., 2021; Mitrovic et al., 2020; Von Kügelgen et al., 2021) conduct data augmentations with self-supervised learning to train a robust model that can handle distribution shifts and corruptions.

Revised: In general, HOOD has two major differences from existing methods in OOD applications and causality.

Revised: On one hand, instead of treating an image instance as a whole as commonly done in many approaches, HOOD can properly leverage OOD examples through their disentangled contents and styles.

Revised: Moreover, augmenting content and style can help improve generalization and robustness simultaneously.

Revised: On the other hand, current causal approaches are incapable of dealing with malign

Revised: OOD data, but HOOD is able to learn style-invariant features from benign OOD data, meanwhile avoiding the damage brought by malign OOD data.

Revised: C C OMPLEMENTARY D ETAILS

Original: Each trial of our experiments is conducted in one single TITAN V GPU.
Revised: Each trial of our experiments is conducted in one single NVIDIA 3090 GPU.

Original: In the open-set SSL and
Revised: In the open-set SSL and

Original: OOD detection tasks, we follow (Saito et al., 2021) by using the Wide ResNet-28-2 (Zagoruyko &
Revised: OOD detection tasks, we follow (Saito et al., 2021) by using the Wide ResNet-28-2 (Zagoruyko &

Original: Komodakis, 2016) as the network backbone and train from scratch.
Revised: Komodakis, 2016) as the network backbone and train from scratch.

Original: In open-set DA application, we follow (Saito et al., 2018) to fine-tune ResNet50 pre-trained on Imagenet (Russakovsky et al., 2015).
Revised: In open-set DA application, we follow (Saito et al., 2018) to fine-tune ResNet50 pre-trained on Imagenet (Russakovsky et al., 2015).

Original: The employed Stochastic Gradient Descent (SGD) optimizer starts with an initial learning rate 3 e −which is decayed by following the cosine function cos ( const × current _ iteration 500 ,) without warm-up, in which const is constant, we follow (Saito et al., 2021) to set it as 7 16 π .
Revised: The employed Stochastic Gradient Descent (SGD) optimizer starts with an initial learning rate 3 e −which is decayed by following the cosine function cos ( const × current _ iteration 500 ,) without warm-up, in which const is constant, we follow (Saito et al., 2021) to set it as 716 π .

Original: The momentum factor is set to 0 .
Revised: The momentum factor is set to 0 .

Original: 9 , which is also the same as (Saito et al., 2021).
Revised: 9 , which is also the same as (Saito et al., 2021).

Original: For choosing the pseudo labels of unlabeled data, we follow (Lee, 2013) to set the pseudo label threshold as 0 .
Revised: For choosing the pseudo labels of unlabeled data, we follow (Lee, 2013) to set the pseudo label threshold as 0 .

Original: The unlabeled examples with confidence smaller than 0 .
Revised: The unlabeled examples with confidence smaller than 0 .

Original: 95 are excluded from training the class branch.
Revised: 95 are excluded from training the class branch.

Original: However, all of them are leveraged to optimize the domain branch.
Revised: However, all of them are leveraged to optimize the domain branch.

Original: Moreover, the content and style features are the output of the penultimate layer, which are further fed into the fully-connected layer to produce the class and domain prediction.
Revised: Moreover, the content and style features are the output of the penultimate layer, which are further fed into the fully-connected layer to produce the class and domain prediction.

Original: Here the class number is decided by the class labels, and the domain number is the number of augmentations plus one logit that stands for the original input.
Revised: Here the class number is decided by the class labels, and the domain number is the number of augmentations plus one logit that stands for the original input.

Original: C D ERIVATION OF ELBO
Revised: D D ERIVATION OF ELBO

Original: In the main paper, the modified structural causal model is factorized as:
Revised: In the main paper, the modified structural causal model is factorized as:

Original: [Equation]
Revised: [Equation]

Original: We employ two encoders to model the distribution of content and style, respectively:
Revised: We employ two encoders to model the distribution of content and style, respectively:

Original: [Equation]
Revised: [Equation]

Original: Moreover [Equation] we utilize two classifiers to model the posteriors of class and domain, respectively:
Revised: Moreover, we utilize two classifiers to model the posteriors of class and domain, respectively:

Original: [Equation]
Revised: [Equation]

Original: Additionally [Equation] a decoder is employed to reconstruct the input instance through following distribution:
Revised: Additionally, a decoder is employed to reconstruct the input instance through following distribution:

Original: [Equation]
Revised: [Equation]

Original: Then [Equation] our goal is to maximize the log-likelihood of the joint distribution p ( x , y, d ) :
Revised: Then, our goal is to maximize the log-likelihood of the joint distribution p ( x , y, d ) :

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: By applying Eq.
Revised: By applying Eq.

Original: 2 to p ′
Revised: 2 to p ′

Original: ( x , y, d, c, s ) , we have:
Revised: ( x , y, d, c, s ) , we have:

Original: ELBO ( x , y, d ) =
Revised: ELBO ( x , y, d ) =

Original: E ( c,s ) ∼ q θ ( C,S | x ) (cid:20) [Equation] [Equation] ) q 
Revised: E ( c,s ) ∼ q θ ( C,S | x ) (cid:20) log p ( c ) p ( s ) q ϕ c ( y | c ) q ϕ s ( d | s )

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: d

Original: [Equation]

Original: c ) (cid:21) =

Original: E

Original: [Equation] c [Equation] s )
Revised: p ψ ( x | c, s )

Original: [Equation] q θ ( [Equation] S | x ) (cid:20) log
Revised: q θ ( c, s | x ) q ϕ c ( y | s ) q ϕ s ( d | c ) (cid:21)

Original: [Equation]

Original: c )

Original: [Equation] s ) q [Equation]
Revised: =

Original: ( c

Original: [Equation]

Original: x )

Original: [Equation]

Original: s

Original: [Equation] x ) (cid:21) [Equation] E ( c,s ) ∼ q θ ( C,S | x ) (cid:20) log q ϕ c ( y | c ) q ϕ s ( d | c ) (cid:21) + E ( c,s ) ∼ q θ ( C,S | x ) (cid:20) log q ϕ s ( d | s ) q ϕ c ( y | s ) (cid:21) + E ( c,s ) ∼ q θ ( C,S | x ) [log p ψ ( x | c, s )]
Revised: E ( c,s ) ∼ q θ ( C,S | x ) (cid:20) log p ( c ) p ( s ) q θ c ( c | x ) q θ s ( s | x ) (cid:21) + E ( c,s ) ∼ q θ ( C,S | x ) (cid:20) log q ϕ c ( y | c ) q ϕ s ( d | c ) (cid:21) + E ( c,s ) ∼ q θ ( C,S | x ) (cid:20) log q ϕ s ( d | s ) q ϕ c ( y | s ) (cid:21) + E ( c,s ) ∼ q θ ( C,S | x ) [log p ψ ( x | c, s )]

Original: =
Revised: =

Original: E ( c ) ∼ q θc ( C | x ) (cid:20) log p ( c ) q [Equation] c ( c 
Revised: E ( c ) ∼ q θc ( C | x ) (cid:20) log p ( c ) q θ c ( c | x ) (cid:21) + E ( s ) ∼ q θs

Original: [Equation] x ) (cid:21) [Equation] E ( s ) ∼ q 
Revised: ( S | x ) (cid:20) log p ( c ) p ( s ) q θ s ( s | x ) (cid:21) + E ( c ) ∼ q θs

Original: [Equation] ( S | x ) (cid:20) log [Equation] ( s ) q θ s (  s  | x )  (cid:21) + E ( c ) ∼ q θs ( 
Revised: ( C | x ) (cid:20) log q ϕ c ( y | c ) q ϕ s ( d | c ) (cid:21) + E ( s ) ∼ q θs ( S | x ) (cid:20) log q ϕ s ( d | s ) q ϕ c ( y | s ) (cid:21) + E ( c,s ) ∼ q θ ( C,S | x ) [log p ψ ( x | c

Original: [Equation]
Revised: [Equation]

Original: which is the final ELBO in our main paper. 
Revised: which is the final ELBO in our main paper. D S X

Original: C. V ISUALIZATION OF H IGHER R ESOLUTION I MAGES
Revised: Y C [Figure] S

Revised: X

Revised: Y

Revised: [Figure]

Revised: E D ISCUSSION OF THE C AUSAL R ELATIONSHIP BETWEEN S TYLE AND D OMAIN

Revised: There are two different assumptions for the causal relationship between style S and domain D : a)

Revised: Style is caused by domain; and b) Style causes domain (as shown in Figure 6).

Revised: The first one is more commonly used in existing literature Liu et al.

Revised: ; Lu et al.

Revised: ; Suter et al.

Revised: (2019), which usually assume that the style of data depends on the sampling environment.

Revised: Such an assumption perfectly fits the natural data generating process.

Revised: However, we follow Mitrovic et al.

Revised: Von Kügelgen et al.

Revised: (2021) by using data augmentation to obtain different styles.

Revised: During this process, each data augmentation only introduces a specific style, then we manually assign a domain label based on the style type.

Revised: Hence, in this scenario, assuming that the style causes the domain label as scenario b) is more appropriate than the first one.

Revised: Furthermore, the causal direction between style and domain depends on the problem context.

Revised: If the domain is partitioned according to the style in the images, content is the cause of domain.

Revised: For example, in image recognition, one can partition images into domains according to the illumination levels.

Revised: The domain is determined by the brightness of the images and the causal relation is brightness → domain.

Revised: On the other hand, if one partition the domains according to time (daytime or night), the brightness level will be an effect of the domain (or time), i.e ., domain → brightness.

Revised: Therefore, our assumption b) holds in certain real-world scenarios.

Revised: F A DDITIONAL A NALYSIS

Revised: [Figure]

Revised: [Figure]

Revised: V ISUALIZATION OF H IGHER R ESOLUTION I MAGES

Original: In the main paper, we have provided the visualization of augmented CIFAR10 images.
Revised: In the main paper, we have provided the visualization of augmented CIFAR10 images.

Original: To testify that our positive data augmentation and negative data augmentation are still effective on higher resolution images, we conduct experiments using ImageNet30 with resolution 256 × 256 , and show the augmented benign data and malign data in Fig.
Revised: To testify that our positive data augmentation and negative data augmentation are still effective on higher resolution images, we conduct experiments using ImageNet30 with resolution 256 × 256 , and show the augmented benign data and malign data in Fig.

Original: We can see the similar phenomenon as in
Revised: We can see the similar phenomenon as in

Original: CIFAR10:
Revised: CIFAR10:

Original: In negative data augmentation, the objects are completely unidentifiable which lead to erroneous model predictions.
Revised: In negative data augmentation, the objects are completely unidentifiable which lead to erroneous model predictions.

Original: On the contrary, the positive data augmentation only changes the style (there are three style types are presented: purple style, green style, and sharp-texture style) but leave the objects intact.
Revised: On the contrary, the positive data augmentation only changes the style (there are three style types are presented: purple style, green style, and sharp-texture style) but leave the objects intact.

Original: As a result, the model predictions are usually correct.
Revised: As a result, the model predictions are usually correct.

Original: Therefore, our augmentation method can be effectively deployed to high resolution images.[Figure]
Revised: Therefore, our augmentation method can be effectively deployed to high resolution images.

Original: C. T -SNE V ISUALIZATION
Revised: F. T -SNE V ISUALIZATION

Original: To further demonstrate the effectiveness of our disentanglement, we show the t-SNE (Maaten &
Revised: To further demonstrate the effectiveness of our disentanglement, we show the t-SNE (Maaten &

Original: Hinton, 2008) visualization of the feature extracted with or without disentanglement, as shown in
Revised: Hinton, 2008) visualization of the feature extracted with or without disentanglement, as shown in

Original: Fig. 
Revised: Fig. 8.

Original: We can see that when training without disentanglement, the features are closely gathered in each cluster.
Revised: We can see that when training without disentanglement, the features are closely gathered in each cluster.

Original: However, the malign OOD data represented by gray and pink points are also intensely aligned in the clusters, which would damage the model robustness.
Revised: However, the malign OOD data represented by gray and pink points are also intensely aligned in the clusters, which would damage the model robustness.

Original: In contrast, when training with disentanglement, there is a slight gap between OOD data and ID data, which means that the model trained with disentanglement can avoid overfitting to specific styles, and show better robustness against OOD data.
Revised: In contrast, when training with disentanglement, there is a slight gap between OOD data and ID data, which means that the model trained with disentanglement can avoid overfitting to specific styles, and show better robustness against OOD data.

Original: [Table]
Revised: [Figure]

Original: D A DDITIONAL A NALYSIS D. OOD S CORE
Revised: F. E FFECT OF A DDING B ENIGN AND M ALIGN OOD D ATA INTO T RAINING

Revised: To give a illustrative comparison of adding benign OOD data and malign OOD data into training, we conduct experiments under the open-set SSL setting and separately augmenting benign OOD data and malign OOD data to compare their effects. Moreover, we conduct plain training as a baseline result which do not use either augmentations. The results are shown in Fig.

Revised: We can see that after adding augmented data, the effect of malign OOD data causes sudden performance degradation.

Revised: On the contrary, benign OOD data can further improve the learning result compared to the plain training baseline.

Revised: Which again shows that preserving content and augmenting style is beneficial for improving generalization, and eliminating content is harmful for learning.

Revised: [Figure]

Original: To show the effectiveness of identifying malign OOD data from benign OOD data, we test the performance of HOOD on three applications to observe the OOD scores of benign OOD data and malign OOD data and show the averaged OOD scores in Table 5.
Revised: To show the effectiveness of identifying malign OOD data from benign OOD data [Table] we test the performance of HOOD on three applications to observe the OOD scores of benign OOD data and malign OOD data and show the averaged OOD scores in Table 5.

Original: We can see that the OOD score produced by our one-vs-all classifier can clearly distinguish benign and malign OOD data during the test phase, which again validates the effectiveness of HOOD. 
Revised: We can see that the OOD score produced by our one-vs-all classifier can clearly distinguish benign and malign OOD data during the test phase, which again validates the effectiveness of HOOD. [Table] F. E XECUTION E FFICIENCY Additionally, to give a quantitative comparison on the execution efficiency of HOOD

Revised: [Table]

Revised: here we provide the running time on 3090 GPU compared to some typical baseline methods.

Original: E L IMITATION AND S OCIAL I MPACT
Revised: The results are shown in Table 6. Note that our method involves causal disentanglement as well as adversarial training, therefore, the training time is more than other opponents. [Table] G L IMITATION AND S OCIAL I MPACT

Original: The proposed HOOD method has many advantages which have been demonstrated in the main paper.
Revised: The proposed HOOD method has many advantages which have been demonstrated in the main paper.

Original: However, HOOD is still limited in some aspects.
Revised: However, HOOD is still limited in some aspects.

Original: Firstly, HOOD contains an extra phase that computes the gradient to produce the augmented data, so it cannot be conducted in an endto-end manner.
Revised: Firstly, HOOD contains an extra phase that computes the gradient to produce the augmented data, so it cannot be conducted in an endto-end manner.

Original: Therefore, it is feasible to improve HOOD by designing a more compact method and incorporating augmentation into the training process.
Revised: Therefore, it is feasible to improve HOOD by designing a more compact method and incorporating augmentation into the training process.

Original: Secondly, HOOD utilizes existing data augmentation techniques to simulate different styles, which cannot perfectly cover all the possible styles in the real world.
Revised: Secondly, HOOD utilizes existing data augmentation techniques to simulate different styles, which cannot perfectly cover all the possible styles in the real world.

Original: Hence, better style simulation might further improve the learning performance of HOOD.
Revised: Hence, better style simulation might further improve the learning performance of HOOD.

Original: Regardless of the limitations, our method could have some positive social impacts.
Revised: Regardless of the limitations, our method could have some positive social impacts.

Original: First, HOOD can be safely deployed into many open situations where many unknown classes exist.
Revised: First, HOOD can be safely deployed into many open situations where many unknown classes exist.

Original: As practical problems contain lots of uncertainties and novel instances occur constantly.
Revised: As practical problems contain lots of uncertainties and novel instances occur constantly.

Original: Thanks to the negative data augmentation of HOOD, the novel instances can be successfully identified and would not harm the prediction accuracy.
Revised: Thanks to the negative data augmentation of HOOD, the novel instances can be successfully identified and would not harm the prediction accuracy.

Original: Secondly, in many non-stationary environments, the knowledge of the backbone model can be easily transferred thanks to the positive data augmentation of HOOD, which further broadens the practical usage of HOOD in modern industry.
Revised: Secondly, in many non-stationary environments, the knowledge of the backbone model can be easily transferred thanks to the positive data augmentation of HOOD, which further broadens the practical usage of HOOD in modern industry.