Original: LASSO: L ATENT S UB - SPACES O RIENTATION FOR D OMAIN G ENERALIZATION
Revised: LASSO: L ATENT S UB - SPACES O RIENTATION FOR D OMAIN G ENERALIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: To achieve a satisfactory generalization performance on prediction tasks in an unseen domain, existing domain generalization (DG) approaches often rely on the assumption of the existence of fixed domain-invariant features and common hypotheses learned from a set of training domains.
Revised: To achieve a satisfactory generalization performance on prediction tasks in an unseen domain, existing domain generalization (DG) approaches often rely on the assumption of the existence of fixed domain-invariant features and common hypotheses learned from a set of training domains.

Original: While it is a natural and important premise to ground generalization capacity on the target domain, we argue that this assumption could be overly strict and sub-optimal.
Revised: While it is a natural and important premise to ground generalization capacity on the target domain, we argue that this assumption could be overly strict and sub-optimal.

Original: It is particularly evident when source domains share little information or the target domains leverages information from selective source domains in a compositional way instead of relying on a unique invariant hypothesis across all source domains.
Revised: It is particularly evident when source domains share little information or the target domains leverages information from selective source domains in a compositional way instead of relying on a unique invariant hypothesis across all source domains.

Original: Unlike most existing approaches, instead of constructing a single hypothesis shared among domains, we propose a LAtent Sub-Space Orientation (LASSO) method that explores diverse latent sub-spaces and learning individual hypotheses on those sub-spaces.
Revised: Unlike most existing approaches, instead of constructing a single hypothesis shared among domains, we propose a LAtent Sub-Space Orientation (LASSO) method that explores diverse latent sub-spaces and learning individual hypotheses on those sub-spaces.

Original: Moreover, in LASSO, since the latent sub-spaces are formed by the label-informative features captured in source domains, they allow us to project target examples onto appropriate sub-spaces while preserving crucial label-informative features for the label prediction task.
Revised: Moreover, in LASSO, since the latent sub-spaces are formed by the label-informative features captured in source domains, they allow us to project target examples onto appropriate sub-spaces while preserving crucial label-informative features for the label prediction task.

Original: Finally, we empirically evaluate our method on several wellknown DG benchmarks, where it achieves state-of-the-art results.
Revised: Finally, we empirically evaluate our method on several wellknown DG benchmarks, where it achieves state-of-the-art results.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: One of the most challenging problems in applying machine learning to real-world problems is to address the domain shift encountered when test data at inference time come from different distributions compared to training data, often causing unexpectedly imperfect generalization performance.
Revised: One of the most challenging problems in applying machine learning to real-world problems is to address the domain shift encountered when test data at inference time come from different distributions compared to training data, often causing unexpectedly imperfect generalization performance.

Original: To handle this issue, many out-of-distribution learning settings have been investigated, notably domain adaptation (DA) and domain generalization (DG).
Revised: To handle this issue, many out-of-distribution learning settings have been investigated, notably domain adaptation (DA) and domain generalization (DG).

Original: In particular, DA setting (Mansour et al., 2009; Ben-David et al., 2010; Zhao et al., 2019) takes the assumption that both labeled source data and unlabeled target data are available at the training phase, while DG setting (Blanchard et al., 2011; Muandet et al., 2013; Ganin et al., 2016) is much more challenging due to the complete absence of any target data at training time, and the learned model is expected to perform a zero-shot prediction on test samples.
Revised: In particular, DA setting (Mansour et al., 2009; Ben-David et al., 2010; Zhao et al., 2019) takes the assumption that both labeled source data and unlabeled target data are available at the training phase, while DG setting (Blanchard et al., 2011; Muandet et al., 2013; Ganin et al., 2016) is much more challenging due to the complete absence of any target data at training time, and the learned model is expected to perform a zero-shot prediction on test samples.

Original: While being more challenging than DA, DG is arguably more versatile and applicable to real-world scenarios where there is a need to rapidly deploy a prediction model on a new target domain without any access to target data.
Revised: While being more challenging than DA, DG is arguably more versatile and applicable to real-world scenarios where there is a need to rapidly deploy a prediction model on a new target domain without any access to target data.

Original: A common premise on which most of the existing DG approaches rely to address the domain shift problem is to learn a shared set of features among different source domains, e.g., domain-invariant features (Muandet et al., 2013; Ganin et al., 2016; Motiian et al., 2017; Ghifary et al., 2015; Xie et al., 2017; Wang et al., 2019; Piratla et al., 2020; Zhao et al., 2020).
Revised: A common premise on which most of the existing DG approaches rely to address the domain shift problem is to learn a shared set of features among different source domains, e.g., domain-invariant features (Muandet et al., 2013; Ganin et al., 2016; Motiian et al., 2017; Ghifary et al., 2015; Xie et al., 2017; Wang et al., 2019; Piratla et al., 2020; Zhao et al., 2020).

Original: By learning domain-invariant features, these approaches identify a latent representation from multiple source domains which is also expected encompass unseen target domains, hence reducing the domain shift.
Revised: By learning domain-invariant features, these approaches identify a latent representation from multiple source domains which is also expected encompass unseen target domains, hence reducing the domain shift.

Original: However, since this latent space is often high-dimensional and unseen target domains could vary greatly and unexpectedly, these domain invariance approaches could become ineffective.
Revised: However, since this latent space is often high-dimensional and unseen target domains could vary greatly and unexpectedly, these domain invariance approaches could become ineffective.

Original: We argue that using a single hypothesis which requires to be optimal for all source and target domains is rather a strong assumption and might not have an effective algorithmic solution, hence presenting a key weakness to these existing approaches.
Revised: We argue that using a single hypothesis which requires to be optimal for all source and target domains is rather a strong assumption and might not have an effective algorithmic solution, hence presenting a key weakness to these existing approaches.

Original: In this paper, we propose a new approach to efficiently reduce the domain shift.
Revised: In this paper, we propose a new approach to efficiently reduce the domain shift.

Original: Our idea is to optimally decompose the high-dimensional latent feature space to lower-dimensional sub-spaces, after which an individual hypothesis is learned (optimally) for each latent sub-space.
Revised: Our idea is to optimally decompose the high-dimensional latent feature space to lower-dimensional sub-spaces, after which an individual hypothesis is learned (optimally) for each latent sub-space.

Original: In particular, these latent sub-spaces are formed by label-informative features which are learned from source latent representations so that sub-space latent representations can capture sufficient and necessary information to effectively train sub-space hypotheses.
Revised: In particular, these latent sub-spaces are formed by label-informative features which are learned from source latent representations so that sub-space latent representations can capture sufficient and necessary information to effectively train sub-space hypotheses.

Original: We then perform extensive experimental evaluations to demonstrate the merits of our methodology and framework.
Revised: We then perform extensive experimental evaluations to demonstrate the merits of our methodology and framework.

Original: In sum, our contributions are as follows:
Revised: In sum, our contributions are as follows:

Original: • We propose LAtent SubSpace Orientation (LASSO), a novel approach for DG, which aims to learn label-informative latent sub-spaces from source latent representations, preserving crucial label information for training qualified individual sub-space hypotheses.
Revised: • We propose LAtent SubSpace Orientation (LASSO), a novel approach for DG, which aims to learn label-informative latent sub-spaces from source latent representations, preserving crucial label information for training qualified individual sub-space hypotheses.

Original: • We develop a rigorous theoretical analysis to explain how our approach can learn meaningful sub-spaces by learning an indicator function Γ that helps to reduce latent domain shift.
Revised: • We develop a rigorous theoretical analysis to explain how our approach can learn meaningful sub-spaces by learning an indicator function Γ that helps to reduce latent domain shift.

Original: Moreover, by linking to information theory (cf.
Revised: Moreover, by linking to information theory (cf.

Original: Theorem 3), we theoretically justify the rationale behind learning the sub-space indicator function Γ as a mechanism for element-wise selection of latent features to gather sufficient label information.
Revised: Theorem 3), we theoretically justify the rationale behind learning the sub-space indicator function Γ as a mechanism for element-wise selection of latent features to gather sufficient label information.

Original: Finally, we empirically demonstrate that our proposed method can achieve favorable results when evaluating our model on domain generalization benchmarks in the comparison with state-of-the-art methods.
Revised: Finally, we empirically demonstrate that our proposed method can achieve favorable results when evaluating our model on domain generalization benchmarks in the comparison with state-of-the-art methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Domain generalization (DG) approaches can be categorized into several groups: domain-invariant representation learning , meta-learning , and augmentation/self-supervision .
Revised: Domain generalization (DG) approaches can be categorized into several groups: domain-invariant representation learning , meta-learning , and augmentation/self-supervision .

Original: The works in the first group aim to learn a domain-invariant representation with the hope of transferring well to unseen domains.
Revised: The works in the first group aim to learn a domain-invariant representation with the hope of transferring well to unseen domains.

Original: Notably, Muandet et al.
Revised: Notably, Muandet et al.

Original: (2013) construct shared components by minimizing the discrepancy of the source domain marginal distributions using a kernel-based algorithm.
Revised: (2013) construct shared components by minimizing the discrepancy of the source domain marginal distributions using a kernel-based algorithm.

Original: Xie et al.
Revised: Xie et al.

Original: (2017) employ an adversarial training strategy to preserve the desired invariance and eliminate variations of domain factors from features.
Revised: (2017) employ an adversarial training strategy to preserve the desired invariance and eliminate variations of domain factors from features.

Original: Seo et al.
Revised: Seo et al.

Original: (2020) combine batch normalization and instance normalization to remove domain-specific styles while preserving semantic category information.
Revised: (2020) combine batch normalization and instance normalization to remove domain-specific styles while preserving semantic category information.

Original: Other works (Ghifary et al., 2015; Li et al., 2018b; Ilse et al., 2020) employ auto-encoders to support domain-invariant feature extraction.
Revised: Other works (Ghifary et al., 2015; Li et al., 2018b; Ilse et al., 2020) employ auto-encoders to support domain-invariant feature extraction.

Original: Another efficient approach for DG is meta-learning.
Revised: Another efficient approach for DG is meta-learning.

Original: For example, the studies in (Li et al., 2018a;
Revised: For example, the studies in (Li et al., 2018a;

Original: Balaji et al., 2018) use source domains to simulate meta-train/meta-test, which encourages the trained model to generalize better on the meta-test data, leading to a more plausible global generalization performance on the unseen target domain.
Revised: Balaji et al., 2018) use source domains to simulate meta-train/meta-test, which encourages the trained model to generalize better on the meta-test data, leading to a more plausible global generalization performance on the unseen target domain.

Original: Dou et al.
Revised: Dou et al.

Original: (2019) then extend the work of Li et al.
Revised: (2019) then extend the work of Li et al.

Original: (2018a)
Revised: (2018a)

Original: by combining it with metric learning loss to encourage domain-independent semantic feature space.
Revised: by combining it with metric learning loss to encourage domain-independent semantic feature space.

Original: Self-supervised learning and data augmentation have been also applied to DG.
Revised: Self-supervised learning and data augmentation have been also applied to DG.

Original: Typically, Carlucci et al.
Revised: Typically, Carlucci et al.

Original: (2019) propose to solve the pretext task of Jigsaw Puzzles to improve the generalization performance on unseen domains.
Revised: (2019) propose to solve the pretext task of Jigsaw Puzzles to improve the generalization performance on unseen domains.

Original: Shankar et al.
Revised: Shankar et al.

Original: (2018) augment training data with instances perturbed along with directions of domain change.
Revised: (2018) augment training data with instances perturbed along with directions of domain change.

Original: In addition, Zhou et al.
Revised: In addition, Zhou et al.

Original: (2020a) employ a classifier that can learn the generalization on additional augmented samples of diversity pseudo-novel domains by leveraging optimal transport theory.
Revised: (2020a) employ a classifier that can learn the generalization on additional augmented samples of diversity pseudo-novel domains by leveraging optimal transport theory.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2020b) augment the original training data of source domains with synthetic data from unseen domains such that augmented data are correctly classified by the classifier while fooling the domain classifier to make the task model intrinsically more domain-generalizable.
Revised: (2020b) augment the original training data of source domains with synthetic data from unseen domains such that augmented data are correctly classified by the classifier while fooling the domain classifier to make the task model intrinsically more domain-generalizable.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2021) propose to mix the styles of different source domains based on normalization-based style-transfer technique to effectively increase the diversity of domains during training.
Revised: (2021) propose to mix the styles of different source domains based on normalization-based style-transfer technique to effectively increase the diversity of domains during training.

Original: Recently, learning domain-specific information to boost classification performance in domain generalization has attracted more attention.
Revised: Recently, learning domain-specific information to boost classification performance in domain generalization has attracted more attention.

Original: For example, Huang et al.
Revised: For example, Huang et al.

Original: (2020) iteratively discard the dominant features to exploit all useful features (including both invariant and specific features) that highly correlate with labels.
Revised: (2020) iteratively discard the dominant features to exploit all useful features (including both invariant and specific features) that highly correlate with labels.

Original: Chattopadhyay et al.
Revised: Chattopadhyay et al.

Original: (2020) propose a domain-specific mask learned from the domain label to balance domain-invariant and domain-specific features.
Revised: (2020) propose a domain-specific mask learned from the domain label to balance domain-invariant and domain-specific features.

Original: As a result, source domain classification can benefit from the specialized features while retaining the generalizing applicability of domain-invariant features.
Revised: As a result, source domain classification can benefit from the specialized features while retaining the generalizing applicability of domain-invariant features.

Original:  P ROPOSED METHOD 3. P ROBLEM SETTING
Revised:  P ROPOSED METHOD 3. P ROBLEM SETTING

Original: Let D Sk
Revised: Let D Sk

Original: = { ( x ki , y ki ) } N Sk i =1 , k = 1 , ..., K be the source labeled datasets, where k is the domain index,
Revised: = { ( x ki , y ki ) } N Sk i =1 , k = 1 , ..., K be the source labeled datasets, where k is the domain index,

Original: N Sk is the number of examples in D Sk and
Revised: N Sk is the number of examples in D Sk and

Original: y ki ∈ Y :
Revised: y ki ∈ Y :

Original: = { 1 , ..., C } is the set of C classes.
Revised: = { 1 , ..., C } is the set of C classes.

Original: Let P Sk and p
Revised: Let P Sk and p

Original: Sk be the (marginal) data distribution and corresponding density of the source domain k .
Revised: Sk be the (marginal) data distribution and corresponding density of the source domain k .

Original: Similarly, let p Sk ( y | x ) denote the probabilistic labeling distribution of this domain.
Revised: Similarly, let p Sk ( y | x ) denote the probabilistic labeling distribution of this domain.

Original: A domain is then defined by a pair of data distribution and labeling function, in which its data ( x ki , y ki ) is generated by first sampling x ki ∼ p
Revised: A domain is then defined by a pair of data distribution and labeling function, in which its data ( x ki , y ki ) is generated by first sampling x ki ∼ p

Original: Sk ( · ) and subsequently y ki ∼ p
Revised: Sk ( · ) and subsequently y ki ∼ p

Original: Sk (cid:0) · | x ki (cid:1) .
Revised: Sk (cid:0) · | x ki (cid:1) .

Original: With a little abuse of notation, let us also denote k -th domain by D Sk , and the data generation process is ( x, y ) ∼ D Sk .
Revised: With a little abuse of notation, let us also denote k -th domain by D Sk , and the data generation process is ( x, y ) ∼ D Sk .

Original: For target domain D T ,
Revised: For target domain D T ,

Original: P T and p T are its data distribution and density respectively, while p T ( y | x ) is the target labeling distribution.
Revised: P T and p T are its data distribution and density respectively, while p T ( y | x ) is the target labeling distribution.

Original: A mixture of the source domains is denoted by D Sπ := (cid:80)
Revised: A mixture of the source domains is denoted by D Sπ := (cid:80)

Original: Kk =1 π k D Sk .
Revised: Kk =1 π k D Sk .

Original: In order to sample data from this mixture, one first sample a domain index from the categorical distribution k ∼ Cat ( π ) , where π specifies the mixture weights, and then sample the actual data from the corresponding domain k .
Revised: In order to sample data from this mixture, one first sample a domain index from the categorical distribution k ∼ Cat ( π ) , where π specifies the mixture weights, and then sample the actual data from the corresponding domain k .

Original: The empirical datasets obtained could hint us to estimate π k = (cid:104) N Sk N S (cid:105) K k
Revised: The empirical datasets obtained could hint us to estimate π k = (cid:104) N Sk N S (cid:105) K k

Original: =1 with N S =
Revised: =1 with N S =

Original: (cid:80)
Revised: (cid:80)

Original: K k
Revised: K k

Original: =1 N Sk .
Revised: =1 N Sk .

Original: Finally, a mixture of data distribution is defined by P Sπ =
Revised: Finally, a mixture of data distribution is defined by P Sπ =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 π k P Sk .
Revised: Kk =1 π k P Sk .

Original: A hypothesis f : X → ∆ C is a map from the common data space X to the the C -simplex label space ∆ C :
Revised: A hypothesis f : X → ∆ C is a map from the common data space X to the the C -simplex label space ∆ C :

Original: = (cid:8) α ∈ R C : ∥
Revised: = (cid:8) α ∈ R C : ∥

Original: α ∥ 1 = 1 ∧ α ≥ 0 (cid:9) .
Revised: α ∥ 1 = 1 ∧ α ≥ 0 (cid:9) .

Original: Let ℓ ( f ( x ) , y ) be the loss incurred by using this hypothesis to predict x ∈ X , given its ground-truth label y ∈ Y .
Revised: Let ℓ ( f ( x ) , y ) be the loss incurred by using this hypothesis to predict x ∈ X , given its ground-truth label y ∈ Y .

Original: Therefore, the general loss of the hypothesis f w.r.t.
Revised: Therefore, the general loss of the hypothesis f w.r.t.

Original: the joint distribution D is:
Revised: the joint distribution D is:

Original: [Equation]
Revised: [Equation]

Original: We examine the composite hypothesis f = h ◦ g where g : X → Z is the feature extractor mapping the data space to a latent space and h :
Revised: We examine the composite hypothesis f = h ◦ g where g : X → Z is the feature extractor mapping the data space to a latent space and h :

Original: Z → Y is the classifier on this latent space.
Revised: Z → Y is the classifier on this latent space.

Original: With respect to the latent space, given a probabilistic labeling distribution p ( y | x ) , let denote p g
Revised: With respect to the latent space, given a probabilistic labeling distribution p ( y | x ) , let denote p g

Original: ( y | z ) as the probabilistic labeling distribution induced by p ( y | x ) and g , that is, for any z ∈ g ( X ) , p g ( y | z ) = (cid:82)
Revised: ( y | z ) as the probabilistic labeling distribution induced by p ( y | x ) and g , that is, for any z ∈ g ( X ) , p g ( y | z ) = (cid:82)

Original: [Equation]
Revised: [Equation]

Original: p ( y | x
Revised: p ( y | x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: (Johansson et al., 2019).
Revised: (Johansson et al., 2019).

Original: Basically, p g ( y | z ) can be regarded as a weighted sum of p ( y | x ) for x ∈ g − 1 ( z ) .
Revised: Basically, p g ( y | z ) can be regarded as a weighted sum of p ( y | x ) for x ∈ g − 1 ( z ) .

Original: We further define p S,gk ( y = c | z ) and p T,g ( y = c | z ) with z ∈ g ( X ) ⊂
Revised: We further define p S,gk ( y = c | z ) and p T,g ( y = c | z ) with z ∈ g ( X ) ⊂

Original: Z as the induced conditional labeling distributions of source and target domains on the latent space via the feature extractor g .
Revised: Z as the induced conditional labeling distributions of source and target domains on the latent space via the feature extractor g .

Original: Finally, we consider the DG setting, in which we only possess the source labeled data and do not have any prior information of any target data during the training process.
Revised: Finally, we consider the DG setting, in which we only possess the source labeled data and do not have any prior information of any target data during the training process.

Original: In what follows, we develop a rigorous theory to promote our LASSO, whose proofs can be found in the Appendix A.
Revised: In what follows, we develop a rigorous theory to promote our LASSO, whose proofs can be found in the Appendix A.

Original: 3. S INGLE HYPOTHESIS AND FULL SPACE FOR DOMAIN GENERALIZATION
Revised: 3. S INGLE HYPOTHESIS AND FULL SPACE FOR DOMAIN GENERALIZATION

Original: Most existing works in DG use a single hypothesis acting on the entire latent space Z for predicting unseen target domains.
Revised: Most existing works in DG use a single hypothesis acting on the entire latent space Z for predicting unseen target domains.

Original: In what follows, we develop an upper-bound which is relevant to the data shift on the latent space of the target loss L (cid:0) f, D T (cid:1) with f = h ◦ g .
Revised: In what follows, we develop an upper-bound which is relevant to the data shift on the latent space of the target loss L (cid:0) f, D T (cid:1) with f = h ◦ g .

Original: Theorem 1.
Revised: Theorem 1.

Original: If the loss function ℓ is upper-bounded by a positive constant L , for any hypothesis f : X → ∆ C where f = h ◦ g with g : X → Z and h : Z → ∆ C , the target general loss is upper-bounded by:
Revised: If the loss function ℓ is upper-bounded by a positive constant L , for any hypothesis f : X → ∆ C where f = h ◦ g with g : X → Z and h : Z → ∆ C , the target general loss is upper-bounded by:

Original: [Equation]
Revised: [Equation]

Original: where g # P Sπ and g # P T are pushed-forward distributions induced by applying g on P Sπ and P T , R α is the α -divergence ( α > 1 ), and ∆ p k ( y | x ) := (cid:2)(cid:12)(cid:12) p
Revised: where g # P Sπ and g # P T are pushed-forward distributions induced by applying g on P Sπ and P T , R α is the α -divergence ( α > 1 ), and ∆ p k ( y | x ) := (cid:2)(cid:12)(cid:12) p

Original: [Equation]
Revised: [Equation]

Original: y =
Revised: y =

Original: [Equation]
Revised: [Equation]

Original: | x )
Revised: | x )

Original: [Equation]
Revised: [Equation]

Original: p T
Revised: p T

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: = c
Revised: = c

Original: | x )
Revised: | x )

Original: [Equation]
Revised: [Equation]

Original: C c =1 represents the label shift between the labeling assignment mechanisms of an individual source domain and target domain on the input space.
Revised: C c =1 represents the label shift between the labeling assignment mechanisms of an individual source domain and target domain on the input space.

Original: The bound developed in Theorem 1 supports us in characterizing the factors influencing the loss on the target domain of a hypothesis f : (i) the label shift : E P Sk [ ∥ ∆ p k
Revised: The bound developed in Theorem 1 supports us in characterizing the factors influencing the loss on the target domain of a hypothesis f : (i) the label shift : E P Sk [ ∥ ∆ p k

Original: ( y | x ) ∥ 1 ] , (ii) the source loss :
Revised: ( y | x ) ∥ 1 ] , (ii) the source loss :

Original: L (cid:0) f, D Sπ (cid:1) , and (iii) the latent data shift : R α (cid:0) g
Revised: L (cid:0) f, D Sπ (cid:1) , and (iii) the latent data shift : R α (cid:0) g

Original: # P T , g # P Sπ (cid:1) .
Revised: # P T , g # P Sπ (cid:1) .

Original: Based on this theorem, we aim to find the hypothesis f such that the upper bound of target loss is minimized.
Revised: Based on this theorem, we aim to find the hypothesis f such that the upper bound of target loss is minimized.

Original: Note that the label shift term is a natural characteristic of domains, hence it is almost unchangeable.
Revised: Note that the label shift term is a natural characteristic of domains, hence it is almost unchangeable.

Original: Therefore, one seeks to minimize the source loss, or the discrepancy between latent distributions, or both of them.
Revised: Therefore, one seeks to minimize the source loss, or the discrepancy between latent distributions, or both of them.

Original: Certainly, if we know target data (e.g., samples from P T ), we can minimize the divergence between g # P Sπ and g # P T directly.
Revised: Certainly, if we know target data (e.g., samples from P T ), we can minimize the divergence between g # P Sπ and g # P T directly.

Original: However, this is almost impossible for the DG setting because the target domain is unknown beforehand when training.
Revised: However, this is almost impossible for the DG setting because the target domain is unknown beforehand when training.

Original: To address this problem, a large number of works propose learning domain-invariant features on a full high-dimensional latent space together with a single hypothesis on top of these domaininvariant features.
Revised: To address this problem, a large number of works propose learning domain-invariant features on a full high-dimensional latent space together with a single hypothesis on top of these domaininvariant features.

Original: Nonetheless, due to the great variance of unseen target distributions on the full high-dimensional latent space, the latent data shift is possibly high in many cases, which hurts the generalization ability of the single hypothesis on unseen target domains.
Revised: Nonetheless, due to the great variance of unseen target distributions on the full high-dimensional latent space, the latent data shift is possibly high in many cases, which hurts the generalization ability of the single hypothesis on unseen target domains.

Original: Evidently, given a latent representation z with a label y in the high-dimensional latent space, only a small portion of its features known as label-informative features is highly relevant to the label y , while the remaining ones are redundant.
Revised: Evidently, given a latent representation z with a label y in the high-dimensional latent space, only a small portion of its features known as label-informative features is highly relevant to the label y , while the remaining ones are redundant.

Original: By eliminating irrelevant features and grouping the latent representations of data examples across multiple domains with the same set of label-informative features, we can form latent label-informative sub-spaces to reduce the latent data shift, whereas preserving sufficient label-information for training good hypotheses on those latent sub-spaces .
Revised: By eliminating irrelevant features and grouping the latent representations of data examples across multiple domains with the same set of label-informative features, we can form latent label-informative sub-spaces to reduce the latent data shift, whereas preserving sufficient label-information for training good hypotheses on those latent sub-spaces .

Original: This can be explained from the fact that unseen target and source examples with the same labelinformative features are projected onto the same latent sub-space on which the latent data shift between mixture of source domains and target domain becomes smaller due to the compactness of this sub-space compared to the full high-dimensional latent space.
Revised: This can be explained from the fact that unseen target and source examples with the same labelinformative features are projected onto the same latent sub-space on which the latent data shift between mixture of source domains and target domain becomes smaller due to the compactness of this sub-space compared to the full high-dimensional latent space.

Original: Furthermore, by learning multiple hypotheses, each of which corresponds to a sub-space, LASSO allows sub-spaces to instance-wisely explore different groups of label-informative features, hence encouraging the diversity of latent representations for achieving better generalization ability, which concurs with the principle in (Huang et al., 2020; Chattopadhyay et al., 2020; Blanchard et al., 2021).
Revised: Furthermore, by learning multiple hypotheses, each of which corresponds to a sub-space, LASSO allows sub-spaces to instance-wisely explore different groups of label-informative features, hence encouraging the diversity of latent representations for achieving better generalization ability, which concurs with the principle in (Huang et al., 2020; Chattopadhyay et al., 2020; Blanchard et al., 2021).

Original: This intuitively boosts the generalization ability of hypotheses.
Revised: This intuitively boosts the generalization ability of hypotheses.

Original: The reason is that given source examples with their labels, LASSO aims to explore possible compact groups of label-informative features which can predict accurately the labels.
Revised: The reason is that given source examples with their labels, LASSO aims to explore possible compact groups of label-informative features which can predict accurately the labels.

Original: In the inference time, given a target example, if the feature extractor can successfully activate a group of label-informative features, this target example is projected and matched with corresponding source examples on a sub-space in which a good hypothesis is used to predict a label for the target example.
Revised: In the inference time, given a target example, if the feature extractor can successfully activate a group of label-informative features, this target example is projected and matched with corresponding source examples on a sub-space in which a good hypothesis is used to predict a label for the target example.

Original: 3. L ATENT SUB - SPACES WITH INDIVIDUAL HYPOTHESES FOR DOMAIN GENERALIZATION
Revised: 3. L ATENT SUB - SPACES WITH INDIVIDUAL HYPOTHESES FOR DOMAIN GENERALIZATION

Original: For a data sample ( x, y ) ∼ D S π , we consider latent representation z
Revised: For a data sample ( x, y ) ∼ D S π , we consider latent representation z

Original: = g ( x ) ∈ R D and propose to learn a sub-space indicator Γ which renders Γ( z ) ∈ { 0 , 1 } D .
Revised: = g ( x ) ∈ R D and propose to learn a sub-space indicator Γ which renders Γ( z ) ∈ { 0 , 1 } D .

Original: Specifically, Γ( z ) specifies the sub-spac e which we project z onto via Π Γ ( z ) =
Revised: Specifically, Γ( z ) specifies the sub-spac e which we project z onto via Π Γ ( z ) =

Original: z ⊙ Γ( z ) , where ⊙ is the element-wise product operator.
Revised: z ⊙ Γ( z ) , where ⊙ is the element-wise product operator.

Original: Our intuition is that Γ is able to keep the most label informative features of latent vector z (i.e., the d th element of Γ( z ) : Γ d ( z ) = 1 means that z d is label-informative and should be selected when projecting z onto its sub-space).
Revised: Our intuition is that Γ is able to keep the most label informative features of latent vector z (i.e., the d th element of Γ( z ) : Γ d ( z ) = 1 means that z d is label-informative and should be selected when projecting z onto its sub-space).

Original: Given a sub-space index m ∈ { 0 , 1 } D , we denote the region on data space which has the same index m as A m = { x : Γ( g ( x )) = m } ⊂ X .
Revised: Given a sub-space index m ∈ { 0 , 1 } D , we denote the region on data space which has the same index m as A m = { x : Γ( g ( x )) = m } ⊂ X .

Original: Let P
Revised: Let P

Original: Sm be the distribution restricted by P Sπ over the set
Revised: Sm be the distribution restricted by P Sπ over the set

Original: A m and P Tm as the distribution restricted by P T over A m .
Revised: A m and P Tm as the distribution restricted by P T over A m .

Original: Eventually, we define p
Revised: Eventually, we define p

Original: Sm ( y | x ) as the probabilistic labeling distribution on the sub-space (cid:0) A m , P Sm (cid:1) , meaning that if x ∼ P Sm , p Sm ( y | x ) =
Revised: Sm ( y | x ) as the probabilistic labeling distribution on the sub-space (cid:0) A m , P Sm (cid:1) , meaning that if x ∼ P Sm , p Sm ( y | x ) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =

Revised: Kk =

Original: π k p
Revised: π k p

Original: Sk ( y | x ) .
Revised: Sk ( y | x ) .

Original: Similarly, we define if x ∼ P Tm , p Tm ( y | x ) =
Revised: Similarly, we define if x ∼ P Tm , p Tm ( y | x ) =

Original: p T ( y | x ) .
Revised: p T ( y | x ) .

Original: Due to this construction, any data sampled from P Sm or P T m have the same index m
Revised: Due to this construction, any data sampled from P Sm or P T m have the same index m

Original: =
Revised: =

Original: Γ ( z ) =
Revised: Γ ( z ) =

Original: Γ( g ( x )) , which specifies the set of relevant feature elements for classifying data from A m .
Revised: Γ( g ( x )) , which specifies the set of relevant feature elements for classifying data from A m .

Original: This hints us to employ an individual hypothesis f m = h m ◦ g Γ , where f m specializes on (cid:0) A m , P Sm (cid:1) and h m specializes on the sub-space (cid:0) g Γ
Revised: This hints us to employ an individual hypothesis f m = h m ◦ g Γ , where f m specializes on (cid:0) A m , P Sm (cid:1) and h m specializes on the sub-space (cid:0) g Γ

Original: ( A m ) , g Γ# P Sm (cid:1) with g Γ :=
Revised: ( A m ) , g Γ# P Sm (cid:1) with g Γ :=

Original: Π Γ ◦ g for each m .
Revised: Π Γ ◦ g for each m .

Original: We note that h m operates on the sub-space indexed by m , while f m operates on its induced data space.
Revised: We note that h m operates on the sub-space indexed by m , while f m operates on its induced data space.

Original: Additionally, since each data point x ∈ X corresponds to only a single Γ( g ( x )) , the data space is partitioned into disjoint sets, i.e., X = (cid:83) M m =

Revised: Additionally, since each data point x ∈ X corresponds to only a single Γ( g ( x )) , the data space is partitioned into disjoint sets, i.e., X = (cid:83) M m =

Original: A m , where A m ∩ A n = ∅ , ∀ m ̸ = n .
Revised: A m , where A m ∩ A n = ∅ , ∀ m ̸ = n .

Original: Therefore, these different hypotheses h m make use of different features, as reflected in the different sub-space index m .
Revised: Therefore, these different hypotheses h m make use of different features, as reflected in the different sub-space index m .

Original: We now present the way to efficiently formulate sub-space hypotheses.
Revised: We now present the way to efficiently formulate sub-space hypotheses.

Original: Let F := { f = h ◦ g : g ∈ G ∧ h ∈ H} be a hypothesis class of f : X → ∆ C , where f = h ◦ g with g : X → Z ∈ G and h : Z → ∆ C ∈ H .
Revised: Let F := { f = h ◦ g : g ∈ G ∧ h ∈ H} be a hypothesis class of f : X → ∆ C , where f = h ◦ g with g : X → Z ∈ G and h : Z → ∆ C ∈ H .

Original: Each hypothesis f = h ◦ g induces the subspace hypotheses f M =
Revised: Each hypothesis f = h ◦ g induces the subspace hypotheses f M =

Original: [ f m ] m ∈M for which f m ( x )
Revised: [ f m ] m ∈M for which f m ( x )

Original: =
Revised: =

Original: h m (Γ( g ( x )) ⊙ g ( x )) for x ∈ A m with h m ( m ⊙ g ( x ))
Revised: h m (Γ( g ( x )) ⊙ g ( x )) for x ∈ A m with h m ( m ⊙ g ( x ))

Original: =
Revised: =

Original: h ( m ⊙ g
Revised: h ( m ⊙ g

Original: ( x )) can be viewed as a hypothesis on the sub-space indexed by m .
Revised: ( x )) can be viewed as a hypothesis on the sub-space indexed by m .

Original: We define the general loss of f M =
Revised: We define the general loss of f M =

Original: [ f m ] m ∈M on the target domain as
Revised: [ f m ] m ∈M on the target domain as

Original: [Equation]
Revised: [Equation]

Original: where M ⊂ { 0 , 1 } D is the set of all feasible sub-space indices m conducted from the observed data, |M| specifies the cardinality of M , and D Tm is the joint distribution of ( x, y ) where x ∼ P Tm and y ∼ p
Revised: where M ⊂ { 0 , 1 } D is the set of all feasible sub-space indices m conducted from the observed data, |M| specifies the cardinality of M , and D Tm is the joint distribution of ( x, y ) where x ∼ P Tm and y ∼ p

Original: Tm ( · | x ) .
Revised: Tm ( · | x ) .

Original: [Figure]
Revised: [Figure]

Original: Figure 1a depicts the generative process of our LASSO in which the sub-space indicator Γ specifies the sub-space to project z
Revised: Figure 1a depicts the generative process of our LASSO in which the sub-space indicator Γ specifies the sub-space to project z

Original: = g ( x ) onto and the projected sample is then predicted by a sub-space hypothesis.
Revised: = g ( x ) onto and the projected sample is then predicted by a sub-space hypothesis.

Original: In Figure 1b, we intuitively demonstrate why projecting onto appropriate sub-spaces is useful to reduce latent data shift.
Revised: In Figure 1b, we intuitively demonstrate why projecting onto appropriate sub-spaces is useful to reduce latent data shift.

Original: For the red points on the latent space, the dimension 1 is commonly label-informative, hence projecting onto this dimension is helpful to preserve label information for constructing a good hypothesis on this sub-space.
Revised: For the red points on the latent space, the dimension 1 is commonly label-informative, hence projecting onto this dimension is helpful to preserve label information for constructing a good hypothesis on this sub-space.

Original: The same observation occurs for the blue points for the dimension 2.
Revised: The same observation occurs for the blue points for the dimension 2.

Original: The green points symbolize target data for which we observe that the latent data shift on the full latent space is significantly greater than that on the sub-spaces.
Revised: The green points symbolize target data for which we observe that the latent data shift on the full latent space is significantly greater than that on the sub-spaces.

Original: In addition,
Revised: In addition,

Original: Figure 1b also represents some notions in our developed theory, i.e., A m =[1 , 0] with the distribution P m =[1 , 0] includes the data samples whose latent representations are projected onto the sub-space indexed by m =
Revised: Figure 1b also represents some notions in our developed theory, i.e., A m =[1 , 0] with the distribution P m =[1 , 0] includes the data samples whose latent representations are projected onto the sub-space indexed by m =

Original: [1 , 0] and similar notions for another sub-space.
Revised: [1 , 0] and similar notions for another sub-space.

Original: Additionally, each sub-space index m associates with two equivalent hypotheses f m on A m and h m on the sub-space m .
Revised: Additionally, each sub-space index m associates with two equivalent hypotheses f m on A m and h m on the sub-space m .

Original: The following theorems explain the intuition that matching source and target domains on good subspaces helps reduce the latent data shift w.r.t.
Revised: The following theorems explain the intuition that matching source and target domains on good subspaces helps reduce the latent data shift w.r.t.

Original: a general unseen target domain; and offer us the training process of the sub-space indicator Γ , which extracts the most label informative features of latent representations z .
Revised: a general unseen target domain; and offer us the training process of the sub-space indicator Γ , which extracts the most label informative features of latent representations z .

Original: Theorem 2.
Revised: Theorem 2.

Original: Given a sub-space indicator Γ , if the loss function ℓ is upper-bounded by a positive constant L , the sub-space target general loss is upper-bounded by:
Revised: Given a sub-space indicator Γ , if the loss function ℓ is upper-bounded by a positive constant L , the sub-space target general loss is upper-bounded by:

Original: [Equation]
Revised: [Equation]

Original: (4) where D S m is the joint distribution of ( x, y ) with x ∼ P S m
Revised: (4) where D S m is the joint distribution of ( x, y ) with x ∼ P S m

Original: and y ∼ p S m ( · | x ) , and ∆ p m :=
Revised: and y ∼ p S m ( · | x ) , and ∆ p m :=

Original: E x ∼ P
Revised: E x ∼ P

Original: Sm (cid:2)(cid:13) (cid:13) p
Revised: Sm (cid:2)(cid:13) (cid:13) p

Original: Sm ( · | x ) − p Tm (
Revised: Sm ( · | x ) − p Tm (

Original: [Table]
Revised: [Table]

Original: Theorem 2 can be viewed as an extension of Theorem 1 in the context of LASSO with multiple sub-spaces.
Revised: Theorem 2 can be viewed as an extension of Theorem 1 in the context of LASSO with multiple sub-spaces.

Original: This theorem shows an upper-bound of the sub-space target loss which consists of three terms: (i) the sub-space label shift: ∆ p m , (ii) the source sub-space hypothesis losses : L (cid:0) f m ,
Revised: This theorem shows an upper-bound of the sub-space target loss which consists of three terms: (i) the sub-space label shift: ∆ p m , (ii) the source sub-space hypothesis losses : L (cid:0) f m ,

Original: D Sm (cid:1) , and (iii) the sub-space latent data shift : R α (cid:0) g Γ # P Tm , g Γ # P Sm (cid:1) .
Revised: D Sm (cid:1) , and (iii) the sub-space latent data shift : R α (cid:0) g Γ # P Tm , g Γ # P Sm (cid:1) .

Original: Since the sub-space label shift depends on the natural characteristics of domains and the source subspace hypothesis losses are trainable, the sub-space latent data shift (i.e., the divergence between the mixture of source domains: g Γ# P Sm and target domain: g Γ# P Tm on a sub-space) is essential to lower the upper-bound in (4).
Revised: Since the sub-space label shift depends on the natural characteristics of domains and the source subspace hypothesis losses are trainable, the sub-space latent data shift (i.e., the divergence between the mixture of source domains: g Γ# P Sm and target domain: g Γ# P Tm on a sub-space) is essential to lower the upper-bound in (4).

Original: As demonstrated in Figure 1b, by projecting corresponding latent representations onto low-dimensional sub-spaces conducted from label-informative features, we can reduce the latent data shift on the sub-spaces due to the compression effect when projecting to the sub-spaces.
Revised: As demonstrated in Figure 1b, by projecting corresponding latent representations onto low-dimensional sub-spaces conducted from label-informative features, we can reduce the latent data shift on the sub-spaces due to the compression effect when projecting to the sub-spaces.

Original: Moreover, if we can appropriately choose the good sets of label-informative features for the sub-spaces, we can preserve sufficient label information on the sub-spaces for training good hypotheses f m or h m with a low source sub-space hypothesis loss L (cid:0) f m , D Sm (cid:1) with the aim to lower the upper-bound in (4).
Revised: Moreover, if we can appropriately choose the good sets of label-informative features for the sub-spaces, we can preserve sufficient label information on the sub-spaces for training good hypotheses f m or h m with a low source sub-space hypothesis loss L (cid:0) f m , D Sm (cid:1) with the aim to lower the upper-bound in (4).

Original: The next arising question regarding how to train a qualified sub-space indicator Γ for choosing good sets of label-informative features to conduct low-dimensional and label-preserving sub-spaces is addressed in the following theorem.
Revised: The next arising question regarding how to train a qualified sub-space indicator Γ for choosing good sets of label-informative features to conduct low-dimensional and label-preserving sub-spaces is addressed in the following theorem.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let X is a random variable of source sample (i.e., drawn from P Sπ ) and Y is a random variable of ground-truth labels.
Revised: Let X is a random variable of source sample (i.e., drawn from P Sπ ) and Y is a random variable of ground-truth labels.

Original: Denote N = (cid:80) m ′ ∈M P Sπ
Revised: Denote N = (cid:80) m ′ ∈M P Sπ

Original: A m ′ ) , we then have
Revised: A m ′ ) , we then have

Original: [Equation]
Revised: [Equation]

Original: where L (cid:0) f m , D Sm (cid:1) is defined based on the cross-entropy loss and I denotes mutual information.
Revised: where L (cid:0) f m , D Sm (cid:1) is defined based on the cross-entropy loss and I denotes mutual information.

Original: Theorem 3 gives us a hint of how to train a sub-space indicator Γ together with the sub-space hypotheses f M =
Revised: Theorem 3 gives us a hint of how to train a sub-space indicator Γ together with the sub-space hypotheses f M =

Original: [ f m ] m ∈M .
Revised: [ f m ] m ∈M .

Original: As suggested by Theorem 3, we minimize the source sub-space hypothesis losses L (cid:0)
Revised: As suggested by Theorem 3, we minimize the source sub-space hypothesis losses L (cid:0)

Original: f m , D Sm (cid:1) by min Γ
Revised: f m , D Sm (cid:1) by min Γ

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: Furthermore, Theorem 3 indicates that by solving the optimization problem in (6), we implicitly learn Γ , g to maximize the mutual information I (Γ ( g ( X )) ⊙ g ( X ) , Y ) , which enables the subspace indicator Γ to preserve the label-informative features of z = g ( x ) .
Revised: Furthermore, Theorem 3 indicates that by solving the optimization problem in (6), we implicitly learn Γ , g to maximize the mutual information I (Γ ( g ( X )) ⊙ g ( X ) , Y ) , which enables the subspace indicator Γ to preserve the label-informative features of z = g ( x ) .

Original: 3. L ATENT SUB - SPACES LEARNING F RAMEWORK
Revised: 3. L ATENT SUB - SPACES LEARNING F RAMEWORK

Original: In what follows, we present the technical details of our proposed method which solves the optimization problem in equation 6.
Revised: In what follows, we present the technical details of our proposed method which solves the optimization problem in equation 6.

Original: In particular, we discuss the way to formulate and learn the sub-space indicator Γ , the feature extractor g : X → Z , and the classifier h : Z → ∆ C , in which Γ and ( g, h ) are updated alternatively.
Revised: In particular, we discuss the way to formulate and learn the sub-space indicator Γ , the feature extractor g : X → Z , and the classifier h : Z → ∆ C , in which Γ and ( g, h ) are updated alternatively.

Original: 3.4. S UB - SPACE INDICATOR
Revised: 3.4. S UB - SPACE INDICATOR

Original: We employ a probabilistic sub-space indicator Γ ( z ) ∈ [0 , 1] D in which Γ d ( z ) ∈ [0 , 1] represents the probability that z d is a label-informative feature.
Revised: We employ a probabilistic sub-space indicator Γ ( z ) ∈ [0 , 1] D in which Γ d ( z ) ∈ [0 , 1] represents the probability that z d is a label-informative feature.

Original: Additionally, in our implementation, we consider a 2D tensor z ∈ R D ′ × D , whereas each z d ∈ R D ′ known as an attribute consists of D ′ features which are simultaneously selected or unselected as the dimensions of the projection sub-space.
Revised: Additionally, in our implementation, we consider a 2D tensor z ∈ R D ′ × D , whereas each z d ∈ R D ′ known as an attribute consists of D ′ features which are simultaneously selected or unselected as the dimensions of the projection sub-space.

Original: The probabilistic sub-space indicator is formulated as Γ ( z ) =
Revised: The probabilistic sub-space indicator is formulated as Γ ( z ) =

Original: [Γ 1 ( z 1 ) , ..., Γ D ( z D )] ∈ [0 , 1] D .
Revised: [Γ 1 ( z 1 ) , ..., Γ D ( z D )] ∈ [0 , 1] D .

Original: The element-wise product Γ ( z ) ⊙ z is defined as z diag (Γ ( z )) , wherein we multiply each element in Γ ( z ) to a column of z to select or unselect the group of features in this column.
Revised: The element-wise product Γ ( z ) ⊙ z is defined as z diag (Γ ( z )) , wherein we multiply each element in Γ ( z ) to a column of z to select or unselect the group of features in this column.

Original: Consequently, Γ d ( z ) =
Revised: Consequently, Γ d ( z ) =

Original: Γ d ( z d ) is computed based solely on the group of D ′ features in the attribute z d itself rather than full latent z .
Revised: Γ d ( z d ) is computed based solely on the group of D ′ features in the attribute z d itself rather than full latent z .

Original: That means attribute-based Γ d depends on attributes which are shared across domains instead of domains, hence, becomes more independent from domain information.
Revised: That means attribute-based Γ d depends on attributes which are shared across domains instead of domains, hence, becomes more independent from domain information.

Original: (Ablation study about attribute-based Γ d is presented in Appendix B.1).

Original: Moreover, by Theorem 3, model can still learn meaningful attributes as long as the performance on source domains is guaranteed.
Revised: Moreover, by Theorem 3, model can still learn meaningful attributes as long as the performance on source domains is guaranteed.

Original: In unseen domains, the target of sub-space indicator Γ is to detect label-informative attributes which are learned in source domains instead of identifying all label-informative attributes.
Revised: In unseen domains, the target of sub-space indicator Γ is to detect label-informative attributes which are learned in source domains instead of identifying all labelinformative attributes.

Revised: For further discussion about attribute-based Γ d please refer to Appendix B.1.

Original: Finally, we minimize the following loss for updating Γ :
Revised: Finally, we minimize the following loss for updating Γ :

Original: [Equation]
Revised: [Equation]

Original: 3.4. S UB - SPACE HYPOTHESES
Revised: 3.4. S UB - SPACE HYPOTHESES

Original: To encourage the sub-space exploration and strengthen the sub-space hypotheses, given a data-label pair ( x, y ) ∼ D Sπ , we sample mask indicators from the Berboulli distribution with parameter Γ d ( z d ) , i.e,.
Revised: To encourage the sub-space exploration and strengthen the sub-space hypotheses, given a data-label pair ( x, y ) ∼ D Sπ , we sample mask indicators from the Berboulli distribution with parameter Γ d ( z d ) , i.e,.

Original: m d ∼ Ber (Γ d ( z d )) to gather m = [ m d ] Dd =1 , and then project onto a sub-space by m ⊙ z .
Revised: m d ∼ Ber (Γ d ( z d )) to gather m = [ m d ] Dd =1 , and then project onto a sub-space by m ⊙ z .

Original: Then, g, h are updated by minimizing the following loss function:
Revised: Then, g, h are updated by minimizing the following loss function:

Original: [Equation]
Revised: [Equation]

Original: where m ∼ Ber (Γ ( z )) means m d ∼ Ber (Γ d ( z d )) , d = 1 , ..., D .
Revised: where m ∼ Ber (Γ ( z )) means m d ∼ Ber (Γ d ( z d )) , d = 1 , ..., D .

Original: (Additional discussion about mask sampling can be found in Appendix B.2.)
Revised: For further discussion about mask sampling please refer to Appendix B.2.

Original: Finally, the pseudcode of our LASSO is summarized in Algorithm 1.
Revised: Finally, the pseudcode of our LASSO is summarized in Algorithm 1.

Original: Algorithm do 3: do 4:

Original: Sample

Original: Optimize Optimize 7: endfor 8: end9: Return
Revised: end end

Original: 3.4. I NFERENCE PROCESS
Revised: 3.4. I NFERENCE PROCESS

Original: At the testing phase, we use two inference strategies: threshold and ensemble strategies
Revised: At the testing phase, we use two inference strategies: threshold and ensemble strategies

Original: • Ensemble : for each target example x , we could compute z = g ( x ) , then sample T masks m 1 , ..., m T ∼ Ber
Revised: • Ensemble : for each target example x , we could compute z = g ( x ) , then sample T masks m 1 , ..., m T ∼ Ber

Original: (Γ ( z )) , and ensemble as h En = 1 T (cid:80)
Revised: (Γ ( z )) , and ensemble as h En = 1 T (cid:80)

Original: Tt =1 h ( m t ⊙ z ) .
Revised: Tt =1 h ( m t ⊙ z ) .

Original: However, for efficient computation, we approximate the ensemble prediction by: h En ≈ h (Γ ( z ) ⊙ z ) .
Revised: However, for efficient computation, we approximate the ensemble prediction by: h En ≈ h (Γ ( z ) ⊙ z ) .

Original: Threshold: we employ a threshold τ ∈ [0 , 1] to compute the mask.
Revised: Threshold: we employ a threshold τ ∈ [0 , 1] to compute the mask.

Original: Specifically, for each target example x , we compute z
Revised: Specifically, for each target example x , we compute z

Original: = g ( x ) , then evaluate the mask m = (cid:2) 1 Γ d (z d ) >
Revised: = g ( x ) , then evaluate the mask m = (cid:2) 1 Γ d (z d ) >

Original: τ (cid:3) D d =1 , where 1 is the indicator function, and predict as h ( m ⊙ z ) .
Revised: τ (cid:3) D d =1 , where 1 is the indicator function, and predict as h ( m ⊙ z ) .

Original:  E XPERIMENTS 4. E VALUATION ON B ENCHMARK D ATASETS
Revised:  E XPERIMENTS 4. E VALUATION ON B ENCHMARK D ATASETS

Original: We report the empirical results on PACS (Li et al., 2017), VLCS (Torralba & Efros, 2011), and
Revised: We report the empirical results on PACS (Li et al., 2017), VLCS (Torralba & Efros, 2011), and

Original: Office-Home (Venkateswara et al., 2017) datasets using standard backbones such as AlexNet,
Revised: Office-Home (Venkateswara et al., 2017) datasets using standard backbones such as AlexNet,

Original: ResNet18, and ResNet50 for the feature extractor.
Revised: ResNet18, and ResNet50 for the feature extractor.

Original: Due to the paper space limit, we leave the details of evaluation protocol, experimental settings and implementation in Appendix C.
Revised: Due to the paper space limit, we leave the details of evaluation protocol, experimental settings and implementation in Appendix C.

Original: The results on VLCS using AlexNet are reported in Table 1, the results on Office-Home using
Revised: The results on VLCS using AlexNet are reported in Table 1, the results on Office-Home using

Original: ResNet18 and ResNet50 are reported in Table 2 and the results on PACS using three backbones are reported in Table 3 respectively.
Revised: ResNet18 and ResNet50 are reported in Table 2 and the results on PACS using three backbones are reported in Table 3 respectively.

Original: Those empirical results clearly show that our LASSO provides competitive classification accuracy compared to the baselines.
Revised: Those empirical results clearly show that our LASSO provides competitive classification accuracy compared to the baselines.

Original: Especially, the average results on
Revised: Especially, the average results on

Original: Office-Home and PACS in Table 2 and Table 3 consistently show that our proposed method LASSO outperforms the ERM baseline with large margins on both backbones ResNet18 and ResNet50.
Revised: Office-Home and PACS in Table 2 and Table 3 consistently show that our proposed method LASSO outperforms the ERM baseline with large margins on both backbones ResNet18 and ResNet50.

Original: Although some other baselines also get comparative or better gains than ours with ResNet18, the gains shrink with ResNet50 since larger ResNet backbones are known to generalize better (Gulrajani & Lopez-Paz, 2021).
Revised: Although some other baselines also get comparative or better gains than ours with ResNet18, the gains shrink with ResNet50 since larger ResNet backbones are known to generalize better (Gulrajani & Lopez-Paz, 2021).

Original: Additionally, the threshold variant LASSO- τ aims to select and retain the top most relevant and label-informative features (the most appropriate sub-space), while the ensemble variant LASSO-En aggregates the predictions of possible sub-space hypotheses.
Revised: Additionally, the threshold variant LASSO- τ aims to select and retain the top most relevant and label-informative features (the most appropriate sub-space), while the ensemble variant LASSO-En aggregates the predictions of possible sub-space hypotheses.

Original: Therefore, the slight superior of the threshold variant to the ensemble one is possibly due to the fact that the former with an appropriate τ can select more compact and label-informative sets of features.
Revised: Therefore, the slight superior of the threshold variant to the ensemble one is possibly due to the fact that the former with an appropriate τ can select more compact and label-informative sets of features.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Sub-space representation:
Revised: Sub-space representation:

Original: To better understand the benefits of learning latent sub-spaces, we visualize the distribution of the learned features to analyze the latent space generated by ERM and
Revised: To better understand the benefits of learning latent sub-spaces, we visualize the distribution of the learned features to analyze the latent space generated by ERM and

Original: LASSO- τ using t-SNE (van der Maaten & Hinton, 2008) in Figure 2.
Revised: LASSO- τ using t-SNE (van der Maaten & Hinton, 2008) in Figure 2.

Original: Furthermore, we decide to choose ”sketch” which contains only colorless images as the target domain.
Revised: Furthermore, we decide to choose ”sketch” which contains only colorless images as the target domain.

Original: This domain can be considered as the most distant domain from the others and hence results in the largest source-target divergence.
Revised: This domain can be considered as the most distant domain from the others and hence results in the largest source-target divergence.

Original: Figure 2a shows feature distributions of different domains on latent space, in which samples are colored accordingly to domain label, with source domains are in red, green and back, while target domain is in blue.
Revised: Figure 2a shows feature distributions of different domains on latent space, in which samples are colored accordingly to domain label, with source domains are in red, green and back, while target domain is in blue.

Original: As it can be clearly seen, ERM training results in representations with high cross-domain separation for both target and source domains.
Revised: As it can be clearly seen, ERM training results in representations with high cross-domain separation for both target and source domains.

Original: On the other hands, the domain representations on a sub-latent space of LASSO is more homogeneously mixed, since samples from different domains are hardly distinguished.
Revised: On the other hands, the domain representations on a sub-latent space of LASSO is more homogeneously mixed, since samples from different domains are hardly distinguished.

Original: Figure 2b further shows that features from differ-
Revised: Figure 2b further shows that features from differ-

Original: [Figure]
Revised: [Figure]

Original: ent classes are more compact and clearly separated on LASSO’s sub-latent space than on full single latent space.
Revised: ent classes are more compact and clearly separated on LASSO’s sub-latent space than on full single latent space.

Original: These results indicate that feature distributions are better aligned on latent sub-space with our proposed LASSO approach.
Revised: These results indicate that feature distributions are better aligned on latent sub-space with our proposed LASSO approach.

Original: Additionally, we further verify the data shift reduction on sub-spaces by measuring the divergence between the source and target distributions of ERM and LASSO- τ via Jensen-Shannon (JS) divergence ( R JS ).
Revised: Additionally, we further verify the data shift reduction on sub-spaces by measuring the divergence between the source and target distributions of ERM and LASSO- τ via Jensen-Shannon (JS) divergence ( R JS ).

Original: We employ a domain discriminator h d which distinguishes the source domain:
Revised: We employ a domain discriminator h d which distinguishes the source domain:

Original: [Equation]
Revised: [Equation]

Original: where G is the feature extractor, which is g for ERM and g Γ for LASSO- τ .
Revised: where G is the feature extractor, which is g for ERM and g Γ for LASSO- τ .

Original: It is well-known that if we search h d in a family with infinite capacity then R JS ( G # P Sπ ||
Revised: It is well-known that if we search h d in a family with infinite capacity then R JS ( G # P Sπ ||

Original: G # P T )
Revised: G # P T )

Original: =
Revised: =

Original: max h d L ( h d )
Revised: max h d L ( h d )

Original: +2log (Goodfellow et al., 2014).
Revised: +2log (Goodfellow et al., 2014).

Original: The empirical results R ERM JS = 0 .
Revised: The empirical results R ERM JS = 0 .

Original: 247 and R LASSO- τ JS = 

Revised: 247 and R LASSO- τ JS =

Original: .show that the latent distribution divergence between sources and target domains is decreased on sub-spaces.
Revised: .show that the latent distribution divergence between sources and target domains is decreased on sub-spaces.

Original: This observation once again provides supporting evidences for our approach.
Revised: This observation once again provides supporting evidences for our approach.

Original: [Table]
Revised: [Table]

Original: Effect of τ : we present the performance on validation set (Val) of source domains and target domain with different thresholds τ to analyze its effect along with the average number of selected features.
Revised: Effect of τ : we present the performance on validation set (Val) of source domains and target domain with different thresholds τ to analyze its effect along with the average number of selected features.

Original: The Table 4 demonstrates that by choosing reasonable τ , the model performance can be improved since latent is mapped to appropriate subspace (i.e., sub-space being compact to reduce the data shift while preserving sufficient informative features for label prediction).
Revised: The Table 4 demonstrates that by choosing reasonable τ , the model performance can be improved since latent is mapped to appropriate subspace (i.e., sub-space being compact to reduce the data shift while preserving sufficient informative features for label prediction).

Original: Alternatively, in the case of large τ , the performance can be dropped since very few features remain.
Revised: Alternatively, in the case of large τ , the performance can be dropped since very few features remain.

Original: It also can be seen that with τ = 0 . 6 and τ = 0 .
Revised: It also can be seen that with τ = 0 . 6 and τ = 0 .

Original: 7 , the model utilizes only a small number of features (i.e., 59 . 9 and 14 . 4 on target domain respectively) but nonetheless still yields good results.
Revised: 7 , the model utilizes only a small number of features (i.e., 59 . 9 and 14 . 4 on target domain respectively) but nonetheless still yields good results.

Original: This indicates that sub-indicator Γ is able to identify essential labelinformative features.
Revised: This indicates that sub-indicator Γ is able to identify essential labelinformative features.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we first theoretically analyze the upper bound of target loss on latent space in domain generalization settings and the main factors that affect the performance.
Revised: In this paper, we first theoretically analyze the upper bound of target loss on latent space in domain generalization settings and the main factors that affect the performance.

Original: Then we point out that learning on appropriate latent sup-space can derive better generalization performance for domain generalization task.
Revised: Then we point out that learning on appropriate latent sup-space can derive better generalization performance for domain generalization task.

Original: Motivated by this, we have proposed a novel LASSO framework which can: (1) learn diverse latent sub-spaces and corresponding individual hypotheses and (2) project source and target examples onto appropriate sub-spaces preserving crucial label-informative features for label prediction, while reducing the latent data shift due to the sub-spaces compactness compared to the entire latent space.
Revised: Motivated by this, we have proposed a novel LASSO framework which can: (1) learn diverse latent sub-spaces and corresponding individual hypotheses and (2) project source and target examples onto appropriate sub-spaces preserving crucial label-informative features for label prediction, while reducing the latent data shift due to the sub-spaces compactness compared to the entire latent space.

Original: Experiments on benchmark datasets verify that LASSO achieves competitive and even state-of-the-art performances.
Revised: Experiments on benchmark datasets verify that LASSO achieves competitive and even state-of-the-art performances.

Original: A T HEORETICAL DEVELOPMENT
Revised: A T HEORETICAL DEVELOPMENT

Original: In this Section, we present all proofs relevant to theory developed in our paper.
Revised: In this Section, we present all proofs relevant to theory developed in our paper.

Original: We begin with a crucial proposition for our theory development.
Revised: We begin with a crucial proposition for our theory development.

Original: Proposition A.1.
Revised: Proposition A.1.

Original: Let f : X
Revised: Let f : X

Original: (cid:55)→ Y ∆ where f = h ◦ g with g : X (cid:55)→ Z and h : Z (cid:55)→ Y ∆ .
Revised: (cid:55)→ Y ∆ where f = h ◦ g with g : X (cid:55)→ Z and h : Z (cid:55)→ Y ∆ .

Original: i)
Revised: i)

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where p g ( y | z ) = (cid:82)
Revised: where p g ( y | z ) = (cid:82)

Original: g − 1( z ) p ( y | x ) p ( x ) dx (cid:82) g − 1( z ) p
Revised: g − 1( z ) p ( y | x ) p ( x ) dx (cid:82) g − 1( z ) p

Original: ( x ) dx for any z ∈ g ( X ) and p g ( z )
Revised: ( x ) dx for any z ∈ g ( X ) and p g ( z )

Original: =
Revised: =

Original: (cid:82) g − 1 ( z ) p ( x ) dx is the density of the push-forward distribution P g = g # P .
Revised: (cid:82) g − 1 ( z ) p ( x ) dx is the density of the push-forward distribution P g = g # P .

Original: ii)
Revised: ii)

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: where p g ( z ) =
Revised: where p g ( z ) =

Original: (cid:82) g − 1 ( z ) p ( x ) dx is the density of the push-forward distribution P g = g # P and p 1 ,g ( y | z ) =
Revised: (cid:82) g − 1 ( z ) p ( x ) dx is the density of the push-forward distribution P g = g # P and p 1 ,g ( y | z ) =

Original: (cid:82) g − 1( z ) p 1 ( y | x ) p ( x ) dx (cid:82) g − 1( z ) p
Revised: (cid:82) g − 1( z ) p 1 ( y | x ) p ( x ) dx (cid:82) g − 1( z ) p

Original: ( x ) dx and p 2 ,g ( y | z ) = (cid:82)
Revised: ( x ) dx and p 2 ,g ( y | z ) = (cid:82)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: y | x
Revised: y | x

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: x ) dx (cid:82) g
Revised: x ) dx (cid:82) g

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: dx for any z ∈ g ( X ) .
Revised: dx for any z ∈ g ( X ) .

Original: Proof.
Revised: Proof.

Original: i)
Revised: i)

Original: We first prove for any y ∈
Revised: We first prove for any y ∈

Original: [ C ] that
Revised: [ C ] that

Original: [Equation]
Revised: [Equation]

Original: In fact, we have
Revised: In fact, we have

Original: [Equation]
Revised: [Equation]

Original: where note that 1 condition is the indicator function which returns 1 if condition is true and
Revised: where note that 1 condition is the indicator function which returns 1 if condition is true and

Original: 0 otherwise, the equality in (1) =
Revised: 0 otherwise, the equality in (1) =

Original: due to Fubini theorem.
Revised: due to Fubini theorem.

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we reach
Revised: we reach

Original: [Equation]
Revised: [Equation]

Original: ii)
Revised: ii)

Original: We then prove that
Revised: We then prove that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We derive as
Revised: We derive as

Original: [Equation]
Revised: [Equation]

Original: where in the derivation of (1) =
Revised: where in the derivation of (1) =

Original: [Equation]
Revised: [Equation]

Original: we use Fubini theorem again.
Revised: we use Fubini theorem again.

Original: Theorem 4.
Revised: Theorem 4.

Original: (Theorem 1 in the main paper)
Revised: (Theorem 1 in the main paper)

Original: If the loss function ℓ is upper-bounded by a positive constant L , the target general loss is upper-bounded by:
Revised: If the loss function ℓ is upper-bounded by a positive constant L , the target general loss is upper-bounded by:

Original: [Equation]
Revised: [Equation]

Original: where g # P Sπ and g # P T are pushed-forward distributions induced by applying g on P Sπ and P T , R α is the α -divergence ( α > 1 ), and ∆ p k ( y | x ) := (cid:2)(cid:12)(cid:12)
Revised: where g # P Sπ and g # P T are pushed-forward distributions induced by applying g on P Sπ and P T , R α is the α -divergence ( α > 1 ), and ∆ p k ( y | x ) := (cid:2)(cid:12)(cid:12)

Original: p
Revised: p

Original: Sk ( y = c
Revised: Sk ( y = c

Original: | x ) − p T ( y = c
Revised: | x ) − p T ( y = c

Original: | x ) (cid:12)(cid:12)(cid:3) C c =1 represents the label shift between the labeling assignment mechanisms of an individual source domain and target domain on the input space.
Revised: | x ) (cid:12)(cid:12)(cid:3) C c =1 represents the label shift between the labeling assignment mechanisms of an individual source domain and target domain on the input space.

Original: Proof.
Revised: Proof.

Original: Let D S,gπ be the joint distribution including the pairs ( y, z ) where k ∼ Cat ( π ) , z ∼ p S,gk ( z ) , and y ∼ p S,gk ( y | z ) .
Revised: Let D S,gπ be the joint distribution including the pairs ( y, z ) where k ∼ Cat ( π ) , z ∼ p S,gk ( z ) , and y ∼ p S,gk ( y | z ) .

Original: The density p S,gπ ( y, z ) of this distribution is as follows:
Revised: The density p S,gπ ( y, z ) of this distribution is as follows:

Original: [Equation]
Revised: [Equation]

Original: Let D T,g be the joint distribution including the pairs ( y, z ) where z ∼ p T,g ( z ) and y ∼ p T,g ( y | z ) .
Revised: Let D T,g be the joint distribution including the pairs ( y, z ) where z ∼ p T,g ( z ) and y ∼ p T,g ( y | z ) .

Original: The density p T,g ( y, z ) of this distribution is as follows:
Revised: The density p T,g ( y, z ) of this distribution is as follows:

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: Eventually, we denote D h,g as the joint distribution including the pairs ( y, z ) where z ∼ p S,gπ ( z ) :
Revised: Eventually, we denote D h,g as the joint distribution including the pairs ( y, z ) where z ∼ p S,gπ ( z ) :

Original: = (cid:80)
Revised: = (cid:80)

Original: Kk =1 π k p S,gk ( z ) and y ∼ p T,g ( y | z ) .
Revised: Kk =1 π k p S,gk ( z ) and y ∼ p T,g ( y | z ) .

Original: The density p h,g ( y, z ) of this distribution is as follows:
Revised: The density p h,g ( y, z ) of this distribution is as follows:

Original: [Equation]
Revised: [Equation]

Original: With the above equipment, we have
Revised: With the above equipment, we have

Original: [Equation]
Revised: [Equation]

Original: We further note that
Revised: We further note that

Original: [Equation]
Revised: [Equation]

Original: where we have (1) = by using Proposition A.1 (i)
Revised: where we have (1) = by using Proposition A.1 (i)

Original: [Equation]
Revised: [Equation]

Original: where we have (2) = by using Proposition A.1 (ii).
Revised: where we have (2) = by using Proposition A.1 (ii).

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we manipulate L (cid:0) h, D T,g (cid:1) as
Revised: we manipulate L (cid:0) h, D T,g (cid:1) as

Original: [Equation]
Revised: [Equation]

Original: The Holder inequality gives us
Revised: The Holder inequality gives us

Original: [Equation]
Revised: [Equation]

Original: Referring to the definition of the R´enyi divergence and note that ℓ ( h ( z )
Revised: Referring to the definition of the R´enyi divergence and note that ℓ ( h ( z )

Original: [Equation]
Revised: [Equation]

Original: y ) ≤ L
Revised: y ) ≤ L

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: We further derive
Revised: We further derive

Original: [Equation]
Revised: [Equation]

Original: Therefore, combining (12) and (13)
Revised: Therefore, combining (12) and (13)

Original: [Equation]
Revised: [Equation]

Original: we reach the following inequality:
Revised: we reach the following inequality:

Original: [Equation]
Revised: [Equation]

Original: Subsequently, by noting that
Revised: Subsequently, by noting that

Original: [Equation]
Revised: [Equation]

Original: we reach
Revised: we reach

Original: [Equation]
Revised: [Equation]

Original: Finally, referring to the inequality in (11)
Revised: Finally, referring to the inequality in (11)

Original: [Equation]
Revised: [Equation]

Original: we reach the conclusion.
Revised: we reach the conclusion.

Original: Remark: We would like to highlight that our Theorem 1 despite developing in a general context of DG is also novel due to two following reasons: (i) it proposes the target loss upper-bound in a general setting of multi-class classification with a sufficiently general loss, which can be viewed as a non-trivial generalization of existing works in DA and DG (Mansour et al., 2009; Ben-David et al., 2010; Zhao et al., 2019) and (ii) our bound is interweaving both input and latent spaces, which is novel and appropriate for theoretical analyses in deep learning
Revised: Remark: We would like to highlight that our Theorem 1 despite developing in a general context of DG is also novel due to two following reasons: (i) it proposes the target loss upper-bound in a general setting of multi-class classification with a sufficiently general loss, which can be viewed as a non-trivial generalization of existing works in DA and DG (Mansour et al., 2009; Ben-David et al., 2010; Zhao et al., 2019) and (ii) our bound is interweaving both input and latent spaces, which is novel and appropriate for theoretical analyses in deep learning

Original: [Equation]
Revised: [Equation]

Original: while the bounds in previous works only involve input space.
Revised: while the bounds in previous works only involve input space.

Original: Theorem 5.
Revised: Theorem 5.

Original: (Theorem 2 in the main paper)
Revised: (Theorem 2 in the main paper)

Original: Given a deterministic sub-space indicator Γ , if the loss function ℓ is upper-bounded by a positive constant L , the sub-space target general loss is upperbounded by:
Revised: Given a deterministic sub-space indicator Γ , if the loss function ℓ is upper-bounded by a positive constant L , the sub-space target general loss is upperbounded by:

Original: [Equation]
Revised: [Equation]

Original: 14) where D S m is the joint distribution of ( x, y ) with x ∼ P S m
Revised: 14) where D S m is the joint distribution of ( x, y ) with x ∼ P S m

Original: and y ∼ p S m ( · | x ) , and ∆ p m :=
Revised: and y ∼ p S m ( · | x ) , and ∆ p m :=

Original: E x ∼ P
Revised: E x ∼ P

Original: Sm (cid:2)(cid:13)(cid:13)
Revised: Sm (cid:2)(cid:13)(cid:13)

Original: p S m ( · | x ) − p Tm ( · | x ) (cid:13)(cid:13) 1 (cid:3) .
Revised: p S m ( · | x ) − p Tm ( · | x ) (cid:13)(cid:13) 1 (cid:3) .

Original: sub-space indices m .
Revised: sub-space indices m .

Original: Proof.
Revised: Proof.

Original: Given a sub-space index m ∈ M , by noting that f m ( x ) = h (cid:0) g Γ ( x ) (cid:1) with x ∼ P Sm over
Revised: Given a sub-space index m ∈ M , by noting that f m ( x ) = h (cid:0) g Γ ( x ) (cid:1) with x ∼ P Sm over

Original: A m , using the same proof for a single space in Theorem 4, we obtain
Revised: A m , using the same proof for a single space in Theorem 4, we obtain

Original: [Equation]
Revised: [Equation]

Original: Finally, taking average over m ∈ M , we reach
Revised: Finally, taking average over m ∈ M , we reach

Original: [Equation]
Revised: [Equation]

Original: Theorem 6.
Revised: Theorem 6.

Original: (Theorem 3 in the main paper)
Revised: (Theorem 3 in the main paper)

Original: Let X is a random variable of source sample (i
Revised: Let X is a random variable of source sample (i

Original: [Equation]
Revised: [Equation]

Original: e., drawn from P Sπ ) and Y is a random variable of ground-truth labels.
Revised: e., drawn from P Sπ ) and Y is a random variable of ground-truth labels.

Original: Denote N = (cid:80) m
Revised: Denote N = (cid:80) m

Original: [Equation]
Revised: [Equation]

Original: P Sπ ( A m ′ ) , we then have
Revised: P Sπ ( A m ′ ) , we then have

Original: [Equation]
Revised: [Equation]

Original: where the loss L (cid:0) f m , D Sm (cid:1) is defined based on the cross-entropy loss and I denotes the mutual information.
Revised: where the loss L (cid:0) f m , D Sm (cid:1) is defined based on the cross-entropy loss and I denotes the mutual information.

Original: Proof.
Revised: Proof.

Original: Denote T =
Revised: Denote T =

Original: Γ ( g ( X )) ⊙ g ( X ) , we have
Revised: Γ ( g ( X )) ⊙ g ( X ) , we have

Original: [Equation]
Revised: [Equation]

Original: where H specifies the entropy, D KL is Kullback-Leibler (KL) divergence, h ( y | t )
Revised: where H specifies the entropy, D KL is Kullback-Leibler (KL) divergence, h ( y | t )

Original: =
Revised: =

Original: h ( t, y )
Revised: h ( t, y )

Original: =
Revised: =

Original: h (Γ ( g ( x )) ⊙ g ( x ) , y ) for any h : Z → Y ∆ , and h ( t, y ) returns the y -th element of h ( t ) .
Revised: h (Γ ( g ( x )) ⊙ g ( x ) , y ) for any h : Z → Y ∆ , and h ( t, y ) returns the y -th element of h ( t ) .

Original: We further derive
Revised: We further derive

Original: [Equation]
Revised: [Equation]

Original: Note that we have (1) = because Γ ( g ( x )) ⊙ g ( x ) pushes forward X ∼ p Sπ ( x ) to T ∼ p ( t ) .
Revised: Note that we have (1) = because Γ ( g ( x )) ⊙ g ( x ) pushes forward X ∼ p Sπ ( x ) to T ∼ p ( t ) .

Original: Moreover, according to our definitions: P Sπ = (cid:80) m ∈M P Sπ ( A m )
Revised: Moreover, according to our definitions: P Sπ = (cid:80) m ∈M P Sπ ( A m )

Original: N
Revised: N

Original: P Sm , we hence obtain
Revised: P Sm , we hence obtain

Original: [Equation]
Revised: [Equation]

Original: B A DDITIONAL EXPERIMENTS B. E FFECT OF THE ATTRIBUTE - BASED INDICATOR Γ d
Revised: B A DDITIONAL EXPERIMENTS B. E FFECT OF THE ATTRIBUTE - BASED INDICATOR Γ d

Original: [Table]
Revised: We consider a feature vector z = [ z 1 , ..., z D ] ∈ R D ′ × D , whereas each z d ∈ R D ′ known as an attribute and model sub-space indicator as Γ ( z ) = [Γ 1 ( z 1 ) , ..., Γ D ( z D )] ∈ [0 , 1] D , Γ d ( z ) =

Revised: Γ d ( z d ) is computed based solely on the group of D ′ features in the attribute z d itself rather than full latent z .

Revised: Consequently, attribute-based Γ d depends on attributes which are shared across domains, hence, becomes more independent to domain.

Revised: Moreover, by Theorem 3, model can learn meaningful attributes as long as the performance on source domains is guaranteed.

Revised: In unseen domains, the target of sub-space indicator Γ is to detect label-informative attributes which is learned in source domains instead of identifying all label-informative attributes.

Revised: More specifically, in training phase, attributes z d come from different source domains, corresponding Γ d plays as a domain invariant hypothesis which output { 0 , 1 } identifying label-informative attributes across source domains.

Revised: In unseen domain, Γ d ( z d ) only predict z d is label-informative attribute which are learned in source domains or not.

Revised: [Table]

Original: In order to analyze the benefit of attribute-based Γ d , we examine the performance on the setting of Γ d ( z ) which uses full latent z as input for Γ d and Γ d ( z d ) which uses only attribute z d as input for Γ d .
Revised: In order to analyze the benefit of attribute-based Γ d , we examine the performance on the setting of Γ d ( z ) which uses full latent z as input for Γ d and Γ d ( z d ) which uses only attribute z d as input for Γ d .

Original: As can be seen from Table 5, both Γ d ( z d ) and Γ d ( z ) achieve competitive performances on target domain: ”Photo”, ”Art-painting” and ”cartoon” domain but Γ d ( z d ) setting outperforms on ”sketch” domain.
Revised: As can be seen from Table 5, both Γ d ( z d ) and Γ d ( z ) achieve competitive performances on target domain: ”Photo”, ”Art-painting” and ”cartoon” domain but Γ d ( z d ) setting outperforms on ”sketch” domain.

Original: Remind that using ”sketch” as target domain leads to the largest source-target divergence.
Revised: Remind that using ”sketch” as target domain leads to the largest source-target divergence.

Original: The results of ”sketch” domain demonstrate that Γ d ( z ) becomes less efficient when source-target divergence.
Revised: The results of ”sketch” domain demonstrate that Γ d ( z ) becomes less efficient when source-target divergence.

Original: In contrast, Γ d ( z ) shows its better generalization as not be interfered by other noisy attributes.
Revised: In contrast, Γ d ( z ) shows its better generalization as not be interfered by other noisy attributes.

Original: B. M ASK SAMPLING FOR MODEL TRAINING
Revised: B. M ASK SAMPLING FOR MODEL TRAINING

Original: In training phase, sampling m ∼ Ber (Γ ( z )) would help hypotheses explore more useful subspaces.
Revised: In training phase, sampling m ∼ Ber (Γ ( z )) would help hypotheses explore more useful subspaces.

Original: The motivation is that target domains might have or share a small sub-set of learned features in source domains e.g. in PACS dataset, many label-informative features can be learned in rich information domain such as ”Photo” domain while ”Sketch” domain only has edge/shape information for classification.
Revised: The motivation is that target domains might have or share a small sub-set of learned features in source domains e.g. in PACS dataset, many label-informative features can be learned in rich information domain such as ”Photo” domain while ”Sketch” domain only has edge/shape information for classification.

Original: Therefore, in training phase, we would like to train hypotheses on different sub-set of learned features (i.e., sub-spaces of sub-space) which would strengthen sub-space hypotheses on arbitrary target domains..
Revised: Therefore, in training phase, we would like to train hypotheses on different sub-set of learned features (i.e., sub-spaces of sub-space) which would strengthen sub-space hypotheses on arbitrary target domains..

Original: [Table]
Revised: [Table]

Original: To demonstrate the benefit of sampling [Table] we conduct additional experiments using deterministic manner which uses threshold τ to determine mask m in model training.
Revised: To demonstrate the benefit of sampling, we conduct additional experiments using deterministic manner which uses threshold τ to determine mask m in model training.

Original: The results in Table 6 indicate sampling significantly enhance performance.
Revised: The results in Table 6 indicate sampling significantly enhance performance.

Original: B. D ISCUSSION AND COMPARISON WITH R ANDOM D ROPOUT
Revised: B. D ISCUSSION AND COMPARISON WITH R ANDOM D ROPOUT

Original: We would like to report additional experimental results in comparison with ”Random Dropout”
Revised: We would like to report additional experimental results in comparison with ”Random Dropout”

Original: Baseline (Srivastava et al., 2014) for different dropping rate as follows:
Revised: Baseline (Srivastava et al., 2014) for different dropping rate as follows:

Original: [Table]
Revised: [Table]

Original:  The results in Table 7 and Table 8 indicate that ERM with an appropriate dropping rate (i.e., 50% for ResNet18 and 30% for ResNet50) is able to achieve comparative performance to most current baselines.[Table] In fact
Revised: [Table] The results in Table 7 and Table 8 indicate that ERM with an appropriate dropping rate (i.e., 50% for ResNet18 and 30% for ResNet50) is able to achieve comparative performance to most current baselines.

Original: [Table] ”Dropout” has a quite similar effect to the representation as LASSO [Table] which increases the diversity of latent representations.
Revised: In fact, ”Dropout” has a quite similar effect to the representation as LASSO, which increases the diversity of latent representations.

Original: However, LASSO goes one step further compared to ”Dropout”.
Revised: However, LASSO goes one step further compared to ”Dropout”.

Original: Specifically, representation is used as it is during inference time, without making the use of information in the target data, e [Table] g., using the full network for inference [Table]
Revised: Specifically, representation is used as it is during inference time, without making the use of information in the target data, e.g., using the full network for inference, or average over a set of randomly ”Dropout” networks (MC dropout).

Original: or average over a set of randomly ”Dropout” networks (MC dropout).

Original: On the contrary, the sub-space indicator in LASSO additionally takes into account the information from the target domain to select the suitable masked network which is likely to generalize better to this particular target domain.
Revised: On the contrary, the sub-space indicator in LASSO additionally takes into account the information from the target domain to select the suitable masked network which is likely to generalize better to this particular target domain.

Original: This is the motivation behind the selection of appropriate sub-space
Revised: This is the motivation behind the selection of appropriate sub-space.

Original: [Table]

Original: C E XPERIMENTAL S ETTINGS C. D ATASET D ETAILS
Revised: C E XPERIMENTAL S ETTINGS C. D ATASET D ETAILS

Original: To evaluate the effectiveness of the proposed method, we utilize three datasets : PACS (Li et al., 2017), VLCS (Torralba & Efros, 2011), and Office-Home (Venkateswara et al., 2017), which are the common DG benchmarks with multi-source domains.
Revised: To evaluate the effectiveness of the proposed method, we utilize three datasets: PACS (Li et al., 2017), VLCS (Torralba & Efros, 2011), and Office-Home (Venkateswara et al., 2017), which are the common DG benchmarks with multi-source domains.

Original: • PACS (Li et al., 2017): 9991 images of seven classes in total, over four domains:Art painting (A), Cartoon (C), Sketches (S), and Photo (P).
Revised: • PACS (Li et al., 2017): 9991 images of seven classes in total, over four domains:Art painting (A), Cartoon (C), Sketches (S), and Photo (P).

Original: • VLCS (Torralba & Efros, 2011): five classes over four domains with a total ofsamples.
Revised: • VLCS (Torralba & Efros, 2011): five classes over four domains with a total ofsamples.

Original: The domains are defined by four image origins, i.e., images were taken from the PASCAL VOC 2007 (V), LabelMe (L), Caltech (C) and Sun (S) datasets.
Revised: The domains are defined by four image origins, i.e., images were taken from the PASCAL VOC 2007 (V), LabelMe (L), Caltech (C) and Sun (S) datasets.

Original: • Office-Home
Revised: • Office-Home

Original: (Venkateswara et al., 2017): 65 categories of 15500 daily objects fromdomains: Art, Clipart, Product (vendor website with white-background) and Real-World (real-object collected from regular cameras).
Revised: (Venkateswara et al., 2017): 65 categories of 15500 daily objects fromdomains: Art, Clipart, Product (vendor website with white-background) and Real-World (real-object collected from regular cameras).

Original: C. E VALUATION PROTOCOL
Revised: C. E VALUATION PROTOCOL

Original: For PACS and VLCS, we follow the setting of (Li et al., 2017) as these two datasets provide specific “train” and “validation” splits for each domain to ensure a fair comparison.
Revised: For PACS and VLCS, we follow the setting of (Li et al., 2017) as these two datasets provide specific “train” and “validation” splits for each domain to ensure a fair comparison.

Original: We use the validation subsets from all training domains to create an overall validation set.
Revised: We use the validation subsets from all training domains to create an overall validation set.

Original: On the target domain, we evaluate on the entire dataset.
Revised: On the target domain, we evaluate on the entire dataset.

Original: Office-Home also follows the leave-one-domain-out protocol in (Li et al., 2017).
Revised: Office-Home also follows the leave-one-domain-out protocol in (Li et al., 2017).

Original: In addition, the generalization of our method is based on the accuracy of the subspace indicator Γ .
Revised: In addition, the generalization of our method is based on the accuracy of the subspace indicator Γ .

Original: Therefore, we choose the model when the sub-space indicator Γ converged.
Revised: Therefore, we choose the model when the sub-space indicator Γ converged.

Original: In particular, we train all models for the same fixed number of steps and consider only the final checkpoint.
Revised: In particular, we train all models for the same fixed number of steps and consider only the final checkpoint.

Original: Finally, we report the average performance with error bars over 5 runs for all experimental settings and compare two variants of our proposed method LASSO − τ (the threshold variant) and
Revised: Finally, we report the average performance with error bars over 5 runs for all experimental settings and compare two variants of our proposed method LASSO − τ (the threshold variant) and

Original: LASSO-En (the ensemble variant) with the recent domain generalization baselines.
Revised: LASSO-En (the ensemble variant) with the recent domain generalization baselines.

Original: C. B ASELINES
Revised: C. B ASELINES

Original: We compare with the following recent domain generalization baselines (with reproducible public codes):
Revised: We compare with the following recent domain generalization baselines (with reproducible public codes):

Original: • ERM (Vapnik, 1999):minimizes the sum of errors across domains and examples.
Revised: • ERM (Vapnik, 1999):minimizes the sum of errors across domains and examples.

Original: For our experiments, we employ the implementation from (Gulrajani & Lopez-Paz, 2021), a strong baseline that can achieve competitive accuracies on DG benchmarks.
Revised: For our experiments, we employ the implementation from (Gulrajani & Lopez-Paz, 2021), a strong baseline that can achieve competitive accuracies on DG benchmarks.

Original: • MLDG
Revised: • MLDG

Original: (Li et al., 2018a):
Revised: (Li et al., 2018a):

Original: Learning to generalize: Meta-learning for domain generalization.
Revised: Learning to generalize: Meta-learning for domain generalization.

Original: • MetaReg (Balaji et al., 2018):
Revised: • MetaReg (Balaji et al., 2018):

Original: Metareg: Towards domain generalization using metaregularization.
Revised: Metareg: Towards domain generalization using metaregularization.

Original: • MASF
Revised: • MASF

Original: (Dou et al., 2019): Domain generalization via model-agnostic learning of semantic features.
Revised: (Dou et al., 2019): Domain generalization via model-agnostic learning of semantic features.

Original: • JiGen (Carlucci et al., 2019):
Revised: • JiGen (Carlucci et al., 2019):

Original: Domain Generalization by Solving Jigsaw Puzzles.
Revised: Domain Generalization by Solving Jigsaw Puzzles.

Original: • DMG
Revised: • DMG

Original: (Chattopadhyay et al., 2020):
Revised: (Chattopadhyay et al., 2020):

Original: Learning to balance specificity and invariance for in and out of domain generalization.
Revised: Learning to balance specificity and invariance for in and out of domain generalization.

Original: • L2A-OT
Revised: • L2A-OT

Original: (Zhou et al., 2020a): Learning to Generate Novel Domains for Domain Generalization.
Revised: (Zhou et al., 2020a): Learning to Generate Novel Domains for Domain Generalization.

Original: • RSC
Revised: • RSC

Original: (Huang et al., 2020): Self-challenging improves cross-domain generalization.
Revised: (Huang et al., 2020): Self-challenging improves cross-domain generalization.

Original: • GroupDRO (Sagawa et al., 2019): Distributionally robust neural networks for group shifts:
Revised: • GroupDRO (Sagawa et al., 2019): Distributionally robust neural networks for group shifts:

Original: On the importance of regularization for worst-case generalization.
Revised: On the importance of regularization for worst-case generalization.

Original: • SFA-A
Revised: • SFA-A

Original: (Li et al., 2021):
Revised: (Li et al., 2021):

Original: A Simple Feature Augmentation for Domain Generalization.
Revised: A Simple Feature Augmentation for Domain Generalization.

Original: • MTL
Revised: • MTL

Original: (Blanchard et al., 2021):
Revised: (Blanchard et al., 2021):

Original: Domain Generalization by Marginal Transfer Learning.
Revised: Domain Generalization by Marginal Transfer Learning.

Original: • SagNets (Nam et al., 2021):
Revised: • SagNets (Nam et al., 2021):

Original: Reducing Domain Gap by Reducing Style Bias.
Revised: Reducing Domain Gap by Reducing Style Bias.

Original: • MatchDG (Mahajan et al., 2021):
Revised: • MatchDG (Mahajan et al., 2021):

Original: Domain generalization using causal matching.
Revised: Domain generalization using causal matching

Original: C. I MPLEMENTATION DETAILS
Revised: C. I MPLEMENTATION DETAILS

Original: We detail our algorithm’s implementation sub-space Indicator Γ : given latent vector z = g ( x ) ∈
Revised: We detail our algorithm’s implementation sub-space Indicator Γ : given latent vector z = g ( x ) ∈

Original: R D × D ′ (i.e. z =
Revised: R D × D ′ (i.e. z =

Original: [ z 1 , .., z D ] | z d ∈ R D ′ ), we model Γ( . ) as a composition of independent indicator on each attribute: Γ( · ) =
Revised: [ z 1 , .., z D ] | z d ∈ R D ′ ), we model Γ( . ) as a composition of independent indicator on each attribute: Γ( · ) =

Original: { Γ d ( · ) } d =1 ...D where each Γ d is neural network.
Revised: { Γ d ( · ) } d =1 ...D where each Γ d is neural network.

Original: In current work, we model Γ d as a Full-Connected (FC) layer [ D ′ × 1] prior to a sigmoid function σ and Γ d takes z d as input.
Revised: In current work, we model Γ d as a Full-Connected (FC) layer [ D ′ × 1] prior to a sigmoid function σ and Γ d takes z d as input.

Original: (Code for experiments is available in Supplementary Material.)
Revised: (Code for experiments is available in Supplementary Material.)

Original: The following are adapted versions on backbones, in which the last FC layer is used as the classification network.
Revised: The following are adapted versions on backbones, in which the last FC layer is used as the classification network.

Original: The summary of the additional parameter is presented in Table.9.
Revised: The summary of the additional parameter is presented in Table.9.

Original: AlexNet: For AlexNet backbone, the extracted feature dimensions are z ∈ R 4096 .
Revised: AlexNet: For AlexNet backbone, the extracted feature dimensions are z ∈ R 4096 .

Original: In current implementation, we partition z into 256 features z d ∈ R 16 as input for Γ d .
Revised: In current implementation, we partition z into 256 features z d ∈ R 16 as input for Γ d .

Original: ResNet18 & ResNet50:
Revised: ResNet18 & ResNet50:

Original: For ResNet18 & ResNet50 backbones, instead of partitioning the feature tensor, we utilize direcly feature tensor before avepooling layer.
Revised: For ResNet18 & ResNet50 backbones, instead of partitioning the feature tensor, we utilize direcly feature tensor before avepooling layer.

Original: In particular, the extracted feature are z ∈ R 512 × 7 × 7 for ResNet18 and z ∈ R 2048 × 7 × 7 for ResNet50.
Revised: In particular, the extracted feature are z ∈ R 512 × 7 × 7 for ResNet18 and z ∈ R 2048 × 7 × 7 for ResNet50.

Original: We use z d ∈ R 7 × 7=49 as input for Γ d .
Revised: We use z d ∈ R 7 × 7=49 as input for Γ d .

Original: [Table] Optimization: We adopt the SDG optimizer for training with: the model learning-rate (lr h,g ) and sub-space indicator Γ learning-rate (lr Γ ) for different bacbones and datasets presented in Table 10, mini-batch size of 64 for 100 epochs.
Revised: Optimization: We adopt the SDG optimizer for training with: the model learning-rate (lr h,g ) and sub-space indicator Γ learning-rate (lr Γ ) for different bacbones and datasets presented in Table 10, mini-batch size of 64 for 100 epochs.

Original: The learning rate (lr h,g and lr Γ ) decayed by 0 .
Revised: The learning rate (lr h,g and lr Γ ) decayed by 0 .

Original: 1 after every 80% epochs.
Revised: 1 after every 80% epochs.

Original: The values of hyper-parameters lr h,g , lr Γ and τ are chosen based on the performance on source validation set.
Revised: The values of hyper-parameters lr h,g , lr Γ and τ are chosen based on the performance on source validation set.

Original: Additionally, in order to the sub-space indicator Γ quickly adapt to features induced by h, g , lr Γ is set much larger than lr h,g . 
Revised: Additionally, in order to the sub-space indicator Γ quickly adapt to features induced by h, g , lr Γ is set much larger than lr h,g . [Table] [Table]

Original: The number of sampling times in Eq. 8 is also a hyper-parameter in our LASSO.
Revised: The number of sampling times in Eq. 8 is also a hyper-parameter in our LASSO.

Original: When trying with several possible values of this hyper-parameter, we observed that (i) when increasing the number of sampling times, the performance slightly improves but the training time more significantly increases, and (ii) when setting this hyper-parameter to 1 , the performance consistently is satisfactory[Table] for all experimental datasets.
Revised: When trying with several possible values of this hyper-parameter, we observed that (i) when increasing the number of sampling times, the performance slightly improves but the training time more significantly increases, and (ii) when setting this hyper-parameter to 1 , the performance consistently is satisfactory for all experimental datasets.

Original: Therefore, in all experiments, we sample once per mini-batch for computation efficiency.
Revised: Therefore, in all experiments, we sample once per mini-batch for computation efficiency.

Original: Additionally, at inference time, we conveniently choose τ = 0 .
Revised: Additionally, at inference time, we conveniently choose τ = 0 .

Original: 6 for ResNet & ResNet50 and τ = 0 .
Revised: 6 for ResNet & ResNet50 and τ = 0 .

Original: 5 for AlexNet on all datasets.
Revised: 5 for AlexNet on all datasets.

Original: However, from the results in Table 4, it is reasonable to choose τ based on the performance of the validation data of source domains although sometimes the best performance on validation data does not reflect the best performance on target domain.
Revised: However, from the results in Table 4, it is reasonable to choose τ based on the performance of the validation data of source domains although sometimes the best performance on validation data does not reflect the best performance on target domain.

Original: Data augmentation: We follow the image augmentation protocol introduced in (Gulrajani &
Revised: Data augmentation: We follow the image augmentation protocol introduced in (Gulrajani &

Original: Lopez-Paz, 2021) which is increasingly standard in state-of-the-art DG work: crops of random size and aspect ratio, resizing to 224 × 224 pixels, random horizontal flips, random color jitter, grayscaling the image with 10% probability, and normalization using the ImageNet channel statistics.
Revised: Lopez-Paz, 2021) which is increasingly standard in state-of-the-art DG work: crops of random size and aspect ratio, resizing to 224 × 224 pixels, random horizontal flips, random color jitter, grayscaling the image with 10% probability, and normalization using the ImageNet channel statistics.

Original: Finally, we use system of GPU NVIDIA Tesla V100 with dual CPUs Intel Xeon E5-2698 v4 to conduct our experiments.
Revised: Finally, we use system of GPU NVIDIA Tesla V100 with dual CPUs Intel Xeon E5-2698 v4 to conduct our experiments.