Original: G ATED D OMAIN U NITS FOR M ULTI - SOURCE D OMAIN G ENERALIZATION
Revised: G ATED D OMAIN U NITS FOR M ULTI - SOURCE D OMAIN G ENERALIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Distribution shift (DS) is a common problem that deteriorates the performance of learning machines.
Revised: Distribution shift (DS) is a common problem that deteriorates the performance of learning machines.

Original: To tackle this problem, we postulate that real-world distributions are composed of elementary distributions that remain invariant across different environments.
Revised: To tackle this problem, we postulate that real-world distributions are composed of elementary distributions that remain invariant across different environments.

Original: We call this an invariant elementary distribution (I.E.D.) assumption.
Revised: We call this an invariant elementary distribution (I.E.D.) assumption.

Original: The I.E.D. assumption implies an invariant structure in the solution space that enables knowledge transfer to unseen domains.
Revised: The I.E.D. assumption implies an invariant structure in the solution space that enables knowledge transfer to unseen domains.

Original: To exploit this property in domain generalization (DG), we developed a modular neural network layer that consists of Gated Domain Units (GDUs).
Revised: To exploit this property in domain generalization (DG), we developed a modular neural network layer that consists of Gated Domain Units (GDUs).

Original: Each GDU learns an embedding of an individual elementary distribution that allows us to encode the domain similarities during the training.
Revised: Each GDU learns an embedding of an individual elementary distribution that allows us to encode the domain similarities during the training.

Original: During inference, the GDUs compute similarities between an observation and each of the corresponding elementary distributions which are then used to form a weighted ensemble of learning machines.
Revised: During inference, the GDUs compute similarities between an observation and each of the corresponding elementary distributions which are then used to form a weighted ensemble of learning machines.

Original: Because our layer is trained with backpropagation, it can naturally be integrated into existing deep learning frameworks.
Revised: Because our layer is trained with backpropagation, it can naturally be integrated into existing deep learning frameworks.

Original: Our evaluation on image, text, graph, and time-series data shows a significant improvement in the performance on out-of-training target domains without domain information and any access to data from the target domains.
Revised: Our evaluation on image, text, graph, and time-series data shows a significant improvement in the performance on out-of-training target domains without domain information and any access to data from the target domains.

Original: This finding supports the practicality of the I.E.D. assumption and demonstrates that our GDUs can learn to represent these elementary distributions.
Revised: This finding supports the practicality of the I.E.D. assumption and demonstrates that our GDUs can learn to represent these elementary distributions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A fundamental assumption in machine learning is that training and test data are independently and identically distributed (I.I.D.).
Revised: A fundamental assumption in machine learning is that training and test data are independently and identically distributed (I.I.D.).

Original: This assumption ensures consistency-results from statistical learning theory, meaning that the learning machine obtained from an empirical risk minimization (ERM) attains the lowest achievable risk as sample size grows (Vapnik, 1998;
Revised: This assumption ensures consistency-results from statistical learning theory, meaning that the learning machine obtained from an empirical risk minimization (ERM) attains the lowest achievable risk as sample size grows (Vapnik, 1998;

Original: Sch ¨ olkopf, 2019).
Revised: Sch ¨ olkopf, 2019).

Original: Unfortunately, a considerable amount of research and real-world applications in the past decades has provided a staggering evidence against this assumption (Zhao et al., 2018; 2020; Ren et al., 2019; Taori et al., 2020) (see D’Amour et al.
Revised: Unfortunately, a considerable amount of research and real-world applications in the past decades has provided a staggering evidence against this assumption (Zhao et al., 2018; 2020; Ren et al., 2019; Taori et al., 2020) (see D’Amour et al.

Original: for case studies).
Revised: for case studies).

Original: The violation of the I.I.D. assumption is usually caused by a distribution shift (DS) and can result in inconsistent learning machines (Sugiyama &
Revised: The violation of the I.I.D. assumption is usually caused by a distribution shift (DS) and can result in inconsistent learning machines (Sugiyama &

Original: Kawanabe, 2012), implying the loss of performance guarantee of machine learning models in the real world.
Revised: Kawanabe, 2012), implying the loss of performance guarantee of machine learning models in the real world.

Original: Therefore, to tackle DS, recent work advocates for domain generalization (DG) (Blanchard et al., 2011; Muandet et al., 2013; Li et al., 2017; 2018b; Zhou et al., 2021a).
Revised: Therefore, to tackle DS, recent work advocates for domain generalization (DG) (Blanchard et al., 2011; Muandet et al., 2013; Li et al., 2017; 2018b; Zhou et al., 2021a).

Original: This generalization to utterly unseen domains is crucial for robust deployment of the models in practice, especially when new, unforeseeable domains emerge after model deployment.
Revised: This generalization to utterly unseen domains is crucial for robust deployment of the models in practice, especially when new, unforeseeable domains emerge after model deployment.

Original: However, the most important question that DG seeks to answer is how to identify the right invariance that allows for generalization.
Revised: However, the most important question that DG seeks to answer is how to identify the right invariance that allows for generalization.

Original: The contribution of this work is twofold.
Revised: The contribution of this work is twofold.

Original: First, we advocate that real-world distributions are composed of smaller “units” called invariant elementary distributions that remain invariant across different domains; see Section 2.1.
Revised: First, we advocate that real-world distributions are composed of smaller “units” called invariant elementary distributions that remain invariant across different domains; see Section 2.1.

Original: Second, we propose to implement this hypothesis through so-called gated domain units (GDUs).
Revised: Second, we propose to implement this hypothesis through so-called gated domain units (GDUs).

Original: Specifically, we developed a modular neural network layer that consists of
Revised: Specifically, we developed a modular neural network layer that consists of

Original: GDUs.
Revised: GDUs.

Original: Each GDU learns an embedding of an individual elementary domain that allows us to express the domain similarities during training.
Revised: Each GDU learns an embedding of an individual elementary domain that allows us to express the domain similarities during training.

Original: For this purpose, we adopt the theoretical framework of reproducing kernel Hilbert space (RKHS) to retrieve a geometrical representation of each distribution in the form of a kernel mean embedding (KME) without information loss (Berlinet & Thomas-Agnan, 2004; Smola et al., 2007; Sriperumbudur et al., 2010; Muandet et al., 2017).
Revised: For this purpose, we adopt the theoretical framework of reproducing kernel Hilbert space (RKHS) to retrieve a geometrical representation of each distribution in the form of a kernel mean embedding (KME) without information loss (Berlinet & Thomas-Agnan, 2004; Smola et al., 2007; Sriperumbudur et al., 2010; Muandet et al., 2017).

Original: This representation accommodates methods based on analytical geometry to measure similarities between distributions.
Revised: This representation accommodates methods based on analytical geometry to measure similarities between distributions.

Original: We show that these similarity measures can be learned and utilized to improve the generalization capability of deep learning models to previously unseen domains.
Revised: We show that these similarity measures can be learned and utilized to improve the generalization capability of deep learning models to previously unseen domains.

Original: The remainder of this paper is organized as follows: Our theoretical framework is laid out in Sectionwith our modular DG layer implementation shown in Section 3.
Revised: The remainder of this paper is organized as follows: Our theoretical framework is laid out in Sectionwith our modular DG layer implementation shown in Section 3.

Original: In Section 4, we outline related work.
Revised: In Section 4, we outline related work.

Original: Our experimental evaluations are presented in Section 5.
Revised: Our experimental evaluations are presented in Section 5.

Original: Finally, we discuss potential limitations of our approach and future work in Section 6.
Revised: Finally, we discuss potential limitations of our approach and future work in Section 6.

Original:  D OMAIN G ENERALIZATION WITH I NVARIANT E LEMENTARY D ISTRIBUTIONS
Revised:  D OMAIN G ENERALIZATION WITH I NVARIANT E LEMENTARY D ISTRIBUTIONS

Original: We assume a mixture component shift for the multi-source DG setting.
Revised: We assume a mixture component shift for the multi-source DG setting.

Original: This shift refers to the most common DS stating that the data is made up of different sources, each with its own characteristics, and their proportions vary between the training and test scenario (Quinonero-Candela et al., 2022).
Revised: This shift refers to the most common DS stating that the data is made up of different sources, each with its own characteristics, and their proportions vary between the training and test scenario (Quinonero-Candela et al., 2022).

Original: Our work thus differs in the assumption from related work in DG, in which the central assumption is the covariate shift (i.e., the conditional distribution of the source and test data stays the same)
Revised: Our work thus differs in the assumption from related work in DG, in which the central assumption is the covariate shift (i.e., the conditional distribution of the source and test data stays the same)

Original: (David et al., 2010).
Revised: (David et al., 2010).

Original: In the following, let X and Y be the input and output space, with a joint distribution P .
Revised: In the following, let X and Y be the input and output space, with a joint distribution P .

Original: We are given a set of D labeled source datasets {D si } Di =1 with D si ⊆
Revised: We are given a set of D labeled source datasets {D si } Di =1 with D si ⊆

Original: X × Y .
Revised: X × Y .

Original: Each of the source datasets is assumed to be I.I.D. generated by a joint distribution P si with support on X × Y , henceforth denoted domain .
Revised: Each of the source datasets is assumed to be I.I.D. generated by a joint distribution P si with support on X × Y , henceforth denoted domain .

Original: The set of probability measures with support on X × Y is denoted by P .
Revised: The set of probability measures with support on X × Y is denoted by P .

Original: The multi-source dataset D s comprises the merged individual source datasets {D sj } Dj =1 .
Revised: The multi-source dataset D s comprises the merged individual source datasets {D sj } Dj =1 .

Original: We aim to minimize the empirical risk, see Section 3.3 for details.
Revised: We aim to minimize the empirical risk, see Section 3.3 for details.

Original: Important notation is summarized in Table 1.
Revised: Important notation is summarized in Table 1.

Original: 2. I NVARIANT E LEMENTARY D ISTRIBUTIONS
Revised: 2. I NVARIANT E LEMENTARY D ISTRIBUTIONS

Original: [Table]
Revised: [Table]

Original: V j α sj coefficient for P sj α j
Revised: V j α sj coefficient for P sj α j

Original: [Table]
Revised: [Table]

Original: for
Revised: for

Original: [Table]
Revised: [Table]

Original: β
Revised: β

Original: [Table]
Revised: [Table]

Original: coefficient for
Revised: coefficient for

Original: [Table]
Revised: [Table]

Original: x i and
Revised: x i and

Original: [Table]
Revised: [Table]

Original: Similar to Mansour et al.
Revised: Similar to Mansour et al.

Original: (2009; 2012); Hoffman et al.
Revised: (2009; 2012); Hoffman et al.

Original: (2018a)
Revised: (2018a)

Original: , we assume that the distribution of the source dataset can be described as a convex combination P s = (cid:80)
Revised: , we assume that the distribution of the source dataset can be described as a convex combination P s = (cid:80)

Original: Dj =

Revised: Dj =

Original: α sj P sj where α s =
Revised: α sj P sj where α s =

Original: ( α s 1 , . . . , α sD ) is an element of the probability simplex, i.e., α s ∈ ∆ D := { α ∈ R D | α j ≥ 

Revised: ( α s 1 , . . . , α sD ) is an element of the probability simplex, i.e., α s ∈ ∆ D := { α ∈ R D | α j ≥

Original: ∧ (cid:80) Dj =

Revised: ∧ (cid:80) Dj =

Original: α j = 1 } .
Revised: α j = 1 } .

Original: In other words, α j quantifies the contribution of each individual source domain to the combined source domain.
Revised: In other words, α j quantifies the contribution of each individual source domain to the combined source domain.

Original: In contrast, we generalize their problem descriptions: We express the distribution of each domain as a convex combination of K elementary distributions { P j } Kj =1 ⊂ P , meaning that
Revised: In contrast, we generalize their problem descriptions: We express the distribution of each domain as a convex combination of K elementary distributions { P j } Kj =1 ⊂ P , meaning that

Original: P s = (cid:80) Kj =1 α j P j where α ∈ ∆ K .
Revised: P s = (cid:80) Kj =1 α j P j where α ∈ ∆ K .

Original: Our main assumption is that these elementary distributions remain invariant across the domains .
Revised: Our main assumption is that these elementary distributions remain invariant across the domains .

Original: The advantage is that we can find an invariant subspace at a more elementary level, as opposed to when we consider the source domains as some sort of basis for all unseen domain.
Revised: The advantage is that we can find an invariant subspace at a more elementary level, as opposed to when we consider the source domains as some sort of basis for all unseen domain.

Original: Figure 1 illustrates this idea.
Revised: Figure 1 illustrates this idea.

Original: Theoretically speaking, the I.E.D assumption is appealing because it implies the invariant structure in the solution space, as shown in the following lemma.
Revised: Theoretically speaking, the I.E.D assumption is appealing because it implies the invariant structure in the solution space, as shown in the following lemma.

Original: The proof is given in Appendix A.1.
Revised: The proof is given in Appendix A.1.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let L :
Revised: Let L :

Original: Y × Y → R + be a non-negative loss function, F a hypothesis space of functions f : X → Y , and P s ( X, Y ) a data distribution.
Revised: Y × Y → R + be a non-negative loss function, F a hypothesis space of functions f : X → Y , and P s ( X, Y ) a data distribution.

Original: Suppose that the I.E.D assumption holds, i.e., there exist K elementary distributions P 1 , . .
Revised: Suppose that the I.E.D assumption holds, i.e., there exist K elementary distributions P 1 , . .

Original: , P K such that any data distribution can be expressed as P s ( X, Y ) =
Revised: , P K such that any data distribution can be expressed as P s ( X, Y ) =

Original: (cid:80) Kj =

Revised: (cid:80) Kj =

Original: α j P j
Revised: α j P j

Original: ( X, Y )
Revised: ( X, Y )

Original: for some α ∈ ∆ K .
Revised: for some α ∈ ∆ K .

Original: Then, the corresponding Bayes predictor f ∗ ∈ arg min f
Revised: Then, the corresponding Bayes predictor f ∗ ∈ arg min f

Original: [Equation]
Revised: [Equation]

Original: E (
Revised: E (

Original: [Equation]
Revised: [Equation]

Original: ,Y )
Revised: ,Y )

Original: [Equation]
Revised: [Equation]

Original: P [ L ( Y, f ( X ))] is Pareto-optimal with respect to a vector of elementary risk functionals ( R 1 , . .
Revised: P [ L ( Y, f ( X ))] is Pareto-optimal with respect to a vector of elementary risk functionals ( R 1 , . .

Original: , R K ) :
Revised: , R K ) :

Original: F → R K + where R j ( f ) :
Revised: F → R K + where R j ( f ) :

Original: = E ( X,Y ) ∼ P j [ L ( Y, f ( X ))] .
Revised: = E ( X,Y ) ∼ P j [ L ( Y, f ( X ))] .

Original: Lemma 1 implies that, under the I.E.D assumption, Bayes predictors must belong to a subspace of F called the Pareto set F Pareto ⊂ F which consists of Pareto-optimal models.
Revised: Lemma 1 implies that, under the I.E.D assumption, Bayes predictors must belong to a subspace of F called the Pareto set F Pareto ⊂ F which consists of Pareto-optimal models.

Original: The model f is said to be Pareto-optimal if there exists no g ∈ F such that R j ( g ) ≥ R j ( f ) for all j ∈ { 1 , . . .
Revised: The model f is said to be Pareto-optimal if there exists no g ∈ F such that R j ( g ) ≥ R j ( f ) for all j ∈ { 1 , . . .

Original: , K } with R j ( g ) >
Revised: , K } with R j ( g ) >

Original: R j ( f ) for some j ; see, e.g., Sener & Koltun (2018, Definition 1).
Revised: R j ( f ) for some j ; see, e.g., Sener & Koltun (2018, Definition 1).

Original: In other words, the
Revised: In other words, the

Original: I.E.D assumption allows us to translate the invariance property of data distributions to the solution space.
Revised: I.E.D assumption allows us to translate the invariance property of data distributions to the solution space.

Original: Since Bayes predictors of all future test domains must lie within the Pareto set, which is a
Revised: Since Bayes predictors of all future test domains must lie within the Pareto set, which is a

Original: −   60 .0 .0 .0 .
Revised: −   60 .0 .0 .0 .

Original: − 6 − 4 − 2   80 .0 .0 .0 .
Revised: − 6 − 4 − 2   80 .0 .0 .0 .

Original: V• µ V• µ V
Revised: V• µ V• µ V

Original: Vx i • ϕ
Revised: Vx i • ϕ

Original: ( x i ) β
Revised: ( x i ) β

Original: iβ i
Revised: iβ i

Original: During training
Revised: During training

Original: During inference
Revised: During inference

Original: Hence, the first challenge during butions during strict subset of the original hypothesis space, it is still possible to identify the optimal predictors of future test domains, even without additional data from the test domains, except the I.E.D. assumption itself.
Revised: Hence, the first challenge during butions during strict subset of the original hypothesis space, it is still possible to identify the optimal predictors of future test domains, even without additional data from the test domains, except the I.E.D. assumption itself.

Original: Hence, given data from the training domains, it is sufficient for the purpose of generalization to maintain only solutions within this Pareto set during the training time.
Revised: Hence, given data from the training domains, it is sufficient for the purpose of generalization to maintain only solutions within this Pareto set during the training time.

Original: Unfortunately, neither the elementary distributions nor the weights α are known in practice.
Revised: Unfortunately, neither the elementary distributions nor the weights α are known in practice.

Original: Motivated by this theoretical insight, our DG layer presented in Section 3 is designed to uncover them from a multi-source training dataset D s .
Revised: Motivated by this theoretical insight, our DG layer presented in Section 3 is designed to uncover them from a multi-source training dataset D s .

Original: While Lemma 1 shows the theoretical appeal of the I.E.D. assumption, we discuss below a situation in which it might hold in practice.
Revised: While Lemma 1 shows the theoretical appeal of the I.E.D. assumption, we discuss below a situation in which it might hold in practice.

Original: The limitations will be discussed later in Section 6.
Revised: The limitations will be discussed later in Section 6.

Original: Real-world example.
Revised: Real-world example.

Original: In this work, we postulate that the elementary domain bases are the invariant subspaces that allow us to generalize to unseen domains.
Revised: In this work, we postulate that the elementary domain bases are the invariant subspaces that allow us to generalize to unseen domains.

Original: In practice, the question arises if and when elementary domains evolve.
Revised: In practice, the question arises if and when elementary domains evolve.

Original: Consider that we aim to learn to predict the risk of developing Diabetes from laboratory data from Europe and then infer the risk from data from the United States of America.
Revised: Consider that we aim to learn to predict the risk of developing Diabetes from laboratory data from Europe and then infer the risk from data from the United States of America.

Original: Naturally, factors influencing the data-generating process may change, such as the level of physical activity and nutritional habits.
Revised: Naturally, factors influencing the data-generating process may change, such as the level of physical activity and nutritional habits.

Original: While, to a certain degree, these common factors remain invariant across continents, each of these factors’ contributions may differ.
Revised: While, to a certain degree, these common factors remain invariant across continents, each of these factors’ contributions may differ.

Original: In terms of our assumptions, we model each of these factors with a corresponding elementary distribution P j .
Revised: In terms of our assumptions, we model each of these factors with a corresponding elementary distribution P j .

Original: For a previously unseen individual, we can then determine the coefficients α sj and quantify each factor’s contribution without any information about the individual’s origin.
Revised: For a previously unseen individual, we can then determine the coefficients α sj and quantify each factor’s contribution without any information about the individual’s origin.

Original: 2. K ERNEL M EAN E MBEDDING OF D ISTRIBUTIONS
Revised: 2. K ERNEL M EAN E MBEDDING OF D ISTRIBUTIONS

Original: We leverage the KME of distributions (Berlinet & Thomas-Agnan, 2004; Smola et al., 2007; Muandet et al., 2017) to discover the elementary distributions and evaluate similarities between them.
Revised: We leverage the KME of distributions (Berlinet & Thomas-Agnan, 2004; Smola et al., 2007; Muandet et al., 2017) to discover the elementary distributions and evaluate similarities between them.

Original: Let H be a reproducing kernel Hilbert space (RKHS) of real-valued functions on X with a reproducing kernel k : X × X
Revised: Let H be a reproducing kernel Hilbert space (RKHS) of real-valued functions on X with a reproducing kernel k : X × X

Original: → R (Sch¨olkopf et al., 2001).
Revised: → R (Sch¨olkopf et al., 2001).

Original: The KME of a probability measure P ∈ P in the
Revised: The KME of a probability measure P ∈ P in the

Original: RKHS H is defined by a mapping ϕ ( P ) = µ P :=
Revised: RKHS H is defined by a mapping ϕ ( P ) = µ P :=

Original: (cid:82) X k
Revised: (cid:82) X k

Original: ( x , · ) d P ( x ) .
Revised: ( x , · ) d P ( x ) .

Original: We assume that the kernel k is characteristic, i.e., the mapping µ P is injective (Fukumizu et al., 2004; Sriperumbudur et al., 2008).
Revised: We assume that the kernel k is characteristic, i.e., the mapping µ P is injective (Fukumizu et al., 2004; Sriperumbudur et al., 2008).

Original: Theoretically, this essential assumption ensures that there is no information loss when mapping the distribution into H .
Revised: Theoretically, this essential assumption ensures that there is no information loss when mapping the distribution into H .

Original: Given the samples { x 1 , . . .
Revised: Given the samples { x 1 , . . .

Original: , x n } generated I.I.D. from P , µ P can be approximated by the empirical KME ˆ µ P =
Revised: , x n } generated I.I.D. from P , µ P can be approximated by the empirical KME ˆ µ P =

Original: (1 /n )
Revised: (1 /n )

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =1 k ( x i , · ) =
Revised: =1 k ( x i , · ) =

Original: (1 /n ) (cid:80) ni
Revised: (1 /n ) (cid:80) ni

Original: =1 ϕ ( x i ) .
Revised: =1 ϕ ( x i ) .

Original: We refer non-expert readers to Muandet et al.
Revised: We refer non-expert readers to Muandet et al.

Original: (2017) for a thorough review on this topic.
Revised: (2017) for a thorough review on this topic.

Original: Challenges.
Revised: Challenges.

Original: Figure 1 depicts two challenges that come with our assumption of elementary distributions.
Revised: Figure 1 depicts two challenges that come with our assumption of elementary distributions.

Original: First, since we do not have access to the samples from the hidden elementary distributions, the elementary KME cannot be estimated directly from the samples at hand.
Revised: First, since we do not have access to the samples from the hidden elementary distributions, the elementary KME cannot be estimated directly from the samples at hand.

Original: To overcome this challenge, we instead seek a proxy KME µ V j := (1 /N )
Revised: To overcome this challenge, we instead seek a proxy KME µ V j := (1 /N )

Original: (cid:80) Nk
Revised: (cid:80) Nk

Original: =1 ϕ ( v jk ) =
Revised: =1 ϕ ( v jk ) =

Original: (1 /N )
Revised: (1 /N )

Original: (cid:80) Nk
Revised: (cid:80) Nk

Original: =1 k ( v jk , · ) for each elementary KME µ P j from a domain basis V j , where V j = { v j 1 , . . .
Revised: =1 k ( v jk , · ) for each elementary KME µ P j from a domain basis V j , where V j = { v j 1 , . . .

Original: , v jN }
Revised: , v jN }

Original: ⊆ X for all j ∈ { 1 , . . .
Revised: ⊆ X for all j ∈ { 1 , . . .

Original: , M } .
Revised: , M } .

Original: Hence, the KME µ V j can be interpreted as the KME of the empirical probability measure ˆ P V
Revised: Hence, the KME µ V j can be interpreted as the KME of the empirical probability measure ˆ P V

Original: j = (1 /N )
Revised: j = (1 /N )

Original: (cid:80) Nk
Revised: (cid:80) Nk

Original: =

Revised: =

Original: δ v jk .
Revised: δ v jk .

Original: Here, we assume that M = K .
Revised: Here, we assume that M = K .

Original: The sets V j are referred to as elementary domain basis .
Revised: The sets V j are referred to as elementary domain basis .

Original: Intuitively, the elementary domain basis V 1 , . . .
Revised: Intuitively, the elementary domain basis V 1 , . . .

Original: , V M represents each elementary distribution by a set of vectors that mimic samples generated from the corresponding distribution.
Revised: , V M represents each elementary distribution by a set of vectors that mimic samples generated from the corresponding distribution.

Original: In Figure 1, V 1 and V 2 as well as their mapping in H visualize this first challenge.
Revised: In Figure 1, V 1 and V 2 as well as their mapping in H visualize this first challenge.

Original: The second challenge is the objective of learning the unknown similarity between a single sample x i and an elementary domain V j , which we denote by β ij .
Revised: The second challenge is the objective of learning the unknown similarity between a single sample x i and an elementary domain V j , which we denote by β ij .

Original: Considering the advantage of KMEs, that is to tackle this challenge from a geometrical viewpoint, we quantify similarities between KMEs.
Revised: Considering the advantage of KMEs, that is to tackle this challenge from a geometrical viewpoint, we quantify similarities between KMEs.

Original: For example, in Figure 1, the similarity between ϕ ( x i ) and µ V 1 ( β i 1 ) and µ V 2 ( β i 2 ) could be quantified as their distance or angle.
Revised: For example, in Figure 1, the similarity between ϕ ( x i ) and µ V 1 ( β i 1 ) and µ V 2 ( β i 2 ) could be quantified as their distance or angle.

Original: These similarity coefficients enable our Domain Generalization Layer to represent a convex combination of elementary domain-specific learning machines, commonly known as ensembles.
Revised: These similarity coefficients enable our Domain Generalization Layer to represent a convex combination of elementary domain-specific learning machines, commonly known as ensembles.

Original: We introduce our layer in the following Section 3.
Revised: We introduce our layer in the following Section 3.

Original:  D OMAIN G ENERALIZATION L AYER
Revised:  D OMAIN G ENERALIZATION L AYER

Original: This section aims to transfer the theoretical ideas presented in Section 2 into a deep learning framework.
Revised: This section aims to transfer the theoretical ideas presented in Section 2 into a deep learning framework.

Original: For the purpose of implementation, let x ∈ R h × w denote the input data point and h ξ : R h × w → R e the feature extractor (FE) that maps the input into a low-dimensional representation ˜ x ∈ R e .
Revised: For the purpose of implementation, let x ∈ R h × w denote the input data point and h ξ : R h × w → R e the feature extractor (FE) that maps the input into a low-dimensional representation ˜ x ∈ R e .

Original: Then the prediction layer g θ : R e
Revised: Then the prediction layer g θ : R e

Original: → Y infers the label y .
Revised: → Y infers the label y .

Original: To tackle the DG problem, we introduce a layer module called the gated domain unit (GDU).
Revised: To tackle the DG problem, we introduce a layer module called the gated domain unit (GDU).

Original: A GDU consists of three main components: (1) a similarity function γ : H × H
Revised: A GDU consists of three main components: (1) a similarity function γ : H × H

Original: → R that is the same for all elementary domains, (2) an elementary basis V j and (3) a learning machine f
Revised: → R that is the same for all elementary domains, (2) an elementary basis V j and (3) a learning machine f

Original: (˜ x, θ j ) for each elementary domain j ∈ { 1 , . . .
Revised: (˜ x, θ j ) for each elementary domain j ∈ { 1 , . . .

Original: , M } .
Revised: , M } .

Original: The architecture of the layer proposed herein is depicted in Figure 2.
Revised: The architecture of the layer proposed herein is depicted in Figure 2.

Original: [Figure]
Revised: [Figure]

Original: The DG layer consists of several GDUs that represent the elementary distributions.
Revised: The DG layer consists of several GDUs that represent the elementary distributions.

Original: During training, these GDUs learn the elementary domain bases V 1 , . .
Revised: During training, these GDUs learn the elementary domain bases V 1 , . .

Original: , V M that approximate these distributions.
Revised: , V M that approximate these distributions.

Original: Essentially, the process is as follows:
Revised: Essentially, the process is as follows:

Original: First, the j -th GDU takes ˜ x i as an input and yields β ij as an output.
Revised: First, the j -th GDU takes ˜ x i as an input and yields β ij as an output.

Original: The KME of each domain basis V j is required in order to apply γ to compute similarity between ˜ x i and V j .
Revised: The KME of each domain basis V j is required in order to apply γ to compute similarity between ˜ x i and V j .

Original: These KMEs are obtained by ϕ ( V j ) :
Revised: These KMEs are obtained by ϕ ( V j ) :

Original: = µ V j = (1 /N )
Revised: = µ V j = (1 /N )

Original: (cid:80) Nk
Revised: (cid:80) Nk

Original: =1 ϕ ( v jk ) =
Revised: =1 ϕ ( v jk ) =

Original: (1 /N )
Revised: (1 /N )

Original: (cid:80) N k
Revised: (cid:80) N k

Original: =1 k ( v j k , · ) .
Revised: =1 k ( v j k , · ) .

Original: The GDU, therefore, has the task to allocate coefficients β ij for each elementary domain based on a similarity function γ .
Revised: The GDU, therefore, has the task to allocate coefficients β ij for each elementary domain based on a similarity function γ .

Original: The function γ outputs the β ij = γ ( ϕ (˜ x i ) , µ V j ) coefficients that in turn represent similarities between the KME of both, the corresponding domain basis V j and the input ˜ x i .
Revised: The function γ outputs the β ij = γ ( ϕ (˜ x i ) , µ V j ) coefficients that in turn represent similarities between the KME of both, the corresponding domain basis V j and the input ˜ x i .

Original: Theoretically speaking, µ V j and the feature mapping ϕ
Revised: Theoretically speaking, µ V j and the feature mapping ϕ

Original: (˜ x i ) are elements of the associated RKHS H , which allow us to evaluate similarities of non-linear features in a higher dimensional feature space.
Revised: (˜ x i ) are elements of the associated RKHS H , which allow us to evaluate similarities of non-linear features in a higher dimensional feature space.

Original: Each GDU is then connected to a learning machine f
Revised: Each GDU is then connected to a learning machine f

Original: (˜ x i , θ j ) that yields an elementary domain-specific inference.
Revised: (˜ x i , θ j ) that yields an elementary domain-specific inference.

Original: The final prediction of the layer is then an ensemble of these learning machines g θ (˜ x i ) =
Revised: The final prediction of the layer is then an ensemble of these learning machines g θ (˜ x i ) =

Original: (cid:80) Mj
Revised: (cid:80) Mj

Original: =1 β ij f (˜ x i , θ j ) where θ =
Revised: =1 β ij f (˜ x i , θ j ) where θ =

Original: ( θ 1 , . .
Revised: ( θ 1 , . .

Original: , θ M ) .
Revised: , θ M ) .

Original: In Figure 2, we give an overview of how data is processed and information is stored in the GDU.
Revised: In Figure 2, we give an overview of how data is processed and information is stored in the GDU.

Original: In summary, GDUs leverage the invariant elementary distribution (I.E.D.) assumption and represent our algorithmic contribution: The elementary domain bases are stored as weights in the layer.
Revised: In summary, GDUs leverage the invariant elementary distribution (I.E.D.) assumption and represent our algorithmic contribution: The elementary domain bases are stored as weights in the layer.

Original: Storing information as a weight matrix (i.e., domain memory) allows to learn the elementary domain bases efficiently using backpropagation.
Revised: Storing information as a weight matrix (i.e., domain memory) allows to learn the elementary domain bases efficiently using backpropagation.

Original: Hence, we avoid the dependency on problem-adaptive methods (e.g., domain-adversarial training) and domain information (e.g., domain labels).
Revised: Hence, we avoid the dependency on problem-adaptive methods (e.g., domain-adversarial training) and domain information (e.g., domain labels).

Original: 3. D OMAIN S IMILARITY M EASURES
Revised: 3. D OMAIN S IMILARITY M EASURES

Original: For the similarity function γ , we consider two similarity measures H ( ϕ (˜ x ) , µ V j ) , namely the cosine similarity (CS) (Kim et al., 2019) and maximum mean discrepancy (MMD) (Borgwardt et al., 2006;
Revised: For the similarity function γ , we consider two similarity measures H ( ϕ (˜ x ) , µ V j ) , namely the cosine similarity (CS) (Kim et al., 2019) and maximum mean discrepancy (MMD) (Borgwardt et al., 2006;

Original: Gretton et al., 2012).
Revised: Gretton et al., 2012).

Original: To ensure that the resulting coefficients β i lie on the probability simplex, we apply the kernel softmax function (Gao et al., 2019) and interpret its output as the similarity between an observation ˜ x and an elementary domain basis V i .
Revised: To ensure that the resulting coefficients β i lie on the probability simplex, we apply the kernel softmax function (Gao et al., 2019) and interpret its output as the similarity between an observation ˜ x and an elementary domain basis V i .

Original: We get
Revised: We get

Original: [Equation]
Revised: [Equation]

Original: (1) where κ > 0 is a positive softness parameter for the kernel softmax.
Revised: (1) where κ > 0 is a positive softness parameter for the kernel softmax.

Original: Geometrically speaking, these similarities correspond to the angle and distance of two KMEs in the RKHS H .
Revised: Geometrically speaking, these similarities correspond to the angle and distance of two KMEs in the RKHS H .

Original: The function ϕ maps the observation ˜ x and domain basis V j into H meaning that ϕ (˜ x ) = µ δ ˜ x
Revised: The function ϕ maps the observation ˜ x and domain basis V j into H meaning that ϕ (˜ x ) = µ δ ˜ x

Original: = k (˜ x, · ) is the KME of a
Revised: = k (˜ x, · ) is the KME of a

Original: Dirac measure δ
Revised: Dirac measure δ

Original: ˜ x and ϕ ( V j ) = µ V j = (1 /N )
Revised: ˜ x and ϕ ( V j ) = µ V j = (1 /N )

Original: (cid:80) Nk
Revised: (cid:80) Nk

Original: =1 k ( v jk , · ) .
Revised: =1 k ( v jk , · ) .

Original: CS.
Revised: CS.

Original: The CS function H ( ϕ (˜ x i ) , µ V j ) = ⟨ ϕ (˜ x i ) ,µ Vj ⟩ H ∥ ϕ
Revised: The CS function H ( ϕ (˜ x i ) , µ V j ) = ⟨ ϕ (˜ x i ) ,µ Vj ⟩ H ∥ ϕ

Original: (˜ x i ) ∥
Revised: (˜ x i ) ∥

Original: H ∥ µ Vj ∥
Revised: H ∥ µ Vj ∥

Original: H is used as an angle-based similarity.
Revised: H is used as an angle-based similarity.

Original: MMD.
Revised: MMD.

Original: We consider the MMD for calculating a distance-based similarity measure.
Revised: We consider the MMD for calculating a distance-based similarity measure.

Original: The distance is then given as ∥ ϕ (˜ x i ) − µ V j ∥ H .
Revised: The distance is then given as ∥ ϕ (˜ x i ) − µ V j ∥ H .

Original: Subsequently, the similarity function H is the negative MMD:
Revised: Subsequently, the similarity function H is the negative MMD:

Original: H ( ϕ (˜ x i ) , µ V j ) = −∥ ϕ (˜ x i ) − µ V j ∥ H .
Revised: H ( ϕ (˜ x i ) , µ V j ) = −∥ ϕ (˜ x i ) − µ V j ∥ H .

Original: The intuition behind the negative MMD is to put higher weights on samples that are closer to the KME of an elementary domain basis.
Revised: The intuition behind the negative MMD is to put higher weights on samples that are closer to the KME of an elementary domain basis.

Original: 3. P ROJECTION - BASED G ENERALIZATION
Revised: 3. P ROJECTION - BASED G ENERALIZATION

Original: For classification tasks, we introduce an alternative approach to infer the β i coefficients that is based on the idea of kernel sparse coding (Gao et al., 2010; 2013).
Revised: For classification tasks, we introduce an alternative approach to infer the β i coefficients that is based on the idea of kernel sparse coding (Gao et al., 2010; 2013).

Original: Herein the goal is to find an approximated representation of each feature mapping ϕ
Revised: Herein the goal is to find an approximated representation of each feature mapping ϕ

Original: (˜ x i ) using the elements of a dictionary { µ V j } Mj =1 .
Revised: (˜ x i ) using the elements of a dictionary { µ V j } Mj =1 .

Original: This approach allows us to approximate the feature mapping with these elements by ϕ (˜ x i ) ≈ (cid:80)
Revised: This approach allows us to approximate the feature mapping with these elements by ϕ (˜ x i ) ≈ (cid:80)

Original: Mj
Revised: Mj

Original: =1 β ij µ V j .
Revised: =1 β ij µ V j .

Original: In contrast to the aforementioned approaches, an elementary domain KME µ V j does not necessarily represent the KME of an elementary distribution µ P j .
Revised: In contrast to the aforementioned approaches, an elementary domain KME µ V j does not necessarily represent the KME of an elementary distribution µ P j .

Original: Therefore, we present another approach that aims to find a set { µ V j } Mj =1 that permits µ P s to be represented as a linear combination.
Revised: Therefore, we present another approach that aims to find a set { µ V j } Mj =1 that permits µ P s to be represented as a linear combination.

Original: Since P is assumed to be a convex combination of elementary distributions, we can find a linear combination to represent µ P s by the domain KMEs µ V j , as long as µ P s ∈ H M :=
Revised: Since P is assumed to be a convex combination of elementary distributions, we can find a linear combination to represent µ P s by the domain KMEs µ V j , as long as µ P s ∈ H M :=

Original: span { µ V j | j = 1 , . . .
Revised: span { µ V j | j = 1 , . . .

Original: , M } .
Revised: , M } .

Original: The RKHS H M is a subspace of the actual RKHS H , which allows us to represent elements of H at least approximately in the subspace H M .
Revised: The RKHS H M is a subspace of the actual RKHS H , which allows us to represent elements of H at least approximately in the subspace H M .

Original: By keeping the H M large, we gain more representative power.
Revised: By keeping the H M large, we gain more representative power.

Original: To make H M as large as possible, we have to ensure its spanning elements are linearly independent or, even better, orthogonal.
Revised: To make H M as large as possible, we have to ensure its spanning elements are linearly independent or, even better, orthogonal.

Original: Orthogonal KMEs ensure two desirable properties.
Revised: Orthogonal KMEs ensure two desirable properties.

Original: First, pairwise orthogonal elements in H M guarantee no redundancy.
Revised: First, pairwise orthogonal elements in H M guarantee no redundancy.

Original: Second, having orthogonal elements allows us to make use of the orthogonal projection.
Revised: Second, having orthogonal elements allows us to make use of the orthogonal projection.

Original: This projection geometrically yields the best approximation of ϕ (˜ x ) in H M .
Revised: This projection geometrically yields the best approximation of ϕ (˜ x ) in H M .

Original: In other words, we can achieve the best possible approximation of the feature mapping by using its orthogonal components (see Proposition 3.1).
Revised: In other words, we can achieve the best possible approximation of the feature mapping by using its orthogonal components (see Proposition 3.1).

Original: The orthogonal projection is given by
Revised: The orthogonal projection is given by

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For a KME µ P of a given mixture distribution P the following holds µ P ∈ span { µ V j | V j , ∀ j = 1 , . . .
Revised: For a KME µ P of a given mixture distribution P the following holds µ P ∈ span { µ V j | V j , ∀ j = 1 , . . .

Original: , M } , where ⟨ µ V i , µ V j ⟩ H = 0 , ∀ i ̸ =
Revised: , M } , where ⟨ µ V i , µ V j ⟩ H = 0 , ∀ i ̸ =

Original: j (i.e., the KME of the elementary domains basis are pairwise orthogonal).
Revised: j (i.e., the KME of the elementary domains basis are pairwise orthogonal).

Original: The value of the function (cid:80) Mj =1 ∥ µ P − β j µ V j ∥ 2 H k is minimal if the coefficients are set as β ∗ j = ⟨ µ P , µ V j ⟩ H / ∥ µ V j ∥ 2 H .
Revised: The value of the function (cid:80) Mj =1 ∥ µ P − β j µ V j ∥ 2 H k is minimal if the coefficients are set as β ∗ j = ⟨ µ P , µ V j ⟩ H / ∥ µ V j ∥ 2 H .

Original: The Proposition 3.1 can be used to give an approximation of µ P by projecting it into H M , i.e., µ P ≈ (cid:80)
Revised: The Proposition 3.1 can be used to give an approximation of µ P by projecting it into H M , i.e., µ P ≈ (cid:80)

Original: Mj
Revised: Mj

Original: =

Revised: =

Original: β j µ V j , where β j = ⟨ µ P , µ V j ⟩ H / ∥ µ V j ∥ 2 H .
Revised: β j µ V j , where β j = ⟨ µ P , µ V j ⟩ H / ∥ µ V j ∥ 2 H .

Original: This best approximation property is the main advantage of our assumption in Proposition 3.1 (i.e., having orthogonal KME) and thus a potential advantage of projection-based DG.
Revised: This best approximation property is the main advantage of our assumption in Proposition 3.1 (i.e., having orthogonal KME) and thus a potential advantage of projection-based DG.

Original: Appendix A.2 provides the proof of Proposition 3.1.
Revised: Appendix A.2 provides the proof of Proposition 3.1.

Original: 3. M ODEL T RAINING
Revised: 3. M ODEL T RAINING

Original: For model training, we adapt the domain adaptation (DA) framework from Zhuang et al.
Revised: For model training, we adapt the domain adaptation (DA) framework from Zhuang et al.

Original: Thus, our learning objective function is formalized as L ( g ) + λ D Ω D ( ∥ g ∥ H ) .
Revised: Thus, our learning objective function is formalized as L ( g ) + λ D Ω D ( ∥ g ∥ H ) .

Original: The goal of the training can be described in terms of the two components of this function.
Revised: The goal of the training can be described in terms of the two components of this function.

Original: Consider a batch of training data { x 1 , . .
Revised: Consider a batch of training data { x 1 , . .

Original: , x b } , where b is the batch size.
Revised: , x b } , where b is the batch size.

Original: During training, we minimize the loss function
Revised: During training, we minimize the loss function

Original: L ( g ) = 1 b (cid:80) b i =

Revised: L ( g ) = 1 b (cid:80) b i =

Original: L
Revised: L

Original: (ˆ y i , y i ) = 1 b (cid:80) b
Revised: (ˆ y i , y i ) = 1 b (cid:80) b

Original: i =1 L ( (cid:80) M j =1 γ ( ϕ (˜ x i ) , µ V j ) f j (˜ x i ) , y i ) for an underlying task and the respective batch size.
Revised: i =1 L ( (cid:80) M j =1 γ ( ϕ (˜ x i ) , µ V j ) f j (˜ x i ) , y i ) for an underlying task and the respective batch size.

Original: In addition, our objective is that the model learns to distinguish between different domains.
Revised: In addition, our objective is that the model learns to distinguish between different domains.

Original: Thus, the regularization Ω D is introduced to control the domain basis.
Revised: Thus, the regularization Ω D is introduced to control the domain basis.

Original: In our case, we require the regularization Ω D to ensure that the KMEs of the elementary domain basis are able to represent the KMEs of the elementary domains.
Revised: In our case, we require the regularization Ω D to ensure that the KMEs of the elementary domain basis are able to represent the KMEs of the elementary domains.

Original: Therefore, we minimize the
Revised: Therefore, we minimize the

Original: MMD between the feature mappings ϕ (˜ x i ) and the associated representation (cid:80) Mj
Revised: MMD between the feature mappings ϕ (˜ x i ) and the associated representation (cid:80) Mj

Original: =1 β ij µ V j .
Revised: =1 β ij µ V j .

Original: Note that β ij = γ
Revised: Note that β ij = γ

Original: ( ϕ (˜ x i ) , µ V j ) .
Revised: ( ϕ (˜ x i ) , µ V j ) .

Original: Hence, the regularization Ω D = Ω OLSD is defined as Ω OLSD (cid:0) ∥ g ∥ H (cid:1)
Revised: Hence, the regularization Ω D = Ω OLSD is defined as Ω OLSD (cid:0) ∥ g ∥ H (cid:1)

Original: = 1 b (cid:80) bi =1 ∥
Revised: = 1 b (cid:80) bi =1 ∥

Original: ϕ
Revised: ϕ

Original: (˜ x i ) − (cid:80)
Revised: (˜ x i ) − (cid:80)

Original: Mj
Revised: Mj

Original: =1 β ij µ V j ∥ 2 H (see Appendix B.1 for details).
Revised: =1 β ij µ V j ∥ 2 H (see Appendix B.2 for details).

Original: The intuition is the objective to represent each feature mapping ϕ
Revised: The intuition is the objective to represent each feature mapping ϕ

Original: (˜ x i ) by the domain KMEs µ V j .
Revised: (˜ x i ) by the domain KMEs µ V j .

Original: Thus, we try to minimize the MMD between the feature map and a combination of µ V j .
Revised: Thus, we try to minimize the MMD between the feature map and a combination of µ V j .

Original: The minimum of the stated regularization can be interpreted as the ordinary least square-solution of a regression-problem of ϕ (˜ x i ) by the components of H M .
Revised: The minimum of the stated regularization can be interpreted as the ordinary least square-solution of a regression-problem of ϕ (˜ x i ) by the components of H M .

Original: In other words, we want to ensure that the basis V j is contained in feature mappings ϕ
Revised: In other words, we want to ensure that the basis V j is contained in feature mappings ϕ

Original: (˜ x i ) .
Revised: (˜ x i ) .

Original: In the particular case of projection, we want the KME of the elementary domain to be orthogonal to ensure high expressive power.
Revised: In the particular case of projection, we want the KME of the elementary domain to be orthogonal to ensure high expressive power.

Original: For this purpose, the additional term Ω ⊥ D will be introduced to ensure the desired orthogonality.
Revised: For this purpose, the additional term Ω ⊥ D will be introduced to ensure the desired orthogonality.

Original: Considering a kernel function with k ( x, x ) = 1 , orthogonality would require the Gram matrix K ij = ⟨ µ V i , µ V j ⟩ H to be close to the identity matrix I .
Revised: Considering a kernel function with k ( x, x ) = 1 , orthogonality would require the Gram matrix K ij = ⟨ µ V i , µ V j ⟩ H to be close to the identity matrix I .

Original: There are a variety of methods for regularizing matrices available (Xie et al., 2017; Bansal et al., 2018).
Revised: There are a variety of methods for regularizing matrices available (Xie et al., 2017; Bansal et al., 2018).

Original: A well-known method to ensure orthogonality is the soft orthogonality (SO) regularization Ω ⊥ D = λ
Revised: A well-known method to ensure orthogonality is the soft orthogonality (SO) regularization Ω ⊥ D = λ

Original: ∥ K − I ∥ 2 F (Bansal et al., 2018).
Revised: ∥ K − I ∥ 2 F (Bansal et al., 2018).

Original: As pointed out by Bansal et al.
Revised: As pointed out by Bansal et al.

Original: (2018), the spectral restricted isometry property (SRIP) and mutual coherence (MC) regularization can be a promising alternative for SO and thus are additionally implemented in the DG layer.
Revised: (2018), the spectral restricted isometry property (SRIP) and mutual coherence (MC) regularization can be a promising alternative for SO and thus are additionally implemented in the DG layer.

Original: Hence, in the case of projection, the regularization is given by Ω
Revised: Hence, in the case of projection, the regularization is given by Ω

Original: [Equation]
Revised: [Equation]

Original: Lastly, sparse coding is an efficient technique to find the least possible basis to recover the data subject to a reconstruction error (Olshausen & Field, 1997).
Revised: Lastly, sparse coding is an efficient technique to find the least possible basis to recover the data subject to a reconstruction error (Olshausen & Field, 1997).

Original: Several such applications yield strong performances, for example in the field of computer vision (Lee et al., 2007; Yang et al., 2009).
Revised: Several such applications yield strong performances, for example in the field of computer vision (Lee et al., 2007; Yang et al., 2009).

Original: Kernel sparse coding transfers the reconstruction problem of sparse coding into H by using the mapping ϕ , and, by applying a kernel function, the reconstruction error is quantified as the inner product (Gao et al., 2010; 2013).
Revised: Kernel sparse coding transfers the reconstruction problem of sparse coding into H by using the mapping ϕ , and, by applying a kernel function, the reconstruction error is quantified as the inner product (Gao et al., 2010; 2013).

Original: To ensure sparsity, we apply the L 1 -norm on the coefficients β and add Ω L 1 D ( ∥ γ ∥ ) :
Revised: To ensure sparsity, we apply the L 1 -norm on the coefficients β and add Ω L 1 D ( ∥ γ ∥ ) :

Original: = ∥ γ ( ϕ (˜ x i ) , µ V j ) ∥ 1 to the regularization term Ω D with the corresponding coefficient λ L 1 .
Revised: = ∥ γ ( ϕ (˜ x i ) , µ V j ) ∥ 1 to the regularization term Ω D with the corresponding coefficient λ L 1 .

Original: Appendix B.2 gives a visual overview of the model training.
Revised: Appendix B.3 gives a visual overview of the model training.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: DG, also known as out-of-distribution (OOD) generalization, is among the hardest problems in machine learning (Blanchard et al., 2011; Muandet et al., 2013; Arjovsky et al., 2019).
Revised: DG, also known as out-of-distribution (OOD) generalization, is among the hardest problems in machine learning (Blanchard et al., 2011; Muandet et al., 2013; Arjovsky et al., 2019).

Original: In contrast,
Revised: In contrast,

Original: DA, which predates DG and OOD problems, deals with a slightly simpler scenario in which some data from the test distribution are available (Ganin et al., 2015).
Revised: DA, which predates DG and OOD problems, deals with a slightly simpler scenario in which some data from the test distribution are available (Ganin et al., 2015).

Original: Hence, based on the available data, the task is to develop learning machines that transfer knowledge learned in a source domain specifically to the target domain.
Revised: Hence, based on the available data, the task is to develop learning machines that transfer knowledge learned in a source domain specifically to the target domain.

Original: Approaches pursued in DA can be grouped primarily into (1) discrepancy-based
Revised: Approaches pursued in DA can be grouped primarily into (1) discrepancy-based

Original: DA (Sun et al., 2016; Peng & Saenko, 2018; Ben-David et al., 2010; Fang et al., 2020; Tzeng et al., 2014; Long et al., 2015; Baktashmotlagh et al., 2016) (2) adversary-based DA (Tzeng et al., 2017; Liu & Tuzel, 2016; Ganin et al., 2015; Long et al., 2018), and (3) reconstruction-based DA (Bousmalis et al., 2016; Hoffman et al., 2018b; Kim et al., 2017; Yi et al., 2017; Zhu et al., 2017; Ghifary et al., 2014).
Revised: DA (Sun et al., 2016; Peng & Saenko, 2018; Ben-David et al., 2010; Fang et al., 2020; Tzeng et al., 2014; Long et al., 2015; Baktashmotlagh et al., 2016) (2) adversary-based DA (Tzeng et al., 2017; Liu & Tuzel, 2016; Ganin et al., 2015; Long et al., 2018), and (3) reconstruction-based DA (Bousmalis et al., 2016; Hoffman et al., 2018b; Kim et al., 2017; Yi et al., 2017; Zhu et al., 2017; Ghifary et al., 2014).

Original: In DA, learning the domain-invariant components requires access to unlabeled data from the target domain.
Revised: In DA, learning the domain-invariant components requires access to unlabeled data from the target domain.

Original: Unlike problems in DA, where the observed data from the test domains can be used to find the most appropriate invariant structures (Ben-David et al., 2010), the lack thereof in DG calls for a postulation of invariant structure that will enable the OOD generalization.
Revised: Unlike problems in DA, where the observed data from the test domains can be used to find the most appropriate invariant structures (Ben-David et al., 2010), the lack thereof in DG calls for a postulation of invariant structure that will enable the OOD generalization.

Original: To enable generalization to unseen domains without any access to data from them, researchers have made significant progress in the past decade and developed a broad spectrum of methodologies (Zhou et al., 2021a;c; Li et al., 2019; Blanchard et al., 2011).
Revised: To enable generalization to unseen domains without any access to data from them, researchers have made significant progress in the past decade and developed a broad spectrum of methodologies (Zhou et al., 2021a;c; Li et al., 2019; Blanchard et al., 2011).

Original: For thorough review see, e.g., Zhou et al.
Revised: For thorough review see, e.g., Zhou et al.

Original: (2021a); Wang et al.
Revised: (2021a); Wang et al.

Original: Existing works can be categorized into methods based on domaininvariant representation learning (Muandet et al., 2013; Li et al., 2018b;d), meta-learning (Li et al., 2018a;
Revised: Existing works can be categorized into methods based on domaininvariant representation learning (Muandet et al., 2013; Li et al., 2018b;d), meta-learning (Li et al., 2018a;

Original: Balaji et al., 2018), data augmentation (Zhou et al., 2020), to name a few.
Revised: Balaji et al., 2018), data augmentation (Zhou et al., 2020), to name a few.

Original: Another recent stream of research from a causal perspective includes invariant risk minimization (Arjovsky et al., 2019), invariant causal prediction (Peters et al., 2016), and causal representation learning (Sch¨olkopf et al., 2021).
Revised: Another recent stream of research from a causal perspective includes invariant risk minimization (Arjovsky et al., 2019), invariant causal prediction (Peters et al., 2016), and causal representation learning (Sch¨olkopf et al., 2021).

Original: The overall motivation here is to learn the representation that is robust to domain-specific spurious correlations.
Revised: The overall motivation here is to learn the representation that is robust to domain-specific spurious correlations.

Original: In other words, it is postulated that “causal” features are the right kind of invariance that will enable OOD generalization.
Revised: In other words, it is postulated that “causal” features are the right kind of invariance that will enable OOD generalization.

Original: Despite the successful applications, DG remains a challenging research gap.
Revised: Despite the successful applications, DG remains a challenging research gap.

Original: We differentiate our work from existing ones as follow.
Revised: We differentiate our work from existing ones as follow.

Original: First, we postulate the existence of domaininvariant structure at the distributional level rather than at the data representation, which is a common assumption in DG.
Revised: First, we postulate the existence of domaininvariant structure at the distributional level rather than at the data representation, which is a common assumption in DG.

Original: This is motivated by theoretical results (Mansour et al., 2009; Hoffman et al., 2018a), stating that a distribution-weighted combination of source hypotheses represents the ideal hypothesis.
Revised: This is motivated by theoretical results (Mansour et al., 2009; Hoffman et al., 2018a), stating that a distribution-weighted combination of source hypotheses represents the ideal hypothesis.

Original: Furthermore, our distributional assumption, as we argued in Section 2, generalizes previous work that proposes to use domain-specific knowledge to tackle the problem of DG from a more elementary setting.
Revised: Furthermore, our distributional assumption, as we argued in Section 2, generalizes previous work that proposes to use domain-specific knowledge to tackle the problem of DG from a more elementary setting.

Original: For example, approaches such as Piratla et al.
Revised: For example, approaches such as Piratla et al.

Original: ; Monteiro et al.
Revised: ; Monteiro et al.

Original: (2021) can be compared to our GDUs as domain-specific predictors, in the special case, where each elementary domain represents a single source domain.
Revised: (2021) can be compared to our GDUs as domain-specific predictors, in the special case, where each elementary domain represents a single source domain.

Original: However, GDUs do not assume the existence of a single common classifier for all the domains, providing a combination of multiple common classifiers shared between different source domains.
Revised: However, GDUs do not assume the existence of a single common classifier for all the domains, providing a combination of multiple common classifiers shared between different source domains.

Original: Second, we incorporate the I.E.D. assumption directly into our model’s architecture, as shown in
Revised: Second, we incorporate the I.E.D. assumption directly into our model’s architecture, as shown in

Original: Figure 2.
Revised: Figure 2.

Original: Designing effective architectures for DG has been largely neglected (Zhou et al., 2020, Sec.
Revised: Designing effective architectures for DG has been largely neglected (Zhou et al., 2020, Sec.

Original: Last, we do not assume access to domain information.
Revised: Last, we do not assume access to domain information.

Original: Although obtaining such information can be difficult in practice,see our short discussion in Appendix C.4 (Niu et al., 2017), DG methods that can deal with their absence (e.g., Huang et al.
Revised: Although obtaining such information can be difficult in practice, see our short discussion in Appendix C.4 (Niu et al., 2017), DG methods that can deal with their absence (e.g., Huang et al.

Original: ; Carlucci et al.
Revised: ; Carlucci et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2018c)) are yet scarce (Zhou et al., 2020, Sec. 4.2).
Revised: (2018c)) are yet scarce (Zhou et al., 2020, Sec. 4.2).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Since ERM is one of the strongest baselines in DG (Gulrajani & Lopez-Paz, 2020; Koh et al., 2021), we, first, compare our approach compared to ERM and ensemble learning (Table 2 and Appendix
Revised: Since ERM is one of the strongest baselines in DG (Gulrajani & Lopez-Paz, 2020; Koh et al., 2021), we, first, compare our approach compared to ERM and ensemble learning (Table 2 and Appendix

Original: C.1).
Revised: C.1).

Original: Second, we benchmark our approach to state-of-the-art DG (e.g., CORAL, LISA, IRM, FISH,
Revised: Second, we benchmark our approach to state-of-the-art DG (e.g., CORAL, LISA, IRM, FISH,

Original: Group DRO) methods focusing on image, graph, and text data (Table 3 and Appendix C.4).
Revised: Group DRO) methods focusing on image, graph, and text data (Table 3 and Appendix C.4).

Original: Third, we analyse the GDUs robustness gainst DS that occurs in daily clinical practice (Table 12 and Appendix
Revised: Third, we analyse the GDUs robustness gainst DS that occurs in daily clinical practice (Table 12 and Appendix

Original: C.3).
Revised: C.3).

Original: Finally, in Appendix C.2, we conduct an ablation study focusing on the representation learned during training (Appendix C.2.2).
Revised: Finally, in Appendix C.2, we conduct an ablation study focusing on the representation learned during training (Appendix C.2.2).

Original: In our experiments, we distinguish two modes of training the DG layer: fine tuning (FT), where we extract features using a pre-trained model, and end-to-end training (E2E), where the FE and the DG layer are jointly trained 1 .
Revised: In our experiments, we distinguish two modes of training the DG layer: fine tuning (FT), where we extract features using a pre-trained model, and end-to-end training (E2E), where the FE and the DG layer are jointly trained 1 .

Original: 5. P ROOF - OF - CONCEPT BASED ON DIGITS CLASSIFICATION
Revised: 5. P ROOF - OF - CONCEPT BASED ON DIGITS CLASSIFICATION

Original: Following Feng et al.
Revised: Following Feng et al.

Original: (2020) among others, we create a multi-source dataset by combining five publicly available digits image datasets, namely MNIST (Lecun et al., 1998), MNIST-M (Ganin & Lempitsky, 2015), SVHN (Netzer et al., 2011), USPS, and Synthetic Digits (SYN) (Ganin &
Revised: (2020) among others, we create a multi-source dataset by combining five publicly available digits image datasets, namely MNIST (Lecun et al., 1998), MNIST-M (Ganin & Lempitsky, 2015), SVHN (Netzer et al., 2011), USPS, and Synthetic Digits (SYN) (Ganin &

Original: Lempitsky, 2015).
Revised: Lempitsky, 2015).

Original: The task is to classify digits between zero and nine.
Revised: The task is to classify digits between zero and nine.

Original: Each of these datasets is considered an out-of-training target domain which is inaccessible during training, and the remaining four are the source domains.
Revised: Each of these datasets is considered an out-of-training target domain which is inaccessible during training, and the remaining four are the source domains.

Original: Details are given in Appendix C.1.
Revised: Details are given in Appendix C.1.

Original: Table 2 summarizes the results for the most challenging out-of-training target domain, namely MNIST-M. In Appendix C.1, we provide the results on the remaining target domains in Table 7 and a discussion heuristics for choosing hyperparameters for our GDUs.
Revised: Table 2 summarizes the results for the most challenging out-of-training target domain, namely MNIST-M. In Appendix C.1, we provide the results on the remaining target domains in Table 7 and a discussion heuristics for choosing hyperparameters for our GDUs.

Original: Our method noticeably improves for all datasets mean accuracy and decreases the standard deviation in comparison to the ERM and ensemble baselines, making the results more stable across the ten iterations reported.
Revised: Our method noticeably improves for all datasets mean accuracy and decreases the standard deviation in comparison to the ERM and ensemble baselines, making the results more stable across the ten iterations reported.

Original: [Table]
Revised: [Table]

Original: We also compare our methods with related work that uses domain information and data augmentation, based on the results of Li et al.
Revised: We also compare our methods with related work that uses domain information and data augmentation, based on the results of Li et al.

Original: (Table 9 in Appendix C.1).
Revised: (Table 9 in Appendix C.1).

Original: Although data augmentation in DG is a comparatively strong approach and, at the same time, we do not use domain information; we obtain comparable results to the baselines reported by Li et al.
Revised: Although data augmentation in DG is a comparatively strong approach and, at the same time, we do not use domain information; we obtain comparable results to the baselines reported by Li et al.

Original: Ablation study We chose the digits dataset to analyze each component of our DG layer in 1st paragraph in Appendix C. and C.2.
Revised: Ablation study We chose the digits dataset to analyze each component of our DG layer in 1st paragraph in Appendix C. and C.2.

Original: We (A) vary M , N on Figure 7, and the strength of the regularization terms on Figure 4, Figure 5, and Figure 6 to assess the sensitivity of the DG layer to the choice of hyperparameters, (B) visualize the output of the FE (Figure 9).
Revised: We (A) vary M , N on Figure 9, and the strength of the regularization terms on Figure 6, Figure 7, and Figure 8 to assess the sensitivity of the DG layer to the choice of hyperparameters, (B) visualize the output of the FE (Figure 11).

Original: Our ablation study in (A) reveals stable results across different sets of hyper-parameters.
Revised: Our ablation study in (A) reveals stable results across different sets of hyper-parameters.

Original: While the layer is not sensitive to the choice of regularization strength, we recommend not to omit the regularization completely, although the computational expenses decrease without the orthogonal regularization.
Revised: While the layer is not sensitive to the choice of regularization strength, we recommend not to omit the regularization completely, although the computational expenses decrease without the orthogonal regularization.

Original: As an illustration in (B), we project the output of the FE trained with a dense layer (ERM) and with the DG layer by t-SNE (t-distributed stochastic neighbor embedding).
Revised: As an illustration in (B), we project the output of the FE trained with a dense layer (ERM) and with the DG layer by t-SNE (t-distributed stochastic neighbor embedding).

Original: The GDU-trained FE yields more concentrated and bounded clusters in comparison to the one trained by ERM.
Revised: The GDU-trained FE yields more concentrated and bounded clusters in comparison to the one trained by ERM.

Original: Hence, we observe a positive effect on the representation learned by the FE.
Revised: Hence, we observe a positive effect on the representation learned by the FE.

Original: 5. WILDS B ENCHMARK
Revised: 5. WILDS B ENCHMARK

Original: To challenge the I.E.D. assumption and the OOD generalization capabilities of the GDUs, we use
Revised: To challenge the I.E.D. assumption and the OOD generalization capabilities of the GDUs, we use

Original: WILDS, a curated set of real-world experiments for benchmarking DG methods (Koh et al., 2021).
Revised: WILDS, a curated set of real-world experiments for benchmarking DG methods (Koh et al., 2021).

Original: Further, WILDS is a semi-synthetic dataset set that operates under similiar assumptions as the source component shift (Koh et al., 2021).
Revised: Further, WILDS is a semi-synthetic dataset set that operates under similiar assumptions as the source component shift (Koh et al., 2021).

Original: We consider the following eight datasets: Camelyon17 , FMoW ,
Revised: We consider the following eight datasets: Camelyon17 , FMoW ,

Original: FMoW , Amazon , iWildCam , and RxRx1 , OGB-MolPCBA , Civil-Comments , and PovertyMap , which represent the task of real-world DG.
Revised: FMoW , Amazon , iWildCam , and RxRx1 , OGB-MolPCBA , Civil-Comments , and PovertyMap , which represent the task of real-world DG.

Original: We closely follow Koh et al.
Revised: We closely follow Koh et al.

Original: (2021) for the experiments.
Revised: (2021) for the experiments.

Original: Details on datasets and benchmark methods are given in Appendix C.4.
Revised: Details on datasets and benchmark methods are given in Appendix C.4.

Original: We present our benchmarking in Table 3.
Revised: We present our benchmarking in Table 3.

Original: Our results are achieved out-of-the-box (i.e., default parameters) since hyperparameter optimization has a substantial impact on the generalization performance (Gulrajani & Lopez-Paz, 2020), and we aim to highlight the improvements solely attributable to our GDUs.
Revised: Our results are achieved out-of-the-box (i.e., default parameters) since hyperparameter optimization has a substantial impact on the generalization performance (Gulrajani & Lopez-Paz, 2020), and we aim to highlight the improvements solely attributable to our GDUs.

Original: First, we observe the strengths and weaknesses of the benchmarks in the different data sets, all of which are lower than ERM at least once.
Revised: First, we observe the strengths and weaknesses of the benchmarks in the different data sets, all of which are lower than ERM at least once.

Original: In contrast, although GDUs show similar behavior across the datasets, performing very well for some datasets (e.g., FMoW, Poverty Map), they, however, do not fall below ERM across all GDU experiments conducted.
Revised: In contrast, although GDUs show similar behavior across the datasets, performing very well for some datasets (e.g., FMoW, Poverty Map), they, however, do not fall below ERM across all GDU experiments conducted.

Original: In addition, the baselines require domain information.
Revised: In addition, the baselines require domain information.

Original: Our approach requires less information, yet, achieving comparable results to the benchmarks.
Revised: Our approach requires less information, yet, achieving comparable results to the benchmarks.

Original: 5. ECG E XPERIMENT
Revised: 5. ECG E XPERIMENT

Original: The PhysioNet/Computing in Cardiology Challenge 2020 (Perez Alday et al., 2021; Goldberger et al., 2000; Perez Alday et al., 2020) aims to identify clinical diagnoses from 12-lead ECG recordings from 6 different databases.
Revised: The PhysioNet/Computing in Cardiology Challenge 2020 (Perez Alday et al., 2021; Goldberger et al., 2000; Perez Alday et al., 2020) aims to identify clinical diagnoses from 12-lead ECG recordings from 6 different databases.

Original: This publicly available pooled dataset contains 43,101 recordings sampled with various sampling frequencies and lengths.
Revised: This publicly available pooled dataset contains 43,101 recordings sampled with various sampling frequencies and lengths.

Original: Each recording is labeled as having one or more of 24 cardiac abnormalities; hence, the task is to perform a multi-label binary classification.
Revised: Each recording is labeled as having one or more of 24 cardiac abnormalities; hence, the task is to perform a multi-label binary classification.

Original: For our experiment, we iterate over the databases, taking one at a time as the test domain while utilizing
Revised: For our experiment, we iterate over the databases, taking one at a time as the test domain while utilizing

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the remaining five databases for training.
Revised: the remaining five databases for training.

Original: The performance was measured according to the original
Revised: The performance was measured according to the original

Original: PhysioNet challenge score.
Revised: PhysioNet challenge score.

Original: This generalized intersection-over-union score assigns partial credit to misdiagnoses that result in similar treatments or outcomes.
Revised: This generalized intersection-over-union score assigns partial credit to misdiagnoses that result in similar treatments or outcomes.

Original: The score is then adjusted for a solution that always selects the normal/majority class and normalized for the perfect solution.
Revised: The score is then adjusted for a solution that always selects the normal/majority class and normalized for the perfect solution.

Original: Therefore, the score can have negative values and a best possible score of 1.
Revised: Therefore, the score can have negative values and a best possible score of 1.

Original: Table 4 reports results for the ECG experiments (see Appendix C.3 for details).
Revised: Table 4 reports results for the ECG experiments (see Appendix C.3 for details).

Original: For this clinical time-series data, we observe an improvement in mean score and a reduction in standard deviation over the ERM and ERM ensemble baselines across all DG tasks.
Revised: For this clinical time-series data, we observe an improvement in mean score and a reduction in standard deviation over the ERM and ERM ensemble baselines across all DG tasks.

Original: We attribute poorer performance for the PTB dataset to the fact that it contains considerably longer recordings than other datasets (except for INCART which, however, contains only 75 samples) and a higher sampling rate (1000Hz vs. 500Hz and 257Hz).
Revised: We attribute poorer performance for the PTB dataset to the fact that it contains considerably longer recordings than other datasets (except for INCART which, however, contains only 75 samples) and a higher sampling rate (1000Hz vs. 500Hz and 257Hz).

Original: The negative challenge score for the PTB-XL dataset is due to the presence of previously unobserved labels in other datasets as well as a considerably smaller amount of data for training since the PTB-XL dataset comprises the majority of all samples (21,837 out of 43,101).
Revised: The negative challenge score for the PTB-XL dataset is due to the presence of previously unobserved labels in other datasets as well as a considerably smaller amount of data for training since the PTB-XL dataset comprises the majority of all samples (21,837 out of 43,101).

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION AND D ISCUSSIONS
Revised:  C ONCLUSION AND D ISCUSSIONS

Original: We introduced the I.E.D. assumption, postulating that real-world distributions are composed of elementary distributions that remain invariant across different domains and showed that it implies an invariant structure in the solution space that enables knowledge transfer to unseen domains.
Revised: We introduced the I.E.D. assumption, postulating that real-world distributions are composed of elementary distributions that remain invariant across different domains and showed that it implies an invariant structure in the solution space that enables knowledge transfer to unseen domains.

Original: Empirical results based on real-world data support the practicality of the I.E.D. assumption and that we can learn such a representation.
Revised: Empirical results based on real-world data support the practicality of the I.E.D. assumption and that we can learn such a representation.

Original: Further, we presented a modular neural network layer consisting of Gated
Revised: Further, we presented a modular neural network layer consisting of Gated

Original: Domain Units (GDUs) that leverage the I.E.D. assumption.
Revised: Domain Units (GDUs) that leverage the I.E.D. assumption.

Original: Our GDUs can substantially improve the downstream performance of learning machines in real-world DG tasks.
Revised: Our GDUs can substantially improve the downstream performance of learning machines in real-world DG tasks.

Original: Across our experiments, we observed that for some datasets FT is better than E2E and vice versa.
Revised: Across our experiments, we observed that for some datasets FT is better than E2E and vice versa.

Original: In E2E training, the feature extractor (encoder) is jointly trained with GDUs.
Revised: In E2E training, the feature extractor (encoder) is jointly trained with GDUs.

Original: Hence, the latent representation is stochastic during training, meaning that we have variability in the representation fed into GDUs between epochs.
Revised: Hence, the latent representation is stochastic during training, meaning that we have variability in the representation fed into GDUs between epochs.

Original: In contrast, in FT, the feature extractor is pretrained and always produces the same embedding.
Revised: In contrast, in FT, the feature extractor is pretrained and always produces the same embedding.

Original: Especially with large feature extractors such as ResNet-50, learning the elementary domains can be more effective when we avoid any stochasticity in the latent representation.
Revised: Especially with large feature extractors such as ResNet-50, learning the elementary domains can be more effective when we avoid any stochasticity in the latent representation.

Original: Limitations.
Revised: Limitations.

Original: A major limitation of our I.E.D. assumption is to provide theoretical evidence that this assumption holds in practice.
Revised: A major limitation of our I.E.D. assumption is to provide theoretical evidence that this assumption holds in practice.

Original: We aim to expand the scope of the theoretical understanding of the I.E.D. assumption and the GDUs.
Revised: We aim to expand the scope of the theoretical understanding of the I.E.D. assumption and the GDUs.

Original: In addition, the particular theoretical setting of Albuquerque et al.
Revised: In addition, the particular theoretical setting of Albuquerque et al.

Original: (2019) (i.e., each elementary domain represents a source domain) seems promising to extend their generalization guarantee to cases where our I.E.D. assumption holds.
Revised: (2019) (i.e., each elementary domain represents a source domain) seems promising to extend their generalization guarantee to cases where our I.E.D. assumption holds.

Original: Second, our GDU layer induces additional computational overhead due to the regularization and model size that increases as a function of the number of elementary domains.
Revised: Second, our GDU layer induces additional computational overhead due to the regularization and model size that increases as a function of the number of elementary domains.

Original: Noteworthy, our improvement is achieved with a relatively small number of elementary domains indicating that the increased complexity is not a coercive consequence of applying the DG layer.
Revised: Noteworthy, our improvement is achieved with a relatively small number of elementary domains indicating that the increased complexity is not a coercive consequence of applying the DG layer.

Original: Also, the results achieved are not a consequence of increased complexity, as the ensemble baseline shows.
Revised: Also, the results achieved are not a consequence of increased complexity, as the ensemble baseline shows.

Original: Future work We expect the I.E.D. assumption and GDUs to be adapted, yielding novel applications that tackle DG.
Revised: Future work We expect the I.E.D. assumption and GDUs to be adapted, yielding novel applications that tackle DG.

Original: For example, we suggest dynamically increasing the number of elementary domains during learning until their distributional variance reaches a plateau as a measure of their heterogeneity.
Revised: For example, we suggest dynamically increasing the number of elementary domains during learning until their distributional variance reaches a plateau as a measure of their heterogeneity.

Original: Hence, one would learn the number of elementary domains instead of fixing the number of elementary domains prior to training.
Revised: Hence, one would learn the number of elementary domains instead of fixing the number of elementary domains prior to training.

Original: Appendices 
Revised: Appendices Table

Original: Table of Contents
Revised: of Contents

Original: A Proofs
Revised: A Proofs

Original: A.1 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Proof of Proposition 3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2 Proof of Proposition 3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B Details on the Gated Domain Units
Revised: B Details on the Gated Domain Units

Original: B.1 Detailed View of the Regularization Term Ω OLSD . . . . . . . . . . . . . . . . . 
Revised: B.1 Real-world example: Visualizations . . . . . . . . . . . . . . . . . . . . . . . . B [Table] [Table]

Original: B.2 Visualization of DG Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: of the [Table] Term Ω OLSD . . . . . . . . . . . . . . . . . [Table]

Original: C Experiments
Revised: C Experiments

Original: C.1 Digits Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: C.1 Digits Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.

Revised: [Table]

Original: C.2 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.3 ECG Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.3 ECG Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.4 WILDS Benchmarking Experiments . .
Revised: C.4 WILDS Benchmarking Experiments . .

Original: A P ROOFS A. P ROOF OF L EMMA
Revised: A P ROOFS A. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: The result holds trivially for K = 1 .
Revised: The result holds trivially for K = 1 .

Original: For K ≥ 2 and by the I.E.D assumption,
Revised: For K ≥ 2 and by the I.E.D assumption,

Original: P s ( X, Y ) =
Revised: P s ( X, Y ) =

Original: (cid:80) Kj =

Revised: (cid:80) Kj =

Original: α j P j
Revised: α j P j

Original: ( X, Y )
Revised: ( X, Y )

Original: for some α ∈ ∆ K .
Revised: for some α ∈ ∆ K .

Original: Then, we can write the risk functional for each f ∈ F as R ( f ) =
Revised: Then, we can write the risk functional for each f ∈ F as R ( f ) =

Original: (cid:82) L ( y, f ( x ))
Revised: (cid:82) L ( y, f ( x ))

Original: d P s ( x, y ) =
Revised: d P s ( x, y ) =

Original: (cid:82) L ( y, f ( x ))
Revised: (cid:82) L ( y, f ( x ))

Original: d ( (cid:80) Kj =1 α j P j ( x, y ))
Revised: d ( (cid:80) Kj =1 α j P j ( x, y ))

Original: = (cid:80) Kj =1 α j (cid:82) L ( y, f ( x ))
Revised: = (cid:80) Kj =1 α j (cid:82) L ( y, f ( x ))

Original: d P j ( x, y ) =
Revised: d P j ( x, y ) =

Original: (cid:80) Kj
Revised: (cid:80) Kj

Original: =

Revised: =

Original: α j R j ( f ) where R j : F → R + is the elementary risk functional associated with the elementary distribution P j ( X, Y ) .
Revised: α j R j ( f ) where R j : F → R + is the elementary risk functional associated with the elementary distribution P j ( X, Y ) .

Original: Hence, the Bayes predictors satisfy
Revised: Hence, the Bayes predictors satisfy

Original: [Equation]
Revised: [Equation]

Original: Since the rhs of equation A.3 corresponds to the linear scalarization of a multi-objective function ( R 1 , . .
Revised: Since the rhs of equation A.3 corresponds to the linear scalarization of a multi-objective function ( R 1 , . .

Original: , R K ) , its solution (i.e., a stationary point) is Pareto-optimal with respect to these objective functions (Ma et al., 2020, Definition 3.1); see, also, (Hillermeier, 2001a;b).
Revised: , R K ) , its solution (i.e., a stationary point) is Pareto-optimal with respect to these objective functions (Ma et al., 2020, Definition 3.1); see, also, (Hillermeier, 2001a;b).

Original: That is, the Bayes predictors for the data distribution that satisfies the I.E.D assumption must belong to the Pareto set
Revised: That is, the Bayes predictors for the data distribution that satisfies the I.E.D assumption must belong to the Pareto set

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF P ROPOSITION 3. 
Revised: A. P ROOF OF P ROPOSITION 3.

Original: Proof.
Revised: Proof.

Original: Suppose we have a representation,
Revised: Suppose we have a representation,

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. { µ V 1 , . .
Revised: e. { µ V 1 , . .

Original: . , µ V m } are pairwise orthogonal.
Revised: . , µ V m } are pairwise orthogonal.

Original: We want to minimize the MMD by minimizing ∥ µ P −
Revised: We want to minimize the MMD by minimizing ∥ µ P −

Original: M (cid:88)
Revised: M (cid:88)

Original: [Equation]
Revised: [Equation]

Original: By defining
Revised: By defining

Original: [Equation]
Revised: [Equation]

Original: we can simply find the optimal β j by using the partial derivative
Revised: we can simply find the optimal β j by using the partial derivative

Original: [Equation]
Revised: [Equation]

Original: Please note that the function Φ is convex. 
Revised: Please note that the function Φ is convex. DE CH AT

Revised: US

Revised: Task :

Original: B D ETAILS ON THE G ATED D OMAIN U NITS B. D ETAILED V IEW OF THE R EGULARIZATION T ERM Ω OLSD
Revised: Forecast glucose values from past observations OOD generalization requires to find the “right” invariance ! DE CH

Revised: AT

Revised: US

Revised: Elementary distributions

Revised: DE

Revised: CH

Original: First, consider the following single term ∥
Revised: AT USA Problems: Ø Learn invariant structure based on observed data Ø Domains are defined too broad (countries vs. subpopulations within countries) Prior research

Revised: Motivation GDUs

Revised: B D ETAILS ON THE G ATED D OMAIN U NITS B. R EAL - WORLD EXAMPLE : V ISUALIZATIONS

Revised: As written in Section 2.1, we postulate that the elementary domain bases are the invariant subspaces that allow us to generalize to unseen domains.

Revised: In practice, the question arises if and when elementary domains evolve.

Revised: Consider that we aim to learn to predict the risk of developing Diabetes from laboratory data from Europe and then infer the risk from data from the United States of America.

Revised: Naturally, factors influencing the data-generating process may change, such as the level of physical activity and nutritional habits.

Revised: While, to a certain degree, these common factors remain invariant across continents, each of these factors’ contributions may differ.

Revised: In terms of our assumptions, we model each of these factors with a corresponding elementary distribution.

Revised: Figure 3 depicts our assumption and how it differs from existing works 2 .

Revised: To exploit this assumption in out-of-distribution (OOD) generalization, we developed a modular neural network layer that consists of so-called Gated Domain Units (GDUs).

Revised: In Figure 4, we visualized the fundamental concept of the GDUs.

Revised: Each GDU learns an embedding of an individual elementary domain that allows us to encode the domain similarities during the training.

Revised: During inference, the GDUs compute similarities between observation and each of the corresponding elementary distributions, which are then used to form a weighted ensemble of learning machines.

Revised: In other words, for a previously unseen individual, we aim to determine the coefficients and quantify each factor’s contribution without any information about the individual’s origin

Revised: . B. D ETAILED V IEW OF THE R EGULARIZATION T ERM Ω OLSD

Revised: First, consider the following single term ∥

Original: ϕ
Revised: ϕ

Original: (˜ x i ) − (cid:80)
Revised: (˜ x i ) − (cid:80)

Original: Mj
Revised: Mj

Original: =1 β ij µ V j ∥ 2 H that can be expressed as ∥ 
Revised: =1 β ij µ V j ∥ 2 H that can be expressed as ∥ ϕ

Original: [Equation]
Revised: [Equation]

Original: A D (1):
Revised: i [Equation] − M

Original:  We begin with Term (1) and write ∥ ϕ (˜ x i ) ∥ 2 H as ∥ ϕ
Revised: [Equation] A D [Equation] We begin with Term (1) and write ∥ ϕ (˜ x i ) ∥ 2 H as ∥ ϕ

Original: (˜ x i ) ∥ 2 H = ⟨ ϕ (˜ x i ) , ϕ (˜ x i ) ⟩ H =
Revised: (˜ x i ) ∥ 2 H = ⟨ ϕ (˜ x i ) , ϕ (˜ x i ) ⟩ H =

Original: k
Revised: k

Original: (˜ x i , ˜ x i ) .
Revised: (˜ x i , ˜ x i ) .

Original: We could evaluate this term using the kernel function k for each data point in the batch b . 
Revised: We could evaluate this term using the kernel function k for each data point in the batch b . However, since this Of Figure

Revised: , and we do not want to make medical implications any way.

Original: However, since this term does not depend on the the elementary domains { V 1 , . .
Revised: [Figure] term does not depend on the the elementary domains { V 1 , . .

Original: . , V M } , it is unnecessary to compute this value to minimize the penalty.
Revised: . , V M } , it is unnecessary to compute this value to minimize the penalty.

Original: Thus, we obtain a similar result by minimizing the penalty without considering ∥ ϕ
Revised: Thus, we obtain a similar result by minimizing the penalty without considering ∥ ϕ

Original: (˜ x i ) ∥ 2 H in the regularization.
Revised: (˜ x i ) ∥ 2 H in the regularization.

Original: A D (2):
Revised: A D (2):

Original: Term (2) can be expressed as
Revised: Term (2) can be expressed as

Original: [Equation]
Revised: [Equation]

Original: Implementation-wise, the evaluation of this term requires the calculation of the inner product ⟨ ϕ (˜ x i ) , µ V j ⟩ H .
Revised: Implementation-wise, the evaluation of this term requires the calculation of the inner product ⟨ ϕ (˜ x i ) , µ V j ⟩ H .

Original: Since our CS and projection-based methods involve this inner product to determine the coefficients β ij , we pre-compute the inner product ⟨ ϕ (˜ x i ) , µ V j ⟩ H once for a mini-batch and store these information during training to avoid multiple calculations of the same term.
Revised: Since our CS and projection-based methods involve this inner product to determine the coefficients β ij , we pre-compute the inner product ⟨ ϕ (˜ x i ) , µ V j ⟩ H once for a mini-batch and store these information during training to avoid multiple calculations of the same term.

Original: Moreover, the projection-based method does not apply softmax and has a linear form.
Revised: Moreover, the projection-based method does not apply softmax and has a linear form.

Original: Therefore, the term (2) can be simplified even further:
Revised: Therefore, the term (2) can be simplified even further:

Original: [Equation]
Revised: [Equation]

Original: A D
Revised: A D

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Last, we express the term (3) as follows
Revised: Last, we express the term (3) as follows

Original: [Equation]
Revised: [Equation]

Original: and calculate the inner product of the domains ⟨ µ V j , µ V k ⟩ H by
Revised: and calculate the inner product of the domains ⟨ µ V j , µ V k ⟩ H by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: B.8) where N represents the number of vectors per domain basis.
Revised: where N represents the number of vectors per domain basis.

Original: Note that this term does not depend on the input data x i and, hence, matrix K jk can be calculated once at the beginning of the optimization step and stored to be re-used for all the data point of a batch.
Revised: Note that this term does not depend on the input data x i and, hence, matrix K jk can be calculated once at the beginning of the optimization step and stored to be re-used for all the data point of a batch.

Original: Combining Equation B.6 and Equation B.8 yields
Revised: Combining Equation B.6 and Equation B.8 yields

Original: [Equation]
Revised: [Equation]

Revised: As a final step

Original: As a final step, we use the results for Term (1) , (2) , and (3) to obtain the desired regularization term
Revised: [Equation] we use the results for Term (1) , (2) [Equation] and (3) to obtain the desired regularization term

Original: [Equation]
Revised: [Equation]

Revised: B.

Revised: [Equation]

Revised: [Equation]

Original: As mentioned above, ∥ ϕ (˜ x i ) ∥
Revised: As mentioned above, ∥ ϕ (˜ x i ) ∥

Original: 2 H is independent from the elementary domains, and, thus a constant in the regularization.
Revised: 2 H is independent from the elementary domains, and, thus a constant in the regularization.

Original: Hence, we can exclude this term, which avoids additional computational effort.
Revised: Hence, we can exclude this term, which avoids additional computational effort.

Original: B. V ISUALIZATION OF DG L AYER
Revised: B. V ISUALIZATION OF DG L AYER

Original: Figure 3 depicts the layout of our DG layer.[Figure] [Equation] [Figure]
Revised: Figure 5 depicts the layout of our DG layer.

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: C E XPERIMENTS
Revised: C E XPERIMENTS

Original: In this section, we provide a detailed description of the DG experiment presented in Section 5.
Revised: In this section, we provide a detailed description of the DG experiment presented in Section 5.

Original: Our
Revised: Our

Original: Digits and ECG experiments are implemented using TensorFlow 2.4.1 and TensorFlow Probability 0.12.1.
Revised: Digits and ECG experiments are implemented using TensorFlow 2.4.1 and TensorFlow Probability 0.12.1.

Original: For the WILDS benchmarking we use our PyTorch (version 1.11.0).
Revised: For the WILDS benchmarking we use our PyTorch (version 1.11.0).

Original: All source code will be made available on GitHub https://github.com/ (TensorFlow) and https://github.
Revised: All source code will be made available on GitHub https://github.com/ (TensorFlow) and https://github.

Original: com/ (PyTorch).
Revised: com/ (PyTorch).

Original: Overall, our experiments aim to show the validity of the invariant elementary distribution (I.E.D.) assumption and the Gated Domain Units (GDUs).
Revised: Overall, our experiments aim to show the validity of the invariant elementary distribution (I.E.D.) assumption and the Gated Domain Units (GDUs).

Original: For the DG layer, we considered two modes of model training: fine tuning (FT) and end-to-end training (E2E).
Revised: For the DG layer, we considered two modes of model training: fine tuning (FT) and end-to-end training (E2E).

Original: In FT scenario, we first pre-train the FE in the ERM single fashion.
Revised: In FT scenario, we first pre-train the FE in the ERM single fashion.

Original: Then, we extract features using the pre-trained model and pass them to the DG layer for training the latter.
Revised: Then, we extract features using the pre-trained model and pass them to the DG layer for training the latter.

Original: For the
Revised: For the

Original: E2E training, however, the whole model including the FE and DG layer is trained jointly from the very beginning. 
Revised: E2E training, however, the whole model including the FE and DG layer is trained jointly from the very beginning. h θ x Σ

Revised: H M = span { µ V i | i = 1 ,...,M }

Revised: L (ˆ y , y ) + Ω( ∥ f ∥ H ) + µ Vβ 1 ( W 1 x + b 1 ) µ

Revised: Vβ 2 ( W 2 x + b 2 ) ...

Original:  C. D IGITS E XPERIMENT
Revised: µ V M β M ( W M x + b M ) ϕ ( x ) γ ( ϕ ( x ) ,µ V M ) V M ϕ ( V M ) µ V M x ( x ,β M ) C. D IGITS E XPERIMENT

Original: Our experiment setup is closely related to Peng et al.
Revised: Our experiment setup is closely related to Peng et al.

Original: ; Feng et al.
Revised: ; Feng et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: Zhao et al.
Revised: Zhao et al.

Original: Each dataset, except USPS, is split into training and test sets of 25,000 and 9, images, respectively.
Revised: Each dataset, except USPS, is split into training and test sets of 25,000 and 9, images, respectively.

Original: For USPS, we take the whole dataset for the experiment since it contains only 9,298 images 2 .
Revised: For USPS, we take the whole dataset for the experiment since it contains only 9,298 images 3 .

Original: Our experimental setup regarding datasets, data loader, and FE are based on existing work (Feng et al., 2020; Peng et al., 2019).
Revised: Our experimental setup regarding datasets, data loader, and FE are based on existing work (Feng et al., 2020; Peng et al., 2019).

Original: The structure of the FE is summarized in Table 5 and the subsequent learning machine is a dense layer.
Revised: The structure of the FE is summarized in Table 5 and the subsequent learning machine is a dense layer.

Original: In the Empirical Risk Minimization (ERM) single experiment, we add a dense layer with 10 outputs (activation= tanh ) as a classifier to the FE.
Revised: In the Empirical Risk Minimization (ERM) single experiment, we add a dense layer with 10 outputs (activation= tanh ) as a classifier to the FE.

Original: In the Empirical Risk Minimization (ERM) ensemble experiment, we add M classification heads (a dense layers with 10 outputs and tanh activation each) to the FE and average their output for the final prediction.
Revised: In the Empirical Risk Minimization (ERM) ensemble experiment, we add M classification heads (a dense layers with 10 outputs and tanh activation each) to the FE and average their output for the final prediction.

Original: This sets a baseline for our DG layer to show performance gain against the ERM model with the same number of learning machines.
Revised: This sets a baseline for our DG layer to show performance gain against the ERM model with the same number of learning machines.

Original: For training, we resorted to the Adam optimizer with a learning rate of 0.001.
Revised: For training, we resorted to the Adam optimizer with a learning rate of 0.001.

Original: We used early stopping and selected the best model weights according to the validation accuracy.
Revised: We used early stopping and selected the best model weights according to the validation accuracy.

Original: For the validation data, we used the combined test splits only of the respective source datasets.
Revised: For the validation data, we used the combined test splits only of the respective source datasets.

Original: The batch size was set to 512.
Revised: The batch size was set to 512.

Original: Although the DG layer requires more computation resources than the ERM models, all digits experiments were conducted on a single GPU (NVIDIA GeForce RTX 3090).
Revised: Although the DG layer requires more computation resources than the ERM models, all digits experiments were conducted on a single GPU (NVIDIA GeForce RTX 3090).

Original: Heuristics for main parameter of DG layer From a practical perspective, our layer requires choosing two main hyper-parameters: the number of elementary domains M and since we use the characteristics Gaussian kernel the corresponding parameter σ .
Revised: Heuristics for main parameter of DG layer From a practical perspective, our layer requires choosing two main hyper-parameters: the number of elementary domains M and since we use the characteristics Gaussian kernel the corresponding parameter σ .

Original: The parameter M determines the size of the ensemble of learning machines and, thus, for deep learning models, their overall network size.
Revised: The parameter M determines the size of the ensemble of learning machines and, thus, for deep learning models, their overall network size.

Original: As a heuristic to choose M , we suggest to cluster the output of a pre-trained FE.
Revised: As a heuristic to choose M , we suggest to cluster the output of a pre-trained FE.

Original: In the following, we provide an example.
Revised: In the following, we provide an example.

Original: We pre-trained the FE for the test domain MNIST-M and pass the source data through this FE, which we cluster with the k-means algorithm.
Revised: We pre-trained the FE for the test domain MNIST-M and pass the source data through this FE, which we cluster with the k-means algorithm.

Original: Subsequently, we analyse three different metrics (Calinski Harabasz score, Davies Bouldinn score, and Silhouette score) to select the optimal number of clusters as the basis to choose M .
Revised: Subsequently, we analyse three different metrics (Calinski Harabasz score, Davies Bouldinn score, and Silhouette score) to select the optimal number of clusters as the basis to choose M .

Original: All scores yielded an accordance between four to five clusters.
Revised: All scores yielded an accordance between four to five clusters.

Original: Therefore, we set M to five and observed in Table 2 in Section 5 strong results in the generalizing to the unseen test domain MNIST-M . 
Revised: Therefore, we set M to five and observed in Table 2 in Section 5 strong results in the generalizing to the unseen test domain MNIST-M . [Table]

Original: As for the parameter σ , we resort to the median heuristic proposed in (Muandet et al., 2016) that is σ 2 = median { ∥ ˜ x i − ˜ x
Revised: As for the parameter σ , we resort to the median heuristic proposed in (Muandet et al., 2016) that is σ 2 = median { ∥ ˜ x i − ˜ x

Original: j ∥ 2 : i, j = 1 , . . . , n } .
Revised: j ∥ 2 : i, j = 1 , . . . , n } .

Original: While both heuristics require a pre-trained
Revised: While both heuristics require a pre-trained

Original: FE, cross-validation can act as a reasonable alternative.
Revised: FE, cross-validation can act as a reasonable alternative.

Original: The hyper-parameters relevant for the DG layer are summarized in Table 6.
Revised: The hyper-parameters relevant for the DG layer are summarized in Table 6.

Original: In the FT setting, we applied the median heuristics presented above to estimate σ of the Gaussian kernel function, where the estimator is denoted as ˆ σ .
Revised: In the FT setting, we applied the median heuristics presented above to estimate σ of the Gaussian kernel function, where the estimator is denoted as ˆ σ .

Original: Since median heuristic is not applicable for the E2E scenario, σ was fixed to 7.5 for E2E.
Revised: Since median heuristic is not applicable for the E2E scenario, σ was fixed to 7.5 for E2E.

Original: Note that our approach to choose the relevant parameters was kept very general to show the feasibility of the I.E.D. assumption and the generalization ability of GDUs and, most importantly, to provide easy-to-reproduce results.
Revised: Note that our approach to choose the relevant parameters was kept very general to show the feasibility of the I.E.D. assumption and the generalization ability of GDUs and, most importantly, to provide easy-to-reproduce results.

Original: During training, additional epoch metrics can be subscribed using our[Table] custom DG layer callback , which may help to choose the model parameters.
Revised: During training, additional epoch metrics can be subscribed using our custom DG layer callback , which may help to choose the model parameters.

Original: Furthermore, we observed that the elementary domains become naturally orthogonal during the experiments [Table] and thus, we set λ ORTH relatively small.
Revised: Furthermore, we observed that the elementary domains become naturally orthogonal during the experiments, and thus, we set λ ORTH relatively small.

Original: Since the orthogonal regularization puts additional computational burden, one could omit this term completely to speed up training
Revised: Since the orthogonal regularization puts additional computational burden, one could omit this term completely to speed up training.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] In previous research, the aforementioned digits data is not only used for domain adaptation (DA), but also for domain generalization (DG) methods.
Revised: Digit-DG Benchmark In previous research, the aforementioned digits data is not only used for domain adaptation (DA), but also for domain generalization (DG) methods.

Original: For the latter, Zhou et al.
Revised: For the latter, Zhou et al.

Original: (2021b) and Li et al.
Revised: (2021b) and Li et al.

Original: introduced Digit-DG dataset and the evaluation protocol to benchmark seven DG methods and ERM 3 .
Revised: introduced Digit-DG dataset and the evaluation protocol to benchmark seven DG methods and ERM 4 .

Original: Unlike the Digits experiment described above, Digit-DG dataset from
Revised: Unlike the Digits experiment described above, Digit-DG dataset from

Original: Zhou et al.
Revised: Zhou et al.

Original: (2021b)
Revised: (2021b)

Original: and
Revised: and

Original: Li et al.
Revised: Li et al.

Original: (2021) consists of only four datasets (without USPS) and a different
Revised: (2021) consists of only four datasets (without USPS) and a different

Original: FE summarized in Table 8.
Revised: FE summarized in Table 8.

Original: Therefore, we follow their instructions to conduct a fair comparison and ensure reproducibility.
Revised: Therefore, we follow their instructions to conduct a fair comparison and ensure reproducibility.

Original: For the hyper-parameters, however, we kept the same values that we used for the Digits experiment, see Table 
Revised: For the hyper-parameters, however, we kept the same values that we used for the Digits experiment, see Table 6.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: As a first method

Original: [Table]
Revised: [Table]

Original:  we consider the CCSA (Classification and Contrastive Semantic Alignment) method [Table] which learns a domain-invariant representation by utilizing the CCSA loss (Motiian et al., 2017).Second
Revised: As a first method, we consider the CCSA (Classification and Contrastive Semantic Alignment) method, which learns a domain-invariant representation by utilizing the CCSA loss (Motiian et al., 2017).

Original: [Table] MMD-AAE (Maximum Mean Discrepancy-based Adverserial Autoencoders) extends adverserial autoencoders by a maximum mean discrepancy regularization to learn a domain-invariant feature representation (Li et al., 2018b).
Revised: Second, MMD-AAE (Maximum Mean Discrepancy-based Adverserial Autoencoders) extends adverserial autoencoders by a maximum mean discrepancy regularization to learn a domain-invariant feature representation (Li et al., 2018b).

Original: CrossGrad (Cross-Gradient) augments data by perturbating the input space using the cross-gradients of a label and domain predictor (Shankar et al., 2018).
Revised: CrossGrad (Cross-Gradient) augments data by perturbating the input space using the cross-gradients of a label and domain predictor (Shankar et al., 2018).

Original: Another augmentation-based DG method is L2A-OT (Learning to Augment by Optimal Transport)
Revised: Another augmentation-based DG method is L2A-OT (Learning to Augment by Optimal Transport)

Original: (Zhou et al., 2021b).
Revised: (Zhou et al., 2021b).

Original: Specifically, a data generator trained to maximize the optimal transport distance between source and pseudo domains, is used to augment the source data.
Revised: Specifically, a data generator trained to maximize the optimal transport distance between source and pseudo domains, is used to augment the source data.

Original: All aforementioned methods rely on the availability of domain information such as domain labels.
Revised: All aforementioned methods rely on the availability of domain information such as domain labels.

Original: To benchmark our layer to a method for DG without domain information, we resort to the JiGen (Jigsaw puzzle based
Revised: To benchmark our layer to a method for DG without domain information, we resort to the JiGen (Jigsaw puzzle based

Original: Generalization) method (Carlucci et al., 2019).
Revised: Generalization) method (Carlucci et al., 2019).

Original: JiGen introduces an auxiliary loss for solving jigsaw task during training.
Revised: JiGen introduces an auxiliary loss for solving jigsaw task during training.

Original: Further, we use the adaptive and non-adaptive stochastic feature augmentation (SFA-S and SFA-A, respectively) method proposed by Li et al.
Revised: Further, we use the adaptive and non-adaptive stochastic feature augmentation (SFA-S and SFA-A, respectively) method proposed by Li et al.

Original: In principle, both method augment the latent feature embedding of a FE using random noise.
Revised: In principle, both method augment the latent feature embedding of a FE using random noise.

Original: Our results are summarized in Table 9.
Revised: Our results are summarized in Table 9.

Original: As noted by Li et al.
Revised: As noted by Li et al.

Original: (2021), it is challenging to outperform augmentation-based DG methods.
Revised: (2021), it is challenging to outperform augmentation-based DG methods.

Original: In addition, SFA-A and SF-S are computationally light (i.e., only adding random noise to the feature embedding) and do not require domain information (Li et al., 2021).
Revised: In addition, SFA-A and SF-S are computationally light (i.e., only adding random noise to the feature embedding) and do not require domain information (Li et al., 2021).

Original: Nevertheless, our layer achieves competitive results even against the strongest baselines in all
Revised: Nevertheless, our layer achieves competitive results even against the strongest baselines in all

Original: DG tasks without requiring domain information.
Revised: DG tasks without requiring domain information.

Revised: [Table]

Revised: B ✗

Revised: [Table]

Original:  C. A BLATION S TUDY C.2. M AIN COMPONENTS OF THE G ATED D OMAIN U NIT
Revised: B ✗ C. A BLATION S TUDY C.2. M AIN COMPONENTS OF THE G ATED D OMAIN U NIT

Original: We chose the Digits dataset to conduct an ablation study, which is organized as follows: (1) ablation of the regularization terms presented in Section 3, (2) effect of the orthogonal regularization for projection-based generalization, and (3) affect on the FE’s output.
Revised: We chose the Digits dataset to conduct an ablation study, which is organized as follows: (1) ablation of the regularization terms presented in Section 3, (2) effect of the orthogonal regularization for projection-based generalization, and (3) affect on the FE’s output.

Original: As a reminder, we introduced the regularization to be dependent on the form of generalization (i.e., domain similarity measures or projection-based generalization in Section 3).
Revised: As a reminder, we introduced the regularization to be dependent on the form of generalization (i.e., domain similarity measures or projection-based generalization in Section 3).

Original: For the domain similarity measure case, the regularization is
Revised: For the domain similarity measure case, the regularization is

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: B ✗ P ROJECTION 97.

Original: B ✗ where λ OLS , λ L 1 ≥ 0 .
Revised: where λ OLS , λ L 1 ≥ 0 .

Original: In the case of projection, the regularization is given by
Revised: In the case of projection, the regularization is given by

Original: [Equation]
Revised: [Equation]

Original: with λ OLS , λ ORTH ≥ 0 .
Revised: with λ OLS , λ ORTH ≥ 0 .

Original: Although one can additionally choose the sparse regularization in projection-based generalization, we set the focus in the ablation study on the two main regularization terms that are the OLS and orthogonal regularization.
Revised: Although one can additionally choose the sparse regularization in projection-based generalization, we set the focus in the ablation study on the two main regularization terms that are the OLS and orthogonal regularization.

Original: For (1) we vary in Equation C.1 and Equation
Revised: For (1) we vary in Equation C.1 and Equation

Original: C.2 the corresponding weights λ 1 and λ 2 in the interval of [0; 0 . 1] and display the mean classification accuracy for the most challenging classification task of MNSIT-M in the form of a heatmap.
Revised: C.2 the corresponding weights λ 1 and λ 2 in the interval of [0; 0 . 1] and display the mean classification accuracy for the most challenging classification task of MNSIT-M in the form of a heatmap.

Original: In
Revised: In

Original: Figures 4-6, we see that the classification accuracy remains on an overall similar level which indicates that the DG layer is not very sensitive to the hyper-parameter change for MNIST-M as the test domain.
Revised: Figures 6-8, we see that the classification accuracy remains on an overall similar level which indicates that the DG layer is not very sensitive to the hyper-parameter change for MNIST-M as the test domain.

Original: Nevertheless, we observe that ablating the regularization terms by setting the corresponding weights to zero decreases the classification results and the peaks in performance occur when the regularization is included during training of the DG layer.
Revised: Nevertheless, we observe that ablating the regularization terms by setting the corresponding weights to zero decreases the classification results and the peaks in performance occur when the regularization is included during training of the DG layer.

Original: Applying the DG layer comes with additional overhead, especially the regularization that ensures the orthogonality of the elementary domain bases.
Revised: Applying the DG layer comes with additional overhead, especially the regularization that ensures the orthogonality of the elementary domain bases.

Original: This additional effort raises a question whether ensuring the theoretical assumptions outweigh the much higher computational effort.
Revised: This additional effort raises a question whether ensuring the theoretical assumptions outweigh the much higher computational effort.

Original: Thus, in a second step, we analyze how the orthogonal regularization affects the orthogonality of the elementary domain bases (i.e., spectral restricted isometry property (SRIP) value) and the loss function (i.e., categorical cross-entropy).
Revised: Thus, in a second step, we analyze how the orthogonal regularization affects the orthogonality of the elementary domain bases (i.e., spectral restricted isometry property (SRIP) value) and the loss function (i.e., categorical cross-entropy).

Original: In Figure 8, we depict the mean and standard deviation of the SRIP value and loss over five runs for 
Revised: In Figure 10, we depict the mean and standard deviation of the SRIP value and loss over five runs for 40 epochs.

Revised: The SRIP value can be tracked during training with the DG layer’s callback functionalities.

Original: First, we observe that the elementary domains are almost orthogonal when initialized.
Revised: First, we observe that the elementary domains are almost orthogonal when initialized.

Original: Training the layer leads in the first epochs to a decrease in orthogonality.
Revised: Training the layer leads in the first epochs to a decrease in orthogonality.

Original: This initial decrease happens because cross-entropy has a stronger influence on the optimization than regularization in the first epochs.
Revised: This initial decrease happens because cross-entropy has a stronger influence on the optimization than regularization in the first epochs.

Original: After five epochs, the cross-entropy decrease to a threshold when the regularization becomes more effective and the orthogonality of the elementary domain bases increases again.
Revised: After five epochs, the cross-entropy decrease to a threshold when the regularization becomes more effective and the orthogonality of the elementary domain bases increases again.

Original: In Figure 8, we also observe that ablating the orthogonal regularization, while leading to better orthogonality of the domains, does not significantly affect the overall cross-entropy during training.
Revised: In Figure 10, we also observe that ablating the orthogonal regularization, while leading to better orthogonality of the domains, does not significantly affect the overall cross-entropy during training.

Original: Finally, we project the output of the FE trained with a dense layer (ERM) and with the DG layer by t-SNE (t-distributed stochastic neighbor embedding) in Figure 9.
Revised: Finally, we project the output of the FE trained with a dense layer (ERM) and with the DG layer by t-SNE (t-distributed stochastic neighbor embedding) in Figure 11.

Original: The GDU-trained FE yields more concentrated and bounded clusters in comparison to the one trained by ERM.
Revised: The GDU-trained FE yields more concentrated and bounded clusters in comparison to the one trained by ERM.

Original: Hence, we observe a positive effect on the representation learned by the FE.
Revised: Hence, we observe a positive effect on the representation learned by the FE.

Original: C.2. I NTERPRETATION OF THE E LEMENTARY D OMAINS
Revised: C.2. I NTERPRETATION OF THE E LEMENTARY D OMAINS

Original: We analyze the learned elementary domains in the digits experiment based on two visualizations, and choose the maximum mean discrepancy (MMD) as the similarity measure and MNIST-M as the test domain.
Revised: We analyze the learned elementary domains in the digits experiment based on two visualizations, and choose the maximum mean discrepancy (MMD) as the similarity measure and MNIST-M as the test domain.

Original: The first visualization depicts the MMD between the datasets (i.e., MNIST , MNIST-M ,
Revised: The first visualization depicts the MMD between the datasets (i.e., MNIST , MNIST-M ,

Original: SVHN , USPS , and Synthetic Digits (SYN) ) and the learned elementary domains (i.e., V 1 − V 5 ) as a heatmap (see Figure 10 (left)).
Revised: SVHN , USPS , and Synthetic Digits (SYN) ) and the learned elementary domains (i.e., V 1 − V 5 ) as a heatmap (see Figure 12 (left)).

Original: The heatmap indicates that the source and test domains are close to one another in terms of the MMD.
Revised: The heatmap indicates that the source and test domains are close to one another in terms of the MMD.

Original: Hence, we expect that their closeness reflects in the learning of the elementary domains.
Revised: Hence, we expect that their closeness reflects in the learning of the elementary domains.

Original: In other words, we expect that each elementary domains contributes similarly to the source and test domains (i.e., the coefficients β are similar for each of these domains).
Revised: In other words, we expect that each elementary domains contributes similarly to the source and test domains (i.e., the coefficients β are similar for each of these domains).

Original: In Section 3.1, we derive the coefficients by applying a kernel softmax function to the negative MMD distances.
Revised: In Section 3.1, we derive the coefficients by applying a kernel softmax function to the negative MMD distances.

Original: Since the MMD distances between the source / test domains and the elementary domains are similar, the coefficients will be similar too.
Revised: Since the MMD distances between the source / test domains and the elementary domains are similar, the coefficients will be similar too.

Original: We conclude that the learned elementary domains represent the same distributional characteristics that existed among the source and test domains.
Revised: We conclude that the learned elementary domains represent the same distributional characteristics that existed among the source and test domains.

Original: In the second visualisation, we show the t-SNE (t-distributed stochastic neighbor embedding) of the feature extractor output for each source and test domain alongside the elementary domains in Figure 10 (right).
Revised: In the second visualisation, we show the t-SNE (t-distributed stochastic neighbor embedding) of the feature extractor output for each source and test domain alongside the elementary domains in Figure 12 (right).

Original: First, we observe that the learned elementary domain bases form distinctive clusters.
Revised: First, we observe that the learned elementary domain bases form distinctive clusters.

Original: We see these clusters as a validation of our hypothesis that each GDU learns to mimic samples generated from a corresponding elementary distribution as pointed out in Section 2.2.
Revised: We see these clusters as a validation of our hypothesis that each GDU learns to mimic samples generated from a corresponding elementary distribution as pointed out in Section 2.2.

Original: However, we can not answer whether and where these elementary distributions occur in the real world.
Revised: However, we can not answer whether and where these elementary distributions occur in the real world.

Original: Moreover, these elementary distributions yet lack interpretability.
Revised: Moreover, these elementary distributions yet lack interpretability.

Original: In summary, the MMD heatmap and t-SNE embeddings of the learned elementary and source domains on Figure 10 indicate that the GDUs learn to represent distributional structures in the dataset.
Revised: In summary, the MMD heatmap and t-SNE embeddings of the learned elementary and source domains on Figure 12 indicate that the GDUs learn to represent distributional structures in the dataset.

Original: C. ECG E XPERIMENT
Revised: C. ECG E XPERIMENT

Original: We adopted the task of multi-label binary classification of 12-lead electrocardiogram (ECG) signals combined from 6 different sources introduced in the PhysioNet/Computing in Cardiology Challenge 2020 4 ( Alday al al Alday  al., 
Revised: We adopted the task of multi-label binary classification of 12-lead electrocardiogram (ECG) signals combined from 6 different sources introduced in the PhysioNet/Computing in Cardiology Challenge 2020 5 (Perez Alday et al., 2021; Goldberger et al., 2000; Perez Alday et al., 2020).

Original: Each ECG [Figure] N I S T [Figure]
Revised: Each ECG recordings is annotated with 24 binary labels indicating whether or not a certain cardiac abnormality is present.

Original: N I S T M S V H N S Y N
Revised: The data is aggregated from 6 different databases and contains 43,101 recordings sampled with various sampling frequencies, number of subjects, and lengths.

Original: [Figure] S P S [Figure] [Table]
Revised: Table 10 summarizes most important details about the data sources for this experiment.

Original: [Figure] [Table] [Figure]
Revised: According to the original challenge score, we measure the performance in terms of the generalized Intersection-over-Union (IoU) score where partial credit is assigned to misdiagnoses that result in similar treatments or outcomes.

Original: SNEembeddingfor:
Revised: The score is defined as

Original: [Figure]
Revised: [Equation]

Original:  0 V 1 V 2 V [Figure]
Revised: where y, ˆ y ∈ { 0 , 1 } 24 represent actual labels and predicted labels and W stands for the partial credit-assignment matrix provided as a part of the challenge description. Note that in case of identity matrix W the score is exactly the Intersection-over-Union (IoU) score.

Original: [Table] [Figure]
Revised: The score is then adjusted for a solution y majority , which always predicts the normal/majority class, and is moreover normalized

Original: V  1 V [Table] 4 MNISTMNISTMSVHN [Table]
Revised: V 0 V 1 V 2 V 3 V 4 MNISTMNISTMSVHN SYN USPS

Original: [Figure]

Original: V V
Revised: V V

Original: V V
Revised: V V

Original: V
Revised: V

Original: M N I S T M N I S T M
Revised: M N I S T M N I S T M

Original: S V H N
Revised: S V H N

Original: S Y N
Revised: S Y N

Original: {+U S P S 0.141.4 1.61.6 1.3 1.71.7 1.6 0.68 1.60.34 0.57 0.71 1.3 1.20.
+}
Revised: {+U S P S 0.141.4 1.61.6 1.3 1.71.7 1.6 0.68 1.60.34 0.57 0.71 1.3 1.20.
+}

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: P S
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]

Original: [Table] [Figure]
Revised: [Table] [Table]

Original: recordings is annotated with 24 binary labels indicating whether or not a certain cardiac abnormality is present. The data is aggregated from 6 different databases and contains 43,101 recordings sampled with various sampling frequencies, number of subjects, and lengths. Table 10 summarizes most important details about the data sources for this experiment. According to the original challenge score, we measure the performance in terms of the generalized
Revised: 0.68 1.5 0.58 1.4 0.54 0.029 0.12 0.17 0.079 0 MMDheatmapfor:PROJECTED − − −     80 −60 −40 −20 0 20t-SNEembeddingfor:PROJECTED mnistmnistmsvhnsynusps V 0 V 1 V 2 V 3 V0.

Original: Intersection-over-Union (IoU) score where partial credit is assigned to misdiagnoses that result in similar treatments or outcomes. The score is defined as
Revised: MNIST-M

Original: [Equation]
Revised: [Table]

Original: where y, ˆ y ∈ { 0 , 1 } 24 represent actual labels and predicted labels and W stands for the partial credit-assignment matrix provided as a part of the challenge description. Note that in case of identity matrix W the score is exactly the Intersection-over-Union (IoU) score. The score is then adjusted for a solution y majority , which always predicts the normal/majority class, and is moreover normalized for the perfect solution y .
Revised: for the perfect solution y .

Original: Therefore, the final score can have negative values and the best possible score of 1 and is formalized as
Revised: Therefore, the final score can have negative values and the best possible score of 1 and is formalized as

Original: [Equation]
Revised: [Equation]

Original: As a pre-processing step, we down-sampled all the signals to 125 Hz and applied Z-score, random amplification and random stretching according to Vicar et al.
Revised: As a pre-processing step, we down-sampled all the signals to 125 Hz and applied Z-score, random amplification and random stretching according to Vicar et al.

Original: For that we partially adopted the code provided by the authors 5 .
Revised: For that we partially adopted the code provided by the authors 6 .

Original: Additionally, we cropped each signal to its first 15,000 points if the signal was too long (mostly applied to INCART database).
Revised: Additionally, we cropped each signal to its first 15,000 points if the signal was too long (mostly applied to INCART database).

Original: Each dataset was randomly split into train and validation parts with 3:1 ratio.
Revised: Each dataset was randomly split into train and validation parts with 3:1 ratio.

Original: During each experiment, we used the train splits of [Table] databases for training and utilized the validation splits of the training databases for early stopping.
Revised: During each experiment, we used the train splits ofdatabases for training and utilized the validation splits of the training databases for early stopping.

Original: The hold-out 6-th database was used for inference and testing only
Revised: The hold-out 6-th database was used for inference and testing only.

Original: [Table] Table 11 describes the architecture of FE used for the task.
Revised: Table 11 describes the architecture of FE used for the task.

Original: Since the provided ECG recordings have different lengths, we used TensorFlow padded batching, which is padding all the recordings in a batch to the length of the longest sequence in the batch.
Revised: Since the provided ECG recordings have different lengths, we used TensorFlow padded batching, which is padding all the recordings in a batch to the length of the longest sequence in the batch.

Original: Therefore, input from different batches can have different lengths so the spatial dimensions of the 1D-Convolutional layers are not predefined and are presented as *. 
Revised: Therefore, input from different batches can have different lengths so the spatial dimensions of the 1D-Convolutional layers are not predefined and are presented as *. variable lengths, therefore, the spatial dimension is not defined and denoted as *.

Original: [Table]
Revised: [Table]

Original: We used the Adam optimizer to optimize weighted binary cross-entropy loss defined as − ( w pos · y · log ˆ y )
Revised: We used the Adam optimizer to optimize weighted binary cross-entropy loss defined as − ( w pos · y · log ˆ y )

Original: + (1 − y ) · log (1 − ˆ y ) .
Revised: + (1 − y ) · log (1 − ˆ y ) .

Original: Positive weights w pos are defined per class based on the training split data inversely proportional to the frequency of positive labels for each class.
Revised: Positive weights w pos are defined per class based on the training split data inversely proportional to the frequency of positive labels for each class.

Original: A learning rate was initially set to 0.001 and during the training reduced by the factor of 0.2 if the training loss was not improving for 10 epochs.
Revised: A learning rate was initially set to 0.001 and during the training reduced by the factor of 0.2 if the training loss was not improving for 10 epochs.

Original: We also applied early stopping and restored model weights to the best model according to the validation accuracy after the training end.
Revised: We also applied early stopping and restored model weights to the best model according to the validation accuracy after the training end.

Original: Since each input samples for this experiment have a larger size than the previous one, we decreased the batch size to 64.
Revised: Since each input samples for this experiment have a larger size than the previous one, we decreased the batch size to 64.

Original: Each ECG experiment was performed on a single GPU (Nvidia GTX 1080 Ti).
Revised: Each ECG experiment was performed on a single GPU (Nvidia GTX 1080 Ti).

Original: The parameters relevant for the DG layer are summarized in Table 12.
Revised: The parameters relevant for the DG layer are summarized in Table 12.

Original: We have to emphasize that we did not perform extensive hyper-parameter tuning since our goal was to show the feasibility of the I.E.D. assumption and GDUs while keeping the experiments reproducible. 
Revised: We have to emphasize that we did not perform extensive hyper-parameter tuning since our goal was to show the feasibility of the I.E.D. assumption and GDUs while keeping the experiments reproducible. [Table]

Original: C. WILDS B ENCHMARKING E XPERIMENTS
Revised: C. WILDS B ENCHMARKING E XPERIMENTS

Original: For comparison of our approach and benchmarking, we followed the standard procedure of WILDS experiments, described in Koh et al.
Revised: For comparison of our approach and benchmarking, we followed the standard procedure of WILDS experiments, described in Koh et al.

Original: As a technical note, all WILDS experiments have been implemented in Pytorch (version > = 1.7.0) based on the specifications made in Koh et al.
Revised: As a technical note, all WILDS experiments have been implemented in Pytorch (version > = 1.7.0) based on the specifications made in Koh et al.

Original: (2021) and their code published on https://github.com/p-lambda/wilds [last accessed on 2022-05-17, available under MIT License].
Revised: (2021) and their code published on https://github.com/p-lambda/wilds [last accessed on 2022-05-17, available under MIT License].

Original: The results for the benchmarks were retrieved from the official leaderboard https://wilds.stanford.edu/leaderboard/ [last accessed on 2022-09-26].
Revised: The results for the benchmarks were retrieved from the official leaderboard https://wilds.stanford.edu/leaderboard/ [last accessed on 2022-09-26].

Original: Camelyon

Revised: Camelyon

Original: In medical applications, the goal is to apply models trained on a comparatively small set of hospitals to a larger number of hospitals.
Revised: In medical applications, the goal is to apply models trained on a comparatively small set of hospitals to a larger number of hospitals.

Original: For this application, we study images of tissue slides under a microscope to determine whether a patient has cancer or not.
Revised: For this application, we study images of tissue slides under a microscope to determine whether a patient has cancer or not.

Original: Shifts in patient populations, slide staining, and image acquisition can impede model accuracy in previously unseen hospitals.
Revised: Shifts in patient populations, slide staining, and image acquisition can impede model accuracy in previously unseen hospitals.

Original: [Table]

Original: Camelyon17 comprises images of tissue patches from five different hospitals.
Revised: Camelyon17 comprises images of tissue patches from five different hospitals.

Original: While the first three hospitals are the source domains (302,436 examples), the forth and fifth are the validation (34, examples) and test domain (85,054 examples), respectively.
Revised: While the first three hospitals are the source domains (302,436 examples), the forth and fifth are the validation (34, examples) and test domain (85,054 examples), respectively.

Original: We deviate from the specifications made in (Koh et al., 2021) in terms of the FE.
Revised: We deviate from the specifications made in (Koh et al., 2021) in terms of the FE.

Original: We use the FE from Feng et al.
Revised: We use the FE from Feng et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: (2019) since we observed a higher mean accuracy and faster training than with the by Koh et al.
Revised: (2019) since we observed a higher mean accuracy and faster training than with the by Koh et al.

Original: (2021) originally proposed DenseNet-121 FE (Huang et al., 2017).
Revised: (2021) originally proposed DenseNet-121 FE (Huang et al., 2017).

Original: We trained the FE from scratch.
Revised: We trained the FE from scratch.

Original: Both, ERM and the DG were trained over 250 epochs with early stopping, a learning rate of 0.001, which is reduced by a factor of 0.2 if the cross-entropy loss has not improved after 10 epochs.
Revised: Both, ERM and the DG were trained over 250 epochs with early stopping, a learning rate of 0.001, which is reduced by a factor of 0.2 if the cross-entropy loss has not improved after 10 epochs.

Original: All results were aggregated over ten runs.
Revised: All results were aggregated over ten runs.

Original: FMoW Analyzing satellite images with machine learning (ML) models may enable novel possibilities in tackling global sustainability and economic challenges such as population density mapping and deforestation tracking.
Revised: FMoW Analyzing satellite images with machine learning (ML) models may enable novel possibilities in tackling global sustainability and economic challenges such as population density mapping and deforestation tracking.

Original: However, satellite imagery changes over time due to human behavior (e.g., infrastructure development), and the extent of change is different in each region.
Revised: However, satellite imagery changes over time due to human behavior (e.g., infrastructure development), and the extent of change is different in each region.

Original: The Functional
Revised: The Functional

Original: Map of the World (FMoW) dataset consists of satellite images from different continents and years: training (76,863 images; between 2002–2013), validation (19,915 images; between 2013 and 2016), and test (22,108 images, between 2016–2017).
Revised: Map of the World (FMoW) dataset consists of satellite images from different continents and years: training (76,863 images; between 2002–2013), validation (19,915 images; between 2013 and 2016), and test (22,108 images, between 2016–2017).

Original: The objective is to determine one of 62 building types (e.g., shopping malls) and land-use.
Revised: The objective is to determine one of 62 building types (e.g., shopping malls) and land-use.

Original: As instructed in Koh et al.
Revised: As instructed in Koh et al.

Original: (2021), we used the DenseNet-121 pre-trained on ImageNet without
Revised: (2021), we used the DenseNet-121 pre-trained on ImageNet without

Original: L2-regularization.
Revised: L2-regularization.

Original: For the optimization, we use the Adam optimizer with a learning rate of 1e-4, which is decayed by a factor of 0.96 per epoch.
Revised: For the optimization, we use the Adam optimizer with a learning rate of 1e-4, which is decayed by a factor of 0.96 per epoch.

Original: The models were trained for 50 epochs with early stopping and a batch size of 64.
Revised: The models were trained for 50 epochs with early stopping and a batch size of 64.

Original: Additionally, we report the worst-region accuracy, which is a specific metric used for FMoW.
Revised: Additionally, we report the worst-region accuracy, which is a specific metric used for FMoW.

Original: This worst-region accuracy reports the worst accuracy across the following regions: Asia, Europe, Africa, America, and Oceania (see Koh et al.
Revised: This worst-region accuracy reports the worst accuracy across the following regions: Asia, Europe, Africa, America, and Oceania (see Koh et al.

Original: (2021) for the details).
Revised: (2021) for the details).

Original: Again, we report the results over three runs.
Revised: Again, we report the results over three runs.

Original: Amazon Recent research shows that consumer-facing machine learning application large performance disparities across different set of users.
Revised: Amazon Recent research shows that consumer-facing machine learning application large performance disparities across different set of users.

Original: To study this performance disparities, WILDS (Koh et al., 2021) leverages a variant of the Amazon Review dataset.
Revised: To study this performance disparities, WILDS (Koh et al., 2021) leverages a variant of the Amazon Review dataset.

Original: The Aamazon-WILDS dataset is composed of data from 3,920 domains (number of reviewers) and the task is a multi-class sentiment classification, where the model receives a review text and has to predict the rating from one to five.
Revised: The Aamazon-WILDS dataset is composed of data from 3,920 domains (number of reviewers) and the task is a multi-class sentiment classification, where the model receives a review text and has to predict the rating from one to five.

Original: To split this dataset, a between training, validation, and test disjoint set of reviewers is used: training (245,502 reviews from 1,252 reviewers), validation (100,050 reviews from 1,334 reviewers), test (100,050 reviews from 1,334 reviewers).
Revised: To split this dataset, a between training, validation, and test disjoint set of reviewers is used: training (245,502 reviews from 1,252 reviewers), validation (100,050 reviews from 1,334 reviewers), test (100,050 reviews from 1,334 reviewers).

Original: For the experiments and baseline models, we use the specifications made in Koh et al.
Revised: For the experiments and baseline models, we use the specifications made in Koh et al.

Original: As for the FE, we used DistilBERT-base-uncased models.
Revised: As for the FE, we used DistilBERT-base-uncased models.

Original: For ERM, we use a batch size of 8, learning rate 1e-5, L2 regularization of 0.01, 3 epochs with early stopping and 512 as the maximum length of tokens.
Revised: For ERM, we use a batch size of 8, learning rate 1e-5, L2 regularization of 0.01, 3 epochs with early stopping and 512 as the maximum length of tokens.

Original: For training the DG layer, we used the same specifications as made for ERM.
Revised: For training the DG layer, we used the same specifications as made for ERM.

Original: The performance is measured in 10th percentile accuracy.
Revised: The performance is measured in 10th percentile accuracy.

Original: iWildsCam Wildlife camera traps offer an excellent possibility to understand and monitor biodiversity loss.
Revised: iWildsCam Wildlife camera traps offer an excellent possibility to understand and monitor biodiversity loss.

Original: However, images from different camera traps differ in illumination, color, camera angle, background, vegetation, and relative animal frequencies.
Revised: However, images from different camera traps differ in illumination, color, camera angle, background, vegetation, and relative animal frequencies.

Original: We use the iWildsCam dataset consisting of 323 different camera traps positioned in different locations worldwide.
Revised: We use the iWildsCam dataset consisting of 323 different camera traps positioned in different locations worldwide.

Original: In the dataset, we refer to different locations of camera traps as different domains, in particular 243 training traps (129, images), 32 validation traps (14,961 images), and 48 test traps (42,791 images).
Revised: In the dataset, we refer to different locations of camera traps as different domains, in particular 243 training traps (129, images), 32 validation traps (14,961 images), and 48 test traps (42,791 images).

Original: The objective is to classify one of 182 animal species.
Revised: The objective is to classify one of 182 animal species.

Original: Following the instructions by Koh et al.
Revised: Following the instructions by Koh et al.

Original: (2021), we used again the ResNet50 pre-trained on ImagNet (He et al., 2016).
Revised: (2021), we used again the ResNet50 pre-trained on ImagNet (He et al., 2016).

Original: For ERM, we used a learning rate of 3e-5 and no L2-regularization.
Revised: For ERM, we used a learning rate of 3e-5 and no L2-regularization.

Original: The models were trained for 12 epochs with a batch size of 16 with the Adam optimizer.
Revised: The models were trained for 12 epochs with a batch size of 16 with the Adam optimizer.

Original: In addition to the accuracy, we report the macro F1-score to evaluate the performance on rare species (see Koh et al.
Revised: In addition to the accuracy, we report the macro F1-score to evaluate the performance on rare species (see Koh et al.

Original: (2021) for details).
Revised: (2021) for details).

Original: All results were aggregated over three runs.
Revised: All results were aggregated over three runs.

Original: RxRx

Revised: RxRx

Original: In biomedical research areas such as genomics or drug discovery, high-throughput screening techniques generate a vast amount of data in several batches.
Revised: In biomedical research areas such as genomics or drug discovery, high-throughput screening techniques generate a vast amount of data in several batches.

Original: Because experimental designs cannot fully mitigate the effects of confounding variables like temperature, humidity, and measurements across batches, this creates heterogeneity in the observed datasets (commonly known as batch effect).
Revised: Because experimental designs cannot fully mitigate the effects of confounding variables like temperature, humidity, and measurements across batches, this creates heterogeneity in the observed datasets (commonly known as batch effect).

Original: The RxRx1 dataset comprises images obtained by fluorescent microscopy from 51 domains (disjoint experiments): training (40,612 images, 33 domains), validation (9,854 images, 4 domains), and test (34,432 images, 14 domains).
Revised: The RxRx1 dataset comprises images obtained by fluorescent microscopy from 51 domains (disjoint experiments): training (40,612 images, 33 domains), validation (9,854 images, 4 domains), and test (34,432 images, 14 domains).

Original: The aim is to classify one of 1,139 genetic treatments.
Revised: The aim is to classify one of 1,139 genetic treatments.

Original: All results were aggregated over three runs.
Revised: All results were aggregated over three runs.

Original: We conducted the RxRx1 experiments in accordance with the specifications made in (Koh et al., 2021).
Revised: We conducted the RxRx1 experiments in accordance with the specifications made in (Koh et al., 2021).

Original: As for the FE, we, thus, used the ResNet50 pre-trained on ImagNet (He et al., 2016).
Revised: As for the FE, we, thus, used the ResNet50 pre-trained on ImagNet (He et al., 2016).

Original: We trained the models using AdamW with default parameters β 1 = 0 .
Revised: We trained the models using AdamW with default parameters β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 using a learning rate of 1e-4 and a L2-regularization with strength 1e-5 for 90 epochs with a batch size of 75.
Revised: 999 using a learning rate of 1e-4 and a L2-regularization with strength 1e-5 for 90 epochs with a batch size of 75.

Original: We scheduled the learning rate to linearly increase in the first ten epochs and then decreased it following a cosine rate.
Revised: We scheduled the learning rate to linearly increase in the first ten epochs and then decreased it following a cosine rate.

Original: For trainingthe DG layer, we chose the same parameters as for the ERM.
Revised: For trainingthe DG layer, we chose the same parameters as for the ERM.

Original: All results were aggregated over three runs.
Revised: All results were aggregated over three runs.

Original: OBG-MolPCBA In biomedical research, machine learning has the potential to accelerate drug discovery while reducing the experimental overhead due to lowering the number of experiments required.
Revised: OBG-MolPCBA In biomedical research, machine learning has the potential to accelerate drug discovery while reducing the experimental overhead due to lowering the number of experiments required.

Original: However, to leverage the potential of machine learning, the models need to generalize to molecules structurally different from those seen during training.
Revised: However, to leverage the potential of machine learning, the models need to generalize to molecules structurally different from those seen during training.

Original: To study this OOD generalization across molecule scaffolds, we use the OGB-MolPCBA dataset.
Revised: To study this OOD generalization across molecule scaffolds, we use the OGB-MolPCBA dataset.

Original: This dataset is split into the following subsets according to the scaffold structure: training (44,930 domains), validation (31,361 domains), and test (43,739 domains).
Revised: This dataset is split into the following subsets according to the scaffold structure: training (44,930 domains), validation (31,361 domains), and test (43,739 domains).

Original: The task is to classify the presence/absence of 128 biological activities based on a graph representation of a molecule.
Revised: The task is to classify the presence/absence of 128 biological activities based on a graph representation of a molecule.

Original: In line with Koh et al. (2021), we use a Graph Isomorphism Network (GIN) combined with virtual nodes as the FE.
Revised: In line with Koh et al. (2021), we use a Graph Isomorphism Network (GIN) combined with virtual nodes as the FE.

Original: For training ERM and our DG, we use the default parameters: five GNN layers with a dimensionality of 300 and a learning rate of 0.001.
Revised: For training ERM and our DG, we use the default parameters: five GNN layers with a dimensionality of 300 and a learning rate of 0.001.

Original: We train for 100 epochs using early stopping.
Revised: We train for 100 epochs using early stopping.

Original: As for the performance, we report the mean and standard deviation of the average precision across all scaffolds (domains) over three runs.
Revised: As for the performance, we report the mean and standard deviation of the average precision across all scaffolds (domains) over three runs.

Original: CivilComments
Revised: CivilComments

Original: In the last decades, users have generated a vast amount of text on the Internet, some of which contain toxic comments.
Revised: In the last decades, users have generated a vast amount of text on the Internet, some of which contain toxic comments.

Original: Machine learning has been leveraged for automatic text review to flag toxic comments.
Revised: Machine learning has been leveraged for automatic text review to flag toxic comments.

Original: However, the models are prone to learn spurious correlations between toxicity and information on demographics in the comment, which causes the model performance to drop in specific subpopulations.
Revised: However, the models are prone to learn spurious correlations between toxicity and information on demographics in the comment, which causes the model performance to drop in specific subpopulations.

Original: To study this OOD task, we leverage the modified CivilComment dataset from Koh et al.
Revised: To study this OOD task, we leverage the modified CivilComment dataset from Koh et al.

Original: Based on text input, the task is to predict a binary label, toxic or non-toxic.
Revised: Based on text input, the task is to predict a binary label, toxic or non-toxic.

Original: The domains are defined according to eight demographic identities: male, female, LGBTQ,
Revised: The domains are defined according to eight demographic identities: male, female, LGBTQ,

Original: Christian, Muslim, other religions, Black, and White.
Revised: Christian, Muslim, other religions, Black, and White.

Original: All comments were randomly split into a disjoint training (269,038 comments), validation (45,180 comments), and test (133,782 comments) set.
Revised: All comments were randomly split into a disjoint training (269,038 comments), validation (45,180 comments), and test (133,782 comments) set.

Original: Again, we follow Koh et al.
Revised: Again, we follow Koh et al.

Original: and use a DistillBERT-base-uncased model with the following parameters: batch size = 16, learning rate
Revised: and use a DistillBERT-base-uncased model with the following parameters: batch size = 16, learning rate

Original: = 1e-5, AdamW optimizer, number of epochs = 5, L regularization 0.01, and the maximum number of tokens of 300.
Revised: = 1e-5, AdamW optimizer, number of epochs = 5, L regularization 0.01, and the maximum number of tokens of 300.

Original: We use these default parameters for training our DG layer.
Revised: We use these default parameters for training our DG layer.

Original: The performance is measured in the worst-group accuracy and we report mean and standard deviation across five runs.
Revised: The performance is measured in the worst-group accuracy and we report mean and standard deviation across five runs.

Original: PovertyMap As the FMoW example shows, satellite images in combination with machine learning models can been used to monitor sustainability and economic challenges on a global scale.
Revised: PovertyMap As the FMoW example shows, satellite images in combination with machine learning models can been used to monitor sustainability and economic challenges on a global scale.

Original: Another application of these satellite images is poverty estimation across different spatial regions.
Revised: Another application of these satellite images is poverty estimation across different spatial regions.

Original: However, there exists a lack of labels for developing countries since obtaining the ground truth is expensive, which makes this application attractive for machine learning models.
Revised: However, there exists a lack of labels for developing countries since obtaining the ground truth is expensive, which makes this application attractive for machine learning models.

Original: To study the OOD generalization to unseen countries, we use a modified version of the poverty mapping dataset of WILDS (Koh et al., 2021).
Revised: To study the OOD generalization to unseen countries, we use a modified version of the poverty mapping dataset of WILDS (Koh et al., 2021).

Original: The task is to predict a real-valued aset wealth index between 1 and 5 based on a multi-spectral satellite image.
Revised: The task is to predict a real-valued aset wealth index between 1 and 5 based on a multi-spectral satellite image.

Original: The domain refers to the country and whether the the the image is from a rural or urban are.
Revised: The domain refers to the country and whether the the the image is from a rural or urban are.

Original: In contrast to the other datasets, this dataset is split in five different folds, whereby in each fold the the training, valdiation and test set contains a disjoint set of countries, however, data from both rural and urban regions.
Revised: In contrast to the other datasets, this dataset is split in five different folds, whereby in each fold the the training, valdiation and test set contains a disjoint set of countries, however, data from both rural and urban regions.

Original: The avergae size of each set across the 5 folds is for the training ˜10, images (13-14 countries), ˜4,000 images (4-5 different countries), and for the test set ˜4,000 images (13-14 countries).
Revised: The avergae size of each set across the 5 folds is for the training ˜ 10, images (13-14 countries), ˜ 4,000 images (4-5 different countries), and for the test set ˜ 4,000 images (13-14 countries).

Original: On the challenge of obtaining domain labels.
Revised: On the challenge of obtaining domain labels.

Original: In the example of hospitals (e.g. Camelyon dataset), domain labels come, in fact, for free.
Revised: In the example of hospitals (e.g. Camelyon dataset), domain labels come, in fact, for free.

Original: However, other examples, such as the CivilComments dataset, show the opposite.
Revised: However, other examples, such as the CivilComments dataset, show the opposite.

Original: This dataset requires additional annotations (i.e., demographic identities), which can be tedious to obtain in practice.
Revised: This dataset requires additional annotations (i.e., demographic identities), which can be tedious to obtain in practice.

Original: Some algorithms need these domain annotations to achieve superior performance on each subgroup.
Revised: Some algorithms need these domain annotations to achieve superior performance on each subgroup.

Original: Furthermore, the task of subgroup detection in itself is a difficult and relevant problem.
Revised: Furthermore, the task of subgroup detection in itself is a difficult and relevant problem.

Original: Coming back to our hospital example, even people from the same hospital might belong to different subpopulation (e.g. gender, race, age) and these demographic subgroups are often more relevant for diagnosis than which hospital a patient comes from.
Revised: Coming back to our hospital example, even people from the same hospital might belong to different subpopulation (e.g. gender, race, age) and these demographic subgroups are often more relevant for diagnosis than which hospital a patient comes from.

Original: This information, however, is not always available (due to anonymization standards, for instance) and, therefore, the relevant domain annotation might be hard to obtain.
Revised: This information, however, is not always available (due to anonymization standards, for instance) and, therefore, the relevant domain annotation might be hard to obtain.

Original: We follow Koh et al.
Revised: We follow Koh et al.

Original: and use a pre-trained ResNet-18 model minimizing the sqarred error loss.
Revised: and use a pre-trained ResNet-18 model minimizing the sqarred error loss.

Original: For the optimization, we rely on the Adam optimizer with the following parameters: learning rate of 1e-3 with a decay of 0.96 per epoch, batch size of 64 and early stopping based on the OOD evaluation score.
Revised: For the optimization, we rely on the Adam optimizer with the following parameters: learning rate of 1e-3 with a decay of 0.96 per epoch, batch size of 64 and early stopping based on the OOD evaluation score.

Original: For evaluation, we report the Pearson correlation (r) between the predicted and actual asset wealth indices across the five different folds.
Revised: For evaluation, we report the Pearson correlation (r) between the predicted and actual asset wealth indices across the five different folds.

Original: General benchmark methods Following the WILDS benchmarking procedure (Koh et al., 2021), we compare our proposed DG layer to the following baselines.
Revised: General benchmark methods Following the WILDS benchmarking procedure (Koh et al., 2021), we compare our proposed DG layer to the following baselines.

Original: First, empirical risk minimization (ERM), which minimizes the average training loss over the pooled dataset.
Revised: First, empirical risk minimization (ERM), which minimizes the average training loss over the pooled dataset.

Original: Second, a group of
Revised: Second, a group of

Original: DG algorithms provided by the WILDS benchmark, namely, Coral, Fish, IRM, and DRO.
Revised: DG algorithms provided by the WILDS benchmark, namely, Coral, Fish, IRM, and DRO.

Original: The
Revised: The

Original: Coral algorithm introduces a penalty for differences in means and covariances of the domains feature distributions.
Revised: Coral algorithm introduces a penalty for differences in means and covariances of the domains feature distributions.

Original: The Fish algorithm achieves DG by approximating an inter-domain gradient matching objective, i.e., maximizing the inner product between gradients from different domains (Shi et al., 2021).
Revised: The Fish algorithm achieves DG by approximating an inter-domain gradient matching objective, i.e., maximizing the inner product between gradients from different domains (Shi et al., 2021).

Original: Conceptually, Fish learns feature representations that are invariant across domains.
Revised: Conceptually, Fish learns feature representations that are invariant across domains.

Original: Invariant risk minimization (IRM) introduces a penalty for feature distributions with different optimal classifiers for each domain (Arjovsky et al., 2019).
Revised: Invariant risk minimization (IRM) introduces a penalty for feature distributions with different optimal classifiers for each domain (Arjovsky et al., 2019).

Original: The idea is to enable OOD generalization by learning domain-invariant causal predictors.
Revised: The idea is to enable OOD generalization by learning domain-invariant causal predictors.

Original: Lastly, group distributionally robust optimization (DRO) explicitly minimizes the training loss on the worst-case domain (Sagawa et al., 2020; Hu et al., 2018).
Revised: Lastly, group distributionally robust optimization (DRO) explicitly minimizes the training loss on the worst-case domain (Sagawa et al., 2020; Hu et al., 2018).

Original: In addition to the baselines originally presented in Koh et al.
Revised: In addition to the baselines originally presented in Koh et al.

Original: (2021), we consider the following more recent DG baselines.
Revised: (2021), we consider the following more recent DG baselines.

Original: First, we describe LISA, which instead of regularizing the internal representations for generalization, seeks to learn domain-invariant predictors with selective data  augmentation ? .
Revised: First, we describe LISA, which instead of regularizing the internal representations for generalization, seeks to learn domain-invariant predictors with selective data [Table] augmentation Yao et al.

Original: Common Gradient Descent (CGD), introduced by Piratla et al.
Revised: Common Gradient Descent (CGD), introduced by Piratla et al.

Original: (2021), is based on Group-DRO.
Revised: (2021), is based on Group-DRO.

Original: However, it proposes to focus not on groups with the worst regularization but on common groups that enable generalization.
Revised: However, it proposes to focus not on groups with the worst regularization but on common groups that enable generalization.

Original: Last, Adaptive Risk Minimization using batch normalization (ARM-BN) by ? is different from the methods presented since it adapts to previously unseen domains during test time using unlabeled observations from this test domain.
Revised: Last, Adaptive Risk Minimization using batch normalization (ARM-BN) by Zhang et al. (2021) is different from the methods presented since it adapts to previously unseen domains during test time using unlabeled observations from this test domain.

Original: [Table] [Table][Table]
Revised: [Table] [Table]