Original: I DENTIFYING L ATENT C AUSAL C ONTENT FOR M ULTIS OURCE D OMAIN A DAPTATION
Revised: I DENTIFYING L ATENT C AUSAL C ONTENT FOR M ULTIS OURCE D OMAIN A DAPTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multi-source domain adaptation (MSDA) learns to predict the labels in target domain data, under the setting that data from multiple source domains are labelled and data from the target domain are unlabelled.
Revised: Multi-source domain adaptation (MSDA) learns to predict the labels in target domain data, under the setting that data from multiple source domains are labelled and data from the target domain are unlabelled.

Original: Most methods for this task focus on learning invariant representations across domains.
Revised: Most methods for this task focus on learning invariant representations across domains.

Original: However, their success relies heavily on the assumption that the label distribution remains consistent across domains, which may not hold in general real-world problems.
Revised: However, their success relies heavily on the assumption that the label distribution remains consistent across domains, which may not hold in general real-world problems.

Original: In this paper, we propose a new and more flexible assumption, termed latent covariate shift , where a latent content variable z c and a latent style variable z s are introduced in the generative process, with the marginal distribution of z c changing across domains and the conditional distribution of the label given z c remaining invariant across domains.
Revised: In this paper, we propose a new and more flexible assumption, termed latent covariate shift , where a latent content variable z c and a latent style variable z s are introduced in the generative process, with the marginal distribution of z c changing across domains and the conditional distribution of the label given z c remaining invariant across domains.

Original: We show that although (completely) identifying the proposed latent causal model is challenging, the latent content variable can be identified up to scaling by using its dependence with labels from source domains, together with the identifiability conditions of nonlinear ICA.
Revised: We show that although (completely) identifying the proposed latent causal model is challenging, the latent content variable can be identified up to scaling by using its dependence with labels from source domains, together with the identifiability conditions of nonlinear ICA.

Original: This motivates us to propose a novel method for MSDA, which learns the invariant label distribution conditional on the latent content variable, instead of learning invariant representations.
Revised: This motivates us to propose a novel method for MSDA, which learns the invariant label distribution conditional on the latent content variable, instead of learning invariant representations.

Original: Empirical evaluation on simulation and real data demonstrates the effectiveness of the proposed method.
Revised: Empirical evaluation on simulation and real data demonstrates the effectiveness of the proposed method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Traditional machine learning requires the training and testing data to be independent and identically distributed (Vapnik, 1999).
Revised: Traditional machine learning requires the training and testing data to be independent and identically distributed (Vapnik, 1999).

Original: This strict assumption may not be fulfilled in various potential real-world applications.
Revised: This strict assumption may not be fulfilled in various potential real-world applications.

Original: For example, in medical applications, it is common to seek to train a model on patients from a few hospitals and generalize it to a new hospital (Zech et al., 2018).
Revised: For example, in medical applications, it is common to seek to train a model on patients from a few hospitals and generalize it to a new hospital (Zech et al., 2018).

Original: In this case, it is often reasonable to consider that the distributions of data from training hospitals are different from the new hospital (Koh et al., 2021).
Revised: In this case, it is often reasonable to consider that the distributions of data from training hospitals are different from the new hospital (Koh et al., 2021).

Original: Domain adaptation is a promising research area to handle such problems.
Revised: Domain adaptation is a promising research area to handle such problems.

Original: In this work, we focus on multi-source DA (MSDA) settings where source domain data are collected from multiple domains.
Revised: In this work, we focus on multi-source DA (MSDA) settings where source domain data are collected from multiple domains.

Original: Formally, let x denote the input, e.g. image, y denote the labels in source and target domains, and D denote the domain index.
Revised: Formally, let x denote the input, e.g. image, y denote the labels in source and target domains, and D denote the domain index.

Original: We observe labeled data pairs ( x S , y S ) from the multiple joint distributions p ( x , y | D = 1) , ..., p ( x , y | D
Revised: We observe labeled data pairs ( x S , y S ) from the multiple joint distributions p ( x , y | D = 1) , ..., p ( x , y | D

Original: = m ) , ..., p ( x , y | D = M ) in source domains, and unlabeled input data samples x T from the joint distribution p ( x , y | D T ) in the target domain.
Revised: = m ) , ..., p ( x , y | D = M ) in source domains, and unlabeled input data samples x T from the joint distribution p ( x , y | D T ) in the target domain.

Original: The training phase of MSDA is to use the sets of ( x S , y S ) and x T , to train a predictor so that it can provide a satisfactory estimation for y T in the target domain.
Revised: The training phase of MSDA is to use the sets of ( x S , y S ) and x T , to train a predictor so that it can provide a satisfactory estimation for y T in the target domain.

Original: The key for MSDA is to understand how the joint distribution p D ( x , y ) change across all different source and target domains.
Revised: The key for MSDA is to understand how the joint distribution p D ( x , y ) change across all different source and target domains.

Original: Most early methods assume that the change of the joint distribution results from Covariate Shift (Huang et al., 2006; Bickel et al., 2007;
Revised: Most early methods assume that the change of the joint distribution results from Covariate Shift (Huang et al., 2006; Bickel et al., 2007;

Original: Sugiyama et al., 2007; Wen et al., 2014), e.g. , p D ( x , y )
Revised: Sugiyama et al., 2007; Wen et al., 2014), e.g. , p D ( x , y )

Original: =
Revised: =

Original: p D ( y | x ) p D ( x ) , as depicted by Figure 1(a).
Revised: p D ( y | x ) p D ( x ) , as depicted by Figure 1(a).

Original: This setting assumes that p D ( x ) changes across domains, while the conditional distribution p D ( y | x ) is invariant across domains.
Revised: This setting assumes that p D ( x ) changes across domains, while the conditional distribution p D ( y | x ) is invariant across domains.

Original: Such assumption may not always hold for some real applications, e.g. , image classification.
Revised: Such assumption may not always hold for some real applications, e.g. , image classification.

Original: For example, the assumption of invariant p D ( y | x ) implies that p D ( y ) should change as p D ( x ) changes.
Revised: For example, the assumption of invariant p D ( y | x ) implies that p D ( y ) should change as p D ( x ) changes.

Original: However, we can easily change style information ( e.g. , hue, view) in the images to change p D ( x ) and keep p D ( y ) unchanged, which is common in classification but violates the assumption.
Revised: However, we can easily change style information ( e.g. , hue, view) in the images to change p D ( x ) and keep p D ( y ) unchanged, which is common in classification but violates the assumption.

Original: In contrast to covariate shift, most current works consider Conditional Shift as depicted by Figure 1(b).
Revised: In contrast to covariate shift, most current works consider Conditional Shift as depicted by Figure 1(b).

Original: It assumes that the conditional p D ( x | y ) changes while p D ( y ) is invariant across domains
Revised: It assumes that the conditional p D ( x | y ) changes while p D ( y ) is invariant across domains

Original: [Figure] (Zhang et al., 2013; 2015; Sch¨olkopf et al., 2012; Stojanov et al., 2021; Peng et al., 2019).
Revised: (Zhang et al., 2013; 2015; Sch¨olkopf et al., 2012; Stojanov et al., 2021; Peng et al., 2019).

Original: This situation motivates a popular class of methods focusing on learning invariant representations across domains to approach the latent content variable z c in Figure 1(b) (Ganin et al., 2016;
Revised: This situation motivates a popular class of methods focusing on learning invariant representations across domains to approach the latent content variable z c in Figure 1(b) (Ganin et al., 2016;

Original: Zhao et al., 2018;
Revised: Zhao et al., 2018;

Original: Saito et al., 2018; Mancini et al., 2018;
Revised: Saito et al., 2018; Mancini et al., 2018;

Original: Yang et al., 2020; Wang et al., 2020; Li et al., 2021;
Revised: Yang et al., 2020; Wang et al., 2020; Li et al., 2021;

Original: Stojanov et al., 2021).
Revised: Stojanov et al., 2021).

Original: However, the label distribution p D ( y ) may change across domains in many real application scenarios (Tachet des Combes et al., 2020; Lipton et al., 2018; Zhang et al., 2013), for which learning invariant representations may be resulting in degenerating performance.
Revised: However, the label distribution p D ( y ) may change across domains in many real application scenarios (Tachet des Combes et al., 2020; Lipton et al., 2018; Zhang et al., 2013), for which learning invariant representations may be resulting in degenerating performance.

Original: In theory, there exists an upper bound on the performance of learning invariant representations when label distribution changes across domains (Zhao et al., 2019). 
Revised: In theory, there exists an upper bound on the performance of learning invariant representations when label distribution changes across domains (Zhao et al., 2019). [Figure]

Original: In some real-world scenarios, the label distribution p D ( y ) and the corresponding p D ( z c ) may vary across domains.
Revised: In some real-world scenarios, the label distribution p D ( y ) and the corresponding p D ( z c ) may vary across domains.

Original: For example, in image classification, z c represents the essential visual semantic features for discriminating classes ( e.g. , fur texture or color for animal classification) may vary across domains ( e.g. , geographical location) but are also stably associated to specific domains.
Revised: For example, in image classification, z c represents the essential visual semantic features for discriminating classes ( e.g. , fur texture or color for animal classification) may vary across domains ( e.g. , geographical location) but are also stably associated to specific domains.

Original: In such scenarios, if all domain variant features are excluded from z c , z c will be non-informative for deciding y .
Revised: In such scenarios, if all domain variant features are excluded from z c , z c will be non-informative for deciding y .

Original: On the other hand, since such varying features are associated with and covariant with the domain, they can be usable and useful in MSDA.
Revised: On the other hand, since such varying features are associated with and covariant with the domain, they can be usable and useful in MSDA.

Original: To handle this scenario, we propose a new assumption, Latent Covariate Shift (LCS) , as depicted by Figure 1(c).
Revised: To handle this scenario, we propose a new assumption, Latent Covariate Shift (LCS) , as depicted by Figure 1(c).

Original: Unlike the conditional shift,
Revised: Unlike the conditional shift,

Original: LCS assumes that there is a latent content variable z c , whose distribution p D ( z c ) changes across domains.
Revised: LCS assumes that there is a latent content variable z c , whose distribution p D ( z c ) changes across domains.

Original: Meanwhile the label conditional distribution p D ( y | z c ) is invariant.
Revised: Meanwhile the label conditional distribution p D ( y | z c ) is invariant.

Original: By combing these assumptions, LCS enables the label distribution to change across domains.
Revised: By combing these assumptions, LCS enables the label distribution to change across domains.

Original: To understand more deeply and handle LCS, we propose a latent causal model to formulate the data and label generating process, by introducing the latent style variable z s to complement z c as depicted in Figure 2.
Revised: To understand more deeply and handle LCS, we propose a latent causal model to formulate the data and label generating process, by introducing the latent style variable z s to complement z c as depicted in Figure 2.

Original: To analyse the identifiability of the proposed causal model, we also introduce latent noise variables n c and n s , which represent some unmeasured factors influencing z c and z s , respectively.
Revised: To analyse the identifiability of the proposed causal model, we also introduce latent noise variables n c and n s , which represent some unmeasured factors influencing z c and z s , respectively.

Original: As a result, we can leverage recent progress in the identifiability result of nonlinear
Revised: As a result, we can leverage recent progress in the identifiability result of nonlinear

Original: ICA (Hyvarinen et al., 2019; Khemakhem et al., 2020) to analyse the identifiability of the proposed latent causal model.
Revised: ICA (Hyvarinen et al., 2019; Khemakhem et al., 2020) to analyse the identifiability of the proposed latent causal model.

Original: We show that although completely identifying the proposed latent causal model is often not possible without further assumptions due to transitivity in latent space, partially identifying the latent content variables z c up to scaling is tractable, by integrating the identifiability result of nonlinear ICA with the dependence between n c and y .
Revised: We show that although completely identifying the proposed latent causal model is often not possible without further assumptions due to transitivity in latent space, partially identifying the latent content variables z c up to scaling is tractable, by integrating the identifiability result of nonlinear ICA with the dependence between n c and y .

Original: This motivates us to propose a novel method to learn the invariant conditional distribution p D ( y | z c ) for LCS, instead of learning invariant representations.
Revised: This motivates us to propose a novel method to learn the invariant conditional distribution p D ( y | z c ) for LCS, instead of learning invariant representations.

Original: Relying on the guaranteed identifiability on z c , the proposed method provides a principled way to ensure that the covariant z c can be identified on the target domain data, and the learned predictor p D ( y | z c ) can generalize to the target domain.
Revised: Relying on the guaranteed identifiability on z c , the proposed method provides a principled way to ensure that the covariant z c can be identified on the target domain data, and the learned predictor p D ( y | z c ) can generalize to the target domain.

Original: Empirical evaluation on synthetic and real data demonstrates the effectiveness of the proposed method, compared with state-of-the-art methods.
Revised: Empirical evaluation on synthetic and real data demonstrates the effectiveness of the proposed method, compared with state-of-the-art methods.

Revised: Overall, our main contributions can be summarized as follows: (i) Differ from the commonly-used

Revised: Conditional Shift as shown in Figure 1 (b), which assume label distribution to be the same across domains, we propose a new problem setting, latent covariate shift , as shown in Figure 1 (c).

Revised: (ii)

Original:   R ELATED WORK
Revised: We propose a latent causal model for latent covariate shift . Leveraging the existing identifiability results of nonlinear ICA, we provide an analysis about the identifiability of the proposed latent causal graph, which provides guarantee for identifying the latent causal content variable z c . (iii) Under the identifiability result, we design a new method for domain adaptation, and empirically evaluate the proposed method on simulation and real data.  R ELATED WORK

Original: Learning invariant representations .
Revised: Learning invariant representations .

Original: Due to the limitation of covariate shift in image data, most current works for domain adaptation consider the conditional shift, which learns invariant representations across domains (Ganin et al., 2016;
Revised: Due to the limitation of covariate shift in image data, most current works for domain adaptation consider the conditional shift, which learns invariant representations across domains (Ganin et al., 2016;

Original: Zhao et al., 2018;
Revised: Zhao et al., 2018;

Original: Saito et al., 2018; Mancini et al., 2018;
Revised: Saito et al., 2018; Mancini et al., 2018;

Original: Yang et al., 2020; Wang et al., 2020; Li et al.,  2021).
Revised: Yang et al., 2020; Wang et al., 2020; Li et al., 2021; Wang et al., 2022b; Zhao et al., 2021).

Original: Such invariant representations can be obtained by applying suitable linear or nonlinear transformations on the input data.
Revised: Such invariant representations can be obtained by applying suitable linear or nonlinear transformations on the input data.

Original: The key of these methods is how to enforce the invariance of the learned representations.
Revised: The key of these methods is how to enforce the invariance of the learned representations.

Original: For example, the invariance can be enforced by maximum classifier discrepancy (Saito et al., 2018), or by a domain discriminator for adversarial training (Ganin et al., 2016;
Revised: For example, the invariance can be enforced by maximum classifier discrepancy (Saito et al., 2018), or by a domain discriminator for adversarial training (Ganin et al., 2016;

Original: Zhao et al., 2018), or by moment matching (Peng et al., 2019), or by relation alignment loss (Wang et al., 2020).
Revised: Zhao et al., 2018; 2021), or by moment matching (Peng et al., 2019), or by relation alignment loss (Wang et al., 2020), or by pseudo labeling (Wang et al., 2022b).

Original: However, all these methods require label distribution to be invariant across domains.
Revised: However, all these methods require label distribution to be invariant across domains.

Original: As a result, when label distribution is varying across domains, they may perform well only in the overlapping areas among label distributions in different domains, while facing with challenges in the non-overlapping areas.
Revised: As a result, when label distribution is varying across domains, they may perform well only in the overlapping areas among label distributions in different domains, while facing with challenges in the non-overlapping areas.

Original: To overcome this, recent progress focuses on learning invariant representations conditional on the label across domains (Gong et al., 2016;
Revised: To overcome this, recent progress focuses on learning invariant representations conditional on the label across domains (Gong et al., 2016;

Original: Ghifary et al., 2016; Tachet des Combes et al., 2020).
Revised: Ghifary et al., 2016; Tachet des Combes et al., 2020).

Original: One of the challenges in these methods is that the labels in the target domain is unavailable.
Revised: One of the challenges in these methods is that the labels in the target domain is unavailable.

Original: More importantly, these methods do not guarantee that the learnt representations to be consistent with the true relevant information for predicting in the target domain, thus there is no principled way to guarantee that the learned predictor can generalize to the target domain.
Revised: More importantly, these methods do not guarantee that the learnt representations to be consistent with the true relevant information for predicting in the target domain, thus there is no principled way to guarantee that the learned predictor can generalize to the target domain.

Original: Learning invariant conditional distribution p D ( y | z c ) .
Revised: Learning invariant conditional distribution p D ( y | z c ) .

Original: There exist few of works exploring the invariant conditional distribution p D ( y | z c ) for domain adaptation (Kull & Flach, 2014; Bouvier et al., 2019).
Revised: There exist few of works exploring the invariant conditional distribution p D ( y | z c ) for domain adaptation (Kull & Flach, 2014; Bouvier et al., 2019).

Original: Differ from these two works, the proposed method provides the identifiability of z c , so that the learned p D ( y | z c ) in this work can generalize to the target domain in a principled way.
Revised: Differ from these two works, the proposed method provides the identifiability of z c , so that the learned p D ( y | z c ) in this work can generalize to the target domain in a principled way.

Original: Besides, in the context of out-of-distribution generalization, some recent works explore learning invariant conditional distribution p D ( y | z c ) (Arjovsky et al., 2019; Sun et al., 2021;
Revised: Besides, in the context of out-of-distribution generalization, some recent works explore learning invariant conditional distribution p D ( y | z c ) (Arjovsky et al., 2019; Sun et al., 2021;

Original: Liu et al., 2021;
Revised: Liu et al., 2021;

Original: Lu et al., 2021).
Revised: Lu et al., 2021).

Original: For example, Arjovsky et al.
Revised: For example, Arjovsky et al.

Original: (2019) imposes learn the optimal invariant predictor across domains from the viewpoint of an intimate link between invariance and causation, while the proposed method directly explores conditional invariance given the proposed latent causal model.
Revised: (2019) imposes learn the optimal invariant predictor across domains from the viewpoint of an intimate link between invariance and causation, while the proposed method directly explores conditional invariance given the proposed latent causal model.

Original: Sun et al.
Revised: Sun et al.

Original: (2021) mainly focus on single domain, while the proposed method consider multiple domains.
Revised: (2021) mainly focus on single domain, while the proposed method consider multiple domains.

Original: The proposed method is also different from the work in Liu et al.
Revised: The proposed method is also different from the work in Liu et al.

Original: (2021) in that the former assume the latent content variable caused by the style variable, while the latter depends on a confounder to model the causal relation between the latent content variable and the style variable.
Revised: (2021) in that the former assume the latent content variable caused by the style variable, while the latter depends on a confounder to model the causal relation between the latent content variable and the style variable.

Original: Unlike the work in Lu et al.
Revised: Unlike the work in Lu et al.

Original: (2021) that the label is treated as a variable causing the other latent variables, the proposed method assumes that the label have no child nodes.
Revised: (2021) that the label is treated as a variable causing the other latent variables, the proposed method assumes that the label have no child nodes.

Revised: Causality for Domain Generalization It has been shown that there is closely the connection between causality and generalization (Peters et al., 2016).

Revised: Motivated by this, most of current methods leverage to introduce new methods in various applications, e.g. , domain generalization, (Mahajan et al., 2021; Christiansen et al., 2021; Wang et al.

Revised: , 2022a),text classification (Veitch et al., 2021),

Revised: Out-of-Distribution Generalization Ahuja et al.

Original:   T HE P ROPOSED LATENT CAUSAL MODEL FOR L ATENT C OVARIATE S HIFT
Revised: Perhaps the closest to our problem setting is domain generalization, where one can not ‘see’ input data x . In general, because one can not ‘see’ input data x for domain generalization, obtaining identifiability result in the setting of domain generalization is generally not possible. In contrast, this work provides the identifiability result, providing a principled way to guarantee that the learned predictor can generalize to the target domain.  T HE P ROPOSED LATENT CAUSAL MODEL FOR L ATENT C OVARIATE S HIFT

Original: We introduce a latent causal model to formulate features and label generative process to handle LCS as depicted by Figure 2.
Revised: We introduce a latent causal model to formulate features and label generative process to handle LCS as depicted by Figure 2.

Original: It introduces the observed domain variable D to denote in which specific domain data are collected.
Revised: It introduces the observed domain variable D to denote in which specific domain data are collected.

Original: n c and n s represent some unmeasured factors corresponding to latent content noise and latent style noise, respectively.
Revised: n c and n s represent some unmeasured factors corresponding to latent content noise and latent style noise, respectively.

Original: n c and n s influence the latent content variable z c and the latent style variable z s , respectively.
Revised: n c and n s influence the latent content variable z c and the latent style variable z s , respectively.

Original: Generally speaking, z c and z s should be dependent given the domain variable D .
Revised: Generally speaking, z c and z s should be dependent given the domain variable D .

Original: Here we consider that z c causes z s , to model the correlation between z s and y .
Revised: Here we consider that z c causes z s , to model the correlation between z s and y .

Original: In the proposed latent causal model, p D ( z c ) change across domains while p D ( y | z c ) is invariant across domains, which meets the basic assumption in the proposed latent covariate shift.
Revised: In the proposed latent causal model, p D ( z c ) change across domains while p D ( y | z c ) is invariant across domains, which meets the basic assumption in the proposed latent covariate shift.

Original: In the following, we discuss two key causal assumptions, which also highlight the novelty of the proposed latent causal model.
Revised: In the following, we discuss two key causal assumptions, which also highlight the novelty of the proposed latent causal model.

Original: z c causes y :
Revised: z c causes y :

Original: Previous works consider the causal relation between x and y as y → x (Gong et al., 2016; Stojanov et al., 2019; Li et al., 2018), while we employ z c → y .
Revised: Previous works consider the causal relation between x and y as y → x (Gong et al., 2016; Stojanov et al., 2019; Li et al., 2018), while we employ z c → y .

Original: We argue that these two cases are not contradictory since the labels y in these two cases represent two different physical meanings.
Revised: We argue that these two cases are not contradictory since the labels y in these two cases represent two different physical meanings.

Original: To understand this point, let ˆy replace y in the first case (i.e., ˆy → x ) to distinguish from y in the second case (i.e., z c → y ).
Revised: To understand this point, let ˆy replace y in the first case (i.e., ˆy → x ) to distinguish from y in the second case (i.e., z c → y ).

Original: For the first case, consider the following generative process of images.
Revised: For the first case, consider the following generative process of images.

Original: A label should be first sampled, e.g., ˆy , then one may determine content information regarding to the label ˆy , and finally generate a image, which is a reasonable assumption in many real application scenarios.
Revised: A label should be first sampled, e.g., ˆy , then one may determine content information regarding to the label ˆy , and finally generate a image, which is a reasonable assumption in many real application scenarios.

Original: In the proposed latent causal model, n c play a role to replace ˆy and causes the content variable z c .
Revised: In the proposed latent causal model, n c play a role to replace ˆy and causes the content variable z c .

Original: For the second case, z c → y , it formulates the process that experts extract content information from given images and then provide reasonable labels according to their domain knowledge.
Revised: For the second case, z c → y , it formulates the process that experts extract content information from given images and then provide reasonable labels according to their domain knowledge.

Original: This assumption has been employed by some recent works (Mahajan et al., 
Revised: This assumption has been employed by some recent works (Mahajan et al., 2021;

Original: Liu et al., 2021; Sun et al., 2021).
Revised: Liu et al., 2021; Sun et al., 2021).

Original: Particularly, these two different labels, ˆy and y , has been simultaneously considered in Mahajan et al.
Revised: Particularly, these two different labels, ˆy and y , has been simultaneously considered in Mahajan et al.

Original: z c causes z s : Here we consider having the object essence
Revised: z c causes z s : Here we consider having the object essence

Original: z c first, from which a latent style z s springs to render x .
Revised: z c first, from which a latent style z s springs to render x .

Original: The correlation between y and z s can be seen as a spurious correlation that should not contribute to predicting y .
Revised: The correlation between y and z s can be seen as a spurious correlation that should not contribute to predicting y .

Original: We here employ z c as a confounding factor of both y and z s to model the spurious correlation.
Revised: We here employ z c as a confounding factor of both y and z s to model the spurious correlation.

Original: The rationality of this assumption can be further verified from the viewpoint of the converse.
Revised: The rationality of this assumption can be further verified from the viewpoint of the converse.

Original: In particular, if we assume that z s causes z c , all high-level information in input data x , z s and z c would be causally related to the label y , which can not model the spurious correlation and is obviously unreasonable.
Revised: In particular, if we assume that z s causes z c , all high-level information in input data x , z s and z c would be causally related to the label y , which can not model the spurious correlation and is obviously unreasonable.

Original: Therefore, assuming z c → z s is more persuasive and consistent with previous works (Gong et al., 2016; Stojanov et al., 2019; Mahajan et al., 2021).
Revised: Therefore, assuming z c → z s is more persuasive and consistent with previous works (Gong et al., 2016; Stojanov et al., 2019; Mahajan et al., 2021).

Original: One recent work in Sun et al.
Revised: One recent work in Sun et al.

Original: (2021) leverages an additional variable as a confounding factor that influences both the content variable z c and the style variable z
Revised: (2021) leverages an additional variable as a confounding factor that influences both the content variable z c and the style variable z

Original: s to model their relation.
Revised: s to model their relation.

Original: Interestingly, the identifiability result in Sun et al.
Revised: Interestingly, the identifiability result in Sun et al.

Original: (2021) does not depend on the confounding factor.
Revised: (2021) does not depend on the confounding factor.

Original: As a result, the confounding factor can be incorporated into the domain index, which is equivalent to the case where z c and z s are independent given the domain index.
Revised: As a result, the confounding factor can be incorporated into the domain index, which is equivalent to the case where z c and z s are independent given the domain index.

Original: By contrast, the proposed latent causal model assumes a more general setting where z c and z s are dependent given the domain index, as depicted by Figure 2.
Revised: By contrast, the proposed latent causal model assumes a more general setting where z c and z s are dependent given the domain index, as depicted by Figure 2.

Original: We will further verify the advantages of the proposed latent causal model in experiments, compared with Sun et al.
Revised: We will further verify the advantages of the proposed latent causal model in experiments, compared with Sun et al.

Original:  I DENTIFIABILITY A NALYSIS OF THE P ROPOSED LATENT CAUSAL MODEL
Revised:  I DENTIFIABILITY A NALYSIS OF THE P ROPOSED LATENT CAUSAL MODEL

Original: In this section, we provide identifiability analysis for the proposed latent causal model.
Revised: In this section, we provide identifiability analysis for the proposed latent causal model.

Original: We first build a connection between the proposed latent causal model and nonlinear ICA by using the independence among latent noise variables.
Revised: We first build a connection between the proposed latent causal model and nonlinear ICA by using the independence among latent noise variables.

Original: We then show that it is still challenging to completely identify the proposed latent causal model due to transitivity , even with the identifiability result in nonlinear ICA.
Revised: We then show that it is still challenging to completely identify the proposed latent causal model due to transitivity , even with the identifiability result in nonlinear ICA.

Original: We finally show how to partially identify the latent content variable z c , by integrating the identifiability result of nonlinear ICA (Khemakhem et al., 2020) with the dependence between n c and y .
Revised: We finally show how to partially identify the latent content variable z c , by integrating the identifiability result of nonlinear ICA (Khemakhem et al., 2020) with the dependence between n c and y .

Original: 4. R ELATING THE P ROPOSED L ATENT C AUSAL M ODEL WITH N ONLINER ICA
Revised: 4. R ELATING THE P ROPOSED L ATENT C AUSAL M ODEL WITH N ONLINER ICA

Original: The proposed latent causal model splits latent noise variables n into two disjoint parts, n c and n s , as depicted by Figure 3.
Revised: The proposed latent causal model splits latent noise variables n into two disjoint parts, n c and n s , as depicted by Figure 3.

Original: Since n i models the noise information, the latent noise variables n i are assumed to be independent with each other in a causal system (Pearl, 2000; Spirtes et al., 2001).
Revised: Since n i models the noise information, the latent noise variables n i are assumed to be independent with each other in a causal system (Pearl, 2000; Spirtes et al., 2001) 1 .

Original: As a result, it is natural to connect the latent noise variables n i with latent independent variables in nonlinear ICA.
Revised: As a result, it is natural to connect the latent noise variables n i with latent indepen- dent variables in nonlinear ICA.

Original: Specifically, nonlinear ICA aims to separate independent latent variables, e.g. , n i , from observed mixing data, e.g. , x , generated by a nonlinear function.
Revised: Specifically, nonlinear ICA aims to separate independent latent variables conditional on D , e.g. , n i , from observed mixing data, e.g. , x , generated by a nonlinear function.

Original: Recent progress in Khemakhem et al.
Revised: Recent progress in Khemakhem et al.

Original: (2020) shows that one can recover n i up to permutation and scaling with relatively mild conditions, e.g. , there is an auxiliary observed variable, similar as D in Figure 3, which modulates the distributions of all independent latent variables n i .
Revised: (2020) shows that one can recover n i up to permutation and scaling with relatively mild conditions, e.g. , there is an auxiliary observed variable, similar as D in Figure 3, which modulates the distributions of all independent latent variables n i .

Original: The identifiability result of n i holds in both source and target domains, since we have feature data x from both source and target domains in the training phase for domain adaptation.
Revised: The identifiability result of n i holds in both source and target domains, since we have feature data x from both source and target domains in the training phase for domain adaptation.

Original: However, the permutation indeterminacy implies that we can not determine which recovered variables n i correspond to n c (or n s ) without further information.
Revised: However, the permutation indeterminacy implies that we can not determine which recovered variables n i correspond to n c (or n s ) without further information.

Original: Therefore, using the identifiability result of nonlinear ICA only is insufficient to identify n c and n s .
Revised: Therefore, using the identifiability result of nonlinear ICA only is insufficient to identify n c and n s .

Original: We will further discuss how to handle the permutation indeterminacy in Section 4.3.
Revised: We will further discuss how to handle the permutation indeterminacy in Section 4.3.

Original: Before that, we first analyze identifiability of the proposed latent causal model to provide a deep insight.
Revised: Before that, we first analyze identifiability of the proposed latent causal model to provide a deep insight.

Original:  C OMPLETE I DENTIFIABILITY : T HE N ON - IDENTIFIABILITY R ESULT
Revised: 4. C OMPLETE I DENTIFIABILITY : T HE N ON - IDENTIFIABILITY R ESULT

Original: Even with the identifiability result of nonlinear
Revised: Even with the identifiability result of nonlinear

Original: ICA, it is still challenging to completely identify the proposed latent causal model.
Revised: ICA, it is still challenging to completely identify the proposed latent causal model.

Original: In particular, we have the following result:
Revised: In particular, we have the following result:

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: With the identifiability result of nonlinear ICA under certain assumptions (see Khemakhem et al. , the proposed latent causal model is still unidentifiable without additional assumptions, due to transitivity in latent space.
Revised: With the identifiability result of nonlinear ICA under certain assumptions 2 , the proposed latent causal model is still unidentifiable without additional assumptions, due to transitivity in latent space.

Original: Intuitively, the non-identifiability result above is because we can not determine which path is the correct path corresponding to the net effect of n c on x ; e.g., n c → z c → x and n c → z c → z s → x are equivalent causal structures in Figure 4.
Revised: Intuitively, the non-identifiability result above is because we can not determine which path is the correct path corresponding to the net effect of n c on x ; e.g., n c → z c → x and n c → z c → z s → x are equivalent causal structures in Figure 4.

Original: This indeterminacy is due to transitivity , because if z c → z s → x , then z c → x can also be an alternative graph structure to generate same observed data, without further assumptions.
Revised: This indeterminacy is due to transitivity , because if z c → z s → x , then z c → x can also be an alternative graph structure to generate same observed data, without further assumptions.

Original: For example, for simplicity, let us only consider one-dimensional z c and z s .
Revised: For example, for simplicity, let us only consider one-dimensional z c and z s .

Original: According to the graph structure shown in the right column of Figure 4, assume that z c := n c , z s :=
Revised: According to the graph structure shown in the right column of Figure 4, assume that z c := n c , z s :=

Original: z c + n c and x := f
Revised: z c + n c and x := f

Original: ( z 1 , z 2 ) + ε (case 1).
Revised: ( z 1 , z 2 ) + ε (case 1).

Original: We then consider the graph structure shown in the left column of Figure 4, assume that z c := n c , z s := n c and x := f ◦ g
Revised: We then consider the graph structure shown in the left column of Figure 4, assume that z c := n c , z s := n c and x := f ◦ g

Original: ( z c , z s )
Revised: ( z c , z s )

Original: + ε where g ( z c , z s ) =
Revised: + ε where g ( z c , z s ) =

Original: [ z c , z c + z s ] (case 2).
Revised: [ z c , z c + z s ] (case 2).

Original: Interestingly, we find that the causal models in case 1 and case 2 generate the same observed data x , which implies that there are two different causal models to interpret the same observed data, even with the identifiable n c and n s .
Revised: Interestingly, we find that the causal models in case 1 and case 2 generate the same observed data x , which implies that there are two different causal models to interpret the same observed data, even with the identifiable n c and n s .

Original: It often appears in latent causal discovery and seriously hinders the identifiability of latent causal models.
Revised: It often appears in latent causal discovery and seriously hinders the identifiability of latent causal models.

Original: For reader who may be interested in that problem, we recommend recent work by Adams et al.
Revised: For reader who may be interested in that problem, we recommend recent work by Adams et al.

Original: 4. P ARTIAL I DENTIFIABILITY : I DENTIFYING z c UP TO S CALING
Revised: 4. P ARTIAL I DENTIFIABILITY : I DENTIFYING z c UP TO S CALING

Original: Although completely identifying the proposed latent causal model is challenging, for domain adaptation application, we are only interested in the identifiability of z c , instead of the latent style variable z s , since label y is only caused by z c .
Revised: Although completely identifying the proposed latent causal model is challenging, for domain adaptation application, we are only interested in the identifiability of z c , instead of the latent style variable z s , since label y is only caused by z c .

Original: Thanks to the observed y S from source domains, we have the following identifiability result:
Revised: Thanks to the observed y S from source domains, we have the following identifiability result:

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Assume that
Revised: Assume that

Original: (i) all latent noise variables n i can be identified up to permutation and scaling,the
Revised: (i) all latent noise variables n i can be identified up to permutation and scaling,

Original:  latent content variable z c in the proposed latent causal model can be identifiable up to scaling by using the dependence between n S c and y S from source domains.
Revised: the latent content variable z c in the proposed latent causal model can be identifiable up to scaling by using the dependence between n S c and y S from source domains.

Original: Condition (i) can be obtained from the identifiability result of nonlinear ICA (Khemakhem et al., 2020).
Revised: Condition (i) can be obtained from the identifiability result of nonlinear ICA (Khemakhem et al., 2020).

Original: The proposition shows that the content variable z c can be identified up to scaling by using the identifiability result of nonlinear ICA and the dependence between y and n c , in which the identifiability result of nonlinear ICA ensures to recover n i up to permutation and scaling, while the dependence removes the permutation indeterminacy to identify n c and thus z c .
Revised: The proposition shows that the content variable z c can be identified up to scaling by using the identifiability result of nonlinear ICA and the dependence between y and n c , in which the identifiability result of nonlinear ICA ensures to recover n i up to permutation and scaling, while the dependence removes the permutation indeterminacy to identify n c and thus z c .

Original: The scaling indeterminacy of z c results from the scaling indeterminacy of n i and the unknown mapping form n c to z c .
Revised: The scaling indeterminacy of z c results from the scaling indeterminacy of n i and the unknown mapping form n c to z c .

Original: The scaling indeterminacy of z c has no significance and can be ignored in latent space, since it can be ‘absorbed’ by the nonlinear mapping from z c to y .
Revised: The scaling indeterminacy of z c has no significance and can be ignored in latent space, since it can be ‘absorbed’ by the nonlinear mapping from z c to y .

Original: For example, consider the recovered variable ˆz c and its scaling scaling ( ˆz c ) .
Revised: For example, consider the recovered variable ˆz c and its scaling scaling ( ˆz c ) .

Original: When we try to learn a invariant predictor g ( · ) form ˆz c to y , the scaling indeterminacy can be ‘absorbed’ by a composition predictor, e.g., g ( scaling ( · )) .
Revised: When we try to learn a invariant predictor g ( · ) form ˆz c to y , the scaling indeterminacy can be ‘absorbed’ by a composition predictor, e.g., g ( scaling ( · )) .

Original:  L EARNING I NVARIANT p D ( y | z c ) FOR MSDA
Revised:  L EARNING I NVARIANT p D ( y | z c ) FOR MSDA

Original: The identifiable z c provides a principled way to guarantee that we can learn the conditional distribution p D ( y | z c ) that is invariant across domains and thus can be generalized to the target domain.
Revised: The identifiable z c provides a principled way to guarantee that we can learn the conditional distribution p D ( y | z c ) that is invariant across domains and thus can be generalized to the target domain.

Original: Furthermore, since the identifiable n c is the only parent node of z c , learning p D ( y | z c ) can be transferred into learning p D ( y | n c ) , which is also invariant across domains as depicted by Figure 2.
Revised: Furthermore, since the identifiable n c is the only parent node of z c , learning p D ( y | z c ) can be transferred into learning p D ( y | n c ) , which is also invariant across domains as depicted by Figure 2.

Original: In this section, we propose a novel method to show how to learn the invariant conditional distribution p D ( y | n c ) for MSDA.
Revised: In this section, we propose a novel method to show how to learn the invariant conditional distribution p D ( y | n c ) for MSDA.

Original: 5. T HE P ROPOSED M ETHOD FOR L EARNING I NVARIANT p D ( y | n c )
Revised: 5. T HE P ROPOSED M ETHOD FOR L EARNING I NVARIANT p D ( y | n c )

Original: As analyzed in Section 4.3, the identifiability of z c is based on the identifiability result of nonlinear
Revised: As analyzed in Section 4.3, the identifiability of z c is based on the identifiability result of nonlinear

Original: ICA, so we need to identify n i first.
Revised: ICA, so we need to identify n i first.

Original: To meet the conditions of identifiable n i as mentioned in
Revised: To meet the conditions of identifiable n i as mentioned in

Original: Khemakhem et al.
Revised: Khemakhem et al.

Original: , we employ the following Gaussian prior on n c and n s :
Revised: , we employ the following Gaussian prior on n c and n s :

Original: [Equation]
Revised: [Equation]

Original: where µ and Σ denote the mean and variance, respectively.
Revised: where µ and Σ denote the mean and variance, respectively.

Original: Both depend on the domain variable D and can be implemented with multi-layer perceptrons.
Revised: Both depend on the domain variable D and can be implemented with multi-layer perceptrons.

Original: Since n i are to independent noise variables, Σ here is a diagonal matrix.
Revised: Since n i are to independent noise variables, Σ here is a diagonal matrix.

Original: Some other exponential distributions, e.g. , Laplace distribution, also meet the conditions of identifiable n i and thus are also feasible (Khemakhem et al., 2020).
Revised: Some other exponential distributions, e.g. , Laplace distribution, also meet the conditions of identifiable n i and thus are also feasible (Khemakhem et al., 2020).

Original: We here use the Gassian prior since it is easy to leverage the reparameterization trick (Kingma & Welling, 2013).
Revised: We here use the Gassian prior since it is easy to leverage the reparameterization trick (Kingma & Welling, 2013).

Original: The proposed Gaussian prior equation 1 gives rise to the following variational posterior:
Revised: The proposed Gaussian prior equation 1 gives rise to the following variational posterior:

Original: [Equation]
Revised: [Equation]

Original: where µ ′ and Σ ′ denote the mean and variance of the posterior, respectively; both of them depend on the domain variable D and the observed x and can be implemented by multi-layer perceptrons.
Revised: where µ ′ and Σ ′ denote the mean and variance of the posterior, respectively; both of them depend on the domain variable D and the observed x and can be implemented by multi-layer perceptrons.

Original: Combining this with the Gaussian prior in equation 1, we can derive the following evidence lower bound (ELBO):
Revised: Combining this with the Gaussian prior in equation 1, we can derive the following evidence lower bound (ELBO):

Original: [Equation]
Revised: [Equation]

Original:  where D KL denotes the Kullback–Leibler divergence.
Revised: 3) where D KL denotes the Kullback–Leibler divergence.

Original: By maximizing the ELBO equation 3, we can then recover n i up to scaling and permutation.
Revised: By maximizing the ELBO equation 3, we can then recover n i up to scaling and permutation.

Original: To remove the permutation indeterminacy as mentioned in Section 4.3, we can simultaneously maximize the correlation between y S and n S i to identify n S c .
Revised: To remove the permutation indeterminacy as mentioned in Section 4.3, we can simultaneously maximize the correlation between y S and n S i to identify n S c .

Original: Here we employ the mutual information to maximize the dependence.
Revised: Here we employ the mutual information to maximize the dependence.

Original: As a result, we arrive at:
Revised: As a result, we arrive at:

Original: [Equation]
Revised: [Equation]

Original: where I ( n S c , y S ) denotes the mutual information between n S c and y S in source domains, and λ is a regularization hyper-parameter that balances the ELBO and the mutual information (MI).
Revised: where I ( n S c , y S ) denotes the mutual information between n S c and y S in source domains, and λ is a regularization hyper-parameter that balances the ELBO and the mutual information (MI).

Original: The proposed method is termed iLCC-MSDA (identifiable Latent Causal Content for MSDA), including two components: ELBO and mutual information.
Revised: The proposed method is termed iLCC-MSDA (identifiable Latent Causal Content for MSDA), including two components: ELBO and mutual information.

Original: The ELBO component ensures that n i can be recovered up to scaling and permutation.
Revised: The ELBO component ensures that n i can be recovered up to scaling and permutation.

Original: The MI component handles the permutation, and thus ensures which recovered n i corresponds to the latent content variables n c .
Revised: The MI component handles the permutation, and thus ensures which recovered n i corresponds to the latent content variables n c .

Original: In the implementation, we use the variational low bounder of mutual information proposed by Alemi et al.
Revised: In the implementation, we use the variational low bounder of mutual information proposed by Alemi et al.

Original: (2016) to approximate the mutual information in equation 4.
Revised: (2016) to approximate the mutual information in equation 4.

Original: [Equation]
Revised: [Equation]

Original: A graphical depiction of the proposed iLCC-MSDA is shown in Figure 5.[Figure]
Revised: A graphical depiction of the proposed iLCC-MSDA is shown in Figure 5.

Original: Constraining the independence among n i
Revised: Constraining the independence among n i

Original: As we discussed, the performance of the proposed iLCC-MSDA above relies on assumptions on the identifiability of n i , which requires that there are enough domains across which n i changes in order to well capture the statistical independence.
Revised: As we discussed, the performance of the proposed iLCC-MSDA above relies on assumptions on the identifiability of n i , which requires that there are enough domains across which n i changes in order to well capture the statistical independence.

Original: However, in real applications, we may not have sufficient domains.
Revised: However, in real applications, we may not have sufficient domains.

Original: To mitigate this issue, motivated by disentangled representations (Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018), we proposes to use a hyperparameter β to enhance the independence among n i .
Revised: To mitigate this issue, motivated by disentangled representations (Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018), we proposes to use a hyperparameter β to enhance the independence among n i .

Original: Entropy regularization In the loss function in equation 5, we maximize the causal influence between y S and n S c ) in source domains with mutual information.
Revised: Entropy regularization In the loss function in equation 5, we maximize the causal influence between y S and n S c ) in source domains with mutual information.

Original: To encourage such causal influence in target domain, we can also maximize the mutual information between ˆy T and n T c by minimizing the following conditional entropy:
Revised: To encourage such causal influence in target domain, we can also maximize the mutual information between ˆy T and n T c by minimizing the following conditional entropy:

Original: [Equation]
Revised: [Equation]

Original: (6) where ˆy T denotes the estimated label in the target domain.
Revised: (6) where ˆy T denotes the estimated label in the target domain.

Original: This regularization has been empirically used to make label predictions more deterministic in previous works (Wang et al., 2020; Li et al., 2021), while we consider it from the view of causality.
Revised: This regularization has been empirically used to make label predictions more deterministic in previous works (Wang et al., 2020; Li et al., 2021), while we consider it from the view of causality.

Original: Therefore, our final loss function is:
Revised: Therefore, our final loss function is:

Original: [Equation]
Revised: [Equation]

Original: where β, λ, γ are hyper-parameters that trade off the independence of n c and n s , the classifier and the entropy regularization loss terms.
Revised: where β, λ, γ are hyper-parameters that trade off the independence of n c and n s , the classifier and the entropy regularization loss terms.

Original:  E XPERIMENTS 6. E XPERIMENTS ON S YNTHETIC D ATA
Revised:  E XPERIMENTS 6. E XPERIMENTS ON S YNTHETIC D ATA

Original: Dataset We conduct experiments on synthetic data, generated by the following process: we divide the latent variables into 5 segments, which are corresponding to 5 domains.
Revised: Dataset We conduct experiments on synthetic data, generated by the following process: we divide the latent variables into 5 segments, which are corresponding to 5 domains.

Original: Each segment includes 1000 examples.
Revised: Each segment includes 1000 examples.

Original: Within each segment, we first sample the mean and the variance from uniform distributions [1 , 2] and [0 . 3 , 1] for the latent exogenous variables n c and n s , respectively.
Revised: Within each segment, we first sample the mean and the variance from uniform distributions [1 , 2] and [0 . 3 , 1] for the latent exogenous variables n c and n s , respectively.

Original: Then for each segment, we generate z c , z s , x and y according to the following structural causal model:
Revised: Then for each segment, we generate z c , z s , x and y according to the following structural causal model:

Original: [Equation]
Revised: [Equation]

Original: where following (Khemakhem et al., 2020) we mix the latent z c and z s using a multi-layer perceptron to generate x .
Revised: where following (Khemakhem et al., 2020) we mix the latent z c and z s using a multi-layer perceptron to generate x .

Original: Results In implementation, we use the first 4 segments as source domains, and the last segment as target domain.
Revised: Results In implementation, we use the first 4 segments as source domains, and the last segment as target domain.

Original: Figure 6(a) shows the true and recovered distributions of the exogenous variables n c .
Revised: Figure 6(a) shows the true and recovered distributions of the exogenous variables n c .

Original: Due to the support of nonlinear ICA, the proposed iLCC-MSDA obtain the mean correlation coefficient (MCC) 0.96 between the original n c and the recovered.
Revised: Due to the support of nonlinear ICA, the proposed iLCC-MSDA obtain the mean correlation coefficient (MCC) 0.96 between the original n c and the recovered.

Original: Due to the invariant conditional distribution p ( y | n c ) , even with the change of distribution of the exogenous variables n c as shown in Figure 6(a), the learned p ( y | n c ) can generalize to target segment in a principle way as depicted by the Figure 6(b).
Revised: Due to the invariant conditional distribution p ( y | n c ) , even with the change of distribution of the exogenous variables n c as shown in Figure 6(a), the learned p ( y | n c ) can generalize to target segment in a principle way as depicted by the Figure 6(b).

Original: Due to the limited space, Figure 6(b) only shows 200 samples for the true and predicted y . 
Revised: Due to the limited space, Figure 6(b) only shows 200 samples for the true and predicted y . [Figure]

Original: 6. E XPERIMENTS ON R EAL D ATA
Revised: 6. E XPERIMENTS ON R EAL D ATA

Original: Dataset We further evaluate the proposed iLCC-MSDA on benchmark domain adaptation dataset
Revised: Dataset We further evaluate the proposed iLCC-MSDA on benchmark domain adaptation dataset

Original: PACS dataset (Li et al., 2017) and Terra Incognita (Beery et al., 2018).
Revised: PACS dataset (Li et al., 2017) and Terra Incognita (Beery et al., 2018).

Original: To obtain the situation of
Revised: In the original PACS, the label distributions for any two domains is very similar ( i.e. , D KL ≈ 0 . 1 ). This data is suitable for domain adaptation with conditional shift as shown in Figure 1 (b) where the label distribution remains unchanged, while it is not appropriate for the proposed setting where the label distribution changes across domains, as shown in Figure 1 (c).

Original: LCS on PACS dataset, we filtered the original  dataset by re-sampling it to generate three datasets,
Revised: Therefore, we randomly sample the original PACS dataset to provide new PACS dataset where the label distribution changes, to generate three datasets,

Original: PACS ( D KL = 0 . 3 ), PACS ( D KL = 0 . 5 ) and PACS ( D KL = 0 . 7 ), where D KL = 0 .
Revised: PACS ( D KL = 0 . 3 ), PACS ( D KL = 0 . 5 ) and PACS ( D KL = 0 . 7 ). Here D KL = 0 .

Original: 5 , 0 . 7) denotes that KL divergence of label distributions in any two different domains is approximately 0.
Revised: 5 , 0 . 7) denotes that KL divergence of label distributions in any two different domains is approximately 0.

Original: Details of label distributions are depicted by Figure 7. [Figure]
Revised: See APPENDIX for details of label distributions.

Original: Baselines We compare the proposed method with state-of-the-art methods to verify its effectiveness.
Revised: Baselines We compare the proposed method with state-of-the-art methods to verify its effectiveness.

Original: Particularly, we compare the proposed methods with empirical risk minimization (ERM),
Revised: Particularly, we compare the proposed methods with empirical risk minimization (ERM),

Original: MCDA (Saito et al., 2018), M3DA (Peng et al., 2019), LtC-MSDA (Wang et al., 2020), T-SVDNet (Li et al., 2021), IRM (Arjovsky et al., 2019), IWCDAN (Tachet des Combes et al., 2020) and
Revised: MCDA (Saito et al., 2018), M3DA (Peng et al., 2019), LtC-MSDA (Wang et al., 2020), T-SVDNet (Li et al., 2021), IRM (Arjovsky et al., 2019), IWCDAN (Tachet des Combes et al., 2020) and

Original: LaCIM (Sun et al., 2021).
Revised: LaCIM (Sun et al., 2021).

Original: In these methods, MCDA, M3DA, LtC-MSDA and T-SVDNet learn invariant representations for MSDA, while IRM, IWCDAN and LaCIM learn invariant conditional distributions, allowing label distribution to change.
Revised: In these methods, MCDA, M3DA, LtC-MSDA and T-SVDNet learn invariant representations for MSDA, while IRM, IWCDAN and LaCIM learn invariant conditional distributions, allowing label distribution to change.

Original: Details of implementation, including network architectures and hyper-parameter setting, are in the APPENDIX.
Revised: Details of implementation, including network architectures and hyper-parameter setting, are in the APPENDIX.

Original: All the proposed methods are averaged over 3 runs with standard deviation.
Revised: All the proposed methods are averaged over 3 runs with standard deviation.

Original: Ablation studies The bottom of Table 1 and 2 presents the results for ablation studies.
Revised: Ablation studies The bottom of Table 1 and 2 presents the results for ablation studies.

Original: We can observe that entropy regularization equation 6 significantly increases the performance (around 10% and 5% ) of the proposed method on both dataset.
Revised: We can observe that entropy regularization equation 6 significantly increases the performance (around 10% and 5% ) of the proposed method on both dataset.

Original: This justifies the importance of the causal relation between y and n c , which is consistent with our model assumption.
Revised: This justifies the importance of the causal relation between y and n c , which is consistent with our model assumption.

Original: Besides, the hyper-parameter β also boosts the performance by enforcing the independence among the latent variables n i .
Revised: Besides, the hyper-parameter β also boosts the performance by enforcing the independence among the latent variables n i .

Original: Results
Revised: Results

Original: The results by different methods on PACS are presented in Table 1.
Revised: The results by different methods on PACS are presented in Table 1.

Original: We can observe that as the increase of KL divergence of label distribution, the performance of MCDA, M3DA, LtC-MSDA and T-SVDNet, which are based on learning an invariant representations, gradually degenerates.
Revised: We can observe that as the increase of KL divergence of label distribution, the performance of MCDA, M3DA, LtC-MSDA and T-SVDNet, which are based on learning an invariant representations, gradually degenerates.

Original: When the KL divergence is about 0.7, the performance of these methods is worse than traditional
Revised: When the KL divergence is about 0.7, the performance of these methods is worse than traditional

Original: ERM.
Revised: ERM.

Original: Compared with IRM, IWCDAN and LaCIM, which allows label distribution to change across domains, the proposed iLCC-MSDA obtains the best performance, due to our theoretical supports.
Revised: Compared with IRM, IWCDAN and LaCIM, which allows label distribution to change across domains, the proposed iLCC-MSDA obtains the best performance, due to our theoretical supports.

Original: Table 2 depicts the results by different methods on challenging Terra Incognit.
Revised: Table 2 depicts the results by different methods on challenging Terra Incognit.

Original: The proposed iLCCMSDA achieves a significant performance gain on the challenging task → L7.
Revised: The proposed iLCCMSDA achieves a significant performance gain on the challenging task → L7.

Original: Compared with the other methods, the proposed iLCC-MSDA is the only one that is superior to ERM.
Revised: Compared with the other methods, the proposed iLCC-MSDA is the only one that is superior to ERM.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The key for domain adaptation is to understand how the joint distribution of features and label changes across domains.
Revised: The key for domain adaptation is to understand how the joint distribution of features and label changes across domains.

Original: Previous works usually assume covariate shift or conditional shift to interpret the change of the joint distribution, which may be restricted in some real applications.
Revised: Previous works usually assume covariate shift or conditional shift to inter- [Table] [Table] pret the change of the joint distribution, which may be restricted in some real applications.

Original: Hence, this work considers a new and milder assumption, latent covariate shift.
Revised: Hence, this work considers a new and milder assumption, latent covariate shift.

Original: Specifically, we propose a latent causal model to precisely formulate the generative process of input features and label.
Revised: Specifically, we propose a latent causal model to precisely formulate the generative process of input features and label.

Original: We show that the latent content variable in the proposed latent causal model can be identified up to scaling.
Revised: We show that the latent content variable in the proposed latent causal model can be identified up to scaling.

Original: This inspires a new method to learn the invariant label distribution conditional on the latent[Table] [Table] causal variable, resulting in a principled way to guarantee generalization to target domains.
Revised: This inspires a new method to learn the invariant label distribution conditional on the latent causal variable, resulting in a principled way to guarantee generalization to target domains.

Original: Experiments demonstrate the theoretical results and the efficacy of the proposed method, compared with state-of-the- methods across various sets
Revised: Experiments demonstrate the theoretical results and the efficacy of the proposed method, compared with state-of-the-art methods across various data sets.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: Data Details
Revised: Data Details

Revised: The commonly used datasets for multi-source domain adaptation, such as Digitsfive, Office-Home, PACS, and DomainNet, are not considered in this work, because for these dataset the label distributions of any two domains is very similar, which is suitable for domain adaptation with conditional shift as shown in Figure 1 (b).

Revised: However, these datasets are not appropriate for the proposed setting where the label distribution changes across domains, as shown in Figure (c).

Original: The original PACS (Li et al., 2017) is a multiple domain dataset contains 4 domains, 
Revised: Therefore, we resample the original PACS (Li et al., 2017) dataset, which contains 4 domains, Photo,

Original: Photo, Artpainting, Cartoon and Sketch, which shares the same seven categories.
Revised: Artpainting, Cartoon and Sketch, which shares the same seven categories.

Original: The KL divergence of label distributions of any two domains in the original PACS is very small, round 0.1.
Revised: The KL divergence of label distributions of any two domains in the original PACS is very small, round 0.1.

Original: For obtaining  the proposed latent covariate shift, we filter the original dataset by re-sampling it, and obtain three new datasets with different the KL divergences of label distributions as depicted by Figure 7.
Revised: For obtaining dataset that meets the requirement of the proposed latent covariate shift, we filter the original dataset by re-sampling it, and obtain three new datasets with different the KL divergences of label distributions as depicted by Figure 7.

Original: For
Revised: The resampling process just randomly select some sample from the original PACS dataset, so that the labels distribution changes across domains. The labels distribution are depicted by Figure 7 .

Original:  Terra Incognita (Beery et al., 2018), the label distribution is long-tailed at each domain, and each domain has a different label distribution, which is naturally applicable for our setting.
Revised: For Terra Incognita (Beery et al., 2018), the label distribution is long-tailed at each domain, and each domain has a different label distribution, which is naturally applicable for our setting.

Original: This work uses the four domains from the original data, L28, L43, L46 and L7, which shares the same seven categories: bird, bobcat, empty, opossum, rabbit, raccoon, skunk, as depicted by Figure 
Revised: This work uses the four domains from the original data, L28, L43, L and L7, which shares the same seven categories: bird, bobcat, empty, opossum, rabbit, raccoon, skunk, as depicted by Figure 8. (Here ’L8’ denotes the image data is collected from the location 28.) [Figure]

Original: [Figure]
Revised: [Figure]

Original: Implementation Details For the synthetic data, we used a encoder, e.g. 3-layer fully connected network with 30 hidden nodes for each layer, and decoder, e.g. 3-layer fully connected network with 30 hidden nodes for each layer.
Revised: Implementation Details For the synthetic data, we used a encoder, e.g. 3-layer fully connected network with 30 hidden nodes for each layer, and decoder, e.g. 3-layer fully connected network with 30 hidden nodes for each layer.

Original: We use 3-layer fully connected network with 30 hidden nodes for prior model.
Revised: We use 3-layer fully connected network with 30 hidden nodes for prior model.

Original: Since this is a ideal environment to verify the proposed method, for hyper-parameters, we set β = 1 and γ = 0 to remove the heuristic constraints, and we set λ = 1 e − 2 .
Revised: Since this is a ideal environment to verify the proposed method, for hyper-parameters, we set β = 1 and γ = 0 to remove the heuristic constraints, and we set λ = 1 e − 2 .

Original: For the real data, all methods used the same network backbone, ResNet-18 pre-trained on ImageNet.
Revised: For the real data, all methods used the same network backbone, ResNet-18 pre-trained on ImageNet.

Original: Since it can be challenging to train VAE on high-resolution images, we use extracted features by ResNet-18 as our
Revised: Since it can be challenging to train VAE on high-resolution images, we use extracted features by ResNet-18 as our

Original: VAE input.
Revised: VAE input.

Original: We then use 2-layer fully connected networks as the VAE encoder and decoder, use 2-layer fully connected network for the prior model, use 2-layer fully connected network to transfer n c to z c .
Revised: We then use 2-layer fully connected networks as the VAE encoder and decoder, use 2-layer fully connected network for the prior model, use 2-layer fully connected network to transfer n c to z c .

Original: For hyper-parameters, we set β = 4 , γ = 0 .
Revised: For hyper-parameters, we set β = 4 , γ = 0 .

Original: 1 , λ = 1 e − 4 for the proposed method on all datasets.
Revised: 1 , λ = 1 e − 4 for the proposed method on all datasets.

Original: A. T HE P ROOF OF P ROPOSITION 4. 
Revised: A. T HE P ROOF OF P ROPOSITION 4.

Original: To prove non-identifiability, it is sufficient to show that several different graph structures can lead to the same observed data.
Revised: To prove non-identifiability, it is sufficient to show that several different graph structures can lead to the same observed data.

Original: In particular, let us consider the net effect of n c on x .
Revised: In particular, let us consider the net effect of n c on x .

Original: There are two different paths to ‘explain’ the net effect of n c on x .
Revised: There are two different paths to ‘explain’ the net effect of n c on x .

Original: One path is n c → z c → x .
Revised: One path is n c → z c → x .

Original: In this case, since we have no limitation on the function class of edges, we can cut the path z
Revised: In this case, since we have no limitation on the function class of edges, we can cut the path z

Original: c → z s off (e.g., the left sub-figure of Figure 4) and obtain the same observed data depicted by the right sub-figure of Figure 4.
Revised: c → z s off (e.g., the left sub-figure of Figure 4) and obtain the same observed data depicted by the right sub-figure of Figure 4.

Original: Therefore, there is a causal graph depicted by the left sub-figure of Figure 4 are equivalent with the proposed latent causal model as depicted by the right sub-figure of Figure 4.
Revised: Therefore, there is a causal graph depicted by the left sub-figure of Figure 4 are equivalent with the proposed latent causal model as depicted by the right sub-figure of Figure 4.

Original: A. T HE P ROOF OF P ROPOSITION 4. 
Revised: A. T HE P ROOF OF P ROPOSITION 4.

Original: As mentioned in Section 4.1, there are the permutation indeterminacy and scaling indeterminacy in identifiable n i .
Revised: As mentioned in Section 4.1, there are the permutation indeterminacy and scaling indeterminacy in identifiable n i .

Original: The permutation indeterminacy implies that we are uncertain of which the recovered variable ˆ n i are corresponding to the latent content variable n c .
Revised: The permutation indeterminacy implies that we are uncertain of which the recovered variable ˆ n i are corresponding to the latent content variable n c .

Original: If we can solve this permutation problem, since the parent node of z c includes n c only, z c can be identifiable up to scaling, i.e., z c = f ( n c ) where f can be any nonlinear function.
Revised: If we can solve this permutation problem, since the parent node of z c includes n c only, z c can be identifiable up to scaling, i.e., z c = f ( n c ) where f can be any nonlinear function.

Original: Let us consider the relationships between each ˆ n i and y in the proposed causal model, it is clear that the label y depends on n c , and is independent with n s , given the domain variable D .
Revised: Let us consider the relationships between each ˆ n i and y in the proposed causal model, it is clear that the label y depends on n c , and is independent with n s , given the domain variable D .

Original: As a result, we can compute the correlations ( e.g. , by mutual information) between y S from source domains and ˆ n S i from source domains to determine which recovered variables ˆ n i are corresponding to the latent content variable n c . 
Revised: As a result, we can compute the correlations ( e.g. , by mutual information) between y S from source domains and ˆ n S i from source domains to determine which recovered variables ˆ n i are corresponding to the latent content variable n c . [Equation] [Figure] [Equation]

Revised: [Figure]

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Revised: A. A SSUMPTIONS FOR N ONLINEAR ICA

Revised: Our theoretical results, e.g. , Propositions 4.1 and 4.2, rely on the identifiability result of nonlinear ICA.

Revised: For completeness, we present assumptions for the identifiability result Khemakhem et al.

Revised: ; Sorrenson et al.

Revised: (2020), as follows:

Revised: Theorem A.1.

Revised: Khemakhem et al.

Revised: ; Sorrenson et al.

Revised: Suppose the following conditional generative model:

Revised: [Equation]

Revised: Assume the following holds:

Revised: (i)

Revised: The set { x ∈ X| φ ε ( x ) = 0 } has measure zero (i.e., has at most countable number of elements), where φ ε is the characteristic function of the density p ε .

Revised: (ii)

Revised: The function f in Eq.

Revised: 10 is bijective.

Revised: (iii)

Revised: There exist 2 d + 1 distinct points D 0 , d 1 , ..., D 2 d such that the matrix

Revised: [Equation]

Revised: of size 2 d × 2 d is invertible, where

Revised: d

Revised: is the dimension of

Revised: n

Revised: , η ( D ) denotes the vector of their coefficients, which depends on β i, 1 and β i, 2 .

Revised: then the true latent variables n are related to the estimated latent variables ˆn by the following relationship:

Revised: [Equation]

Revised: where P denotes the permutation matrix with scaling, c denotes a constant vector.

Revised: Eq.

Revised: 9 is enforcing Gaussian distributions on the latent noise variables n .

Revised: Note that the assumptions of nonlinear ICA (Khemakhem et al., 2020) on the noise could be broad exponential family distribution, e.g. , Gaussian distributions, Laplace, Gamma distribution and so on.

Revised: This work consider

Revised: Gaussian distribution, mainly because we implement it in our experiment, as