Paper: Identifiable Latent Causal Content for Domain Adaptation under Latent Covariate Shift (ICLR-2024)

Reviewer comment (Theory):
It is unclear to me how this graph relates to the causal or anti-causal tasks typically defined. For instance, how would the graph map to a segmentation task? I believe this relates to the assumption of a lack of direct influence between $X$ and $Y$.

Author response:
Thank you for your valuable suggestion. Regarding causal or anti-causal tasks, we recognize that the complexities of specific tasks may require a more tailored causal graph structure. Nevertheless, we posit that these tasks might benefit from the proposed causal graph. The link between the proposed causal graph and causal or anti-causal tasks could be attributed to the deliberate modeling of latent causal variables within our framework. To illustrate, let us explore its potential application in segmentation tasks, where interpreting graph nodes as distinct regions may be reasonable. In this scenario, justifying the absence of direct connections between nodes may be conceivable; instead, connections should be conceptualized within high-level latent variables, as highlighted in the proposed causal graph. More details can be found in Section A.9 in the revised version.

Reviewer comment (Presentation):
I found the discussion missing, with no mention of limitations.

Author response:
Thank you once again for your insightful suggestions. The most challenging assumption in our model necessitates the presence of substantial changes in latent noise variables, as they play a crucial role in identifiability. However, in practical applications, a prevalent challenge arises from the scarcity of extensive source domain datasets that adequately capture such substantial changes. In this context, the assumptions related to identifiability in our results may not be satisfied, and thus, the performance of the proposed method may benefit from the introduction of certain regularizations. One potential regularization approach involves leveraging the independence among the noise variables, and further details can be found in Appendix A.10.

Reviewer comment (Novelty):
While the obtained method is different, the current work can be related to that of Alabdulmohsin et al., 2023, which also investigates latent shift. It would be great to discuss the $n$ and $z$ variables compared to the proxies in that work and whether some cases would be more adapted to one method or the other, or whether the authors believe their method would be superior (and why).

Author response:
Thank you once again for your valuable suggestions. We have incorporated the referenced work into our related work section. We think that each method possesses its own set of advantages and disadvantages. For instance, proxy-based methods offer the advantage of relaxing model assumptions, but they need a high-quality proxy. In contrast, our approach eliminates the need for a proxy variable but entails making stronger assumptions within the model.

Reviewer comment (Writing):
I would suggest to frame the graph as a set of assumptions, rather than as another "view" of the MSDA problem. This would make the limitations of the work clearer.

Author response:
Thank you for your thoughtful feedback. We agree that presenting our graphical representation in this manner would enhance clarity regarding the limitations of our work. We have added a Section A.8 in the revised version, including assumptions (i)-(iii) in proposition 4.2, and most importantly, assumptions on the proposed causal graph as formulated in Eqs. (1)-(3), to understand our assumptions for identifying $\mathbf{z}_c$. For examples, we assume two-parameter exponential family members for latent noise variables $\mathbf{n}$ as defined in Eq. (1), considering informative latent noise variables $n_i$ can be automatically separated from noise by an estimating model (see [1]). However, in real applications, the distribution of $\mathbf{n}$ could be arbitrary. In this context, assumption Eq. (1) may only serve as an approximation for the true distribution of $\mathbf{n}$. More details please see Section A.8.

[1] Sorrenson, Peter, Carsten Rother, and Ullrich Köthe. "Disentanglement by nonlinear ica with general incompressible-flow networks (gin)." arXiv preprint arXiv:2001.04872 (2020).

Reviewer comment (Theory):
This could include the clarification on how this graph relates to typical causal and anti-causal tasks.

Author response:
Thank you for your valuable suggestion. Regarding causal or anti-causal tasks, we recognize that the complexities of specific tasks may require a more tailored causal graph structure. Nevertheless, we posit that these tasks might benefit from the proposed causal graph. The link between the proposed causal graph and causal or anti-causal tasks could be attributed to the deliberate modeling of latent causal variables within our framework. To illustrate, let us explore its potential application in segmentation tasks, where interpreting graph nodes as distinct regions may be reasonable. In this scenario, justifying the absence of direct connections between nodes may be conceivable; instead, connections should be conceptualized within high-level latent variables, as highlighted in the proposed causal graph. More details can be found in Section A.9 in the revised version.

Reviewer comment (Writing):
Please refrain from using superlatives. Some of the adjectives describing the method felt a bit optimistic, especially in the absence of a proper discussion.

Author response:
Thank you for your valuable suggestions. Due to the constraints of the rebuttal time frame, our priority is to address more complex reviews. We will thoroughly revise inappropriate presentations in the final version. Your understanding is greatly appreciated.

Reviewer comment (Theory):
There is uncertainty about the correctness of the identifiability result. In the derivation of Eq. (19), the authors used the d-separation between $n_s$ (or $\hat{n}_s$) and $y$ given $u$, but this separation does not necessarily imply that $n_c$ only depends on $\hat{n}_c$, since including $\hat{n}_s$ does not break the dependency between $n_c$ and $y$ (given $u$). Additionally, the parameterization of the variational posterior does not respect the dependency constraints from Fig. 2 (a). Specifically, given $x$, $n_c$ and $n_s$ are not independent because $x$ is a collider in the path between them.

Author response:
**R2:** Strongly disagree. **The assertion that the d-separation exploited between $\mathbf{n}_s$ and $\mathbf{y}$ given $\mathbf{u}$ does not imply the identifiability of $\mathbf{n}_c$ is misconstrued**. By leveraging the independence of $\mathbf{n}_s$ from $\mathbf{y}$ given $\mathbf{u}$, one can perform independent tests between each recovered variable ${n}_i$ and $\mathbf{y}$. Those ${n}_i$ showing a statistically correlation with $\mathbf{y}$ are part of $\mathbf{n}_c$, as variations in $\mathbf{n}_c$ are linked to changes in $\mathbf{y}$. Conversely, ${n}_i$ that do not exhibit a correlation with $\mathbf{y}$ are to belong to $\mathbf{n}_s$.

**R3:** Utilizing mean field variational inference, assuming that the variational posterior distribution are independent and factorizes, is a well-established and widely accepted technique in variational inference. This approach serves as a practical solution to address the inherent computational challenges associated with estimating the true posterior. Specifically, in the context of Gaussian distributions, employing mean field variational inference significantly alleviates the computational complexity, as the inverse of variance scales with $n^3$, a critical factor in optimization. This is a well-founded and reasonable strategy adopted by many researchers in various fields.

Reviewer comment (Novelty):
The contributions of this paper are limited. While this paper introduces a different paradigm, it appears to still be a latent representation disentangle mechanism from a causal perspective.

Author response:
We appreciate your time and careful consideration of our work, as reflected in your insightful review. However, we would like to express a disagreement with the notion that the contributions of our paper are limited. We believe our work significantly advances the field, particularly in the context of Latent Covariate Shift, and we would like to address the specific concerns raised.

New Problem Setting - Latent Covariate Shift: We introduce a novel paradigm, latent covariate shift, which deviates from the commonly-used Conditional Shift. This paradigm shift represents a substantial contribution to the field, offering a fresh perspective on addressing covariate shift issues.

Intricate Latent Causal Generative Model: Our paper presents a sophisticated latent causal generative model, introducing latent content variables and latent style variables. This model provides a nuanced understanding of latent covariate shift mechanisms, offering a more comprehensive and realistic representation.

Theoretical Analysis of Identifiability: We conduct a thorough analysis of identifiability in the proposed causal model, presenting both a complete non-identifiability result and a partial identifiability result for latent content variables. This theoretical foundation adds depth to our work and contributes valuable insights to the research community.

New Method for Latent Covariate Shift: Guided by our theoretical results, we design a new method specifically tailored for latent covariate shift. The proposed method is a practical application of our theoretical findings, showcasing the applicability and relevance of our work.

Verification of Advantages: Through rigorous experimentation, we validate the advantages of latent covariate shift and the proposed method. Our empirical results provide concrete evidence of the effectiveness and superiority of our approach.

In summary, the introduction of a new paradigm, theoretical analysis, method design, and empirical verification collectively demonstrate the significance and impact of our work on addressing latent covariate shift challenges.

Furthermore, we wish to emphasize that the fundamental essence of our contribution is encapsulated within the proposed paradigm. Within this framework, each element of our proposed causal graph, learning method, and identifiability analysis has been meticulously crafted to address the nuances of the entirely novel problem paradigm we have introduced. This deliberate tailoring ensures that our methodologies are not merely incremental but intricately designed to suit the distinctive challenges posed by this innovative problem context. Regrettably, we observe a substantial oversight by the reviewer in recognizing the paramount importance of this foundational aspect.

Reviewer comment (Theory):
The authors propose a more versatile domain adaptation paradigm and construct iLCC-LCS based on it. However, the feasibility of their latent variable modeling approach relies on conditions made in Proposition 4.2, which are not adequately explained and ensured against potential violations.

Author response:
We appreciate your time and careful review again. The main assumptions, encompassing i to iii in Proposition 4.2, stem from the realm of nonlinear ICA. Over recent years, nonlinear ICA has found applications in diverse real-world scenarios, such as domain adaptation [1], image generation and translation [2], domain generalization [3]. Consequently, we assert that these three assumptions have been substantiated through practical applications. We have add a new Section A.8 for understanding these assumptions. 

[1] Kong, Lingjing, et al. "Partial Identifiability for Domain Adaptation." ICML 2022.

[2] Xie, Shaoan, et al. "Multi-domain image generation and translation with identifiability guarantees." ICLR 2022.

[3] Wang, Xinyi, et al. "Causal balancing for domain generalization." arXiv preprint arXiv:2206.05263 (2022).

Reviewer comment (Presentation):
Recommend that the authors provide an overview diagram or a detailed description of the implementation of iLCC-LCS to enhance its intuitiveness and readability.

Author response:
We appreciate your suggestion again. The implementation details has been provided in the original paper, see A.3, we also add a overview diagram in new version (See Figure. 7).

Reviewer comment (Evaluation):
t-SNE can distort the high-dimensional geometry of embeddings. While it can help visualization, it's not suitable for evaluating the quality of embeddings. Could the authors consider employing numerical measures for this purpose?

Author response:
Please note that the main propose of t-SNE in A.5 is to show that the distributions of the learned features by the proposed method change across domain, not to evaluate the quality. Please see A.5 for more details.

Reviewer comment (Writing):
The justification of graph 1(c) is not clear. While the authors justify partial edge directions in Section 3, e.g., $z_c\rightarrow y$ and $z_c\rightarrow z_s$, it is not clear whether there is a real-world setting that fits this graph. Specifically, it would be nice to give a motivating example that clearly explains what each variable $(u,z_c,z_s,y,x)$ refers to and when $p_u(y\mid z_c)$ is invariant and other distributions are variant. The difference between $z_c$ and $z_s$ is not clear as well.

Author response:
**R1:** Thank you for your time and efforts in enhancing this work. For clarifying these questions, we would like first further clarify the notations, including the domain index $\mathbf{u}$, latent content variables $\mathbf{z}_c$, latent style variable $\mathbf{z}_s$, label $\mathbf{y}$, and observation $\mathbf{x}$.

In domain adaptation, latent content variables, latent style variables, domain index, and labels play distinct roles in capturing and aligning information across different domains. Let us consider an example in the context of animal data, were collected from different locations:


Domain index $\mathbf{u}$: it denotes an indicator of the source domain from which the data originates, e.g., locations index).

Latent content variables $\mathbf{z}_c$: this denotes unobservable factors that represent the essential content or intrinsic characteristics of the object. For example: latent content variables encompass features such as the species, or specific morphological characteristics that are inherent to each type of animal.

Latent style variables $\mathbf{z}_s$: These variables aim to capture aspects of the data that are not specific to the intrinsic features, but style-related variations in the data. For example: latent style variables could represent variations in background, lighting conditions, environmental context.

Observation $\mathbf{x}$: The actual image data, including both content and style, including all visual information captured by camera.

Labels $\mathbf{y}$: it represents ground truth labels indicating the class or category of each instance. Example: a label refers to the assigned category or class that indicates the identity or type of each animal in the dataset.

Let us illustrate Figure 1 (c) with an example in the context of animal data. The relationship between $\mathbf{u}$ causing $\mathbf{x}$ is modeled to capture the fact that input in domain adaptation changes across different domains (locations). 

Now, considering why $\mathbf{u}$ causes $\mathbf{z}_c$, let us explore how the unique characteristics and environmental factors associated with each location influence the latent content variables for zebras. For instance, the shades and patterns of the stripes on zebras vary based on the specific location, and certain adaptations to the local ecosystem can be reflected in $\mathbf{z}_c$. 

Moving on to why $\mathbf{z}_c$ causes $\mathbf{y}$, $\mathbf{z}_c$ essentially capture features relevant to predicting $\mathbf{y}$ in a classification task. For a more detailed explanation, please refer to the paragraph 'zc causes y:' in the main paper.

$\mathbf{z}_c$ influence observations $\mathbf{x}$ because they serve as a conceptual representation of the underlying, unobservable factors that contribute to the observed data. 

Finally, let us provide a example why $\mathbf{z}_c$ causes $\mathbf{z}_s$ in Figure 2 (a). An apt example is that different animals (corresponding to $\mathbf{z}_c$) often choose different environments (corresponding to $\mathbf{z}_s$) where the basic needs of the animals to survive are met.

Thus far, we have provided comprehensive explanations for all variables and causal directions depicted in Figure 1 (c) and Figure 2 (a). These causal directions not only elucidate the internal relationships within the models but also indicate how we effectively model the changes in various distributions across domains ($\mathbf{u}$). These considerations underscore the holistic nature of our approach, wherein the causal relationships guide our representation of the dynamic shifts in distributions across different domains.

For when $p(\mathbf{y}∣\mathbf{z}_c)$ remains invariant: it aligns with a fundamental principle in causality, specifically the notion of independent causal mechanisms. This principle posits that the conditional distribution of $p(\mathbf{y}∣\mathbf{z}_c)$ remains stable and unaffected by external factors. In the context of domain adaptation, assuming the invariance of $p(\mathbf{y}∣\mathbf{z}_c)$across domains is a common practice. The invariance in $p(\mathbf{y}∣\mathbf{z}_c)$ is crucial for ensuring the model's ability to make consistent and robust predictions in the target domain. This assumption implies that the predictive relationship between the label $\mathbf{y}$ and $\mathbf{z}_c$ remains unchanged despite variations in data distributions across different domains. This stability facilitates a seamless transition from training on source domains to making reliable predictions in the target domain, contributing to the model's adaptability and generalization.

Reviewer comment (Theory):
Proposition 4.2 shows the identifiability of $z_c$. However, from the result, it is not clear whether $p_u(y\mid z_c)$ is identifiable. Hence, it is not clear whether true $p_u(y\mid z_c)$ can be recovered from data.

Author response:
**R2:** With access to label $\mathbf{y}$ in source domains, and the identified $\mathbf{z}_c$ in those domains, it becomes possible to accurately estimate the parameters of the conditional distribution $p(\mathbf{y}∣\mathbf{z}_c)$. This allows for a robust modeling of the relationship between the latent content variables  $\mathbf{z}_c$ and the corresponding labels $\mathbf{y}$ in the source domains. Then since $\mathbf{z}_c$ in target domain is also identified, and leveraging the invariance of  $p(\mathbf{y}∣\mathbf{z}_c)$, the learned conditional distribution $p(\mathbf{y}∣\mathbf{z}_c)$ theoretically be generalized to the target domain. This generalization is grounded in the notion that the causal relationship between $\mathbf{z}_c$ and $\mathbf{y}$ remains consistent across domains.

Reviewer comment (Theory):
$g_c$ and $g_{s_2}$ in Equation (2) are invertible functions. In this case, it seems trivial to introduce variables $z_c$ and $z_s$ since Equation (3) can be rewritten as
$$
x=f(g_c(n_c), g_{s_2}(g_{s_1}(g_c(n_c)) + n_s))+\varepsilon
$$

Author response:
**R3:** We guess that your concern is that  "$\mathbf{g}_c$ and $\mathbf{g}_s$ in Equation (2) are invertible functions, it seems trivial to identify variables $\mathbf{z}_c$ and $\mathbf{z}_s$". If we understand incorrectly, please let us known. Please be aware that $\mathbf{z}_c$ and $\mathbf{z}_s$ are mixed through the mapping $\mathbf{f}$, and even if $\mathbf{f}$ is invertible, disentangling $\mathbf{z}_c$ and $\mathbf{z}_s$ is a well-known challenging task. Without further assumptions, achieving a clear separation of $\mathbf{z}_c$ and $\mathbf{z}_s$ is considered impossible, as asserted in Proposition 4.1.

Reviewer comment (Writing):
What does the index $i$ and $j$ in Equation (1) refer to?

Author response:
**R4:** i denotes the number of latent noise variable $\mathbf{n}$ or $\mathbf{z}$, and j denotes the number of sufficient statistic for ${n}_i$.

Reviewer comment (Presentation):
From Figure 2(b), it looks like $n_c$ and $n_b$ are observed variables as they are shaded. Seems like a typo?

Author response:
**R5:** Thank you for your careful review. We have correctted it in the new version.