Paper: Identifiable Latent Polynomial Causal Models through the Lens of Change (ICLR-2024)

Reviewer comment (Experiments):
The empirical experiments are somewhat artificial, and do not demonstrate real-world relevance (often the case in papers in this domain). The results are only shown for those setups where the method worked: LinearGamma, LinearBeta, PolyGamma, and not for the difficult cases: e.g. inverse Gamma etc. These could have been included in the Appendix at least, even if they didn’t work well (which the paper openly mentions).

Author response:
Thank you for your suggestion. Given the constraints of the rebuttal time frame, we must prioritize addressing more complex reviews. We will add these experiments in the final verision. Your understanding is greatly appreciated.

Reviewer comment (Theory):
Condition (iii) in Thm 3.1 has u_{2l+1} although u’s exist only until 2l. Possible typo?

Author response:
Please note that the indexing of u starts from 0, not 1.

Reviewer comment (Theory):
Could you include additional intuition for conditions (iii) and (iv) of Thm 3.1?

Author response:
Condition (iii) stems from nonlinear ICA, it is basically saying that the auxiliary variable must have a sufficiently strong and diverse effect on the distributions of the latent noises, to facilitate their identifiability. Condition (iv) is derived from the work by Liu et al. (2022), aiming to constrain the function class of lambda to prevent a specific case, e.g., lambda(u) = lambda(u) + b, where b is a constant. In this scenario, the invariant part, e.g., b * Pa (parent node), leads to unidentifiability, as discussed in Corollary 3.3. We have added Section A. 10 in the revised verison for understanding our assumptions. Thanks for your suggestion.

Reviewer comment (Theory):
Proof sketch of Thm 3.1 says “arises from the fact that additive noise models are identifiable”. If I remember correctly, additive linear Gaussian models are not identifiable? Could you clarify how this is compatible with the sentence?

Author response:
It should be "arises from the fact that nonlinear models with additive noise models are identifiable". Thank you for your careful review. In fact, linear Gaussian models in multiple environments are generally identifiable, thanks to the assumption of independent causal mechanisms [1] [2]  

[1] Ghassami, AmirEmad, et al. "Multi-domain causal structure learning in linear systems." Advances in neural information processing systems 31 (2018).  

[2] Huang, Biwei, et al. "Causal discovery from heterogeneous/nonstationary data." The Journal of Machine Learning Research 21.1 (2020): 3482-3534.

Reviewer comment (Theory):
About Corollary 3.2: There is a condition: “if there is an unchanged coefficient in Eq 4 across environments, z is unidentifiable.” What prevented such a condition from happening in Theorem 3.1?

Author response:
Assumption (iv) imposes a constraint on the function class, ensuring that $\lambda_{i,j}(\mathbf{u})$ is not non-zero constant. Corollaries 3.2 and 3.3, "Under the condition that the assumption (i)-**(iv)**" should be "Under the condition that the assumption (i)-**(iii)**". Thank you for your thorough review.

Reviewer comment (Theory):
Second line of Equation (6): Where is z hidden on the RHS of the equation?

Author response:
Again, that is an excellent question. As non-Gaussian noise lacks an analytically tractable solution in general, we model the prior distribution $p(z|u)$ using $p(\lambda, n|u)$, this is because z can be entirely represented by $\lambda$, $n$, and $u$, as further elaborated in Eq. (14). More deeply, from the perspective of independent causal mechanisms, the latent noise variables $n$ need not to be independent of the coefficient $\lambda$, given $u$, since they both correspond to the mechanism generating the effect on latent causal variables. However, considering the challenges in modeling the complicated dependence between $n$ and $\lambda$, and the challenges in optimization, we employ an approximate implementation, as shown in the second line of Eq. (6), which assumes their independence.

Reviewer comment (Theory):
Prior on lambda: you ensure acyclicity by assuming lambda is lower-dimensional. Does this mean that you have to assume the causal order?

Author response:
Indeed, we have already assumed the causal order, to avoid trouble DAG constraint, e.g., proposed by Zheng et al.  
(2018). Essentaily, this is due to the specific permutation indeterminacy in latent space. To illustrate this, let us examine a scenario involving only two latent causal variables, namely size (z1) and color (z2) of an object, with z1 causing z2. Upon obtaining two recovered latent causal variables, z'1 and z'2, permutation indeterminacy arises, allowing z'1 to correspond to either the size (z1) or the color (z2). This flexibility enables us to pre-define a causal order in the inference model, such as z'1 causing z'2, **without specifying semantic information (size or color) for the nodes**. Consequently, the inference model establishes a causal relationship between z'1 and z'2, prompting them to learn the genuine latent semantic information. In other words, the inference model compels z'1 (or z'2) to acquire size (or color) information, effectively sidestepping DAG constraints. For a more comprehensive and detailed explanation, please refer to Section 3.4 of the work by Liu et al. (2022), where the authors provide a thorough clarification of these concepts.

Reviewer comment (Evaluation):
MPC metric: correlation between what exactly?

Author response:
MPC involves computing the correlation between the true $\mathbf{z}$ and the recovered counterpart $\mathbf{\hat z}$. For a $\mathbf{z}$ with a dimension of 5, the computation process begins by generating a 5 $\times$ 5 matrix, where each element represents the Pearson correlation coefficient between $z_i$ and ${\hat z}_j$. Subsequently, a linear sum assignment problem is solved to match each recovered latent variable with the true latent variable that exhibits the highest correlation. This step effectively reverses any permutations in the latent space. The outcome is a 5-dimensional vector, which is then averaged.

Reviewer comment (Theory):
The last sentence of Discussion: “even with identifiability guarantees, it is still challenging to learn causal graphs in latent space”. It would have been interesting to explore how the number of samples affects this. Do I understand it correctly that the theory guarantees that in the limit of large number of samples the correct graph should be recovered?

Author response:
That is an excellent question as well. The challenge stems not only from the influence of sample size, but also, or most importantly, from the optimization problem. In our implementation, we observe that there always be an approximation between the recovered latent causal variables and the true ones, indicating that exact recovery of latent variables is seldom achieved. This prompts another question: could even a minor error in the approximation result in diverse graph structures? To delve deeper, how does the presence of a small error impact the learned graph structure? Regrettably, we are currently unable to provide a definitive answer to this question. Nevertheless, we acknowledge it as a crucial issue deserving attention and further exploration.

Reviewer comment (Theory):
"Identifiability": It is expected that it is impossible to exactly reconstruct the latent variables from the observed ones, e.g. due to the possibilities of permutation and scaling. So I would expect that your objective is identification up to certain transformations. However, a definition is not given and it remains unclear what you mean precisely by "identifiability" (and by "unidentifiability"), rendering the claims of the corollaries unclear.

Author response:
Please note that the definition of identifiability has already been established in our Theorem 3.1 and in Corollary 3.3, as stated "the true latent causal variables $\mathbf{z}$ are related to the estimated latent causal variables ${ \mathbf{\hat z}}$, which are learned by matching the true marginal data distribution $p(\mathbf{x}|\mathbf{u})$, by the following relationship: $\mathbf{z} = \mathbf{P} \mathbf{\hat z} + \mathbf{c},$ where $\mathbf{P}$ denotes the permutation matrix with scaling, $\mathbf{c}$ denotes a constant vector". Naturally, unidentifiability emerges as the complementary concept to identifiability. That is, z can not be recovered up to $\mathbf{z} = \mathbf{P} \mathbf{\hat z} + \mathbf{c}$.

Reviewer comment (Writing):
Changes across environments are mentioned frequently in section 3.3, but it is difficult to determine from the text what the precise meaning is: for a coefficient, is it (A) that there exist two environments such that the coefficient is different between those, or (B) that for each pair of distinct environments, the coefficient is different between them?

Author response:
Please be aware that we have formulated the changes using the function $\lambda_{i,j}(u)$ for the coefficients as defined in Eq. (4). This implies that the changes are entirely dependent on the function class of $\lambda_{i,j}$, as long as the function satisfies assumption (iv). Additionally, it is crucial to note that we do not make any claims regarding the pair of environments in our results. Specifically, we do not require paired environments, distinguishing our approach from some prior works that do necessitate such pairing. We have also clarified this aspect in our introduction.

Reviewer comment (Theory):
Theorem 3.1 (i), "measure zero (i.e., has at the most countable number of elements)" - I assume you are referring to Lebesgue measure. In dimension 2 or higher, Lebesgue measure zero does not imply that the number of elements is countable. For instance, the set $\{(x,0)| x \in \mathbb{R} \}$ has Lebesgue measure zero, but uncountably many elements.

Author response:
Thank you for such carefully check. In our previous version, we included the phrase 'has at most a countable number of elements' merely for the purpose of easier reader understanding. However, we recognize that this constitutes a stronger assumption than measure zero. In our updated version, we have removed this phrase. Essentially, we here aims to emphasize that $\varphi_{\varepsilon}(\mathbf{x}) \neq 0$ almost everywhere. We appreciate your careful review.

Reviewer comment (Presentation):
A "DAG constraint" is first mentioned in the proof sketch of the main theorem. It would be better form to introduce this assumption prior to the proof.

Author response:
Thank you for such carefully check. We have carefully modified these problems in new version.

Reviewer comment (Writing):
Pages 5 and 7: "identifyable" -> "identifiable".

Author response:
Thank you for such carefully check. We have carefully modified these problems in new version.

Reviewer comment (Writing):
Page 5: "polynomial propriety" -> "the property".

Author response:
Thank you for such carefully check. We have carefully modified these problems in new version.

Reviewer comment (Writing):
Appendix A.4: "support that" -> "suppose that".

Author response:
Thank you for such carefully check. We have carefully modified these problems in new version.

Reviewer comment (Presentation):
Appendix A.4, "proof of corollary A.3" -> "proof of corollary 3.2".

Author response:
Thank you for such carefully check. We have carefully modified these problems in new version.

Reviewer comment (Presentation):
End of section 4: "APPENDIX" -> "Appendix".

Author response:
Thank you for such carefully check. We have carefully modified these problems in new version.

Reviewer comment (Theory):
**Latent structure identifiability**: While the paper makes significant strides in latent variable identifiability, it does not discuss the identifiability of the latent structure under the equivalence class. I understand that once the latent variable values are identified, the latent structure can be recovered trivially. However, given that the latent variables are only recovered up to an equivalence class, especially for partial identification case, it is unclear how would these affect the latent structure identification. This is a crucial aspect in causal representation learning, as understanding the underlying causal structure is often more important than identifying individual variables. The paper could benefit from a more thorough discussion or analysis on this aspect.

Author response:
Thank you for your high-level and insightful suggestions. Our primary finding, as presented in Theorem 3.1, demonstrates that latent causal variables can be identified up to permutation and scaling. Importantly, permutations do not affect the underlying latent graph structure, as they are solely related to namespace. Regarding scaling, as mentioned in our paper, the identifiability of nonlinear models with additive noise remains unaffected by scaling, as we claimed in the paragraph following the proof sketch for theorem 3.1. Consequently, the equivalence class, encompassing permutation and scaling, corresponds to the same graph structure. In terms of partial identifiability, we think that our current results only allow for the identifiability of the values of a subset of latent variables. In such cases, asserting the recovery of a portion of the true causal graph is challenging, given our belief that the unidentifiable part may introduce various possible effects on the graph structure within the identifiable portion. We argue that understanding the underlying causal structure is often more crucial than identifying individual variables. However, for certain applications, such as domain adaptation, we may be interested in splitting the latent space into an invariant space and a variant space, as domain adaptation can leverage the invariant part to predict labels, regardless of the graph structure in latent space. We have incorporated these discussions in the new version. Please see **Section A.8**  in the revised version for more details.

Reviewer comment (Theory):
**Functional assumption**: The necessity and expressiveness of the polynomial assumption are not adequately addressed in the paper. While it's understood that this assumption is needed for identifiability, it would be beneficial to have a more detailed discussion on why this specific form is chosen and how expressive it can be in capturing complex causal relationships. It would also be interesting to explore if there are other forms or assumptions that could achieve similar results.

Author response:
Thank you once again for your insightful suggestions. In general, we believe that model assumptions can be extended to a more general case than polynomial, even for non-parametric models, as long as the changes in causal influences are sufficient. However, defining what sufficient changes poses a challenge. In this work, we opted for polynomial models, considering their approximation ability and simple expression, which is beneficial for analysis and formulating the notion of sufficient changes. Simultaneously, we acknowledge the limitations of this choice, particularly with the presence of high-order terms in polynomials, making optimization highly challenging. Please see **Section A.9** in the revised version for more details.

Reviewer comment (Presentation):
**Presentation**: Given that most of the assumptions have a strong graphical interpretation, the paper could benefit from adding more illustrations or visualizations to help readers better understand these assumptions and their implications.

Author response:
Thank you once again. We also recognize the significance of facilitating a better understanding of these assumptions for our readers. Consequently, we have augmented our presentation by including a dedicated in **Section A. 10**  in the revised version to provide detailed insights into these assumptions. For example, two of the main assumptions (iii) and (iv) in Theorem 3.1. Assumption (iii) stems from nonlinear ICA, essentially requiring sufficient changes in latent noise variables to facilitate their identification. Assumption (iv) is derived from the work by Liu et al. (2022), aiming to constrain the function class of lambda to prevent a specific case, e.g., lambda(u) = lambda(u) + b, where b is a constant. In this scenario, the invariant b leads to unidentifiability, as discussed in Corollary 3.3.

Reviewer comment (Novelty):
My major concern with the work is regarding the technical contribution and significance of the proposed identification result. The proof uses common ideas of exponential family distribution [1] to obtain identification guarantees up to permutation and scaling for the noise variables. Further, the idea of using the changing distribution of latent variables identified up to polynomial mixing can be used to obtain permutation and scaling identification guarantees has been explored in prior works [2]. While I agree that the application of these ideas to the task of learning latent causal relationships is not trivial (as noted in the Strengths section above), I am not convinced by the significance of the proposed identification result. In terms of only the latent identification (not recovering causal relationships), the prior works that leverage interventional data can already provide permutation and scaling guarantees with much more general latent structures. For the task of learning the latent causal relationships, the permutation indeterminacy in the learned latents would pose a huge challenge which the authors circumvented by assuming the correct topological order. This makes it difficult for me to assess the importance of the proposed identification result.

Author response:
It appears there might be confusion. We main result, Theorem 3.1, is using the result from [2], not [1], to obtain identifiability up to permutation and scaling for the noise variables.

[1] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859–868. PMLR, 2019.

[2] Sorrenson, Peter, Carsten Rother, and Ullrich Kothe. "Disentanglement by nonlinear ica with general incompressible flow networks." arXiv preprint arXiv: 2001.04872 (2020).

Please be aware of a substantial disparity between the identifiability of latent noise variables n and latent causal variables z. Notably, assuming an absence of all edges among z makes it trivial to apply the results of nonlinear ICA. However, in real-world applications, latent graph structures can be arbitrary. Consequently, the solution space for identifying z is notably more expansive than that for identifying n. This disparity is critical, especially given the complexity introduced by arbitrary graph structures. Essentially, many existing works focusing on the changes of causal influences, e.g., hard intervention or soft intervention, are dedicated to addressing this pronounced gap. 

**R2**: It is essential to highlight that [2] achieves the recovery of latent causal variables identified up to polynomial mixing by assuming a polynomial mapping from latent causal variables z to observed x. In contrast, our approach employs a more general mapping f, as defined in Eq. (3). Consequently, in our setting, the task of recovering latent causal variables identified up to polynomial mixing becomes more challenging due to the increased flexibility and complexity introduced by the general mapping function f.

Most importantly, the distinction between this work and [2] is substantial. [2] utilizes hard interventions to identify latent causal variables, while this work employs soft interventions for the same purpose. To elucidate this significant difference, let us consider a fundamental question: how can hard interventions be practically applied to latent causal variables?

We posit that conducting hard interventions on latent space is almost impractical due to the unobservable nature of latent causal variables. Consequently, hard interventions should be understood as modeling changes generated by self-initiated behavior within causal systems across environments. However, the self-initiated changes in causal systems across environments can be arbitrary. In this context, hard interventions in [2] only allow model specific types of changes, while this work, involving soft interventions, can model a broader range of possible changes. This versatility may prove more attainable for latent variables than the constrained nature of hard interventions. 

**R3**: Please take note that our identifiability results, elucidated in Theorem 3.1, inherently signify the identifiability of the causal graph structure, as explicitly stated in the paragraph following the proof sketch for Theorem 3.1. This assertion is grounded in the inherent identifiability of nonlinear models with additive noise. Please see the detailed clarification provided in the corresponding paragraph for a more thorough and comprehensive understanding of this crucial aspect. Furthermore, it is important to highlight that our results are not contingent upon any assumptions regarding latent structures. This underscores the flexibility of our framework, allowing for the consideration of arbitrary structures.

**R4**: We would like to clarify that we do not circumvent permutation; rather, we strategically leverage it to overcome traditional Directed Acyclic Graph (DAG) constraints, as proposed by Zheng et al. (2018). To illustrate this concept, consider a scenario with only two latent causal variables, namely size (z1) and color (z2) of an object, with z1 causing z2. Upon obtaining two recovered latent causal variables, z'1 and z'2, permutation indeterminacy arises, allowing z'1 to correspond to either the size (z1) or the color (z2). This flexibility empowers us to pre-define a causal order in the inference model, such as z'1 causing z'2, without specifying semantic information (size or color) for the nodes. Consequently, the inference model establishes a causal relationship between z'1 and z'2, prompting them to learn the genuine latent semantic information. In other words, the inference model compels z'1 (or z'2) to acquire size (or color) information, effectively sidestepping DAG constraints. This strategic use of permutation indeterminacy enhances the adaptability of our approach. For a more comprehensive and detailed explanation, please refer to Section 3.4 of the work by Liu et al. (2022), where the authors provide a thorough explanation elucidating the rationale behind the predefinition of a causal order in inference models.

Reviewer comment (Experiments):
While I appreciate the experimentation over several benchmarks, the scale (dimension of the latent variables) of the experiments is too small, with the maximum dimension of the latent space being 5. It would be nice to see the results with the proposed approach for larger latent dimensions ($d= 10$ or $d=20$ at least) for the synthetic benchmark.

Author response:
Currently, causal representation learning is in its initial stage, with a predominant focus on pivotal steps like identifiability in most existing works. The optimization problem in latent space is acknowledged as a significant challenge. Typically, only an approximation of the true latent variables can be obtained, and even a slight error in this approximation can lead to diverse learned causal graphs. This is one reason why many existing works opt for experiments in settings with small dimensions of latent variables. In typical cases, the value of d is less than 10. We anticipate that identifiability results will converge in the coming years, paving the way for more researchers to address the optimization challenges that follow. We will expand our experiments to include latent variables with larger dimensions and offer a more in-depth discussion in final version, even if the results may not be as successful. Thanks for your suggestion.

Reviewer comment (Experiments):
Is there any particular reason why the authors did not compare against several latent identification works that leverage interventional data? That would be a better choice of baselines as compared to Beta-VAE, which does not explicitly utilize the variation of latent variables with auxiliary variables.

Author response:
Response:  The fundamental issue lies in the lack of alignment in assumptions among works employing interventions. For example, there is a divergence in the nature of interventions, with some works assuming hard interventions and others assuming soft interventions. Additionally, discrepancies extend to the choice of models, where some studies assume linearity within latent spaces, while others opt for nonlinear models. There are also variations in the requirement for paired interventional data, whereas this study focuses on unpaired data. These misalignments present significant hurdles for conducting fair and meaningful comparisons, rendering it challenging for most existing works, including this one, to offer comprehensive and meaningful comparisons. Further, we anticipate that the assumptions for identifiability will converge in the coming years, paving the way for fair comparisons. We think this convergence promising, as the utilization of both hard and soft interventions is essential for modeling the changes in causal influences.

Reviewer comment (Evaluation):
It would be good to plot the F1 score along with SHD for the performance of latent causal discovery as well. The authors could maybe set a threshold to determine the latent causal structure from the functional relationships and compare that against the true latent causal graph.

Author response:
We appreciate your input and will add it for the final version. However, given the constraints of the rebuttal time frame, we must prioritize addressing more complex reviews. Your understanding is greatly appreciated.

Reviewer comment (Novelty):
Liu et al. 2022 plays a central role in this work, since this work poses a direct successor in the form of an incremental improvement. While an incremental improvement can be of substantial nature, IMHO this work poses a Corollary-type of discussion over Liu et al. Furthermore, the presentation (especially in direct comparison to Liu et al.) does not provide any further justice to the present manuscript's proposed contribution. There is an abundance of repetition of the form "we've extended it now" accompanied by a lack of how alleged extension was achieved. Consulting the appendix did not resolve this major concern.

Author response:
**Q1: an incremental improvement, a lack of how alleged extension was achieved**

**Response**: We respectfully disagree with that. Our contributions, highlighted throughout the abstract, introduction, and theoretical results, encompass the following advancements:
 
1) The dependencies among latent variables, previously assumed to be linear in Liu's work, are demonstrated to be polynomial in our research.
2) Contrary to the Gaussian assumption in Liu's work, we establish that the noise associated with latent variables follows an exponential distribution.
3) In comparison to Liu's prior work, our proof requires a significantly reduced number of environments (2l+1 environments), a substantial improvement from the previous quadratic requirement in Liu's work.
4) We also provide proofs for partial (un)identifiability, specifically addressing situations where the assumption of a complete change in causal influences is violated.
5) Introducing an estimation method, we validate its efficacy on both synthetic and real-world datasets.


All theoretical contributions are rigorous justified by the Theorem 3.1, Corollary 3.2, Corollary 3.3, and their respective proofs. Further, experimental results serve to affirm not only the soundness of our theoretical contributions but also the practical efficacy of the proposed estimation method. Finally, to understand the motivation behind these theoretical advancements, our Introduction sheds light on the fact that LIU's work primarily focuses on identifying latent causal variables from the perspective of observed data. In contrast, our research is propelled by the realization that we can significantly narrow down the solution space of latent causal variables by considering the perspective of latent noise variables. This new insight motivates us to propose more general models than LIU's work.

Reviewer comment (Writing):
Kronecker product definition is not clear, not the usual definition e.g. $(a_1, a_2) \times B = (a_1\cdot B, a_2\cdot B)$.

Author response:
**Q2: Kronecker product definition is not clear, not the usual definition.**

**Response**: We deviate from the conventional definition by employing the notation ${\bar \otimes}$ in place of the usual ${\otimes}$ (usually denote the Kronecker product) to denote the operator. Additionally, we explicitly elucidate that the operator, denoted as ${\bar \otimes}$, signifies the Kronecker product with all distinct entries.

Reviewer comment (Theory):
Corollary 3.2. seems to carry no value when considering that the whole paradigm of change of causal influence is based on the complete change assumption.

Author response:
**Q5: Corollary 3.2. seems to carry no value when considering that the whole paradigm of change of causal influence is based on the complete change assumption.**

**Response**: There may be a misunderstanding of the Corollary 3.2 in the review. The Corollary aims to explore the necessity of the assumption that requiring all coefficients within polynomial models to change for complete identifiability, without relying on any additional assumptions. Exploring the necessity of the assumption is important for robustness of identifiability results, generalizability, and so no.

Reviewer comment (Writing):
Given the heavy reliance on Liu et al. 2022, the authors should consider actually saying what "transitivity" is within their introductory section and avoid using formulations that Liu et al. use in different contexts (e.g. "absorbs").

Author response:
**Q6: The suggestions for presentation.**

**Response**: We sincerely appreciate these valuable suggestions regarding our presentation and will thoughtfully consider their implementation.

Reviewer comment (Presentation):
Please consider rewriting the introduction s.t. redundancy with Section 2 is being reduced. The extra space that is won through this is valuable presentation space, which avoids having to put otherwise more important results into the appendix.

Author response:
**Q6: The suggestions for presentation.**

**Response**: We sincerely appreciate these valuable suggestions regarding our presentation and will thoughtfully consider their implementation.

Reviewer comment (Presentation):
Please consider removing the pre-text at the beginning of Section 3, in favor of relevant discussions, as it arguably provides no additional value given all the text before (Abstract and Sec.1,2).

Author response:
**Q6: The suggestions for presentation.**

**Response**: We sincerely appreciate these valuable suggestions regarding our presentation and will thoughtfully consider their implementation.

Reviewer comment (Presentation):
Please consider improving accessibility to the Figures by resolving the issues on the following two dimensions: (1) label/legend and general font sizes but also (2) proportions e.g. the choice of content being presented (e.g. Figure 6 and 7 do not convey valuable information w.r.t. this aspect).

Author response:
**Q6: The suggestions for presentation.**

**Response**: We sincerely appreciate these valuable suggestions regarding our presentation and will thoughtfully consider their implementation.