From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: Counterfactual modeling provides a conceptual basis for understanding image modifications. Given the degrees of freedom in such a task, it is relevant to go further on the idea, tapping not only on off-the-shelf causal concepts but specializing them for the task at hand. This paper provides conceptual advances on how to integrate ideas from deep generative models and counterfactual modeling.

Strengths: it provides original foundational advances on the problem of image generating by tapping into its own take of counterfactual modeling.

Weaknesses: there are still degrees of freedom to be decided, and further discussion to which extent just throwing more data at the problem would suffice to fill up some of these gaps.
justification_for_why_not_higher_score: There are still degrees of freedom to be decided, and further discussion to which extent just throwing more data at the problem would suffice to fill up some of these gaps.
justification_for_why_not_lower_score: The paper provides original foundational advances on the problem of image generating by tapping into its own take of counterfactual modeling. It's not only originality for its own sake, but the problem targeted is of particular relevance to ICLR.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper368/Authors
title: Discussion period ending soon
comment: Dear reviewers,

We would like to thank you for spending your time carefully evaluating our submission and providing valuable feedback.
As the initial rebuttal period is expected to conclude soon, we hope our response addresses your concerns. We look forward to hearing your feedback if you have any further requests for changes to the paper. We are happy to engage in further discussions.

Best Regards,

Author

From: ICLR.cc/2023/Conference/Paper368/Authors
title: Reply to reviewer o4KV (weakness 4)
comment: We recognise that effectiveness is the most challenging of the three properties to evaluate faithfully. In most real-world scenarios, it is not feasible to use a human-in-the-loop system due to labour constraints or to hardcode an accurate enough oracle due to the complexity of the problem. Thus we rely on a data-driven approach with its strengths and weaknesses. While the accuracy of the proposed effectiveness metric depends on the accuracy of the trained oracle, the accuracy of the trained oracle is independent of the counterfactual inference model. As a result, we know apriori how good/reliable our pseudo-oracle is before evaluating the model.

Additionally, even though the reliability of data-driven pseudo-oracles is inevitably limited by their accuracy, we have shown empirically in the appendix that for the purpose of relative ranking of models, an imperfect pseudo-oracle seems sufficiently reliable.

From: ICLR.cc/2023/Conference/Paper368/Authors
title: Reply to reviewer o4KV (weakness 3)
comment: Consider the following definition of reversibility (Pearl2009 sec 7.3):

"If setting a variable A to a value a results in a value b for a variable B, and setting B to a value b results in a value a for A, then A and B will take the values a and b."

Consider also the following twin network representation of a (Markovian) SCM:

$Pa → X ← \epsilon → X^* ← Pa^*$

For an invertible mechanism, given $Pa$, setting $X$ to a value $x$ results in a uniquely defined value for the exogenous noise $\epsilon$.
Since $X^*$ is a counterfactual of $X$, given $Pa^*$, setting $X^*$ to $x^*$ must result in the same unique value for the exogenous noise $\epsilon$. Plugging into the definition above:

"If setting a variable $X$ to a value $x$ results in a value $x^*$ for a variable $X$, and setting $X^*$ to a value $x^*$ results in a value $x$ for $X$, then $X$ and $X^*$ will take the values $x$ and $x^*$."

As a result, reversibility implies cycle consistency between the factual and counterfactual for an invertible mechanism.
The proposed reversibility metric is only applicable to invertible mechanisms. In our experiments, we assume that the mechanisms are invertible. We have clarified this in the revised version.

From: ICLR.cc/2023/Conference/Paper368/Authors
title: Reply to reviewer o4KV (weakness 2)
comment: Considering only Markovian causal models (see point 1), we use counterfactual identifiability in the broader sense as the ability to correctly estimate the true counterfactual distribution. In the case of a Markovian SCM, this implies uniquely abducting the exogenous noise (inverse model $\epsilon = f^{-1}(x, pa)$) and uniquely estimating the generative model ($x = f(\epsilon, pa)$).

For deep latent variable models, counterfactuals are not identifiable when the exogenous noise variables are assumed to be jointly independent (which is a constraint of Markovian causal models) (Locatello2020, Khemakhem2020).
This impossibility result is due to the existence of multiple solutions to the latent variable distributions that result in the same observational/counterfactual distribution. Thus, one cannot recover the true joint distribution.

We recognise that the following statement in the paper was confusing:

“composition, effectiveness and reversibility are necessary and sufficient for counterfactual identifiability.”

We have amended it to:

“composition, effectiveness and reversibility are the necessary and sufficient properties of counterfactuals in any causal model.” (Pearl2009 sec 7.3)

When the counterfactual is not underdetermined by the data, we can use these axiomatic properties to measure how close an approximate counterfactual inference model is to the ideal/identifiable model.
We have attempted to clarify this point throughout the paper (see changes highlighted in blue in the pdf). Thanks for raising this point.

From: ICLR.cc/2023/Conference/Paper368/Authors
title: Reply to reviwer o4KV (weakness 1)
comment: Our work considers only Markovian causal models: acyclic and with jointly independent exogenous noise variables. We have clarified this in the revision (changes highlighted in blue). We believe it is an important class of models often considered in ML applications due to convenience and sufficient generality for most use-cases.

From: ICLR.cc/2023/Conference/Paper368/Authors
title: Reply to reviewer o4KV (intro)
comment: We thank the reviewer for the in-depth and thorough review, the positive feedback and the insightful comments. Below we address the comments and suggestions individually.

From: ICLR.cc/2023/Conference/Paper368/Authors
title: Reply to reviwer cqtT
comment: We thank the reviewer for their insightful review and positive feedback. We appreciate the words of encouragement regarding the importance of counterfactual assessment techniques. Below we address the comments on the suggestions.

"It should be more discussed on the properties the distance metric, dx, shown in Equations (1) and (2) should satisfy. It is sure that distance is equal to zero if the counterfactual function is ideal. In case the counterfactual function is not ideal, the distance would be larger than zero and the inference quality is measured by the distance metric quantitatively. The reviewer believes more discussion on the distance between given two images is needed."

Thanks for the suggestion to include further discussion about the distance metric. We have added this to the appendix. We discuss what properties a distance must have to be a metric and motivate our choice of the l1 distance.

"For the check of Effectiveness, the method uses a pseudo-oracle function that infers the parents of a variable. An image generative model often generates artificial images that do not obey the probability density distribution of (real) images. When a generated counterfactual image deviates from the distribution of real images, the reviewer thinks, the pseudo-oracle function should not output any parents of a given image and should degrades the rating of Effectiveness."

Thanks for this thoughtful comment. We recognise that effectiveness is the most challenging of the three properties to evaluate faithfully. In most real-world scenarios, it is not feasible to use a human-in-the-loop system due to labour constraints or to hardcode an accurate enough oracle due to the complexity of the problem. Thus we rely on a data-driven approach with its strengths and weaknesses. We agree that the effectiveness oracle should ideally be able to detect whether a generated counterfactual deviates from the distribution of real images. However, all data-driven methods are susceptible to this type of vulnerability: adversarial examples, distribution shift, robustness etc. Solving these problems falls out of the scope of this work. However, work that improves those areas would directly translate into our approach. Finally, we highlight how the three properties work together to evaluate a counterfactual model. The remaining properties of composition and reversibility can flag inadequate counterfactuals. For example, in the VAE vs GAN 3D shapes experiment, we could argue that the GAN counterfactuals are out-of-distribution due to the distortion introduced in the images. Even though the effectiveness of the VAE and GAN are comparable, we can see that the GAN has lower composition and reversibility. Another example: suppose we train an adversarial counterfactual model which achieves perfect effectiveness from counterfactuals that look like white noise. The model would fail in composition and reversibility, prompting us to dismiss the generated counterfactuals and model.

From: ICLR.cc/2023/Conference/Paper368/Authors
title: Reply to reviwer segb
comment: We thank the reviewer for their time and for recognising the novelty and contributions of the paper. We are encouraged by the positive feedback provided. Below we address the comments.

“It may have been interesting to have a short discussion (maybe in the Appendix) on the choice of metrics.”

Many thanks for the suggestion. We agree, and as suggested, we have added a section (Section B) to the appendix where we discuss and motivate the choice of metrics.

“If feasible, comparison with other counterfactual deep models would have been instructive.”

Thank you for the suggestion. In the paper, we have included an extensive range of experiments using VAEs, GANs and deep hierarchical VAEs on multiple datasets of varying complexity. Current SOTA counterfactual deep models are based on these components with minor modifications to how abduction is performed. Therefore, we believe our choices are representative of the SOTA. Given that our work focuses on demonstrating how one can evaluate approximate counterfactual inference models, we feel that additional models would not alter the takeaway message and contribution.

From: ICLR.cc/2023/Conference/Paper368/Reviewer_o4KV
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes some metrics to evaluate counterfactual generation models, based on Galles and Pearl's axiomatization of counterfactuals that are interpreted according to (a certain class of) structural causal models. Corresponding to each of the three axioms -- effectiveness, composition, and reversibility -- a metric (family) is developed in the paper. Using these metrics, the authors examine the empirical behaviors and performance of some popular generative models, and demonstrate in experiments the potential of these metrics for indicating the quality of counterfactual generation. The extent to which simulated intervention can be used to mitigate confounding is also explored in the experiments. 
strength_and_weaknesses: Strengths:

1. The central idea of designing evaluation metrics based on Galles and Pearl's axioms is novel. 
2. The development of the metrics is well argued and explained.
3. The empirical studies are interesting and seem to provide good evidence for the potential usefulness of the proposed metrics.
4. The demonstration of the utility of simulated intervention is clever and fairly convincing.

Weaknesses:

1. Galles and Pearl's axioms are proposed to characterize a certain class of structural causal models, i.e., those models that imply that the endogenous variables are fully determined by the exogenous variables (including the exogenous noises). This class is on the one hand more general than the usually assumed recursive models and on the other hand quite special among non-recursive models. This point is only alluded to but not made sufficiently clear in the paper. It should be explicitly discussed to properly qualify the intended applications of the proposed metrics. In particular, if more general feedback mechanisms that allow multiple equilibria are of interest, composition and reversibility will need to be weakened to accommodate them. 
2. It is not clear what "counterfactual identifiability" exactly means in this paper. If it means identifiability of counterfactual queries from observational data (or perhaps also interventional data?), I don't see how the soundness and completeness results imply that "composition, effectiveness and reversibility are necessary and sufficient for counterfactual identifiability." It is possible to have a causal model that does not satisfy some of these properties (because it is outside of the class mentioned above), but still allows some counterfactuals to be identified, isn't it? Conversely, when data underdetermine the underlying causal model, even if all the possible candidates are assumed to satisfy these properties, counterfactuals may still not be identified. It seems to me that either the authors intend to use the term "counterfactual identifiability" in a very special sense, or there is some misunderstanding of what the completeness theorem amounts to.
3. The proposed metric for reversibility seems to be much more about the invertibility of a causal mechanism than about reversibility. In particular, I don't see exactly how the proposed metric derives from applying (something like) reversibility to "a twin network conception of the SCM." If it is really just a metric for invertibility, it is doubtful whether it is appropriate to require a counterfactual inference model to respect it (because invertibility may be too strong a requirement in many contexts). 
4. The proposed metric for effectiveness is apparently a metric to evaluate a combination of a counterfactual inference model and a pseudo-oracle function, rather than of the counterfactual inference model by itself. It is hence a little unclear in principle whether it is suitable for evaluating the latter. 
clarity,_quality,_novelty_and_reproducibility: I am quite impressed by the novelty and technical quality of the paper. Overall it is also clearly written, but clarity is a little lacking regarding those conceptual issues I listed above. I don't see any reason to doubt the reproducibility of the empirical results.
summary_of_the_review: This is a novel and interesting paper with quite a few merits. But I am also puzzled by several conceptual aspects of the paper, as indicated in my comments on the weaknesses. (I will raise the score if those comments prove to be my confusions in the subsequent discussions.) 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper368/Reviewer_segb
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes an approach to learn counterfactual functions based on the properties that the axiomatic definition of counterfactuals require (composition, reversibility, effectiveness). Evaluating the degree to which this properties are satisfied allows to assess the distance between an ideal counterfactual and its approximation.
strength_and_weaknesses: + The paper considers a relevant problem from an original perspective
+ The proposed framework is grounded in SEM theory
+ The framework is applied in representative experiments and datasets.
+ Experiments, ranging from simpler to more realistic examples are well designed, discussed and commented, providing an illustration of the application of the author's frameworks.

- It may have been interesting to have a short discussion (maybe in the Appendix) on the choice of metrics.
- If feasible, comparison with other counterfactual deep models would have been instructive.
clarity,_quality,_novelty_and_reproducibility: The paper is clear (explaining thoroughly the reasoning from the definition of conterfactuals to counterfactual functions), and of high quality. To the best of my knowledge, reliance on axiomatic properties for learning counterfactuals is novel. Experiments are well presented and documented in the appendix.
summary_of_the_review: The paper offers a good contribution which may be of interest for anyone working on causality and causal learning.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper368/Reviewer_cqtT
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a method that evaluates counterfactuals.  The method does not focus on the algorithm (or models) for the inference of counterfactuals but on properties the inferred outputs must obey. Based on Pearl’s axiomatic definition of counterfactuals, the authors propose to check three properties: Effectiveness, Composition, and Reversibility. Experimental results demonstrate the utility of the proposed evaluation method. 
strength_and_weaknesses: Strong points:
As mentioned by the authors, there is little methodology available to assess the quality of inferred counterfactuals. The reviewer believes the assessment of counterfactuals is very important and methodologies of the counterfactuals assessment should be discussed more vigorously. The idea to focus on the properties the outputs must obey is novel and the list of the three properties spearheads research on counterfactual inference.


Weak points:
It should be more discussed on the properties the distance metric, dx, shown in Equations (1) and (2) should satisfy. It is sure that distance is equal to zero if the counterfactual function is ideal. In case the counterfactual function is not ideal, the distance would be larger than zero and the inference quality is measured by the distance metric quantitatively. The reviewer believes more discussion on the distance between given two images is needed.

For the check of Effectiveness, the method uses a pseudo-oracle function that infers the parents of a variable. An image generative model often generates artificial images that do not obey the probability density distribution of (real) images. When a generated counterfactual image deviates from the distribution of real images, the reviewer thinks, the pseudo-oracle function should not output any parents of a given image and should degrades the rating of Effectiveness.
clarity,_quality,_novelty_and_reproducibility: The paper is well organized and the proposition is novel.
summary_of_the_review: The reviewer recommends to accept this paper. The reviewer believes the evaluation of counterfactuals is an important research topic and is worth discussed in the conference. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper