From: ICLR.cc/2023/Conference/Paper1889/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1889/Authors
title: Respond to Reviewer GQzM
comment: 1.The main motivation of the paper is not correct. The authors argue that the covariate balancing algorithms are only applicable to linear and binary settings. In fact, there are off-the-shelf generalizations for higher order covariate balancing and non-binary treatments. Here is a list. As a conclusion, I do not see a clear motivation for this paper.

We have read all the references you offered to us. Stable learning and the method(off-the-self generalizations for higher order covariate balancing and non-binary treatments) you mentioned are totally different models and the problems they solved are also totally different. For example, for this paper you mentioned ‘Covariate Balancing Propensity Score for A Continuous Treatment: Application to the Efficacy of Political Advertisements’, they aim to minimize the association between covariates and the treatment. In our paper, we didn’t introduce treatment. The purpose of stable learning is to eliminate the relationships among the features, not the treatment and features. In real world datasets, it is hard to get the treatment variables. For example, stable learning can be applied into deep learning such as Computer Vision:

 [1] Zhang, X., Cui, P., Xu, R., Zhou, L., He, Y. and Shen, Z., 2021. Deep stable learning for out-of-distribution generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5372-5382).

2.The proposed approach is only a heuristic and the authors do not provide any theoretical correctness guarantees.

We added the proof in the appendix.

From: ICLR.cc/2023/Conference/Paper1889/Authors
title: Respond to Reviewer syzi
comment: 1.It is not clear to me whether the assumptions made in the paper are too restrictive. The authors may provide justification and explanations of these assumptions.

Thanks for your suggestions. Our model is a very general causal inference method. These three kinds of assumptions are just corresponding to different parts of methods, and they are not restrictive. For the first assumption: the set of strong causality rules is a subset of the set of rules with strong correlations, it means that strong correlations is not equivalent to strong causality, but strong causality has strong relationships. Hence, that’s the reason why we set low thresholds for association mining methods. We can extract most association rules with strong causality. Meanwhile we also extract lots of redundant rules with high conditional probability(confidence) or high joint probability(support). Because, the Apriori algorithm will always add items into the rules until the confidence or support of this rule is lower than the threshold. Hence, for example, A=>B is the causal rule, but Apriori algorithm will add item C into this rule just like: A, C => B, and C is the redundant item. That’s why the third assumption: causal rules tend to include smaller antecedents, is correct and not restrictive. For the second hypothesis, our paper aims to solve observed confounding. This hypothesis means that if there are lots of unobserved confounding variables, we cannot extract complete causal rules. However, in the datasets, most of these datasets can include lots of observed features, while the unobserved parts are not very enough. This hypothesis is not equal to our model cannot deal with unobserved confounding. In the experiment(Appendix A.2), we construct some unobserved variable to test the robustness of our model, and our model performs well.

2.It is not clear to me how the Taylor expansion techniques can be used to estimate causal relationships.

Taylor expansion techniques are not used to calculate causal relationships. In our paper, we apply Taylor expansion techniques to denote the relationship between two features and to construct regularizer for decorrelation. Hence, our model includes two kinds of parts: decorrelation and calculating the causality. Some references are attached here that can explain why after decorrelation we can get the causality and why stable learning can work:

[1] Xu, R., Cui, P., Shen, Z., Zhang, X. and Zhang, T., 2021. Why stable learning works? A theory of covariate shift generalization. arXiv preprint arXiv:2111.02355.\
[2] Kuang, K., Xiong, R., Cui, P., Athey, S. and Li, B., 2020, April. Stable prediction with model misspecification and agnostic distribution shift. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 4485-4492).\
[3] Xu, R., Zhang, X., Shen, Z., Zhang, T. and Cui, P., 2022, June. A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization. In International Conference on Machine Learning (pp. 24803-24829). PMLR.

For the decorrelation part, we don’t need to calculate the causal relationships. For example, there are two features A and B, the causal relationship between them is A=a1*B+a2*B^2+C, and a1=0.5, a2=0.6. The coefficients calculated by Taylor expansion are a1=0.6, a2=0.55, D is other values: A=a1*B+a2*B^2+D. The only thing we should do is to promise a1=0 and a2=0 after decorrelation. Because after decorrelation we can make the two features nearly independent, even though we cannot calculate the causal relationships.

3.There is a lack of theoretical justification for the proposed method.

Thank you for your suggestions. We have added proof in the appendix(A.5).

From: ICLR.cc/2023/Conference/Paper1889/Authors
title: Respond to Reviewer CHFK
comment: 1.The background is only briefly described, and many terms are used without prior introduction. The work could benefit from a more formal and self-contained problem setting

Thank you for your suggestions. We have revised our background and introduced every term in the introduction and related work.

2.The proposed method is not well motivated and not described clearly. Many terms are never properly introduced (e.g., p1,p2,frequent,xp1(0) etc.). Why the choice of Taylor expansion for modeling non-linearities?

Thanks for your suggestions. We have introduced these terms in the new revision. We choose the Taylor expansion for modeling nonlinearities because we have theoretical guarantees. We added the proof in the appendix.


3.Table 2 - as I see it this is not an ablation study but a (grid) hyperparameter search. What dataset did the authors use for this? Was it a different setting than the one the authors evaluated on? If not, the results of Figure 2 can hardly be objective. Was a similar search performed for the baseline methods?

We applied synthetic datasets in this experiment(as shown in the appendix A.2), and they are the same settings. For the DWR method, we applied the code with parameters which are published on Github by the original author.

4.Section 4.2.1, Feature Selection: Authors say they use 4 different methods to select the features for their model. This procedure is unclear. Are all 4 methods used simultaneously? Are the AUROC curves computed on a separate validation set? Is this feature selection method also applied for baseline methods?

Thanks for your suggestions, we have introduced more details of feature selection in our revised paper. We first applied 4 different methods to compute AUROC. Then we chose the model with the highest AUROC to select the features. Yes, the performance is computed on the separate validation set since this process is based on a 5-fold cross-validation. All baseline methods are based on the same features.

5.Why is there no comparison to DWR in experiments on real-world datasets?

Thanks for your advice. We added this comparison experiment in the revised version.

6.Table 1 is never referenced in text. It is also unclear what the different betas correspond to

Thanks for your suggestion. We have introduced the details of Table 1 in the revised version. In the appendix, we introduce what the difference is between the beta_s and beta_v.

7.Table 3 - results are barely better (often only by the third decimal point) from other methods. Also in case of identical results for two methods, authors only highlight theirs

There might be some misunderstandings. Firstly, Table 3 is used to say that our model is able to achieve better causality results under similar accuracy performance. Secondly, although the other models can get high accuracy, they learn parameters according to probability distributions which are not the same as causality. Our model, on the other hand, is proposed to compute causality. Therefore we compared the weight calculated by each model on the real data set with the expert's score and calculated the Spearman Coefficient to measure the quality. As a result, we found that the weights calculated by our model achieve better match with the doctor's evaluation scores (refer to the causality in Table 3).

From: ICLR.cc/2023/Conference/Paper1889/Authors
title: Respond to Reviewer ownt
comment: We appreciate your comments and suggestions. Please refer to the below answers to your questions.

1.It is better to provide theoretical analysis for Equations 2 and 6 to prove the proposed methods working.
We have introduced a new section in the appendix(A.5) to give the proof of our algorithm.

2.The novelty of the paper is a little bit weak, as it adopts the existing approaches and looks like data processing in sections of Mining Association Rules and Selecting Rules.

There might be some misunderstandings here. Our aim is to build a novel and effective causal rules construction system for real-world healthcare applications. Mining association rules is a step of our algorithm. It is unfair to judge the novelty of our whole system by simply focusing on association rule mining. Besides, we propose a novel causal inference method and object loss function for constructing causal rules (equation (2,6)). This should be considered as an important part of our novelty. The novelty of the paper is not only rules but also more about causal inference. Mining Association Rules is just a minor part of our algorithm.

3.Although the experiment setting is listed in Appendix in section 4.1, it is still not clear.

In the revised version, we explained more about the details of the experiment. Could you please demonstrate clearer about your confusion points if that doesn’t answer your question?

4.The size of data sets is much small, for example, Esophageal Cancer consists of data from 261 patients. So the experimental results are not convincing.

If you are familiar with healthcare datasets, the size of our dataset is not small. Collecting healthcare datasets is nontrivial considering the quantity of patients itself is small. In particular, healthcare data collection needs to go through strict board reviews and privacy policies which usually regularize the dataset scale. On the other hand, we validate our algorithm on various real world datasets, and all of them show promising results indicating our algorithm’s ability to work in various environments.

From: ICLR.cc/2023/Conference/Paper1889/Reviewer_ownt
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes to combine machine learning with association rules to increase model interpretability in healthcare applications. the many-domain generalization problem by treating each patient as a domain in healthcare applications. The paper expands the stable learning problem from a linear to a nonlinear environment. Experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed approach.
strength_and_weaknesses: S1: The methodology part is well addressed the main claims in the introduction part.

S2: It is an interesting idea to apply association rules to increase model interpretability.

S3: The extensive experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed model.

written in a clear way and

W1: It is better to provide theoretical analysis for Equations 2 and 6 to prove the proposed methods working.

W2: The novelty of the paper is a little bit weak, as it adopts the existing approaches and looks like data processing in sections of Mining Association Rules and Selecting Rules.

W3: Although the experiment setting is listed in Appendix in section 4.1, it is still not clear.

W4: The size of data sets is much small, for example, Esophageal Cancer consists of data from 261 patients. So the experimental results are not convincing. 
clarity,_quality,_novelty_and_reproducibility: Clarity: Most parts of the paper are written in a clear way.

Quality: The quality of the paper can be further improved.

Novelty: The novelty is fair.

Originality: Good.

Reproducibility: no source codes
summary_of_the_review: Not good enough. 
Please see Strength And Weaknesses for detail.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1889/Reviewer_CHFK
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose a method for causal inference in association rule mining.
They propose to reduce confounding by selecting rules with fewer antecedents and to handle non-linear relationships between variables based on Taylor expansion.
The approach is evaluated against several simple baseline methods, predictive error and error in estimating the causal relationships, achieving better results in almost all settings.
strength_and_weaknesses: Strengths:
 - Clearly better causality estimates than the baselines
 - Validation of the selected association rules with domain experts. This is a valuable aspect of the paper and I believe it could be moved to the main part of the paper. It would also be beneficial to obtain the scores for the baselines as well

Weaknesses:
 - The paper is not clearly written:
     - The background is only briefly described, and many terms are used without prior introduction. The work could benefit from a more formal and self-contained problem setting. 
     - The proposed method is not well motivated and not described clearly. Many terms are never properly introduced (e.g., $p_{1}, p_{2}, frequent, x_{p_{1}}(0)$ etc.). Why the choice of Taylor expansion for modeling non-linearities?
 - Validity of the results is questionable:
    - Table 2 - as I see it this is not an ablation study but a (grid) hyperparameter search. What dataset did the authors use for this? Was it a different setting than the one the authors evaluated on? If not, the results of Figure 2 can hardly be objective. Was a similar search performed for the baseline methods?
    - Section 4.2.1, Feature Selection: Authors say they use 4 different methods to select the features for their model. This procedure is unclear. Are all 4 methods used simultaneously? Are the AUROC curves computed on a separate validation set? Is this feature selection method also applied for baseline methods?
    - Why is there no comparison to DWR in experiments on real-world datasets?
    - Table 1 is never referenced in text. It is also unclear what the different betas correspond to
    - Table 3 - results are barely better (often only by the third decimal point) from other methods. Also in case of identical results for two methods, authors only highlight theirs
clarity,_quality,_novelty_and_reproducibility: The paper is not clearly written - it lacks a self-contained problem setting and a clear motivation and technical description of the proposed method.

Reproducibility is questionable:
 - Details on hyperparameter selection for both the proposed methods as well as baselines are missing.
 - Feature selection, training details and other experimental settings are missing as well.
 - Algorithms in the appendix are not clearly written and contain visible errors.
summary_of_the_review: The paper suffers from poor readability, which I believe could be a smaller concern for researches well accustomed with the topic.

However, the bigger issue is with unclear experimental settings:
 - it is unclear if the authors fit the best hyperparameters for their model on the same datasets they evaluate on
 - a baseline method is excluded from one of the experiments without any justification
 - in case of tied results in Table 3 authors highlight only their own method
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1889/Reviewer_syzi
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposed a causal inference model that is capable of handling both linear and nonlinear confounding via the use of a decorrelation regularizer.  Furthermore, the method offers interpretability via the use of association rule mining. The proposed method is validated empirically on both synthetic data and real-world data.
strength_and_weaknesses: Strength:
1. the paper is concerned with an important and practical setting: causal inference with an emphasis on nonlinearity and interpretability.
2. Experiments are conducted on both synthetic data and real-world data. An ablation study is conducted. A fair number of alternative methods are compared. The proposed method appeared to perform better than the alternatives in the experiments.

Weakness:
1. it is not clear to me whether the assumptions made in the paper are too restrictive. The authors may provide justification and explanations of these assumptions. 
2. It is not clear to me how the Taylor expansion techniques can be used to estimate causal relationships.
3. There is a lack of theoretical justification for the proposed method.
clarity,_quality,_novelty_and_reproducibility: Quality: the paper is of moderate quality as the rationale of some parts of the method is not well explained. For example, while both the association mining part and the Taylor expansion part make sense, it is not clear to me how these two components are connected with each other. It is not obvious why deploying Taylor expansion for variable decorrelation can yield causal relationships. Some notation does not seem to be defined, such as $\phi$. The introduction of the hyperparameters \gamma, lambda_5, and \lambda_6 are not well explained either. The paper can benefit from providing more explanation on how the three stages work together in the procedure.

Clarity: some minor issues with clarity such as unexplained concepts and  typos:
* what is an antecedent?
* demonstrates the necessary (necessity?) of the stable prediction
* under the (delete?) another environment e_j

originality: while the proposed method makes sense, it is also somewhat non-surprising in the meantime. There is also a lack of theoretical justification for the performance of the proposed method.
summary_of_the_review: While the paper tackles an important problem in healthcare and proposes a method to do so with empirical evaluation, I have some concerns about the assumptions made by the paper, the treatment of the key components of the proposed method, as well as the theoretical rigor of the proposed method. As a result, I am leaning to reject this submission.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1889/Reviewer_GQzM
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper focuses on improving the out-of-domain generalization of the machine learning algorithm using covariate balancing methods. They develop based on the Deep Global Balancing Regression (DGBR) and Decorrelated Weighting Regression (DWR) algorithms. They argue that these algorithms are only applicable to linear and binary settings (in fact, they are wrong here, there are off-the-shelf generalizations). They propose an association rule based algorithm and an algorithm to select rules in Eq. (2). 
strength_and_weaknesses: **Weaknesses**
1. The main motivation of the paper is not correct. The authors argue that the covariate balancing algorithms are only applicable to linear and binary settings. In fact, there are off-the-shelf generalizations for higher order covariate balancing and non-binary treatments. Here is a list. As a conclusion, I do not see a clear motivation for this paper.
* [General reference for continuous treatments and higher-order balancing beyond linear correlation]  
    * Galagate, D. Causal Inference With a Continuous Treatment and Outcome: Alternative Estimators for Parametric Dose-response Functions With Applications. PhD thesis, University of Maryland, 2016.
* [Covariate balancing for continuous treatments using entropy balancing]  
    * Fong, C., Hazlett, C., Imai, K., et al. Covariate balancing propensity score for a continuous treatment: application to the efficacy of political advertisements. The Annals of Applied Statistics, 12(1):156–177, 2018. 
* [Higher-order balancing beyond linear correlation]
    * Wong, R. K. and Chan, K. C. G. Kernel-based covariate functional balancing for observational studies. Biometrika, 105(1):199–213, 2018.
    * Hazlett, C. Kernel balancing: A flexible non-parametric weighting procedure for estimating causal effects. Statistica Sinica, 2020.

2. The writing of this paper is quite unclear. For example, the authors do not describe what clusters mean in Figure 1. The authors do not clearly describe the intuition behind Eq. (2) formulation.

3. The proposed approach is only a heuristic and the authors do not provide any theoretical correctness guarantees.
3. The authors do not report confidence intervals in Table 3. Thus, we cannot assess the significance of the results.
clarity,_quality,_novelty_and_reproducibility: 1. The paper is unclear in different parts.

2. Given the issue with the missing literature on non-linear covariate balancing, I do not consider the paper to be novel.

summary_of_the_review: This paper proposes aims that relaxing the limitations of the Deep Global Balancing Regression (DGBR) and Decorrelated Weighting Regression (DWR) algorithms. However, there are off-the-shelf generalizations for higher order covariate balancing and non-binary treatments. There are other issues about the clarity of the paper too. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough