Paper: DeepROCK: Error-controlled interaction detection in deep neural networks (ICLR-2024)

Reviewer comment (Experiments):
For the real-world experiments in Fig. 3 and 4, there is no comparison with existing methods. It would be interesting to study found interactions without calibration/coupling layer.

Author response:
The reviewer suggested a comparison of DeepROCK with existing methods. As the only baseline method available to us, we employed the Benjamini–Hochberg procedure[1] to achieve false discovery rate (FDR) control. This is done by working with p-values calculated using a permutation procedure tailored for neural networks to assess the significance of interactions[2]. As shown in Figure A.4 in the paper appendix, the permutation-based p-value, coupled with the Benjamini–Hochberg procedure, fails to correctly control the false discovery rate (FDR), rendering the only available baseline method invalid. It is important to note that currently, there is no valid existing method specifically designed for error-controlled explanations in deep neural networks. Addressing this specific problem makes DeepROCK's contribution noteworthy.

The reviewer inquired about the detected interactions in the absence of calibration or a coupling layer. In response to this question, we ran an ablation study in which we modified DeepROCK to exclude either the calibration or the coupling layer. To compare the four different settings, we quantified each interaction in each setting using the averaged q-value across 20 runs. The q-value is defined as the minimum false discovery rate at which an observed score is deemed significant. We applied the modified DeepROCK to the _Drosophila_ enhancer data, for which we have a list of well-characterized interactions in early _Drosophila_ embryos, as summarized by [3], serving as the ground truth. Despite acknowledging that some of the settings fail to control false discovery rate (FDR) or underperform in detection power, we found that all four settings demonstrated proficiency in prioritizing the ground-truth interactions, as illustrated in Figure A.5 in the paper appendix. Furthermore, we observed that the removal of the coupling layer resulted in a reduction in the separation between the ground-truth interactions and others, as expected.

Reviewer comment (Theory):
As the main goal is variable selection and the stated goal is FDR control, it seems necessary that there should be a proof of FDR control. To start here, one example of a definition of an important feature is \( Y \perp X_j \mid X_{-j} \). Is there a version of this in terms of interaction? Possibly, the following  
\[
Y \perp (X_j, X_i) \mid X_{-ji}, (E[Y \mid X_j], E[Y\mid X_i])
\]

Author response:
The reviewer suggested defining an important feature, denoted as $j$, through the condition $Y \perp X_j| X_{-j}$ and inquired about the existence of an analogous definition for important interactions. First, the proposed definition appears to be problematic because, intuitively, an important feature should not be conditionally independent of the response variable $Y$. We reasoned that the reviewer's definition should be modified and applied to a set of important features, denoted as $J \subset \left \\{ 1, 2, \ldots, p \right \\}$, such that, conditioned on $J$, the complementary set of features, $J^c = \left \\{ 1, 2, \ldots, p \right \\}\backslash J$, is independent of the response variable $Y$, i.e., $Y \perp X_{J^c}| X_{J}$. We value the mathematical rigor exhibited by the reviewer and highlight the corresponding definition for significant interactions outlined in Section 2.1, under the heading "Problem Setup". Specifically, we denote the set of important interactions as $\mathcal{S}\subset J \times J$, such that, conditioned on $\mathcal{S}$ and $J$, the response $Y$ is independent of interactions in the complement $\mathcal{S}^c=\\{1,\cdots, p\\}\times \\{1,\cdots, p\\}\backslash \mathcal{S}$, i.e., $Y \perp X_{\mathcal{S}^c}| X_{J}, X_{\mathcal{S}}$. It is worth noting that we regard the features engaged in significant interactions as marginally important; however, a marginally important feature may not necessarily participate in a significant interaction.
Here is an illustrative example supporting the claim: $Y = f(X_1) + g(X_2, X_3)$.

Reviewer comment (Theory):
Without connecting such a definition to how you are using the knockoffs framework, I cannot trust a claim about FDR control. I see a few things that could help — if the knockoff swap property holds, then real-real interactions and knockoff-real interactions also should satisfy the swap property. I think this should be shown but it seems believable.

Author response:
The reviewer asked about the existence of a swap property for real-real interactions and knockoff-real interactions, analogous to the knockoff swap property. We do not believe that the swap property directly applies to the interaction setting. If such a swap property exists, then using off-the-shelf feature interaction importance measures should, in theory, adequately control the false discovery rate (FDR). However, as demonstrated in our paper, this is not the case unless we implement the calibration procedure (Section 4.1 under the heading "Performance on simulated data"). Our claim is further substantiated by the disparity in score distribution between the non-ground-truth real-real interactions and knockoff-real interactions. As shown in Figure A.1 in the paper appendix, the uncalibrated non-ground-truth real-real interactions and knockoff-real interactions exhibit a notable distributional disparity, suggesting a lack of adherence to the swap property. Moreover, Figure A.1 indicates that the calibration effectively mitigates this distribution disparity, thereby enhancing the practical utility of knockoff-real interactions as a control for estimating the false discovery rate (FDR). It is also noteworthy that, based on the cumulative distribution function, the calibration has not completely eliminated the distribution disparity between real-real interactions and knockoff-real interactions. Investigating alternative calibration strategies would be an intriguing avenue for further exploration.

Reviewer comment (Evaluation):
It has been suggested that integrated gradients do not have fidelity when it comes to explaining models. Then, what kind of conclusions can I make from scores based on them?

Author response:
The reviewer raised concerns about the reliability of using integrated gradient-based interaction importance for model explanations and subsequently questioned the validity of the conclusions we drew. First, it is essential to note that we employ expected gradients[1], a state-of-the-art method refined from integrated gradients.  Importantly, expected gradients is guaranteed to adhere to the interpretability axioms. Second, we concur with the reviewer that relying on a single method is not advisable, because no method is flawless. We therefore propose here an alternative, instance-based importance measure by substituting expected gradients with SmoothGrad[2] another state-of-the-art and commonly used method. As shown in Figure A.2 in the paper appendix, the interaction importance derived from SmoothGrad produces results similar to those obtained from the expected gradient-based method. These findings, coupled with the results from model-based interaction importance, not only reinforce the conclusions drawn from the expected gradient-based method but also underscore the robustness of DeepROCK.

Reviewer comment (Theory):
The model-dependent score seems to be architecture specific. Are there concerns about multiplying weight matrices across layers in, for example, deep residual networks?

Author response:
The reviewer raised a question about whether the model-based importance is applicable only to MLPs or if it extends to non-MLP architectures such as deep residual networks. While we acknowledge that the model-based importance is specific to MLPs, we want to emphasize that our intention was not to address all architectures using this method. Instead, we introduced the model-based importance as an example to showcase the robustness of DeepROCK across various types of importance scores, similar to the previously mentioned SmoothGrad-based interaction importance. From a practitioner's perspective, we recommend instance-based importance measures because they are model-agnostic.

Reviewer comment (Theory):
Is there something formal to understand this better — "Intuitively, the interaction between two marginally important features naturally has a higher importance score than random interactions, even though they are all false"?

Author response:
The reviewer inquired about the formal understanding of the assertion that the interaction between two marginally important features naturally yields a higher importance score than random interactions even though they have no interaction. First, we began with the uncalibrated interaction scores presented in Figure A.1 in the paper appendix and organized the interactions into distinct groups. To be specific, utilizing the ground truth labels, we partitioned the original features into two groups (marginal and irrelevant) based on their marginally important status, resulting in a total of seven interaction groups, as depicted in Figure A.3 in the paper appendix. From the figure, we observe that the uncalibrated and non-ground-truth interactions between two marginally important features are substantially higher than other non-ground-truth interactions, which could potentially pose challenges for FDR estimation. Second, we offer an explanation for the biased importance of interactions between two marginally important features in the setting of model-based importance. As depicted in Section 3.2 of the paper, under the heading "Feature interaction importance", the model-based interaction importance is determined by two factors:
(1) the relative importance between the original feature and its knockoff counterpart, and
(2) the relative importance among all features.
These two factors are inherently large for marginally important features by definition, even if they do not participate in any interactions.

Reviewer comment (Experiments):
The generality of the method across different DNN architectures in not developed. In fact, the method only seems to work with MLPs.

Author response:
1. The reviewer expressed concerns regarding the generalizability of DeepROCK to DNN architectures other than MLP. It is crucial to clarify that DeepROCK is designed to be applicable to any off-the-shelf DNN architecture beyond MLP. We recommend using DeepROCK with instance-based importance measures because they are model-agnostic. While we acknowledge that model-based importance is specific to MLPs, it’s important to note that our intention was not to address all architectures using this particular method. Instead, we introduced model-based importance as an illustrative example to demonstrate the robustness of DeepROCK across various types of importance scores.

Reviewer comment (Theory):
The method seems somewhat heuristic. As pointed out by reviewer 1, the sentence 'Intuitively, the interaction between two marginally important features naturally has a higher importance score than random interactions' is used to motivate the calibration in section 3.2, but is not very well formalized.

Author response:
2. The reviewer raised concerns about the motivation behind feature interaction importance calibration and sought a formal understanding of the claim that the interaction between two marginally important features inherently results in a higher importance score than random interactions, even in the absence of any genuine interaction. First, we began with the uncalibrated interaction scores in Simulation Function 1 and organized the interactions into distinct groups. To be specific, utilizing the ground truth labels, we partitioned the original features into two groups (marginal and irrelevant) based on their marginally important status, resulting in a total of seven interaction groups, as depicted in Figure A.1 in the paper appendix. From the figure, we observe that the uncalibrated and non-ground-truth interactions between two marginally important features are substantially higher than other non-ground-truth interactions, which could potentially pose challenges for FDR estimation. Second, we offer an explanation for the biased importance of interactions between two marginally important features in the setting of model-based importance. As depicted in Section 3.2 of the paper, under the heading “Feature interaction importance”, the model-based interaction importance is determined by two factors: (1) the relative importance between the original feature and its knockoff counterpart, and (2) the relative importance among all features. These two factors are inherently large for marginally important features by definition, even if they do not participate in any interactions.

Reviewer comment (Theory):
As I understand it, there are no statistical guarantees due to the use of function approximation in the KnockoffGAN and MLP.

Author response:
4. The reviewer expressed concerns about the absence of statistical guarantees arising from the use of function approximation in both KnockoffGAN and MLP. First, it is crucial to clarify that both KnockoffGAN [1][2][3] and MLP [4] have established a track record of controlling the FDR with statistical guarantees, even in the presence of function approximation. Moreover, the selection of KnockoffGAN and MLP is further substantiated by the comprehensive simulation studies presented in the paper (Section 4.1, under the heading “Performance on simulated data”). Finally, it’s crucial to acknowledge that there is currently no established method specifically designed for error-controlled explanations in deep neural networks. Therefore, any effective methods in this realm would be highly significant. We believe that DeepROCK represents just the beginning of a more extensive effort to further develop and enhance this concept with a better theoretical understanding.

__References__

[1] J. Jordon, J. Yoon, and M. van der Schaar. KnockoffGAN: Generating knockoffs for feature selection using generative adversarial networks. In International Conference on Learning Representations, 2018.

[2] Y. Romano, M. Sesia, and E. Candès. Deep knockoffs. Journal of the American Statistical Association, 115(532):1861–1872, 2020.

[3] M. Sudarshan, W. Tansey, and R. Ranganath. Deep direct likelihood knockoffs. In Advances in Neural Information Processing Systems, volume 33, pages 5036–5046, 2020.

[4] Y. Y. Lu, Y. Fan, J. Lv, and W. S. Noble. DeepPINK: reproducible feature selection in deep neural networks. In Advances in Neural Information Processing Systems, 2018.