Paper:
DeepPINK: reproducible feature selection in deep neural networks
Deep learning has become increasingly popular in both supervised and unsupervised machine learning thanks to its outstanding empirical performance. However, because of their intrinsic complexity, most deep learning methods are largely treated as black box tools with little interpretability. Even though recent attempts have been made to facilitate the interpretability of deep neural networks (DNNs), existing methods are susceptible to noise and lack of robustness. Therefore, scientists are justifiably cautious about the reproducibility of the discoveries, which is often related to the interpretability of the underlying statistical models. In this paper, we describe a method to increase the interpretability and reproducibility of DNNs by incorporating the idea of feature selection with controlled error rate. By designing a new DNN architecture and integrating it with the recently proposed knockoffs framework, we perform feature selection with a controlled error rate, while maintaining high power. This new method, DeepPINK (Deep feature selection using Paired-Input Nonlinear Knockoffs), is applied to both simulated and real data sets to demonstrate its empirical utility.

Review:
The paper proposes a method for feature selection in neural networks using a method for a controlled error rate, quantified through the False Discovery Rate. To control FDR the paper is using the model-X knockoffs framework [2, 3, 10]: construct random features that obey distributional properties with respect to the true features, and extract statistics (filters) of pairwise importance measures between true-knockoff dimensions. The choice of the importance function and the knockoff filters is flexible. The novelty of this paper lies in using a neural network (MLP) to get the importance measure through a linear layer that couples the true and knockoff features pairwise. The final statistic depends both on the trainable linear layer weights and the rest of the network weights.   Results are provided on real and synthetic data: for the latter (noisy linear and nonlinear processes with increasing number of dimensions/features), the proposed method performs better (in terms of FDR control and power or feature relevance certainty) compared to MLP, RF and SVMs. Notably, compared to MLP the difference comes from the coupling initial, linear layer. The paper also compares feature selection to other knockoffs-related methods ([2], [4], [10]), on a real dataset for mining interpretable dimensions features (HIV-1) and provide qualitative analysis on the discovered features from gut microbiome data.    General comments: Feature selection is an important topic related to interpretability, transparency and robustness in neural network models (and machine learning more general) and the paper is an effort in that direction. The framework is an extension of the model-X knockoff framework [10] and the novelty lies in adding a hard-wired, linear, pairwise-connected layer on the knockoff features before an MLP. The benefit is higher statistical power for feature relevance. I like the idea of coupling feature weights and algorithm weights in a single importance measure, which ends up ranking a feature j high if both the filter coefficients difference and the NN output dimension j is high. The paper has a clear exposure of ideas and background (e.g. on the knockoff statistics models provided) and the paper is easy to follow from motivation, to background to empirical evaluation. My concerns with the paper are mainly around: a) the amount of novelty (and generality) on the proposed coupling/pairwise layer and b) the broader applicability of the proposed method.   Strengths: - The framework is an application/extension of an existing statistical framework for error control with theoretical guarantees [10]. - The coupling pairs + parametrization idea is generic and could be applied to different types of algorithms (as a learned representation layer).  - The paper is well written, reads smoothly and the ideas are well exposed (supported, explained or cited).  Weaknesses:  - It would help to have some element of theoretical analysis on the chosen definition of the importance measures Z (Sec. 3). For example, deriving this as an explicit regularizer on the layer weights. - The framework (proposed as a general feature selection method?), is restricted to Gaussian design and the use of an estimated precision matrix, for which there might not be any robustness guarantees. Some discussion (or paths for alternative designs?) might be helpful.  - There is some disconnect, in my opinion, between motivating this work as a feature selection method for NN and using a NN as a way to measure importance (given a prediction task). - The empirical validation, even though convincing in some respect: the benefit of the novelty here (i.e. adding a pairwise connected layer vs. having a vanilla MLP) is only explored on the synthetic data.   Specific comments: - The method is motivated by the need for interoperability in NN and error control in NN, though only the latter is explored (in a synthetic scenario).  - Similarly, the title states “reproducible” feature selection and this might something that needs further explaining. In what sense reproducible?  - Did the authors explore/consider different definitions of the importance measures  on the knockoff statistics?  - What is the dependency of some aspects of the performance on the knock-off feature generation (e.g., s for the Gaussian case)? Is there a way to make vanilla MLP + knockoff features better by changing (or tuning) s? - Is there a specific rationale for fixing the number of neurons in the hidden layer to be p (equal to the input dimension)? As long as the output layer is p-dimensional the reasoning should still hold with a different ‘factorization’ of the input to output mapping that defines w. The same holds for using L1 regularization (and with a specific regularization parameter). - How dependent on the specific parametrization of the MLP (or NN more general) is the performance?  - The FDR target error level is fixed/set to q=0.2 everywhere. Would it be insightful to vary this and re-interpret, e.g. the results of Sec. 5.1? - It would be helpful to see some prediction results with real data, based on the selected features. E.g. a two-pass process where the NN is re-trained on the selected, error-control features to predict the target function values or class.   - How much False Discovery errors are relevant for NN performance in practical applications (and in high-dimensions)? An open question that might be worth discussing. - Did the authors try the knockoffs+vanilla MLP approach on the real data in 5.1 (or the qualitative analysis in 5.2)? The pairwise-connected layer is the novelty of this paper and this should be further highlighted.   =========================  Comments after author feedback:  Thank you for the replies to all reviews and most of the comments. I agree with the points of your clarifications and can see the value of the points made in the response. I also agree with Reviewer 1 that this can inspire future work by the authors and others. I am changing my score to a 6 to reflect these. At the same time my main concern on general applicability remains, and in some cases has been strengthened by some of the responses.   - “in the sense that small changes in the data can yield dramatic changes in the selected features. “ If that is the case, perturbations in the data (and thus reproducibility of the framework) are not explored. What type of changes?   - “We consider the primary virtue of the robustness to be improved reproducibility of the selected features;” As above. I don’t feel that this aspect of the authors’ work is highlighted or supported experimentally in the paper.   - “Given the universal approximation property of DNNs, practitioners can now avoid having to handcraft feature importance measures with the help of DeepPINK.” This is a good point, thank you for making this connection explicit. Is the addition of DeepPINK layer however ‘altering’ the network definition, i.e. are you selecting features for the original network or the modified one? How dependent is this selection on initialization, parametrization, and as the authors state, changes in the input data (e.g. transformations, subsampling, noise etc).  - “using interpretability alone as a measurement is far less reliable than using FDR and power with synthetic data as measurements.”  This is also a good point, and provides statistical guarantees (as Reviewer 1 also points out).   - “it is the fundamental architecture (instead of the tuning) of vanilla MLP that prevents the knockoff filter from performing well.” I am not sure what this claim implies. That MLP is not a good choice for being subjected to knockoff filters? This defies the purpose of the paper. Or that knockoffs can only add value coupled in the learning process (like for DeepPINK)? I wonder, and I made the same point in my initial review on which the authors replied, how much those findings are generally applicable to any ‘neural network’ parametrization or just MLP-type. How about basic linear or (shallow) non-linear regression with the PINK coupling and solved through SGD?   - “Since there is no existing method available for robust feature selection in DNNs, any effective methods would be significant.” I agree. However the idea of input features and feature representation (hidden layers) is coupled in NNs. As the authors point out, interpretability and selection is conditioned on the prediction algorithm. So this is not ‘feature selection’, but input dimension selection. One could envision however a PINK layer attached on higher hidden layers for representation dimension selection, interpretability and transferability.   - “The proposed network architecture may seem simple, but our journey to finding it was nontrivial.” So then does the paper involve a single, fixed architecture? Or is it generally applicable for any parametrization that one can add on top of the pairwise linear layer?