F EATURE S ELECTION USING S TOCHASTIC G ATES
Anonymous authors
Paper under double-blind review
A BSTRACT
Feature selection problems have been extensively studied in the setting of linear estimation, for instance LASSO, but less emphasis has been placed on feature selection for non-linear functions.
In this study, we propose a method for feature selection in non-linear function estimation problems.
The new procedure is based on directly penalizing the (cid:96) 0 norm of features, or the count of the number of selected features.
Our (cid:96)
based regularization relies on a continuous relaxation of the Bernoulli distribution, which allows our model to learn the parameters of the approximate Bernoulli distributions via gradient descent.
The proposed framework simultaneously learns a non-linear regression or classiﬁcation function while selecting a small subset of features.
We provide an information-theoretic justiﬁcation for incorporating Bernoulli distribution into our approach.
Furthermore, we evaluate our method using synthetic and real-life data and demonstrate that our approach outperforms other embedded methods in terms of predictive performance and feature selection.
I NTRODUCTION
Feature selection is a fundamental task in machine learning and statistics.
Feature selection leads to a number of potential beneﬁts: reducing experimental costs Min et al.
(2014), enhancing interpretability (Ribeiro et al., 2016), computational speed up and even improving model generalization on unseen data (Chandrashekar & Sahin, 2014). In biomedicine, scientists collect multitude datasets comprising of many biomarkers (e.g., genes or proteins) that require development of effective diagnostics or prognostics models.
For instance, in Genome wide association studies (GWAS), feature selection can help identify such models and lead to improved risk assessment and reduced cost.
Feature selection methods may be classiﬁed into three major categories: ﬁlter methods, wrapper methods, and embedded methods.
Filter methods attempt to remove irrelevant features prior to learning a model.
These methods ﬁlter features using a per-feature relevance score that is created based on some statistical measure (Battiti, 1994; Peng et al., 2005; Estévez et al., 2009; Song et al., 2007; 2012;
Chen et al., 2017).
Wrapper methods (Kohavi & John, 1997b; Stein et al., 2005; Zhu et al., 2007; Reunanen, 2003; Allen, 2013) use the outcome of a classiﬁer to determine the relevance of each feature, which requires recomputing the classiﬁer for each subset of features.
This becomes computationally expensive for neural network based wrapper methods (Verikas & Bacauskiene, 2002;
Kabir et al., 2010; Roy et al., 2015).
Embedded methods aim to remove this burden by learning the model while simultaneously selecting the subset of relevant features.
The Least Absolute Shrinkage and Selection Operator (LASSO) (Tibshirani, 1996) is a well-known embedded method, whose objective is to minimize the loss while enforcing an (cid:96) 1 constraint on the weights of the features.
Although LASSO is scalable and widely used (Hans, 2009; Li et al., 2011; 2006), it is restricted to the domain of linear functions.
To allow the model to capture nonlinear interaction, it is appealing to consider the non-convex extension of the LASSO formulation using neural networks.
We develop a fully embedded feature selection method for nonlinear models. To the best of our knowledge it is the ﬁrst (cid:96) 0 embedded feature selection method. Our method improves upon the
LASSO formulation in two aspects: a) it captures nonlinear interactions between the features via neural network modeling and {+b) it employs an (cid:96) 
 -like regularization+} using gates whose weights are parametrized by a smooth variant of a Bernoulli distribution. Altogether these twofold improvements are formulated as a fully differentiable neural network.
Speciﬁcally, our contributions are as follows:
By utilizing a recent development of continuous and differentiable approximation to discrete distributions (Maddison et al., 2016), (Jang et al., 2017), (Louizos et al., 2017), we introduce a solution to the long standing problem of feature selection with an (cid:96) 0 regularization.
We present a novel simple relaxation of Bernoulli distribution to sparsify the input layer (the feature space) which we call stochastic gate (STG) and show its advantage over the distribution presented by (Louizos et al., 2017) both in performance and convergence time.
By applying these two relaxations to an input layer of a neural network, we perform embedded feature selection in classiﬁcation, regression or survival analysis tasks and demonstrate its capabilities on artiﬁcial and real data sets.
We justify our probabilistic approach by analyzing the constrained Mutual Information maximization objective of feature selection.
We demonstrate the applicability of our method using numerous examples (see Section 6 and Appendix).
Notation: We refer to vectors as bold lowercase x and random vectors as bold uppercase letters
X .
Scalars are non-bold case y , while random variables are capital case Y .
A set is represented by script fonts X , Y , S .
For example the n th vector-valued observation is denoted as x n whereas X d represents the d th feature of the vector-valued random variable X .
Let [ n ] = 1 , 2 , . . . , n .
For a set
S ⊂
[ D ] let the vector s ∈ { 0 , 1 } D be the characteristic function for the set.
That is s i
=
if i ∈ S and 0 otherwise.
For two vectors x and z we denote x (cid:12) z to be the element-wise product between x and z .
Thus, if we let s ∈ { 0 , 1 } D be the characteristic vector of S , then we may deﬁne x S = x (cid:12) s .
The (cid:96) 1 norm of a vector is denoted as (cid:107) x (cid:107) 1 = (cid:80) Di =1 | x i | .
Finally, the (cid:96) 0 norm of a vector is denoted as (cid:107) x (cid:107) 0 and counts the total number of non-zero entries in the vector x .
P ROBLEM S ETUP AND B ACKGROUND
Let X ∈ R D be the input domain with corresponding response domain Y .
Given realizations from some unknown data distribution P X,Y , the goal of embedded feature selection methods is to simultaneously ﬁnd a subset of indices S ⊂ { 1 , ...D } and construct a model that predicts Y based on the selected features X S .
2. R ISK MINIMIZATION OBJECTIVE
We assume that we are given a family of functions F such that any function f θ ∈ F is indexed by a set of parameters θ .
Given some loss L , a selection of features S ⊂
[ D ] , and a choice of parameters θ , we denote the risk of our model as
[Equation]
where we recall that s = { 0 , 1 } D is a vector of indicator variables for the set S and (cid:12) denotes the point-wise product.
Thus, the goal of the feature selection problem is to ﬁnd the parameters θ and s that minimize R ( θ , s ) such that (cid:107) s (cid:107) 0 is small compared to D .
2. F EATURE S ELECTION FOR L INEAR M ODELS
Before proceeding with our proposed method, we review the feature selection problem in the linear regression setting with the least squares loss.
Thus, we restrict F to be the space of linear functions and the loss function to be the quadratic loss.
Given observations { x n , y n } Nn =1 we may consider the constrained empirical risk minimization problem
[Equation]
Since the above problem is intractable, a number of authors replace the (cid:96) 0 constraint with a surrogate function Ω( θ ) : R D
→ R + designed to penalize the number of selected features in θ .
A natural choice for Ω is the (cid:96) 1 norm, which yields a convex problem and more precisely the LASSO optimization problem. The (cid:96) 1 is known to be the closest convex relaxation to the (cid:96) 0 .
{+In fact, in certain settings the LASSO optimization and the (cid:96) 
 based objective+} have the same solution.
While the original LASSO problem focuses on the constrained optimization problem, the regularized least squares problem, which is often used in practice, yields the following minimization objective:
[Equation]
The hyperparameter λ trades off the amount of regularization versus the ﬁt of the objective 1 .
The (cid:96)regularized method is very effective for feature selection and prediction; however, it achieves this through shrinkage of the coefﬁcients.
As a result, (Fan & Li, 2001) have considered non-convex choices for Ω that perform well both theoretically and empirically for prediction and feature selection.
Our goal is to apply such regularization techniques to perform feature selection while learning a non-linear function.
Kernel methods have been considered (Yamada et al., 2014), but scale quadratically in the number of observations.
To alleviate this burden, (Gregorová et al., 2018) use random Fourier features to approximate the kernel.
(Li et al., 2016) and (Scardapane et al., 2017) take an alternative approach by modeling f θ using a neural network with (cid:96) 1 regularization on the input weights.
However, in practice, introducing an (cid:96) 1 penalty into gradient descent does not provide sufﬁcient sparsiﬁcation.
Below, we discuss our method that works to directly use an (cid:96) 0 penalty.
P ROPOSED M ETHOD
We take a probabilistic approach to approximate the (cid:96) 0 norm, which can extend to non-linear models while remaining computationally efﬁcient.
To motivate such probabilistic approach, we provide theoretical support (see Section 4) based on a Mutual Information perspective of the feature selection problem.
To view the (cid:96) 0 regularized version of the risk (Eq. 1) from a probabilistic perspective, one can introduce a Bernoulli random vector ˜ S whose entries are independent and the d th entry satisﬁes π d = P
( ˜ S d = 1) for d ∈ [ D ] .
If we denote the empirical expectation over our observations as ˆ E X,Y , then, the empirical regularized risk (Eq. 1) becomes
[Equation]
where we have E ˜ S
|| ˜ S
||
= (cid:80) Dd =1 π d
and we constrain π d ∈ { 0 , 1 } .
Clearly, this formulation is equivalent to Eq.
1, with a regularized penalty on cardinality rather than an explicit constraint.
We may then relax the discrete constraint on π d to be π d ∈ [0 , 1] .
Now, our goal is to ﬁnd the model parameters θ ∗ and Bernoulli parameters π ∗ that minimize the empirical risk ˆ R ( θ , π ) via gradient descent.
However, an optimization of a loss function which includes discrete random variables suffers from high variance (see F in the Appendix for more details).
Therefore, inspired by a recently developed continuous approximation for discrete random variables, suggested by (Jang et al., 2017; Maddison et al., 2016), we develop and use a novel and simple continuous distribution that is fully differentiable and suited for the task of feature selection.
3. C ONTINUOUS R ELAXATION
Our continuous relaxation for the Bernoulli variables ˜ S d for d ∈
[ D ] is termed stochastic gate (STG).
The STG relies on the reparametrization trick, which is widely used for reducing the variance of gradient estimators (Miller et al., 2017;
Figurnov et al., 2018).
To construct a continuous approximation to Bernoulli random variable via the reparametrization trick, we deﬁne z
d = g ( µ d + (cid:15) d ) = max(0 , min(1 , (cid:15) d + µ d + 0 . 5))
where (cid:15) d is drawn from a Gaussian distribution N (0 , σ d ) , where σ d is ﬁxed throughout training.
This approximation can be viewed as a clipped, mean-shifted, Gaussian random vector.
Furthermore, the gradient of the objective with respect to µ d can be computed via the chain rule.
We can now rewrite the objective in Eq.
4 as
[Equation]
where Z is a random vector with D independent variables z
d for [ D ] .
To optimize the empirical surrogate of the objective (Eq. 5), we ﬁrst differentiate it with respect to µ .
Then, Monte Carlo sampling leads us to the following gradient estimator
[Equation]
where K is the number of Monte Carlo samples.
Thus, we can update the parameters µ d for [ D ] via gradient descent.
We note that if we replace ∂z kd ∂µ d with 1, the above gradient estimator for L (cid:48) is reduced to the Straight-Through estimator (Bengio et al., 2013).
Under the continuous relaxation, the expected regularization term in the objective ˆ R ( θ , µ ) is simply the sum of the probability that the gates { z d } Dd =1 are active, which is equal to (cid:80) Dd =1 Φ (cid:16) µ d
+ 12 σ d (cid:17) , where Φ is the standard Gaussian CDF.
To conclude, we can now optimize the objective in Eq.using gradient descent over the model parameters θ and the parameters µ representing the Gaussian
’s mean (instead of the Bernoulli parameters π ).
After training, to remove the stochasticity from the learned gates, we set ˆ z d = max(0 , min(1 , µ d + 0 . 5)) , which informs what features are selected.
Note that when | µ d | is less than 1 2 , ˆ z d returns the value between (0 , 1) .
In such a case, we can treat the value of ˆ z d as feature importance or employ an additional thresholding (i.e.
if ˆ z d > 0 .
5 and 0 otherwise)
depending on application-speciﬁc needs.
In the Appendix, we provide the pseudo-code of our algorithm as well as the discussion of the choice of σ d .
C ONNECTION TO M UTUAL I NFORMATION
In this section we show an equivalence between the Bernoulli formulation of the feature selection problem and the (cid:96) 0 regularized approach.
4. M UTUAL I NFORMATION BASED OBJECTIVE
From an information theoretic perspective, the goal of feature selection is to ﬁnd a subset of features
S that has the highest Mutual Information (MI) with the target variable Y .
Recall that the MI between two random variables can be deﬁned as I ( X ; Y )
=
H ( Y ) − H
( Y | X ) where H ( Y ) ,
H ( Y | X ) are the entropy of p Y ( Y ) and the conditional entropy of p Y | X ( Y | X ) , respectively (Cover & Thomas, 2006).
Then we can formulate the task as selecting S such that the mutual information between X S and Y are maximized:
[Equation]
where k is the hypothesised number of relevant features.
4. I NTRODUCING RANDOMNESS
We ﬁrst demonstrate that under mild assumptions we can replace the deterministic search over the set S (or corresponding indicator vector s ), by a search over the parameters of the distributions that model s .
Our proposition is based on the following two assumptions:
Assumption 1: There exists a subset of indices S ∗ with cardinality equal to k such that for any i ∈ S ∗ we have I ( X i ; Y | X \{ i } ) > 0 .
Assumption 2: I ( X S ∗ c ; Y | X S ∗ )
= 0 .
Discussion of assumptions: Assumption 1 that including an element from S ∗ improves prediction accuracy.
This assumption is equivalent to stating that feature i is strongly relevant (Kohavi & John, 1997a; Brown et al., 2012).
Assumption 2 simply states that S ∗ is a superset of the Markov Blanket of the variable Y (Brown et al., 2012).
The assumptions are quite benign.
For instance they are satisﬁed if X is drawn from a Gaussian with a non-degenerate covariance matrix and Y =
f
( X S ∗ ) + w , where w is noise independent of X and f is not degenerate.
With these assumptions in hand, we may present our result.
Proposition 1.
Suppose that the above assumptions hold for the model.
Then, solving the optimization in Eq. 6 is equivalent to solving the optimization
[Equation]
where the coordinates ˜ S i are drawn independently at random according to a Bernoulli distribution with parameter π i .
Due to length constraints, we leave the proof of this proposition and how it bridges the MI maximization (Eq. 6) and risk minimization (Eq. 2) to the Appendix (see Section C and D).
R ELATED W ORK
The two most related works to this study are (Louizos et al., 2017) and (Chen et al., 2018).
In (Louizos et al., 2017), they introduce the Hard-Concrete (HC) distribution as a continuous surrogate for Bernoulli distributions in the context of model compression.
The HC distribution is induced by applying a hard sigmoid z = min(1 , max(0 , ¯ s )) to ¯ s, where we construct ¯ s by applying Sigmoid function to the logistic distribution as follows:
[Equation]
The interval ( τ, ζ ) , with τ < 0 and ζ > 1 , allows the distribution to have more probability mass on the edge of the support.
The authors demonstrate how incorporating the HC in deep neural networks leads to fast convergence and improved generalization due to the sparsiﬁcation effect.
In this study, we introduce this type of sparsiﬁcation for the long standing problem of feature selection with (cid:96)regularization.
We demonstrate that a simple relaxation of Bernoulli distributions is sufﬁcient and works better than the HC distribution for feature selection.
We ﬁnd empirically that the HC yields high variance gradient estimates compared to the STG.
This high-variance does not affect model sparsiﬁcation as the sparsity pattern does not matter.
Contrarily, for feature selection it is crucial as practitioners require a stable procedure.
Furthermore, higher variance gradients will result in a slower convergence of the model (this has been demonstrated empirically in the supp.
material).
In (Chen et al., 2018), the Gumbel-softmax trick is used to develop a framework for interpreting pre-trained models.
Their method is focused on ﬁnding a subset of features given a particular instance, and therefore is not appropriate for general feature selection.
Some studies tackle embedded feature selection problems by extending LASSO and group LASSO to neural network models.
The authors (Li et al., 2016; Scardapane et al., 2017) and (Feng &
Simon, 2017) have a similar goal as ours in performing feature selection, but instead rely on the (cid:96)relaxation to the (cid:96) 0 .
Our approach, which utilizes stochastic gates coupled with the (cid:96) 0 norm achieves substantially better empirical performance compared against other (cid:96) 1 based baselines.
This point is demonstrated in the next section.
E XPERIMENTS
Here we provide empirical evaluation of our method in a wide range of settings.
We begin with simple linear regression evaluating the capabilities of our approach to select a sparse set of relevant features (section 6.1). We then evaluate our method in nonlinear regression tasks (section 6.2). Next, we demonstrate the applicability of the method in a highly nonlinear classiﬁcation task (section 6.3. Finally, we present two potential applications (section 7).
We implement our method using both the STG and HC distributions and compare their applicability for feature selection.
The hyperparamters of all the methods are optimized using validation sets via Optuna (Takuya Akiba & Koyama, 2019).
In the Appendix, we provide all details of the Optuna based parameter tuning procedure. Additional experiments also appear in the Appendix, including high dimensional datasets and an extensive comparisons between the STG and HC (Louizos et al., 2017) distributions.
6. S UPPORT R ECOVERY IN L INEAR R EGRESSION
In the setting of noisy linear regression, (Wainwright, 2009) have analyzed the probability of LASSO to correctly identify a sparse subset of active variables. The problem is know as support recovery and is formulated as follows; let β ∗ ∈
R D be a ﬁxed sparse vector, such that β ∗ i ∈ {− 0 .
5 } (with equal probability) if i ∈ S , and β ∗ i =
otherwise.
Suppose the cardinality of the support |S| = k is known.
Given a matrix of measurements X ∈ R N × D with values drawn independently from N (0 , 1) , the response y is deﬁned as
[Equation]
(8) where the values of the noise w i , i = 1 , ..., N are drawn independently from N (0 , 0 . 5) .
Here, we reproduce this setting to evaluate the probability of the proposed approach to perfectly recover the support of β ∗ .
As suggested by (Wainwright, 2009), we use a sparsity that scales with D such that k = (cid:100) 0 .
4 D 0 .
75 (cid:101) .
For each number of samples N in the range [10 , 500] , we runsimulations and count the portion of correctly recovered supports.
We repeat this process fordifferent values of D and compare our performance to LASSO.
For LASSO, the regularization parameter was set to its optimal value α N = (cid:113) [Equation]
2 log
[Equation]
N (Wainwright, 2009).
For STG and
HC we set λ N = Cα N , such that C is a constant, which is selected using a grid search in the range [0.1,10].
As evident from Fig.
1, even when restricting to linear functions our method has a clear advantage over LASSO.
This implies that the (cid:96) 0 based penalty, although is non convex in nature, allows perfect recovery of the support using less samples.
Furthermore, the proposed STG requires even less samples than the HC distribution and suffers from a smaller variance.
[Figure]
6. N ONLINEAR R EGRESSION USING SYNTHETIC AND REAL DATASETS
In this section, we evaluate our method for regression tasks against two other embedded feature selection methods: LASSO and Sparse Random Fourier Features (Gregorová et al., 2018).
Following the same format as (Gregorová et al., 2018), the following functions are used to generate synthetic data: (SE1: 100/5)
y
= sin ( x 1 + x 3 ) 2 sin ( x 7 x 8 x 9 ) + N (0 , 0 . 1) .
(SE2: 18/5)
y
= log(( (cid:80) 15 s =11 x s ) 2 )
+
N (0 , 0 . 1) .
(SE3: 1000/10)
y =
10( z 21 + z 23 )
e − 2( z 21 + z 23 ) + N (0 , 0 . 01) , where each x i is drawn from the standard Gaussian N (0 , 1) .
For (SE3), the ﬁve consecutive coordinates are generated by x 5( j − 1)+ i =
z
j + N (0 , 0 . 1) , where z j ∼ N (0 , 1) for i = 1 , ..., 5 and j = 1 , ..., 200 .
The numbers next to the experiment code indicate total dimensions/relevant dimensions in the feature space.
We also evaluate our method using three real datasets: (RCP: 21/-), of computer systems activity, (REL 17/-) of F16 elevators and (RAI 39/-) of F16 ailernos all taken from the LIACC repository 2 .
For each dataset, we generate 30 different replications and randomly split the data into train, validation, and test set (see Appendix for more details).
The root mean squared error on the test set averaged overrandom replicated datasets are reported in Table 1.
Our method outperforms all alternative methods for most cases.
Note that (SE1) is generated using a sine function, which is in favor of the random Fourier feature based method (SRFF). [Table]
6. N OISY B INARY XOR C LASSIFICATION
In the following evaluation, we consider the problem of learning a binary XOR function for classiﬁcation task.
The ﬁrst two coordinates x 1 , x 2 are drawn from a binary "fair" Bernoulli distribution.
The response variable is set as an XOR of the ﬁrst coordinates, such that y = x 1 ⊕ x 2 . The coordinates x i , i = 3 , ..., D are nuisance features, also drawn from a binary "fair" Bernoulli distribution. The number of points we generate is N = 1 , 500 , of which 70 % are reserved for test and 10% of the remaining training set was reserved for validation. We compare the proposed method to four embedded feature selection methods (LASSO (Tibshirani, 1996), C-support vectors (SVC) (Chang & Lin, 2011), deep feature selection (DFS) (Li et al., 2016), sparse group regularized NN (SG-L1-NN)
(Scardapane et al., 2017)).
To provide more benchmarks, we also compare our embedded method against three wrapper methods (Extremely Randomized Trees (Tree) (Rastogi & Shim, 2000), Random Forests (RF) (Strobl et al., 2008)) and Extreme Gradient Boosting (XGBOOST) (Chen & Guestrin, 2016).
To evaluate the feature selection performance, we calculate the Informative Features Weight Ratio (IFWR).
IFWR is deﬁned as the sum of weights W d over the informative features divided by the sum over all weights.
In the case of binary weights the IFWR is in fact a recall measure for the relevant features (See the Appendix for more details.)
The experiment is repeated 20 times for different values of D , and the average test classiﬁcation accuracy and standard deviation are presented in Fig. 2(a), followed by the IFWR in Fig. 2(b). The number of selected features affects the accuracy. Therefore, to treat all the methods in a fair manner, we tune the hyperparameter that controls the sparsity level using Optuna (Takuya Akiba & Koyama, 2019) which optimizes the overall accuracy across different D s. For instance, the wrapper methods (Tree, RF and XGBOOST) has a threshold value to retain features.
We retrain them using only such features whose weight is higher than the threshold. In terms of feature ranking (see Fig. 2(c), only the tree based methods and the proposed (STG and HC based) provide the optimal median rank (which is 1 . 5 ) for the two relevant features. Nonetheless, the ranking provided by STG is the most stable comparing to all the alternative methods. [Figure]
In (Zheng et al., 2017), the authors have subjected more than 90 , 000 puriﬁed cell populations of peripheral blood monocytes (PBMCs) to Single-cell RNA sequencing (scRNA-seq) analysis.
Here
[Table]
we use this data and focus on classifying two subpopulations of T-cells, namely Naive and regulatory
T-cells. We use the proposed method to select a subset of genes for which the network discriminates between Naive and regulatory T-cells.
We ﬁrst ﬁlter out the genes that are lowly expressed in the cells, which leaves us with D = 2538 genes (features).
The total number of cells in these two classes is N = 20742 , of which we only use 10% of the data for training.
We apply the proposed method for different values of λ and report the number of selected features and classiﬁcation accuracy on the test set.
Here we compare our performance (STG and HC) to RF and LASSO.
A least squares polynomial ﬁt plot of the accuracy vs. number of selected features is presented in Fig.
The accuracy obtained by a NN without feature selection is 91 .
09% , which is comparable to what we achieve with a small fraction of the features.
We have also evaluated the performance of the Hard-Concrete applied to all layers (HC-Full), following the procedure in (Louizos et al., 2017).
Empirically we observed that using this type of regularization across all layers provides inferior capabilities in terms of feature selection.
Moreover, when the HC-Full converges to a larger subset of features ( d > 50) , it does not generalize at all and the test accuracy is around 0 .
C OX P ROPORTIONAL H AZARD M ODELS FOR S URVIVAL A NALYSIS
We incorporate our STG into DeepSurv (Katzman et al., 2018) to examine how this procedure improves survival analysis.
The input to this analysis are gene expression proﬁles (Curtis et al., 2012) along with additional commonly used clinical variables, the goal is to predict survival time. In this setting, identifying a small subset of predictive covariates is important as it may lead to cheaper and more stable medical assays.
See the Appendix for additional description of the data and the experimental setup.
We compared our method (Cox-STG and COX-HC) against three other methods: Cox model with (cid:96)regularization (Cox-LASSO), Random Survival Forest (RSF) (Ishwaran et al., 2008) and the original
DeepSurv.
We evaluate the predictive ability of the learned models based on the concordance index (CI), a standard performance metric for model assessment in survival analysis, which measures the agreement between the rankings of the predicted and observed survival times.
The performance in terms of the CI is reported in Table 2.
We see that Cox-HC and Cox-STG outperform the alternative methods, indicating that our method shrinks the feature size while learning a predictive model.
C ONCLUSION
In this paper, we propose a novel embedded feature selection method for neural networks and linear models based on stochastic gates.
It has an advantage over previous (cid:96) 1 based regularization methods in terms of achieving a high level of sparsity while learning effective non linear models.
We justify our probabilistic feature selection framework from the information theoretic perspective.
In experiments, we demonstrate that our method consistently outperforms existing embedded feature selection methods in both synthetic and real datasets.
A A PPENDIX B A LGORITHM
Here we present the pseudo code of our method, presented in Algorithm 1.
The loss L is typically negative-log likelihood for classiﬁcation and squared loss for regression.
N is the sample size, D is the number of features, K is the number of Monte Carlo samples.
Input: X ∈ R N × D , target variables y ∈ R N , regularization parameter λ , number of epochs M , learning rate γ ,
Output: Trained model f θ and parameter µ ∈ R D .
1: Initialize the model parameter θ .
Set µ = 0 .
2: for i = 1 , ..., M do 3: for n = 1 , ..., N do 4: for d = 1 , ..., D do 5: for k = 1 , ..., K do 6:
Sample (cid:15) ( k ) d ∼ N (0 , σ d ) 7:
Compute the gate z ( k ) d = max(0 , min(1 , µ d + (cid:15) ( k ) d + 0 . 5)) 8: end for 9:
Set z d = 1 K (cid:80)
Kk =1 z ( k ) d 10: end for 11:
Set z
=
[ z 1 , ..., z D ] T 12: end for 13:
Compute the loss ˆ L = 1 N (cid:80) Nn
=1 L ( f θ ( x n (cid:12) z ) , y n ) 14:
Compute the regularization term R = λ
(cid:80) Dd =1 Φ( µ d +0 .
5 σ d ) 15:
Update θ := θ − γ ∇ θ ˆ L and µ := µ − γ ∇ µ (ˆ L + R ) 16: end for 17:
We empirically observe that setting the number of Monte Carlo samples K = 1 and the standard deviation of the Gaussian distribution σ d = 0 .
5 for d = 1 , ..., D sufﬁces for feature selection in our experiments.
See Section E in the Appendix for more details about the speciﬁc choice of σ d .
After training, the set of indices for selected features is: { d : min(1 , max(0 , µ d + 0 . 5)) > 0 } .
C P ROOF OF P ROPOSITION
We now provide a proof for Proposition 1, showing the equivalence between the stochastic optimization (Eq.7) and the deterministic one (Eq. 6).
Let ˜ S be a subset such that S ∗ \ ˜ S (cid:54) = ∅ .
That is there exists some element in S ∗ that is not in ˜ S .
For any such set ˜ S we have that I ( X ˜ S ; Y ) <
I ( X ; Y ) .
Indeed, if we let i ∈ S ∗ ∩ ˜ S
c then we have
[Equation]
where the ﬁnal inequality follows by Assumption 1.
Assumption 2 also yields that for any set ˜ S such that S ∗ ⊂ ˜ S , we have I ( X ˜ S ; Y )
=
I ( X ; Y ) .
Now, when we consider the Bernoulli optimization problem we have
[Equation]
The mutual information can be expanded as
[Equation]
where we have used the fact that S is independent of everything else.
Recall that in optimization (Eq.
7) the coordinates of S are sampled at random.
Therefore, the distribution that is being optimized over p π is a product distribution.
Our goal is to understand the form of this distribution.
To that end, we will consider a problem dropping the independence constraint.
If we can show that the distribution found by solving this new optimization problem with less constraints is still a product distribution, then we obtain a solution to the original optimization (Eq. 6).
Now, from above we know that the optimal value of the optimization is I ( X (cid:12) ˜ S ; Y ) for any set
S ∗ ⊂ ˜ S .
Hence, any unconstrained distribution should place all of its mass on such subsets in order to maximize the mutual information.
As a result (cid:80) l ∈ S ∗ p ( S l = 1) = k .
However, there is an optimization constraint that E [ (cid:80) l S l ] ≤ k .
Therefore, E [ S l ] = 0 for any l / ∈ S ∗ .
Hence, the optimal solution is to select the distribution so that all of the mass is placed on the subset S ∗ and no mass elsewhere.
As this is also a product distribution, this complete the proof of the claim.
D B RIDGING THE T WO P ERSPECTIVES
To motivate the introduction of randomness into the risk, we have looked at the feature selection problem from a MI perspective.
Based on the MI objective, we have observed that introducing randomness into the constrained maximization procedure, does not change the objective (Proposition 1).
Here we provide a relation between the MI objective (Eq. 6) to the empirical risk (Eq. 1), which supports our proposed procedure.
We ﬁrst note that the MI maximization over the set S can be reformulated as the minimization of the conditional entropy
H
( Y | X S ) since H ( Y ) does not depend on S :
[Equation]
Recall that X S
= X (cid:12) ˜ S .
By Proposition 1, we can rewrite the deterministic search over the set S by a search over the Bernoulli parameters π :
[Equation]
where the expectation is over X , Y ∼ P θ ∗ , which is the true data distribution, and ˜ S ∼ Bern ( ˜ S | π ) .
Put our model distribution as P θ , then we can rewrite the right hand side as:
[Equation]
Since KL ( P θ ∗ ( Y | X (cid:12) ˜ S )
||
P θ ( Y | X (cid:12) ˜ S )) is non-negative, E ˜ S KL ( P θ ∗ ( Y | X (cid:12) ˜ S )
||
P θ ( Y | X (cid:12) ˜ S )) is also non-negative because it is a weighted sum of non-negative terms.
Noting that
[Equation]
we can conclude that
[Equation]
If we consider the negative log likelihood of the target given the observations (i.e. − log P θ ( Y | X (cid:12) ˜ S ) ) as a loss function L (which encodes the classiﬁcation or regression function f θ ), then we see that minimizing the risk approximately maximizes the MI objective in Eq. 6.
E D ETAILS OF R EGULARIZATION T ERM
Here we provide the detail description of the regularization term.
For the vector of stochastic gates z ∈ R D , the regularization term is expressed as follows:
[Equation]
The derivative of the regularization term with respect to the distribution parameter µ d is simply the
Gaussian PDF:
[Equation]
The effect of σ can be understood by looking at the value of ∂∂µ d E Z
||
Z
|| 0 .
In the ﬁrst iteration during the training, µ d is 0.
Therefore, during the initial phase of training, it is close to
[Equation]
In order to remove irrelevant features, this term has to be greater than the derivative of the loss with respect to µ d because otherwise µ d is updated in the incorrect direction.
To encourage such behavior, we set σ = 0 .
5 , which is around the maximum of the gradient during the initial phase as shown in Fig.
Although the point that attains the maximum moves as µ changes, we empirically observe that setting σ = 0 .
5 performs well in our experiments when the regularization parameter λ is appropriately {+tuned (see Section I). Z || 
+}
| µ =0 = 1 √ 2 πσ 2 e − 18 σ 2 for σ = [0 .
F I SSUES IN G RADIENT E STIMATION OF D ISCRETE R ANDOM V ARIABLES
In Section 3.1, we have introduced Bernoulli random ˜ s d , d = 1 , ..., D variables with corresponding parameters π d into the risk objective (Eq. 4).
Taking the expectation over the (cid:96) 0 norm of ˜ S boils down to the sum of the Bernoulli parameters π d .
However, the optimization of the resulting objective suffers from high variance due to the discrete nature of ˜ S .
Here, we attempt to convey this problem by analyzing the risk term in the objective in Eq. 4.
Using the Bernoulli paramterization the empirical risk ˆ R ( θ , π ) is expressed as
[Equation]
In practice, as the outer sum involves enumerating 2 D possibilities of the indicator variables, one can replace the outer sum with Monte Carlo samples from the product of Bernoulli distributions B ( z | π ) .
However, a Monte Carlo estimate of ∂∂π d ˆ R ( θ , π ) suffers from high variance.
To see this, consider the following exact gradient of the empirical risk with respect to π d , which is
[Equation]
where p ( z i (cid:54) =
d ) = (cid:81) Di (cid:54) =
d π z i i (1 − π i )
1 − z i , by absorbing the model f θ ( · ) and the data into L ( · ) .
Due to the discrete nature of z , we see that even the sign of the gradient estimate becomes inaccurate if we can only access a small number of Monte Carlo samples.
While a score-function estimator such as REINFORCE (Williams, 1992) can be used, it is known that the reparametrization trick reduces the variance more in practice.
G C OMPARISON TO H ARD -C ONCRETE DISTRIBUTION
To evaluate the strength of the proposed continuous relaxation of the Bernoulli distribution, described in Subsection 3.1, we compare it with the Hard-Concrete distribution (Louizos et al., 2017), another continuous surrogate for Bernoulli distributions, which was originally developed for neural network model compression.
The details of the Hard-Concrete distribution is described below.
G. H ARD -C ONCRETE DISTRIBUTION
The authors in (Louizos et al., 2017) introduce a modiﬁcation of Binary Concrete, whose sampling procedure is as follows:
[Equation]
where ( τ, ζ ) is an interval, with τ < 0 and ζ > 1 .
This induces a new distribution, whose support is [0 , 1] instead of (0 , 1) .
With 0 < β < 1 , the probability density concentrates its mass near the end points, since values larger than 1 − τ ζ − τ are rounded to one, whereas values smaller than − τ ζ − τ are rounded to zero.
The CDF of s is
[Equation]
and so the CDF of ¯ s is
[Equation] [Equation]
where φ = ( β, log α, ζ, τ )
Now, the probability of being the gate z i being active is 1 − Q ¯ s (
| φ ) and can be written as
[Equation]
G. T HE EFFECT OF THE HEAVY TAIL DISTRIBUTION
The main difference between our proposed distribution (STG) and the Hard-Concrete (Louizos et al., 2017) distribution is that the latter is based on the logistic distribution, which has a heavier tail than the Gaussian distribution we have employed.
As shown in Fig 5, the heavy-tailness results in instability during training.
Furthermore, our method converges much faster and more reliably than the feature selection method using the Hard-Concrete distribution on a the two-moons, XOR and
MADELON datasets (see Subsection J.1 and J.2).
A similar phenomenon is also demonstrated in the MNIST experiment (see Subsection J.3).
[Figure]
H N ON -C ONVEX D ETERMINISTIC R EGULARIZATION
Our proposed objective for feature selection (summarized in Eq. 5) consists of three major differences compared to the LASSO formulation.
• Non-linearity, which is obtained by learning parameters of a multi-layer network with non linear activation’s.
• Feature gates, these are a speciﬁc form on non convex regularization.
• Stochasticity, which is achieved by injecting noise via the reparametrization trick.
In section 6.1, we have demonstrated that even in the linear regime the proposed formulation outperforms the LASSO in the task of identifying active features (see Fig. 1(a)).
The stochastic property of the gates proposed in this paper is justiﬁed using a MI prospective (see
Section 4).
Nonetheless, in this section we evaluate whether the practical performance gain of our method stems from the non-convex regularization or from it’s combination with the injected noise.
We deﬁne a deterministic gate using ˜ z
d = g ( µ d ) = max(0 , min(1 , µ d + 0 . 5)) , where µ d is a parameter learned for each feature d = 1 , ..., D .
In the linear regression setting we can deﬁne the non-convex deterministic (NCD) as
[Equation]
where Φ is the standard Gaussian CDF.
Combined with ˜ z
d this non-convex regularized objective is deterministic and differentiable, and its solution can be approximated via gradient decent.
We evaluate the applicability of such alternative objective for the task of feature selection in linear regression.
The experimental setting follows the description in Section 6.1 using D = 64 .
In Fig.
6(b), we compare the values of the deteministic gates (DNC) and stochastic gates (STG) using
N = 60 .
As depicted from this ﬁgure, the STG correctly identiﬁes the active features (ﬁrst 10 ) and sparsiﬁes the non-active features.
The deterministic version DNC does induce partial sparsity, but fails to remove some of the irrelevant features.
Furthemore, it does not push the gates values of active features up to 1 .
Clearly, in the deterministic setting, shrinking the value of µ d while compensating with a large coefﬁcient in θ is preferable for the objective in Eq.
To evaluate the probability of support recovery using DNC, we apply additional thresholding to the gates.
Speciﬁcally, we use a threshold of 0 .
5 to deﬁne if a feature is active or not.
A comparison with STG, HC and LASSO in this setting appears in Fig. 6(a).
The DNC is superior to LASSO, but inferior to the HC and STG.
Which means that there is a true advantage of stochacity in our formulation.
One aspect that is crucially difference between the STG and DNC is what we call “second chance”.
For the deterministic formulation, if a feature is zeroed out in an early training phase, the gradient of the feature vanishes for further training and it will never be pulled back.
For the STG, even if the feature has been pulled to zero, the gate may still be pushed back up to one at a later training phase.
This is due to the injected noise which allows to reevaluate the gradient of each gate.
In Fig. 6(c), we demonstrate this “second chance” effect using N = 60 samples and presenting the gate’s values (throughout training) for an active feature.
To conclude, this form of deterministic gates (Eq. 12) is not suitable for an embedded feature selection method.
Using thresholding seems somewhat effective; however such thresholding was not required for the STG in any of our experiments.
While the shrinkage phenomenon might be avoided using further regularization on θ this is out of the scope of the current study.
[Figure]
I T UNING THE REGULARIZATION PARAMETER λ
There have been several studies (Lederer & Müller, 2015; Coen et al., 2006) on the problem of tuning regularization parameter in regression and classiﬁcation.
Perhaps the most related regularized objective is the LASSO formulation.
Studies such as (Chand, 2012; Fan & Tang, 2013) develop methods for tuning the regularization parameter in LASSO.
In Eq.
5 as in the LASSO formulation, increasing the value of λ effectively forces a sparser solution with less features.
For simplicity, we focus on one popular way for tuning λ in the LASSO.
The idea is to minimize the generalization error using a cross validation procedure over a range of λ values.
In this section, we demonstrate that this procedure is also applicable for the proposed approach.
We apply the STG to the RAI dataset and evaluate the MSE using a 5 -folds cross validation procedure.
We compute the MSE on the validation data using 100 values of λ logarithmically scaled in [10 − 3 , 10 1 ] .
As demonstrated in Fig.
7, using a cross validation procedure for tuning λ provides a clear optimal value for this example.
J A DDITIONAL E XPERIMENTS J. MADELON DATASET
The MADELON dataset, ﬁrst suggested for the NIPS 2003 feature selection problem, is a multivariate highly nonlinear binary classiﬁcation problem.
The MADELON dataset is generated using 32 groups of data points placed on a 5 dimensional hyper-cube and randomly labeling them by one of the two class labels.
The ﬁrst 5 informative features are then used to create 15 additional coordinates that are formed based on a random linear transformation of the ﬁrst 5 .
A Gaussian noise N (0 , 1) is added to each feature.
Next, additional 480 nuisance coordinates are added in the same manner.
These features have no effect on the class label.
Finally, 1% of the labels are ﬂipped.
To be consistent with the
XOR and two moons experiments, we use 1 , 500 points from this dataset, and evaluate our proposed method in terms of its predictive power and ability to detect the informative features.
We vary the regularizaton parameter λ in the range [0 . 01 , 10] and evaluate the classiﬁcation accuracy using 5 folds cross validation.
In this example, we restrict our comparison to Random Forest and LASSO.
We focus on Random Forest as it was the strongest competitor to our method in all of our experiments, while LASSO is evaluated because it is a widely used embedded feature selection method.
As evident from Fig. 8(a), our method achieves the highest accuracy while using less features.
Moreover, as depicted from this ﬁgure, peak performence occurs when selecting 5 features, thus, our method provides a clear indication to the true number of informative features.
Both LASSO and RF on the other hand, do not provide a clear indication of the true number of relevant features.
In Fig.
8(b), we evaluate the effect of λ on the number and quality of selected features.
As shown in this plot, there is a wide range of λ ’s such that our method only selects relevant features.
Finally, as evident from the plato on the right hand side of the red plot, there is a range of λ ’s such that exactly 5 features are selected. [Figure]
J. T WO M OONS CLASSIFICATION WITH NUISANCE FEATURES
In this experiment, we construct a dataset based on "two moons" shape classes, concatenated with noisy features.
The ﬁrst two coordinates x 1 , x 2 are generated by adding a Gaussian noise with zero mean and the variance of σ 2 r = 0 .
1 onto two nested half circles, as presented in Fig.
9(a).
Nuisance features x i , i = 3 , ..., D , are drawn from a Gaussian distribution with zero mean and variance of σ 2 n = 1 .
We reserve the 70% as a test set, and use 10% of the remaining training set as a validation set.
We follow the same hyperparameter tuning procedure as in the XOR experiment.
The classiﬁcation accuracy is in Fig. 9(b).
Based on the classiﬁcation accuracies, it is evident that for a small number of nuisance dimensions all methods correctly identify the most relevant features.
The proposed method (STG) and Random Forest (RF) are the only methods that achieve near perfect classiﬁcation accuracy for a wide range of nuisance dimensions.
The other NN based method (DFS) seem to converge to sub-optimal solutions.
We note that the median rank for all the methods is 1.5.
[Figure]
J. S PARSE H ANDWRITTEN DIGITS CLASSIFICATION
In the following toy example, we attempt to distinguish between images of handwritten digits of 3’s and 8’s using samples from MNIST (LeCun et al., 1998).
The orientation and location of the digits is more or less the same throughout this dataset, therefore for these two classes (3’s and 8’s), we expect that some of the left side features (pixels) would be sufﬁcient for the separation.
The experiment is performed as followed.
We reserve 90% of the data as the test set, and train on the remaining 10% .
We then apply STG and evaluate the classiﬁcation accuracy and the number of selected features.
We use the architecture [200, 50, 10] with tanh activations.
The experiment was repeated 10 times, the extracted features and accuracies were consistent over 20 trials.
We noticed a relatively small number of selected features, which are positioned southwest and close to the center of the images, achieve very high classiﬁcation accuracy.
An example of 9 randomly selected samples overlaid with the weights of the selected features is presented in Fig. 10(a).
Furthermore, we also evaluate the effect of λ on the the number of selected features and accuracy of the method.
We apply our method (STG) and its variant using the Hard-Concrete distribution to a randomly sampled training set of size N = 1500 and vary λ in the range of [0 . 001 , 0 . 01] .
In Fig.
10(b)
we present the accuracy and sparsity level vs. the λ parameter.
This experiment demonstrates the improved performance of the proposed distribution compared to the Hard-Concrete (HC (Louizos et al., 2017)), which was designed for neural net model compression.
Not only that the overall accuracy is superior, but also it seems that the transition as a function of λ is smoother, which suggests that the method is less sensitive to the choice of λ . J. G ISETTE DATASET
The Gisette is a handwritten dataset also appears in the NIPS 2003 feature selection challenge.
The data consists of 13 , 500 handwritten digits of ’4’ and ’9’.
The original digits have 28 × 28 pixels, which were modiﬁed for the feature selection challenge.
Speciﬁcally, a random subset of the features are embedded in a 2500 dimensional space and additional 2500 irrelevant probes are added.
The train, validation and test dimensions are 6000 / 1000 / 7500 respectively. We apply the proposed STG based feature selection method and compare its performance to Random Forests (RF) and Extremely Randomized Trees (Tree). As evident from Fig.
11, we obtain a high accuracy even for a dramatic reduction in the feature size.
LASSO is not presented in this experiment as its performance is dramatically inferior to the alternatives (accuracy < 0 . 6 ).
[Figure]
J. R EUTERS C ORPUS V OLUME I
The Reuters Corpus Volume I (RCV1) consists of 800 , 000 newswire stories manually labeled by 103 categories.
This is a multilable regime, i.e. each story is assigned to multiple categories.
Here we focus on a binary subset of this corpus, with 23 , 203 stories.
The total number of feature is 47 ,and the train, validation and test portions are 10% , 8 .
5% and 81 .
5% respectively.
We evaluate the performance of our method using a 5 -fold cross validation.
A comparison the LASSO and RF appears in Fig.
This example demonstrates that our method is also effective in an extremely high dimensional regime of non linear function estimation.
J. D ETAILS OF C OX P ROPORTIONAL H AZARD M ODEL
Survival times are assumed to follow a distribution, which is characterized by the survival function
S ( t ) =
P ( T > t ) .
A hazard function, which measures the instantaneous rate of death, is deﬁned by h ( t ) =
lim ∆ t → 0 P ( t<T ≤ t +∆ t | T>t ) ∆ t = p ( t ) S ( t ) .
We can relate the two functions in the following way:
[Equation]
Proportional hazard models assume a multiplicative effect of the covariates x on the hazard function such that
[Equation]
where h 0 ( t ) is a baseline hazard function, which is often the exponential or Weibull distribution, and θ is the parameter of interests.
One of the difﬁculties in estimating θ in survival analysis is that a large portion of the available data is censored.
However, in order to obtain estimates, Cox observed that it is sufﬁcient to maximize the partial-likelihood, which is deﬁned as follows:
L ( θ ) =
(cid:81) T i uncensored
[Figure]
In (Katzman et al., 2018), the authors propose DeepSurv, which uses a deep neural network model to replace the linear relations between the covariate x and θ , demonstrating improvements of survival time prediction over existing models such as CPH and the random survival forest (Ishwaran & Kogalur, 2007), (Ishwaran et al., 2008).
The Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) dataset consists of gene expression data and clinical features for 1 , 980 patients, and 57 .
72% have an observed death due to breast cancer with a median survival time of 116 months.
The METABRIC dataset involves 24,368 features (genes).
Most genes are irrelevant for outcome prediction.
To demonstrate the advantage of STG in the context of survival analysis, we selectedwell-known genes relevant for survival (out of the 24,368 genes) that correspond to the Oncotype
DX test, a gene panel used for treatment decision making.
We also include ﬁve additional clinical features (hormone treatment indicator, radiotherapy indicator, chemotherapy indicator, ER-positive indicator, and age at diagnosis).
We then added 200 additional irrelevant gene variables that we selected randomly from the remaining list of genes.
After we omit the null values, the number of samples is 1969.
We reserve the 20% for test, and use the 20% of the remaining training set as validation.
(That is, Train: 1260, Valid: 315, Test:samples.)
Experimental Detail For DeepSurv, we manually select the architecture using the validation set so that we obtain a similar performance reported in (Katzman et al., 2018).
The learning rate decay is set to 1.
The learning rate and the regularization parameter are optimized via Optuna using the validation set, where the search range is set LR : [1 e − 3 , 1] and λ : [1 e − 3 , 1] .
The hyperparameters used in the experiment are the following: architecture :[60, 20, 3], activation: selu (as suggested by (Katzman et al., 2018)), learning rate : 0.152, learning rate decay: 1 . 0 , σ : 0 .
5 , λ : 0 . 023 , training epoch: 2000.
Note that to see the effect of feature selection, we used the hyperparameters optimized for DeepSurv to test our method (STG-DeepSurv).
J. A DDITIONAL E XPERIMENTAL D ETAILS
Here we provide a full description of the procedures we have performed in the experimental parts of the paper.
For synthetic datasets are ﬁrst split into train, validation and test.
Validation is always 10% of the train, while the exact ratios between train and test is detailed for each experiment separately.
All the neural network weights are initialized by drawing from N (0 , 0 . 1) and bias terms are set to 0.
All the batch sizes are equal to the number of training samples.
In table 3, we detail the search range of hyperparamters as well as the exact values used in our experiments.
We set n-trials = 1000 for
Optuna, which is a deﬁne-by-run based hyperparameter optimization software equipped with efﬁcient search and pruning strategies.
We use SGD for all the experiments, except for the Cox model where we use Adam.
All the experiments are conducted using Intel(R)
Xeon(R) CPU E5-2620 v3 @2.4Ghz x2 (12 cores total).
[Table]
For the Phase Transition experiment [Table] we use 0.1 as a learning rate.
For all the experiments when we use Tree and RF, we use the default value for max-depth, so that nodes are expanded based on the purity.
For the XOR problem, the exact architectures used for the NN based methods are: (STG/HC): [476 , 490 , 14] with Tanh, (DFS): [100 , 10] with Tanh, (SG-L1-NN): [100 , 10 , 5] with Tanh.
For the two moons we use (STG): [490 , 406 , 18] with Tanh, (DFS): [158 , 27 , 224] with
Tanh, (SG-L1-NN): [88 , 28 , 27] with Tanh.
For the XOR problem, we attempted to use Optuna to optimize parameters of DFS and SG-L1-NN, but we ended up using the architecture suggested by the authors (Li et al., 2016; Scardapane et al., 2017) as they outperform the values suggested by
Optuna.
The number of epochs used for the XOR problem is 20 K, 14 K, 800 for STG/HC, DFS and SG-L1-NN respectively.
Regularization parameters are 0 .
3 e − 5 and 3 e − 5 respectively.
The number of epochs used for the two-moons problem is 20 K, 1570 , 708 for STG/HC, DFS and
SG-L1-NN
respectively.
Regularization parameters are 0 .
48 , 9 e − 3 and 1 e − 3 respectively.
We note that the regularization parameters and learning procedure is different in nature, as we use an (cid:96) 0 type penalty.
For the PBMC experiment, the architecture was hand-tuned using the validation set and set as [200 , 100 , 50 , 10] with Tanh.
Learning rate was 0 .
2 and the number of epochs 4000 .
The hyperparameter λ varies in the range [0 . 001 , 0 . 11] to achieve different levels of sparsity.
For
MADELON, we use the architecture optimized for the binary XOR classiﬁcation.
The number of epochs used is 20 K , the learning rate is 0 .
06 and λ varies in the range [0 . 01 , 10] .
For regression, the learning rate and the λ are optimized via Optuna using the search range LR : [1 e − 4 , 1] and λ : [1 e − 4 , 10] based on validation.
The parameters used are the following: (SE1) architecture [600, 200, 100, 50, 1] with ReLu activations, num epochs: 5, λ : 5 , LR : 0 .
(SE2) architecture [600, 300, 150, 60, 20] with ReLu activations, num epochs=2000, λ : 5 , LR : 0 .
001 (SE3) architecture [600, 300, 150, 60, 20] with ReLu activations, num epochs : 1000, λ : 1 , LR : 0 .
(RCP) architecture [1000, 300, 150, 60, 20] with ReLu activation, num-epochs: 2000, λ : 5 . 0 , LR : 0 .
(REL) architecture [26,91,63] with ReLu activation, num-epochs: 1600, λ : 0 .
031 , LR : 0 .
(RAI) architecture [10,177] with ReLu activation, num-epochs: 1800, λ : 0 .
019 , LR : 0 . 002 .
Architectures for SE1-SE3 and RCP where tuned manually.
The ratio of train/test/valid split is 1:1:1 for synthetic data.
For the real data (RCP and REL), the train size is 6000, the test and valid size is 1000 samples.
For RAI the train size is 5000, the test and valid size is 1000 samples.
In order to deﬁne the IFWR, for STG, the d th feature weight is set to max(0 , min(1 , µ d + 0 . 5)) .
For other neural net based methods, it is given by (cid:80) j W dj , where W is the weight matrix of the ﬁrst layer.
For other methods we just used the feature relevance returned by the trained model.
Finally, the LASSO’s IFWR in the XOR experiment was omitted from the manuscript as it suffered from high variance.
Regarding the comparison performed in the two-moons and XOR problem, we believe that adding
IFWR along with classiﬁcation accuracy versus number of feature selected provides a complementary perspective in demonstrating the efﬁcacy of feature selection techniques.
We emphasize that our goal is not to just rank features but select features by assigning the weight of 0 to irrelevant features while simultaneously obtaining good predictive accuracy.