S ELF -S UPERVISION E NHANCED F EATURE S ELECTION WITH C ORRELATED G ATES
Anonymous authors
Paper under double-blind review
A BSTRACT
Discovering relevant input features for predicting a target variable is a key scientiﬁc question.
However, in many domains, such as medicine and biology, feature selection is confounded by a scarcity of labeled samples coupled with signiﬁcant correlations among features.
In this paper, we propose a novel deep learning approach to feature selection that addresses both challenges simultaneously.
First, we pre-train the network using unlabeled samples within a self-supervised learning framework by solving pretext tasks that require the network to learn informative representations from partial feature sets.
Then, we ﬁne-tune the pre-trained network to discover relevant features using labeled samples.
During both training phases, we explicitly account for the correlation structure of the input features by generating correlated gate vectors from a multivariate Bernoulli distribution.
Experiments on multiple real-world datasets including clinical and omics demonstrate that our model discovers relevant features that provide superior prediction performance compared to the state-of-the-art benchmarks, in practical scenarios where there is often limited labeled data and high correlations among features.
I NTRODUCTION
High-dimensional datasets are increasingly prevalent in a variety of ﬁelds, including critical domains such as medicine and biology.
Discovering features responsible for the target variable is an essential but challenging step in the analysis of such data.
For example, while next generation sequencing can detect the expression of tens of thousands of genes per sample (e.g. RNA-seq), many genetic disorders stem from the variation in only a few groups of related genes (Jackson et al., 2018).
Identiﬁcation of such disease-related factors is crucial for the design of therapeutic treatments.
Furthermore, feature selection has additional beneﬁts, including reduced experimental costs, greater interpretability, and improved model generalization (Min et al., 2014; Ribeiro et al., 2016; Chandrashekar & Sahin, 2014).
However, the high-dimensional and often noisy nature of such data prevents relevant features from being readily discovered.
Moreover, in many domains there is a scarcity of label information due to cost, privacy reasons, or experimental study design.
While deep learning approaches have resulted in improvements in feature selection (e.g. Yamada et al.
(2020); Lemhadri et al.
(2021), see Related
Work), such methods typically assume access to many labeled samples.
With low sample size, feature selection methods, in particular those employing deep learning, have shown a propensity to overﬁt high-dimensional data (Kuncheva et al., 2020).
This issue is further exacerbated by the inherent structure of such data.
Many (high-dimensional) datasets exhibit substantial inter-correlations or multicollinearity – i.e. there exist features that are (highly) correlated among themselves – which impacts the performance of feature selection methods (Chong & Jun, 2005; Katrutsa & Strijov, 2017; Belsley et al., 2005).
In particular, this structure can cause parameter estimates to be unstable (Dobson & Barnett, 2018) and, in the extreme, can prevent variable effects from being separated, confounding the problem of feature selection (Meloun et al., 2002).
Existing deep learning-based methods for feature selection do not (explicitly) consider the correlations between features which will often result in the selection of redundant features.
When labeled samples are scarce, large numbers of unlabeled samples are often available (e.g.
Perez-Riverol et al.
However, the importance of unlabeled data has been overlooked in feature selection despite its potential to prevent the model selection process from overﬁtting by allowing informative feature representations to be learned.
Therefore, the development of methods for feature selection that can exploit both labeled and unlabeled data is of great practical importance.
Contributions.
We propose a novel method for feature selection that addresses two key challenges in real-world scenarios: limited labeled data and correlated features.
We use a self-supervised approach to train an encoder using unlabeled data via two pretext tasks: feature vector reconstruction and gate vector estimation.
This pre-conditions the encoder to learn informative representations from partial feature sets, aligning the self-supervision with the model selection process of the downstream feature selection task.
In addition, we introduce a novel gating procedure that accounts for the correlation structure of the input features.
This ensures the pretext tasks remain challenging by preventing the model from memorizing trivial relations between features.
Moreover, unlike previous deep learning-based feature selection methods, the correlated gate vectors encourage our method to select the most relevant features by making multiple correlated features compete against each other.
We validate our approach through experiments on synthetic and multiple real-world datasets, including clinical and omics, where only a small number of labeled samples are available.
Our model discovers relevant features that provide superior prediction performance compared to state-of-the-art benchmarks, and we corroborate these features with supporting medical and scientiﬁc literature.
R ELATED W ORK
Feature Selection Methods.
Feature selection is a well-studied problem, with a number of proposed solutions including wrapper (Kohavi & John, 1997) and ﬁlter (Liu & Setiono, 1996; Kira & Rendell, 1992) methods.
Recent advances in deep learning provide an elegant way of training embedded feature selection methods by jointly learning to perform feature selection while training a prediction network (Huang et al., 2020).
These methods learn to perform the non-differentiable process of selecting feature subsets by approximating it either with Lasso or elastic net penalization (Li et al., 2016), using an MCMC sampling approach (Liang et al., 2018), or more recently with continuous relaxation using independent Gaussian random variables (Yamada et al., 2020).
However, supervised feature selection methods can fail to identify relevant features when limited labeled samples are available due to overﬁtting (Kuncheva et al., 2020), impacting their suitability in many real-world scenarios.
Moreover, these methods do not consider the underlying correlation structure of the input features which can be problematic when selecting relevant features among (highly) correlated ones.
Relatively few feature selection methods use unlabeled samples.
Abid et al.
(2019) used autoencoders to identify a pre-speciﬁed number of features that are sufﬁcient for reconstructing the data.
Lindenbaum et al.
(2020) improved the well-known Laplacian score (He et al., 2005) by selecting feature subsets that better capture the “local structure” of the data.
However, both approaches are fully unsupervised and, without the guidance of label information, can fail to identify features relevant to the target outcome.
While a number of semi-supervised feature selection methods have been proposed (Sheikhpour et al., 2017), they have typically been extensions of traditional methods, such as a Laplacian score with modiﬁed afﬁnity scores using labeled samples (Zhao et al., 2008) or manifold regularization based on linear SVMs (Dai et al., 2013).
To the best of our knowledge, this is the ﬁrst deep learning framework that fully utilizes both labeled and unlabeled samples for feature selection.
Self-Supervised Learning.
Self-supervised learning methods create (weak) supervised signals from unlabeled data, employing contrastive learning or pretext task(s) to provide surrogate labels.
While self-supervised learning has found success in computer-vision (Chen et al., 2020) and natural language processing (Devlin et al., 2018), the tabular domain, which is most relevant in feature selection, has been largely neglected.
Methods for tabular data seek to reconstruct data either based on a corrupted sample alone (Vincent et al., 2008; Arik & Pﬁster, 2019;
Yin et al., 2020) or with knowledge of which entries have been corrupted (Pathak et al., 2016).
Recently, Yoon et al.
(2020) jointly learn to recover the original sample and predict the mask vector used to corrupt the data.
While our pretext tasks are also reconstruction-based, we propose a novel method for generating the gate vector used to produce the input feature vector.
This is particularly important when there exists substantial correlation between features (see Section 5).
All existing methods have used independent, uniform sampling; however, this allows highly correlated features to be readily reconstructed since it is likely that not all such features will be corrupted.
We prevent this by incorporating the correlation structure into the gating procedure.
In this work, we employ pretext tasks that mirror the goal of the feature selection process: the input to the encoder is a subset of the features selected at random.
Employing partial feature sets for self-supervised learning encourages the encoder to learn better representations of these partial feature sets.
This in turn beneﬁts learning both the model selection function and the feature selection step since they are also trained using partial features sets.
P ROBLEM F ORMULATION
Let X =
( X 1 , · · · , X p ) ∈
X p and Y ∈ Y be random variables for the high-dimensional input features (e.g., gene expressions) and the target outcome (e.g., disease traits) whose realizations are denoted as x = ( x 1 , · · · , x p ) and y , respectively.
Throughout the paper, we will often use lowercase letters to denote realizations of random variables.
Embedded feature selection aims to select a subset, S ⊂
[ p ] , of features that are relevant for predicting the target as part of the model selection process.
Denote ∗ be any point not in X and deﬁne
X S =
( X ∪{∗} ) p .
Then, given X ∈ X p , the selected subset of features can be denoted as X S ∈ X S where x S ,k
= x k if k ∈ S and x k = ∗ if k / ∈ S .
Let f : X S
→ Y be a function in F that takes as input the subset X S and outputs Y .
Then, selecting relevant features can be achieved by solving the following optimization problem:
[Equation]
subject to |
[Equation]
(1) where δ constrains the number of selected features and (cid:96) Y ( y, y (cid:48) )
= − (cid:80)
Cc =1 y c log y (cid:48)
c for C -way classiﬁcation tasks (i.e., Y = { 1 , · · · , C } ) and (cid:96) Y ( y, y (cid:48) )
= (cid:107) y − y (cid:48)
(cid:107) 22 for regression tasks (i.e.,
Y = R ).
Unfortunately, the combinatorial problem in (1) becomes intractable for high-dimensional data as the search space increases exponentially with p .
Hence, we instead focus on a relaxation by converting the combinatorial search in (1) into a search over the space of possibly correlated binary random variables.
It is worth highlighting that unlike existing work (e.g. Yamada et al.
(2020); Yoon et al.
, we do not assume independence among these random variables.
Let M =
( M 1 , · · · , M p ) ∈ { 0 , 1 } p be binary random variables governed by distribution p M , whose realization m is referred to as the gate vector , for indicating selection of the corresponding features.
Then, the selected features given gate vector m can be written as
[Equation]
where ¯ x = E x ∼ p
X
[ x ] and (cid:12) indicates element-wise multiplication.
Here, we replace not-selected features with their means to resolve any issue when a feature having zero value (e.g., turned-off genes) has speciﬁc meaning.
Finally, we can approximately achieve (1) by jointly learning the model selection f and the gate vector distribution p M based on the following optimization problem:
[Equation]
where f is implemented as a neural network and β is a balancing coefﬁcient that controls the number of features to be selected.
Challenges.
In practice (especially in the healthcare domain), there are two main challenges that confound selecting relevant features for predicting the target: (i) inter-correlation or multicollinearity, which is the existence of features that are (highly) correlated among themselves, and (ii) the absence of sufﬁcient labeled samples.
These challenges make embedded feature selection vulnerable to overﬁtting.
More speciﬁcally, model selection (i.e., learning f ) and feature selection (i.e., learning p M ) are conducted jointly.
Either one being overﬁt to correlated or noisy irrelevant features will end up discovering spurious relations and poor prediction.
For instance, if the network is biased toward irrelevant features, the selection probability (i.e., importance) of those features will be increased as if those features were “discriminative” or “predictive”, and vice versa.
M ETHOD : S ELF -S UPERVISION E NHANCED F EATURE S ELECTION
In this section, we describe our novel feature selection method, which we refer to as S elf-supervision
E nhanced F eature S election (SEFS).
To address the challenges described above, we propose a novel gate vector generation process modeled by a multivariate Bernoulli distribution where dependencies among gates are transferred from the correlation structure of the input features, and a two-step training procedure that utilizes both labeled and unlabeled samples.
Our approach is deﬁned by the selection probability π , that governs the gate vector generation process, and the model selection function f , that we decompose into an encoder f θ and a predictor f φ , i.e. f = f φ ◦ f θ , as illustrated in Figure 1.
To utilize both labeled and unlabeled samples, these components are updated via a two-step training procedure:
• Self-Supervision Phase : the encoder f θ is pre-trained to learn favorable representations for feature selection using unlabeled samples via solving two pretext tasks speciﬁcally designed to take only subsets of the features as input.
• Supervision Phase : the learned representations from the Self-Supervision Phase are adopted to solve the objective in (3) – i.e. updating the encoder f θ , the predictor f φ , and the selection probability π – for discovering task-relevant features using labeled samples.
4. M ULTIVARIATE B ERNOULLI G ATE V ECTORS USING G AUSSIAN C OPULA
Instead of assuming independence in the gating mechanism (2), we model the gate vector distribution p M as a multivariate Bernoulli distribution where dependencies among the gates are determined by the correlation structure of the input features.
To this goal, we use Gaussian copula (Nelsen, 2007) to construct a valid multivariate Bernoulli distribution.
The Gaussian copula is a multivariate cumulative distribution function (CDF) of random variables ( U 1 , · · · , U p ) over the unit cube [0 , 1] p with uniform marginals, where U k ∼ Uniform (0 , 1) for k ∈ [ p ] .
Formally, given the correlation matrix R ∈ [ − 1 , 1] p that captures the correlation structure of X , the Gaussian copula is deﬁned as:
[Equation]
where Φ R stands for the joint CDF of a multivariate Gaussian distribution with mean zero vector and correlation matrix R , and Φ − 1 is the inverse CDF of the standard univariate Gaussian distribution.
Deﬁne π
=
( π 1 , · · · , π p ) ∈ [0 , 1] p to be the selection probability that governs the multivariate
Bernoulli distribution used to generate the gate vector m .
Then, given C R , we can generate the gate vector from a multivariate Bernoulli distribution, which is denoted as m ∼ MultiBern ( π ; R ) , based on the correlated random variables ( U 1 , · · · , U p ) that preserve the correlation structure of the input features; formally, m k
= 1 if U k ≤ π k and m k =
if U k > π k for k ∈ [ p ] .
Using the correlation structure of the input features when generating the gate vector has the following two advantages: (i) during the Self-Supervision Phase, correlated gate vectors encourage the network not to rely on trivial signals from highly correlated features when solving the pretext tasks, which may lead to inefﬁcient pre-training and sub-optimal downstream performance; (ii) during the Supervision Phase, correlated gate vectors encourage the network to select only the most rele- vant features by increasing the chance that correlated features are selected jointly and thus compete against each other.
4. S ELF -S UPERVISION P HASE
Self-supervised learning uses unlabeled samples to automatically generate a (weak) supervisory signal by solving relevant pretext tasks in the absence of labeled data.
The learned representations can then be used to solve downstream tasks.
To this goal, we begin by deﬁning two pretext tasks motivated by recent success in self-supervised learning for tabular data (Yoon et al., 2020).
The pretext tasks to be jointly solved are: (i) reconstructing the original input x based on a randomly selected subset of input features ˜ x , and (ii) simultaneously estimating the gate vector m that deﬁned which features were selected and which features were not (i.e., replaced by their mean value).
To solve these pretext tasks, we formally deﬁne the encoder f θ and two auxiliary network components h ψ 1 , h ψ 2 that are temporarily employed during the Self-Supervision Phase:
• Encoder , f θ : X p → Z , that takes as input either the selected subset of features ˜ x or the original feature vector x , and outputs latent representations z ∈ Z .
• Feature vector estimator , h ψ 1 : Z → X p , that takes z = f θ (˜ x ) as input and outputs a vector ˆ x which is an estimate of the original input x . • Gate vector estimator , h ψ 2 : Z → [0 , 1] p , that takes z
= f θ (˜ x ) as input and outputs a vector ˆ m which is a prediction of which features have been selected (i
e., m k = 1 ) and which features have been replaced by the mean (i
e., m k = 0 ).
When generating the subset of features ˜ x , each feature is selected based on the multivariate Bernoulli distribution with an equal probability π , i.e., π k = π for k ∈ [ p ] , which is a hyper-parameter ﬁxed throughout the Self-Supervision Phase.
The adoption of an equal selection probability allows us to make no assumptions about the relative feature importance, which is not known a priori .
The purpose of the Self-Supervision Phase is to pre-train the encoder.
As such, both estimators take as input the output of the same encoder f θ , which is the only component that will be retained in the
Supervision Phase.
Formally, given the latent representations z
= f θ (˜ x ) , the reconstructed input feature and the gate vector are deﬁned as ˆ x (cid:44) h ψ 1 ◦ f θ (˜ x ) and ˆ m (cid:44)
h
ψ 2 ◦ f θ (˜ x ) , respectively.
The encoder and the two estimators are trained jointly based on the following objective function:
[Equation]
where α is a coefﬁcient chosen to balance between the two pretext task losses
: (cid:96) X
( x , ˆ x ) = (cid:107) x − ˆ x
(cid:107)and (cid:96) M
( m , ˆ m )
= − (cid:80) pk =1 m k log
ˆ m k + (1 − m k ) log(1 − ˆ m k ) .
Compared to previous work (Yoon et al., 2020), the correlated gating procedure in the proposed method prevents the pretext tasks from being solved by only exploiting trivial relationships among features by increasing the probability that highly-correlated features are masked simultaneously.
Further, the pretext tasks directly mirror the Supervision Phase: in both phases, the input to the encoder is a subset of the features ˜ x .
This trains the encoder to produce better representations of partial feature sets which in turn prevents the model selection process in the Supervision Phase from overﬁtting to spurious, noisy features, beneﬁting feature selection.
4. S UPERVISION P HASE
During the Supervision Phase, the encoder, predictor, and the selection probability, which we refer to as the feature selection network , are jointly updated to solve the feature selection objective (3).
The predictor is formally deﬁned as follows:
• Predictor , f φ : Z → Y , that takes as input the latent representations of the selected subset of features, i.e., z = f θ (˜ x ) , and outputs predictions on the target outcome.
Together with the encoder, the model selection function is given by f = f φ ◦ f θ .
To account for the correlations between features, we generate gate vectors using a multivariate
Bernoulli distribution.
This is important since the correlated gate vectors encourage the network to select only the most relevant features by making multiple correlated features compete against each other.
However, solving (3) directly using such gate vectors is intractable since p M ( m ) has no differentiable closed-form expression.
Instead, we adopt a continuous relaxation of the multivariate
Bernoulli distribution (Wang & Yin, 2020) by applying the reparameterization trick to the correlated uniform random variables ( U 1 , · · · , U p ) that preserve the correlation structure R .
Formally, given selection probability π = ( π 1 , · · · , π p ) and the multivariate uniform random variables ( U 1 , · · · , U p ) from Gaussian copula C R , we can generate relaxed gate vector ˜ m
= ( ˜ m 1 , · · · , ˜ m p ) ∈ (0 , 1) p based on the following reparameterization trick (Wang & Yin, 2020):
[Equation]
where σ ( x ) = (1 + exp( − x )) − 1 is the sigmoid function.
Such a relaxation is parameterized by π , a pre-speciﬁed covariance matrix R , and a temperature hyper-parameter τ ∈ (0 , ∞ ) .
Similar to the
Relaxed Bernoulli distribution (Maddison et al., 2017), (6) is differentiable with respect to π .
In practice, the gate vector generation process in (6) can be handled as a deterministic transformation of samples from a standard multivariate Gaussian random variable.
More speciﬁcally, we ﬁrst draw samples from the multivariate Gaussian distribution v ∼ N (0 , R ) by v = ( v 1 , · · · , v p ) =
L (cid:15) where (cid:15) ∼ N (0 , I ) and L is a lower triangular matrix with positive diagonal entries, which is derived by the Cholesky factorization of the covariance matrix, i.e., R = LL T .
Next, we generate correlated uniform random variables
u k = Φ( v k ) for k ∈ [ p ] .
Finally, we can generate a relaxed multivariate
Bernoulli variable ˜ m k ∈ (0 , 1) by plugging in u k and π k into (6).
Under the continuous relaxation, the regularization term in (3) that induces sparsity of selected features can be simply derived as E m ∼ p M (cid:107) m (cid:107)
= (cid:80) pk =1 P ( U k ≤ π k ) =
(cid:80) pk =1 π k .
Overall, we can rewrite our objective as
[Equation]
The model selection function f = f φ ◦ f θ and the selection probability π are updated jointly via gradient descent.
Pseudo-code of SEFS can be found in Appendix A.
E XPERIMENTS
In this section, we evaluate the performance of SEFS and multiple feature selection methods using a synthetic and several real-world datasets.
Further experiments together with detailed information regarding all experiments, benchmarks, and datasets can be found in the Appendix.
Benchmarks.
We compare SEFS with 7 feature selection methods including 3 conventional methods and 4 state-of-the-art methods.
The conventional methods include LASSO regularized linear/logistic model ( Lasso , Tibshirani (1996)), extremely randomized tree ensembles ( Tree , Geurts et al.
and Laplacian Score that preserves locality structure ( L-Score , He et al.
Stateof-the-art deep learning methods include Bayesian neural networks for feature selection ( BNNsel , (Liang et al., 2018)), feature selection based on continuous relaxation ( STG , Yamada et al.
unsupervised feature selection based on gated Laplacian ( DUFS , Lindenbaum et al.
We also include an extension of STG to the semi-supervised setting by augmenting the loss with a reconstruction task that estimates the original features from the gated inputs ( STG (SS) ).
Among the benchmarks, three are able to use both labeled and unlabeled samples: DUFS is fully unsupervised, while both L-Score and STG (SS) are semi-supervised approaches that explicitly use the label.
To highlight our novel two-step training process, we also include a variant of our method that is trained without employing Self-Supervision Phase, which is denoted as SEFS (no SS) .
Performance Metrics.
For synthetic experiments, we utilize the true positive rate (TPR) to assess whether discovered features are truly relevant compared to the ground-truth.
With real-world data, however, it is not possible to calculate the TPR since the ground truth relevance is often not known.
Hence, we instead evaluate the prediction performance of the discovered features obtained by different feature selection methods.
For classiﬁcation tasks, we use the area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC) and for regression tasks, mean squared error (MSE).
This is an indirect way of assessing the relevance of the discovered
[Figure]
the learned
[Table]
features on the target tasks: we ﬁrst conduct feature selection based on each method and then train a multi-layer perceptron (MLP) to perform predictions on top of the feature-selected data.
Training a separate MLP isolates the effect of having different model selections – such as linear models for
Lasso, ensemble trees for Tree, and deep neural networks for BNNsel, STG and SEFS – and thereby provides a fair comparison of the discovered features.
Experimental Setup.
For each dataset, we divide the available samples into labeled and unlabeled.
Labeled samples are randomly split into training and testing sets.
Within the training set, n l labeled samples from the training set are used to train the feature selection methods and the entire labeled samples are used to evaluate the discovered features.
As a result, while features are selected based on limited sample sizes, a signiﬁcantly larger number of labeled samples are used to validate the discovered features.
Details of the data construction process is given in Appendix C.
Implementation
Implementation details, including hyper-parameters and the construction of covariance matrix R , together with details for all benchmarks can be found in Appendix B
. 5. S YNTHETIC : B LOCK -S TRUCTURED N OISY T WO -M OONS D ATASET
To motivate and conﬁrm our intuition of applying self-supervision and correlated gate vectors in feature selection, we begin by evaluating SEFS via a set of synthetic experiments that addresses two key challenges – i.e. limited labeled data and correlated features – in real-world scenarios.
Dataset Description.
We consider the well-known 2-dimensional “Two-Moons” dataset with additive noise with variance σ 2 = 0 .
The outcome label (i.e., Y = { 0 , 1 } ) is generated solely based on these 2 features (i.e., x 1 and x 2 ).
We augment 8 auxiliary feature dimensions that are irrelevant to target outcomes by sampling from the standard Normal distribution.
Finally, we introduce a correlated block-structure by augmenting each feature with 9 highly-correlated features (correlation: 0.94) by injecting small white noise (i.e., N (0 , 0 .
3 2 ) ), yielding 100 features in total.
Quantitative Analysis.
In Figure 3(a), we compare the average TPR with varying numbers of labeled samples n l = { 10 , 20 , 40 , 60 , 80 } while ﬁxing the number of unlabeled samples n u = 1000 .
We deﬁne x 1 (or x 2 ) as correctly discovered if and only if x 1 (or x 2 ) has the ﬁrst or second highest feature importance.
SEFS beneﬁts from learning the data structure using unlabeled samples and provides signiﬁcant gain over all baselines (see Appendix D.1 for a full comparison).
The majority of benchmarks struggle to identify x 1 in particular due to x 2 being more discriminative than x 1 , and therefore noisy features that are correlated with x 2 are often selected (Table S.4).
We highlight the beneﬁt of the correlated gating procedure by comparing SEFS (no SS) and SEFS (no SS, indep), a variant that uses the independent Bernoulli distribution to generate gate vectors.
Our novel gating procedure signiﬁcantly improves the TPR even in the fully supervised setting (Figure 3(a)).
We further investigate the utility of the Self-Supervision Phase in SEFS.
To this goal, we introduce two counterparts that replace our self-supervised training with (i) a conventional auto-encoder (SEFS (AE)) and (ii) a variant of our model with an uncorrelated gating procedure using independent Bernoulli random variables (SEFS (indep)).
In all cases, we employ the same feature selection process as SEFS to isolate the impact of self-supervision.
In Figure 3(b), we evaluate these variants in terms of the TPR with a varying number of unlabeled samples n u = 200 , 500 , 1000 , 2000 , and 4000 while ﬁxing the number of labeled samples n l = 10 .
(i.e., 5 for each label).
Our method signiﬁcantly outperforms the other variants, improving the TPR over SEFS (no SS) even with a relative small number of unlabeled samples.
Moreover, we further evaluate the quality of the learned representation by assessing the discriminative performance of the learned representations given only the ground-truth features (i.e., setting m 1 = m 2 = 1 and m k = 0 for k / ∈ { 1 , 2 } ).
As shown in Table 1, the proposed gating procedure in SEFS helps the encoder to learn meaningful structure and results in the most performant representations, further validating the self-supervised training procedure.
5. C LINICAL : UKCF D ATASET
[Table]
Dataset Description.
The UK Cystic Fibrosis registry (UKCF) 3 records annual follow-ups for 6,754 adults over the period 2008–2015.
We include patients with observations in at least three consecutive years.
Each patient is associated with p =245 clinical variables ( 11 static and 3 × 78 time-varying features), including demographics, genetic mutations, clinical tests, and therapeutic management.
We set respiratory failure (death or lung transplant) in the next 5 years as the label.
Quantitative Results.
In Table 2, we compare the prediction performance of feature selection methods with n l = 32 , n u = 4754 and |S| = 10 .
As expected,
SEFS beneﬁts from learning the data structure utilizing the unlabeled samples and discovers features that provide signiﬁcantly greater discriminative performance over all other methods (Table 2).
Fully supervised feature selection methods struggle to discover relevant features with only a few labeled samples.
The beneﬁt of self-supervision can be clearly seen by comparing SEFS with SEFS (no
SS).
Our results also highlight that fully unsupervised feature selection methods (i.e., L-Score and
DUFS) struggle to discover relevant features as these methods cannot select features that effectively distinguish the target outcomes without the guidance of label information.
Qualitative Results.
Two interesting features identiﬁed by SEFS but that were not discovered by other methods were PI Allele 1 and 2 that are related to pancreatic functions.
More speciﬁcally, pancreatic function studies have demonstrated that (types of) mutations in the CFTR gene, which causes CF, are highly associated with dysfunction in organ systems and the pathology of CF patients (Sosnay et al., 2013; Gibson-Corley et al., 2016).
Further details of the most frequently discovered features and supporting evidence of their clinical relevance can be found in Appendix D.2.
5. P ROTEOMICS : CCLE D ATASET
provided
[Table]
Dataset Description.
We study the response of heterogeneous cancer cell lines to 11 different drugs where the goal is to identify proteins associated with the cell line response based on proteomic measurements from the Cancer Cell Line Encyclopedia (CCLE, Barretina et al.
CCLE is a small dataset containing 899 cancer cell lines (i.e., samples) described by p =196 protein expressions.
The real-valued drug response (i.e.,
Y = R ) is available for 458 samples and is missing for the remaining 441 samples (thus unlabeled).
To beneﬁt from selfsupervised learning, we integrate the RPPA measurements on 7,329 samples from The Cancer Genome Atlas (TCGA) 4 , creating overall 7,770 unlabeled samples (i.e, n u = 7770 ).
While the two datasets are not generated from the same distribution, we expect some beneﬁt from learning basic correlations among features, even if not all correlations from CCLE are preserved.
Quantitative Results.
In Figure 4, we compare the ranking of SEFS and the benchmarks acrossdrugs ( n l = 20 , n u = 7770 , |S| = 10 ).
SEFS is the best performing method for 9 drugs and is always in the top 3 (median rank: 1, average rank: 1.36).
Despite the majority of unlabeled data originating from a different source, SEFS outperforms SEFS (no SS) in every experiment.
While
[Table]
we would expect further gain from more similar unlabeled data, our results highlight that there is potential beneﬁt even when unlabeled data is only partially related to the labeled samples.
Qualitative Results.
Now we have demonstrated the superior predictive performance of the features discovered using SEFS, we seek to validate these features in the scientiﬁc literature.
Here, we focus our analysis on panobinostat , a histone deacetylase (HDAC) inhibitor used in the treatment of several cancers.
We found strong supporting evidence for 9 of the top 10 ranked features selected by
SEFS (Table 3, see Appendix D.3 for full details).
In addition, the remaining feature (ﬁbronectin) is strongly implicated in cancer, but we did not ﬁnd literature speciﬁcally relating it to panobinostat or
HDAC-mediated pathways.
We provide supporting evidence for other drugs in Appendix D.3.
5. T RANSCRIPTOMICS : PBMC D ATASET
Dataset Description.
We focus on distinguishing sub-populations of T-cells (i.e., Y = { 0 , 1 } ), namely naive and regulatory T-cells, from puriﬁed populations of peripheral blood monocytes (PBMCs) 5 based on transcriptomic measurements (i.e., mRNA sequence).
The PBMC dataset consists of 20,742 samples described by p =19256 protein-coding genes.
We primarily used n l = 20 , n u = 15557 , with the remaining 5,165 labeled samples used to evaluate the discovered features.
Quantitative and Qualitative Results.
Similarly to the performance on other datasets, SEFS signiﬁcantly outperforms all benchmarks (Table 4).
Figure S.4 displays only features selected with a frequency ≥ 0 .
It is immediately apparent that only SEFS consistently identiﬁes the same features, while Lasso and Tree typically select different features, demonstrating the robustness of our approach.
We provide supporting evidence for the importance of these features in Appendix D.4.
C ONCLUSION
In this paper, we proposed SEFS, a self-supervised feature selection framework that is able to leverage the abundant quantities of unlabeled data that are often available, achieving state-of-the-art performance when limited labeled data is available.
Using synthetic data, we motivate and conﬁrm our intuition as to why self-supervision and our novel gating procedure can beneﬁt feature selection methods.
Through experiments on multiple real-world datasets, we validate that our model discovers features that provide superior prediction performance and corroborate the vast majority of the discovered features with supporting medical and scientiﬁc literature.
As with all feature selection methods, the discovered features based on SEFS should be further experimentally veriﬁed or evaluated by a domain expert prior to deployment in practice.
R EPRODUCIBILITY S TATEMENT
The source code of SEFS and the synthetic dataset are provided in the Supplementary Material for reproducibility.
All details of our method and the experimental setup are included in either the main manuscript or the Appendix: a description of the method can be found in Section 4, pseudo-code for
SEFS is described in Appendix A, Appendix B provides implementation details for both SEFS and the benchmarks, Appendix C provides details about the real-world datasets.
A P SEUDO -C ODE OF SEFS
SEFS is trained via a two-step training procedure.
We provide pseudo-codes for the Self-Supervision
Phase in Algorithm 1 and for the Supervision Phase in Algorithm 2.
Algorithm
Input: Dataset D u
= { x i } n u i =1 , coefﬁcient α , selection probability π , mini-batch size n mb , learning rate η
Output: SEFS parameter θ
Initialize parameters ( θ, ψ 1 , ψ 2 )
Compute the mean value: ¯ x = 1 n u (cid:80) n u i =
1 x i
Compute the correlation matrix: R where R kj = | C kj | √ C kk · C jj and C kj
=
(cid:80) nui
=1 ( x ik − ¯ x ik )( x ij − ¯ x ij ) n u −repeat
Sample a mini-batch of n mb unlabeled data samples: { x i } n mb i =1 ∼ D u for i = 1 , · · · , n mb do
Sample gate vector: m i ∼ MultiBern ( π ; R )
Generate feature subset: ˜ x
i ← m (cid:12) x
i + (1 − m ) (cid:12) ¯ x
Estimate feature vector: ˆ x i ← h ψ 1 ◦ f θ (˜ x i )
Estimate gate vector: ˆ m i ← h ψ 2 ◦ f θ (˜ x i ) end for
Update the encoder parameter θ : θ ← θ − η ∇ θ (cid:32)n mb n mb (cid:88)
i = (cid:96) X
( x i , ˆ x i ) + α · (cid:96) M
( m i , ˆ m i ) (cid:33)
Update the feature vector estimator parameter ψ 1 : ψ 1 ← ψ 1 − η ∇ ψ(cid:32)n mb n mb (cid:88) i
= (cid:96) X
( x i , ˆ x i ) (cid:33)
Update the gate vector estimator parameter ψ 2 : ψ 2 ←
[Equation]
−
[Equation]
∇ ψ(cid:32)n
[Equation]
n mb (cid:88) i = α · (cid:96) M (
[Equation]
i ) (cid:33) until convergence
B I MPLEMENTATION D ETAILS
The hyper-parameters of SEFS and those of the benchmarks are chosen via a grid search.
For all methods, we use 20% of the overall training set as the validation set, which will then be unseen for training feature selection methods with chosen hyper-parameters.
Once feature selection methods are trained utilizing a small subset of the overall training set as illustrated in Figure S.1, we train MLPs (with 3 layers and 100 nodes) with feature-selected data.
As described in Section 5, training a separate MLP isolates the effect of having different model selections and thereby provides a fair comparison among the discovered features.
It is worth highlighting that generating correlated gate vectors can be computationally burdensome for high-dimensional data due to the matrix multiplication in Algorithm 3 and 4.
To avoid such an issue, we ﬁrst apply thresholding (e.g. 0.7 for the PBMC dataset) to the correlation matrix R , group features based on the agglomerative clustering 6 using the correlation matrix as the similarity measure, and then conduct block-wise matrix multiplication.
Overall, the generated gates for features
Input: Dataset D l = { x i , y i } n l i =1 , pre-trained encoder θ , coefﬁcient β , mini-batch size n mb , learning rate η
Output: SEFS parameters ( θ, φ, π )
Initialize parameters ( φ, π )
Compute the mean value: ¯ x = 1 n u (cid:80) n u i =
1 x i
Compute the correlation matrix: R where R kj = | C kj | √ C kk · C jj and C kj
=
(cid:80) nui
=1 ( x ik − ¯ x ik )( x ij − ¯ x ij ) n u −repeat
Sample a mini-batch of n mb labeled data samples: { x i , y i } n mb i =1 ∼ D l for i = 1 , · · · , n mb do
Sample gate vector:
˜ m i ∼ Relaxed-MultiBern ( π ; R )
Generate feature subset: ˜ x i ← ˜ m (cid:12) x i + (1 − ˜ m ) (cid:12) ¯ x
Predict outcome given the feature subset: ˆ y i ←
f φ ◦ f θ (˜ x i ) end for
Fine-tune the encoder parameter θ : θ ← θ − η ∇ θ (cid:32)n mb n mb (cid:88) i
= (cid:96)
Y
( y i , ˆ y i ) (cid:33)
Update the predictor parameter φ : φ ← φ − η ∇ φ (cid:32)n mb n mb (cid:88) i
= (cid:96)
Y
( y i , ˆ y i ) (cid:33)
Update the selection probability parameter π : π ← π − η ∇ π (cid:32)n mb n mb (cid:88) i = (cid:96) Y
( y i , ˆ y i )
+ β p (cid:88) k
= π k (cid:33) until convergence
Input: selection probability π , correlation matrix R
Output: correlated gate vector m
Draw a standard normal sample: (cid:15) ∼ N (0 , I )
Compute L = Cholesky-Decomposition ( R )
Generate a multivariate Gaussian vector v = L (cid:15) for k = 1 , · · · , p do
Apply element-wise Gaussian CDF:
u k = Φ( v k )
Generate correlated gate: m k = 1 ( u k ≤ π k ) end for within the same group will maintain the correlation structure.
This signiﬁcantly reduces the computational complexity in Algorithm 3 and 4 as it scales quadratically with respect to the largest block size (i.e., the number of features grouped in the largest block). Thus, if the largest block size remains the same, the complexity of generating the correlated gate vectors will only increase linearly with the feature dimension (since the number of blocks would increase linearly).
Throughout the experiments, training SEFS takes approximately 30 minutes to 1 hour for each phase (similar to that of STG) on a single GPU machine 7 .
Input: selection probability π , correlation matrix R , temperature τ
Output: correlated gate vector ˜ m
Draw a standard normal sample: (cid:15) ∼ N (0 , I )
Compute L = Cholesky-Decomposition ( R )
Generate a multivariate Gaussian vector v = L (cid:15) for k = 1 , · · · , p do
Apply element-wise Gaussian CDF:
u k = Φ( v k )
Apply reparameterization trick (Maddison et al., 2017):
[Equation]
where σ ( x ) = 1 1+exp( − x ) .
end for
B. SEFS
The overall training process of SEFS consists of 4 different network components that are implemented as neural networks: (i) encoder f θ , (ii) predictor f φ , (iii), feature vector estimator h φ 1 , and (iv) gate vector estimator h φ 2 .
We use a fully-connected network as the baseline architecture for all the network components.
For the Semi-Supervision Phase, coefﬁcients ( α , π ) and the hyper-parameters – including the number of hidden units, the number of nodes, and the number of layers – of the encoder, the feature vector estimator, and the gate vector estimator are chosen utilizing a grid search.
We choose hyperparameter values that achieve the lowest validation loss.
For the Supervision Phase, coefﬁcient β and the hyper-parameters – including the number of nodes and the number of layers – of the predictor are chosen utilizing a grid search (note that the hyperparameters of the encoder is ﬁxed after the Semi-Supervision Phase).
We choose hyper-parameter values that achieve the best validation performance by using the outputs of the predictor (i.e., ˆ y =
f φ ◦ f θ ( x ) ).
The permitted values of coefﬁcients ( α , β , π ) and the hyper-parameters are listed in Table S.
[Table] [Table]
B. B ENCHMARKS
Throughout the experiments, we compared SEFS with feature selection methods ranging from conventional approaches – such as Lasso (Tibshirani, 1996) and Tree (Geurts et al., 2006) – to the state-of-the-art approaches – such as STG (Yamada et al., 2020) and DUFS (Lindenbaum et al., 2020).
Further descriptions of the benchmarks and implementation details are as follows:
• Lasso 8 : Lasso is a well-known embedded method whose objective is to minimize the prediction loss (i.e., cross-entropy loss for classiﬁcation tasks and mean squared error for regression tasks) while enforcing the l 1 -penalty to achieve sparsity among input features (Tibshirani, 1996).
We implement Lasso regression for regression tasks and Logistic regression with l 1 -penalty for classiﬁcation tasks.
The hyper-parameter to control the input sparsity is chosen from { 0.001, 0.01, 0.1, 1, 10, 100 } .
• Tree 9 : ExtraTree (Geurts et al., 2006) is an ensemble of decision trees that has become a popular choice for classiﬁcation and regression tasks for tabular data.
We use the feature importance obtained by ExtraTree to conduct feature selection.
The number of trees and the maximum depth are chosen from { 10, 50, 100, 300, 500 } and { 1, 2, 3, 4 } , respectively.
• L-Score 10 : The Laplacian Score (L-Score)
(He et al., 2005) is an unsupervised (ﬁlter) feature selection method that quantiﬁes the importance of input features by the ability to preserve local structure of the data, which is captured by the Laplacian matrix.
We further modify the afﬁnity score between a pair of labeled samples following the description in Sheikhpour et al.
(2017), i.e., the afﬁnity score becomes 1 if the two samples have the same label and 0 otherwise.
We construct the Laplacian matrix based on the Gaussian kernel using both labeled and unlabeled samples with 10 -nearest neighbors and setting the kernel bandwidth as the median Euclidean distance.
• BNNsel 11 : BNNsel is an embedded feature selection method based on Bayesian neural networks that overcomes the non-differentiability in the selection process by utilizing MCMC sampling (Liang et al., 2018).
The number of hidden units is chosen among { 3 (default value), 5, 10, 30, 50 } with a ﬁxed prior probability 0.025 (default value).
• STG 12 : STG is an embedded feature selection method using deep neural networks that overcomes the non-differentiable process of selecting feature subsets via continuous relaxation using Gaussian random variables (Yamada et al., 2020).
We use a fully-connected network as the baseline architecture where the number of nodes and the number of layers are chosen among { 10, 30, 50, 100, 300, 500 } and { 1,2,3,4,5 } , respectively.
The hyper-parameter to control the sparsity of input features is chosen among { 0.001, 0.01, 0.1, 1, 10, 100 } .
• STG (SS):
We propose an extension of STG to the semi-supervised setting.
To this goal, we introduce a reconstruction task in addition to the prediction task of the original STG.
More speciﬁcally, we utilize an auxiliary network, i.e., decoder, which reconstructs the original input features given the gated features by minimizing the reconstruction loss.
The overall network is trained based on the combination of the prediction loss and the reconstruction loss.
(Here, we adopt a hyper-parameter α ∈ { 1 ., 0 . 1 , 0 . 01 , 0 . 001 } to balance the two losses).
We use a fullyconnected network as the baseline architecture where the number of nodes and the number of layers are chosen among { 10, 30, 50, 100, 300, 500 } and { 1,2,3,4,5 } , respectively.
The hyperparameter to control the sparsity of input features is chosen among { 0.001, 0.01, 0.1, 1, 10, 100 } .
• DUFS 13 : DUFS is an unsupervised feature selection method that improves the Laplacian score (He et al.,
2005) by replacing it with a gated variant computed on a subset of features (Lindenbaum et al., 2020).
We use the parameter-free version of DUFS which construct the Laplacian matrix with 2 -nearest neighbors and setting the kernel bandwidth as the median Euclidean distance.
Both labeled and unlabeled samples are used for training
[Table]
[Figure]
C D ETAILED D ATA D ESCRIPTIONS
In Section 5 of the manuscript
[Figure]
we evaluate SEFS and the benchmarks with multiple healthcare datasets from three different sources, i
[Figure]
e., UKCF (clinical), CCLE (proteomics), and PBMC (transcriptomics).
The summary statistics of these datasets are provided in Table S.2.
Throughout the experiments on the real-world healthcare and omics datasets, we consider a practical scenario for feature selection under low labeled data regime where only a small number of labeled samples are available while having a sufﬁcient amount of unlabeled sample.
To this goal, we construct datasets based on the following steps: First, we obtain enough unlabeled samples by utilizing samples without labels that are readily present in the original data (e.g., UKCF dataset has censored patients and CCLE dataset has samples without drug response), treating a portion of labeled samples as unlabeled (e.g., UKCF and PBMC datasets), and by integrating samples from different sources (e.g., CCLE dataset).
Second, we randomly split the labeled samples into training and testing sets.
Here, both the training and the testing sets are used for evaluating discovered features from different feature selection methods.
This assures that the discovered features generalize well to unseen samples.
Third, we randomly select a small subset of training samples for training different feature selection methods.
The overall data construction steps are illustrated in Figure S.1.
Correlation Structure.
The correlation structure of the datasets are illustrated in Figure S.2.
We construct the correlation matrix R as deﬁned in Algorithm 1 and 2: formally, each element of R is given as R kj =
[Equation]
where
[Equation]
It is worth highlighting that the PBMC dataset contains many features that have low correlation with other features; thus, after thresholding at 0.7, only 426 features with high correlation remain.
To quantify the amount of collinearity, we calculated the variance inﬂation factor (VIF), which is an index that measures how much the variance (the square of the estimate’s standard deviation) of an estimated regression coefﬁcient is increased because of collinearity.
While there is not a strict cut-off indicating collinearity, a general rule of thumb is that VIFs exceeding 10 are signs of serious multicollinearity (Menard, 2001; Vittinghoff et al., 2011; James et al., 2013).
We note this is a conservative cut-off, with guidelines suggesting that VIFs greater than 5 (Menard, 2001; Vittinghoff et al., 2011) indicate considerable collinearity.
In Table S.3, we provide the VIFs of features in each dataset; here the VIFs for the PBMC dataset are not available due to the computational complexity.
As seen in the table, there are strong signs of multicollinearity in both datasets with the VIFs for 23.5% of the features in the UKCF dataset and all the features in the CCLE dataset exceeding 10.
[Table]
D A DDITIONAL E XPERIMENTS D. S YNTHETIC : B LOCK -S TRUCTURED N OISY T WO -M OONS D ATASET
Quantitative Results.
Table S.4 shows the average TPRs for all benchmarks.
As can be seen in the table, SEFS outperforms all benchmarks, with the majority of methods struggling to identify Feature 1 in particular.
This is due to Feature 2 being more discriminative than Feature 1 and therefore the noisy features that are correlated with Feature 2 are often selected.
It is worth highlighting that
L-Score, DUFS
[Table]
and STG (SS) fail to discover discriminative features in almost all cases due to the correlation structure of the data that cannot be addressed by the similarity metrics or reconstruction employed by these methods.
[Table]
D. C LINICAL : UKCF D ATASET
Quantitative Results.
Figure S.3 displays features discovered with a frequency ≥ 0.4, where we consider a feature is discovered (for all three feature selection methods) if that feature is within the top 20 highest feature importance.
It is immediately apparent that SEFS consistently identiﬁes the same features, while Lasso and Tree typically select different features, demonstrating the robustness of our approach.
Qualitative Results.
Other than PI Allele 1 and 2 that are related to pancreatic functions as described in Section 5.2, SEFS consistently discovered features related to the lung function scores (i.e., FEV 1 % pred and FEV 1 ).
It is worth highlighting that, features related to the lung function scores are frequently selected despite their relatively high correlation.
We conjecture that the trajectory of the lung functions scores measured at different times (with the highest frequency of the latest measurement) plays an important role for predicting the respiratory failure as supported in
Taylor-Robinson et al.
; Adler & Liou (2016).
D. P ROTEOMICS : CCLE D ATASET
Quantitative Results.
In Table S.5, we compare the prediction performance of 5 and 10 discovered features (i.e., |S| = 5 and 10 ) for the 11 drugs – AZD6244, Erlotinib, Irinotecan, L-685458,
Lapatinib, PD-0325901, PF2341066, PLX4720, Paclitaxel, Panobinostat, and Topotecan – reported in the manuscript.
Despite the majority of unlabeled data originating from a different source, our method beneﬁts from learning the underlying data structure from unlabeled samples.
SEFS consistently displays improvements (except for Erlotinib with |S| = 5 and 10 , L-685458 with |S| = 5 , and PF2341066 with |S| = 5 and 10 ) in performance for different drugs and varying numbers of discovered features, outperforming SEFS (no SS) in every experiment.
While we would expect further gain from more similar unlabeled data, our results highlight the effectiveness of self-supervision even when the unlabeled data is only partially related to the labeled samples.
Qualitative
Results.
In the main manuscript, we validated the features identiﬁed by SEFS for panobinostat in the scientiﬁc literature.
In Table S.6, we provide details of the references for panobinostat and also provide supporting evidence for the selected features for lapatinib and irinotecan.
Lapatinib is a drug used primarily to treat breast cancer, as well as other solid tumours.
It is a dual tyrosine kinase inhibitor which inhibits the epidermal growth factor receptor (EGFR) and human epidermal growth factor receptor 2 (HER2) receptors.
In total, we found supporting literature forof the top 10 ranked features selected by SEFS (Table S.6b), with ﬁve such features not proposed by SEFS (no SS).
The other two features selected by SEFS are both established cancer biomarkers with evidence of their importance in breast cancer, but we did not ﬁnd literature speciﬁcally relating them to Lapatinib.
Irinotecan is a chemotherapy agent used primarily to treat colon cancer in addition to small cell lung cancer.
All of the top 10 features identiﬁed by SEFS had supporting scientiﬁc literature for their impact on the mechanism and effectiveness of irintecan or colon cancer (Table S.6c), of which six were not proposed by SEFS (no SS).
[Table]
S
[Table]
S
[Table]
S|
[Table]
S
[Table]
S
[Table]
D. T RANSCRIPTOMICS : PBMC D ATASET
Qualitative Results.
In Figure S.4, we compare the frequency of the discovered features based on
SEFS with that of the discovered features based on Lasso and Tree.
The ﬁgure displays only the features that were selected with a frequency equal to or greater than 0.4 by one of the methods.
It is worth highlighting that SEFS (no SS) does not discover any feature with frequency equal to or greater than 0.4, which shows the importance of the Self-Supervision Phase.
The lack of consistency of discovering features suggests that Lasso and Tree could be overﬁt to spurious relations in different splits of the data.
Only 2 features are discovered at least 40% of the time by either Lasso or Tree (one feature each), compared to 9 for SEFS.
SEFS almost always discovers S100A4 and IL32 as relevant features.
Both of these genes have been shown to be only expressed, or preferentially expressed, in certain types of T-cells – S100A4 :
Weatherly et al.
(2015) and IL32 : Goda et al.
(2006) – which strongly validates the selection of these features.
These features are also the most frequent features discovered by Tree, but with much lower frequency, while Lasso does not discover either feature, despite their high relevance.
Similarly, while DUSP1 (also known as MKP-1 ) is selected by Lasso, Tree, and SEFS, only our method discovers this feature in the majority of the experiments, despite its importance for T-cell activation and function (Zhang et al., 2009).
In addition, we also ﬁnd evidence of the importance of JUN (Riera-Sans & Behrens, 2007),
SH3BGRL
(Deng et al., 2006), CCR7 (Schneider et al., 2007), and ribosomal genes RPS13 , RPL21 , and RPS3A (Procaccini et al., 2016).
[Table]
[Figure]
D. C ORRELATED VS I NDEPENDENT G ATE V ECTOR G ENERATION IN F EATURE S ELECTION
In this subsection, we investigate how the correlated gate vectors using a multivariate Bernoulli distribution in SEFS improves the feature selection process over independently generated gate vectors.
To this goal, in Figure S.5, we provide trajectories of selection probabilities for the ground-truth feature x 2 and its block-correlated noisy features with respect to the number of training iterations.
Note that the results are generated only utilizing the Supervision Phase.
As claimed in the manuscript, the correlated gate vectors encourage the network to select only the most relevant feature by making multiple correlated features compete against each other.
This is highlighted in Figure S.5.
However, when independent gate vectors are used, it is less likely to make correlated features to compete each other and increase the selection probabilities of the correlated noisy features because these features are still informative about the target compared to the augmented features – i.e., ( x 3 , · · · , x 10 ) – and their block-correlated features that are purely noisy.
[Figure]
[Figure]
D. S ENSITIVITY A NALYSIS ON C OEFFICIENT β
In this subsection, we provide sensitivity analysis using the Block-Structured Noisy Two-Moons dataset to see the effect of coefﬁcient β that controls the sparsity of the selected features.
Figure S. shows (a) the true discovery rate as deﬁned in Section 5.1 – where we deﬁne x 1 (or x 2 ) as correctly discovered if and only if x 1 (or x 2 ) has the ﬁrst or the second highest feature importance – and (b) the number of features discovered whose selection probability is higher than 0.5 (i.e., π k > 0 . 5 ).
Here, multiple instances of SEFSwith different β ’s are trained utilizing n l = 20 labeled samples (i.e., 10 labeled samples for each label) and pre-trained with n u = 1000 .
The results are averaged over 100 iterations as described in Section 5.1 of the manuscript.
Figure S.6(a) shows that SEFS trained with setting β = 5 , 10 provides the highest TPR.
However, as can be seen in Figure S.6(b), setting the coefﬁcient too small (here, β ≤ 1 ) will end up discovering too many irrelevant features as if they were relevant – i.e., having many features whose selection probability is above a certain level – and will mislead the discovery of relevant features.
Contrarily, setting the coefﬁcient too high (here, β = 10 ) can restrict the selection process and may fail to identify relevant features.