Composite Feature Selection Using Deep Ensembles
Anonymous Author(s)
AffiliationAddress email
Abstract
In many real world problems, features do not act alone but in combination witheach other.
For example, in genomics, diseases might not be caused by any singlemutation but require the presence of multiple mutations.
Prior work on featureselection either seeks to identify individual features or can only determine relevantgroups from a predefined set.
We investigate the problem of discovering groupsof predictive features without predefined grouping.
To do so, we define predictivegroups in terms of linear and non-linear interactions between features.
We introducea novel deep learning architecture that uses an ensemble of feature selection modelsto find predictive groups, without requiring candidate groups to be provided.
Theselected groups are sparse and exhibit minimum overlap.
Furthermore, we proposea new metric to measure similarity between discovered groups and the ground truth.
We test our model on multiple synthetic tasks, semi-synthetic chemistry datasetsand image datasets to demonstrate its utility
Introduction
Feature selection is a key problem permeating statistics, machine learning and broader science.
Typically in high-dimensional datasets, the majority of features will not be responsible for the targetresponse and thus an important goal is to identify which variables are truly predictive.
For example,in healthcare there may be many features (such as age, sex, medical history, etc.)
that could beconsidered, while only a small subset might in fact be relevant for predicting the likelihood ofdeveloping a specific disease.
By eliminating irrelevant variables, feature selection algorithms can beused to drive discovery, improve model generalisation/robustness, and improve interpretability [16].
However, features often do not act alone but instead in combination .
In genetics, for instance, it hasbeen noted that understanding the origins of many diseases may require methods able to identify morecomplex genetic models than single variants [57].
While feature selection might be able to identify aset of features associated with a particular response, the underlying structure of how features interactis not captured.
Further, the resulting predictive models can be complex, hard to interpret, and notamenable to the generation of hypotheses that can be experimentally tested [41].
This limits theimpact such models can have in furthering scientific understanding across many domains wherevariables are known to interact, such as genetics [57, 61, 54], medicine [79, 11], and economics [7].
Group feature selection is a generalisation of standard feature selection, where instead of selectingindividual features, groups of features are either entirely chosen or entirely excluded.
A primaryapplication of group feature selection is when features are jointly measured, for example by differentinstruments.
In such scenarios, groups are readily defined as features measured by the same instrument.
A natural question is which instruments give the most meaningful measurements.
Group featureselection has also been applied in situations where there is extensive domain knowledge regarding the group structure [64] or where groups are defined by the correlation structure between features (e.g.neighbouring pixels in images are highly correlated).
The pervasive issue with current group featureselection methods is that a predetermined grouping must be provided, and the groups are selectedfrom the given candidates.
In reality, we may not know how to group the variables.
In this paper we seek to solve a related but ultimately different and more challenging problem, whichwe call Composite Feature Selection .
We wish to find groups of variables without prior knowledge,where each group acts as a separate predictive subset of the features and the overall predictive poweris greatest when all groups are used in unison.
We call each group of features a composite feature.
By imposing this structure on the discovered features, we attempt to isolate pathways from features tothe response variable.
Discovering groups of features offers deeper insights into why specific featuresare important than standard feature selection.
Contributions.
(1) We formalise composite feature selection as an extension of standard featureselection, defining composite features in terms of linear and non-linear interactions between variables(Sec. 3).
(2) We propose a new deep learning architecture for composite feature selection using anensemble-based approach (Sec. 4).
(3) To assess our solution, we introduce a metric for assessingcomposite feature similarity based on Jaccard similarity.
(4) We demonstrate the utility of our modelon a range of synthetic and semi-synthetic tasks where the ground truth group features are known(Sec.
We see that our model not only frequently recovers the relevant features, but also oftendiscovers the underlying group structure.
Related Work
Significant attention has been placed on feature selection (see Appendix B for further discussion ofstandard feature selection), and several approaches have been extended to select predefined groupsinstead of individual features.
For example, LASSO [67, 73] is a linear method that uses an L1 penaltyto impose sparsity among coefficients.
Group LASSO [82] generalises this to allow predefined groupsto be selected or excluded jointly, rather than single features, by replacing the L1 penalty with L2penalties on each group.
Other feature selection methods, such as SLOPE [10], have been similarlyextended to group feature selection to give Group-SLOPE [12].
Further examples of group featureselection using adapted loss functions are SCAD-L2 [84] and hierarchical LASSO [89].
Similarly,
Bayesian approaches to feature selection [25] have also been generalised to the group setting [31].
Finally, the Knockoff procedure [8, 15, 35, 51, 65, 70] is a generative procedure that creates fakecovariates (knockoffs), obeying certain symmetries under permutations of real and knockoff features.
By subsequently carrying out Feature Selection on the combined real and knockoff data, there areguarantees on the False Discovery Rate.
Generalisations of the Knockoff procedure to the groupsetting also exist [21, 90], where symmetries under permutations of entire groups must exist.
The key commonality is that none of these methods discover groups, but instead can only selectgroups from a set of predefined candidates.
Therefore, while they may be applicable when we cansplit inputs into groups, they are not able to find groups of predictors on their own.
Our work differsfrom these methods by considering the challenge of finding such groups in the absence of priorknowledge.
Additionally, unlike prior work, we do not make assumptions about correlations betweenfeatures or restrictions on groups, such as requiring that the candidate groups partition the features.
Problem Description
Let X ∈ X p be a p -dimensional signal (such as gene expressions or patient covariates) and Y ∈ Y bea response (such as disease traits).
Informally, we wish to group features into the maximum numberof subsets, G i ⊂ [ p ] , where the predictive power of any single group significantly decreases when anyfeature is removed, allowing us to separate the groups into different pathways from the signal to theresponse.
Note that we do not enforce assumptions on the groups such as non-overlapping groups or every feature being in at least one group.
In this section, we begin with a description of traditionalfeature selection before formalizing composite feature selection.
3.1 Feature Selection
The goal of traditional feature selection is to select a subset, S ⊂
[ p ] , of features that are relevant forpredicting the response variable.
In particular, in the case of embedded feature selection [28], this isconducted jointly with the model selection process.
Let ∗ denote any point not in X and define X S =
( X ∪ {∗} ) p .
Then, given X ∈ X p , the selectedsubset of features can be denoted as X S ∈ X S where x S ,k
= x k if k ∈ S and x S ,k
= ∗ if k / ∈ S .
Let f : X S
→ Y be a function in some space F (such as the space of neural networks) taking subset
X S as input to yield Y .
Then, selecting relevant features for predicting a response can be achievedby solving the following optimization problem:
[Equation]
where δ constrains the number of selected features and ℓ Y ( y, y ′ ) is a task-specific loss function.
This can be solved by introducing a selection vector M =
( M 1 , · · · , M p ) ∈ { 0 , 1 } p , consisting ofbinary random variables governed by distribution p M , with realization m indicating selection of thecorresponding features.
Then, the selected features given vector m can be written as
[Equation]
where ⊙ indicates element-wise multiplication and ˆ x are the values assigned to features that are notselected (typically ˆ x ≡ 0 or ¯ x ).
(1) can be (approximately) solved by jointly learning the model fand the selection vector distribution p M based on the following optimization problem:
[Equation]
where β is a balancing coefficient that controls the number of features to be selected.
3.2 Composite Feature Selection
The goal of composite feature selection is to not only find the predictive features, but also to groupthem based on how they are predictive.
For example, assume features x 1 and x 2 are only predictivewhen both are known by the model, but make the same prediction independent of x 3 .
Then wewish to group x 1 , x 2 from x 3 .
In this section, we define the embedded composite feature selectionproblem; that is, we want to find a valid model f and groups {G 1 , . .
. , G N } in parallel.
A model isonly valid when the group representations are combined in a way where we can view each group ascontributing an independent piece of information for the final prediction.
A valid model acts on aset of groups [83], thus when combining groups, we require order not to matter.
Therefore, we mustcombine the representations using a permutation invariant aggregator.
Let A : ( (cid:81) i R n ) −→ R N be a general permutation invariant aggregation function.
It is well establishedthat for a specific choice of ϕ : R n −→ R m and ρ : R m −→ R N , A can be decomposed as ρ [ (cid:80) i ϕ ( · )]
(see [83] for examples).
This gives f ( x ) =
g (cid:0) ρ (cid:2) (cid:80) i ϕ
( f i ( x G i ))
(cid:3)(cid:1) , where f i encodes group i , ρand ϕ give the permutation invariant aggregation and g is any final non-linear function, softmax forinstance.
The function composition of ϕ and f i can be relabelled as ˜ f i =
ϕ ◦ f i , and the compositionof g and ρ can be relabelled as ˜ ρ
=
g ◦ ρ .
This leads to f ( x ) =
˜
ρ (cid:2) (cid:80) i ˜
f i ( x G i ) (cid:3) , this gives a definitionfor what a valid model structure can be in composite feature selection.
Definition 3.1.
The most general valid model for acting on N composite features is given by:
[Equation]
That is, the groups must interact only once, all groups must be included and the interaction is asummation; all other interactions can (and often should) be non-linear.
Depending on the task, a specific permutation invariant aggregation may be chosen (e.g. Max() ).
However, any permutation invariant aggregator can be (approximately) expressed in the form of Def.3.1; thus, when learning from data, the general structure of Def.
3.1 means that this is not necessary.
The embedded composite feature selection problem can now be phrased in an analogous way totraditional feature selection.
Let ∗ denote some point not in X and define X G i =
( X ∪ {∗} ) p .
Then,given X ∈ X p , the selected group of features is denoted as X G i ∈
X G i where x G i ,k = x k if k ∈ G iand x k = ∗ if k / ∈ G i .
Let f i : X
G i
→ Z be a function in F that takes as input the subset X G i andoutputs a latent representation z i .
Then, finding the groups of features can be achieved by solving theoptimization problem:
[Equation]
subject to |G
[Equation]
≤
[Equation] [Equation]
where δ constrains the number of selected features in each group and ∆ gives the minimum numberof groups.
This objective leads to multiple smaller groups, rather than one group containing allfeatures, which is consistent with our motivation of the problem.
Continuing to expand from traditional feature selection, we can also extend the solution to thecomposite setting.
For N groups we can introduce a selection matrix M ∈ { 0 , 1 } N × p , governed bydistribution p M .
For a realization m , the selected features from group i are given by
[Equation]
(6) where m i is the i th row of M .
We can approximately solve (5) by solving the optimization problem:minimize
[Equation]
where f ( x ) obeys Def.
(4) and R e is a regularisation term which controls how features are selectedin each group.
R e should capture both group size (i.e. encourage as few features as possible to beselected) but also the relationships between groups (i.e. groups should be distinct and not redundant).
3.3 Challenges
There are various challenges in solving the composite feature selection problem.
While the ultimatetask is to find predictive groups of features, there remains the necessity simply to identify predictivefeatures, which is already an NP-hard problem [2].
Composite feature selection not only inheritsthis property but introduces additional complexity since we can think of each group as solving aseparate feature selection problem.
Consider the number of potential solutions: in traditional featureselection (assuming not all features are selected), there are 2 n − 2 ways of selecting a subset from nfeatures; even restricting to at most m << n quickly becomes unfeasible for even modest valuesof m .
In composite feature selection, every group has the same number of solutions as traditionalfeature selection, drastically increasing the total number of possible solutions.
A challenge specificto composite feature selection arises when the ground truth group structure contains groups withoverlapping features (e.g. feature x 1 interacts independently with both x 2 and x 3 ).
In this scenario, itis difficult to separate these two effects while penalizing the inclusion of additional features.
Method: CompFS
In this section, we propose a novel architecture for finding predictive groups of features, which werefer to as Comp osite F eature S election (CompFS).
In order to discover groups of features, our modelis composed of a set of group selection models and an aggregate predictor.
Our approach resemblesan ensemble of “weak” feature selection models, where each learner attempts to solve the task usinga sparse set of features (Figure 1).
These models are then trained in such a way as to discover distinctpredictive groups.
We first consider the group selection models in more detail before describing howthe group selection models are combined and the training procedure.
GroupEncoder ( 𝒇 𝜽 𝒊 )
Group Predictor ( 𝒉 𝝋 𝒊 )
Predictor ( 𝒉 𝝋 ) 𝐗 𝛑 𝒊 ෩𝐗 𝒊 ෥𝐦 𝐢
Group Generator
S a m p l e ෝ𝒚 𝒊 𝒛 𝒊
GroupEncoder ( 𝒇 𝜽 𝒋 )
Group Predictor ( 𝒉 𝝋 𝒋 ) 𝐗 𝛑 𝒋 ෩𝐗 𝒋 ෥𝐦 𝐣
Group Generator
S a m p l e ෝ𝒚 𝒋 𝒛 𝒋 ...
ෝ𝒚
G a t e
G a t e
4.1 Group Selection Models
CompFS is composed of a set of group selection models, each of which primarily aims to solve thetraditional feature selection problem specified by (1).
We achieve this by solving (3) using a neuralnetwork-based approach with stochastic gating of the input features.
Each group selection modelconsists of the following three components (Figure 1):
• Group Selection Probability , π i =
( π 1 ,i , · · · , π p,i ) ∈ [0 , 1] p , which is a trainable vector thatgoverns the Bernoulli distribution used to generate the gate vector m i .
Each element of the
selection probability π k,i indicates the importance of the corresponding feature to the target.
• Group Encoder , f θ i : X p
→ Z , that takes as input the selected subset of features ˜ x i and outputslatent representations z i ∈ Z .
• Group Predictor , h ϕ
i : Z → Y , that takes as input the latent representations of the selected subsetof features, z i =
f θ i
(˜ x i ) , and outputs predictions on the target outcome.
Solving (3) directly is not possible since the sampling step has no differentiable inverse.
Instead, weuse the relaxed Bernoulli distribution [53, 34] and apply the reparameterization trick as follows.
Formally, given selection probability π = ( π 1 , · · · , π p ) and independent Uniform (0 , 1) randomvariables ( U 1 , · · · , U p ) , we can generate a relaxed gate vector ˜ m
= ( ˜ m 1 , · · · , ˜ m p ) ∈ (0 , 1) p basedon the following reparameterization trick [53]:
[Equation]
where σ ( x ) = (1 + exp( − x )) − 1 is the sigmoid function.
This relaxation is parameterized by π andtemperature τ ∈ (0 , ∞ ) .
Further, as τ → 0 , the gate vectors ˜ m k converge to Bernoulli ( π k ) randomvariables.
Crucially this is differentiable with respect to π .
Given group selection probability π i , we first sample relaxed Bernoulli random variable ˜
m i accordingto (8) and then use ˜ m i in a gating procedure to select the group of features.
The output of the gate is:
[Equation]
where we replace the variables that were not selected by their mean value ¯ x .
The mean is used becausein particular tasks a feature having a value of 0 may be particularly meaningful.
The gate output ˜ x
i isthen fed into the group encoder f
θ i to yield representation
z i =
f θ i
(˜ x i ) .
This representation is finallypassed to the group predictor h ϕ i to produce the prediction for an individual learner, ˆ y i
=
h ϕ i ( z i ) .
4.2 Group Aggregation
The final component necessary for CompFS is a way to aggregate the individual group selectionmodels.
This is achieved via an overall predictor , h ϕ : Z → Y , that takes as input the set of latentrepresentations { z 1 , . . .
, z N } produced by the individual learners and outputs predictions on thetarget outcome.
For simplicity, we apply a linear prediction head to the latent representations and useelement-wise summation to aggregate.
Thus, the prediction of the ensemble is given by:
[Equation]
where N is the number of members of the ensemble and ρ is a suitable transformation (e.g. softmax).
Note that by using element-wise summation, our model satisfies (4) for acting on composite features.
4.3 Loss Functions
Group Selection Models.
The individual learners can be trained to perform (traditional) featureselection (1) by minimizing the following loss function:
[Equation]
where ℓ Y is a suitable loss function for the prediction task (e.g. cross-entropy for classification tasksand MSE for regression tasks) and β ≥ 0 balances the two terms.
Note the selections probabilities π iare not regularized with the typical L1 penalty.
Instead, we apply an L2 penalty to the mean selectionprobability ⟨ π i ⟩ for each individual learner.
This is justified as follows.
Recall the optimizationproblem given by (5).
We desire a solution with the maximal number of predictive groups N , whileminimizing the number of selected features per group (cid:80) Ni =1 |G i | .
The standard L1 penalty term doesnot achieve this goal since adding an additional feature to either group G i or G j incurs the samepenalty.
In contrast, the L2 penalty imposed on ⟨ π i ⟩ penalizes adding extra features to already largegroups, favoring the construction of smaller groups over larger ones.
Aggregate Predictor.
The aggregate predictor can be trained jointly with the group feature selectionmodels by minimizing a standard prediction loss (where ℓ Y is the same as in (11)):
[Equation]
Additional Regularization.
If we simply apply the losses given by Eqs.
(11), (12), there will belimited (or even no) differentiation among the individual learners and the optimal solution wouldbe for each learner to simply solve the traditional feature selection problem (1).
This results inall learners selecting the same features, which does not achieve our aim of discovering groups ofpredictive features.
In order to encourage differentiation between the models, we introduce anadditional loss that penalizes the selection of the same features in multiple groups:
[Equation]
Overall Loss.
Combining the above, our overall loss function therefore can be written as follows:
[Equation]
where β E , β R ≥ 0 are hyperparameters to balance the losses.
Training CompFS with the loss given by (14) is designed to achieve the following: (1) The overallensemble network should be a good predictor ( L E ).
Each individual learner should to solve thetraditional feature selection problem ( L G i ), which requires the group predictor to be accurate whileselecting minimal features.
(3) Finally, we want the groups to be distinct and thus discourage highlysimilar groups ( L R ).
However, note that we do not exclude the possibility of some overlap of featuresbetween groups.
The model is end-to-end differentiable, so we train with gradient descent.
Evaluation.
During evaluation, only the gating procedure changes.
The way features can be selectedis chosen by the user.
A standard solution which we adopt in this paper is using a threshold λ andcalculating the gate vector
m i as follows: m k,i = 1 , if π k,i > λ and 0 otherwise.
Experiments
We evaluate CompFS using several synthetic and semi-synthetic datasets where ground truth featureimportances and group structure are known.
Specific architectural details are given in App.
C.
Additional information regarding experiments, benchmarks, and datasets can be found in App.
D.
Additional ablations and sensitivity analysis are in App. A.
Benchmarks.
The primary goal of our experiments is to demonstrate the utility of discoveringcomposite features over traditional feature selection.
Our main benchmark is an oracle featureselection method (“Oracle") that perfectly selects the ground truth features but provides no structure,giving all features as one group.
We also include comparisons to a linear feature selection method(LASSO)
[73] and two non-linear, state of the art approaches, Stochastic Gates (STG) [81] and
Supervised Concrete Autoencoder (Sup-CAE) [6].
Finally, we compare with Group LASSO [82],where we enumerate all groups with 1 or 2 features as predefined groups.
Note this represents asignificant simplification of the task for Group Lasso (see App.
G for additional baselines).
Metrics.
Since the ground truth feature groups G 1 , . . .
, G N are known, we use True Positive Rate(TPR) and False Discovery Rate (FDR) to assess the discovered features against the ground truth.
Toassess composite features, i.e. grouping, we define the Group Similarity (G sim ) as the normalized
Jaccard similarity between ground truth feature groups and the most similar proposed group:
[Equation]
where J is the Jaccard index [33] and ˆ G 1 , .
, ˆ G K are the discovered groups.
G sim ∈ [0 , 1] , where
G sim
= 1 corresponds to perfect recovery of the ground truth groups, while G sim
= 0 when none of thecorrect features are discovered.
See App.
E for additional details together with examples.
We assessthe models by seeing if the ground truth features have been correctly discovered, using TPR and FDR.
We then see if the underlying grouping has been uncovered (and correct features) using G sim .
5.1 Synthetic Experiments.
Dataset Description.
We begin by evaluating our method on a range of synthetic datasets wherethe ground truth feature importance is known (Table 1).
We generate synthetic datasets by samplingfrom the Gaussian distribution with initially no correlations across the features ( X ∼ N (0 , I ) ).
Weconstruct binary classification tasks, where the class y is determined by the following decision rules:
• (Syn1) y
=
if x 1 > 0 .
55 or x 2 > 0 .
55 , 0 otherwise.
The ground truth groups are {{ 1 } , { 2 }} .This
task assesses whether the model can separate two features rather than group them together.
• (Syn2) y
=
if x 1 x 2 > 0 . 30 or x 3 x 4 > 0 . 30 , 0 otherwise.
The ground truth groups are{{ 1 , 2 } , { 3 , 4 }} .
This task requires identifying groups consisting of more than one variable.
• (Syn3) y
=
if x 1 x 2 > 0 . 30 or x 1 x 3 > 0 . 30 , 0 otherwise.
The ground truth groups are
This task investigates whether a model can split the features into two overlapping
groups of two, rather than one group with all three features.
• (Syn4)
y
=
if x 1 x 4 > 0 . 30 or x 7 x 10 > 0 . 30 , 0 otherwise.
The ground truth groups are{{ 1 , 4 } , { 7 , 10 }} .
This task is equivalent to Syn2 , however, here the features exhibit strongcorrelation in collections of 3.
This task demonstrates the difficulty of carrying out group feature
selection (and indeed standard feature selection) when the features are highly correlated.
The decision rules are created such that there is minimal class imbalance.
We use signals with 500dimensions to demonstrate the utility in the high dimensional regime.
We use 20000 samples to trainand 200 to test.
Each experiment is repeated 10 times.
Analysis.
On both Syn1 and Syn2, CompFS achieves high TPR with no false discoveries (0%
FDR) and significantly higher G sim than the Oracle.
Despite allowing CompFS to discover up to5 groups, CompFS typically finds the correct number of groups (2), demonstrating that it is notnecessary for the number of allowed composite features to match the ground truth, which is vitalin real-world use cases where this is unknown.
Syn3 is significantly more challenging due to theoverlapping structure and we observe essentially the same performance as Oracle.
Despite finding allthe correct features and no false discoveries, CompFS typically finds the union { 1 , 2 , 3 } rather thanthe underlying group structure {{ 1 , 2 } , { 1 , 3 }} .
Finally, for Syn4, while CompFS has a relativelyhigh FDR, it frequently finds the ground truth relevant features and groups with similar G sim to Oracle.
[Table]
This is a challenging task with significant correlation between features.
Despite this, CompFS isable to uncover the underlying group structure, providing additional insight over traditional featureselection.
STG typically performs well in terms of traditional feature selection, but scores poorly interms of G sim due to not providing any group information.
5.2 Semi-Synthetic Experiments.
Dataset Description.
Next, we assess our ability to identify composite features using semi-syntheticmolecular datasets.
These tasks are analogs of real-world problems, such as identifying biologicallyactive chemical groups; however, the labels are determined by a synthetic “binding logic” so that theground truth feature relevance is known.
We use several of the datasets constructed by [56], some ofwhich were also used by [66].
The synthetic “binding logics” are expressed as a combination ofmolecular fragments that must either be present or absent for binding to occur and are used to labelmolecules from the ZINC database [32].
Each logic includes up to four functional groups (Table 6).
Molecules are featurized using a set of 84 functional groups, where feature x i = 1 if the moleculecontains functional group i and 0 otherwise.
The specific binding logics are given in App.
F.
[Table]
[Figure]
Analysis.
All methods are able to identify the ground truth relevant features; however, only CompFSprovides deeper insights.
Unlike for Syn1-4, LASSO correctly selects the ground truth featuressince the dataset consists of binary variables and thus it is possible to find performant linear models.
However, while discovering the correct features, Group LASSO selects all possible combinations ofthese features, adding no benefit over standard feature selection.
For Chem1-2, CompFS perfectly recovers the group structure in the majority of experiments, leadingto high G sim far exceeding traditional feature selection.
On Chem3, we occasionally discoveradditional features that are not part of the binding logic.
However, a number of molecular fragmentsare strongly correlated with the binding logic, even though they are not themselves included.
In fact,some features contain information about multiple functional groups.
For example, esters containa carbonyl and an ether; both are in the binding logic for Chem3, while ester is not, despite beinghighly informative, and thus occasionally CompFS incorrectly selects this feature.
In spite of this,
CompFS achieves significantly higher G sim than even Oracle.
This demonstrates the benefit ofthe grouping discovered by CompFS, even with a modest number of false discoveries.
As before,
CompFS typically finds the correct number of groups (2), despite being able to discover up to 5groups, further demonstrating that the number of composite features need not be known a priori ,which is the case in real-world applications.
5.3 Real-World Data: METABRIC
We compare CompFS and STG using 25 features to an MLP using all 489 features.
[Table]
Dataset Description.
Finally, we assess CompFS on a realworld dataset, METABRIC [19, 60], where the ground truthgroup structure is unknown .
METABRIC contains gene expression, mutation, and clinical data for 1,980 primary breastcancer samples.
We evaluated the ability to predict the progesterone receptor PR) status of the tissue based on the gene ex- pression data
which consists of measurements for
genes.
Analysis.
CompFS suffers limited performance degradation compared to using all features, despiteonly using 5% of the features (Table 3).
Despite imposing a more rigid structural form on howfeatures can interact in the predictive model, STG only had marginally greater predictive power than
CompFS.
However, CompFS provides greater insight into how the features interact than STG.
We found supporting evidence in the scientific literature for all but 1 of the genes discovered by
CompFS (Table 10).
In addition, within each group, we found further evidence of the interactionsbetween genes, demonstrating the ability for CompFS to learn informative groups of features.
Forexample, in Group 1, CXCR1 and PEN-2 (the protein encoded by PSENEN) are known to interact[5].
In Group 2, BMP6 encodes a member of the TGF- β superfamily of proteins, and TGF- βtriggers activation of SMAD3 [17].
In the same group, MAPK1 activity is dependent on the activityof PRKCQ in breast cancer cells [13], while MAPK1 is also known to interact with MAPT [45],
SMAD3 [23], and BMP6 [85].
Additional supporting evidence can be found in Appendix H.
Conclusion
In this paper, we introduced CompFS, an ensemble-based approach that tackles the newly proposedchallenge of composite feature selection.
Using synthetic and semi-synthetic data, we assess ourability to go beyond traditional feature selection and recover deeper underlying connections betweenvariables.
CompFS is not without limitations: as with other methods, points of difficulty arise whenfeatures are highly correlated, or if predictive composites contain overlapping features.
Future workmay overcome this by using correlated gates.
Further, as with many traditional feature selectionmethods, there are no guarantees on false discovery rate.
This could be tackled by first proposingcandidate composite features, and then using the Group Knockoff procedure.
Additionally, to discovergroups, CompFS requires the introduction of additional hyperparameters which could be challengingto tune in practice.
More broadly, as with standard feature selection, groups found under compositefeature selection must be verified by domain experts (both features but additionally interactions).
However, we believe the additional structure provided by composite feature selection could be ofsignificant benefit to a wide variety of practitioners.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Conclusion for details.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] In theConclusion, we caution that as with any feature selection method, discovered featuresmust be verified or evaluated by domain experts.
This verification or evaluation might
be costly, and should the method perform poorly, could result in wasted resources.
Inaddition, without additional oversight (primarily in dataset construction but also when
validating features), features that contain bias could remain and be identified by featureselection algorithms.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide allcode needed to reproduce all results in the supplemental material.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Hyperparameters for each experiment are provided in Table 4.Architecture details are provided in Appendix C and further experimental details areprovided in Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All experiments are repeated 10 times and results arereported along with standard deviations.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] All experiments can be run
easily on a commercially-available laptop.
We provide further details of the compute
resources used in Appendix D.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We used several
existing methods and datasets (see Experiments).
All benchmark methods and datasets
are clearly cited.
(b) Did you mention the license of the assets?
[Yes] Licenses of assets (benchmark methods
and datasets) is provided in Appendix D and F.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
The code to run our experiments is included in the supplementary material.
The code
will also be released publicly after the review period.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]