Composite Feature Selection Using Deep Ensembles
Anonymous Author(s)
AffiliationAddress email
Abstract
In many real world problems, features do not act alone but in combination witheach other.
For example, in genomics, diseases might not be caused by any singlemutation but require the presence of multiple mutations.
Prior work on featureselection either seeks to identify individual features or can only determine relevantgroups from a predefined set.
We investigate the problem of discovering groupsof predictive features without predefined grouping.
To do so, we define predictivegroups in terms of linear and non-linear interactions between features.
We introducea novel deep learning architecture that uses an ensemble of feature selection modelsto find predictive groups, without requiring candidate groups to be provided.
Theselected groups are sparse and exhibit minimum overlap; additionally, we canrank the groups based on their predictive power for downstream decision making.
Furthermore, we propose a new metric to measure similarity between discoveredgroups and the ground truth.
We test our model on multiple synthetic tasks, semisynthetic chemistry datasets and image datasets to demonstrate its utility.
Introduction
Feature selection is a key problem permeating statistics, machine learning and broader science.
Typically in high-dimensional datasets, the majority of features will not be responsible for the targetresponse and thus an important goal is to identify which variables are truly predictive.
For example,in healthcare there may be many features (such as age, sex, medical history, etc.)
that could beconsidered, while only a small subset might in fact be relevant for predicting the likelihood ofdeveloping a specific disease.
By eliminating irrelevant variables, feature selection algorithms can beused to drive discovery, improve model generalisation/robustness, and improve interpretability [9].
However, features often do not act alone but instead in combination .
In genetics, for instance, it hasbeen noted that understanding the origins of many diseases may require methods able to identify morecomplex genetic models than single variants [30].
While feature selection might be able to identify aset of features associated with a particular response, the underlying structure of how features interactis not captured.
Further, the resulting predictive models can be complex, hard to interpret, and notamenable to the generation of hypotheses that can be experimentally tested [20].
This limits theimpact such models can have in furthering scientific understanding across many domains wherevariables are known to interact, such as genetics [30, 31, 28], medicine [38, 6], and economics [3].
Group feature selection is a generalisation of standard feature selection, where instead of selectingindividual features, groups of features are either entirely chosen or entirely excluded.
A primaryapplication of group feature selection is when features are jointly measured, for example by differentinstruments.
In such scenarios, groups are readily defined as features measured by the same instrument.
A natural question is which instruments give the most meaningful measurements.
Group feature selection has also been applied in situations where there is extensive domain knowledge regarding thegroup structure [32] or where groups are defined by the correlation structure between features (e.g.neighbouring pixels in images are highly correlated).
The pervasive issue with current group featureselection methods is that a predetermined grouping must be provided, and the groups are selectedfrom the given candidates.
In reality, we may not know how to group the variables.
In this paper we seek to solve a related but ultimately different and more challenging problem, whichwe call Composite Feature Selection .
We wish to find groups of variables without prior knowledge,where each group acts as a separate predictive subset of the features and the overall predictive poweris greatest when all groups are used in unison.
We call each group of features a composite feature.
By imposing this structure on the discovered features, we attempt to isolate pathways from featuresto the response variable.
Doing so offers deeper insights into why specific features are important thanstandard feature selection, with the goal of improving data-driven discovery.
Contributions.
(1) We formalise composite feature selection as an extension of standard featureselection, defining composite features in terms of linear and non-linear interactions between variables(Sec. 3).
(2) We propose a new deep learning architecture for composite feature selection using anensemble-based approach (Sec. 4).
(3) To assess our solution, we introduce a metric for assessingcomposite feature similarity based on Jaccard similarity.
(4) We demonstrate the utility of our modelon a range of synthetic and semi-synthetic tasks where the ground truth group features are known(Sec.
We see that our model not only frequently recovers the relevant features, but also oftendiscovers the underlying group structure.
Related Work
Significant attention has been placed on feature selection, with a range of solutions includingtraditional methods (e.g. [25, 21, 13]) and deep learning approaches (e.g. [24, 39, 2]). Several ofthese approaches have been extended to select predefined groups instead of individual features.
Forexample, LASSO [35, 37] is a linear feature selection method that uses an L1 penalty to imposesparsity among coefficients.
Group LASSO [40] generalises this to allows predefined groups to beselected or excluded jointly, rather than single features, by replacing the L1 penalty with L2 penaltieson each group.
Other feature selection methods, such as SLOPE [5, 23], have been similarly extendedto group feature selection to give Group-SLOPE [7].
Further examples of group feature selectionusing adapted loss functions are SCAD-L2 [42] and hierarchical LASSO [43].
Feature selection for linear regression can also be approached from a Bayesian perspective byemploying latent variables w i ∈ { 0 , 1 } to indicate whether feature i is relevant to the model or not[11].
This class of approaches has been generalised to the group setting such that the latent variablew does not encode the relevance of each feature, but of each group from a predefined set [14].
Finally, the Knockoff procedure [4, 8] is a separate feature selection method that provides guaranteeson the False Discovery Rate that has been utilized by a number of feature selection methods [18, 26,33, 36].
Generalizations of the Knockoff procedure to the group setting also exist [10, 44].
The key commonality is that none of these methods discover groups, but instead can only selectgroups from a set of predefined candidates.
Therefore, while they may be applicable when we cansplit inputs into groups, they are not able to find groups of predictors on their own.
Our work differsfrom these methods by considering the challenge of finding such groups in the absence of priorknowledge.
Additionally, unlike prior work, we do not make assumptions about correlations betweenfeatures or restrictions on groups, such as requiring that the candidate groups partition the features.
Problem Description
Let X ∈ X p be a p -dimensional signal (such as gene expressions or patient covariates) and Y ∈ Ybe a response (such as disease traits).
Informally, we wish to group the features into the maximum number of subsets, G i ⊂ [ p ] , where complex interactions may happen within groups but only simpleinteractions between groups, allowing us to separate the groups into different pathways from the signalto the response.
Note that we do not enforce assumptions on the groups such as non-overlappinggroups or every feature being in at least one group.
In this section, we begin with a description oftraditional feature selection before formalizing composite feature selection.
3.1 Feature Selection
The goal of traditional feature selection is to select a subset, S ⊂
[ p ] , of features that are relevant forpredicting the response variable.
In particular, in the case of embedded feature selection [12], this isconducted jointly with the model selection process.
Let ∗ denote any point not in X and define X S =
( X ∪ {∗} ) p .
Then, given X ∈ X p , the selectedsubset of features can be denoted as X S ∈ X S where x S ,k
= x k if k ∈ S and x S ,k
= ∗ if k / ∈ S .
Let f : X S
→ Y be a function in F taking subset X S as input to yield Y .
Then, selecting relevantfeatures for predicting a response can be achieved by solving the following optimization problem:
[Equation]
where δ constrains the number of selected features and ℓ Y ( y, y ′ ) is a task-specific loss function.
This can be solved by introducing a selection vector M =
( M 1 , · · · , M p ) ∈ { 0 , 1 } p , consisting ofbinary random variables governed by distribution p M , with realization m indicating selection of thecorresponding features.
Then, the selected features given vector m can be written as
[Equation]
where ⊙ indicates element-wise multiplication and ˆ x are the values assigned to features that are notselected (typically ˆ x ≡ 0 or ¯ x ).
(1) can be (approximately) solved by jointly learning the model fand the selection vector distribution p M based on the following optimization problem:
[Equation]
where β is a balancing coefficient that controls the number of features to be selected.
3.2 Composite Feature Selection
The goal of composite feature selection is to not only find the predictive features, but also to groupthem based on how they are predictive.
For example, assume features x 1 and x 2 are only predictivewhen both are known, but make the same prediction independent of x 3 .
Then we wish to groupx 1 , x 2 from x 3 .
In this section, we define the embedded composite feature selection problem; that is,we want to find a valid model f and groups {G 1 , . .
. , G N } in parallel.
The model is valid when onlysimple interactions happen between groups and interactions within groups can be arbitrarily complex.
This raises the question, what is a simple interaction? We can view each group as contributing adifferent piece of information to the final prediction.
When combining groups, we require order notto matter, as otherwise more complex interactions exist between groups.
Therefore we must combinethe representations using a permutation invariant aggregator .
Separately, we can also view our modelas acting on a set of groups [41], which naturally leads to permutation invariant aggregation.
It is well established that any permutation invariant aggregation function can be decomposed asρ [ (cid:80) ϕ ( · )] , for an appropriate choice of ρ and ϕ .
This gives f ( x ) =
g (cid:0) ρ (cid:2) (cid:80) i ϕ
( f i ( x G i ))
(cid:3)(cid:1) , wheref i encodes group i , ρ and ϕ give the permutation invariant aggregation and g is any final non-linearfunction, softmax for instance.
The function composition of ϕ and f i can be relabelled as ˜ f i =
ϕ ◦ f i ,and the composition of g and ρ can be relabelled as ˜ ρ
=
g ◦ ρ .
This leads to f ( x ) =
˜
ρ (cid:2) (cid:80) i ˜
f i ( x G i ) (cid:3) ,this gives a definition for what a valid model structure can be in composite feature selection.
Definition 3.1.
The most general valid model for acting on N composite features is given by:
[Equation]
That is, the groups must interact only once, all groups must be included and the interaction is asummation; all other interactions can (and often should) be non-linear.
Depending on the task, a specific permutation invariant aggregation may be chosen (e.g. Max() ).
However, any permutation invariant aggregator can be (approximately) expressed in the form of Def.3.1; thus, when learning from data, the general structure of Def.
3.1 means that this is not necessary.
The embedded composite feature selection problem can now be phrased in an analogous way totraditional feature selection.
Let ∗ denote some point not in X and define X G i =
( X ∪ {∗} ) p .
Then,given X ∈ X p , the selected group of features is denoted as X G i ∈
X G i where x G i ,k = x k if k ∈ G iand x k = ∗ if k / ∈ G i .
Let f i : X
G i
→ Z be a function in F that takes as input the subset X G i andoutputs a latent representation z i .
Then, finding the groups of features can be achieved by solving theoptimization problem:
[Equation]
subject to |G i
[Equation]
N
[Equation] [Equation]
where δ constrains the number of selected features in each group and ∆ gives the minimum numberof groups.
This objective leads to multiple smaller groups, rather than one group containing allfeatures, which is consistent with our motivation of the problem.
Continuing to expand from traditional feature selection, we can also extend the solution to thecomposite setting.
For N groups we can introduce a selection matrix M ∈ { 0 , 1 } N × p , governed bydistribution p M .
For a realization m , the selected features from group i are given by
[Equation]
where m i is the i th row of M .
We can approximately solve (5) by solving the optimisation problem:
[Equation]
where f ( x ) obeys Def.
(4) and R e is a regularisation term which controls how features are selectedin each group.
R e should capture both group size (i.e. encourage as few features as possible to beselected) but also the relationships between groups (i.e. groups should be distinct and not redundant).
3.3 Challenges
There are various challenges in solving the composite feature selection problem.
While the ultimatetask is to find predictive groups of features, there remains the necessity simply to identify predictivefeatures, which is already an NP-hard problem [1].
Composite feature selection not only inheritsthis property but introduces additional complexity since we can think of each group as solving aseparate feature selection problem.
Consider the number of potential solutions: in traditional featureselection (assuming not all features are selected), there are 2 n − 2 ways of selecting a subset from nfeatures; even restricting to at most m << n quickly becomes unfeasible for even modest valuesof m .
In composite feature selection, every group has the same number of solutions as traditionalfeature selection, drastically increasing the total number of possible solutions.
A challenge specificto composite feature selection arises when the ground truth group structure contains groups withoverlapping features (e.g. feature x 1 interacts independently with both x 2 and x 3 ).
In this scenario, itis difficult to separate these two effects while penalizing the inclusion of additional features.
Method: CompFS
In this section, we propose a novel architecture for finding predictive groups of features, which werefer to as Comp osite F eature S election (CompFS).
In order to discover groups of features, our modelis composed of a set of group selection models and an aggregate predictor.
Our approach resemblesan ensemble of “weak” feature selection models, where each learner attempts to solve the task usinga sparse set of features (Figure 1).
These models are then trained in such a way as to discover distinctpredictive groups.
We first consider the group selection models in more detail before describing howthe group selection models are combined and the training procedure.
GroupEncoder ( 𝒇 𝜽 𝒊 )
Group Predictor ( 𝒉 𝝋 𝒊 )
Predictor ( 𝒉 𝝋 ) 𝐗 𝛑 𝒊 ෩𝐗 𝒊 ෥𝐦 𝐢
Group Generator
S a m p l e ෝ𝒚 𝒊 𝒛 𝒊
GroupEncoder ( 𝒇 𝜽 𝒋 )
Group Predictor ( 𝒉 𝝋 𝒋 ) 𝐗 𝛑 𝒋 ෩𝐗 𝒋 ෥𝐦 𝐣
Group Generator
S a m p l e ෝ𝒚 𝒋 𝒛 𝒋 ...
ෝ𝒚
G a t e
G a t e
4.1 Group Selection Models
CompFS is composed of a set of group selection models, each of which primarily aims to solve thetraditional feature selection problem specified by (1).
We achieve this by solving (3) using a neuralnetwork-based approach with stochastic gating of the input features.
Each group selection modelconsists of the following three components (Figure 1):
• Group Selection Probability , π i =
( π 1 ,i , · · · , π p,i ) ∈ [0 , 1] p , which is a trainable vector thatgoverns the Bernoulli distribution used to generate the gate vector m i .
Each element of the
selection probability π k,i indicates the importance of the corresponding feature to the target.
• Group Encoder , f θ i : X p
→ Z , that takes as input the selected subset of features ˜ x i and outputslatent representations z i ∈ Z .
• Group Predictor , h ϕ
i : Z → Y , that takes as input the latent representations of the selected subset of features, z i =
f θ i
˜ x i ) , and outputs predictions on the target outcome.
Solving (3) directly is not possible since the sampling step has no differentiable inverse.
Instead, weuse the relaxed Bernoulli distribution [27, 17] and apply the reparameterization trick as follows.
Formally, given selection probability π = ( π 1 , · · · , π p ) and independent Uniform (0 , 1) randomvariables ( U 1 , · · · , U p ) , we can generate a relaxed gate vector ˜ m
= ( ˜ m 1 , · · · , ˜ m p ) ∈ (0 , 1) p basedon the following reparameterization trick [27]:
[Equation]
where σ ( x ) = (1 + exp( − x )) − 1 is the sigmoid function.
This relaxation is parameterized by π andtemperature τ ∈ (0 , ∞ ) .
Further, as τ → 0 , the gate vectors ˜ m k converge to Bernoulli ( π k ) randomvariables.
Crucially this is differentiable with respect to π .
Given group selection probability π i , we first sample relaxed Bernoulli random variable ˜
m i accordingto (8) and then use ˜ m i in a gating procedure to select the group of features.
The output of the gate is:
[Equation]
where we replace the variables that were not selected by their mean value ¯ x .
The mean is used becausein particular tasks a feature having a value of 0 may be particularly meaningful.
The gate output ˜ x
i isthen fed into the group encoder f
θ i to yield representation
z i =
f θ i
(˜ x i ) .
This representation is finallypassed to the group predictor h ϕ i to produce the prediction for an individual learner, ˆ y i
=
h ϕ i ( z i ) .
4.2 Group Aggregation
The final component necessary for CompFS is a way to aggregate the individual group selectionmodels.
This is achieved via an overall predictor , h ϕ : Z → Y , that takes as input the set of latentrepresentations { z 1 , . . .
, z N } produced by the individual learners and outputs predictions on thetarget outcome.
For simplicity, we apply a linear prediction head to the latent representations and useelement-wise summation to aggregate.
Thus, the prediction of the ensemble is given by:
[Equation]
where N is the number of members of the ensemble and ρ is a suitable transformation (e.g. softmax).
Note that by using element-wise summation, our model satisfies (4) for acting on composite features.
4.3 Loss Functions
Group Selection Models.
The individual learners can be trained to perform (traditional) featureselection (1) by minimizing the following loss function:
[Equation]
where ℓ Y is a suitable loss function for the prediction task (e.g. cross-entropy for classification tasksand MSE for regression tasks) and β ≥ 0 balances the two terms.
Note the selections probabilities π i are not regularized with the typical L1 penalty.
Instead, we applyan L2 penalty to the mean selection probability ⟨ π i ⟩ for each individual learner.
This is justified asfollows.
Recall the optimization problem given by (5).
We desire a solution with the maximal numberof predictive groups N , while minimizing the number of selected features per group (cid:80) Ni =1 |G i | .
Thestandard L1 penalty term does not achieve this goal since adding an additional feature to either group G i or G j incurs the same penalty.
In contrast, the L2 penalty imposed on ⟨ π i ⟩ penalizes adding extrafeatures to already large groups, favoring the construction of smaller groups over larger ones.
Aggregate Predictor.
The aggregate predictor can be trained jointly with the group feature selectionmodels by minimizing a standard prediction loss:
[Equation]
where ℓ Y is the same as in (11).
Additional Regularization.
If we simply apply the losses given by Eqs.
(11), (12), there will belimited (or even no) differentiation among the individual learners and the optimal solution wouldbe for each learner to simply solve the traditional feature selection problem (1).
This results inall learners selecting the same features, which does not achieve our aim of discovering groups ofpredictive features.
In order to encourage differentiation between the models, we introduce anadditional loss that penalizes the selection of the same features in multiple groups:
[Equation]
Overall Loss.
Combining the above, our overall loss function therefore can be written as follows:
[Equation]
where β E , β R ≥ 0 are hyperparameters to balance the losses.
Training CompFS with the loss given by (14) is designed to achieve the following: (1) The overallensemble network should be a good predictor ( L E ).
Each individual learner should attemptto solve the traditional feature selection problem ( L G i ), which requires the group predictor to beaccurate while selecting a minimal number of features.
(3) Finally, we want the groups to be distinctand thus discourage highly similar groups ( L R ).
However, note that we do not exclude the possibilityof some overlap of features between groups.
The model is end-to-end differentiable, so we train withgradient descent.
Evaluation.
During evaluation, only the gating procedure changes.
The way features can be selectedis chosen by the user.
A standard solution which we adopt in this paper is using a threshold λ andcalculating the gate vector
m i as follows: m k,i = 1 , if π k,i > λ and 0 otherwise.
[Equation]
Experiments
We evaluate CompFS using several synthetic and semi-synthetic datasets where ground truth featureimportances and group structure are known.
Specific architectural details are given in App.
B.
Additional information regarding experiments, benchmarks, and datasets can be found in App.
C.
Additional ablations and sensitivity analysis are in App. A.
Benchmarks.
The primary goal of our experiments is to demonstrate the utility of discoveringcomposite features over traditional feature selection.
Our main benchmark is an oracle featureselection method (“Oracle") that perfectly selects the ground truth features but provides no structure,giving all features as one group.
As a result, we do not conduct an exhaustive comparison withtraditional feature selection, since the upper bound on the performance of any such method is Oracle. However, for illustrative purposes, we include comparisons to a linear feature selection method(“LASSO”, [37]) and an ablation of CompFS where we only allow a single group (“CompFS(1)”).
Additional comparisons to STG [39] and Concrete Autoencoder [2] can be found in App.
F. Finally,we compare with Group LASSO [40], where we enumerate all groups with 1 or 2 features aspredefined groups.
Note this represents a significant simplification of the task for Group Lasso.
Metrics.
Since the ground truth feature groups G 1 , . . .
, G N are known, we use True Positive Rate(TPR) and False Discovery Rate (FDR) to assess the discovered features against the ground truth.
Toassess composite features, i.e. grouping, we define the Group Similarity (G sim ) as the normalized
Jaccard similarity between ground truth feature groups and the most similar proposed group:
[Equation]
where J is the Jaccard index [16] and ˆ G 1 , .
, ˆ G K are the discovered groups.
G sim ∈ [0 , 1] , where
G sim
= 1 corresponds to perfect recovery of the ground truth groups, while G sim
= 0 when none of thecorrect features are discovered.
See App.
D for additional details together with examples.
TPR and FDR assess whether the ground truth features have been correctly identified but do nottake into account the group structure, which is our main goal.
On the other hand, G sim takes bothinto account and requires both the correct features to be identified and the underlying grouping to beuncovered. Finally, models should have strong predictive performance, which we assess via accuracy.
5.1 Synthetic Experiments.
Dataset Description.
We begin by evaluating our method on a range of synthetic datasets wherethe ground truth feature importance is known (Table 1).
We generate synthetic datasets by samplingfrom the Gaussian distribution with initially no correlations across the features ( X ∼ N (0 , I ) ).
Weconstruct binary classification tasks, where the class y is determined by the following decision rules:
• (Syn1) y
=
if x 1 > 0 .
55 or x 2 > 0 .
55 , 0 otherwise.
The ground truth groups are {{ 1 } , { 2 }} .This
task assesses whether the model can separate two features rather than group them together
• (Syn2) y
=
if x 1 x 2 > 0 . 30 or x 3 x 4 > 0 . 30 , 0 otherwise.
The ground truth groups are
This task requires identifying groups consisting of more than one variable.
• (Syn3) y
=
if x 1 x 2 > 0 . 30 or x 1 x 3 > 0 . 30 , 0 otherwise.
The ground truth groups are{{ 1 , 2 } , { 1 , 3 }} .
This task investigates the model when groups overlap, and whether it can split
into two overlapping groups of two features each, rather than one group with three features
• (Syn4)
y
=
if x 1 x 4 > 0 . 30 or x 7 x 10 > 0 . 30 , 0 otherwise.
The ground truth groups are{{ 1 , 4 } , { 7 , 10 }} .
This task is equivalent to Syn2 , however, here the features exhibit strongcorrelation.
We enforce a block diagonal covariance matrix, with 1 on the diagonals
, and 0.99 onthe off diagonal entries in the blocks.
The block matrices are 3 × 3 and cover the first 12 features.
This task demonstrates the difficulty of carrying out group feature selection (and indeed standardfeature selection) when the features are highly correlated. [Table]
The decision rules are created such that there is minimal class imbalance.
We use signals with 500dimensions to demonstrate the utility in the high dimensional regime.
We use 20000 samples duringtrain and 200 to test.
Each experiment is repeated 10 times.
For all experiments we use a threshold ofλ = 0 .
7 to determine whether a feature is included by CompFS.
Analysis.
On both Syn1 and Syn2, CompFS achieves high TPR with no false discoveries (0% FDR)and significantly higher G sim than oracle feature selection.
Despite allowing CompFS to discover upto 5 groups, CompFS typically finds the correct number of groups (2), demonstrating that it is notnecessary for the number of allowed composite features to match the ground truth, which is vital inreal-world use cases where this is unknown.
Syn3 is significantly more challenging due to the overlapping structure and we observe essentially thesame performance as Oracle.
Despite finding all the correct features and no false discoveries, CompFStypically finds the union { 1 , 2 , 3 } rather than the underlying group structure {{ 1 , 2 } , { 1 , 3 }} .
Finally, for Syn4, while CompFS has a relatively high FDR, it frequently finds the ground truthrelevant features and groups, outperforming the ablation of our method (CompFS(1)) by all metricswith similar G sim to Oracle.
This is a challenging task with significant correlation between features.
Despite this, CompFS is able to uncover the underlying group structure, providing additional insightover traditional feature selection.
5.2 Semi-Synthetic Experiments.
Dataset Description.
Next, we assess our ability to identify composite features using semi-syntheticmolecular datasets.
These tasks are analogs of real-world problems, such as identifying biologicallyactive chemical groups; however, the labels are determined by a synthetic “binding logic” so that theground truth feature relevance is known.
We use several of the datasets constructed by [29], some ofwhich were also used by [34].
The synthetic “binding logics” are expressed as a combination of molecular fragments that musteither be present or absent for binding to occur and are used to label molecules from the ZINCdatabase [15].
Each logic includes up to four functional groups (Table 5).
Molecules are featurizedusing a set of 84 functional groups, where feature x i = 1 if the molecule contains functional group iand 0 otherwise.
The specific binding logics are given in App.
E. Finally, note that many molecularfragments correlate with the binding logic but are not themselves responsible for binding.
[Table]
Analysis.
All methods are able to identify the ground truth relevant features; however, only CompFSprovides deeper insights.
Unlike for Syn1-4, LASSO correctly selects the ground truth featuressince the dataset consists of binary variables and thus it is possible to find performant linear models.
However, while discovering the correct features, Group LASSO selects all possible combinations ofthese features, adding no benefit over standard feature selection.
For Chem1-2, CompFS perfectly recovers the group structure in the majority of experiments, leadingto high G sim far exceeding traditional feature selection.
On Chem3, we occasionally discoveradditional features that are not part of the binding logic.
However, a number of molecular fragmentsare strongly correlated with the binding logic, even though they are not themselves included.
In fact,some features contain information about multiple functional groups.
For example, esters containa carbonyl and an ether; both are in the binding logic for Chem3, while ester is not, despite beinghighly informative, and thus occasionally CompFS incorrectly selects this feature.
In spite of this,
CompFS achieves significantly higher G sim than even Oracle.
This demonstrates the benefit ofthe grouping discovered by CompFS, even with a modest number of false discoveries.
As before,
CompFS typically finds the correct number of groups (2), despite being able to discover up to 5groups, further demonstrating that the number of composite features need not be known a priori ,which is the case in real-world applications.
[Figure]
MNIST.
Finally, we investigate CompFS on
MNIST [22], where the ground truth group structure is unknown .
CompFS achieves overall accuracy of 95.2%, with four groups using 38 features( < 5% of pixels) on average.
This is comparable toother studies [2], with individual learners achievingc.
80% accuracy and only minor overlap.
Thus,while individual learners might not be able to solve the problem themselves, they provide differentpieces of information to an overall ensemble.
We visualise the pixels that each learner uses in Figure2.
We see that learners target different aspects of the problem; in particular, groups 1 and 2 havedistinct features, with group 1 focusing more on the left and bottom to detect loops, while group 2 onthe top right thus cannot make this distinction.
See App. F.2 for additional details and comparisons.
Conclusion
In this paper, we introduced CompFS, an ensemble-based approach that tackles the newly proposedchallenge of composite feature selection.
Using synthetic and semi-synthetic data, we assess ourability to go beyond traditional feature selection and recover deeper underlying connections betweenvariables.
As with any feature selection method, discovered features must be verified or evaluated bydomain experts; however, we believe the additional structure provided by composite feature selectioncould be of significant benefit to a wide variety of practitioners.