Few-shot Learning for Feature Selection with Hilbert-Schmidt Independence Criterion
Abstract
We propose a few-shot learning method for feature selection that can select relevant features given a small number of labeled instances.
Existing methods require many labeled instances for accurate feature selection.
However, sufﬁcient instances are often unavailable.
We use labeled instances in multiple related tasks to alleviate the lack of labeled instances in a target task.
To measure the dependency between each feature and label, we use the Hilbert-Schmidt Independence Criterion, which is a kernel-based independence measure.
By modeling the kernel functions with neural networks that take a few labeled instances in a task as input, we can encode the taskspeciﬁc information to the kernels such that the kernels are appropriate for the task.
Feature selection with such kernels is performed by using iterative optimization methods, in which each update step is obtained as a closed-form.
This formulation enables us to directly and efﬁciently minimize the expected test error on features selected by a small number of labeled instances.
We experimentally demonstrate that the proposed method outperforms existing feature selection methods.
Introduction
Feature selection is a fundamental problem in machine learning that aims to ﬁnd the subset of relevant features [7].
By extracting a small subset of features, we can interpret the characteristics of datasets, accelerate the learning processes of subsequent problems such as regression and classiﬁcation, and eliminate the cost of collecting irrelevant or redundant features.
Thanks to these beneﬁcial properties, feature selection methods have been used in various applications such as biomarker discovery [52], document categorization [10], disease diagnosis [1] and drug development [32].
Many feature selection methods have been proposed.
An unsupervised approach aims to ﬁnd relevant features without label information [3, 30, 5].
Although it is useful, it has difﬁculty ﬁnding features that are responsible for predicting responses without labels.
On the other hand, a supervised approach can ﬁnd such features by using labeled instances [48, 34, 53, 54, 29].
Therefore, this paper focuses on supervised feature selection.
Existing supervised feature selection methods often require many labeled instances to accurately select relevant features [21].
However, sufﬁcient instances can be difﬁcult to collect in many applications.
For example, although feature selection is used for analysis in healthcare [36, 31], sufﬁcient instances (e.g., patients) for prognosis analysis might be difﬁcult to collect in a hospital.
In marketing, when data scientists want to analyze sales by using past data in each area, sufﬁcient instances might be difﬁcult to collect from underdeveloped or new areas.
[Figure]
Even if many labeled instances are difﬁcult to collect in a task of interest, called a target task, they might be obtainable in different but related tasks, called source tasks.
For example, sufﬁcient labeled instances might be obtainable from other hospitals or urban areas.
When target and source tasks are related, we can transfer useful knowledge in source tasks to target ones [17].
In this paper, we propose a few-shot learning method for supervised feature selection.
Few-shot learning is usually formulated as meta-learning, which learns how to learn from a few data using data in multiple tasks.
Speciﬁcally, the proposed method meta-learns with labeled instances in multiple source tasks and uses the learned knowledge for feature selection with a small number of labeled instances in unseen target tasks.
Figure 1 illustrates the overview of our meta-learning procedure.
In feature selection, the dependency between each feature and response needs to be measured.
To this end, we use the Hilbert-Schmidt Independence Criterion (HSIC), which is a non-parametric kernel-based independence measure [15].
Since HSIC can measure the non-linear dependencies in a principle way without going through density estimation, it is successfully used for various applications including feature selection [53, 11, 47, 24].
To accurately measure the dependencies from a few labeled instances, the kernel functions need to be suitably designed.
With the proposed method, the kernel functions are modeled by permutation-invariant neural networks [56] that take a small number of labeled instances, called a support set, as input.
By this modeling, our model can encode the information of the support set to the kernels, and the kernels are meta-learned with labeled instances in source tasks so that our model can accurately select relevant features even from a few labeled instances.
With such kernels, feature selection problems with the support set can be formulated as non-negative least square problems with an (cid:96) 1 regularizer, which is introduced in a representative feature selection method, HSIC Lasso [53].
To obtain the global optimal solution (coefﬁcient parameters) in this optimization problem, we use the Iterative Shrinkage
Thresholding Algorithm (ISTA) [8], in which each update step can be obtained as a closed-form and is differentiable.
This differentiability enables us to efﬁciently backpropagate the expected test feature selection error in terms of HSIC, which is calculated with testing instances (a query set) drawn from the same task as the support set, after adapting to the support set with gradient-based optimization methods.
The proposed method further treats initial coefﬁcient parameters in the optimization problems as trainable parameters, and they are also meta-learned such that only a few update steps by ISTA lead to appropriate coefﬁcient parameters for each task.
Since our neural network and initial coefﬁcient parameters are shared across all tasks, the learned model can be applied to unseen target tasks.
As a result, the learned model can accurately select features from a small number of labeled instances in a target task.
Our main contributions are summarized as follows: (1) To the best of our knowledge, our work is the ﬁrst attempt at few-shot learning for supervised feature selection.
(2) We propose a novel method that combines neural networks with HSIC Lasso in the meta-learning framework.
Our formulation enables us to perform accurately few-shot feature selection with the closed-from update steps, which is preferable for the efﬁcient meta-learning.
(3) We empirically show that the proposed method outperformed existing feature selection methods when there are no sufﬁcient target data in both synthetic and real-word datasets.
Related Work
Many supervised feature selection methods have been proposed [7, 6].
Least Absolute Shrinkage and Selection Operator (Lasso) is a representative method that uses a sparse linear model for feature selection [48].
Although this method is useful, it cannot ﬁnd a non-linear relationship between feature and response.
To perform non-linear feature selection, several kernel-based methods have been proposed [53, 47, 42].
Among them, HSIC Lasso has performed excellently despite its simple formulation [53, 18, 11, 24].
We use this formulation in our framework since it enables effective and efﬁcient support set adaptation to be performed as described later.
Neural network-based feature selection methods have recently been proposed [54, 29, 3, 33].
With the high expressive capability of neural networks, they can ﬁnd non-linear complex interaction between feature and response.
However, they cannot perform well due to overﬁtting when there are insufﬁcient instances [25, 31].
Although our model also uses neural networks in kernel functions, they are shared across all tasks and are learned with all instances in source tasks.
Thus, the proposed method can accurately select relevant features without overﬁtting.
We discuss the merits of feature selection in detail in Appendix A.
Multi-task and transfer feature selection methods improve feature selection performances by sharing instances in several tasks [37, 57, 2].
They require instances in target tasks at the training phase.
In contrast, the proposed method does not.
Our method can quickly adapt to unseen target tasks given target labeled instances at the testing phase.
Meta-learning has recently been attracting a lot of attention, which trains a model such that it generalizes well after adapting to few instances (support set) [17, 50, 46, 44, 19].
Although many methods have been proposed, existing methods are not designed for few-shot supervised feature selection and thus they cannot be applied to our problems.
Some studies use the term “meta-learning” for selecting a feature selection algorithm taking meta-features such as number of classes and number of features [45, 22, 39].
They are not methods for few-shot learning, and thus, the scope of these studies is different to that of our study.
One method uses the meta-learning framework to select features in hidden features obtained through CNNs for stable prediction [55].
This method cannot perform task-speciﬁc feature selection or feature selection from a few data.
The gradient-based meta-learning methods such as model-agnostic meta-learning (MAML) [9] adapt to the support set by iteratively updating the whole neural network parameters with gradient descent methods.
These methods require second-order derivatives of the parameters of the whole neural network when backpropagating the loss, which imposes considerable computational and memory burdens [4, 43].
Although the proposed method also iteratively adapts to the support set by ISTA, our formulation can adapt to support sets by updating only coefﬁcient parameters for ISTA whose dimension is equal to the feature vector size, and its update step can be obtained as a closed-form.
This closed-form update step enables calculation of the second-order derivative of the parameters to be evaded in an automatic differentiation framework such as Pytorch [40], which realizes more efﬁcient adaptation.
Encoder-decoder meta-learning such as neural processes [12, 13, 19] efﬁciently adapts to a support set by forwarding the support set to neural networks.
They use only neural networks, and do not explicitly minimize the loss of the support set when adapting to new tasks.
Thus, they might have difﬁculty performing effective adaptation.
In contrast, the proposed method explicitly minimize the loss of the support set by solving the optimization problem for feature selection, which achieves accurate adaptation.
One few-shot learning method for unsupervised feature selection has recently been proposed that selects features from few unlabeled instances [25].
Unlike our method, it cannot use labels.
This method is categorized as an encoder-decoder method, which selects features with only neural networks.
Thus, it might be difﬁcult to perform effective adaptation.
We show that our method outperforms this method in our experiments.
Some methods use deep kernels, which combine neural networks with kernels, in the meta-learning framework [41, 20, 49].
They are designed for classiﬁcation or regression problems, and cannot be used for feature selection.
We design novel kernel functions with neural networks that are appropriate for feature selection, which are described in Section 4.
Preliminary
In this section, we brieﬂy review HSIC, which can measure non-linear dependencies between two random variables without going through density estimation.
Let X
and Y be two random variables on domains X and Y , respectively, and sample ( x, y ) is drawn from a joint distribution p ( x, y ) .
HSIC is given by
[Equation]
where K : X × X → R and L : Y × Y → R are positive-deﬁne kernels, x (cid:48)
( y (cid:48) ) is a i.i.d.
copy of x ( y ) , and E x,x (cid:48) ,y,y (cid:48) means the expectation over independent pairs ( x, y ) and ( x (cid:48) , y (cid:48) )
drawn from p ( x, y ) .
When kernels are characteristic such as RBF kernels, HSIC takes zero if and only if X and
Y are statistically independent [15, 47].
In other words, a large value of HSIC indicates that there is a strong dependency between X and Y .
Therefore, HSIC can be used as a dependence measure.
Proposed Method 4.1 Problem Formulation
Let D t := { x tn , y tn } N t n =1 be a set of labeled instances in the t -th task, where x tn = ( x (1) tn , . . .
, x ( D ) tn ) (cid:62) is the D -dimensional feature vector of the n -th instance in the t -th task and y tn is its response.
The proposed method can be applied in both regression and classiﬁcation settings.
In this section, for simplicity, we explain our method with regression problems, i.e., y tn ∈ R .
Our model for classiﬁcation is described in Appendix B. We assume that each task has the same feature vector size
D , but each joint distribution p t ( x , y ) can differ, which is the standard assumption used in transfer learning studies [38].
Suppose that T source tasks D :=
{D t } Tt =1 are given at the training phase.
At the testing phase, we are given a small number of labeled instances S := { x n , y n } N S n =1 , called a support set, in a target task, which is different from source tasks.
Our goal is to select features that are appropriate for the target task from S .
4.2 Model
We explain our model that outputs the importance of each feature given S .
By selecting the top- K features in accordance with the importance values, we can select K relevant features.
Optimization Problem To estimate importance values α =(
α 1 , . .
. , α D ) ∈ R D ≥ 0 , we consider the following optimization problem with support set S :
[Equation]
where (cid:107) · (cid:107) F is the Frobenius norm, (cid:107) · (cid:107) 1 is the (cid:96) 1 norm, λ ∈ R > 0 is a regularization parameter, ¯
L S :
= Γ N S L S Γ N S ∈ R N S × N S is the centered Gram matrix, ( L S ) ij :
=
L ( S )( y i ,
y j ) ∈
R is the
S -dependent kernel for the response in S , Γ N S :
= I N S − 1 N S 1 N S 1 (cid:62) N S is the centering matrix,
I N S is the N S -dimension identity matrix, 1 N S is the N S -dimension vector whose elements are all one, ¯ K ( d ) S :
= Γ N S K ( d ) S Γ N S ∈ R N S × N S is the centered Gram matrix for the d -th feature, and ( K ( d ) S ) ij :=
K ( d ) ( S )( x ( d ) i , x ( d ) j ) ∈ R is the S -dependent kernel for the d -th feature in S .
Since kernels K ( d ) ( S ) and L ( S ) depend on support set S , we can reﬂect information of the support set to the kernels so that our model can accurately select features from the support set.
The speciﬁc form of the kernels will be explained later.
We explain the interpretation of the problem (2).
The ﬁrst term in Eq.
(2) can be rewritten as
[Equation]
where (cid:92) HSIC S
( X ( d ) , Y ) :
= 1 N
2 S Tr( ¯ K ( d ) S ¯ L S ) is the empirical estimate of HSIC with support set
S [15], Tr( · ) is the trace, and C is a constant term.
If there is a strong dependency between the d -th feature X ( d ) and response Y , (cid:92) HSIC S
( X ( d ) , Y ) takes a large value and α d should also take a large value to minimize Eq.
If X ( d ) is independent of Y , (cid:92) HSIC S
( X ( d ) , Y ) is close to zero and thus α d tends to become zero by the (cid:96) 1 regularizer.
Thus, we can regard α d as the importance of the d -th feature.
Moreover, if X ( d ) and X ( d (cid:48) ) are strongly dependent, i.e., redundant features, (cid:92)
HSIC S ( X ( d ) , X ( d (cid:48) ) ) takes a large value and thus, either of α d or α (cid:48)
d tends to zero.
This means that redundant features are automatically removed.
Therefore, by minimizing Eq.
(2), we can select relevant features for predicting the response while removing redundant features.
However, when there are insufﬁcient instances, the empirical estimate of HSIC will become inaccurate, and thus, relevant features will be difﬁcult to accurately select.
Support Set-dependent Kernel Functions
To alleviate this problem, we propose kernel functions appropriate for meta-learning feature selection with a small number of labeled instances.
First, we consider obtaining a latent representation of a task from support set S by using permutation-invariant neural networks [56]:
[Equation]
where f and g are any feed-forward neural network, and [ · , · ] is concatenation.
Since averaging operation is permutation-invariant, the neural network in Eq.
(4) outputs the same vector even when the order of instances in the support set varies.
Moreover, this neural network can handle different numbers of instances.
Thus, the neural network in Eq.
(4) is well deﬁned as functions for set inputs.
Latent task representation z contains information of the empirical distribution of instances in support set S .
Although we use averaging operation for simplicity, we can use any other permutation-invariant function such as summation, max [56], and set transformer [28] to obtain latent task representations.
By using latent task representation z , our kernel functions are deﬁned by
[Equation]
where β ( z ) :=
( β (1) ( z ) , . .
β ( D ) ( z ))
∈ R D> 0 and σ ( z ) :
=
( σ (1) ( z ) , . . .
, σ ( D ) ( z )) ∈ R D> 0 are modeled by feed-forward neural networks taking z as input, respectively, and σ y > 0 is a hyperparameter.
We use a S -independent RBF kernel for y for simplicity although any S -dependent kernel can be used.
β ( d ) ( z ) and σ ( d ) ( z ) can vary in accordance with z .
Intuitively, we expect to obtain inductive bias appropriate for feature selection in each task, i.e., each feature is easily selected or it is not selected, by meta-learning β ( z ) and σ ( z ) .
Speciﬁcally, for example, when β ( d ) ( z ) is learned to zero, both (cid:92) HSIC S
( X ( d ) , Y ) and (cid:92) HSIC S
( X ( d ) , X ( d (cid:48) ) ) become zero in Eq.
Thus, corresponding α d tends to be made zero by the (cid:96) 1 regularizer.
Similarly, σ ( d ) ( z ) can be learned to ﬁt the scale of the feature in each task.
These kernel parameters are meta-learned such that our model can accurately select relevant features from a small number of labeled instances, which will be explained in detail in the next subsection.
We note that the kernels in Eq.
(5) are characteristic if β ( d ) ( z ) and σ ( d ) ( z ) are bounded since RBF kernel is characteristic.
Support Set Adaptation by ISTA To solve the optimization problem (2), we use ISTA, which is a simple and efﬁcient algorithm for solving (cid:96) 1 -regularized problems in a mathematically elegant way [8].
By using ISTA, we can derive the closed-form update steps for optimization.
Speciﬁcally, we can solve the problem (2) by repeating the following steps:
[Equation]
where A := (cid:104) vec( ¯ K (1) S ) , . . .
, vec( ¯ K ( D ) S )
(cid:105) ∈ R N 2 S × D , b :=
vec(¯ L S )
∈ R N 2 S , where vec( · ) is the vectorization operator, R λµ is a non-negative soft-thresholding operator, and [ · ] + = ReLU( · ) .
The
Require:
Datasets in source tasks D , support set size N S , query set size N Q , step size for ISTA µ , and the number of iterations for ISTA
I
Ensure:
Parameters of our model Θ 1: repeat 2:
Randomly sample task t from { 1 , . . .
, T } 3:
Randomly sample support set S with size N S from D t 4:
Randomly sample query set Q with size N Q from D t \ S 5:
Calculate latent task representation z from support set S by Eq.
Calculate kernel matrices (5) with task representation z and support set S 7: for l := 1 to I do 8:
Update coefﬁcient parameters (importance values) α by Eqs.
(6) and (7) 9: end for 10:
Calculate the loss J ( Q ; S ) on query set Q by Eqs.
(9) and (10) 11:
Update parameters Θ with the gradients of the loss J ( Q ; S ) 12: until End condition is satisﬁed; ﬁrst step (6) minimizes the ﬁrst term in Eq. (2) by using gradient descent with step size µ > 0 . The second step (7) reﬂects the second term in Eq. (2) and non-negative constants.
It shrinks α towards zero while satisfying non-negative constraints.
We treat the initial coefﬁcient parameters (importance values)
α 0 for ISTA as trainable parameters like the MAML [9].
By learning good initial parameters with multiple tasks, we can obtain appropriate parameters for each target task even with a few iterations of ISTA.
4.3 Training
We explain our training procedure.
In this subsection, symbol S is used as a support set in source tasks.
In our model, the parameters to be estimated, Θ , are parameters of neural networks f , g , β , σ , initial coefﬁcient parameters α 0 , and regularization parameter λ .
We estimate these parameters by minimizing the expected test error on features selected from support set S , where support set and test instances Q , called the query set, are randomly generated from source tasks:
[Equation]
where
[Equation]
where α ∗ =
( α ∗ 1 , . .
, α ∗ D ) ∈ R D ≥ 0 is coefﬁcient parameters obtained from S by ISTA, ¯ L Q :=
Γ N Q LΓ N Q ∈ R N Q × N Q is the Gram matrix for the response on query set Q with size N Q , L ij :=
L ( y i ,
y j ) ∈
R is the kernel for the response in Q , ¯ K ( d ) Q :=
Γ N Q K ( d ) Γ N Q ∈ R N Q × N Q is the Gram matrix for the d -th feature on Q , and K ( d ) ij :=
K ( d ) ( x ( d ) i , x ( d ) j ) ∈ R is the kernel for the d -th feature in Q .
When query size N Q is sufﬁciently large, the empirical estimate of HSIC with the RBF kernels becomes accurate [15].
Thus, we used the following RBF kernels for all features in Eq.
(9) as in [53],
[Equation]
where σ x > 0 is a hyperparameter.
For the kernel of the response L , we used the same RBF kernel as that for the support set adaptation in Eq.
By using the same kernels for all features, we can treat each feature equally.
That is, the proposed method can fairly evaluate the importance of each feature obtained from S on query set Q .
Algorithm 1 shows the pseudocode for our training procedure.
For each iteration, we randomly sample task t from source tasks (Line 2).
From D t , we randomly sample support set S and query set
Q (Lines 3 and 4).
We calculate task representation z from S (Line 5), and the S -dependent kernel matrices (Line 6).
We iteratively update coefﬁcient parameters α with support set S by using ISTA (Lines 7 – 9).
Using the adapted coefﬁcient parameter α ∗ , we calculate loss J ( Q ; S ) on query set Q (Line 10).
Lastly, the parameters of our model Θ are updated with the gradient of the loss (Line 11).
Since each update step of ISTA (Eqs.
(6) and (7)) is obtained as a closed-form that is differentiable, the whole training can be performed by using gradient-based optimization methods.
This training explicitly matches the selected features from few labeled instances S and those from sufﬁcient labeled instances Q on the basis of HSIC.
Thus, the learned model can select relevant features for each target task given a small number of labeled target instances (speciﬁcally, by running lines 5 through 9 of Algorithm 1 with the target instances.)
Experiments
In this section, we demonstrate the effectiveness of the proposed method using one synthetic and three real-world datasets.
5.1 Comparison Methods
We compared the proposed method with seven existing feature selection methods: Lasso, HSIC
Lasso (HLasso), feature selection with stochastic gate (STG) [54], STG-S, ﬁne-tuning variant of
STG (STG-FT), meta-learning method for feature selection based on Gumbel-softmax distribution (MetaGS) [25], and a supervised variant of MetaGS (SMetaGS).
Lasso is a well-known linear feature selection method.
HLasso is a kernel-based non-linear feature selection method.
STG is a neural network-based non-linear feature selection method that has been reported to outperform various existing methods in various datasets [54].
STG learns a mask vector that determines which features are selected while simultaneously learning predictors.
Lasso, HLasso, and STG uses the target support set for feature selection.
STG-S trains the model of STG with labeled instances in source tasks.
STG-FT ﬁne-tunes the model trained by STG-S with the target support set.
MetaGS is a neural network-based encoder-decoder meta-learning method for unsupervised feature selection that uses the reconstruction error for feature selection.
SMetaGS is a supervised extension of MetaGS that uses the cross-entropy loss as the objective function.
(S)MetaGS meta-learns the mask vector for feature selection.
Speciﬁcally, the task-speciﬁc mask is modeled by the permutation-invariant neural network that takes support set (a few labeled data) as input and is meta-learned to improve test feature selection performance.
All methods except for MetaGS use label information.
The proposed method, STG-FT, MetaGS, and SMetaGS use both the target support set and source instances for feature selection.
5.2 Simulation Experiments
Data We ﬁrst used simple synthetic regression problems with 100 -dimensional feature and onedimensional response spaces.
Response y in the t -th task was generated from the following non-linear function:
y = w t 1 x ( i 1 ) exp(2 w t 2 x ( i 2 ) )
+ w t 3 ( x ( i 3 ) ) 2 , where task-speciﬁc parameters w t 1 , w t 2 , and w t 3 were uniform randomly generated from [ − 1 , 1] , and features ( x (1) , . . .
, x (100) ) was generated from 100 -dimensional standard Gaussian distribution.
Feature indexes ( i 1 , i 2 , i 3 ) represent important feature indexes in the t -th task, and were uniform randomly selected for each task from 10 predeﬁned feature indexes { (1 , 2 , 3) , (4 , 5 , 6) , . .
We generated 300 labeled instances for each task.
We standardized the values of response.
We randomly created 300 training, 20 validation, and 100 target tasks.
Settings We evaluated the average fraction of correctly selected features on each target task with different numbers of target support instances within { 10 , 20 , 30 } .
For each method, we selected the top-3 features by ranking their importance scores.
For the proposed method, we selected the hyperparameter on the basis of mean validation loss.
For comparison methods, the best test results are reported from their hyperparameter candidates.
The details of the experimental settings such as hyperparameter candidates are described in Appendix C.
Results Table 1 shows average and standard errors of the fraction of correctly selected features with different numbers of target support instances.
The proposed method clearly outperformed the other methods.
HLasso outperformed Lasso since it is the non-linear method that ﬁts on the non-linear synthetic data.
Although STG is also a non-linear method, it did not perform well since training data was too small to train its neural networks.
STG-FT, which is the ﬁne-tuning method, improved
[Table]
[Figure] performance compared with STG-S. Although MetaGS and SMetaGS are meta-learning methods like the proposed method, it performed worse.
This result suggests that task-speciﬁc features are difﬁcult to select from few instances only by using neural networks.
The proposed method can select features by explicitly solving Eq.
(2) with useful knowledge in source tasks, and thus, it worked well.
Figure 2 shows one example of feature importance α ∗ and kernel parameters, β ( z ) and σ ( z ) , obtained by the proposed method.
As explained in Section 4, β d ( z ) is expected to indicate how likely the d -th feature is to be selected before solving Eq.
That is, when β d ( z ) is small, the d -th feature is less likely to be selected.
β d ( z ) took a small value when d was in { 31 , . .
Since there are no important features within { 31 , . .
, 100 } in all tasks of this synthetic data as described above, β ( z ) was learned as expected.
Similarly, σ d ( z ) tends to take different values in the two ranges { 1 , . .
. , 30 } and { 31 , . .
This result means that appropriate scales were automatically learned for feature groups that are easy to select (i.e., d ∈ { 1 , . .
and those that are not (i.e., d ∈ { 31 , . .
After adaptation to the support set, α ∗ d took a large value when d was in { 1 , 2 , 3 } , which is the true important feature indexes of this example.
This result shows that the proposed method worked well as expected.
We also conﬁrmed that the proposed method works well on more difﬁcult synthetic data in Appendix D.10.
5.3 Real-world Data Experiments
Data We used there real-world datasets: Mnistr 1 , Isolet 2 , and IoT 3 , which have been widely used in previous studies [3, 25, 54].
Mnistr is derived from Mnist by rotating the images [14].
This dataset has six tasks (six rotation angles), and its feature dimension is 256 .
We created the binary classiﬁcation problem by regarding even digits as positive and odd digits as negative, respectively.
Isolet consists of letters (classes) spoken by 150 speakers, and speakers are grouped into ﬁve groups (tasks) by speaking similarity.
Each instance is represented as a 617-dimensional vector.
We created the binary classiﬁcation problem by regarding class labels { 1 , . . .
, 13 } as positive and { 14 , . .
. , 26 } as negative, respectively.
IoT is real network trafﬁc data, which are gathered from nine IoT devices (tasks) infected by malware [35].
Each task has normal and malicious trafﬁc instances, and each instance is represented by a 115-dimensional vector.
For each task, we randomly used 500 normal and 500 malicious instances.
For all datasets, we standardized the values of each response and feature.
We randomly select one task for the target task, one task for the validation task, and the rest for the source tasks.
For each dataset, we created 10 different target/validation/source task splits.
[Table] [Table] Settings To evaluate the quality of selected features, we trained classiﬁers by using the target support set with the selected features and evaluated the test classiﬁcation accuracy, which is the standard evaluation protocol used in the previous feature selection studies [53, 47, 29, 54].
Since the proposed method, HLasso, and MetaGS cannot directly classify test instances, we used support vector machines with RBF kernels as the classiﬁers for all methods for a fair comparison.
We did not use neural networks as classiﬁers since they have many hyperparameters to be tuned, and target training instances are too small to train without overﬁtting.
The details of the settings such as hyperparameters are described in Appendix C.
Results Table 2 shows average and standard errors of test accuracies over different numbers of selected features within { 10 , 30 , 50 } when changing the numbers of target support instances within { 10 , 20 , 30 } .
The proposed method achieved the best average accuracies on all datasets.
Lasso,
HLasso, and STG, which used only the target support instances, did not perform well since it was difﬁcult to select relevant features from only few labeled instances.
STG-FT outperformed these methods for all datasets, which indicates that using information on related tasks is useful to alleviate the lack of target data.
MetaGS did not work well on all datasets, which indicates the effectiveness of label information.
SMetaGS did not perform well with Mnistr and IoT since the support set adaptation was difﬁcult with only neural networks.
In contrast, the proposed method worked well by appropriately unifying neural networks and kernel methods in a meta-learning framework.
[Figure] Figure 3 shows average and standard errors of test accuracies when changing the number of selected features K on Isolet.
The proposed method performed well across different K values.
In particular, when K is small, the proposed method outperformed the others by a large margin.
In general, when K is small, it is more important to select informative features to improve performance.
Thus, this result suggests that the proposed method was able to assign high importance to informative features.
Table 3 shows the results of ablation studies.
We compared our model with seven models: w/o Latent, w/o Feature, w/o LFeature, w/o S -kernel, w/o Initial,
w/ Deep, and w/o S -adapt.
For support set adaptation,
[Figure] w/o Latent uses the kernels without task representation z , i.e., it uses the task-invariant featurespeciﬁc kernel parameters β ( d ) and σ ( d ) in Eq.
w/o Feature uses the task-speciﬁc feature-invariant kernel parameters β ( z ) and σ ( z ) for all features.
w/o LFeature uses the task and feature-invariant kernel parameters β and σ for all features.
w/o S -kernel uses the S -independent kernels of Eq.
This method adapts to the support set by only learning initial coefﬁcient parameters.
w/o Initial uses the same kernels as the proposed method, but does not learn initial coefﬁcient parameters. w/ Deep uses the deep kernels [51], K ( d ) ( S ) ( x ( d ) i , x ( d ) j ) := exp( − ( h ( x ( d ) i , z ) − h ( x ( d ) j , z )) 2 ) , where h is a three-layered feed-forward neural networks with one output node.
w/o S -adapt does not perform support set adaptation (i.e., I = 0 ).
This method learns task-invariant initial coefﬁcient parameters for feature selection on any tasks.
The proposed method showed the best average results over all datasets.
Especially, it outperformed w/o Latent, which indicates the usefulness of inferring task-speciﬁc kernels for each feature from the support set.
In IoT, w/o Latent worked well.
This was probably because each task has similar properties as described in [26], and thus, task-invariant kernels might be sufﬁcient.
w/o Feature and w/o LFeature performed worse than the proposed method, which indicates that feature-speciﬁc kernel parameters are useful to select features.
Since w/o S -kernel and w/o Initial did not work well, we found that both the kernel and initial parameter adaptations need to be considered simultaneously.
w/ Deep did not perform well.
This result indicates that the form of our kernel function in Eq.
(5) is essential for our problems.
The proposed method also outperformed w/o S -kernel in Mnistr and Isolet, which means that support set adaptation needs to be performed to select task-speciﬁc features.
Overall, these results indicate the effectiveness of our model design.
We visualized the selected features by the proposed method with Mnistr dataset.
Figure 4 showsselected features on each task of Mnistr by the proposed method when N S = 30 .
Since the selected features differ between tasks, we found that the learned model can select task-speciﬁc features by adapting to the support set on each task as expected.
In addition, the selected features can identify digits ‘3’ and ‘8’ on each task since they captured the bottom left white pixels of digit ‘8’.
Therefore, the proposed method can identify relevant features that are responsible for the responses.
Conclusion
In this paper, we proposed a few-shot learning method for feature selection based on Hilbert-Schmidt
Independence Criterion (HSIC).
Experiments showed that the proposed method outperformed existing feature selection methods when there was insufﬁcient data in target tasks.
Although the proposed method is useful as a feature selection method, it cannot train a classiﬁcation or regression model that can predict unseen instances simultaneously.
As future work, we plan to combine HSIC
Lasso and neural network-based classiﬁcation/regression model in our meta-learning framework to simultaneously perform feature selection and classiﬁcation/regression model learning from a few labeled data.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Please see Section 6 and Appendix E. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] Please see Appendix F. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The code is proprietary.
The datasets and experimental procedures are described in Section 5.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Please see Section 5 and Appendix C. (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] We performed experiments multiple times with different training/validation/testing splits and reported average results with standard errors in Tables 1, and 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Appendix C. 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Please see Section 5.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
The datasets used in the paper do not contain such information.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Merits of Feature Selection
We explain the merits of feature selection by comparing it to feature extraction.
Feature extraction aims to learn instance representations to improve the performance of prediction models such as classiﬁcation or regression models [27, 16].
The learned representations are represented by the dense combination of all features.
Thus, they cannot explain/analyze the dataset itself or reduce the cost of collecting irrelevant or redundant features by identifying the important feature subset from original features, which are the major objectives (merits) of feature selection.
Existing meta-learning methods perform feature extraction to improve the classiﬁcation or regression performance [17, 9, 46, 44].
Therefore, they cannot be used for feature selection purposes.
To the best of our knowledge, our work is the ﬁrst attempt at meta-learning for supervised feature selection, which enables us to select task-speciﬁc important feature subsets from a few labeled data.
B Proposed Model for Classiﬁcation Problems
We explain how to apply our model to classiﬁcation problems.
To this end, we slightly modify kernel functions for responses and the way to construct latent task representations described in the main paper.
For the kernel functions for responses y in Eq.
(5), following a previous study [53], we can use delta kernels, L ( S )
( y i , y j ) :
= 1 |S yi | if y i = y j and zero otherwise, where S c := { ( x , y ) ∈ S| y
= c } .
We can also use this kernel for the objective function with the query set in Eq.
For latent task representation z in Eq.
(4), we change the construction procedure of z depending on whether the response spaces are the same across tasks or not.
When the response space is the same across tasks, we can use the same neural network architecture as that for regression problems by using one-hot vector representations of responses (classes) y .
When the response spaces are different across tasks, we cannot use the one-hot vector representations for classes because we do not know classes in unseen target tasks at the training phase.
To deal with this, for each support set, we ﬁrst calculate class-speciﬁc task representation
z c where c is a class in the support set,
[Equation]
and then, we construct z from the class-speciﬁc task representations by using permutation-invariant neural networks, [Equation] u
[Equation]
, where u and v are any feed-forward neural networks, and C is the total number of classes in the support set.
By the average operation, we can obtain the same vector z
even if the order of the class-speciﬁc task representations varies.
C Experimental Setting Details
We explain the neural network architectures used in our simulation and real-world data experiments.
For the proposed method, we used a three(two)-layered feed-forward neural network for latent task representations f ( g ) in Eq.
The hidden and output sizes of f were 32 and ReLU activations were used.
The output size of g was ﬁxed to 4 ( J = 4) .
For functions of kernel parameters β and σ in Eq.
(5), two-layered feed-forward neural networks with 32 hidden nodes and ReLU activations were used, respectively.
To ensure the non-negativeness of β and σ , softplus functions were used for output nodes.
For prediction networks of STG, STG-S, STG-FT, MetaGS, and SMetaGS, we used three-layered neural networks with 32 hidden nodes and ReLU activations, respectively.
For
MetaGS and SMetaGS, we used the temperature annealing from 10 to 0.01 as in the original paper.
All methods were implemented on the basis of Pytorch [40].
All experiments were conducted on a
Linux server with an Intel Xeon CPU and a NVIDIA GeForce GTX 1080 GPU.
For the simulation experiments, the hyperparameters of the proposed method was determined on the basis of mean validation loss.
Step size for ISTA µ was selected from { 10 − 3 , 10 − 2 , 10 − 1 , 1 } .
The number of iterations for ISTA I was selected from { 1 , 3 , 10 } .
Gaussian widths σ x and σ y were set to one.
For all comparison methods, the best test results are reported from the following hyperparameter candidates.
For Lasso, HLasso, STG, STG-T, and STG-FT, the regularization parameter was chosen from { 10 − 6 , 10 − 5 , . . .
For HLasso, Gaussian width was chosen from { 0 . 1 , 0 . 5 , 1 , 2 } .
For STGFT, the number of ﬁne-tune iterations was chosen from { 10 , 50 , 100 } .
For all neural network-based methods, we used the Adam optimizer [23] with a learning rate of 10 − 3 .
The mini-batch size was(for the proposed method, we set N Q + N S =64).
The validation loss was also used for early stopping to avoid over-ﬁtting, where the maximum number of training iterations was 10,000.
For the real-word data experiments, we used the same hyperparameter candidates used in the synthetic dataset.
For the proposed method, we used the delta kernel for kernel functions of responses as described in Appendix B. Since each dataset used in our experiments has the same response (class) space across tasks, we used one-hot vector representations for classes to obtain latent task representations as described in Appendix B. For all methods, to tune their hyperparameters and the regularization parameter of support vector machine (SVM), we used 10 labeled instances from the target task as validation data, and its validation accuracy was used.
We used the scikit-learn implementations of the
SVM.
The regularization parameter candidates of SVM were { 10 − 4 , 10 − 3 , . .
The Gaussian width of the RBF kernel was automatically ﬁtted to training data by scikit-learn.
D Additional Experimental Results D.1 Test Accuracies with Different Selected Feature Sizes
Figure 5 shows the average and standard errors of test accuracies over different target support instances when changing the selected feature size K on each dataset.
The proposed method consistently performed well with different K values on Mnistr and Isolet.
In particular, when K is small, the proposed method outperformed the others by a large margin.
In general, it is more important to select informative features when K is small.
Therefore, this result indicates that the proposed method can select important features accurately.
For IoT, the proposed method, STG-S, and STG-FT showed similar results.
Especially, these methods performed well when K is small.
This would be because this dataset has some noisy features, and thus they deteriorate performance.
Overall, the proposed method performed well.
[Figure] [Figure] D.2 Dependency of the Dimension of Latent Task Representations
We investigated the dependency of the dimension of latent task representations z on the proposed method, which are used for inferring the task-speciﬁc kernel functions for feature selection.
Figure 6 shows the average and standard errors of test accuracies of the proposed method with different dimensions of latent task representations J within { 2 , 4 , 8 , 16 , 32 } .
The proposed method constantly outperformed HLasso and performed relatively stably with different J values.
D.3 Dependency of the Iteration Numbers of ISTA
We investigated the dependency of the iteration numbers of ISTA on the proposed method.
Figure 7 shows the average and standard errors of test accuracies of the proposed method with different iteration numbers I within { 1 , 3 , 5 , 10 , 15 } .
The proposed method consistently outperformed HLasso and performed relatively stably with different I values.
This result indicates that the proposed method can select task-speciﬁc features with a few training iterations of ISTA.
D.4 Dependency of the Step Size of ISTA
We investigated the dependency of the step size of ISTA on the proposed method.
Figure 8 shows the average and standard errors of test accuracies of the proposed method with different step sizes µ within { 10 − 3 , 10 − 2 , 10 − 1 , 1 } .
The proposed method consistently outperformed HLasso with different µ values.
The appropriate step sizes varied across datasets.
For Mnistr and Isolet, small step sizes performed well.
In contrast, for IoT, large step sizes tended to perform well.
D.5 Computation Cost
We investigated the computation time of the proposed method.
We used a computer with a 2.20 GHz
CPU.
The target support set size was set to 30 and the iteration number of ISTA was set to 3 .
Tablesand 5 show the training time with source tasks and feature selection time on a target task with Mnistr, [Figure] [Figure] [Figure]
[Figure]
cc u [Figure] c [Figure]
c y
OursHLasso
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
cc u
[Figure]
y
[Figure]
[Figure]
Ours
STG-S
STG-FT
MetaGS
SMetaGS 170.
[Table]
[Table]
respectively.
Since Lasso, HLasso
[Table]
and STG do not use source tasks, we omitted these methods in
Table 4.
Since STG-S directly uses selected features from source tasks for the target task, we omitted this method in Table 5.
Although the proposed method took time for training with source tasks, it was able to select features on the target task as fast as other methods
[Table]
D.6 Detailed Analysis of Learning Initial Coefﬁcient Parameters
We showed the effectiveness of learning initial coefﬁcient parameters α 0 in the ablation study (Table 3).
We further analyzed its effect in details.
Speciﬁcally, we investigated whether w/o Initial, which is our method without learning initial coefﬁcient parameters, can achieve good performance when the number of the iterations for ISTA I is increased.
Table 6 shows the mean test accuracy and the training time with source tasks.
As the iteration number I was increased, the training time [Table] [Table] of w/o Initial signiﬁcantly increased.
However, w/o Initial with these higher iteration numbers ( I = 10 , 20 , 30 , 40 , 50 ) performed worse than the proposed method with I = 3 .
This would be because many iterations expand the computation graph of neural networks and it might makes the optimization difﬁcult.
This result shows the effectiveness of learning initial coefﬁcient parameters.
D.7 Experiments on Tasks with Different Response Spaces
In the main paper [Table] we conﬁrmed that the proposed method works well with the real-world datasets, in which the response (class) space is the same across tasks.
However, the proposed method is also applicable to the case that the response space differs between tasks as described in Appendix B.
We additionally evaluated the proposed method on Mnistr with different response spaces.
In this experiment, for each task, we created a binary-classiﬁcation problem by randomly choosing half of the 10 digits as positive and the rest as negative.
That is, the positive (negative) classes for different tasks consist of different digits.
Table 7 shows the results.
In this experiment, we used the identity functions for task representation neural networks u and v in Appendix B and set the dimension of class-speciﬁc task representation
z c to two.
Due to the difﬁculty of the problem, all methods degraded performance compared with the case of the same response space (Table 2).
However, the proposed method outperformed the other methods.
These results indicate that the proposed method works well even when the response spaces are different across tasks.
D.8 Classiﬁcation without Feature Selection
To evaluate the effectiveness of feature selection, we investigated classiﬁcation performance without feature selection on the real-world datasets.
Table 8 shows the results of SVM without feature selection (NoFS).
The proposed method that trains SVM with selected features clearly outperformed
NoFS on all datasets.
This is because training data are too small to train SVMs with original highdimensional features.
Feature selection often improves performance when there are insufﬁcient training data.
D.9 Comparison with Multi-task Extensions of Lasso and HSIC Lasso
To further demonstrate the effectiveness of the proposed method, we considered multi-task extensions of Lasso and HSIC Lasso that use labeled data in both source and target tasks for training.
We call these methods MLasso and MHLasso, respectively.
Table 9 shows the results.
Although MLasso and MHLasso were able to improve performance by using labeled data in source tasks, the proposed method performed better than MLasso and MHLasso on all datasets.
Since MLasso and MHLasso are not designed for few-shot feature selection, they did not work well. [Table] [Table] [Table]
[Table]
D.10 Simulation Experiments with More Difﬁcult Settings
In the main paper, we used the synthetic data, in which the important feature indexes were selected from { (1 , 2 , 3) , (4 , 5 , 6) , . . .
, (28 , 29 , 30) } in both source and target tasks.
We additionally investigated the case that the important feature indexes of target tasks are randomly selected from { 1 , . .
. , 30 } such as (3 , 10 , 26) and (5 , 7 , 29) .
This is a more challenging but realistic setting.
Table 10 shows the results.
The proposed method clearly outperformed the other methods even in this difﬁcult setting.
In addition, we investigated whether the proposed method works well on high-dimensional data.
We created high-dimensional data by changing the feature dimension of the synthetic data described in the main paper from 100 to 3000 .
Table 11 shows the results.
The proposed method clearly outperformed
Lasso and HLasso by a large margin.
Especially, although Lasso and HLasso signiﬁcantly deteriorated the performance compared to the case with 100 features (Table 1) due to the curse of dimensionality, the proposed method was able to maintain the performance to some extent by meta-learning.
These results indicate that the proposed method works well on high-dimensional data.
D.11 Experiments with Larger Target Support Instances
This paper focuses on feature selection from a small number of target instances.
However, it is interesting to investigate the performance of the proposed method when there are larger target instances.
Table 12 shows the results with number of target support instances N S = 50 and 80 .
When
N S = 50 , the proposed method performed better the others.
When N S = 80 , the proposed method and HLasso performed almost the same.
This result is reasonable because when there is enough target data, the effect of source data generally becomes small.
E Limitations
The proposed method has a few limitations to overcome.
First, the proposed method uses multiple source tasks to improve feature selection performance on unseen target tasks.
However, when source and target tasks are signiﬁcantly different, the performance on the target tasks risks degrading.
This is known as “negative transfer,” and overcoming it is one of the important problems in transfer/metalearning studies.
Developing methods to automatically remove negative effects of such tasks is a promising research direction.
Second, the proposed method assumes that the input feature space is the same across all tasks, which may hinder its practicality in some applications.
However, this assumption is not unique to the proposed method but is common to almost all meta-learning methods [46, 44, 9, 4, 43, 25, 26].
Also, multi-task and transfer feature selection methods usually require the same assumption [37, 57, 2].
Even if the input feature space is the same, there are many possible practical applications.
For example, as mentioned in Section 1, imagine that a company wants to analyze the features of its product ( x ) that affect sales (y) in each store/region (task).
Within the same company, product x is often registered in the database using the same features (price, ingredients, etc).
If we want to analyze important features from text documents in different tasks, the feature (word) space is identical within the same language.
Developing meta-learning methods for feature selection that can handle different feature spaces is also a promising challenge.
Third, HSIC-based feature selection methods, including the proposed method, cannot capture some nonlinear feature interactions although HSIC can capture the nonlinear relationship between one variable x and the response y .
For example, if the predictive function is an XOR of two variables x 1 and x 2 , HSIC cannot capture such interaction.
Extending the proposed method to capture such relationships is also one of the important research directions.
F Social Impacts
The proposed method has some potential risks to be addressed when it is deployed to real-world applications.
First, although we experimentally demonstrated the proposed method outperformed existing methods, it is not perfect; that is, it may select irrelevant features, which may lead to wrong analysis/interpretation in some cases.
To mitigate this, people can use the proposed method as a support tool for their detailed analysis/interpretation of data.
Second, the proposed method needs to access datasets obtained from multiple tasks like almost all transfer/meta-learning methods.
When each dataset is provided from different owners such as companies, sensitive information in the dataset risks being stolen and abused by malicious people that use the proposed method.
To evade this risk, we suggest promoting research for developing transfer/meta-learning without accessing raw datasets.