S ECOND -O RDER U NSUPERVISED F EATURE S ELECTION VIA K NOWLEDGE C ONTRASTIVE D ISTILLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised feature selection aims to select a subset from the original features that are most useful for the downstream tasks without external guidance information.
While most unsupervised feature selection methods focus on ranking features based on the intrinsic properties of data, they do not pay much attention to the relationships between features, which often leads to redundancy among the selected features.
In this paper, we propose a two-stage Second-Order unsupervised Feature selection via knowledge contrastive disTillation (SOFT) model that incorporates the second-order covariance matrix with the ﬁrst-order data matrix for unsupervised feature selection.
In the ﬁrst stage, we learn a sparse attention matrix that can represent second-order relations between features.
In the second stage, we build a relational graph based on the learned attention matrix and perform graph segmentation for feature selection.
Experimental results on 12 public datasets show that SOFT outperforms classical and recent state-of-the-art methods, which demonstrates the effectiveness of our proposed method.
I NTRODUCTION
In the digital world, huge amounts of high-dimensional data (Guyon et al., 2004; Bajwa et al., 2016;
Yang et al., 2019; Maoying et al., 2020) are captured every day.
Due to the existence of irrelevant or redundant features, data in high dimensions may signiﬁcantly increase the computational costs and bring challenges for efﬁcient and effective data management.
Dimensionality reduction is one of the most well-known techniques to address the above issue, which can be categorized into feature transformation and feature selection.
Feature transformation, also known as representation learning, aims to project the original high-dimensional features into a new low-dimensional feature space.
The new feature space is usually a linear or nonlinear combination of the original features and does not have semantic meanings, so it is hard to be interpreted.
On the other hand, feature selection methods try to select a subset of relevant features from all available features based on a predeﬁned criterion, maintaining physical meanings of the original features for better interpretability.
Without label information, unsupervised feature selection methods aim to select a feature subset that can preserve the intrinsic structure of the whole feature set accurately.
There are many algorithms designed to solve the unsupervised feature selection problem.
ReliefF (Robnik-ˇSikonja & Kononenko, 2003), HSIC (Gretton et al., 2005), Laplacian Score (He et al., 2005), SPEC (Zhao & Liu, 2007),
SPFS (Zhao et al., 2011) evaluate features by their capability in preserving the pairwise sample similarity.
UDFS (Yang et al., 2011), FSASL (Du & Shen, 2015), and TSFS (Mirzaei et al., 2020) employ pseudo labels as the supervision to guide the feature selection along with a sparse constraint.
Most of these methods apply the linear feature selection matrices and select the representative features by ranking their feature weight vector.
Such operations treat the feature set independently and fail to tackle the complex high-order relationship (Zhang et al., 2017; Zhu et al., 2020) among original features which inevitably brings in the redundancy among the selected features.
Contributions : We propose a two-stage Second-Order unsupervised Feature selection via knowledge contrastive disTillation (SOFT) model that incorporates the second-order covariance matrix with the ﬁrst-order data.
In the ﬁrst stage, SOFT learns a sparse attention matrix to explore the second-order feature relationships.
In the second stage, we perform graph segmentation on the learned attention matrix for feature selection.
In summary, we highlight our contributions as follows: (1) We consider the second-order feature relationship in the unsupervised feature selection problem and propose the SOFT algorithm to distill knowledge from both original features and their second-order feature covariance matrix; (2) Our SOFT learns a mask matrix on or off the covariance matrix and obtains the attention matrix and masked matrix.
Throughout distilling the structural knowledge, the sparse attention matrix contains such knowledge as much as possible while excluding that from the masked matrix as well; (3) Different from selecting features according to weights, we propose a graph segmentation-based feature selection method on the attention matrix, where only one representative feature is selected from each segment to avoid redundancy; and (4) Experimental results validate that our SOFT model outperforms classical and recent state-of-the-art models on 12 public datasets.
We also provide in-depth exploration for both stages to demonstrate the effectiveness of SOFT.
M ETHODOLOGY
Motivation .
Unsupervised feature selection aims to select a small portion of the original features that are most useful for the downstream tasks without external guidance.
Most previous methods focus on ranking features based on the values of individual features (He et al., 2005; Zhao & Liu, 2007;
Yang et al., 2011) while neglect the high-order relationships between features.
Unfortunately, this might lead to the redundancy of the selected features and further deteriorate downstream tasks.
Figure 1 provides an illustrative example of selection results on Sonar (Rossi &
Ahmed, 2015) and Waveform (Breiman et al., 1984) by Laplacian Score (LapScore)
(He et al., 2005).
The heatmap shows the initial relationships between features based on the covariance matrix of features.
We remove the diagonal values and calculate the absolute values of covariance, which are normalized for visualization.
In this example, we select the top four features on Sonar and Waveform by LapScore, which are highlighted with red lines in Figure 1.
Obviously, features a and b , features c and d on Sonar are from two groups within of high similarity, indicating that one feature might be denoted by the other.
Similarly, On Waveform , features e , f , and h contain highly relevant information as well.
However, features from the same feature group might lead to redundancy, which disobeys the purpose of feature selection.
This drives us to explore the complex relationships derived from the second-order feature covariance matrix in the unsupervised feature selection problem.
Motivated by the above situation, we design a method that can avoid such redundancy by considering both ﬁrst-order data and second-order data.
Framework Overview .
To incorporate the correlation feature information, we propose a two-stage model SOFT, which takes the ﬁrst-order data matrix and second-order feature correlation matrix as inputs.
In the ﬁrst stage, SOFT learns a mask on the feature correlation matrix via knowledge contrastive distillation, which is beneﬁcial to preserve the data structure.
In the second stage, we select the features on the masked correlation matrix via graph segmentation.
Figure 2 shows the overview of our SOFT framework.
With the ﬁrst-order data matrix and secondorder correlation feature matrix as inputs, we aim to learn a mask matrix applying to the feature correlation matrix for feature selection.
By this means, we can obtain the attention matrix and masked matrix by the learnable mask on or off the feature matrix, respectively.
The key idea of
SOFT is to distill the structural knowledge via making the attention matrix contain such knowledge as much as possible while excluding that from the masked matrix as well.
The green and red lines in
Figure 2 demonstrate the knowledge contrastive distilling process.
To achieve this, we adopt a shared
Graph Convolution Network (GCN) (Kipf & Welling, 2016) to generate attention/original/masked representations from the attention/feature/masked matrices, respectively.
Then we use pseudo labels generated by attention representation as positive guidance for original representation and negative guidance for masked representation so that attention representation and original representation are close to each other and far away from masked representation.
Throughout the above knowledge contrastive distilling process, we can get a sparse and effective feature relation matrix that represent the second-order correlation, where each node denotes a feature and weights of edges are the corresponding values in the learned attention matrix.
In the second stage, different from the existing work, which calculates the weight of each feature and suffers from the redundant feature issue (shown in Figure 1), our attention matrix delivers the weight for a pair of features.
To proceed with the feature selection, we use graph segmentation to cut the attention matrix into partitions and select one feature that has the highest relationship to others from each partition as the ﬁnal selection result.
By this means, the redundancy among the selected features can be mitigated.
[Table]
Attention Matrix Learning .
Given n data instances with d features, we have the ﬁrst-order n × d data matrix X and the d × d second-order feature matrix M F = X (cid:62) × X .
Our SOFT model calculates a learnable mask for feature selection.
In the ﬁrst stage, SOFT consists of four components, attention layer, shared GCN, pseudo label generation, and contrastive learning.
We use Θ = {
θ M , θ G , θ C } to denote the trainable parameters set in the SOFT model,
where θ M , θ G , and θ C denote the parameters of the attention layer, shared GCN and contrastive learning, respectively
Note that the pseudo labels are generated from the attention matrix and shared GCN, which are controlled by θ M and θ G .
Therefore, no learnable parameters are needed for pseudo label generation.
Table 1 shows the notations used in our SOFT model and their descriptions.
Each part is detailed as follows.
Attention Layer .
The initialized feature matrix M F may not be good enough to represent the relationship between features, so we add an attention layer to better capture second-order feature interactions by highlighting the important relations and reducing others through a learnable mask θ M .
The attention matrix M A , which represents the important part of M F , is calculated as follows:
[Equation]
where (cid:12) is the element-wise product. Due to the symmetry of M F , θ M is forced to be symmetric by adding a d × d parameter matrix with its transpose.
With the learnable mask θ M and the attention matrix M A , We assume that the input feature matrix M F can be decomposed as a summation formulation of attention matrix M A and masked matrix M M . Then we deﬁne the masked matrix M M by the following function:
[Equation]
To make the learned attention matrix M A sparse enough to identify crucial and principle relations, we apply (cid:96) 2 , 1 -norm to learnable mask θ M on both row and column level, which is:
[Equation]
Shared Graph Convolutional Network .
GCN (Kipf & Welling, 2016) helps generate embeddings that contain both data information and feature relationship information.
We apply a shared 2-layer GCN to extract the attention/original/masked representations from the attention/feature/masked matrices, respectively.
In Kipf & Welling (2016), nodes of graphs denote samples, while in our case, nodes of graphs are features.
Therefore, the computation of the GCN layer is a little different from Kipf &
Welling (2016) and is described by the following equation:
[Equation]
where M is the input relationship matrix,
˜ M = M + I d , I d ∈ R d
× d is the identity matrix, ˜ D is the degree matrix of ˜ M , and θ ( l ) G is a layer-speciﬁc trainable weight matrix.
G ( l ) denotes the embeddings in the l -th layer, and G (0) =
X .
In our case, we use a shared 2-layer GCN to process samples with different input relationship matrices M A , M F , and M M .
Thus we have the attention representation
G A , original representation G F , and masked representation G M , respectively.
Pseudo Label Generation .
In the unsupervised feature selection, pseudo labels are usually used as the criterion to guide feature selection.
In our experiments, we use Principal Component Analysis (PCA)
(Wold et al., 1987) and K-means to generate pseudo labels (Caron et al., 2018).
The embeddings processed by the clustering part is generated by attention matrix M A , which is also the matrix we expect to learn in the SOFT model.
PCA helps get principal features, and K-means clustering generates pseudo labels P A for the input samples X .
Contrastive Learning .
With the above representations, we expect that attention representation and original representation are close to each other but far away from masked representation.
To achieve such contrastive learning, we design two losses to measure the predictive abilities of different representations and compare them with the pseudo labels.
Specially, we employ a 2-layer Multi-Layer
Perceptron (MLP) to get predictions of the samples with original representation G F and masked representation G M about which clusters they belong to,
[Equation]
where θ ( l ) C is a layer-speciﬁc trainable weight matrix, P ( l ) denotes the embeddings in the l -th layer, and P (0) =
G F or P (0)
=
G M for the original representation and masked representation, respectively.
We use σ ( · ) =
ReLU( · ) as the activation function for layers before the last layer, and σ ( · ) =
Softmax( · ) for the last layer.
In our experiments, we adopt a shared 2-layer MLP and obtain
P F and P M for original representation G F and masked representation G M .
Pseudo labels P A generated by the clustering part are used for the following positive and negative training.
By positive guidance for P F , we get a cross-entropy loss L F for P F , which is deﬁned as:
[Equation]
where P Fij denotes the probability that the i -th sample belongs to cluster j based on M F .
P Aij
=if the i -th sample belongs to cluster j based on the clustering result,
otherwise P Aij = 0 .
Eq.
(7) is designed to make P F similar to pseudo labels P A .
While by negative guidance for predictions of the masked part P M , we apply attention loss described in Li et al.
(2018b) to reduce the weights of unimportant relations, which is stated as follows based on our notations:
[Equation]
where P Mij denotes the probability that the i -th instance belongs to cluster j based on M M .
Through minimizing Eq.
(7), the model tends to put instances to a cluster that they do not belong to, thus driving predictions P M generated by masked representation different from pseudo labels P A , which would ﬁnally lead to the difference of masked representation G M and attention representation G A .
Objective Function .
Combining Eq.
(3), Eq.
(6), and Eq.
(7), Our overall objective function of SOFT can be written as follows:
[Equation]
where α and β are hyperparameters for L M and L 2 , 1 , respectively.
We adopt Adam optimizer (Kingma & Ba, 2014) to minimize the objective function.
The detailed algorithm can be found in Appendix A.
Feature Selection on Attention Matrix .
Different from the traditional feature selection methods that return a weight vector to choose the top-ranked features, here, our attention matrix provides the weight of a pair of features.
To proceed with the feature selection, we build a feature graph based on the attention matrix.
Nodes of the graph denote features, and edges are the relationship between features in the attention matrix.
Then we perform graph segmentation for feature selection.
Graph Construction .
Instead of transforming the attention matrix M A to a graph immediately, we add two additional processes so that the generated graph is more suitable for graph segmentation and feature selection.
The ﬁrst one is to remove some irrelevant features.
We calculate a score for each feature by S
= (cid:80) di =1 M Ai , where S is the summation of each row or column of the attention matrix, which measures the importance of each feature.
We delete the last 10% features based on the sorted S .
The second step is to set all values smaller than the median value in M A to zero so that the number of unimportant edges is largely reduced.
We also set all negative values in M A to zero to make sure no negative edges would exist in the constructed graph.
Graph Segmentation .
With the processed graph, we apply the graph segmentation method provided by Karypis & Kumar (1998) and cut the graph into k parts, where k is the number of features we aim to select.
For each partition, we choose the feature that has the largest value in S , since we consider features in the same partition are highly related.
Through learning attention matrix in the ﬁrst stage of our method, we incorporate both ﬁrst-order data and second-order data and get a reﬁned feature relation matrix that can better reﬂect feature relations.
Then in the second stage, we build a graph based on the learned attention matrix and perform graph segmentation, which groups high-correlated features together.
Therefore, by selecting features from each partition, our method reduces redundancies among the selected features.
Our proposed method focuses on learning pair-wise relationships of features and uses graph segmentation to select features.
The time complexity of our method is O ( nd 2 ) .
If the number of samples is larger than the number of features ( n > d ), the space complexity of our method is O ( nd ) , otherwise O ( d 2 ) .
E XPERIMENTAL R ESULTS
Experimental Settings .
We introduce the experimental settings below.
Datasets .
We select 12 public feature selection benchmark datasets of different types for evaluation:
COIL
(Nene et al., 1996), Colon (Alon et al., 1999), Gisette (Guyon et al., 2004), Lung-Cancer ( L.Cancer )
(Hong & Yang, 1991), Madelon (Guyon et al., 2007), MovementLibras ( M.Libras ) (Dias et al., 2009), NCI9 (Ross et al., 2000), ORL (Cai et al., 2006), Sonar (Rossi & Ahmed, 2015), UAV and UAV2 (Zhao et al., 2018), and Waveform ( Wave. )
(Breiman et al., 1984).
The instance numbers of these dataset range from 32 to 19937, the feature numbers range from 40 to 9712, and the true cluster numbers are from 2 to 40.
The sample/feature ratios of these datasets range from 0.01 to more than 350, indicating the diversity of the datasets.
Detailed dataset statistics can be found in Appendix B.
Comparative Methods and Implementation .
We choose 10 classical and recent state-of-the-art unsupervised feature selection methods for comparison.
Laplacian Score (LapScore)
(He et al., 2005) selects features by scoring features with a Gaussian Laplacian matrix.
SPEC (Zhao & Liu, 2007) is a more general framework for feature selection based on spectral graph theory, where
LapScore is a special case of it.
MCFS (Cai et al., 2010) uses spectral analysis and sparse regression to select features and capture the multi-cluster data structure.
UDFS (Yang et al., 2011) selects features by discriminative analysis and (cid:96) 2 , 1 minimization.
NDFS (Li et al., 2012) selects the most
[Table]
54 ± 0.
Average 0.
N/A: “ ∗ ” indicates the algorithmic running error, and “ + ” means the out-of-64GB-memory error.
[Figure]
discriminative features with a nonnegative constraint and (cid:96) 2 , 1 regularization.
LRPFS (Zheng et al., 2018) adopts a low-rank constraint to preserve the subspace structure information.
NSSLFS (Zheng et al., 2019) learns the feature weight matrix with the (cid:96) 2 , 1 -norm and the non-negative constraint based on the low-dimensional sparse subspace learning.
TSFS (Mirzaei et al., 2020) employs a teacher-student scheme for deep feature selection.
CAE (Balın et al., 2019) uses the concrete distribution and the reparametrization trick to differentiate through an reconstruction loss and select input features.
InfFS (Roffo et al., 2020) is a fast graph-based approach which ranks and selects features by considering the possible subsets of features as paths on a graph.
For LapScore, SPEC, MCFS, UDFS, and NDFS, we adopt implementations and default settings provided by scikit-feature (Li et al., 2018a).
For LRPFS and NSSLFS, we set the values of hyperparameters in their objective functions to 1.0.
For TSFS, CAE (non-linear version), and InfFS, we use default settings provided in their open-source codes.
The settings of our SOFT model are as follows.
In the stage of Attention Matrix learning, we implement the networks by PyTorch.
The Learnable Mask is initialized by the normal distribution.
The weights α and β in the objective function areand 0.001, respectively.
We adopt adam optimizer (Kingma & Ba, 2014) to minimize the objective function and set the learning rate to 1e-4.
We run a total of 300 epochs to learn the attention matrix.
In the stage of graph segmentation, we ﬁrst remove 10% features as irrelevances based on the learned attention matrix and set the least 50% values of the remaining 90% features in attention matrix to zero before graph construction.
Then we perform graph segmentation and select 1 feature from each partition.
For the re-implementation, the link of code and datasets can be found in Appendix C.
Evaluation Metric .
We employ k-means++ (Arthur & Vassilvitskii, 2006) on the selected features, and compare the obtained partition and ground truth by clustering accuracy, settings of which follows scikit-feature (Li et al., 2018a).
Algorithmic Performance .
Table 2 shows the experimental results of different unsupervised feature selection methods on 10% selected features in terms of accuracy.
The best results are highlighted in bold.
“N/A” means the corresponding method cannot process the dataset successfully due to running error or out of memory error.
We can see that our SOFT model achieves the best on 11 of thedatasets.
One of the possible reasons is that SOFT explores the second-order relationships among features, while other competitive methods only used the ﬁrst-order data, while we incorporated second-order data in the feature selection process.
By grouping features based on the learned attention matrix, our method can avoid the redundancy described in Section 2.
Moreover, the average accuracy of SOFT is signiﬁcantly better than other methods with large margins of 5% to 10%, which demonstrates the positive effects of second-order feature exploration on the unsupervised feature selection problem.
Beyond the algorithmic effectiveness, we also provide the algorithmic execution time of different algorithms in Appendix D.
Next, we show the performance of all the methods on selecting different percents of features in
Figure 3.
LapScore achieves the best performance with 20% selected features on Madelon , LRPFS is always the best among these methods on L.Cancer , NSSLFS outperforms other methods with 15% and 20% selected features on Colon , and TSFS with 15% selected features is ranked the ﬁrst on NCI9 .
It is worthy to note that sometimes the performance of SOFT is not so stable as other methods, such as on L.Cancer when the number of selected features increased.
This is because the comparison methods generate a ranking list for features and select features based on the feature ranking scores, while SOFT select features based on the graph segmentation result.
With the number of selected features increasing, the feature partitions might change notably, which results in a different selection.
In general, SOFT delivers the compromising features with different percents compared with others, especially when the percent of selected features is relatively small.
For results with all percentages and datasets, SOFA outperforms all other methods with an average accuracy of 61%, which is signiﬁcantly better than two recent methods (CAE achieves 54%, and InfFS gets 51%).
In-depth Exploration of SOFT .
We provide the in-depth exploration of SOFT from the following aspects.
Additional experimental results can be found in Appendix E-H.
Parameter Analysis .
There are two parameters in the objective function of SOFT, α and β , denoting the weight of attention loss and (cid:96) 2 , 1 -norm loss, respectively.
We vary α and β from 1 e − 3 to 1 e +to explore the impact of these two parameters on the ﬁnal performance.
Figure 5(a) shows the results [Figure] on Colon with 10% selected features.
We can see that despite the large range of parameter values, the ﬁnal performance does not change much, indicating that the Learnable Mask is well learned and
SOFT might not be sensitive to the value of α and β in a large range.
Visualization of Attention Matrix and Learnable Mask .
To further analyze the performance of SOFT, we visualize the Attention Matrix and the corresponding Learnable Mask in different epochs on
MovementLibras , which is shown in Figure 4.
The darker color indicates the stronger correlation of the corresponding pair of features.
To better recognize feature relations, we remove the diagonal values for visualization.
In the beginning, there is no identiﬁed pattern in the Learnable Mask because the learnable mask is randomly initialized.
As the training epochs increased, the learnable mask becomes sparser and seeks the dataset-dependent patterns.
While the attention matrix is generated by an element-wise product of the original feature matrix and the learnable mask, the attention matrix became sparser as well.
Therefore, the most important part of the attention matrix for representing samples was highlighted through network training.
Then we do a further step to explore the effectiveness of the learned attention matrix by comparing our results with two designated baseline methods.
The ﬁrst one is the First Order method, which only uses intrinsic properties of the data.
We remove the GCN part of SOFT and utilizes a vectorial learnable mask for the ﬁrst order network.
The First Order learns the ranking scores for features, which is denoted by the Learnable Mask.
The second one is the Covariance method, which uses the original feature matrix directly for the second stage of SOFT.
Figure 5(b) shows the experimental result in terms of accuracy.
On almost all the datasets, both SOFT and Covariance performed better than First Order, which demonstrates the effectiveness of incorporating feature relation in feature selection.
While sometimes Covariance achieves a better result than SOFT, SOFT performed the best in most cases and achieved the best on average, proving that SOFT could learn a sparse and effective feature relation matrix to represent the second-order correlation.
Feature Selection Strategy .
We adopt two baseline methods for feature selection from Attention
Matrix as the comparison with the graph segmentation method in SOFT.
The ﬁrst baseline method,
Weight Sum, utilizes a row-sum method to get the total relation value of each feature to all other features as the ranking scores for features.
Features with the highest scores are selected.
The second baseline method, Largest Weight, each time ﬁnds the largest value in Attention Matrix and selects the corresponding feature pair until k features are selected.
Comparison results among Weight Sum,
Largest Weight, and the graph segmentation in SOFT are shown in Figure 5(c).
Overall, the graph segmentation method in SOFT has a signiﬁcant advantage over the baseline methods.
This results from that SOFT avoids selecting highly correlated features by graph segmentation, thus reducing redundancies and bringing in more complementary features for performance boosting.
R ELATED W ORK
We brieﬂy review related work on unsupervised feature selection and deep feature selection below.
Unsupervised Feature Selection .
In the past decades, a large amount of unlabeled data are generated.
To solve the feature selection problem for the unlabeled data, researchers have proposed many unsupervised feature selection methods (Liu et al., 2016; Li et al., 2016a), which can be divided into three main categories: Filter, Wrapper, and Hybrid.
Filter methods evaluate features based on the data itself.
Dash et al.
(1997) propose one of the earliest ﬁlter unsupervised feature selection method, Sequential backward selection method for Unsupervised Data (SUD).
SUD introduces a similarity matrix representing the pair-wise similarity between objects.
By measuring the entropy of the data, the relevance of each feature is quantiﬁed as a ranking score.
Features with the highest scores are selected.
Another example is Laplacian Score (He
et al., 2005), which weights features according to their ability to preserve a predeﬁned manifold structure represented by the Laplacian matrix.
Similarly, SPECtrum decomposition (SPEC) (Zhao & Liu, 2007) also introduces an object similarity matrix.
SPEC measures the consistencies between features and nontrivial eigenvectors of the Laplacian matrix and ranks features based on the consistencies.
Wrapper methods select the most relevant features by using a clustering algorithm.
(Dy & Brodley, 2004) introduce a method to select feature subsets using Expectation Maximization (EM)
(Dempster et al., 1977) clustering and evaluate them with maximum likelihood and the scatter separability criterion.
Kim et al.
(2002) present an evolutionary multi-objective local selection algorithm to search feature subsets with
K-means (Hartigan & Wong, 1979) and EM (Dempster et al., 1977) clustering.
Instead of selecting feature subsets, Law et al.
(2004) propose to estimate a set of real-valued quantities carried out by an
EM algorithm through adopting a minimum message length (Wallace & Dowe, 2000) penalty.
Hybrid methods try to take advantages of both approaches by adopting a two-stage process: ﬁlter stage and wrapper stage.
In the ﬁlter stage, the features are scored based on the intrinsic properties of the data.
And in the wrapper stage, feature sets are generated by a speciﬁc clustering algorithm.
For instance,
Dash & Liu (2000) adopt the method of Dash et al.
(1997) for the ﬁlter stage to sort the features and the method of Dy & Brodley (2004) for the wrapper stage to build clusters.
Solorio-Fern´andez et al.
(2016) combine the spectral feature selection framework using the Laplacian Score (He et al., 2005) ranking and a modiﬁed Calinski–Harabasz index (Cali´nski & Harabasz, 1974).
Different from the above-mentioned methods which are based on ranking, Kim & Gao (2006) propose a method that starts with the wrapper stage by Least-Square-Estimation based evaluation and then selects feature set through a Bayesian network in the ﬁlter stage.
InfFS (Roffo et al., 2020) is a graph-based ﬁltering approach, which evaluates the values of paths in a graph and selects discrete input features by exploiting properties of power series of matrices and the concept of absorbing Markov chains.
Deep Feature Selection .
Recently, deep learning techniques have gained much attention and brought in some studies on deep feature selection (Chang et al., 2017; Taherkhani et al., 2018).
DFS (Li et al., 2016b) adds a weight layer to Multi-layer Perceptron (MLP) together with a sparse regularization term so as to take advantage of deep structures to model nonlinearity.
Zhao et al.
(2015) propose to combine deep neural networks with sparse representation for grouped heterogeneous feature selection.
The model ﬁrst converts the multi-modal data into a uniﬁed representation, then selects features through solving a sparse group lasso (Friedman et al., 2010) problem.
In recent years, some studies also involve data reconstruction error in deep unsupervised feature selection.
AEFS (Han et al., 2018) jointly learns a self-representation autoencoder model and the importance weights of each feature for feature selection.
Furthermore, GAFS (Feng & Duarte, 2018) not only adopts a single layer autoencoder but also incorporates the spectral graph analysis for learning.
UDSFS
(Cong et al., 2016) selects the most discriminative features and meanwhile designates appropriate weights to the feature dimensions by utilizing the group sparsity of the features.
TSFS (Mirzaei et al., 2020) presents a teacher-student scheme for deep feature selection, in which a teacher network is used to learn low-dimensional representations, and a student network is employed for feature selection by minimizing the reconstruction error.
CAE (Balın et al., 2019) uses a concrete selector layer as the encoder and a standard neural network as the decoder, stochastically selects discrete input features by concrete random variables and the reparametrization trick to get a subset of input features.
C ONCLUSION
In this paper, we proposed a two-stage framework named SOFT for unsupervised feature selection, which incorporated second-order data with ﬁrst-order data.
Speciﬁcally, in the ﬁrst stage, we ﬁrst obtained the Attention Matrix and Masked Matrix by applying a learnable mask on and off the input Feature Matrix.
Then we generated Attention/Original/Masked Representations from
Attention/Feature/Masked Matrices by a shared Graph Convolutional Network.
To train the learnable mask, we used pseudo labels generated by Attention Representation as positive guidance for Original
Representation and negative guidance for Masked Representation, such that Attention Representation and Original Representation were similar to each other and different from Masked Representation.
In the second stage, we constructed a graph based on the well-learned Attention Matrix and utilized graph segmentation to separate the graph into several parts.
We chose one feature from each partition as the feature selection result.
Experiments on public datasets demonstrated that our method outperformed classical and recent state-of-the-art methods on tackling the unsupervised feature selection problem.
A A LGORITHM OF SOFT
Algorithm 1 describes the whole process of SOFT.
Through learning Attention Matrix in the ﬁrst stage of our method, we incorporate both ﬁrst-order data and second-order data and get a reﬁned feature relation matrix that can better reﬂect feature relations.
Then in the second stage, we build a graph based on the learned Attention Matrix and adopt the graph segmentation method, which groups high-correlated features together.
B D ATASET S TATISTICS
We select 12 public feature selection benchmark datasets of different types for evaluation:
COIL
(Nene et al., 1996), Colon (Alon et al., 1999), Gisette (Guyon et al., 2004), Lung-Cancer ( L.Cancer )
(Hong & Yang, 1991), Madelon (Guyon et al., 2007), MovementLibras ( M.Libras ) (Dias et al., 2009), NCI9 (Ross et al., 2000), ORL (Cai et al., 2006), Sonar (Rossi & Ahmed, 2015),
UAV1 and UAV2 (Zhao et al., 2018), and Waveform ( Wave. )
(Breiman et al., 1984).
Tabel 3 shows the statistics of these datasets, where the sample/feature ratios range from 0.01 to more than 350, indicating the diversity of the datasets.
[Table]
C R E - IMPLEMENTATION
Codes and datasets of our work can be found at https://github.com/ICLR2022submission/SOFT.
D T IME C OST
We record the running time for each method on 10% selected features, which is detailed in Table 4.
When the number of features is much larger than the number of instances, our method runs relatively slow but faster than methods employing the matrix inverse calculation like UDFS.
But when there are much more samples than features, our method is faster than others except for LapScore, NSSLFS, and InfFS.
This is because SOFT focuses on relations between features, where the number of features contributes more to the time cost than the number of instances.
In general, the speed of SOFT is acceptable even for large-scale datasets
[Table]
[Table]
1034.50 ORL 10.
55.54 Sonar 0.
2.17 UAV 51.
N/A 41.
57.40 UAV 42.
51.52 Wave.
15.26 Note: All experiments ran on a physical machine with Ubuntu 18.04, total memory of 64GB, AMD Ryzen Threadripper 2920X 12-Core Processor, and an NVIDIA GP102 GPU.
E S TABILITY T EST
The “stability” of a feature selection algorithm refers to the robustness of its feature preferences, with respect to data sampling and to its stochastic nature.
An algorithm is “unstable” if a small change in data leads to large changes in the chosen feature subset.
A typical approach to measure stability is to ﬁrst take M bootstrap samples of the provided data set, apply feature selection to each one of them, and then measure the variability in the M feature sets obtained.
Here we conduct the stability tests on COIL20 and L.Cancer .
Speciﬁcally, we generate M = 50 bootstrap folds and run different unsupervised selection methods on these bootstrap folds.
And then, we use the stability measurement proposed by Nogueira et al.
(2017) ranging from -1 to 1, which returns the stability score with conﬁdence intervals.
The large value means more stable.
The collection of the M feature sets can therefore be modeled as a binary matrix Z of size M × d , where d is the dimension of features.
A row in Z represents a feature set, and a column represents the selection of a given feature over the M repeats as follows:
[Equation]
Based on Z , Nogueira et al.
(2017) deﬁne the stability score as follows:
[Equation]
where ¯ k is the average number of features selected over the M feature set, s 2 f is the unbiased sample variance of the selection of the f th feature, and here ˆ p f denotes the mean of the f th column of Z .
Figure 6 shows the stability test of different feature selection methods on COIL20 and L. Cancer .
Our SOFT algorithm performs very stable with high scores and small intervals and excels others by a large margin in terms of stability scores.
F D IFFERENT W AYS TO G ENERATE P SEUDO L ABELS
SOFT is ﬂexible with different pseudo label generation methods.
So far, we use PCA + K-means to generate pseudo labels that are further used to guide feature selection.
Here we test the performance of
SOFT with other ways to generate pseudo labels in Table 5.
Speciﬁcally, two popular deep clustering methods DEC (Xie et al., 2016) and IMSAT (Hu et al., 2017) with their default parameters are used here.
We can see some improvements by the deep clustering methods.
For example, SOFT with
DEC and IMSAT outperforms our default setting (PCA + K-means) on COIL20 and NCI9 , which demonstrates the power of deep methods.
But in general, the default setting is slightly better than the deep methods.
We conjecture that some dedicated network architecture design is needed based on the data property to pursue better performance.
If we compare the results in Table 5 and Table 2, SOFT with DEC and IMSAT to generate pseudo label still excels other baselines by a large margin on the average level.
Moreover, we also report the execution time of SOFT with different ways to generate pseudo labels.
Previously, SOFT might leave an impression of high computational cost just because of the long execution time on Gisette dataset.
Actually, the majority of the running time is occupied by PCA.
The speed of SOFT on Gisette dataset can be dramatically accelerated from 15129.18 to 739.55 or 1068.37 by second with DEC or IMSAT.
[Table]
G D IFFERENT GRAPH S EGMENTATION M ETHODS
Graph segmentation on the attention matrix is a crucial stage of SOFT.
Here we change the graph segmentation algorithms with multilevel recursive bisection on different percents of selected features.
Figure 7 shows our SOFT performance and standard deviations on different percents of selected features without and with multilevel recursive bisection.
We can see that more smoothness over different percents is brought by multilevel recursive bisection on L.Cancer and UAV2 , compared with the default one.
The standard deviations on different percents of selected features are narrowed by multilevel recursive bisection as well.
[Figure]
H A BLATION STUDY ON NOISE REMOVAL FROM THE ATTENTION MATRIX
In the graph construction of SOFT
[Table]
we have two ways to remove noises, including setting the values that are smaller than the median to be zero and deleting 10% features according to their importance.
More details can be referred to in the paragraph of Graph Construction .
Here we provide the ablation study on noise removal from the attention matrix in Table 6.
Our default setting brings improvements on 7 out of 12 datasets compared with the single noise removal way and non-noise removal.
On the average level, our default setting is also slightly better than others
[Table]
[Table]
I A CKNOWLEDGEMENT
We would like to thank the anonymous reviewers for their suggestions and interactions during the review period
[Table]