Original: Where to Pay Attention in Sparse Training for Feature Selection?
Revised: Where to Pay Attention in Sparse Training for Feature Selection?

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: A new line of research for feature selection based on neural networks has recentlyemerged.
Revised: A new line of research for feature selection based on neural networks has recentlyemerged.

Original: Despite its superiority to classical methods, it requires many trainingiterations to converge and detect informative features.
Revised: Despite its superiority to classical methods, it requires many trainingiterations to converge and detect informative features.

Original: For datasets with a largenumber of samples or a very high dimensional feature space, the computational timebecomes prohibitively long.
Revised: For datasets with a largenumber of samples or a very high dimensional feature space, the computational timebecomes prohibitively long.

Original: In this paper, we present a new efficient unsupervisedmethod for feature selection based on sparse autoencoders.
Revised: In this paper, we present a new efficient unsupervisedmethod for feature selection based on sparse autoencoders.

Original: In particular, wepropose a new sparse training algorithm that optimizes a model’s sparse topologyduring training to quickly pay attention to informative features.
Revised: In particular, wepropose a new sparse training algorithm that optimizes a model’s sparse topologyduring training to quickly pay attention to informative features.

Original: The attention-basedadaptation of the sparse topology enables fast detection of informative features aftera few training iterations.
Revised: The attention-basedadaptation of the sparse topology enables fast detection of informative features aftera few training iterations.

Original: We performed extensive experiments on 10 datasets ofdifferent types, including image, speech, text, artificial, and biological.
Revised: We performed extensive experiments on 10 datasets ofdifferent types, including image, speech, text, artificial, and biological.

Original: They covera wide range of characteristics, such as low and high-dimensional feature spaces,as well as few and large training samples.
Revised: They covera wide range of characteristics, such as low and high-dimensional feature spaces,as well as few and large training samples.

Original: Our proposed approach outperformsthe state-of-the-art methods in terms of the selection of informative features whilereducing training iterations and computational costs substantially.
Revised: Our proposed approach outperformsthe state-of-the-art methods in terms of the selection of informative features whilereducing training iterations and computational costs substantially.

Original: Moreover, theexperiments show the robustness of our method in extremely noisy environments 1 .
Revised: Moreover, theexperiments show the robustness of our method in extremely noisy environments 1 .

Original:  Introduction
Revised:  Introduction

Original: Feature selection plays a crucial role in data mining and machine learning tasks with the explosion inthe size and dimensionality of real-world data [65; 10; 28; 41; 21].
Revised: Feature selection plays a crucial role in data mining and machine learning tasks with the explosion inthe size and dimensionality of real-world data [68; 10; 30; 44; 23].

Original: It aims to efficiently select a subsetof the features that are most informative and remove the irrelevant or redundant ones.
Revised: It aims to efficiently select a subsetof the features that are most informative and remove the irrelevant or redundant ones.

Original: It is usefulfor alleviating the curse of dimensionality, interpretability of model-driven decisions, generalizationof downstream tasks, reducing the prohibitive memory and computational costs and avoiding theexpensive costs of collecting the full set of features as in biological studies [11; 36; 29; 49].
Revised: It is usefulfor alleviating the curse of dimensionality, interpretability of model-driven decisions, generalizationof downstream tasks, reducing the prohibitive memory and computational costs and avoiding theexpensive costs of collecting the full set of features as in biological studies [11; 38; 31; 52].

Original: Many methods have been proposed for feature selection in supervised, semi-supervised and unsupervised settings [71; 10; 57; 48; 17; 1].
Revised: Many methods have been proposed for feature selection in supervised, semi-supervised and unsupervised settings [75; 10; 60; 51; 19; 1].

Original: Recently, using neural networks for feature selectionhas received increasing attention due to its power to learn non-linear dependencies among inputfeatures [37].
Revised: Recently, using neural networks for feature selectionhas received increasing attention due to its power to learn non-linear dependencies among inputfeatures [39].

Original: One limitation of current neural network-based methods is that they are computationallyexpensive.
Revised: One limitation of current neural network-based methods is that they are computationallyexpensive.

Original: The computational costs result from training dense models for many training iterationsuntil informative features are recognized.
Revised: The computational costs result from training dense models for many training iterationsuntil informative features are recognized.

Original: The costs increase for datasets with a very large number oftraining samples or high dimensional feature space.
Revised: The costs increase for datasets with a very large numberof training samples or high dimensional feature space.

Original: A recent work proposed by [4] addressed thislimitation by utilizing sparse networks for feature selection.
Revised: This limitation is recently addressed in [4]by utilizing sparse networks for feature selection.

Original: They trained an autoencoder with thesparse training algorithm SET [51], which explores different sparse topologies during training by dropping a portion of the connections and growing the same portion randomly .
Revised: It trains an autoencoder with the sparse trainingalgorithm SET [54], which explores different sparse topologies during training by dropping a portion of the connections and growing the same portion randomly .

Original: The importance of aninput feature is estimated after convergence based on its outgoing sparse connections.
Revised: The importance of an input feature isestimated after convergence based on its outgoing sparse connections.

Original: The usage ofsparse networks decreases memory and computational costs compared tothe previous dense-basedmethods.
Revised: Sparse networks decreasememory and computational costs compared to previous dense-based methods.

Original: Yet, the random exploration of sparse topologies requires many training iterations toidentify informative features.
Revised: Yet, the randomexploration of sparse topologies requires many training iterations to identify informative features.

Original: In this paper, we propose an efficient unsupervised method for feature selection.
Revised: In this paper, we propose an efficient unsupervised method for feature selection.

Original: We introduce anew sparse training algorithm for autoencoders that quickly detects the informative input features.
Revised: We introduce anew sparse training algorithm for autoencoders that quickly detects the informative input features.

Original: Specifically, we optimize the sparse topology to learn W here to pay A ttention during S parse T raining.
Revised: Specifically, we optimize the sparse topology to learn W here to pay A ttention during S parse T raining.

Original: We named our method WAST .
Revised: We named our method WAST .

Original: WAST exploits the information from the reconstruction loss and theweights of sparse connections to guide the topological search.
Revised: WAST exploits the information from the reconstruction loss and theweights of sparse connections to guide the topological search.

Original: It redistributes the sparse connectionsto spot informative features quickly.
Revised: It redistributes the sparse connectionsto spot informative features quickly.

Original: We evaluate our proposed approach on ten datasets from variousdomains, including image, speech, text, artificial, and biological.
Revised: We evaluate our proposed approach on ten datasets from variousdomains, including image, speech, text, artificial, and biological.

Original: Through extensive experiments,we find that WAST identifies the informative features after a few training iterations and outperformsstate-of-the-art unsupervised feature selection methods.
Revised: Through extensive experiments,we find that WAST identifies the informative features after a few training iterations and outperformsstate-of-the-art unsupervised feature selection methods.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We propose a new efficient unsupervised method for feature selection, named WAST.
Revised: • We propose a new efficient unsupervised method for feature selection, named WAST.

Original: WASToptimizes the sparse topology of an autoencoder to detect the informative features quickly.•
Revised: WASToptimizes the sparse topology of an autoencoder to detect the informative features quickly.•

Original: We perform extensive experiments on 10 benchmarks that cover various types and character- istics.
Revised: We perform extensive experiments on 10 benchmarks that cover various types and characteristics.

Original: Experimental results show the effectiveness of our method over the state-of-the-art
Revised: Experimental results show the effectiveness of our method over the state-of-the-art

Original: methods in terms of selecting the informative features.• WAST reduces the computational time compared to neural network-based methods substantially by reducing training iterations and employing highly sparse neural networks.
Revised: methods in terms of selecting the informative features.• WAST reduces the computational time compared to neural network-based methods substantially by reducing training iterations and employing highly sparse neural networks.•

Original: • We illustrate the robustness of our method in extremely noisy environments and its effec-
Revised: We illustrate the robustness of our method in extremely noisy environments and its effec-

Original: tiveness for datasets with very high dimensional feature space and a few training samples.
Revised: tiveness for datasets with very high dimensional feature space and a few training samples.

Original:  Related Work
Revised:  Related Work2.1 Feature Selection

Original: 2.1 Feature Selection

Original: Many feature selection methods were introduced for the supervised, semi-supervised, and unsupervised settings depending on whether the data is labeled, partially labeled, or unlabeled, respectively[71; 10; 57; 1].
Revised: Many feature selection methods were introduced for the supervised, semi-supervised, and unsupervised settings depending on whether the data is labeled, partially labeled, or unlabeled, respectively[75; 10; 60; 1].

Original: Typically, feature selection methods are divided into three categories: filter, wrapper,and embedded methods.
Revised: Typically, feature selection methods are divided into three categories: filter, wrapper,and embedded methods.

Original: Filter methods are independent of the learning algorithms.
Revised: Filter methods are independent of the learning algorithms.

Original: They use rankingtechniques for providing scores to the features, such as Laplacian score [24].
Revised: They use rankingtechniques for providing scores to the features, such as Laplacian score [26].

Original: Despite being fast, theydo not consider the relation between the features, which may result in selecting redundant features[11].
Revised: Despite being fast, theydo not consider the relation between the features, which may result in selecting redundant features[11].

Original: In the supervised setting, filter methods consider the relation between the feature and the classlabel, such as fisher_score [20], CIEF [39], and ICAP [31].
Revised: In the supervised setting, filter methods consider the relation between the feature and the classlabel, such as fisher_score [22], CIEF [41], and ICAP [33].

Original: Wrapper methods exploit the performanceof a predictive model to evaluate the quality of a subset of the features [67; 12; 70].
Revised: Wrapper methods exploit the performanceof a predictive model to evaluate the quality of a subset of the features [70; 12; 74].

Original: They are moreeffective than filter methods, yet they are computationally expensive [37].
Revised: They are moreeffective than filter methods, yet they are computationally expensive [39].

Original: Embedded methods incorporate feature selection into the learning phase of another algorithm.
Revised: Embedded methods incorporate feature selection into the learning phase of another algorithm.

Original: Multi-Cluster Feature Selection(MCFS)
Revised: Multi-Cluster Feature Selection(MCFS)

Original: [9] uses regularization to select the best features that keep the multi-cluster structure ofthe data.
Revised: [9] uses regularization to select the best features that keep the multi-cluster structure ofthe data.

Original: Unsupervised Discriminative Feature Selection (UDFS) [68] uses l 2 , 1 regularization anddiscriminative analysis to select the most discriminative features.
Revised: Unsupervised Discriminative Feature Selection (UDFS) [72] uses l 2 , 1 regularization anddiscriminative analysis to select the most discriminative features.

Original: Similarly, in the supervised setting,
Revised: Similarly, in the supervised setting,

Original: RFS [53] and L1_L21 [42] exploit l 2 , 1 -norm to introduce feature sparsity.
Revised: RFS [56] and L1_L21 [45] exploit l 2 , 1 -norm to introduce feature sparsity.

Original: Another direction under the embedded category has recently emerged.
Revised: Another direction under the embedded category has recently emerged.

Original: It uses neural networks toperform feature selection by learning the non-linear dependencies among input features [38].
Revised: It uses neural networks toperform feature selection by learning the non-linear dependencies among input features [40].

Original: Thesuccess of autoencoder as a tool for feature extraction encourages unsupervised methods to explore itspower for feature selection [7; 6; 64; 23; 16].
Revised: Thesuccess of autoencoder as a tool for feature extraction encourages unsupervised methods to explore itspower for feature selection [7; 6; 67; 25; 18].

Original: AutoEncoder Feature Selector (AEFS)
Revised: AutoEncoder Feature Selector (AEFS)

Original: [23] combinesautoencoder regression and group lasso tasks.
Revised: [25] combinesautoencoder regression and group lasso tasks.

Original: Concrete Autoencoder (CAE) [5] learns a concreteselector layer (encoder) that selects stochastic linear combinations of input features during training.
Revised: Concrete Autoencoder (CAE) [5] learns a concreteselector layer (encoder) that selects stochastic linear combinations of input features during training.

Original: After training, it converges to the target number of features.
Revised: After training, it converges to the target number of features.

Original: Despite the high performance of thesemethods, the large number of iterations required to train dense models increases the computational
Revised: Despite the high performance of thesemethods, the large number of iterations required to train dense models increases the computationalcosts significantly.

Revised: The conceptually closest work to ours is the recent work, Quick Selection (QS) [4].

Original: [Figure]
Revised: [Figure]

Original:  neurons withcosts significantly. The conceptually closest work to ours is the recent work, Quick Selection (QS),by [4].
Revised: After convergence, neurons with the highest importance are selected as the most informative features.

Original: It trains a sparse autoencoder from scratch.
Revised: It trains a sparse autoencoder from scratch.

Original: During training, different sparse topologies areexplored using the SET method [51].
Revised: During training, different sparse topologies are exploredusing the SET method [54].

Original: After each training epoch (i.e., pass on the full data), a fractionof the connections with the least magnitude is dropped, and the same fraction is randomly regrown.
Revised: After each training epoch (i.e., pass on the full data), a fraction of theconnections with the least magnitude is dropped, and the same fraction is randomly regrown.

Original: Few works based on multi-layer perceptron networks are proposed for the supervised setting [35; 58].
Revised: Fewworks based on multi-layer perceptron networks are proposed for the supervised setting [37; 61; 71].

Original: In section 4, we denote non Neural Network-based (NN-based) methods as classical methods. 
Revised: In section 4, we denote non Neural Network-based (NN-based) methods as classical methods. [Figure]

Original: 2.2 Sparse Training
Revised: 2.2 Sparse Training

Original: [Figure]
Revised: [Figure]

Original: Sparse training with dynamic sparsity, also known as Dynamic Sparse Training (DST), is a recentresearch direction that aims at accelerating the training of neural networks without sacrificingperformance [25; 51; 44].
Revised: Sparse training with dynamic sparsity, also known as Dynamic Sparse Training (DST), is a recentresearch direction that aims at accelerating the training of neural networks without sacrificingperformance [27; 54; 47].

Original: Asparse neural network is trained from scratch .
Revised: A neural network is initialized with a random sparse topology from scratch .

Original: The sparse topology(connectivity) and the weights are jointly optimized during training.
Revised: The sparse topology (connectivity) and the weights are jointly optimized during training.

Original: A network is initialized witha random sparse topology.

Original: During training, the sparse topology is changed periodically througha drop-and-grow cycle, where a fraction of the parameters are dropped, and the same portionis regrown among different neurons.
Revised: Duringtraining, the sparse topology is changed periodically through a drop-and-grow cycle, where a fractionof the parameters are dropped, and the same portion is regrown among different neurons.

Original: An update schedule determines the frequency of topologyupdates.
Revised: An updateschedule determines the frequency of topology updates.

Original: Many works have been proposed, focusing on improving the performance of sparse trainingfor supervised image classification tasks by introducing different criteria for connection regrowth[18; 32; 56; 46; 69; 47].
Revised: Many works have been proposed, focusing onimproving the performance of sparse training for supervised image classification tasks by introducingdifferent criteria for connection regrowth [20; 34; 59; 49; 73; 50].

Original: DST demonstrated its success in other fields as well, such as continuallearning [59], feature selection [4], ensembling [43], federated learning [73; 8], adversarial training[54], and deep reinforcement learning [60].
Revised: DST demonstrated its success inother fields as well, such as continual learning [62], feature selection [4], ensembling [46], federatedlearning [77; 8], adversarial training [57], and deep reinforcement learning [63].

Original:  Where to pay Attention in Sparse Training (WAST)?
Revised:  Where to pay Attention in Sparse Training (WAST)?

Original: Problem Formulation.
Revised: Problem Formulation.

Original: Let D be a dataset with samples { x ( j ) } n 1 , where x ( j ) ∈ R m , and m and nare the number of features and samples, respectively.
Revised: Let D be a dataset with samples { x ( j ) } n 1 , where x ( j ) ∈ R m , and m and nare the number of features and samples, respectively.

Original: The goal is to select a subset K ⊂ m of thefeatures, in an unsupervised manner, that is most representative of the whole feature space m .
Revised: The goal is to select a subset K ⊂ m of thefeatures, in an unsupervised manner, that is most representative of the whole feature space m .

Original: WAST is a new sparse training method for neural network-based unsupervised feature selection.
Revised: WAST is a new sparse training method for neural network-based unsupervised feature selection.

Original: Thebasic idea of WAST is to pay attention to input features based on their estimated importance duringtraining to detect the most informative features quickly.
Revised: Thebasic idea of WAST is to pay attention to input features based on their estimated importance duringtraining to detect the most informative features quickly.

Original: An overview of WAST is shown in Figure1.
Revised: An overview of WAST is shown in Figure1.

Original: We use a sparse autoencoder with a single hidden layer of h neurons.
Revised: We use a sparse autoencoder with a single hidden layer of h neurons.

Original: Let f W be an autoencodermodel parameterized by sparse weights W = { W (1) , W (2) }.
Revised: Let f W be an autoencodermodel parameterized by sparse weights W = { W (1) , W (2) }.

Original: The network is initialized with a certainsparsity level s = 1 − ∥ W (1) ∥ 

Revised: The network is initialized with a certainsparsity level s = 1 − ∥ W (1) ∥

Original: m × h , where ∥ .
Revised: m × h , where ∥ .

Original: ∥ is the L 0 norm.
Revised: ∥ is the L 0 norm.

Original: The sparsity level is kept fixed duringtraining.
Revised: The sparsity level is kept fixed duringtraining.

Original: Initially, the sparse weights are uniformly distributed on the neurons.
Revised: Initially, the sparse weights are uniformly distributed on the neurons.

Original: During training,besides weights optimization, the sparse topology is optimized such that the sparse connections aregradually redistributed on the most informative features.
Revised: During training,besides weights optimization, the sparse topology is optimized such that the sparse connections aregradually redistributed on the most informative features.

Original: To achieve this goal, we propose new criteriato update the sparse topology and a new update schedule, as we will explain next.
Revised: To achieve this goal, we propose new criteriato update the sparse topology and a new update schedule, as we will explain next.

Original: Training.
Revised: Training.

Original: An autoencoder model is trained to minimize the reconstruction loss.
Revised: An autoencoder model is trained to minimize the reconstruction loss.

Original: We use mean squareerror to measure the loss between an input sample x and the reconstructed one ˜ x as follows:
Revised: We use mean squareerror to measure the loss between an input sample x and the reconstructed one ˜ x as follows:

Original: [Equation]
Revised: [Equation]

Revised: where ˜ x

Revised: = f W ( x ) = σ ( xW (1) )

Original:  Algorithm 
Revised: W (2) , and σ is a non-linear activation function. After each weightupdate using a batch of the data, we adapt the sparse topology. The effect of the update schedule on Algorithm 1 WAST

Original: ←
Revised: ←

Original: ∀ j ) ∈
Revised: ∀ j ) ∈

Original: B t 
Revised: B t 9:

Original: L ← 1 b (cid:80) bj =

Revised: L ← 1 b (cid:80) bj =

Original: (cid:13)(cid:13)(cid:13) ˜ x
Revised: (cid:13)(cid:13)(cid:13) ˜ x

Original: ( j ) − x ( j ) (cid:13)(cid:13)(cid:13) 210:
Revised: ( j ) − x ( j ) (cid:13)(cid:13)(cid:13) 210:

Original: W ← W − α ∇ W L 11:
Revised: W ← W − α ∇ W L 11:

Original: I O i ← I
Revised: I O i ← I

Original: O i + λ
Revised: O i + λ

Original: | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk
Revised: | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk

Original: =1 | W (2) ki | ∀ i ∈ { 1 , ... m } 12:
Revised: =1 | W (2) ki | ∀ i ∈ { 1 , ... m } 12:

Original: I
Revised: I

Original: I i ←
Revised: I i ←

Original: I
Revised: I

Original: I i + λ | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk
Revised: I i + λ | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk

Original: =1 | W (1) ik | ∀ i ∈ { 1 , ... m } 13:
Revised: =1 | W (1) ik | ∀ i ∈ { 1 , ..., m } 13:

Original: W ( l  ← Drop ( | W l ) |
Revised: W ( l ) ← Drop ( | W l ) |

Original: I O
Revised: I O

Original: I I α) ∀
Revised: I I α

Original:  l ∈ {
Revised: ∀ l ∈ {

Original: W ( l ) ← Grow
Revised: W ( l ) ← Grow

Original: I O 
Revised: I O , I

Original: I α
Revised: , α

Original: ∀ l ∈ 
Revised: ∀ l ∈ {

Original: 16: R top where ˜ x = f W ( x ) = σ ( xW (1) )
Revised: 2 } 15: end top performance is shown in

Original: W (2) , and σ is a non-linear activation function. After each weightupdate using a batch of the data, we adapt the sparse topology. The effect of the update schedule onperformance is shown in Appendix C.
Revised: Appendix E.

Original: The sparse topology is updated through a drop-and-grow cycle.
Revised: The sparse topology is updated through a drop-and-grow cycle.

Original: A fraction α of the connections with the least importance is dropped from each layer { W (1) , W (2) },and the same portion of new connections are regrown.
Revised: A fraction α of the connections with the least importance is dropped from each layer { W (1) , W (2) },and the same portion of new connections are regrown.

Original: New connections are initialized to zero.
Revised: New connections are initialized to zero.

Original: The main contribution of this work lies in where the new connections are regrown.
Revised: The main contribution of this work lies in where the new connections are regrown.

Original: Different from theprevious method (QS) [4], where the connections are randomly regrown, we optimize the topology topay fast attention to informative features during training.
Revised: Different from theprevious method (QS) [4], where the connections are randomly regrown, we optimize the topology topay fast attention to informative features during training.

Original: To this end, we measure the importance ofeach neuron in the input and output layers by leveraging the information available during training.
Revised: To this end, we measure the importance ofeach neuron in the input and output layers by leveraging the information available during training.

Original: The importance of a neuron is estimated by its impact on the reconstructed loss and the magnitude ofits connected weights.
Revised: The importance of a neuron is estimated by its impact on the reconstructed loss and the magnitude ofits connected weights.

Original: Specifically, we measure how sensitive the reconstruction loss L is to changes in the reconstructedoutput ˜ x .
Revised: Specifically, we measure how sensitive the reconstruction loss L is to changes in the reconstructedoutput ˜ x .

Original: The first-order approximation for the change in loss with respect to a small perturbationδ = { δ i } in the output ˜ x
Revised: The first-order approximation for the change in loss with respect to a small perturbationδ = { δ i } in the output ˜ x

Original: = { ˜ x i } can be written as a sum of its individual components as follows:
Revised: = { ˜ x i } can be written as a sum of its individual components as follows:

Original: [Equation]
Revised: [Equation]

Original: where ∂L∂ ˜ x
Revised: where ∂L∂ ˜ x

Original: i is the gradient of the loss with respect to the output neuron ˜ x i .
Revised: i is the gradient of the loss with respect to the output neuron ˜ x i .

Original: We define the importanceof a neuron i in the output layer at training iteration t as:
Revised: We define the importanceof a neuron i in the output layer at training iteration t as:

Original: [Equation]
Revised: [Equation]

Original: where | W (2) ji | is the magnitude of the incoming connection from neuron j to neuron i and λ is ahyperparameter coefficient to balance the two components (See Appendix A).
Revised: where | W (2) ji | is the magnitude of the incoming connection from neuron j to neuron i and λ is ahyperparameter coefficient to balance the two components (See Appendix A).

Original: The same criterionis applied for the input neurons ( I ( t ) I i ), except that the magnitude of input weights ( | W (1) | ) is usedinstead of the output weights.
Revised: The same criterionis applied for the input neurons ( I ( t ) I i ), except that the magnitude of input weights ( | W (1) | ) is usedinstead of the output weights.

Original: This criterion considers two cases: the information from large gradients,especially at the beginning of the training, and the large weights resulting from the optimizationat later stages.
Revised: This criterion considers two cases: the information from large gradients,especially at the beginning of the training, and the large weights resulting from the optimizationat later stages.

Original: The new connections in the input and output layers are regrown on the neuronswith the highest importance.
Revised: The new connections in the input and output layers are regrown on the neuronswith the highest importance.

Original: For instance, the new connections in the input layer are given bytop i ̸∈ ˜ W
Revised: For instance, the new connections in the input layer are given by

Original: (1) ( I I ×
Revised: [Equation]

Original: I ⊤ H , r ) , where top ( Q, k ) gives the indices of the top- k elements in a matrix Q , I His the importance of the hidden neurons, ˜ W (1) is the set of remaining weights after the drop phase, and r is the number of regrown connections based on the fraction α .
Revised: , r ) , where top ( Q, k ) gives the indices of the top- k elements in a matrix Q , I His the importance of the hidden neurons, ˜ W (1) is the set of remaining weights after the drop phase,and r is the number of regrown connections based on the fraction α .

Original: We consider the neurons in thehidden layer to be equally important.
Revised: We consider the neurons in thehidden layer to be equally important.

Original: During the drop phase, the importance of a connection w in the input (output) layer is estimated byits magnitude and the importance of its connected input (output) neuron ( I ( t ) i ) as follows:
Revised: During the drop phase, the importance of a connection w in the input (output) layer is estimated byits magnitude and the importance of its connected input (output) neuron ( I ( t ) i ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where I ( t ) i takes the value of I ( t )
Revised: where I ( t ) i takes the value of I ( t )

Original: I i and I ( t )
Revised: I i and I ( t )

Original: O i for the input and output layers, respectively.
Revised: O i for the input and output layers, respectively.

Original: The effectof each component in the neuron and connection importance criteria is studied in Section 5.3.
Revised: The effectof each component in the neuron and connection importance criteria is studied in Section 5.3.

Original: Theperiodic update of the sparse topology redistributes the connections on the effective features.
Revised: Theperiodic update of the sparse topology redistributes the connections on the effective features.

Original: Feature Selection.
Revised: Feature Selection.

Original: After training, we select the top- K neurons with the highest importance from theinput layer as the most informative K features.
Revised: After training, we select the top- K neurons with the highest importance from theinput layer as the most informative K features.

Original: The details of WAST are provided in Algorithm 1.
Revised: The details of WAST are provided in Algorithm 1.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Baselines
Revised: 4.1 Baselines

Original: We compared our method with several autoencoder-based (QS [4], CAE [5], AEFS [23]) and classicalmethods (lap_score [24], MCFS [9]) for unsupervised feature selection.
Revised: We compared our method with several autoencoder-based (QS [4], CAE [5], AEFS [25]) and classicalmethods (lap_score [26], MCFS [9], DUFS [42]) for unsupervised feature selection.

Original: On top of that, we includecomparisons with six state-of-the-art supervised feature selection methods.
Revised: On top of that,we include comparisons with seven state-of-the-art supervised feature selection methods.

Original: Although the latter is notthe focus of this paper and not a typical comparison in the literature, we prefer to show that WASTperforms competitively also with the supervised methods while being computationally efficient andapplicable for cases where labels are not available or expensive to collect.
Revised: Althoughthe latter is not the focus of this paper and not a typical comparison in the literature, we prefer to showthat WAST performs competitively also with the supervised methods while being computationallyefficient and applicable for cases where labels are not available or expensive to collect.

Original: 4.2 Datasets
Revised: 4.2 Datasets

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We evaluate our method on 10 publicly available datasets, including image, speech, text, time series, biological, and artificial data.
Revised: We evaluate our method on 10 publicly available datasets, including image, speech, text, time series, biological, and artificial data.

Original: They havea variety of characteristics, such aslow and high dimensional features anda small and large number of trainingsamples.
Revised: They havea variety of characteristics, such aslow and high dimensional features anda small and large number of trainingsamples.

Original: Details are in Table 1.
Revised: Details are in Table 1.

Original: 4.3 Experimental Settings
Revised: 4.3 Experimental Settings

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: An efficient feature selection method should have high learning accuracy withless memory and computational costs [10].
Revised: An efficient feature selection method should have high learning accuracy withless memory and computational costs [10].

Original: Hence, we evaluate multiple metrics.
Revised: Hence, we evaluate multiple metrics.

Original: (1) Classification
Revised: (1) Classification

Original: Accuracy : It is typically evaluated by a machine learning model [10].
Revised: Accuracy : It is typically evaluated by a machine learning model [10].

Original: We trained a classifier using the
Revised: We trained a classifier using the

Original: K selected features by the studied methods.
Revised: K selected features by the studied methods.

Original: Here, we use one of the most popular classifiers, supportvector machines (SVM) [14].
Revised: Here, we use one of the most popular classifiers, supportvector machines (SVM) [14].

Original: Other classifiers could be used.
Revised: Other classifiers could be used.

Original: Yet, we choose to use a non-NN-basedclassifier for evaluation to avoid any biased advantages towards the NN-based baselines over theothers.
Revised: Yet, we choose to use a non-NN-basedclassifier for evaluation to avoid any biased advantages towards the NN-based baselines over theothers.

Original: We studied the performance of 6 different values of K .
Revised: We studied the performance of 6 different values of K .

Original: We report the average accuracy of thetest data over 5 seeds.
Revised: We report the average accuracy of thetest data over 5 seeds.

Original: (2) Memory cost: We calculate the number of network parameters (#params)used by each NN-based method.
Revised: (2) Memory cost: We calculate the number of network parameters (#params)used by each NN-based method.

Original: (3) Computational cost: We calculate the number of Floating-pointoperations (FLOPs) consumed to train a neural network.
Revised: (3) Computational cost: We calculate the number of Floating-pointoperations (FLOPs) consumed to train a neural network.

Original: See Appendix A.2 for details.
Revised: See Appendix A.2 for details.

Original: Implementation.
Revised: Implementation.

Original: We implemented WAST and QS [4] with PyTorch [55] (see Appendix E).
Revised: We implemented WAST and QS [4] with PyTorch [58] (see Appendix I).

Original: For CAE [5] and AEFS [23], we used the code provided by the authors of CAE with MIT license 2 .
Revised: For CAE[5] and AEFS [25], we used the code provided by the authors of CAE with MIT license 2 .

Original: For  LassoNet [35], we used the official code of the paper with MIT license 3 .
Revised: For DUFS[42], LassoNet [37] and STG [71], we used the official codes with MIT license 3,4,5 .

Original: For classical baselines, weused the Scikit-Feature library with GNU General Public license [37] 4 .
Revised: For classicalbaselines, we used the Scikit-Feature library with GNU General Public license [39] 6 .

Original: All NN-based methods are
Revised: NN-based and

Original: [Table]
Revised: [Table]

Original: trained on Nvidia GPUs, while classical methods are trained on CPUs.
Revised: classical methods are trained on Nvidia GPUs and CPUs, respectively.

Original: We consider a 12 hours limiton the running time of each experiment.
Revised: We consider a 12 hours limiton the running time of each experiment.

Original: Experiments that exceed this limit are not considered.
Revised: Experiments that exceed this limit are not considered.

Original: Implementation Details.
Revised: Implementation Details.

Original: For all NN-based methods except CAE [5], we use a single hidden layerof 200 neurons.
Revised: For all NN-based methods except CAE [5], we use a single hidden layerof 200 neurons.

Original: The architecture of CAE consists of two layers.
Revised: The architecture of CAE consists of two layers.

Original: The size of the hidden layers isdependent on the chosen K ; [ K, 32 K ].
Revised: The size of the hidden layers isdependent on the chosen K ; [ K, 32 K ].

Original: For WAST and QS, we use a sparsity level of 0.8.
Revised: For WAST and QS, we use a sparsity level of 0.8.

Original: Following[4], we report the accuracy of NN-based baselines after 100 epochs unless stated otherwise.
Revised: Following[4], we report the accuracy of NN-based baselines after 100 epochs unless stated otherwise.

Original: Notethat some baselines reported a higher number of epochs (e.g., CAE [5] uses 200 epochs for somedatasets).
Revised: Notethat some baselines reported a higher number of epochs (e.g., CAE [5] uses 200 epochs for somedatasets).

Original: Yet, we keep 100 epochs for all cases for a fair comparison.
Revised: Yet, we keep 100 epochs for all cases for a fair comparison.

Original: For WAST, we train the modelfor 10 epochs.
Revised: For WAST, we train the modelfor 10 epochs.

Original: Details of the hyperparameters are in Appendix A.1.
Revised: Details of the hyperparameters are in Appendix A.1.

Original: 4.4 Results
Revised: 4.4 Results

Original: Accuracy.
Revised: Accuracy.

Original: Table 2 shows the classification accuracy of the studied datasets.
Revised: Table 2 shows the classification accuracy of the studied datasets.

Original: Here, we report thechallenging case where a few best informative features have to be selected.
Revised: Here, we report thechallenging case where a few best informative features have to be selected.

Original: We use a K of 50 for alldatasets except for Madelon, where a K of 20 is used as the remaining 480 features are pure noise.
Revised: We use a K of 50 for alldatasets except for Madelon, where a K of 20 is used as the remaining 480 features are pure noise.

Original: Experiments with various values for K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } can be found in Appendix B,with a summary provided next.
Revised: Experiments with various values for K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } can be found in Appendix B,with a summary provided next.

Original: Unsupervised NN-based methods outperform the classical ones on all datasets.
Revised: Unsupervised NN-based methods outperform the classical ones on all datasets except one, wherethe performance difference is marginal.

Original: WAST outperformsother unsupervised baselines on all datasets, except MNIST and Fashion MNIST, where CAE isthe best performer.
Revised: WAST outperforms unsupervised NN-based methods onall datasets, except MNIST and Fashion MNIST, where CAE is the best performer.

Original: It is worth emphasizing that the superior performance of WAST is achievedwith a 90% reduction in the number of training iterations.
Revised: It is worthemphasizing that the superior performance of WAST is achieved with a 90% reduction in the numberof training iterations.

Original: For the supervised baselines, LassoNetoutperforms the classical supervised baselines in 6 cases.
Revised: Supervised baselines outperform the classical supervised ones in 6 cases.

Original: Interestingly, WAST achieves a competitiveperformance to the supervised baselines.
Revised: Interestingly, WAST achieves a competitive performance to the supervised baselines.

Original: It outperforms the best-supervised performer in 5cases andis the second-best performer in other four cases.
Revised: It outperformsthe best-supervised performer in 5 cases.

Original: Memory and Computational Costs.
Revised: Memory and Computational Costs.

Original: Table 3 shows the memory and computational costs consumedby each unsupervised NN-based method.
Revised: Table 3 shows the memory and computational costs consumedby each unsupervised NN-based method.

Original: The high memory and computational costs are causedby the high number of training samples as in MNIST or the high-dimensional feature space as in 
Revised: The high memory and computational costs are causedby the high number of training samples as in MNIST or the high-dimensional feature space as in [Figure] [Figure] AEFS

Revised: [Figure]

Revised: [Figure]

Original: GLA.
Revised: GLA.

Original: Training a sparse model from scratch as in WAST and QS reduces these costs substantially.
Revised: Training a sparse model from scratch as in WAST and QS reduces these costs substantially.

Original: QS reduces the computational cost by 80%.
Revised: QS reduces the computational cost by 80%.

Original: Interestingly, besides the reduction resulting from

Original: [Figure] [Figure] [Figure] using sparse neural networks, WAST reduces the computational cost further to 98% due to the fastidentification of the informative features.
Revised: Interestingly, besides the reduction resulting fromusing sparse neural networks, WAST reduces the computational cost further to 98% due to the fastidentification of the informative features.

Original: Both WAST and QS reduce the network size by 80%.
Revised: Both WAST and QS reduce the network size by 80%.

Original: The network size of CAE is dependent on the target value of K .
Revised: The network size of CAE is dependent on the target value of K .

Original: This has two limitations: trainingthe model for every target K and the increase of the memory and computational costs with highervalues for K .
Revised: This has two limitations: trainingthe model for every target K and the increase of the memory and computational costs with highervalues for K .

Original: On the other hand, WAST derives the importance of each feature with a single trainingrun independently of the target K .
Revised: On the other hand, WAST derives the importance of each feature with a single trainingrun independently of the target K .

Original: This is illustrated in Figure 2.
Revised: This is illustrated in Figure 2.

Original: The figure shows the accumulated
Revised: The figure shows the accumulated

Original: FLOPs for all datasets except Madelon at each value of the studied K .
Revised: FLOPs for all datasets except Madelon at each value of the studied K .

Original: Overall Performance.
Revised: Overall Performance.

Original: To give a holistic view of the performance of each method across alldimensions (accuracy, memory cost, and computational cost) on all datasets and all studied valuesof K , we calculate the performance in each case.
Revised: To give a holistic view of the performance of each method across alldimensions (accuracy, memory cost, and computational cost) on all datasets and all studied values of K , we calculate the performance in each case.

Original: Figure 3 illustrates the normalized performanceusing min-max scaling for all cases.
Revised: Figure 3 illustrates the normalized performance usingmin-max scaling for all cases.

Original: The “score” metric represents the number of times a methodis the best performer in terms of accuracy.
Revised: The “score” metric represents the number of times a method is thebest performer in terms of accuracy.

Original: Details are provided in Appendix A.2.
Revised: Details are provided in Appendix A.2.

Original: CAE and AEFShave the same score, while the former is the best only for image datasets with a large number ofsamples (i.e., MNIST and Fashion MNIST) as shown in Figure 4.
Revised: CAE has a higher scorethan AEFS, while the former is the best only for image datasets with a large number of samples (i.e., MNIST and Fashion MNIST) as shown in Figure 4.

Original: AEFS has the highest memoryand computational costs.
Revised: AEFS has the highest memory and computationalcosts.

Original: QS improves the memory and computational costs but has a lower score.
Revised: QS improves the memory and computational costs but has a lower score.

Original: WAST has the highest score and spans more dataset types with different characteristics (e.g., highdimensional data) than the baselines (Figure 4).
Revised: WAST has thehighest score and spans more dataset types with different characteristics (e.g., high dimensionaldata) than the baselines (Figure 4).

Original: The performance gain is accompanied by improvingmemory and computational efficiency significantly.
Revised: The performance gain is accompanied by improving memory andcomputational efficiency significantly.

Original: Detailed results are provided in Appendix B.
Revised: Detailed results are provided in Appendix B.

Original:  Analysis
Revised:  Analysis

Original: 5.1 Effect of Fast Attention During Training
Revised: 5.1 Effect of Fast Attention During Training

Original: To analyze the impact of attention to the informative features in WAST during training, we studythe learning behavior of all unsupervised NN-based methods on the test data for the first 10 epochs.
Revised: To analyze the impact of attention to the informative features in WAST during training, we studythe learning behavior of all unsupervised NN-based methods on the test data for the first 10 epochs.

Original: Since CAE has a parameter that is dependent on the max number of epochs, we performed tenseparate runs and varied the max number of epochs each time.
Revised: Since CAE has a parameter that is dependent on the max number of epochs, we performed tenseparate runs and varied the max number of epochs each time.

Original: Figure 5 shows the accuracy using
Revised: Figure 5 shows the accuracy using

Original: K of 50 for all datasets except Madelon, where K of 20 is used.
Revised: K of 50 for all datasets except Madelon, where K of 20 is used.

Original: The analysis reveals the followingfindings: (1) Robustness to noisy environments: The results on Madelon show the robustness of
Revised: The analysis reveals the followingfindings: (1) Robustness to noisy environments: The results on Madelon show the robustness of

Original: WAST against the noisy features (480 out of 500).
Revised: WAST against the noisy features (480 out of 500).

Original: The figure shows the performance gap between
Revised: The figure shows the performance gap between

Original: WAST and the baseline methods starting from epoch 1.
Revised: WAST and the baseline methods starting from epoch 1.

Original: After 10 epochs, WAST outperforms thesecond-best performer by 22%.
Revised: After 10 epochs, WAST outperforms the

Original: (2) Performance on datasets with a large amount of samples:

Original: WAST outperforms CAE by 3.6% and 1.6% on MNIST and FashionMNIST, respectively, although

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Initialization {+2436480 8 
+}
Revised: Initialization {+2436480 8 
+}

Original: [Figure]

Original: Epoch3042 0 8 

Revised: Epoch3042 0 8

Original: Epoch2040 (c) QS [4].
Revised: Epoch2040 (c) QS [4].

Revised: second-best performer by 22%.

Revised: (2) Performance on datasets with a large amount of samples:

Revised: WAST outperforms CAE by 3.6% and 1.6% on MNIST and FashionMNIST, respectively, although

Original: CAE was the best performer on these datasets after training it for a larger number of epochs (Table2)
Revised: CAE was the best performer on these datasets after training it for a larger number of epochs (Table2).

Original:  Consistency  different dataset types and characteristics:
Revised: (3) Consistency across different dataset types and characteristics:

Original: We observe that somebaselines achieve high performance on some domains while being worse in others.
Revised: We observe that somebaselines achieve high performance on some domains while being worse in others.

Original: WAST has  the highest
Revised: WAST has thehighest consistency on different dataset types and characteristics, outperforming all baselines after 10epochs (Appendix C). (4) Stability: WAST is more stable at the early stage of the training. This isrepresented by the standard deviation across different training runs (shaded region).

Original: 5.2 Visualization
Revised: 5.2 Visualization

Original: [Figure] Figure 6 illustrates how the sparse topology changes during training in WAST and the QS baseline.
Revised: Figure 6 illustrates how the sparse topology changes during training in WAST and QS.

Original: We performed this analysis on MNIST, where handwritten digits are centered in 28 × 28 grayscaleimages (Figure 6a).
Revised: We performedthis analysis on MNIST, where handwritten digits are centered in 28 × 28 grayscale images (Figure6a).

Original: Initially, the sparse connections are uniformly distributed on the input neurons.
Revised: Initially, the sparse connections are uniformly distributed on the input neurons.

Original: With the guided adaptation of a sparse topology via attention, WAST can detect the informativefeatures quickly after one epoch.
Revised: With the guidedadaptation of a sparse topology via attention, WAST detects the informative features quickly after oneepoch.

Original: On the other hand, the random regrowth of the connections in QSrequires more training epochs.
Revised: On the other hand, the random growth of the connections in QS requires more training epochs.

Original: The effect of increasing the frequency of topological exploration in QS is studied in Appendix D.
Revised: The effect of increasing the frequency of topological exploration in QS is studied in Appendix F.

Original: 5.3 Ablation Study
Revised: 5.3 Ablation Study

Original: We performed an ablation study to assess the contribution of each component in the proposed criteriaof neuron and connection importance in the performance of WAST.
Revised: We performed an ablation study to assess the contribution of each component in the proposed criteriaof neuron and connection importance in the performance of WAST.

Original: Neuron Importance.
Revised: Neuron Importance.

Original: (#1) “w/o | ∂L∂ ˜ x | ”: Using only the magnitude of connected weights to determinethe importance of a neuron (i.e., λ = 0 ), (#2) “w/o | W |”: Using only the sensitivity of the neuron tothe change in the loss (i.e., λ = 1 ) (#3) “w/o momentum”:  importance of a neuron is basedonly on  current estimate of its  loss and currently connected weights.
Revised: (#1) “w/o | ∂L∂ ˜ x | ”: Using only the magnitude of connected weights to determinethe importance of a neuron (i.e., λ = 0 ), (#2) “w/o | W |”: Using only the sensitivity of the neuron tothe change in the loss (i.e., λ = 1 ), and (#3) “w/o momentum”: The importance of a neuron is basedonly on the current estimate of its sensitivity to the loss and currently connected weights.

Original: [Table]
Revised: [Table]

Original: Connection Importance.
Revised: Connection Importance.

Original: (#4) “w/o I i ”:
Revised: (#4) “w/o I i ”:

Original: The importance of a connection is estimated by itsmagnitude without considering the importance of its connected neuron I i .
Revised: The importance of a connection is estimated by itsmagnitude without considering the importance of its connected neuron I i .

Original: We performed this analysis on datasets of different types.
Revised: We performed this analysis on datasets of different types.

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 4.

Original: Weobserve that removing the sensitivity of a neuron to the loss (#1) has the biggest effect on performance.
Revised: Weobserve that removing the sensitivity of a neuron to the loss (#1) has the biggest effect on performance.

Original: It has the highest impact on noisy environments, where it reduces the performance by 27.77% on
Revised: It has the highest impact on noisy environments, where it reduces the performance by 27.77% on

Original: Madelon.
Revised: Madelon.

Original: On the other datasets, it results in a 0 .
Revised: On the other datasets, it results in a 0 .

Original: 53 − 4 .
Revised: 53 − 4 .

Original: 93% reduction in the performance.
Revised: 93% reduction in the performance.

Original: Excludingthe magnitude of the weights (#2) affects mainly the datasets with few training samples (e.g., SMK).
Revised: Excludingthe magnitude of the weights (#2) affects mainly the datasets with few training samples (e.g., SMK).

Original: Neglecting the neuron importance in the previous training iterations (#3) reduces the performance by0 .
Revised: Neglecting the neuron importance in the previous training iterations (#3) reduces the performance by0 .

Original: 09 − 5 .
Revised: 09 − 5 .

Original: Aligned with many prior works [18; 51; 25], we find that the weight magnitude isan effective metric for estimating the connection importance (#4).
Revised: Aligned with many prior works [20; 54; 27], we find that the weight magnitude isan effective metric for estimating the connection importance (#4).

Original: Moreover, our results show thatenhancing it with the importance of its connected neuron improves the performance even further onsome datasets without diminishing it on the others.
Revised: Moreover, our results show thatenhancing it with the importance of its connected neuron improves the performance even further onsome datasets without diminishing it on the others.

Original:  Discussion
Revised:  Discussion

Original: Conclusion.
Revised: Conclusion.

Original: In this paper, we propose WAST, a new efficient neural network-based method forunsupervised feature selection.
Revised: In this paper, we propose WAST, a new efficient neural network-based method forunsupervised feature selection.

Original: We train a sparse autoencoder from scratch and optimize the sparsetopology during training to detect the informative features quickly.
Revised: We train a sparse autoencoder from scratch and optimize the sparsetopology during training to detect the informative features quickly.

Original: We performed extensive experiments in which we evaluated 55 cases on various datasets and different values of the number ofselected features.
Revised: We performed extensive experiments in which we evaluated 55 cases on various datasets and different values of the number ofselected features.

Original: WAST achieves the best performance on 20 cases, while the second-best unsupervised performer has a score of 11 cases.
Revised: WAST achieves the best performance on 19 cases, while the second-best unsupervised performer has a score of 11 cases.

Original: More interestingly, the superior performance is achievedwith a few training iterations.
Revised: More interestingly, the superior performance is achievedwith a few training iterations.

Original: WAST reduces the memory and computational costs by 80% and98%, respectively.
Revised: WAST reduces the memory and computational costs by 80% and98%, respectively.

Original: Moreover, we show the robustness of WAST towards very noisy environments,outperforming the state-of-the-art methods by 22% with limited training iterations.
Revised: Moreover, we show the robustness of WAST towards very noisy environments,outperforming the state-of-the-art methods by 22% with limited training iterations.

Original: Finally, we showthat WAST performs competitively with supervised-based methods, outperforming them on imagedatasets.
Revised: Finally, we showthat WAST performs competitively with supervised-based methods, outperforming them on imagedatasets.

Original: This demonstrates the promise of adapting WAST in the supervised setting and motivatesnew sparse training algorithms for supervised and unsupervised feature selection.
Revised: This demonstrates the promise of adapting WAST in the supervised setting and motivatesnew sparse training algorithms for supervised and unsupervised feature selection.

Original: Limitations.
Revised: Limitations.

Original: This work is a step toward exploiting the power of neural networks for feature selectionin a computationally efficient manner.
Revised: This work is a step toward exploiting the power of neural networks for feature selectionin a computationally efficient manner.

Original: Besides the improvement in the accuracy of selecting theinformative feature, fast attention to the important features during training reduces the number oftraining iterations substantially.
Revised: Besides the improvement in the accuracy of selecting theinformative feature, fast attention to the important features during training reduces the number oftraining iterations substantially.

Original: Yet, like most sparse training methods in the literature, this proof-ofconcept does not fully utilize the memory and computational advantages of sparse neural networks.
Revised: Yet, like most sparse training methods in the literature, this proof-ofconcept does not fully utilize the memory and computational advantages of sparse neural networks.

Original: This is due to the lack of hardware support for sparsity and the higher focus of the community on thealgorithmic side [25].
Revised: This is due to the lack of hardware support for sparsity and the higher focus of the community on thealgorithmic side [27].

Original: Nevertheless, there is recent growing attention for the hardware and softwaresupport for sparsity [26; 72; 63; 45; 15; 3; 13] (See Appendix E for discussion).
Revised: Nevertheless, there is recent growing attention for the hardware and softwaresupport for sparsity [28; 76; 66; 48; 15; 3; 13] (See Appendix I for discussion).

Original: This enables the
Revised: This enables pure

Original: Societal Impacts.
Revised: Societal Impacts.

Original: With the emergence of big data, systems that can quickly select the informativefeatures and remove the redundant ones become crucial.
Revised: With the emergence of big data, systems that can quickly select the informativefeatures and remove the redundant ones become crucial.

Original: Besides the performance gain that could beachieved in the downstream tasks by removing irrelevant features, reducing the number of featuresimproves the memory and computational efficiency significantly.
Revised: Besides the performance gain that could beachieved in the downstream tasks by removing irrelevant features, reducing the number of featuresimproves the memory and computational efficiency significantly.

Original: This enables providing energyefficient systems.
Revised: This enables providing energyefficient systems.

Original: Moreover, it is useful for improving the interpretability of model-driven decisions.
Revised: Moreover, it is useful for improving the interpretability of model-driven decisions.

Original: The unsupervised selection of the informative features is effective in the cases where labeled data islimited or very expensive to collect.
Revised: The unsupervised selection of the informative features is effective in the cases where labeled data islimited or very expensive to collect.

Original: We do not expect that there is a negative societal impact.
Revised: We do not expect that there is a negative societal impact.