Original: Where to Pay Attention in Sparse Training for Feature Selection?
Revised: Where to Pay Attention in Sparse Training for Feature Selection?

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: A new line of research for feature selection based on neural networks has recentlyemerged.
Revised: A new line of research for feature selection based on neural networks has recently emerged.

Original: Despite its superiority to classical methods, it requires many trainingiterations to converge and detect informative features.
Revised: Despite its superiority to classical methods, it requires many training iterations to converge and detect informative features.

Original: For datasets with a largenumber of samples or a very high dimensional feature space, the computational timebecomes prohibitively long.
Revised: The computational time becomes prohibitively long for datasets with a large number of samples or a very high dimensional feature space.

Original: In this paper, we present a new efficient unsupervisedmethod for feature selection based on sparse autoencoders.
Revised: In this paper, we present a new efficient unsupervised method for feature selection based on sparse autoencoders.

Original: In particular, wepropose a new sparse training algorithm that optimizes a model’s sparse topologyduring training toquickly pay attention to informative features.
Revised: In particular, we propose a new sparse training algorithm that optimizes a model’s sparse topology during training to pay attention to informative features quickly.

Original: The attention-basedadaptation of the sparse topology enables fast detection of informative features aftera few training iterations.
Revised: The attention-based adaptation of the sparse topology enables fast detection of informative features after a few training iterations.

Original: We performed extensive experiments on 10 datasets ofdifferent types, including image, speech, text, artificial, and biological.
Revised: We performed extensive experiments on 10 datasets of different types, including image, speech, text, artificial, and biological.

Original: They covera wide range of characteristics, such as low and high-dimensional feature spaces,as well as few and large training samples.
Revised: They cover a wide range of characteristics, such as low and high-dimensional feature spaces, and few and large training samples.

Original: Our proposed approach outperformsthe state-of-the-art methods in terms of the selection of informative features whilereducing training iterations and computational costs substantially.
Revised: Our proposed approach outperforms the state-of-the-art methods in terms of selecting informative features while reducing training iterations and computational costs substantially.

Original: Moreover, theexperiments show the robustness of our method in extremely noisy environments 1 .
Revised: Moreover, the experiments show the robustness of our method in extremely noisy environments 1 .

Original:  Introduction
Revised:  Introduction

Original: Feature selection plays a crucial role in data mining and machine learning tasks with the explosion inthe size and dimensionality of real-world data [68; 10; 30; 44; 23].
Revised: Feature selection plays a crucial role in data mining and machine learning tasks with the explosion in the size and dimensionality of real-world data [68; 10; 30; 44; 23].

Original: It aims to efficiently select a subsetof the features that are most informative and remove the irrelevant or redundant ones.
Revised: It aims to efficiently select a subset of the features that are most informative and remove the irrelevant or redundant ones.

Original: It is usefulfor alleviating the curse of dimensionality, interpretability of model-driven decisions, generalizationof downstream tasks, reducing the prohibitive memory and computational costs and avoiding theexpensive costs of collecting the full set of features as in biological studies [11; 38; 31; 52].
Revised: It is useful for alleviating the curse of dimensionality, interpretability of model-driven decisions, generalization of downstream tasks, reducing the prohibitive memory and computational costs and avoiding the expensive costs of collecting the full set of features as in biological studies [11; 38; 31; 52].

Original: Many methods have been proposed for feature selection in supervised, semi-supervised and unsupervised settings [75; 10; 60; 51; 19; 1].
Revised: Many methods have been proposed for feature selection in supervised, semi-supervised, and unsupervised settings [75; 10; 60; 51; 19; 1].

Original: Recently, using neural networks for feature selectionhas received increasing attention due to its power to learn non-linear dependencies among inputfeatures [39].
Revised: Recently, using neural networks for feature selection has received increasing attention due to its power to learn non-linear dependencies among input features [39].

Original: One limitation of current neural network-based methods is that they are computationallyexpensive.
Revised: One limitation of current neural network-based methods is that they are computationally expensive.

Original: The computational costs result from training dense models for many training iterationsuntil informative features are recognized.
Revised: The computational costs result from training dense models for many training iterations until informative features are recognized.

Original: The costs increase for datasets with a very large numberof training samples or high dimensional feature space.
Revised: The costs increase for datasets with a very large number of training samples or high dimensional feature space.

Original: This limitation is recently addressed in [4]by utilizing sparse networks  for feature selection.
Revised: This limitation is recently addressed in [4] by utilizing sparse networks with dynamic sparsity for feature selection.

Original: It trains an autoencoder  with the sparse trainingalgorithm SET [54], which explores different sparse topologies during training by dropping a portion of the connections and growing the same portion randomly .
Revised: An autoencoder is trained with the sparse training algorithm SET [54], which explores different sparse topologies during training by dropping a portion of the connections and growing the same portion randomly .

Original: The importance of an input feature isestimated after convergence based on its outgoing sparse connections.
Revised: The importance of an input feature is estimated after convergence based on its outgoing sparse connections.

Original: Sparse networks decreasememory and computational costs compared to previous dense-based methods.
Revised: Training sparse networks from scratch decreases memory and computational costs compared to previous dense-based methods.

Original: Yet, the randomexploration of sparse topologies requires many training iterations to identify informative features.
Revised: Yet, the random exploration of sparse topologies requires many training iterations to identify informative features.

Original: In this paper, we propose an efficient unsupervised method for feature selection.
Revised: In this paper, we propose an efficient unsupervised method for feature selection.

Original: We introduce anew sparse training algorithm for autoencoders that quickly detects the informative input features.
Revised: We introduce a new sparse training algorithm for autoencoders that quickly detects the informative input features.

Original: Specifically, we optimize the sparse topology to learn W here to pay A ttention during S parse T raining.
Revised: Specifically, we optimize the sparse topology to learn W here to pay A ttention during S parse T raining.

Original: We named our method WAST .
Revised: We named our method WAST .

Original: WAST exploits the information from the reconstruction loss and theweights of sparse connections to guide the topological search.
Revised: WAST exploits the information from the reconstruction loss and the weights of sparse connections to guide the topological search.

Original: It redistributes the sparse connectionsto spot informative features quickly.
Revised: It redistributes the sparse connections to spot informative features quickly.

Original: We evaluate our proposed approach on ten datasets from variousdomains, including image, speech, text, artificial, and biological.
Revised: We evaluate our proposed approach on ten datasets from various domains, including image, speech, text, artificial, and biological.

Original: Through extensive experiments,we find that WAST identifies the informative features after a few training iterations and outperformsstate-of-the-art unsupervised feature selection methods.
Revised: Through extensive experiments, we find that WAST identifies the informative features after a few training iterations and outperforms state-of-the-art unsupervised feature selection methods.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We propose a new efficient unsupervised method for feature selection, named WAST.
Revised: • We propose a new efficient unsupervised method for feature selection, named WAST.

Original: WASToptimizes the sparse topology of an autoencoder to detect the informative features quickly.•
Revised: WAST optimizes the sparse topology of an autoencoder to detect the informative features quickly.

Original:  We perform extensive experiments on 10 benchmarks that cover various types and characteristics.
Revised: • We perform extensive experiments on 10 benchmarks that cover various types and characteristics.

Original: Experimental results show the effectiveness of our method over the state-of-the-art 
Revised: Experimental results show the effectiveness of our method over the state-of-the-art methods in terms of selecting the informative features.

Original: methods in terms of selecting the informative features.• WAST reduces the computational time compared to neural network-based methods substantially by reducing training iterations and employing highly sparse neural networks.•
Revised: • WAST reduces the computational time compared to neural network-based methods substantially by reducing training iterations and employing highly sparse neural networks.

Original:  We illustrate the robustness of our method in extremely noisy environments and its effec- tiveness for datasets with very high dimensional feature space and a few training samples.
Revised: • We illustrate the robustness of our method in extremely noisy environments and its effectiveness for datasets with very high dimensional feature space and a few training samples.

Original:  Related Work2.1 Feature Selection
Revised:  Related Work 2.1 Feature Selection

Original: Many feature selection methods were introduced for the supervised, semi-supervised, and unsupervised settings depending on whether the data is labeled, partially labeled, or unlabeled, respectively[75; 10; 60; 1].
Revised: Many feature selection methods were introduced for the supervised, semi-supervised, and unsupervised settings depending on whether the data is labeled, partially labeled, or unlabeled, respectively [75; 10; 60; 1].

Original: Typically, feature selection methods are divided into three categories: filter, wrapper,and embedded methods.
Revised: Typically, feature selection methods are divided into three categories: filter, wrapper, and embedded methods.

Original: Filter methods are independent of the learning algorithms.
Revised: Filter methods are independent of the learning algorithms.

Original: They use rankingtechniques for providing scores to the features, such as Laplacian score [26].
Revised: They use ranking techniques for providing scores to the features, such as Laplacian score [26].

Original: Despite being fast, theydo not consider the relation between the features, which may result in selecting redundant features[11].
Revised: Despite being fast, they do not consider the relationship between the features, which may result in the selection of redundant features [11].

Original: In the supervised setting, filter methods consider the relation between the feature and the classlabel, such as fisher_score [22], CIEF [41], and ICAP [33].
Revised: In the supervised setting, filter methods consider the relation between the feature and the class label, such as fisher_score [22], CIEF [41], and ICAP [33].

Original: Wrapper methods exploit the performanceof a predictive model to evaluate the quality of a subset of the features [70; 12; 74].
Revised: Wrapper methods exploit the performance of a predictive model to evaluate the quality of a subset of the features [70; 12; 74].

Original: They are moreeffective than filter methods, yet they are computationally expensive [39].
Revised: They are more effective than filter methods, yet they are computationally expensive [39].

Original: Embedded methods incorporate feature selection into the learning phase of another algorithm.
Revised: Embedded methods incorporate feature selection into the learning phase of another algorithm.

Original: Multi-Cluster Feature Selection(MCFS)
Revised: Multi-Cluster Feature Selection (MCFS)

Original: [9] uses regularization to select the best features that keep the multi-cluster structure ofthe data.
Revised: [9] uses regularization to select the best features that keep the multi-cluster structure of the data.

Original: Unsupervised Discriminative Feature Selection (UDFS) [72] uses l 2 , 1 regularization anddiscriminative analysis to select the most discriminative features.
Revised: Unsupervised Discriminative Feature Selection (UDFS) [72] uses l 2 , 1 regularization and discriminative analysis to select the most discriminative features.

Original: Similarly, in the supervised setting, RFS [56] and L1_L21 [45] exploit l 2 , 1 -norm to introduce feature sparsity.
Revised: Similarly, in the supervised setting, RFS [56] and L1_L21 [45] exploit l 2 , 1 -norm to introduce feature sparsity.

Original: Another direction under the embedded category has recently emerged.
Revised: Another direction under the embedded category has recently emerged.

Original: It uses neural networks toperform feature selection by learning the non-linear dependencies among input features [40].
Revised: It uses neural networks to perform feature selection by learning the non-linear dependencies among input features [40].

Original: Thesuccess of autoencoder as a tool for feature extraction encourages unsupervised methods to explore itspower for feature selection [7; 6; 67; 25; 18].
Revised: The success of autoencoders as a tool for feature extraction encourages unsupervised methods to explore their power for feature selection [7; 6; 67; 25; 18].

Original: AutoEncoder Feature Selector (AEFS)
Revised: AutoEncoder Feature Selector (AEFS)

Original: [25] combinesautoencoder regression and group lasso tasks.
Revised: [25] combines autoencoder regression and group lasso tasks.

Original: Concrete Autoencoder (CAE) [5] learns a concreteselector layer (encoder) that selects stochastic linear combinations of input features during training.
Revised: Concrete Autoencoder (CAE) [5] learns a concrete selector layer (encoder) that selects stochastic linear combinations of input features during training.

Original: After training, it converges to the target number of features.
Revised: After training, it converges to the target number of features.

Original: Despite the high performance of thesemethods, the large number of iterations required to train dense models increases the computationalcosts significantly.
Revised: Despite the high performance of these methods, the large number of iterations required to train dense models increases the computational costs significantly.

Original: The conceptually closest work to ours is the recent work, Quick Selection (QS) [4].[Figure]
Revised: The conceptually closest work to ours is the recent work, Quick Selection (QS) [4].

Original: After convergence, neurons with the highest importance are selected as the most informative features.

Original: It trains a sparse autoencoder from scratch.
Revised: It trains a sparse autoencoder from scratch.

Original: During training, different sparse topologies are exploredusing the SET method [54].
Revised: During training, different sparse topologies are explored using the SET method [54].

Original: After each training epoch (i.e., pass on the full data), a fraction of theconnections with the least magnitude is dropped, and the same fractionis randomly regrown.
Revised: After each training epoch (i.e., pass on the full data), a fraction of the connections with the least magnitude is dropped, and the same fraction

Original: Fewworks based on multi-layer perceptron networks are proposed for the supervised setting [37; 61; 71].
Revised: [Figure] is randomly regrown. Few works based on multi-layer perceptron networks are proposed for the supervised setting [37; 61; 71].

Original: In section 4, we denote non Neural Network-based (NN-based) methods as classical methods.[Figure]
Revised: In section 4, we denote non Neural Network-based (NN-based) methods as classical methods.

Original: 2.2 Sparse Training
Revised: 2.2 Sparse Training

Original: [Figure] Sparse training with dynamic sparsity, also known as Dynamic Sparse Training (DST), is a recentresearch direction that aims at accelerating the training of neural networks without sacrificingperformance [27; 54; 47].
Revised: Sparse training with dynamic sparsity, also known as Dynamic Sparse Training (DST), is a recent research direction that aims at accelerating the training of neural networks without sacrificing performance [27; 54; 47].

Original: A neural network is initialized with a random sparse topology from scratch .
Revised: A neural network is initialized with a random sparse topology from scratch .

Original: The sparse topology (connectivity) and the weights are jointly optimized during training.
Revised: The sparse topology (connectivity) and the weights are jointly optimized during training.

Original: Duringtraining, the sparse topology is changed periodically through a drop-and-grow cycle, where a fractionof the parameters are dropped, and the same portion is regrown among different neurons.
Revised: During training, the sparse topology is changed periodically through a drop-and-grow cycle, where a fraction of the parameters are dropped, and the same portion is regrown among different neurons.

Original: An updateschedule determines the frequency of topology updates.
Revised: An update schedule determines the frequency of topology updates.

Original: Many works have been proposed, focusing onimproving the performance of sparse training for supervised image classification tasks by introducingdifferent criteria for connection regrowth [20; 34; 59; 49; 73; 50].
Revised: Many works have been proposed, focusing on improving the performance of sparse training for supervised image classification tasks by introducing different criteria for connection regrowth [20; 34; 59; 49; 73; 50].

Original: DST demonstrated its success inother fields as well, such as continual learning [62], feature selection [4], ensembling [46], federatedlearning [77; 8], adversarial training [57], and deep reinforcement learning [63].
Revised: DST demonstrated its success in other fields as well, such as continual learning [62], feature selection [4], ensembling [46], federated learning [77; 8], adversarial training [57], and deep reinforcement learning [63].

Original:  Where to pay Attention in Sparse Training (WAST)?
Revised:  Where to pay Attention in Sparse Training (WAST)?

Original: Problem Formulation.
Revised: Problem Formulation.

Original: Let D be a dataset with samples { x ( j ) } n 1 , where x ( j ) ∈ R m , and m and nare the number of features and samples, respectively.
Revised: Let D be a dataset with samples { x ( j ) } n 1 , where x ( j ) ∈ R m , and m and n are the number of features and samples, respectively.

Original: The goal is to select a subsetK ⊂ m of thefeatures, in an unsupervised manner, that is most representative of the whole feature space m .
Revised: The goal is to select a subset of the features of a size K in an unsupervised manner, which is most representative of the whole feature space m .

Original: WAST is a new sparse training method for neural network-based unsupervised feature selection.
Revised: WAST is a new sparse training method for neural network-based unsupervised feature selection.

Original: Thebasic idea of WAST is to pay attention to input features based on their estimated importance duringtraining to detect the most informative features quickly.
Revised: The basic idea of WAST is to pay attention to input features based on their estimated importance during training to detect the most informative features quickly.

Original: An overview of WAST is shown in Figure1.
Revised: An overview of WAST is shown in Figure 1.

Original: We use a sparse autoencoder with a single hidden layer of h neurons.
Revised: We use a sparse autoencoder with a single hidden layer of h neurons.

Original: Let f W be an autoencodermodel parameterized by sparse weights W = { W (1) , W (2) }.
Revised: Let f W be an autoencoder model parameterized by sparse weights W = { W (1) , W (2) }.

Original: The network is initialized with a certainsparsity level s = 1 − ∥ W (1) ∥ [-
-] m × h , where ∥ .
Revised: The network is initialized with a certain sparsity level s = 1 − ∥ W (1) ∥ [-
-]0 m × h , where ∥ .

Original: ∥  is the L 0 norm.
Revised: ∥ 0 is the L 0 norm.

Original: The sparsity level is kept fixed duringtraining.
Revised: The sparsity level is kept fixed during training.

Original: Initially, the sparse weights are uniformly distributed on the neurons.
Revised: Initially, the sparse weights are uniformly distributed on the neurons.

Original: During training,besides weights optimization, the sparse topology is optimized such that the sparse connections aregradually redistributed on the most informative features.
Revised: During training, besides weights optimization, the sparse topology is optimized such that the sparse connections are gradually redistributed on the most informative features.

Original: To achieve this goal, we propose new criteriato update the sparse topology and a new update schedule, as we will explain next.
Revised: To achieve this goal, we propose new criteria to update the sparse topology and a new update schedule, as we will explain next.

Original: Training.
Revised: Training.

Original: An autoencoder model is trained to minimize the reconstruction loss.
Revised: An autoencoder model is trained to minimize the reconstruction loss.

Original: We use mean squareerror to measure the loss between an input sample x and the reconstructed one ˜ x as follows:
Revised: We use mean squared error to measure the loss between an input sample x and the reconstructed one ˜ x as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˜ x
Revised: where ˜ x

Original: = f W ( x ) = σ ( xW (1) )
Revised: = f W ( x ) = σ ( xW (1) )

Original: W (2) , and σ is a non-linear activation function.
Revised: W (2) , and σ is a non-linear activation function.

Original: After each weightupdate using a batch of the data, we adapt the sparse topology.
Revised: After each weight update using a batch of the data, we adapt the sparse topology.

Original: The effect of the update schedule on 
Revised: The effect of the update schedule on performance is shown in

Original:  Algorithm1 WAST ←
Revised: Appendix E. The sparse topology is updated through a drop-and-grow cycle. Algorithm

Original: ∀ j ) ∈ 
Revised: ∀ j ) ∈ B

Original: B t 9:

Original: L ← 1 b (cid:80) bj =

Revised: L ← 1 b (cid:80) bj =

Original: (cid:13)(cid:13)(cid:13) ˜ x
Revised: (cid:13)(cid:13)(cid:13) ˜ x

Original: ( j ) − x ( j ) (cid:13)(cid:13)(cid:13) 210:
Revised: ( j ) − x ( j ) (cid:13)(cid:13)(cid:13) 210:

Original: W ← W − α ∇ W L 11:
Revised: W ← W − η ∇ W L 11:

Original: I O i ← I
Revised: I O i ← I

Original: O i + λ
Revised: O i + λ

Original: | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk
Revised: | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk

Original: =1 | W (2) ki | ∀ i ∈ {1 , ... m } 12:
Revised: =1 | W (2) ki | ∀ i ∈ { m } 12:

Original: I
Revised: I

Original: I i ←
Revised: I i ←

Original: I
Revised: I

Original: I i + λ | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk
Revised: I i + λ | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk

Original: =1 | W (1) ik |∀ i ∈ {1 , ..., m } 13:
Revised: =1 | W (1) ik | ∈ {

Original: W( l ) ← Drop( | W l ) | I O
Revised: W ← Drop

Original: I I α
Revised: W I α

Original: ∀ l ∈{
Revised: ∀ l ∈

Original: W( l ) ← GrowI O , I , α
Revised: W ← Grow

Original: ∀l ∈{
Revised: ∀ ∈

Original: 2 } 15: endtop performance is shown in
Revised: 15: end

Original: Appendix E. The sparse topology is updated through a drop-and-grow cycle.
Revised: 16: R top

Original: A fraction α of the connections with the least importance is dropped from each layer { W (1) , W (2) },and the same portion of new connections are regrown.
Revised: A fraction α of the connections with the least importance is dropped from each layer { W (1) , W (2) }, and the same portion of new connections are regrown.

Original: New connections are initialized to zero.
Revised: New connections are initialized to zero.

Original: The main contribution of this work lies in where the new connections are regrown.
Revised: The main contribution of this work lies in where the new connections are regrown.

Original: Different from theprevious method (QS) [4], where the connections are randomly regrown, we optimize the topology topay fast attention to informative features during training.
Revised: Different from the previous method (QS) [4], where the connections are randomly regrown, we optimize the topology to pay fast attention to informative features during training.

Original: To this end, we measure the importance ofeach neuron in the input and output layers by leveraging the information available during training.
Revised: To this end, we measure the importance of each neuron in the input and output layers by leveraging the information available during training.

Original: The importance of a neuron is estimated by its impact on the reconstructed loss and the magnitude ofits connected weights.
Revised: The importance of a neuron is estimated by its impact on the reconstructed loss and the magnitude of its connected weights.

Original: Specifically, we measure how sensitive the reconstruction loss L is to changes in the reconstructedoutput ˜ x .
Revised: Specifically, we measure how sensitive the reconstruction loss L is to changes in the reconstructed output ˜ x .

Original: The first-order approximation for the change in loss with respect to a small perturbationδ = { δ i } in the output ˜ x
Revised: The first-order approximation for the change in loss with respect to a small perturbation δ = { δ i } in the output ˜ x

Original: = { ˜ x i } can be written as a sum of its individual components as follows:
Revised: = { ˜ x i } can be written as a sum of its individual components as follows:

Original: [Equation]
Revised: [Equation]

Original: where ∂L∂ ˜ x
Revised: where ∂L∂ ˜ x

Original: i is the gradient of the loss with respect to the output neuron ˜ x i .
Revised: i is the gradient of the loss with respect to the output neuron ˜ x i .

Original: We define the importanceof a neuron i in the output layer at training iteration t as:
Revised: We define the importance of a neuron i in the output layer at training iteration t as:

Original: [Equation]
Revised: [Equation]

Original: where | W (2) ji | is the magnitude of the incoming connection from neuron j to neuron i and λ is ahyperparameter coefficient to balance the two components (See Appendix A).
Revised: where | W (2) ji | is the magnitude of the incoming connection from neuron j to neuron i and λ is a hyperparameter coefficient to balance the two components (See Appendix A).

Original: The same criterionis applied for the input neurons ( I ( t ) I i ), except that the magnitude of input weights ( | W (1) | ) is usedinstead of the output weights.
Revised: The same criterion is applied for the input neurons ( I ( t ) I i ), except that the magnitude of input weights ( | W (1) | ) is used instead of the output weights.

Original: This criterion considers two cases: the information from large gradients,especially at the beginning of the training, and the large weights resulting from the optimizationat later stages.
Revised: This criterion considers two cases: the information from large gradients, especially at the beginning of the training, and the large weights resulting from the optimization at later stages.

Original: The new connections in the input and output layers are regrown on the neuronswith the highest importance.
Revised: The new connections in the input and output layers are regrown on the neurons with the highest importance.

Original: For instance, the new connections in the input layer are given by 
Revised: For instance, the new connections in the input layer are given by top i ̸∈ ˜ W (1) ( I I ×

Original: [Equation] , r ) , where top ( Q, k ) gives the indices of the top- k elements in a matrix Q , I His the importance of the hidden neurons, ˜ W (1) is the set of remaining weights after the drop phase,and r is the number of regrown connections based on the fraction α .
Revised: I ⊤ H , r ) , where top ( Q, k ) gives the indices of the top- k elements in a matrix Q , I H is the importance of the hidden neurons, ˜ W (1) is the set of remaining weights after the drop phase, and r is the number of regrown connections based on the fraction α .

Original: We consider the neurons in thehidden layer to be equally important. 
Revised: We consider the neurons in the hidden layer to be equally important. [Table]

Original: During the drop phase, the importance of a connection w in the input (output) layer is estimated byits magnitude and the importance of its connected input (output) neuron ( I ( t ) i ) as follows:
Revised: During the drop phase, the importance of a connection w in the input (output) layer is estimated by its magnitude and the importance of its connected input (output) neuron ( I ( t ) i ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where I ( t ) i takes the value of I ( t )
Revised: where I ( t ) i takes the value of I ( t )

Original: I i and I ( t )
Revised: I i and I ( t )

Original: O i for the input and output layers, respectively.
Revised: O i for the input and output layers, respectively.

Original: The effectof each component in the neuron and connection importance criteria is studied in Section 5.3.
Revised: The effect of each component in the neuron and connection importance criteria is studied in Section 5.3.

Original: Theperiodic update of the sparse topology redistributes the connections on the effective features.
Revised: The periodic update of the sparse topology redistributes the connections on the effective features.

Original: Feature Selection.
Revised: Feature Selection.

Original: After training, we select the top- K neurons with the highest importance from theinput layer as the most informative K features.
Revised: After training, we select the top- K neurons with the highest importance from the input layer as the most informative K features.

Original: The details of WAST are provided in Algorithm 1.
Revised: The details of WAST are provided in Algorithm 1.

Original:  Experiments 
Revised:  Experiments 4.1 Baselines

Original: 4.1 Baselines

Original: We compared our method with several autoencoder-based (QS [4], CAE [5], AEFS [25]) and classicalmethods (lap_score [26], MCFS [9], DUFS [42]) for unsupervised feature selection.
Revised: We compared our method with several autoencoder-based (QS [4], CAE [5], AEFS [25]) and classical methods (lap_score [26], MCFS [9], DUFS [42]) for unsupervised feature selection.

Original: On top of that,we include comparisons with seven state-of-the-art supervised feature selection methods.
Revised: On top of that, we include comparisons with seven state-of-the-art supervised feature selection methods.

Original: Althoughthe latter is not the focus of this paper and not a typical comparison in the literature, we prefer to showthat WAST performs competitively also with the supervised methods while being computationallyefficient and applicable for cases where labels are not available or expensive to collect.
Revised: Although the latter is not the focus of this paper and not a typical comparison in the literature, we prefer to show that WAST performs competitively also with the supervised methods while being computationally efficient and applicable for cases where labels are not available or expensive to collect.

Original: 4.2 Datasets
Revised: 4.2 Datasets

Original: [Table] [Figure] [Table] We evaluate our method on 10 publicly available datasets, including image, speech, text, time series, biological, and artificial data.
Revised: We evaluate our method on 10 publicly available datasets, including image, speech, text, time series, biological, and artificial data.

Original: They havea variety of characteristics, such aslow and high dimensional features anda small and large number of trainingsamples.
Revised: They have a variety of characteristics, such as low and high-dimensional features and a small and large number of training samples.

Original: Details are in Table 1.
Revised: Details are in Table 1.

Original: 4.3 Experimental Settings
Revised: 4.3 Experimental Settings

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: An efficient feature selection method should have high learning accuracy withless memory and computational costs [10].
Revised: An efficient feature selection method should have high learning accuracy with less memory and computational costs [10].

Original: Hence, we evaluate multiple metrics.
Revised: Hence, we evaluate multiple metrics.

Original: (1) Classification
Revised: (1) Classification

Original: Accuracy : It is typically evaluated by a machine learning model [10].
Revised: Accuracy : It is typically evaluated by a machine learning model [10].

Original: We trained a classifier using the
Revised: We trained a classifier using the

Original: K selected features by the studied methods.
Revised: K selected features by the studied methods.

Original: Here, we use one of the most popular classifiers, supportvector machines (SVM) [14].
Revised: Here, we use one of the most popular classifiers, support vector machines (SVM) [14].

Original: Other classifiers could be used.
Revised: Other classifiers could be used.

Original: Yet, we choose to use a non-NN-basedclassifier for evaluation to avoid any biased advantages towards the NN-based baselines over theothers.
Revised: Yet, we choose to use a non-NN-based classifier for evaluation to avoid any biased advantages towards the NN-based baselines over the others.

Original: We studied the performance of 6 different values of  K .
Revised: We studied the performance of 6 different values of the number of selected features ( K ).

Original: We report the average accuracy of thetest data over 5 seeds.
Revised: We report the average accuracy of the test data over 5 seeds.

Original: (2) Memory cost: We calculate the number of network parameters (#params)used by each NN-based method.
Revised: (2) Memory cost: We calculate the number of network parameters (#params) used by each NN-based method.

Original: (3) Computational cost: We calculate the number of Floating-pointoperations (FLOPs) consumed to train a neural network.
Revised: (3) Computational cost: We calculate the number of Floating-point operations (FLOPs) consumed to train a neural network.

Original: See Appendix A.2 for details.
Revised: See Appendix A.2 for details.

Original: Implementation.
Revised: Implementation.

Original: We implemented WAST and QS [4] with PyTorch [58] (see Appendix I).
Revised: We implemented WAST and QS [4] with PyTorch [58] (see Appendix H).

Original: For CAE[5] and AEFS [25], we used the code provided by the authors of CAE with MIT license 2 . 
Revised: For CAE [5] and AEFS [25], we used the code provided by the authors of CAE with MIT license 2 . For DUFS

Original: For DUFS[42], LassoNet [37]  and STG [71], we used the official codes with MIT license 3,4,5 .
Revised: [Table] [42], LassoNet [37] [Table] and STG [71], we used the official codes with MIT license 345 .

Original: For classicalbaselines, we used the Scikit-Feature library with GNU General Public license [39] 6 .NN-based and [Table]
Revised: For classical baselines, we used the Scikit-Feature library with GNU General Public license [39] 6 .

Original:  classical methods are trained on Nvidia GPUs and CPUs, respectively.
Revised: NN-based and classical methods are trained on Nvidia GPUs and CPUs, respectively.

Original: We consider a 12 hours limiton the running time of each experiment.
Revised: We consider a 12 hours limit on the running time of each experiment.

Original: Experiments that exceed this limit are not considered.
Revised: Experiments that exceed this limit are not considered.

Original: Implementation Details.
Revised: Implementation Details.

Original: For all NN-based methods except CAE [5], we use a single hidden layerof 200 neurons.
Revised: For all NN-based methods except CAE [5], we use a single hidden layer of 200 neurons.

Original: The architecture of CAE consists of two layers.
Revised: The architecture of CAE consists of two layers.

Original: The size of the hidden layers isdependent on the chosen K ; [ K, 32 K ].
Revised: The size of the hidden layers is dependent on the chosen K ; [ K, 32 K ].

Original: For WAST and QS, we use a sparsity level of 0.8.
Revised: For WAST and QS, we use a sparsity level of 0.8.

Original: Following[4], we report the accuracy of NN-based baselines after 100 epochs unless stated otherwise.
Revised: Following [4], we report the accuracy of NN-based baselines after 100 epochs unless stated otherwise.

Original: Notethat some baselines reported a higher number of epochs (e.g., CAE [5] uses 200 epochs for somedatasets).
Revised: Note that some baselines reported a higher number of epochs (e.g., CAE [5] uses 200 epochs for some datasets).

Original: Yet, we keep 100 epochs for all cases for a fair comparison.
Revised: Yet, we keep 100 epochs for all cases for a fair comparison.

Original: For WAST, we train the modelfor 10 epochs.
Revised: For WAST, we train the model for 10 epochs.

Revised: Following [4], we add a Gaussian noise with a factor of 0.2 to the input in WAST and

Original:  Details of the hyperparameters are in Appendix A.1.
Revised: QS [4]. Details of the hyperparameters are in Appendix A.1.

Original: 4.4 Results
Revised: 4.4 Results

Original: Accuracy.
Revised: Accuracy.

Original: Table 2 shows the classification accuracy of the studied datasets.
Revised: Table 2 shows the classification accuracy of the studied datasets.

Original: Here, we report thechallenging case where a few best informative features have to be selected.
Revised: Here, we report the challenging case where a few best informative features have to be selected.

Original: We use a K of 50 for alldatasets except for Madelon, where a K of 20 is used as the remaining 480 features are pure noise.
Revised: We use a K of 50 for all datasets except for Madelon, where a K of 20 is used as the remaining 480 features are pure noise.

Original: Experiments with various values for K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } can be found in Appendix B,with a summary provided next.
Revised: Experiments with various values for K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } can be found in Appendix B, with a summary provided next.

Original: Unsupervised NN-based methods outperform the classical ones on all datasets except one, wherethe performance difference is marginal. 
Revised: Unsupervised NN-based methods outperform the classical ones on all datasets except one, where the performance difference is marginal. WAST outperforms unsupervised NN-based methods on [Table] [Figure]

Original: WAST outperforms unsupervised NN-based methods onall datasets, except MNIST and Fashion MNIST, where CAE is the best performer.
Revised: values of K . [Figure] [Figure] all datasets except MNIST and Fashion MNIST, where CAE is the best performer.

Original: It is worthemphasizing that the superior performance of WAST is achieved with a 90% reduction in the numberof training iterations.
Revised: It is worth emphasizing that the superior performance of WAST is achieved with a 90% reduction in the number of training iterations.

Original: Supervised baselines outperform the classical supervised ones in 6 cases.
Revised: Supervised baselines outperform the classical supervised ones in 6 cases.

Original: Interestingly, WAST achieves a competitive performance to the supervised baselines.
Revised: Interestingly, WAST achieves a competitive performance to the supervised baselines.

Original: It outperformsthe best-supervised performer in 5 cases.
Revised: It outperforms the best-supervised performer in 5 cases.

Original: Memory and Computational Costs.
Revised: Memory and Computational Costs.

Original: Table 3 shows the memory and computational costs consumedby each unsupervised NN-based method.
Revised: Table 3 shows the memory and computational costs consumed by each unsupervised NN-based method.

Original: The high memory and computational costs are causedby the high number of training samples as in MNIST or the high-dimensional feature space as in[Figure] [Figure] AEFS
Revised: The high memory and computational costs are caused by the high number of training samples, as in MNIST, or the high-dimensional feature space, as in

Original: [Figure]

Original: [Figure]

Original: GLA.
Revised: GLA.

Original: Training a sparse model from scratch as in WAST and QS reduces these costs substantially.
Revised: Training a sparse model from scratch, as in WAST and QS, reduces these costs substantially.

Original: QS reduces the computational cost by 80%.
Revised: QS reduces the computational cost by 80%.

Original: Interestingly, besides the reduction resulting fromusing sparse neural networks, WAST reduces the computational cost further to 98% due to the fastidentification of the informative features.
Revised: Interestingly, besides the reduction resulting from using sparse neural networks, WAST reduces the computational cost further to 98% due to the fast identification of the informative features.

Original: Both WAST and QS reduce the network size by 80%.
Revised: Both WAST and QS reduce the network size by 80%.

Original: The network size of CAE is dependent on the target value of K .
Revised: The network size of CAE is dependent on the target value of K .

Original: This has two limitations: trainingthe model for every target K and the increase of the memory and computational costs with highervalues for K .
Revised: This has two limitations: training the model for every target K and the increase of the memory and computational costs with higher values for K .

Original: On the other hand, WAST derives the importance of each feature with a single trainingrun independently of the target K .
Revised: On the other hand, WAST derives the importance of each feature with a single training run independently of the target K .

Original: This is illustrated in Figure 2.
Revised: This is illustrated in Figure 2.

Original: The figure shows the accumulated
Revised: The figure shows the accumulated

Original: FLOPs for all datasets except Madelon at each value of the studied K .
Revised: FLOPs for all datasets except Madelon at each value of the studied K .

Original: Overall Performance.
Revised: Overall Performance.

Original: To give a holistic view of the performance of each method across alldimensions (accuracy, memory cost, and computational cost) on all datasets and all studied values of K , we calculate the performance in each case.
Revised: To give a holistic view of the performance of each method across all dimensions (accuracy, memory cost, and computational cost) on all datasets and all studied values of K , we calculate the performance in each case.

Original: Figure 3 illustrates the normalized performance usingmin-max scaling for all cases.
Revised: Figure 3 illustrates the normalized performance using min-max scaling for all cases.

Original: The “score” metric represents the number of times a method is thebest performer in terms of accuracy.
Revised: The “score” metric represents the number of times a method is the best performer in terms of accuracy.

Original: Details are provided in Appendix A.2.
Revised: Details are provided in Appendix A.2.

Original: CAE has a higher scorethan AEFS, while the former is the best only for image datasets with a large number of samples (i.e., MNIST and Fashion MNIST) as shown in Figure 4.
Revised: CAE has a higher score than AEFS, while the former is the best only for image datasets with a large number of samples (i.e., MNIST and Fashion MNIST), as shown in Figure 4.

Original: AEFS has the highest memory and computationalcosts.
Revised: AEFS has the highest memory and computational costs.

Original: QS improves the memory and computational costs but has a lower score.
Revised: QS improves the memory and computational costs but has a lower score.

Original: WAST has thehighest score and spans more dataset types with different characteristics (e.g., high dimensionaldata) than the baselines (Figure 4).
Revised: WAST has the highest score and spans more dataset types with different characteristics (e.g., high dimensional data) than the baselines (Figure 4).

Original: The performance gain is accompanied by improving memory andcomputational efficiency significantly.
Revised: The performance gain is accompanied by a significant improvement in memory and computational efficiency.

Original: Detailed results are provided in Appendix B.
Revised: Detailed results are provided in Appendix B.

Original: Analysis
Revised: [Figure]

Original:   5.1 Effect of Fast Attention During Training
Revised:  Analysis 5.1 Effect of Fast Attention During Training

Original: To analyze the impact of attention to the informative features in WAST during training, we studythe learning behavior of all unsupervised NN-based methods on the test data for the first 10 epochs.
Revised: To analyze the impact of attention to the informative features in WAST during training, we study the learning behavior of all unsupervised NN-based methods on the test data for the first 10 epochs.

Original: Since CAE has a parameter that is dependent on the max number of epochs, we performed tenseparate runs and varied the max number of epochs each time.
Revised: Since CAE has a parameter that is dependent on the max number of epochs, we performed ten separate runs and varied the max number of epochs each time.

Original: Figure 5 shows the accuracy using
Revised: Figure 5 shows the accuracy using

Original: K of 50 for all datasets except Madelon, where K of 20 is used.
Revised: K of 50 for all datasets except Madelon, where K of 20 is used.

Original: The analysis reveals the followingfindings: (1) Robustness to noisy environments: The results on Madelon show the robustness of
Revised: The analysis reveals the following findings: (1) Robustness to noisy environments: The results on Madelon illustrate the robustness of

Original: WAST against the noisy features (480 out of 500).
Revised: WAST against the noisy features (480 out of 500).

Original: The figure shows the performance gap between
Revised: The figure shows the performance gap between

Original: WAST and the baseline methods starting from epoch 1.
Revised: WAST and the baseline methods starting from epoch 1.

Original: After 10 epochs, WAST outperforms the [Figure]
Revised: After 10 epochs, WAST outperforms the second-best performer by 22%.

Original: [-[Figure] Initialization 2436480 8 
 Epoch3042 0 8 
 Epoch2040 (c) QS [4].-]
Revised: [-[Figure] Initialization 2436480 8 
 Epoch3042 0 8 
 Epoch2040 (c) QS [4].-]More experiments on noisy enviroments can be found in Appendix G.

Original: second-best performer by 22%. (2) Performance on datasets with a large amount of samples: WAST outperforms CAE by 3.6% and 1.6% on MNIST and FashionMNIST, respectively, although CAE was the best performer on these datasets after training it for a larger number of epochs (Table2).
Revised: (2) Performance on datasets with a large number of samples: WAST outperforms CAE by 3.6% and 1.6% on MNIST and FashionMNIST, respectively, although CAE was the best performer on these datasets after training it for a larger number of epochs (Table 2).

Original: (3) Consistency across different dataset types and characteristics:
Revised: (3) Consistency across different dataset types and characteristics:

Original: We observe that somebaselines achieve high performance on some domains while being worse in others.
Revised: We observe that some baselines achieve high performance in some domains while being worse in others.

Original: WAST has thehighest consistency on different dataset types and characteristics, outperforming all baselines after 10epochs (Appendix C).
Revised: WAST has the highest consistency on different dataset types and characteristics, outperforming all baselines after 10 epochs (Appendix C).

Original: (4) Stability: WAST is more stable at the early stage of the training.
Revised: (4) Stability: WAST is more stable at the early stage of the training.

Original: This isrepresented by the standard deviation across different training runs (shaded region).
Revised: This is represented by the standard deviation across different training runs (shaded region).

Original: 5.2 Visualization
Revised: 5.2 Visualization

Original: Figure 6 illustrates how the sparse topology changes during training in WAST and QS.
Revised: Figure 6 illustrates how the sparse topology changes during training in WAST and QS.

Original: We performedthis analysis on MNIST, where handwritten digits are centered in 28 × 28 grayscale images (Figure6a).
Revised: We performed this analysis on MNIST, where handwritten digits are centered in 28 × 28 grayscale images (Figure 6a).

Original: Initially, the sparse connections are uniformly distributed on the input neurons.
Revised: Initially, the sparse connections are uniformly distributed on the input neurons.

Original: With the guidedadaptation of a sparse topology via attention, WAST detects the informative features quickly after oneepoch.
Revised: With the guided adaptation of a sparse topology via attention, WAST detects the informative features quickly after one [Figure] Initialization Epoch

Revised: Epoch      (b) WAST (ours).

Original: On the other hand, the random growth of the connections in QS requires more training epochs.
Revised: Initialization Epoch Epoch     ( [Figure]

Revised: [Table]

Original:  The effect of increasing the frequency of topological exploration in QS is studied in Appendix F.
Revised: [Table] epoch. On the other hand, the random growth of the connections in QS requires more training epochs. The effect of increasing the frequency of topological exploration in QS is studied in Appendix F.

Original: 5.3 Ablation Study
Revised: 5.3 Ablation Study

Original: We performed an ablation study to assess the contribution of each component in the proposed criteriaof neuron and connection importance in the performance of WAST.
Revised: We performed an ablation study to assess the contribution of each component in the proposed criteria of neuron and connection importance in the performance of WAST.

Original: Neuron Importance.
Revised: Neuron Importance.

Original: (#1) “w/o | ∂L∂ ˜ x | ”: Using only the magnitude of connected weights to determinethe importance of a neuron (i.e., λ = 0 ), (#2) “w/o | W |”: Using only the sensitivity of the neuron tothe change in the loss (i.e., λ = 1 ), and (#3) “w/o momentum”: The importance of a neuron is basedonly on the current estimate of its sensitivity to the loss and currently connected weights.[Table]
Revised: (#1) “w/o | ∂L ∂ ˜ x | ”: Using only the magnitude of connected weights to determine the importance of a neuron (i.e., λ = 0 ), (#2) “w/o | W |”: Using only the sensitivity of the neuron to the change in the loss (i.e., λ = 1 ), and (#3) “w/o momentum”: The importance of a neuron is based only on the current estimate of its sensitivity to the loss and currently connected weights.

Original: Connection Importance.
Revised: Connection Importance.

Original: (#4) “w/o I i ”:
Revised: (#4) “w/o I i ”:

Original: The importance of a connection is estimated by itsmagnitude without considering the importance of its connected neuron I i .
Revised: The importance of a connection is estimated by its magnitude without considering the importance of its connected neuron I i .

Original: We performed this analysis on datasets of different types.
Revised: We performed this analysis on datasets of different types.

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 4.

Original: Weobserve that removing the sensitivity of a neuron to the loss (#1) has the biggest effect on performance.
Revised: We observe that removing the sensitivity of a neuron to the loss (#1) has the biggest effect on performance.

Original: It has the highest impact on noisy environments, where it reduces the performance by 27.77% on
Revised: It has the highest impact on noisy environments, where it reduces the performance by 27.77% on

Original: Madelon.
Revised: Madelon.

Original: On the other datasets, it results in a 0 .
Revised: On the other datasets, it results in a 0 .

Original: 53 − 4 .
Revised: 53 − 4 .

Original: 93% reduction in the performance.
Revised: 93% reduction in the performance.

Original: Excludingthe magnitude of the weights (#2) affects mainly the datasets with few training samples (e.g., SMK).
Revised: Excluding the magnitude of the weights (#2) affects mainly the datasets with few training samples (e.g., SMK).

Original: Neglecting the neuron importance in the previous training iterations (#3) reduces the performance by0 .
Revised: Neglecting the neuron importance in the previous training iterations (#3) reduces the performance by 0 .

Original: 09 − 5 .
Revised: 09 − 5 .

Original: Aligned with many prior works [20; 54; 27], we find that the weight magnitude isan effective metric for estimating the connection importance (#4).
Revised: Aligned with many prior works [20; 54; 27], we find that the weight magnitude is an effective metric for estimating the connection importance (#4).

Original: Moreover, our results show thatenhancing it with the importance of its connected neuron improves the performance even further onsome datasets without diminishing it on the others. 
Revised: Moreover, our results show that enhancing it with the importance of its connected neuron improves the performance even further on some datasets without diminishing it on the others. [Figure] [Figure] cc u r a c

Revised: y

Revised: [ % ] (c) HAR.

Original: Discussion
Revised: 5.4 Effect of the Sparsity Level

Revised: We further study the effect of the sparsity level of the auto-encoder model on the performance of WAST. We evaluated 5 different sparsity levels {20%, 40%, 60%, 80%, 90%}.

Revised: All other settings are the same as the ones stated in Appendix A.

Revised: Figure 7 illustrates the classification accuracy using K = 20 for Madelon and K=50 for other datasets.

Revised: We observe the performance of WAST is robust to the sparsity level.

Revised: Yet, we mostly care about the performance at high sparsity levels, as the goal is to achieve high performance at a low computational cost.

Revised:  Discussion

Original: Conclusion.
Revised: Conclusion.

Original: In this paper, we propose WAST, a new efficient neural network-based method forunsupervised feature selection.
Revised: In this paper, we propose WAST, a new efficient neural network-based method for unsupervised feature selection.

Original: We train a sparse autoencoder from scratch and optimize the sparsetopology during training to detect the informative features quickly.
Revised: We train a sparse autoencoder from scratch and optimize the sparse topology during training to detect the informative features quickly.

Original: We performed extensive experiments in which we evaluated 55 cases on various datasets and different values of the number ofselected features.
Revised: We performed extensive experiments in which we evaluated 55 cases on various datasets and different values of the number of selected features.

Original: WAST achieves the best performance on 19 cases, while the second-best unsupervised performer has a score of 11 cases.
Revised: WAST achieves the best performance on 19 cases, while the second-best unsupervised performer has a score of 11 cases.

Original: More interestingly, the superior performance is achievedwith a few training iterations.
Revised: More interestingly, the superior performance is achieved with a few training iterations.

Original: WAST reduces the memory and computational costs by 80% and98%, respectively.
Revised: WAST reduces the memory and computational costs by 80% and 98%, respectively.

Original: Moreover, we show the robustness of WAST towards very noisy environments,outperforming the state-of-the-art methods by 22% with limited training iterations.
Revised: Moreover, we show the robustness of WAST towards very noisy environments, outperforming the state-of-the-art methods by 22% with limited training iterations.

Original: Finally, we showthat WAST performs competitively with supervised-based methods, outperforming them on imagedatasets.
Revised: Finally, we show that WAST performs competitively with supervised-based methods, outperforming them on image datasets.

Original: This demonstrates the promise of adapting WAST in the supervised setting and motivatesnew sparse training algorithms for supervised and unsupervised feature selection.
Revised: This demonstrates the promise of adapting WAST in the supervised setting and motivates new sparse training algorithms for supervised and unsupervised feature selection.

Original: Limitations.
Revised: Limitations.

Original: This work is a step toward exploiting the power of neural networks for feature selectionin a computationally efficient manner.
Revised: This work is a step toward exploiting the power of neural networks for feature selection in a computationally efficient manner.

Original: Besides the improvement in the accuracy of selecting theinformative feature, fast attention to the important features during training reduces the number oftraining iterations substantially.
Revised: Besides the improvement in the accuracy of selecting the informative feature, fast attention to the important features during training reduces the number of training iterations substantially.

Original: Yet, like most sparse training methods in the literature, this proof-ofconcept does not fully utilize the memory and computational advantages of sparse neural networks.
Revised: Yet, like most sparse training methods in the literature, this proof-ofconcept does not fully utilize the memory and computational advantages of sparse neural networks.

Original: This is due to the lack of hardware support for sparsity and the higher focus of the community on thealgorithmic side [27].
Revised: This is due to the lack of hardware support for sparsity and the higher focus of the community on the algorithmic side [27].

Original: Nevertheless, there is recent growing attention for the hardware and softwaresupport for sparsity [28; 76; 66; 48; 15; 3; 13] (See Appendix I for discussion).
Revised: Nevertheless, there is recent growing attention to hardware and software support for sparsity [28; 76; 66; 48; 15; 3; 13] (See Appendix H for discussion).

Original: This enables  pure 
Revised: This enables the pure sparse implementation of the method in the future.

Original: Societal Impacts.
Revised: Societal Impacts.

Original: With the emergence of big data, systems that can quickly select the informativefeatures and removethe redundant ones become crucial.
Revised: With the emergence of big data, systems that can quickly select informative features and remove redundant ones become crucial.

Original: Besides the performance gain that could beachieved in the downstream tasks by removing irrelevant features, reducing the number of featuresimproves the memory and computational efficiency significantly.
Revised: Besides the performance gain that could be achieved in the downstream tasks by removing irrelevant features, reducing the number of features improves memory and computational efficiency significantly.

Original: This enables providing energyefficient systems.
Revised: This enables providing energy-efficient systems.

Original: Moreover, it is useful for improving the interpretability of model-driven decisions.
Revised: Moreover, it is useful for improving the interpretability of model-driven decisions.

Original: The unsupervised selection of the informative features is effective inthe cases where labeled data islimited or very expensive to collect.
Revised: The unsupervised selection of the informative features is effective in cases where labeled data is limited or very expensive to collect.

Original: We do not expect that there is a negative societal impact. 
Revised: We do not expect that there is a negative societal impact. A Experimental Settings

Revised: A.1 Implementation Details

Revised: For all datasets, we used standard normalization that scales the features to have zero mean and standard deviation of one.

Revised: The architecture of the autoencoder consists of one hidden layer with sigmoid activation.

Revised: A linear activation is used for the output layer.

Revised: We use a hidden layer ofneurons for all datasets.

Revised: We trained each dataset for 10 epochs using stochastic gradient descent with a momentum of 0.9 and a batch size of 128.

Revised: We used a learning rate of 0.1 for all datasets except for the datasets with very few samples.

Revised: We used a learning rate of 0.01 and 0.001 for SMK and GLA, respectively.

Revised: For sparse-based methods, WAST and QS, we used a sparsity level s of 0.8.

Revised: The factor of dropped and regrown connections

Revised: , α , in the two methods is 0.3.

Revised: We used λ of 0.

Revised: for all datasets of type image, and λ of 0.9 for all other types except the biological data.

Revised: For SMK and GLA, we used λ of 0.01 and 0.001, respectively.

Revised: The selection of the value of λ is guided by the magnitude of the gradient of the loss with respect to the reconstructed sample compared to the magnitude of the weights.

Revised: This is illustrated in Figure 8.

Revised: We use λ to control the balance between the two components of the neuron importance in Equation 3.

Revised: For instance, for biological data (SMK and

Revised: GLA), where the absolute gradient is higher than the average sum of absolute weights connected to a neuron, we use a small value for λ .

Revised: The other hyperparameters are selected using a random search.

Revised: For Madelon, we evaluate the performance of K = 20 .

Revised: For all other datasets, we test 6 different values K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } .

Revised: A.2 Evaluation Metrics

Revised: Accuracy.

Revised: To assess how informative are the selected features by each method, we train a support vector machine classifier on the selected features and report the classification accuracy.

Revised: Score.

Revised: To summarize the results for all datasets and various values of the selected features K , we calculate a score for each studied method.

Revised: The score of a method increases by one when it is the best performer in terms of classification accuracy for a certain dataset and a certain value of K .

Revised: Network size (#params).

Revised: To estimate the memory cost of a network, the network size is calculated by the summation of the number of connections allocated in its layers as follows:

Revised: [Equation]

Revised: AEFS

Revised: CAE

Revised: QS WAST (ours)

Revised: S c

Revised: o r e (a) Score ( ↑ ).

Revised: AEFS

Revised: CAE

Revised: QS WAST (ours)

Revised: # P a r a m s ( 1 e) (b) #Params ( ↓ ).

Revised: AEFS

Revised: CAE

Revised: QS WAST (ours)

Revised: F L O P s ( 1 e 12 ) (c) FLOPs ( ↓ ).

Revised: where W ( l ) is the weights in layer l , ∥ .

Revised: ∥ 0 is the standard L 0 norm, and L is the number of layers in the model.

Revised: For sparse neural networks, (cid:13)(cid:13) W ( l ) (cid:13)(cid:13) 0 is controlled by the defined sparsity level for the model.

Revised: Floating-point operations (FLOPs).

Revised: To estimate the computational cost of training a neural network model on a given dataset, we calculate how many FLOPs are performed in the whole course of training.

Revised: We follow the method described in [20] to calculate the FLOPs.

Revised: The FLOPs are estimated by the total number of multiplications and additions performed in the forward and backward passes.

Revised: It is calculated layer by layer in the model and are dependent on the sparsity level of the network.

Revised: FLOPs is the typical used metric in the literature to compare the computational cost of a sparse model against its dense counterpart [27].

Revised: The main motivation is that current sparse training methods are prototyped using masks over dense weights to simulate sparsity [27].

Revised: This is because most deep learning specialized hardware is optimized for dense matrix operations.

Revised: Therefore, the running time using these prototypes would not reflect the actual gain in memory and speed using a truly sparse network.

Revised: See Appendix H for further discussion.

Revised: Performance.

Revised: To assess the efficiency of a method across different dimensions (accuracy, memory cost, and computational cost) on various types of datasets with different characteristics, we include a comparison that gives a holistic view of all datasets and the studied values for K .

Revised: We studycases ((9 datasets × 6 values for K )+ Madelon dataset).

Revised: We calculate the total score for all cases and memory and computational costs.

Revised: Since the memory and computational costs for CAE [5] are dependent on the value of K , we report the average costs across the 6 studied values.

Revised: The memory and computational efficiency are equivalent to ( 1 − cost).

Revised: Normalized values, using min-max scaling, are illustrated in Figure 3.

Revised: B Additional Experiments

Revised: In this appendix, we evaluate the accuracy of the studied unsupervised and supervised methods for various values of the selected features K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } .

Revised: Tables 5 and 6 show the accuracy in each case.

Revised: Consistent with the observations for K of 50 in

Revised: Section 4.4, we find that unsupervised neural network-based methods outperform the classical ones in most cases.

Revised: CAE is the best performer for image datasets with a large number of samples (i.e.,

Revised: MNIST and Fashion MNIST) for all values of K except one.

Revised: WAST is the best performer in 19 cases, while the second-best performer is the best in 11 cases.

Revised: Similarly, the supervised neural network-based methods outperform the classical supervised methods in most cases.

Revised: Yet, for text and high dimensional datasets, classical methods are competitive with the

Revised: NN-based ones, outperforming them in 5 values for K on SMK and PCMAC.

Revised: Interestingly, WAST outperforms the best supervised performer on the image datasets in 14 cases of 24.

Revised: On the other types of datasets, supervised methods achieve higher performance in most cases.

Revised: Figure 9 summarizes the performance of unsupervised methods across various dimensions.

Revised: We report the score (i.e., how many times the method is the best performer) on different values of K (Figure 9a), the memory cost estimated by the number of required network parameters (Figure 9b), and the computational cost (Figure 9c).

Revised: Since the network architecture is dependent on K in CAE [5], we

Revised: [Table]

Revised: report the average memory and computational costs on different K .

Revised: Datasets with a large number of samples, MNIST and FashionMNIST, cause the highest costs.

Revised: AEFS requires the highest memory and computational costs.

Revised: Yet, AEFS performs better than the other dense baseline CAE on various dataset types.

Revised: QS reduces memory and computational costs substantially.

Revised: However, it has a lower score than AEFS.

Revised: WAST achieves the best trade-off across different dimensions.

Revised: It has the highest score and lowest computational cost.

Revised: C Effect of Fast Attention During Training

Revised: We studied the performance of unsupervised NN-based methods during the first 10 epochs in Section 5.1.

Revised: In this appendix, we report the accuracy after performing the 10 epochs

Revised: [Table]

Revised: As illustrated in Table 7, on different dataset types (artificial, image, speech, time series, text

Revised: [Table]

Revised: and biological) and

Revised: different characteristics (high/low dimensional features and few-shot/large data),

Revised: WAST consistently outperforms other NN-based unsupervised methods.

Revised: [Table]

Revised: Supervised

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: D Performance Evaluation on Music Dataset

Revised: In this appendix, we perform additional experiments on a recent music dataset, Free Music Archive (FMA) [16; 17].

Revised: FMA includes a collection of 106574 audio tracks with artist and album information categorized into 161 genres.

Revised: We use FMA-small, which consists of 6400 training samples and 800 testing samples with 140 pre-extracted features.

Revised: We focus on the genre classification task that categorizes the data into eight classes.

Revised: We use the same setting defined in Appendix A. For WAST, we use λ of 0.4.

Revised: We evaluate the classification accuracy of unsupervised NN-based methods using

Revised: K = { 25 , 50 , 75 } .

Revised: The results are reported in Table 8.

Revised: Consistent with our results on other benchmarks, WAST outperforms other unsupervised NN-based methods in two cases out of three, with a marginal performance difference in the third case.

Revised: The performance gain is accompanied by a reduction of the number of epochs by 90%.

Revised: E Effect of the Frequency of Topology Update

Revised: In this section, we analyze the effect of the schedule used to update the sparse topology.

Revised: In WAST, the topology is updated after each weight update using a batch of the data.

Revised: We named this schedule “batch update”.

Revised: We compare this schedule with the one used in the QS baseline, in which the sparse topology is updated after each training epoch (i.e., one pass on the whole data).

Revised: We denote this schedule as “epoch update”.

Revised: [Table]

Original: {+Initialization 2436480 8 
+}
Revised: {+Initialization 2436480 8 
+}

Original: {+Epoch3042 0 8 
+}
Revised: {+Epoch3042 0 8 
+}

Revised: Epoch2040 (a) QS with epoch update.

Original: {+Initialization 2436480 8 
+}
Revised: {+Initialization 2436480 8 
+}

Original: {+Epoch2440 0 8 
+}
Revised: {+Epoch2440 0 8 
+}

Revised: Epoch1545 (

Revised: [Figure]

Revised: update.

Original: {+Initialization 2436480 8 
+}
Revised: {+Initialization 2436480 8 
+}

Original: {+Epoch40120 0 8 
+}
Revised: {+Epoch40120 0 8 
+}

Original: {+Epoch
+}
Revised: {+Epoch
+}

Revised: [Figure]

Original: {+Initialization 2436480 8 
+}
Revised: {+Initialization 2436480 8 
+}

Original: {+Epoch0801600 8 
+}
Revised: {+Epoch0801600 8 
+}

Original: {+Epoch
+}
Revised: {+Epoch
+}

Revised: [Figure]

Revised: Updating the topology more frequently increases the speed of detecting the informative features.

Revised: Table 9 shows the accuracy on datasets from different types.

Revised: Updating the topology more frequently by the batch update schedule improves the performance in most of the cases and helps in finding the informative features quicker (Appendix F).

Revised: The accuracy increases by 0 .

Revised: 52 − 3 .

Revised: 5% in the studied cases.

Revised: F Visualizing the Effect of the Frequency of Topology Update

Revised: In this appendix, we visualize the effect of different update schedules of the sparse topology.

Revised: We analyzed two schedules: “epoch update” and “batch update” (Appendix E).

Revised: We performed this analysis on MNIST.

Revised: Figure 10 illustrates the effect of different topological update schedules in WAST and QS.

Revised: The less frequent update of the topology decreases the speed of identifying the informative features.

Revised: After 1 epoch, the distribution of connections by QS is still almost uniform across different input neurons (Figure 10a).

Revised: On the other hand, WAST starts to identify the important neurons (Figure 10c).

Revised: Updating the topology after each batch update increases the speed of identification in the random exploration setting (Figure 10b) and enables very fast detection in WAST after 1 epoch (Figure 10d).

Revised: G Robustness to Noisy Environments

Revised: Our experiments on the Madelon dataset illustrate the robustness of WAST to noisy features.

Revised: In this appendix, we study the robustness of the unsupervised NN-based methods on noisy samples.

Revised: To this end, we created noisy versions of the datasets by adding Gaussian noise with zero mean to the training data.

Revised: To assess the robustness of different noise levels, we evaluated 4 different values of standard deviation {0.2, 0.4, 0.6, 0.8}.

Revised: We performed this analysis on datasets of different types.

Revised: We use the same experimental setting described in Section 4.3 and Appendix A.

Revised: Figure 11 illustrates the classification accuracy using K = 50 .

Revised: We observe that across different dataset types, methods that are based on sparse models (i.e., WAST and QS) are more robust to noise.

Revised: As expected, the accuracy decreases when the level of noise increases in all methods.

Revised: Yet, sparse-based methods have the least decrease.

Revised: Moreover, WAST outperforms QS in all cases.

Revised: H Advance in Sparsity

Revised: Most of the efforts in previous years are devoted to the algorithmic side of sparse training.

Revised: The aim is to achieve the same or higher performance than the dense models with highly sparse counterparts, as discussed in Section 2.2.

Revised: Existing sparse training methods in the literature simulate sparsity using masks over dense weights since most deep learning specialized hardware is optimized for dense matrix operations.

Revised: Yet, recently, research on the software and hardware that support sparsity has received growing attention.

Revised: NVIDIA

Revised: [Figure]

Revised: cc u r

Revised: [Figure]

Revised: c y [ % ] (c) HAR.

Revised: [Figure]

Revised: released NVIDIA A100, which supports a 50% fixed sparsity level [76], and many other efforts on the hardware side are proposed [66; 3; 13; 43].

Revised: On the software side, libraries that support truly sparse implementations have been started for supervised learning [48; 15].

Revised: With a joint community effort in the algorithmic, software, and hardware directions, we would be able to actually provide faster, memory-efficient, and energy-efficient deep neural networks.

Revised: Further discussion can be found in [29; 53].