Original: Where to Pay Attention in Sparse Training for Feature Selection?
Revised: Where to Pay Attention in Sparse Training for Feature Selection?

Original: Abstract
Revised: Abstract

Original: A new line of research for feature selection based on neural networks has recently emerged.
Revised: A new line of research for feature selection based on neural networks has recently emerged.

Original: Despite its superiority to classical methods, it requires many training iterations to converge and detect informative features.
Revised: Despite its superiority to classical methods, it requires many training iterations to converge and detect informative features.

Original: The computational time becomes prohibitively long for datasets with a large number of samples or a very high dimensional feature space.
Revised: The computational time becomes prohibitively long for datasets with a large number of samples or a very high dimensional feature space.

Original: In this paper, we present a new efficient unsupervised method for feature selection based on sparse autoencoders.
Revised: In this paper, we present a new efficient unsupervised method for feature selection based on sparse autoencoders.

Original: In particular, we propose a new sparse training algorithm that optimizes a model’s sparse topology during training to pay attention to informative features quickly.
Revised: In particular, we propose a new sparse training algorithm that optimizes a model’s sparse topology during training to pay attention to informative features quickly.

Original: The attention-based adaptation of the sparse topology enables fast detection of informative features after a few training iterations.
Revised: The attention-based adaptation of the sparse topology enables fast detection of informative features after a few training iterations.

Original: We performed extensive experiments on 10 datasets of different types, including image, speech, text, artificial, and biological.
Revised: We performed extensive experiments on 10 datasets of different types, including image, speech, text, artificial, and biological.

Original: They cover a wide range of characteristics, such as low and high-dimensional feature spaces, and few and large training samples.
Revised: They cover a wide range of characteristics, such as low and high-dimensional feature spaces, and few and large training samples.

Original: Our proposed approach outperforms the state-of-the-art methods in terms of selecting informative features while reducing training iterations and computational costs substantially.
Revised: Our proposed approach outperforms the state-of-the-art methods in terms of selecting informative features while reducing training iterations and computational costs substantially.

Original: Moreover, the experiments show the robustness of our method in extremely noisy environments 1 .
Revised: Moreover, the experiments show the robustness of our method in extremely noisy environments 1 .

Original:  Introduction
Revised:  Introduction

Original: Feature selection plays a crucial role in data mining and machine learning tasks with the explosion in the size and dimensionality of real-world data [68; 10; 30; 44; 23].
Revised: Feature selection plays a crucial role in data mining and machine learning tasks with the explosion in the size and dimensionality of real-world data [68; 10; 30; 44; 23].

Original: It aims to efficiently select a subset of the features that are most informative and remove the irrelevant or redundant ones.
Revised: It aims to efficiently select a subset of the features that are most informative and remove the irrelevant or redundant ones.

Original: It is useful for alleviating the curse of dimensionality, interpretability of model-driven decisions, generalization of downstream tasks, reducing the prohibitive memory and computational costs and avoiding the expensive costs of collecting the full set of features as in biological studies [11; 38; 31; 52].
Revised: It is useful for alleviating the curse of dimensionality, interpretability of model-driven decisions, generalization of downstream tasks, reducing the prohibitive memory and computational costs and avoiding the expensive costs of collecting the full set of features as in biological studies [11; 38; 31; 52].

Original: Many methods have been proposed for feature selection in supervised, semi-supervised, and unsupervised settings [75; 10; 60; 51; 19; 1].
Revised: Many methods have been proposed for feature selection in supervised, semi-supervised, and unsupervised settings [75; 10; 60; 51; 19; 1].

Original: Recently, using neural networks for feature selection has received increasing attention due to its power to learn non-linear dependencies among input features [39].
Revised: Recently, using neural networks for feature selection has received increasing attention due to its power to learn non-linear dependencies among input features [39].

Original: One limitation of current neural network-based methods is that they are computationally expensive.
Revised: One limitation of current neural network-based methods is that they are computationally expensive.

Original: The computational costs result from training dense models for many training iterations until informative features are recognized.
Revised: The computational costs result from training dense models for many training iterations until informative features are recognized.

Original: The costs increase for datasets with a very large number of training samples or high dimensional feature space.
Revised: The costs increase for datasets with a very large number of training samples or high dimensional feature space.

Original: This limitation is recently addressed in [4] by utilizing sparse networks with dynamic sparsity for feature selection.
Revised: This limitation is recently addressed in [4] by utilizing sparse networks with dynamic sparsity for feature selection.

Original: An autoencoder is trained with the sparse training algorithm SET [54], which explores different sparse topologies during training by dropping a portion of the connections and growing the same portion randomly .
Revised: An autoencoder is trained with the sparse training algorithm SET [54], which explores different sparse topologies during training by dropping a portion of the connections and growing the same portion randomly .

Original: The importance of an input feature is estimated after convergence based on its outgoing sparse connections.
Revised: The importance of an input feature is estimated after convergence based on its outgoing sparse connections.

Original: Training sparse networks from scratch decreases memory and computational costs compared to previous dense-based methods.
Revised: Training sparse networks from scratch decreases memory and computational costs compared to previous dense-based methods.

Original: Yet, the random exploration of sparse topologies requires many training iterations to identify informative features.
Revised: Yet, the random exploration of sparse topologies requires many training iterations to identify informative features.

Original: In this paper, we propose an efficient unsupervised method for feature selection.
Revised: In this paper, we propose an efficient unsupervised method for feature selection.

Original: We introduce a new sparse training algorithm for autoencoders that quickly detects the informative input features.
Revised: We introduce a new sparse training algorithm for autoencoders that quickly detects the informative input features.

Original: Specifically, we optimize the sparse topology to learn W here to pay A ttention during S parse T raining.
Revised: Specifically, we optimize the sparse topology to learn W here to pay A ttention during S parse T raining.

Original: We named our method WAST .
Revised: We named our method WAST .

Original: WAST exploits the information from the reconstruction loss and the weights of sparse connections to guide the topological search.
Revised: WAST exploits the information from the reconstruction loss and the weights of sparse connections to guide the topological search.

Original: It redistributes the sparse connections to spot informative features quickly.
Revised: It redistributes the sparse connections to spot informative features quickly.

Original: We evaluate our proposed approach on ten datasets from various domains, including image, speech, text, artificial, and biological.
Revised: We evaluate our proposed approach on ten datasets from various domains, including image, speech, text, artificial, and biological.

Original: Through extensive experiments, we find that WAST identifies the informative features after a few training iterations and outperforms state-of-the-art unsupervised feature selection methods.
Revised: Through extensive experiments, we find that WAST identifies the informative features after a few training iterations and outperforms state-of-the-art unsupervised feature selection methods.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We propose a new efficient unsupervised method for feature selection, named WAST.
Revised: • We propose a new efficient unsupervised method for feature selection, named WAST.

Original: WAST optimizes the sparse topology of an autoencoder to detect the informative features quickly.
Revised: WAST optimizes the sparse topology of an autoencoder to detect the informative features quickly.

Original: • We perform extensive experiments on 10 benchmarks that cover various types and characteristics.
Revised: • We perform extensive experiments on 10 benchmarks that cover various types and characteristics.

Original: Experimental results show the effectiveness of our method over the state-of-the-art methods in terms of selecting the informative features.
Revised: Experimental results show the effectiveness of our method over the state-of-the-art methods in terms of selecting the informative features.

Original: • WAST reduces the computational time compared to neural network-based methods substantially by reducing training iterations and employing highly sparse neural networks.
Revised: • WAST reduces the computational time compared to neural network-based methods substantially by reducing training iterations and employing highly sparse neural networks.

Original: • We illustrate the robustness of our method in extremely noisy environments and its effectiveness for datasets with very high dimensional feature space and a few training samples.
Revised: • We illustrate the robustness of our method in extremely noisy environments and its effectiveness for datasets with very high dimensional feature space and a few training samples.

Original:  Related Work 2.1 Feature Selection
Revised:  Related Work 2.1 Feature Selection

Original: Many feature selection methods were introduced for the supervised, semi-supervised, and unsupervised settings depending on whether the data is labeled, partially labeled, or unlabeled, respectively [75; 10; 60; 1].
Revised: Many feature selection methods were introduced for the supervised, semi-supervised, and unsupervised settings depending on whether the data is labeled, partially labeled, or unlabeled, respectively [75; 10; 60; 1].

Original: Typically, feature selection methods are divided into three categories: filter, wrapper, and embedded methods.
Revised: Typically, feature selection methods are divided into three categories: filter, wrapper, and embedded methods.

Original: Filter methods are independent of the learning algorithms.
Revised: Filter methods are independent of the learning algorithms.

Original: They use ranking techniques for providing scores to the features, such as Laplacian score [26].
Revised: They use ranking techniques for providing scores to the features, such as Laplacian score [26].

Original: Despite being fast, they do not consider the relationship between the features, which may result in the selection of redundant features [11].
Revised: Despite being fast, they do not consider the relationship between the features, which may result in the selection of redundant features [11].

Original: In the supervised setting, filter methods consider the relation between the feature and the class label, such as fisher_score [22], CIEF [41], and ICAP [33].
Revised: In the supervised setting, filter methods consider the relation between the feature and the class label, such as fisher_score [22], CIEF [41], and ICAP [33].

Original: Wrapper methods exploit the performance of a predictive model to evaluate the quality of a subset of the features [70; 12; 74].
Revised: Wrapper methods exploit the performance of a predictive model to evaluate the quality of a subset of the features [70; 12; 74].

Original: They are more effective than filter methods, yet they are computationally expensive [39].
Revised: They are more effective than filter methods, yet they are computationally expensive [39].

Original: Embedded methods incorporate feature selection into the learning phase of another algorithm.
Revised: Embedded methods incorporate feature selection into the learning phase of another algorithm.

Original: Multi-Cluster
Revised: Multi-Cluster

Original: Feature Selection (MCFS)
Revised: Feature Selection (MCFS)

Original: [9] uses regularization to select the best features that keep the multi-cluster structure of the data.
Revised: [9] uses regularization to select the best features that keep the multi-cluster structure of the data.

Original: Unsupervised Discriminative Feature Selection (UDFS) [72] uses l 2 , 1 regularization and discriminative analysis to select the most discriminative features.
Revised: Unsupervised Discriminative Feature Selection (UDFS) [72] uses l 2 , 1 regularization and discriminative analysis to select the most discriminative features.

Original: Similarly, in the supervised setting, RFS [56] and L1_L21 [45] exploit l 2 , 1 -norm to introduce feature sparsity.
Revised: Similarly, in the supervised setting, RFS [56] and L1_L21 [45] exploit l 2 , 1 -norm to introduce feature sparsity.

Original: Another direction under the embedded category has recently emerged.
Revised: Another direction under the embedded category has recently emerged.

Original: It uses neural networks to perform feature selection by learning the non-linear dependencies among input features [40].
Revised: It uses neural networks to perform feature selection by learning the non-linear dependencies among input features [40].

Original: The success of autoencoders as a tool for feature extraction encourages unsupervised methods to explore their power for feature selection [7; 6; 67; 25; 18].
Revised: The success of autoencoders as a tool for feature extraction encourages unsupervised methods to explore their power for feature selection [7; 6; 67; 25; 18].

Original: AutoEncoder Feature Selector (AEFS)
Revised: AutoEncoder Feature Selector (AEFS)

Original: [25] combines autoencoder regression and group lasso tasks.
Revised: [25] combines autoencoder regression and group lasso tasks.

Original: Concrete Autoencoder (CAE) [5] learns a concrete selector layer (encoder) that selects stochastic linear combinations of input features during training.
Revised: Concrete Autoencoder (CAE) [5] learns a concrete selector layer (encoder) that selects stochastic linear combinations of input features during training.

Original: After training, it converges to the target number of features.
Revised: After training, it converges to the target number of features.

Original: Despite the high performance of these methods, the large number of iterations required to train dense models increases the computational costs significantly.
Revised: Despite the high performance of these methods, the large number of iterations required to train dense models increases the computational costs significantly.

Original: The conceptually closest work to ours is the recent work, Quick
Revised: The conceptually closest work to ours is the recent work, Quick

Original: Selection (QS) [4].
Revised: Selection (QS) [4].

Original: It trains a sparse autoencoder from scratch.
Revised: It trains a sparse autoencoder from scratch.

Original: During training, different sparse topologies are explored using the SET method [54].
Revised: During training, different sparse topologies are explored using the SET method [54].

Original: After each training epoch (i.e., pass on the full data), a fraction of the connections with the least magnitude is dropped, and the same fraction
Revised: After each training epoch (i.e., pass on the full data), a fraction of the connections with the least magnitude is dropped, and the same fraction

Original: [Figure]
Revised: [Figure]

Original: is randomly regrown.
Revised: is randomly regrown.

Original: Few works based on multi-layer perceptron networks are proposed for the supervised setting [37; 61; 71].
Revised: Few works based on multi-layer perceptron networks are proposed for the supervised setting [37; 61; 71].

Original: In section 4, we denote non Neural Network-based (NN-based) methods as classical methods.
Revised: In section 4, we denote non Neural Network-based (NN-based) methods as classical methods.

Original: 2.2 Sparse Training
Revised: 2.2 Sparse Training

Original: Sparse training with dynamic sparsity, also known as Dynamic Sparse Training (DST), is a recent research direction that aims at accelerating the training of neural networks without sacrificing performance [27; 54; 47].
Revised: Sparse training with dynamic sparsity, also known as Dynamic Sparse Training (DST), is a recent research direction that aims at accelerating the training of neural networks without sacrificing performance [27; 54; 47].

Original: A neural network is initialized with a random sparse topology from scratch .
Revised: A neural network is initialized with a random sparse topology from scratch .

Original: The sparse topology (connectivity) and the weights are jointly optimized during training.
Revised: The sparse topology (connectivity) and the weights are jointly optimized during training.

Original: During training, the sparse topology is changed periodically through a drop-and-grow cycle, where a fraction of the parameters are dropped, and the same portion is regrown among different neurons.
Revised: During training, the sparse topology is changed periodically through a drop-and-grow cycle, where a fraction of the parameters are dropped, and the same portion is regrown among different neurons.

Original: An update schedule determines the frequency of topology updates.
Revised: An update schedule determines the frequency of topology updates.

Original: Many works have been proposed, focusing on improving the performance of sparse training for supervised image classification tasks by introducing different criteria for connection regrowth [20; 34; 59; 49; 73; 50].
Revised: Many works have been proposed, focusing on improving the performance of sparse training for supervised image classification tasks by introducing different criteria for connection regrowth [20; 34; 59; 49; 73; 50].

Original: DST demonstrated its success in other fields as well, such as continual learning [62], feature selection [4], ensembling [46], federated learning [77; 8], adversarial training [57], and deep reinforcement learning [63].
Revised: DST demonstrated its success in other fields as well, such as continual learning [62], feature selection [4], ensembling [46], federated learning [77; 8], adversarial training [57], and deep reinforcement learning [63].

Original:  Where to pay Attention in Sparse Training (WAST)?
Revised:  Where to pay Attention in Sparse Training (WAST)?

Original: Problem Formulation.
Revised: Problem Formulation.

Original: Let D be a dataset with samples { x ( j ) } n 1 , where x ( j ) ∈ R m , and m and n are the number of features and samples, respectively.
Revised: Let D be a dataset with samples { x ( j ) } n 1 , where x ( j ) ∈ R m , and m and n are the number of features and samples, respectively.

Original: The goal is to select a subset of the features of a size K in an unsupervised manner, which is most representative of the whole feature space m .
Revised: The goal is to select a subset of the features of a size K in an unsupervised manner, which is most representative of the whole feature space m .

Original: WAST is a new sparse training method for neural network-based unsupervised feature selection.
Revised: WAST is a new sparse training method for neural network-based unsupervised feature selection.

Original: The basic idea of WAST is to pay attention to input features based on their estimated importance during training to detect the most informative features quickly.
Revised: The basic idea of WAST is to pay attention to input features based on their estimated importance during training to detect the most informative features quickly.

Original: An overview of WAST is shown in Figure 1.
Revised: An overview of WAST is shown in Figure 1.

Original: We use a sparse autoencoder with a single hidden layer of h neurons.
Revised: We use a sparse autoencoder with a single hidden layer of h neurons.

Original: Let f W be an autoencoder model parameterized by sparse weights W = { W (1) , W (2) }.
Revised: Let f W be an autoencoder model parameterized by sparse weights W = { W (1) , W (2) }.

Original: The network is initialized with a certain sparsity level s = 1 − ∥ W (1) ∥ 0 m × h , where ∥ .
Revised: The network is initialized with a certain sparsity level s = 1 − ∥ W (1) ∥ 0 m × h , where ∥ .

Original: ∥ 0 is the L 0 norm.
Revised: ∥ 0 is the L 0 norm.

Original: The sparsity level is kept fixed during training.
Revised: The sparsity level is kept fixed during training.

Original: Initially, the sparse weights are uniformly distributed on the neurons.
Revised: Initially, the sparse weights are uniformly distributed on the neurons.

Original: During training, besides weights optimization, the sparse topology is optimized such that the sparse connections are gradually redistributed on the most informative features.
Revised: During training, besides weights optimization, the sparse topology is optimized such that the sparse connections are gradually redistributed on the most informative features.

Original: To achieve this goal, we propose new criteria to update the sparse topology and a new update schedule, as we will explain next.
Revised: To achieve this goal, we propose new criteria to update the sparse topology and a new update schedule, as we will explain next.

Original: Training.
Revised: Training.

Original: An autoencoder model is trained to minimize the reconstruction loss.
Revised: An autoencoder model is trained to minimize the reconstruction loss.

Original: We use mean squared error to measure the loss between an input sample x and the reconstructed one ˜ x as follows:
Revised: We use mean squared error to measure the loss between an input sample x and the reconstructed one ˜ x as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˜ x
Revised: where ˜ x

Original: = f W ( x ) = σ ( xW (1) )
Revised: = f W ( x ) = σ ( xW (1) )

Original: W (2) , and σ is a non-linear activation function.
Revised: W (2) , and σ is a non-linear activation function.

Original: After each weight update using a batch of the data, we adapt the sparse topology.
Revised: After each weight update using a batch of the data, we adapt the sparse topology.

Original: The effect of the update schedule on performance is shown in
Revised: The effect of the update schedule on performance is shown in

Original: Appendix E.
Revised: Appendix E.

Original: The sparse topology is updated through a drop-and-grow cycle.
Revised: The sparse topology is updated through a drop-and-grow cycle.

Original: Algorithm
Revised: Algorithm

Original: ∀ j ) ∈ B
Revised: ∀ j ) ∈ B

Original: L ← 1 b (cid:80) bj =

Revised: L ← 1 b (cid:80) bj =

Original: (cid:13)(cid:13)(cid:13) ˜ x
Revised: (cid:13)(cid:13)(cid:13) ˜ x

Original: ( j ) − x ( j ) (cid:13)(cid:13)(cid:13) 210:
Revised: ( j ) − x ( j ) (cid:13)(cid:13)(cid:13) 210:

Original: W ← W − η ∇ W L 11:
Revised: W ← W − η ∇ W L 11:

Original: I O i ← I
Revised: I O i ← I

Original: O i + λ
Revised: O i + λ

Original: | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk
Revised: | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk

Original: =1 | W (2) ki | ∀ i ∈ { m } 12:
Revised: =1 | W (2) ki | ∀ i ∈ { m } 12:

Original: I
Revised: I

Original: I i ←
Revised: I i ←

Original: I
Revised: I

Original: I i + λ | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk
Revised: I i + λ | ∂L∂ ˜ x i | + (1 − λ ) (cid:80) hk

Original: =1 | W (1) ik |
Revised: =1 | W (1) ik |

Original: ∈ {
Revised: ∈ {

Original: W
Revised: W

Original: ← Drop
Revised: ← Drop

Original: W
Revised: W

Original: I α
Revised: I α

Original: ∀ l ∈
Revised: ∀ l ∈

Original: W
Revised: W

Original: ← Grow
Revised: ← Grow

Original: ∀
Revised: ∀

Original: ∈
Revised: ∈

Original: 15: end
Revised: 15: end

Original: 16: R top
Revised: 16: R top

Original: A fraction α of the connections with the least importance is dropped from each layer { W (1) , W (2) }, and the same portion of new connections are regrown.
Revised: A fraction α of the connections with the least importance is dropped from each layer { W (1) , W (2) }, and the same portion of new connections are regrown.

Original: New connections are initialized to zero.
Revised: New connections are initialized to zero.

Original: The main contribution of this work lies in where the new connections are regrown.
Revised: The main contribution of this work lies in where the new connections are regrown.

Original: Different from the previous method (QS) [4], where the connections are randomly regrown, we optimize the topology to pay fast attention to informative features during training.
Revised: Different from the previous method (QS) [4], where the connections are randomly regrown, we optimize the topology to pay fast attention to informative features during training.

Original: To this end, we measure the importance of each neuron in the input and output layers by leveraging the information available during training.
Revised: To this end, we measure the importance of each neuron in the input and output layers by leveraging the information available during training.

Original: The importance of a neuron is estimated by its impact on the reconstructed loss and the magnitude of its connected weights.
Revised: The importance of a neuron is estimated by its impact on the reconstructed loss and the magnitude of its connected weights.

Original: Specifically, we measure how sensitive the reconstruction loss L is to changes in the reconstructed output ˜ x .
Revised: Specifically, we measure how sensitive the reconstruction loss L is to changes in the reconstructed output ˜ x .

Original: The first-order approximation for the change in loss with respect to a small perturbation δ = { δ i } in the output ˜ x
Revised: The first-order approximation for the change in loss with respect to a small perturbation δ = { δ i } in the output ˜ x

Original: = { ˜ x i } can be written as a sum of its individual components as follows:
Revised: = { ˜ x i } can be written as a sum of its individual components as follows:

Original: [Equation]
Revised: [Equation]

Original: where ∂L∂ ˜ x
Revised: where ∂L∂ ˜ x

Original: i is the gradient of the loss with respect to the output neuron ˜ x i .
Revised: i is the gradient of the loss with respect to the output neuron ˜ x i .

Original: We define the importance of a neuron i in the output layer at training iteration t as:
Revised: We define the importance of a neuron i in the output layer at training iteration t as:

Original: [Equation]
Revised: [Equation]

Original: where | W (2) ji | is the magnitude of the incoming connection from neuron j to neuron i and λ is a hyperparameter coefficient to balance the two components (See Appendix A).
Revised: where | W (2) ji | is the magnitude of the incoming connection from neuron j to neuron i and λ is a hyperparameter coefficient to balance the two components (See Appendix A).

Original: The same criterion is applied for the input neurons ( I ( t ) I i ), except that the magnitude of input weights ( | W (1) | ) is used instead of the output weights.
Revised: The same criterion is applied for the input neurons ( I ( t ) I i ), except that the magnitude of input weights ( | W (1) | ) is used instead of the output weights.

Original: This criterion considers two cases: the information from large gradients, especially at the beginning of the training, and the large weights resulting from the optimization at later stages.
Revised: This criterion considers two cases: the information from large gradients, especially at the beginning of the training, and the large weights resulting from the optimization at later stages.

Original: The new connections in the input and output layers are regrown on the neurons with the highest importance.
Revised: The new connections in the input and output layers are regrown on the neurons with the highest importance.

Original: For instance, the new connections in the input layer are given by top i ̸∈ ˜ W (1) ( I I ×
Revised: For instance, the new connections in the input layer are given by top i ̸∈ ˜ W (1) ( I I ×

Original: I ⊤ H , r ) , where top ( Q, k ) gives the indices of the top- k elements in a matrix Q , I H is the importance of the hidden neurons, ˜ W (1) is the set of remaining weights after the drop phase, and r is the number of regrown connections based on the fraction α .
Revised: I ⊤ H , r ) , where top ( Q, k ) gives the indices of the top- k elements in a matrix Q , I H is the importance of the hidden neurons, ˜ W (1) is the set of remaining weights after the drop phase, and r is the number of regrown connections based on the fraction α .

Original: We consider the neurons in the hidden layer to be equally important.
Revised: We consider the neurons in the hidden layer to be equally important.

Original: [Table]
Revised: [Table]

Original: During the drop phase, the importance of a connection w in the input (output) layer is estimated by its magnitude and the importance of its connected input (output) neuron ( I ( t ) i ) as follows:
Revised: During the drop phase, the importance of a connection w in the input (output) layer is estimated by its magnitude and the importance of its connected input (output) neuron ( I ( t ) i ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where I ( t ) i takes the value of I ( t )
Revised: where I ( t ) i takes the value of I ( t )

Original: I i and I ( t )
Revised: I i and I ( t )

Original: O i for the input and output layers, respectively.
Revised: O i for the input and output layers, respectively.

Original: The effect of each component in the neuron and connection importance criteria is studied in Section 5.3.
Revised: The effect of each component in the neuron and connection importance criteria is studied in Section 5.3.

Original: The periodic update of the sparse topology redistributes the connections on the effective features.
Revised: The periodic update of the sparse topology redistributes the connections on the effective features.

Original: Feature Selection.
Revised: Feature Selection.

Original: After training, we select the top- K neurons with the highest importance from the input layer as the most informative K features.
Revised: After training, we select the top- K neurons with the highest importance from the input layer as the most informative K features.

Original: The details of WAST are provided in Algorithm 1.
Revised: The details of WAST are provided in Algorithm 1.

Original:  Experiments 4.1 Baselines
Revised:  Experiments 4.1 Baselines

Original: We compared our method with several autoencoder-based (QS [4], CAE [5], AEFS [25]) and classical methods (lap_score [26], MCFS [9], DUFS [42]) for unsupervised feature selection.
Revised: We compared our method with several autoencoder-based (QS [4], CAE [5], AEFS [25]) and classical methods (lap_score [26], MCFS [9], DUFS [42]) for unsupervised feature selection.

Original: On top of that, we include comparisons with seven state-of-the-art supervised feature selection methods.
Revised: On top of that, we include comparisons with seven state-of-the-art supervised feature selection methods.

Original: Although the latter is not the focus of this paper and not a typical comparison in the literature, we prefer to show that WAST performs competitively also with the supervised methods while being computationally efficient and applicable for cases where labels are not available or expensive to collect.
Revised: Although the latter is not the focus of this paper and not a typical comparison in the literature, we prefer to show that WAST performs competitively also with the supervised methods while being computationally efficient and applicable for cases where labels are not available or expensive to collect.

Original: 4.2 Datasets
Revised: 4.2 Datasets

Original: We evaluate our method on 10 publicly available datasets, including image, speech, text, time series, biological, and artificial data.
Revised: We evaluate our method on 10 publicly available datasets, including image, speech, text, time series, biological, and artificial data.

Original: They have a variety of characteristics, such as low and high-dimensional features and a small and large number of training samples.
Revised: They have a variety of characteristics, such as low and high-dimensional features and a small and large number of training samples.

Original: Details are in Table 1.
Revised: Details are in Table 1.

Original: 4.3 Experimental Settings
Revised: 4.3 Experimental Settings

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: An efficient feature selection method should have high learning accuracy with less memory and computational costs [10].
Revised: An efficient feature selection method should have high learning accuracy with less memory and computational costs [10].

Original: Hence, we evaluate multiple metrics.
Revised: Hence, we evaluate multiple metrics.

Original: (1) Classification
Revised: (1) Classification

Original: Accuracy : It is typically evaluated by a machine learning model [10].
Revised: Accuracy : It is typically evaluated by a machine learning model [10].

Original: We trained a classifier using the
Revised: We trained a classifier using the

Original: K selected features by the studied methods.
Revised: K selected features by the studied methods.

Original: Here, we use one of the most popular classifiers, support vector machines (SVM) [14].
Revised: Here, we use one of the most popular classifiers, support vector machines (SVM) [14].

Original: Other classifiers could be used.
Revised: Other classifiers could be used.

Original: Yet, we choose to use a non-NN-based classifier for evaluation to avoid any biased advantages towards the NN-based baselines over the others.
Revised: Yet, we choose to use a non-NN-based classifier for evaluation to avoid any biased advantages towards the NN-based baselines over the others.

Original: We studied the performance of 6 different values of the number of selected features ( K ).
Revised: We studied the performance of 6 different values of the number of selected features ( K ).

Original: We report the average accuracy of the test data over 5 seeds.
Revised: We report the average accuracy of the test data over 5 seeds.

Original: (2) Memory cost: We calculate the number of network parameters (#params) used by each NN-based method.
Revised: (2) Memory cost: We calculate the number of network parameters (#params) used by each NN-based method.

Original: (3) Computational cost: We calculate the number of Floating-point operations (FLOPs) consumed to train a neural network.
Revised: (3) Computational cost: We calculate the number of Floating-point operations (FLOPs) consumed to train a neural network.

Original: See
Revised: See

Original: Appendix A.2 for details.
Revised: Appendix A.2 for details.

Original: Implementation.
Revised: Implementation.

Original: We implemented WAST and QS [4] with PyTorch [58] (see Appendix H).
Revised: We implemented WAST and QS [4] with PyTorch [58] (see Appendix H).

Original: For CAE [5] and AEFS [25], we used the code provided by the authors of CAE with MIT license 2 .
Revised: For CAE [5] and AEFS [25], we used the code provided by the authors of CAE with MIT license 2 .

Original: For DUFS
Revised: For DUFS

Original: [Table]
Revised: [Table]

Original: [42], LassoNet [37]
Revised: [42], LassoNet [37]

Original: [Table]
Revised: [Table]

Original: and STG [71], we used the official codes with MIT license 345 .
Revised: and STG [71], we used the official codes with MIT license 345 .

Original: For classical baselines, we used the Scikit-Feature library with GNU General Public license [39] 6 .
Revised: For classical baselines, we used the Scikit-Feature library with GNU General Public license [39] 6 .

Original: NN-based and classical methods are trained on Nvidia GPUs and CPUs, respectively.
Revised: NN-based and classical methods are trained on Nvidia GPUs and CPUs, respectively.

Original: We consider a 12 hours limit on the running time of each experiment.
Revised: We consider a 12 hours limit on the running time of each experiment.

Original: Experiments that exceed this limit are not considered.
Revised: Experiments that exceed this limit are not considered.

Original: Implementation Details.
Revised: Implementation Details.

Original: For all NN-based methods except CAE [5], we use a single hidden layer of 200 neurons.
Revised: For all NN-based methods except CAE [5], we use a single hidden layer of 200 neurons.

Original: The architecture of CAE consists of two layers.
Revised: The architecture of CAE consists of two layers.

Original: The size of the hidden layers is dependent on the chosen K ; [ K, 32 K ].
Revised: The size of the hidden layers is dependent on the chosen K ; [ K, 32 K ].

Original: For WAST and QS, we use a sparsity level of 0.8.
Revised: For WAST and QS, we use a sparsity level of 0.8.

Original: Following [4], we report the accuracy of NN-based baselines after 100 epochs unless stated otherwise.
Revised: Following [4], we report the accuracy of NN-based baselines after 100 epochs unless stated otherwise.

Original: Note that some baselines reported a higher number of epochs (e.g., CAE [5] uses 200 epochs for some datasets).
Revised: Note that some baselines reported a higher number of epochs (e.g., CAE [5] uses 200 epochs for some datasets).

Original: Yet, we keep 100 epochs for all cases for a fair comparison.
Revised: Yet, we keep 100 epochs for all cases for a fair comparison.

Original: For WAST, we train the model for 10 epochs.
Revised: For WAST, we train the model for 10 epochs.

Original: Following [4], we add a Gaussian noise with a factor of 0.2 to the input in WAST and
Revised: Following [4], we add a Gaussian noise with a factor of 0.2 to the input in WAST and

Original: QS [4].
Revised: QS [4].

Original: Details of the hyperparameters are in Appendix A.1.
Revised: Details of the hyperparameters are in Appendix A.1.

Original: 4.4 Results
Revised: 4.4 Results

Original: Accuracy.
Revised: Accuracy.

Original: Table 2 shows the classification accuracy of the studied datasets.
Revised: Table 2 shows the classification accuracy of the studied datasets.

Original: Here, we report the challenging case where a few best informative features have to be selected.
Revised: Here, we report the challenging case where a few best informative features have to be selected.

Original: We use a K of 50 for all datasets except for Madelon, where a K of 20 is used as the remaining 480 features are pure noise.
Revised: We use a K of 50 for all datasets except for Madelon, where a K of 20 is used as the remaining 480 features are pure noise.

Original: Experiments with various values for K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } can be found in Appendix B, with a summary provided next.
Revised: Experiments with various values for K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } can be found in Appendix B, with a summary provided next.

Original: Unsupervised NN-based methods outperform the classical ones on all datasets except one, where the performance difference is marginal.
Revised: Unsupervised NN-based methods outperform the classical ones on all datasets except one, where the performance difference is marginal.

Original: WAST outperforms unsupervised NN-based methods on
Revised: WAST outperforms unsupervised NN-based methods on

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: values of K .
Revised: values of K .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: all datasets except MNIST and Fashion MNIST, where CAE is the best performer.
Revised: all datasets except MNIST and Fashion MNIST, where CAE is the best performer.

Original: It is worth emphasizing that the superior performance of WAST is achieved with a 90% reduction in the number of training iterations.
Revised: It is worth emphasizing that the superior performance of WAST is achieved with a 90% reduction in the number of training iterations.

Original: Supervised baselines outperform the classical supervised ones in 6 cases.
Revised: Supervised baselines outperform the classical supervised ones in 6 cases.

Original: Interestingly, WAST achieves a competitive performance to the supervised baselines.
Revised: Interestingly, WAST achieves a competitive performance to the supervised baselines.

Original: It outperforms the best-supervised performer in 5 cases.
Revised: It outperforms the best-supervised performer in 5 cases.

Original: Memory and Computational Costs.
Revised: Memory and Computational Costs.

Original: Table 3 shows the memory and computational costs consumed by each unsupervised NN-based method.
Revised: Table 3 shows the memory and computational costs consumed by each unsupervised NN-based method.

Original: The high memory and computational costs are caused by the high number of training samples, as in MNIST, or the high-dimensional feature space, as in
Revised: The high memory and computational costs are caused by the high number of training samples, as in MNIST, or the high-dimensional feature space, as in

Original: GLA.
Revised: GLA.

Original: Training a sparse model from scratch, as in WAST and QS, reduces these costs substantially.
Revised: Training a sparse model from scratch, as in WAST and QS, reduces these costs substantially.

Original: QS reduces the computational cost by 80%.
Revised: QS reduces the computational cost by 80%.

Original: Interestingly, besides the reduction resulting from using sparse neural networks, WAST reduces the computational cost further to 98% due to the fast identification of the informative features.
Revised: Interestingly, besides the reduction resulting from using sparse neural networks, WAST reduces the computational cost further to 98% due to the fast identification of the informative features.

Original: Both WAST and QS reduce the network size by 80%.
Revised: Both WAST and QS reduce the network size by 80%.

Original: The network size of CAE is dependent on the target value of K .
Revised: The network size of CAE is dependent on the target value of K .

Original: This has two limitations: training the model for every target K and the increase of the memory and computational costs with higher values for K .
Revised: This has two limitations: training the model for every target K and the increase of the memory and computational costs with higher values for K .

Original: On the other hand, WAST derives the importance of each feature with a single training run independently of the target K .
Revised: On the other hand, WAST derives the importance of each feature with a single training run independently of the target K .

Original: This is illustrated in Figure 2.
Revised: This is illustrated in Figure 2.

Original: The figure shows the accumulated
Revised: The figure shows the accumulated

Original: FLOPs for all datasets except Madelon at each value of the studied K .
Revised: FLOPs for all datasets except Madelon at each value of the studied K .

Original: Overall Performance.
Revised: Overall Performance.

Original: To give a holistic view of the performance of each method across all dimensions (accuracy, memory cost, and computational cost) on all datasets and all studied values of K , we calculate the performance in each case.
Revised: To give a holistic view of the performance of each method across all dimensions (accuracy, memory cost, and computational cost) on all datasets and all studied values of K , we calculate the performance in each case.

Original: Figure 3 illustrates the normalized performance using min-max scaling for all cases.
Revised: Figure 3 illustrates the normalized performance using min-max scaling for all cases.

Original: The “score” metric represents the number of times a method is the best performer in terms of accuracy.
Revised: The “score” metric represents the number of times a method is the best performer in terms of accuracy.

Original: Details are provided in Appendix A.2.
Revised: Details are provided in Appendix A.2.

Original: CAE has a higher score than AEFS, while the former is the best only for image datasets with a large number of samples (i.e., MNIST and Fashion MNIST), as shown in Figure 4.
Revised: CAE has a higher score than AEFS, while the former is the best only for image datasets with a large number of samples (i.e., MNIST and Fashion MNIST), as shown in Figure 4.

Original: AEFS has the highest memory and computational costs.
Revised: AEFS has the highest memory and computational costs.

Original: QS improves the memory and computational costs but has a lower score.
Revised: QS improves the memory and computational costs but has a lower score.

Original: WAST has the highest score and spans more dataset types with different characteristics (e.g., high dimensional data) than the baselines (Figure 4).
Revised: WAST has the highest score and spans more dataset types with different characteristics (e.g., high dimensional data) than the baselines (Figure 4).

Original: The performance gain is accompanied by a significant improvement in memory and computational efficiency.
Revised: The performance gain is accompanied by a significant improvement in memory and computational efficiency.

Original: Detailed results are provided in Appendix B.
Revised: Detailed results are provided in Appendix B.

Original: [Figure]
Revised: [Figure]

Original:  Analysis 5.1 Effect of Fast Attention During Training
Revised:  Analysis 5.1 Effect of Fast Attention During Training

Original: To analyze the impact of attention to the informative features in WAST during training, we study the learning behavior of all unsupervised NN-based methods on the test data for the first 10 epochs.
Revised: To analyze the impact of attention to the informative features in WAST during training, we study the learning behavior of all unsupervised NN-based methods on the test data for the first 10 epochs.

Original: Since CAE has a parameter that is dependent on the max number of epochs, we performed ten separate runs and varied the max number of epochs each time.
Revised: Since CAE has a parameter that is dependent on the max number of epochs, we performed ten separate runs and varied the max number of epochs each time.

Original: Figure 5 shows the accuracy using
Revised: Figure 5 shows the accuracy using

Original: K of 50 for all datasets except Madelon, where K of 20 is used.
Revised: K of 50 for all datasets except Madelon, where K of 20 is used.

Original: The analysis reveals the following findings: (1) Robustness to noisy environments: The results on Madelon illustrate the robustness of
Revised: The analysis reveals the following findings: (1) Robustness to noisy environments: The results on Madelon illustrate the robustness of

Original: WAST against the noisy features (480 out of 500).
Revised: WAST against the noisy features (480 out of 500).

Original: The figure shows the performance gap between
Revised: The figure shows the performance gap between

Original: WAST and the baseline methods starting from epoch 1.
Revised: WAST and the baseline methods starting from epoch 1.

Original: After 10 epochs, WAST outperforms the second-best performer by 22%.
Revised: After 10 epochs, WAST outperforms the second-best performer by 22%.

Original: More experiments on noisy enviroments can be found in Appendix G.
Revised: More experiments on noisy enviroments can be found in Appendix G.

Original: (2) Performance on datasets with a large number of samples: WAST outperforms CAE by 3.6% and 1.6% on MNIST and FashionMNIST, respectively, although CAE was the best performer on these datasets after training it for a larger number of epochs (Table 2).
Revised: (2) Performance on datasets with a large number of samples: WAST outperforms CAE by 3.6% and 1.6% on MNIST and FashionMNIST, respectively, although CAE was the best performer on these datasets after training it for a larger number of epochs (Table 2).

Original: (3) Consistency across different dataset types and characteristics:
Revised: (3) Consistency across different dataset types and characteristics:

Original: We observe that some baselines achieve high performance in some domains while being worse in others.
Revised: We observe that some baselines achieve high performance in some domains while being worse in others.

Original: WAST has the highest consistency on different dataset types and characteristics, outperforming all baselines after 10 epochs (Appendix C).
Revised: WAST has the highest consistency on different dataset types and characteristics, outperforming all baselines after 10 epochs (Appendix C).

Original: (4) Stability:
Revised: (4) Stability:

Original: WAST is more stable at the early stage of the training.
Revised: WAST is more stable at the early stage of the training.

Original: This is represented by the standard deviation across different training runs (shaded region).
Revised: This is represented by the standard deviation across different training runs (shaded region).

Original: 5.2 Visualization
Revised: 5.2 Visualization

Original: Figure 6 illustrates how the sparse topology changes during training in WAST and QS.
Revised: Figure 6 illustrates how the sparse topology changes during training in WAST and QS.

Original: We performed this analysis on MNIST, where handwritten digits are centered in 28 × 28 grayscale images (Figure 6a).
Revised: We performed this analysis on MNIST, where handwritten digits are centered in 28 × 28 grayscale images (Figure 6a).

Original: Initially, the sparse connections are uniformly distributed on the input neurons.
Revised: Initially, the sparse connections are uniformly distributed on the input neurons.

Original: With the guided adaptation of a sparse topology via attention, WAST detects the informative features quickly after one
Revised: With the guided adaptation of a sparse topology via attention, WAST detects the informative features quickly after one

Original: [Figure]
Revised: [Figure]

Original: Initialization
Revised: Initialization

Original: Epoch
Revised: Epoch

Original: Epoch      (b) WAST (ours).
Revised: Epoch      (b) WAST (ours).

Original: Initialization
Revised: Initialization

Original: Epoch
Revised: Epoch

Original: Epoch     (
Revised: Epoch     (

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: epoch.
Revised: epoch.

Original: On the other hand, the random growth of the connections in QS requires more training epochs.
Revised: On the other hand, the random growth of the connections in QS requires more training epochs.

Original: The effect of increasing the frequency of topological exploration in QS is studied in Appendix F.
Revised: The effect of increasing the frequency of topological exploration in QS is studied in Appendix F.

Original: 5.3 Ablation Study
Revised: 5.3 Ablation Study

Original: We performed an ablation study to assess the contribution of each component in the proposed criteria of neuron and connection importance in the performance of WAST.
Revised: We performed an ablation study to assess the contribution of each component in the proposed criteria of neuron and connection importance in the performance of WAST.

Original: Neuron Importance.
Revised: Neuron Importance.

Original: (#1) “w/o | ∂L ∂ ˜ x | ”: Using only the magnitude of connected weights to determine the importance of a neuron (i.e., λ = 0 ), (#2) “w/o | W |”: Using only the sensitivity of the neuron to the change in the loss (i.e., λ = 1 ), and (#3) “w/o momentum”: The importance of a neuron is based only on the current estimate of its sensitivity to the loss and currently connected weights.
Revised: (#1) “w/o | ∂L ∂ ˜ x | ”: Using only the magnitude of connected weights to determine the importance of a neuron (i.e., λ = 0 ), (#2) “w/o | W |”: Using only the sensitivity of the neuron to the change in the loss (i.e., λ = 1 ), and (#3) “w/o momentum”: The importance of a neuron is based only on the current estimate of its sensitivity to the loss and currently connected weights.

Original: Connection Importance.
Revised: Connection Importance.

Original: (#4) “w/o I i ”:
Revised: (#4) “w/o I i ”:

Original: The importance of a connection is estimated by its magnitude without considering the importance of its connected neuron I i .
Revised: The importance of a connection is estimated by its magnitude without considering the importance of its connected neuron I i .

Original: We performed this analysis on datasets of different types.
Revised: We performed this analysis on datasets of different types.

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 4.

Original: We observe that removing the sensitivity of a neuron to the loss (#1) has the biggest effect on performance.
Revised: We observe that removing the sensitivity of a neuron to the loss (#1) has the biggest effect on performance.

Original: It has the highest impact on noisy environments, where it reduces the performance by 27.77% on
Revised: It has the highest impact on noisy environments, where it reduces the performance by 27.77% on

Original: Madelon.
Revised: Madelon.

Original: On the other datasets, it results in a 0 .
Revised: On the other datasets, it results in a 0 .

Original: 53 − 4 .
Revised: 53 − 4 .

Original: 93% reduction in the performance.
Revised: 93% reduction in the performance.

Original: Excluding the magnitude of the weights (#2) affects mainly the datasets with few training samples (e.g., SMK).
Revised: Excluding the magnitude of the weights (#2) affects mainly the datasets with few training samples (e.g., SMK).

Original: Neglecting the neuron importance in the previous training iterations (#3) reduces the performance by 0 .
Revised: Neglecting the neuron importance in the previous training iterations (#3) reduces the performance by 0 .

Original: 09 − 5 .
Revised: 09 − 5 .

Original: Aligned with many prior works [20; 54; 27], we find that the weight magnitude is an effective metric for estimating the connection importance (#4).
Revised: Aligned with many prior works [20; 54; 27], we find that the weight magnitude is an effective metric for estimating the connection importance (#4).

Original: Moreover, our results show that enhancing it with the importance of its connected neuron improves the performance even further on some datasets without diminishing it on the others.
Revised: Moreover, our results show that enhancing it with the importance of its connected neuron improves the performance even further on some datasets without diminishing it on the others.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: cc u r a c
Revised: cc u r a c

Original: y
Revised: y

Original: [ % ] (c) HAR.
Revised: [ % ] (c) HAR.

Original: 5.4 Effect of the Sparsity Level
Revised: 5.4 Effect of the Sparsity Level

Original: We further study the effect of the sparsity level of the auto-encoder model on the performance of
Revised: We further study the effect of the sparsity level of the auto-encoder model on the performance of

Original: WAST.
Revised: WAST.

Original: We evaluated 5 different sparsity levels {20%, 40%, 60%, 80%, 90%}.
Revised: We evaluated 5 different sparsity levels {20%, 40%, 60%, 80%, 90%}.

Original: All other settings are the same as the ones stated in Appendix A.
Revised: All other settings are the same as the ones stated in Appendix A.

Original: Figure 7 illustrates the classification accuracy using K = 20 for Madelon and K=50 for other datasets.
Revised: Figure 7 illustrates the classification accuracy using K = 20 for Madelon and K=50 for other datasets.

Original: We observe the performance of WAST is robust to the sparsity level.
Revised: We observe the performance of WAST is robust to the sparsity level.

Original: Yet, we mostly care about the performance at high sparsity levels, as the goal is to achieve high performance at a low computational cost.
Revised: Yet, we mostly care about the performance at high sparsity levels, as the goal is to achieve high performance at a low computational cost.

Original:  Discussion
Revised:  Discussion

Original: Conclusion.
Revised: Conclusion.

Original: In this paper, we propose WAST, a new efficient neural network-based method for unsupervised feature selection.
Revised: In this paper, we propose WAST, a new efficient neural network-based method for unsupervised feature selection.

Original: We train a sparse autoencoder from scratch and optimize the sparse topology during training to detect the informative features quickly.
Revised: We train a sparse autoencoder from scratch and optimize the sparse topology during training to detect the informative features quickly.

Original: We performed extensive experiments in which we evaluated 55 cases on various datasets and different values of the number of selected features.
Revised: We performed extensive experiments in which we evaluated 55 cases on various datasets and different values of the number of selected features.

Original: WAST achieves the best performance on 19 cases, while the second-best unsupervised performer has a score of 11 cases.
Revised: WAST achieves the best performance on 19 cases, while the second-best unsupervised performer has a score of 11 cases.

Original: More interestingly, the superior performance is achieved with a few training iterations.
Revised: More interestingly, the superior performance is achieved with a few training iterations.

Original: WAST reduces the memory and computational costs by 80% and 98%, respectively.
Revised: WAST reduces the memory and computational costs by 80% and 98%, respectively.

Original: Moreover, we show the robustness of WAST towards very noisy environments, outperforming the state-of-the-art methods by 22% with limited training iterations.
Revised: Moreover, we show the robustness of WAST towards very noisy environments, outperforming the state-of-the-art methods by 22% with limited training iterations.

Original: Finally, we show that WAST performs competitively with supervised-based methods, outperforming them on image datasets.
Revised: Finally, we show that WAST performs competitively with supervised-based methods, outperforming them on image datasets.

Original: This demonstrates the promise of adapting WAST in the supervised setting and motivates new sparse training algorithms for supervised and unsupervised feature selection.
Revised: This demonstrates the promise of adapting WAST in the supervised setting and motivates new sparse training algorithms for supervised and unsupervised feature selection.

Original: Limitations.
Revised: Limitations.

Original: This work is a step toward exploiting the power of neural networks for feature selection in a computationally efficient manner.
Revised: This work is a step toward exploiting the power of neural networks for feature selection in a computationally efficient manner.

Original: Besides the improvement in the accuracy of selecting the informative feature, fast attention to the important features during training reduces the number of training iterations substantially.
Revised: Besides the improvement in the accuracy of selecting the informative feature, fast attention to the important features during training reduces the number of training iterations substantially.

Original: Yet, like most sparse training methods in the literature, this proof-ofconcept does not fully utilize the memory and computational advantages of sparse neural networks.
Revised: Yet, like most sparse training methods in the literature, this proof-ofconcept does not fully utilize the memory and computational advantages of sparse neural networks.

Original: This is due to the lack of hardware support for sparsity and the higher focus of the community on the algorithmic side [27].
Revised: This is due to the lack of hardware support for sparsity and the higher focus of the community on the algorithmic side [27].

Original: Nevertheless, there is recent growing attention to hardware and software support for sparsity [28; 76; 66; 48; 15; 3; 13] (See Appendix H for discussion).
Revised: Nevertheless, there is recent growing attention to hardware and software support for sparsity [28; 76; 66; 48; 15; 3; 13] (See Appendix H for discussion).

Original: This enables the pure sparse implementation of the method in the future.
Revised: This enables the pure sparse implementation of the method in the future.

Original: Societal Impacts.
Revised: Societal Impacts.

Original: With the emergence of big data, systems that can quickly select informative features and remove redundant ones become crucial.
Revised: With the emergence of big data, systems that can quickly select informative features and remove redundant ones become crucial.

Original: Besides the performance gain that could be achieved in the downstream tasks by removing irrelevant features, reducing the number of features improves memory and computational efficiency significantly.
Revised: Besides the performance gain that could be achieved in the downstream tasks by removing irrelevant features, reducing the number of features improves memory and computational efficiency significantly.

Original: This enables providing energy-efficient systems.
Revised: This enables providing energy-efficient systems.

Original: Moreover, it is useful for improving the interpretability of model-driven decisions.
Revised: Moreover, it is useful for improving the interpretability of model-driven decisions.

Original: The unsupervised selection of the informative features is effective in cases where labeled data is limited or very expensive to collect.
Revised: The unsupervised selection of the informative features is effective in cases where labeled data is limited or very expensive to collect.

Original: We do not expect that there is a negative societal impact.A Experimental Settings
Revised: We do not expect that there is a negative societal impact.

Original: A.1 Implementation Details

Original: For all datasets, we used standard normalization that scales the features to have zero mean and standard deviation of one.

Original: The architecture of the autoencoder consists of one hidden layer with sigmoid activation.

Original: A linear activation is used for the output layer.

Original: We use a hidden layer ofneurons for all datasets.

Original: We trained each dataset for 10 epochs using stochastic gradient descent with a momentum of 0.9 and a batch size of 128.

Original: We used a learning rate of 0.1 for all datasets except for the datasets with very few samples.

Original: We used a learning rate of 0.01 and 0.001 for SMK and GLA, respectively.

Original: For sparse-based methods, WAST and QS, we used a sparsity level s of 0.8.

Original: The factor of dropped and regrown connections

Original: , α , in the two methods is 0.3.

Original: We used λ of 0.

Original: for all datasets of type image, and λ of 0.9 for all other types except the biological data.

Original: For SMK and GLA, we used λ of 0.01 and 0.001, respectively.

Original: The selection of the value of λ is guided by the magnitude of the gradient of the loss with respect to the reconstructed sample compared to the magnitude of the weights.

Original: This is illustrated in Figure 8.

Original: We use λ to control the balance between the two components of the neuron importance in Equation 3.

Original: For instance, for biological data (SMK and

Original: GLA), where the absolute gradient is higher than the average sum of absolute weights connected to a neuron, we use a small value for λ .

Original: The other hyperparameters are selected using a random search.

Original: For Madelon, we evaluate the performance of K = 20 .

Original: For all other datasets, we test 6 different values K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } .

Original: A.2 Evaluation Metrics

Original: Accuracy.

Original: To assess how informative are the selected features by each method, we train a support vector machine classifier on the selected features and report the classification accuracy.

Original: Score.

Original: To summarize the results for all datasets and various values of the selected features K , we calculate a score for each studied method.

Original: The score of a method increases by one when it is the best performer in terms of classification accuracy for a certain dataset and a certain value of K .

Original: Network size (#params).

Original: To estimate the memory cost of a network, the network size is calculated by the summation of the number of connections allocated in its layers as follows:

Original: [Equation]

Original: AEFS

Original: CAE

Original: QS WAST (ours)

Original: S c

Original: o r e (a) Score ( ↑ ).

Original: AEFS

Original: CAE

Original: QS WAST (ours)

Original: # P a r a m s ( 1 e) (b) #Params ( ↓ ).

Original: AEFS

Original: CAE

Original: QS WAST (ours)

Original: F L O P s ( 1 e 12 ) (c) FLOPs ( ↓ ).

Original: where W ( l ) is the weights in layer l , ∥ .

Original: ∥ 0 is the standard L 0 norm, and L is the number of layers in the model.

Original: For sparse neural networks, (cid:13)(cid:13) W ( l ) (cid:13)(cid:13) 0 is controlled by the defined sparsity level for the model.

Original: Floating-point operations (FLOPs).

Original: To estimate the computational cost of training a neural network model on a given dataset, we calculate how many FLOPs are performed in the whole course of training.

Original: We follow the method described in [20] to calculate the FLOPs.

Original: The FLOPs are estimated by the total number of multiplications and additions performed in the forward and backward passes.

Original: It is calculated layer by layer in the model and are dependent on the sparsity level of the network.

Original: FLOPs is the typical used metric in the literature to compare the computational cost of a sparse model against its dense counterpart [27].

Original: The main motivation is that current sparse training methods are prototyped using masks over dense weights to simulate sparsity [27].

Original: This is because most deep learning specialized hardware is optimized for dense matrix operations.

Original: Therefore, the running time using these prototypes would not reflect the actual gain in memory and speed using a truly sparse network.

Original: See Appendix H for further discussion.

Original: Performance.

Original: To assess the efficiency of a method across different dimensions (accuracy, memory cost, and computational cost) on various types of datasets with different characteristics, we include a comparison that gives a holistic view of all datasets and the studied values for K .

Original: We studycases ((9 datasets × 6 values for K )+ Madelon dataset).

Original: We calculate the total score for all cases and memory and computational costs.

Original: Since the memory and computational costs for CAE [5] are dependent on the value of K , we report the average costs across the 6 studied values.

Original: The memory and computational efficiency are equivalent to ( 1 − cost).

Original: Normalized values, using min-max scaling, are illustrated in Figure 3.

Original: B Additional Experiments

Original: In this appendix, we evaluate the accuracy of the studied unsupervised and supervised methods for various values of the selected features K ∈ { 25 , 50 , 75 , 100 , 150 , 200 } .

Original: Tables 5 and 6 show the accuracy in each case.

Original: Consistent with the observations for K of 50 in

Original: Section 4.4, we find that unsupervised neural network-based methods outperform the classical ones in most cases.

Original: CAE is the best performer for image datasets with a large number of samples (i.e.,

Original: MNIST and Fashion MNIST) for all values of K except one.

Original: WAST is the best performer in 19 cases, while the second-best performer is the best in 11 cases.

Original: Similarly, the supervised neural network-based methods outperform the classical supervised methods in most cases.

Original: Yet, for text and high dimensional datasets, classical methods are competitive with the

Original: NN-based ones, outperforming them in 5 values for K on SMK and PCMAC.

Original: Interestingly, WAST outperforms the best supervised performer on the image datasets in 14 cases of 24.

Original: On the other types of datasets, supervised methods achieve higher performance in most cases.

Original: Figure 9 summarizes the performance of unsupervised methods across various dimensions.

Original: We report the score (i.e., how many times the method is the best performer) on different values of K (Figure 9a), the memory cost estimated by the number of required network parameters (Figure 9b), and the computational cost (Figure 9c).

Original: Since the network architecture is dependent on K in CAE [5], we

Original: [Table]

Original: report the average memory and computational costs on different K .

Original: Datasets with a large number of samples, MNIST and FashionMNIST, cause the highest costs.

Original: AEFS requires the highest memory and computational costs.

Original: Yet, AEFS performs better than the other dense baseline CAE on various dataset types.

Original: QS reduces memory and computational costs substantially.

Original: However, it has a lower score than AEFS.

Original: WAST achieves the best trade-off across different dimensions.

Original: It has the highest score and lowest computational cost.

Original: C Effect of Fast Attention During Training

Original: We studied the performance of unsupervised NN-based methods during the first 10 epochs in Section 5.1.

Original: In this appendix, we report the accuracy after performing the 10 epochs

Original: [Table]

Original: As illustrated in Table 7, on different dataset types (artificial, image, speech, time series, text

Original: [Table]

Original: and biological) and

Original: different characteristics (high/low dimensional features and few-shot/large data),

Original: WAST consistently outperforms other NN-based unsupervised methods.

Original: [Table]

Original: Supervised

Original: [Table]

Original: [Table]

Original: [Table]

Original: D Performance Evaluation on Music Dataset

Original: In this appendix, we perform additional experiments on a recent music dataset, Free Music Archive (FMA) [16; 17].

Original: FMA includes a collection of 106574 audio tracks with artist and album information categorized into 161 genres.

Original: We use FMA-small, which consists of 6400 training samples and 800 testing samples with 140 pre-extracted features.

Original: We focus on the genre classification task that categorizes the data into eight classes.

Original: We use the same setting defined in Appendix A. For WAST, we use λ of 0.4.

Original: We evaluate the classification accuracy of unsupervised NN-based methods using

Original: K = { 25 , 50 , 75 } .

Original: The results are reported in Table 8.

Original: Consistent with our results on other benchmarks, WAST outperforms other unsupervised NN-based methods in two cases out of three, with a marginal performance difference in the third case.

Original: The performance gain is accompanied by a reduction of the number of epochs by 90%.

Original: E Effect of the Frequency of Topology Update

Original: In this section, we analyze the effect of the schedule used to update the sparse topology.

Original: In WAST, the topology is updated after each weight update using a batch of the data.

Original: We named this schedule “batch update”.

Original: We compare this schedule with the one used in the QS baseline, in which the sparse topology is updated after each training epoch (i.e., one pass on the whole data).

Original: We denote this schedule as “epoch update”.

Original: [Table]

Original: [-Initialization 2436480 8 
-]
Revised: [-Initialization 2436480 8 
-]

Original: [-Epoch3042 0 8 
-]
Revised: [-Epoch3042 0 8 
-]

Original: Epoch2040 (a) QS with epoch update.

Original: [-Initialization 2436480 8 
-]
Revised: [-Initialization 2436480 8 
-]

Original: [-Epoch2440 0 8 
-]
Revised: [-Epoch2440 0 8 
-]

Original: Epoch1545 (

Original: [Figure]

Original: update.

Original: [-Initialization 2436480 8 
-]
Revised: [-Initialization 2436480 8 
-]

Original: [-Epoch40120 0 8 
-]
Revised: [-Epoch40120 0 8 
-]

Original: [-Epoch
-]
Revised: [-Epoch
-]

Original: [Figure]

Original: [-Initialization 2436480 8 
-]
Revised: [-Initialization 2436480 8 
-]

Original: [-Epoch0801600 8 
-]
Revised: [-Epoch0801600 8 
-]

Original: [-Epoch
-]
Revised: [-Epoch
-]

Original: [Figure]

Original: Updating the topology more frequently increases the speed of detecting the informative features.

Original: Table 9 shows the accuracy on datasets from different types.

Original: Updating the topology more frequently by the batch update schedule improves the performance in most of the cases and helps in finding the informative features quicker (Appendix F).

Original: The accuracy increases by 0 .

Original: 52 − 3 .

Original: 5% in the studied cases.

Original: F Visualizing the Effect of the Frequency of Topology Update

Original: In this appendix, we visualize the effect of different update schedules of the sparse topology.

Original: We analyzed two schedules: “epoch update” and “batch update” (Appendix E).

Original: We performed this analysis on MNIST.

Original: Figure 10 illustrates the effect of different topological update schedules in WAST and QS.

Original: The less frequent update of the topology decreases the speed of identifying the informative features.

Original: After 1 epoch, the distribution of connections by QS is still almost uniform across different input neurons (Figure 10a).

Original: On the other hand, WAST starts to identify the important neurons (Figure 10c).

Original: Updating the topology after each batch update increases the speed of identification in the random exploration setting (Figure 10b) and enables very fast detection in WAST after 1 epoch (Figure 10d).

Original: G Robustness to Noisy Environments

Original: Our experiments on the Madelon dataset illustrate the robustness of WAST to noisy features.

Original: In this appendix, we study the robustness of the unsupervised NN-based methods on noisy samples.

Original: To this end, we created noisy versions of the datasets by adding Gaussian noise with zero mean to the training data.

Original: To assess the robustness of different noise levels, we evaluated 4 different values of standard deviation {0.2, 0.4, 0.6, 0.8}.

Original: We performed this analysis on datasets of different types.

Original: We use the same experimental setting described in Section 4.3 and Appendix A.

Original: Figure 11 illustrates the classification accuracy using K = 50 .

Original: We observe that across different dataset types, methods that are based on sparse models (i.e., WAST and QS) are more robust to noise.

Original: As expected, the accuracy decreases when the level of noise increases in all methods.

Original: Yet, sparse-based methods have the least decrease.

Original: Moreover, WAST outperforms QS in all cases.

Original: H Advance in Sparsity

Original: Most of the efforts in previous years are devoted to the algorithmic side of sparse training.

Original: The aim is to achieve the same or higher performance than the dense models with highly sparse counterparts, as discussed in Section 2.2.

Original: Existing sparse training methods in the literature simulate sparsity using masks over dense weights since most deep learning specialized hardware is optimized for dense matrix operations.

Original: Yet, recently, research on the software and hardware that support sparsity has received growing attention.

Original: NVIDIA

Original: [Figure]

Original: cc u r

Original: [Figure]

Original: c y [ % ] (c) HAR.

Original: [Figure]

Original: released NVIDIA A100, which supports a 50% fixed sparsity level [76], and many other efforts on the hardware side are proposed [66; 3; 13; 43].

Original: On the software side, libraries that support truly sparse implementations have been started for supervised learning [48; 15].

Original: With a joint community effort in the algorithmic, software, and hardware directions, we would be able to actually provide faster, memory-efficient, and energy-efficient deep neural networks.

Original: Further discussion can be found in [29; 53].