Dynamic Sparse Network for Time Series Classification: Learning What to “See”
Abstract
The receptive field (RF), which determines the region of time series to be “seen” and used, is critical to improve the performance for time series classification (TSC).
However, the variation of signal scales across and within time series data, makes it challenging to decide proper RF sizes for TSC.
In this paper, we propose a dynamic sparse network (DSN) with sparse connections for TSC, which can learn to cover various RF without cumbersome hyper-parameters tuning.
The kernels in each sparse layer are sparse and can be explored under the constraint regions by dynamic sparse training, which makes it possible to reduce the resource cost.
The experimental results show that the proposed DSN model can achieve the state-ofart performance on both univariate and multivariate TSC datasets with less than 50% computational cost compared with recent baseline methods, opening the path towards more accurate resource-aware methods for time series analyses.
Our code is publicly available at: https://github.com/QiaoXiao7282/DSN .
Introduction
Time series classification (TSC) as an important research topic in the data mining communities [26], has a wide range of applications from health monitoring [27, 14] and public security [60], to grid energy [43] and remote sensing [45].
In the last decade, neural networks, especially deep neural networks, have achieved competitive or even better performance than traditional TSC approaches (e.g. DTW [48], BOSS [49], COTE [1]) in many cases [58, 26].
However, how to discover and exploit the various scaled signals hidden in time series (TS) is still a significant challenge for TSC tasks [54, 39, 21].
One main reason lies in several variances, like sampling rate and record length, which are natural during time-series collection [49, 34, 5].
Furthermore, the amplitude offset, warping and occlusion of data points are unavoidable (see Figure 1).
So determining the optimal scales for feature extraction is difficult but significant in the TSC task.
One main solution is to cover as many receptive field (RF) scales as possible, so it does not ignore any useful signals from the time series inputs [54, 7, 8, 28].
Grounding on the motivation that multiple kernels in combination can capture discriminative patterns despite complex warping, Fawaz et al.
[13] tried to ensemble convolutional neural networks with a large number and variety of kernels.
Tang et al.
[55] have stacked several OS-CNN blocks, each of ∗ Equal contribution. which consisted of a list of kernel sizes to cover all scales of RF and achieve better performance on several TSC tasks.
In Multi-scale Convolutional Neural Network (MCNN) [4], a grid search was applied to find suitable kernel sizes.
Without elaborate setting and fine-grid search, Dempster et al.
[7] transformed time series by using random convolutional kernels and then trained a time series classifier.
Even though the use of kernels with various sizes can help to extract latent hierarchical features of multiple resolutions, it tends to cause a huge computational problem and overfits the datasets when a limited amount of training data is available [28].
To address this problem, dilated convolution [62] has been used to reduce the computational cost while keeping the receptive field size.
Franceschi et al.
[15] presented an unsupervised learning model with convolutional kernels for time series feature transformation, in which the dilation factors of kernels increased exponentially layer by layer.
To date, there is still an unresolved challenge with hyperparameter selections (e.g. kernel sizes and dilated factors) to capture patterns at different scales.
This raises a foundational question: Can we design a scalable method for TSC that achieves a trade-off between the computation and performance without cumbersome hyperparameter tuning?
[Figure]
does not
need
to
be
configured with a series of kernel sizes k . Inspired by sparse neural network models which can match the performance of dense counterparts with fewer connections [42, 37, 35, 63], we propose a dynamic sparse network (DSN) that is composed of CNN layers with large but dynamic sparse kernel for TSC tasks, which can automatically learn sparse connections in terms of covering various RF.
The proposed DSN is trained by a dynamic sparse training (DST) strategy, which makes it possible to reduce the computational cost (e.g. floating-point operations (FLOPs)) for training and inference as well.
Different from the traditional dynamic sparse training methods, which explore the connections in a layer-wise manner [42, 36, 12] and can hardly cover smaller RF, we propose a more fine-grained sparse training strategy for TSC.
Specifically, the
CNN kernels in each layer are split into several groups, each of which is sparse and can be explored under a constraint region during the course of sparse training.
In this way, each layer can extract features in a more scalable and diverse way, and then by cascading, the model can cover more various scales of RF.
To sum up, our main contributions are:
• We propose a novel dynamic sparse CNN layer with various effective neighbour RF, which can learn to “see” selectively and adaptively for more accurate TSC.
• By stacking the above dynamic sparse CNN layers, we introduce a novel DSN model natively for TSC, which is trained by DST to reduce computational and memory cost.
• To bridge gap between resource awareness and accuracy, we propose a more fine-grained training strategy for DSN, which can more easily cover diverse effective RF for TSC.
• We perform extensive empirical evaluation on both univariate and multivariate TSC tasks, showing that DSN achieves superior performance in terms of accuracy and resource cost.
Related Work 2.1 Time Series Classification
In the last decade, the success of deep learning encourages researchers to explore and extend its application on TSC tasks [31, 26, 58, 24, 32].
For univariate TSC, deep learning-based models attempt to directly transform raw time series data into low-dimensional feature representation via convolutional or recurrent networks [28, 47, 44].
For multivariate TSC, LSTM and attention layers are often stacked with CNN layers to extract complementary features from TS data [29, 65].
Recently, many works attempt to extract features in a more scalable manner as TS data is unavoidably composed of signals on various scales [49, 5, 54].
One main solution is to configure suitable kernels of various sizes such that they increase the probability to capture the proper scales [54, 55, 28].
Rocket-based methods [7, 8, 9] aim to use random kernels with several sizes and dilation factors to cover diverse RF for TSC.
Different from these works, our proposed dynamic sparse CNN layer can adaptively learn the various RF with deformable dilation factors and trade-off both computation and performance without cumbersome hyperparameter tuning.
2.2 Sparse Training
Recently, the lottery ticket hypothesis has shown that it is possible to train a sparse sub-network to match the performance of dense counterpart with less computational cost [16].
Rather than iteratively pruning from a dense network [18, 66, 16], recent works try to find an initial mask with one-shot pruning based on gradient information during training [33, 57].
After pruning, the topology of the neural network will be fixed during training.
However, this kind of models can hardly match the accuracy achieved by their dense counterparts [41, 57, 53].
Introduced as a new training paradigm before the lottery ticket hypothesis, DST starts from a sparse network and allows the sparse connectivity to be evolved dynamically with a fixed number of parameters throughout training [42, 12, 37, 63].
Nowadays, DST attracts increasing attentions from other research fields such as reinforcement learning [51] and continual learning [50], due to its potential of outperforming dense neural networks training [42, 63, 37].
Different from the traditional
DST methods, our proposed DSN is trained with a fine-grained sparse training strategy rather than a traditional layer-wise manner to capture more diverse RF for TSC.
2.3 Adaptive Receptive Field
The RF, which can be adaptively changed during the training, has proven to be effective in many domains [56, 46, 59].
The adaptive RF can be usually captured by learning the optimal kernel sizes or the masks of kernels during the course of training [52, 19, 59].
However, neither the kernels nor the masks are sparse, which may cause the huge computational problem when larger RF needed.
Different from these methods, our dynamic sparse CNN layers in DSN can be trained with DST, which can learn to capture variable RF with deformable dilation factors.
What’s more, the kernels are always sparse during training and inference, which makes it possible to reduce the computational cost.
The Proposed Model 3.1 Problem Definition
Definition 1.
( Time Series Classification (TSC) ).
Given a TS instance X
= { X 1 , . .
X n } ∈ R n × m , where m denotes the number of variates and n denotes the number of time steps, TSC aims to accurately predict the class label y ∈ { 1 , . .
. , c } from c classes.
When m equals 1, TSC is univariate and otherwise it is multivariate.
Definition 2.
( Time Series (TS) Training Set ).
A training set D
= { (cid:0) X (1) , y (1) (cid:1) , . . .
, (cid:0) X ( N ) , y ( N ) (cid:1) } consists of N time series instances,
where X ( i ) ∈ R n × m could either be a univariate or multivariate time series instance with its corresponding label y
( i ) ∈ { 1 , . .
, c } .
Note that, in our case, all instances have the same number of time steps in a TS dataset.
Without the loss of generality, given a training set, we aim to train a CNN classifier with adaptive and various RF with a low resource cost (e.g. memory and computation) for the TSC task.
3.2 Dynamic Sparse CNN Layer with Adaptive Receptive Field
A straightforward strategy to cover various RF is to apply multi-sized kernels in each CNN layer, but it has several limitations.
Firstly, TS instances from different TS datasets do not have the same length and cycles 2 in most cases, making it difficult to set a fixed kernel configuration for all the datasets even with prior knowledge.
Secondly, obtaining a large receptive field commonly requires a large kernel or stacking more layers, which introduces more parameters and thus increases storage and computational costs.
To tackle those challenges, the proposed dynamic sparse CNN layer possesses large but sparse kernels, which are learnable to capture adaptive RF.
Specifically, given the input feature map x l ∈ R c l − 1 × h × w in the l th layer ( h equals 1 for univariate TSC and c l − 1 is the number of input channels) , and kernel weights Θ [Equation] ( k is the kernel size and c l is the number of output channels), the convolution with stride 1 and padding in our proposed dynamic sparse CNN layer is formulated as
[Equation]
where O j ∈ R h × w denotes the output feature representation at the j -th output channel, Z denotes the set of integers, I l ( · ) :
R c l − 1 × c l × 1 × k → { 0 , 1 } c l − 1 × c l × 1 × k is an indicator function,
I l
(Θ l ) i,j indicates activated weights for Θ li,j ∈ R 1 × k which is the kernel in the ( i, j ) -th channel, ⊙ denotes the element-wise product, and · denotes the convolution operator.
The indicator function I l ( · ) , which is learned during the training of the proposed DSN, satisfies that ∥
I l ( · ) ∥ 0 ≤ (1 − S ) N l where 0 ≤ S < 1 is the sparsity ratio, ∥ .
∥ 0 denotes the L 0 norm and N l = c l − 1 ∗ c l ∗ 1 ∗ k .
When S > 0 , the kernel is sparse, and we can use a large k in the dynamic sparse CNN layer to obtain a large RF with reduced computational cost.
Remark 1.
( Effective Neighbour Receptive Field (eNRF) size in the dynamic sparse CNN layer ).
The receptive field is defined as the region in the input that the feature of a CNN model is looking at.
We defined the Neighbour Receptive Field (NRF) as the region considered by each feature in the successive layer.
Specifically, the NRF size is equivalent to the kernel size in the standard CNN layer (Considering the case that dilation equals to 1).
Differently, the NRF size of the proposed dynamic sparse CNN layer is smaller than the kernel size, when the first or last weight in a kernel is not activated, e.g., ∃ i ∈ { 1 , . . .
, c l − 1 } and j ∈ { 1 , . .
c l } , such that I l (Θ l ) i,j, 1 , 1 = 0 or
I l
(Θ l ) i,j, 1 ,k = 0 .
As illustrated in Figure 2, we defined the eNRF size f li,j of the kernel Θ li,j as the distance between the first activated weight and last activated weight in the l -th CNN layer as
[Equation]
otherwise
[Equation]
where Ind li,j denotes the set of indices corresponding to non-zero weights in the kernel Θ li,j .
Obviously we have 0 ≤ f li,j ≤ k .
= =
= =
= =
=
= 3 =
The eNRF size set of l -th dynamic sparse CNN layer is denoted by F ( l ) , which satisfies that 0 ≤ min( F ( l ) ) and max( F ( l ) ) ≤ k .
Take the case in the Figure 2 as an simple example for l -th layer, then F ( l ) = { 0 , 1 , 2 , 3 , 4 , 5 , 6 } .
Each dynamic sparse CNN layer is likely to cover various eNRF sizes ranging from 1 to k .
When global information is expected, I l ( · ) could activate weights more dispersed to get a larger eNRF and exploit the input features selectively.
To capture the local context, a smaller eNRF is expected, thus the activated weights tend to be concentrated.
By taking k = 5 as an example, I l (Θ l ) i,j may be [1, 0, 1, 0, 1] for global context, while it may be [0, 0, 1, 1, 0] for local context.
In this way, the eNRF can be adaptively adjusted.
3.3 Architecture in DSN
The proposed DSN model consists of three sparse CNN modules, each of which composes of a dynamic sparse CNN layer and a 1 × 1 CNN layer.
Following the stacked sparse CNN modules, there is an additional dynamic sparse CNN layer, two adaptive average pooling layers, and one 1 ×convolution layer acting as the classifier in the DSN model.
The overall architecture is shown in
Figure 3.
[Figure]
The set of eNRF sizes S ( l ) in the l th sparse CNN module is equal to that in the dynamic sparse
CNN layer, since the eNRF size of 1 × 1 convolution is constantly equal to 1 .
S ( l ) satisfies that 0 ≤ min( S ( l ) ) and max( S ( l ) ) ≤ k .
Then, the set of eNRF sizes for the three successively stacked sparse CNN modules can be described by RF as
[Equation]
i ) ∈ S ( i ) , i ∈ { 1 , 2 , 3 } (cid:111) .
According to Eq.
(3), we can see that stacking multiple sparse CNN modules can increase the size of RF linearly, where l th dynamic sparse CNN layer increases by the size of S ( l ) .
For simplicity, the kernel size in each dynamic sparse CNN layer is consistently set to k in our study.
Then, RF satisfies that max( RF ) ≤ 3 k − 2 and 0 ≤ min( RF ) .
Thus, a large k in sparse CNN modules can increase the range of the eNRF sizes for stacked sparse CNN modules. [Figure]
3.4 Dynamic Sparse Training for DSN Model
In this section, we present the training strategy to discovery the weights required to be activated in order to ensure well-performing RF.
That is, we need to study how to update the indicator function
I l ( · ) during the training of the proposed DSN.
Input: Dataset D , Network f Θ , Sparsity ratio S Exploration regions in layer l : {R l 1 , ..., R lN }
Exploration schedule: T , ∆ T , α , f decay 1: θ li ← initialize activated weights in Θ li using S and R li 2: for t = 1 to T do 3:
Sample a batch B t ∼ D L t = (cid:80) i ∼ B t L (( f θ ( x i ) , y i ) 4: if ( t mod ∆ T ) =
=
then 5: for each layer l = 1 to L do 6: for each group i = 1 to N do 7: u = n ( R li )
f decay ( t ; α, T ) (1 − S ) 8:
I prune = ArgTopK (cid:0) − (cid:12)(cid:12)
θ li (cid:12)(cid:12) , u (cid:1) 9:
I grow =
RandomK (cid:0) R li \ θ li , u (cid:1) 10:
I l ( . )
← Update I l ( . )
using I prune
and I grow 11: θ li ← Update activated weights I l (Θ l ) i ⊙ Θ li 12: end for 13: end for 14: else θ = θ − α ∇ θ L t 15: end if 16: end for
We sparsely train the proposed DSN model from scratch to keep kernel sparse following the main idea of DST methods [42].
By design, during the training phase, the total number of activated weights must not exceed N l (1 − S ) .
However, we observe that the tiny eNRF is hard to be captured by
DST methods with a layer-wise exploration manner, namely activated weights are discovered layer by layer, especially when the sparsity ratio S is undersized (more analysis in Section 4.5).
Motivated by this observation, the kernels in each dynamic sparse CNN layer are divided into different groups, whose corresponding exploration regions are of different sizes as shown in
Figure 4.
Contrary to the DST methods, in the DSN the exploration of activated weights is separately performed in different kernel groups, which is a more fine-grained strategy.
Specifically, the kernel weights Θ l ∈ R c l − 1 × c l × 1 × k in the l th layer are split into N groups along the output channel, that is, Θ l 1 , . . .
, Θ lN ∈ R c
l − 1 × clN × 1 × k with corresponding exploration regions R l 1 , . . .
, R lN .
The exploration region R li in the i th group of the l th layer is defined as the first i × kN positions of each kernel in this group.
By taking k = 6 and N = 3 for example, activated weights in the first group are only within the first two positions of each kernel in Θ l 1 , while they are within the whole kernel for the last group (as shown in Figure 4).
In this way, the various eNRF can be covered, and exploration space can be reduced (more details can be found in Appendix A.1) to improve exploration efficiency.
Given the weight exploration regions R l 1 , . . .
, R lN and the sparsity ratio S , we train
the proposed
DSN model as shown in Algorithm 1.
The activated weights are explored within the exploration regions and updated every ∆ t iterations.
The fraction of updated weight decays over time according the function f decay ( t ; α, T ) , which follows the cosine annealing [10] as
[Equation]
where α is the initial fraction of activated weights updated, t is the current training iteration, and T is the number of training iterations.
Thus, during the t th iteration, the number of updated activated weights in the i th group of the l th layer is n ( R li ) f decay ( t ; α, T ) (1 − S ) , where n ( R li ) is the number of weights can be explored in region R li .
During the update of activated weights, we firstly prune the activated weights determined by ArgTopK (cid:0) − (cid:12) (cid:12) θ li (cid:12) (cid:12) , u (cid:1) and then randomly grow new activated weights by RandomK (cid:0) R li \ θ li , u (cid:1) , where ArgTopK( v , u ) gives the indices of top u elements with larger values in a vector v , RandomK( v , u ) outputs the indices of random u elements in a vector v , and R li \ θ li denotes the weights within R li but except θ li .
The exploration for activated weights is straightforward.
Firstly, pruning weights with small magnitudes is intuitive, because the contribution of weights with smaller magnitudes is insignificant or even negligible.
Considering the pruning recoverability, we randomly regrow new weights with the same number as the pruned weights to achieve better activated weight exploration.
In this way, the exploration of weights is dynamic and of plasticity, compared with the methods of pruning weights before and after training (more explanations can be found in Appendix A.3).  Experiments In this section, we evaluate the resource cost and accuracy of the proposed DSN model with recent baseline methods on both univariate and multivariate TSC data.
4.1 Datasets and Baselines
The details of each kind of dataset are as follows.
• Univariate TS dataset from UCR 85 archive [3] : This archive consists of 85 univariate TS datasets, which are collected from various domains (e.g. health monitoring and remote sensing), have distinguishable characteristics, and various levels of complexity.
The number of instances in the training set varies from 16 to 8926, while their time step resolutions range from 24 to 2709.
• Three multivariate TS datasets from UCI [11] : The EEG2 dataset contains 1200 instances of 2 categories and 64 variates.
The Human Activity Recognition (HAR) dataset consists of 10,299 instances with 6 categories, and the number of variates is 9.
The Daily Sport dataset contains 9,120 instances of 19 categories, and its number of variates is 45.
The processing of these dataset is the same as in [29].
Univariate TSC baselines: The performance of the following three baseline methods are reported,
OS-CNN [54], InceptionTime [28], and ResNet[58], as they are widely used for univariate TSC.
Multivariate TSC baselines: Similar with [65], the following three multivariate baselines are selected: OS-CNN, TapNet [65], MLSTM-FCN [29].
To avoid an unfair comparison, we let outside the ensemble methods.
However, based on our best knowledge, our performance analysis accounts for most of state-of-the-art methods.
4.2 Experimental Settings and Implementation Details
For optimization, we use the Adam optimizer with an initial learning rate of 3 × 10 − 4 , cosine decayed to 10 − 4 .
Our model is trained for 1,000 epochs with a mini-batch size of 16.
As in [28, 29], the best model, which corresponds to the minimum training loss, is used to evaluate performance over the testing sets.
Inspired by [28], the kernel sizes k in each dynamic sparse CNN layer are set to 39.
The number of kernel groups N in Algorithm 1 is set to 3, which helps to cover the small, medium and large eNRF.
Each setting is repeated for five times and the average results are reported.
4.3 Results and Analysis
We show the performance on both univariate and multivariate TSC benchmarks in Table 1-2.
For univariate TSC, we can see that our method outperforms the baseline methods in most cases with a smaller number of parameters (e.g. Params 3 ) and less computation cost (e.g. FLOPs) on UCRarchive datasets.
For multivariate TSC, our proposed DSN method achieves better performance on
EEG2 and HAR datasets, while for Daily Sport, MLSTM-FCN was with 0.46% more accurate.
It is worth highlighting that the resource cost (i.e. Params and FLOPs) of our proposed DSN is much smaller than that of baseline methods.
Due to the page limitation, the detailed results including test accuracy and resource cost on each dataset of UCR 85 archive can be found in Appendix C.
[Table]
4.4 Sensitivity Analysis
Effect of Sparsity Ratio: From Table 3, we can find out how the sparsity ratio of dynamic sparse
CNN layer affects the final test accuracy and the resource cost in multivariate TS datasets.
Here, we analyze the trade-off between test accuracy and the resource cost under various sparsity ratios
[Table]
(i.e. S ∈ [50% , 80% , 90%] ) and a dense DSN model.
Note that the dense model is defined exactly as our DSN model with dense connections along entire kernels.
We can see that it is not that the more parameters of the model, the better performance it will achieve.
This is mainly because more parameters will lead to overfitting and the proper sparse ratio will easily cover the desired receptive field (more demonstrations can be seen in Appendix B.1).
Taking EEG2 data set as an example, with the decrease of sparse rate, the performance on it drops sharply.
We hypothesise that this data set has more expectations for features from small receptive fields (the validation can be found in
Appendix B.2).
According to the experiments, a 80% sparsity ratio reflects a good trade-off between accuracy and resource cost, which is the default setting for the dynamic sparse CNN layer in our
DSN model.
TS
[Table]
Effect of Architecture: To show how the architecture of DSN affects the final performance, we adopt the critical difference diagram for a detailed comparison, which is usually used for evalution in
TSC [54, 28].
From Figure 5, we can find that the DSN model with 4 dynamic sparse CNN layers and 177 output channels in each layer performs consistently better than the other structures.
The proposed model with more output channels seems to be crucial for preserving the capacity of the network, while requires more parameters and FLOPs compared to other counterparts.
This highlights an interesting trade-off between accuracy and computational efficiency.
According to the experiments, the DSN model with 4 dynamic sparse CNN layers and 141 output channels in each layer is the trade-off between the performance and resource cost, which is the default setting for our DSN model.
Effect of Kernel Group: As shown in Figure 6, training DSN without kernel group hampers the capture of small eNRF, which can not guarantee satisfactory performance on the datasets such as EEG2 expecting more local information. With the decrease in sparsity ratio S , the phenomenon that large RF occupies the majority becomes more severe (more demonstrations see in Appendix B.1), which increases the performance gap between training with group and without it as shown in Table 4. Grouping the kernels can achieve various RF coverage and adorable accuracy for all datasets.
[Figure]
4.5 Ablation Study
[Table]
Effect of Dynamic Sparse Training:
Furthermore, we study the effect of the dynamic sparse training in our DSN method.
We conduct ablation studies to compare DSN with its static variant (i.e. DSN fix ), which fixes its topology during training after activated weights initialization.
In other words, the indicator function I l ( · ) in Eq.
(1) will not update during the training process.
According to the results shown in Figure 7 (a), we can find that the DSN model mostly outperforms the DSN fix on UCR 85 archive, indicating that what are learned by dynamic sparse training is the suitable activated weights together with their values.
[Figure]
4.6 Case Study on other DST Method
[Figure]
Likewise, the effectiveness of the proposed DSN method is also analyzed in the presence of other DST methods, such as RigL [12].
Rather than growing weights randomly (as in the line 9 of Algorithm 1), RigL grows them with the highest magnitude gradients.
From Figure 7 (c), we can see that growing weights according to gradients (i.e. DSN grad ) can outperforms the OS-CNN method in most cases on
UCR 85 archive.
Furthermore, as shown in Figure 7 (b), we can find that DSN grad has a similar performance as growing weights randomly (i.e. DSN rand ).
Note that, DSN rand is the main setting of the proposed method in this paper with the reason that we don’t need to compute the full gradients at any moment in time.
In summary, the proposed DSN with fine-grained sparse strategy can be easily combined with the existing DST techniques, showing the potential for further improvements by merging it with other advanced DST methods in the future.
4.7 Visualization of Activated Weights
Figure 8 shows the normalized activated weights of the DSN model after sparse training on HAR dataset.
(a), (b) and (c) correspond to the activated weights of the three kernel groups in the first dynamic sparse CNN layer, and each row of the figure represents one kernel.
We can see that the weights in each kernel are sparse, and can be activated within the constraint region defined in
Section 3.4.
Conclusions
This paper proposes dynamic sparse network (DSN) to cover diverse effective neighbour receptive field (eNRF) for time series classification (TSC) without the cumbersome hyper-parameters tuning.
The proposed DSN can achieve about 2 × reduction in memory and computational cost, while achieving superior performance in terms of accuracy on both univariate and multivariate TSC datasets.
Furthermore, DSN explores the activated weights in a more fine-grained strategy, which can be easily merged with existing DST techniques, showing a potential for further improvements when combined with advanced technology.
DSN provides a feasible solution to bridge gap between resource awareness and various eNRF coverage for TSC, hoping to inspire other researches in different fields.
It is worthwhile extending DSN to other research domains (e.g. time series forecasting and computer vision).
Our DSN model has not yet been executed on real edge devices, and the performance in terms of accuracy and memory footprint on edge devices is not investigated.
In the future, we hope to extend our work to other fields and decentralise training on edge devices.
Acknowledgements
This work has been partly funded by the NWO Perpectief project MEGAMIND.
A Preliminaries A.1 Preliminaries on Comparison of Exploration Space
The size of exploration space for the layer-wise manner in SET algorithm is:
[Equation]
If we divide the kernels in one layer into N groups and explore N l (1 − S ) N activated weights in each group [Equation] then the size of the exploration space becomes:
[Equation]
When the corresponding exploration region of each group is defined as in Section 3.4, the exploration space can be further reduced to:
[Equation]
A.2 Preliminaries on Calculation of Parameters
The summation of the number of non-zero weights (activated weights in sparse CNN layer + weights in other layers) in the network is used to estimate the size of network as follows:
[Equation]
where L contains a set of sparse CNN layer indexes, and θ is the parameter of our DSN which includes an amounts of zero values.
∥
I l (Θ l ) i ⊙ Θ li ∥ 0 is controlled by the sparsity ratio S and the corresponding exploration regions R li .
A.3 Preliminaries on Existing Compression Solutions
The existing compression solutions can be categorized into after-training, during-training and beforetraining methods.
• After-training methods obtain a sparse or small neural network from a pre-trained network by pruning the unimportant weights [20], quantifying the weights [61] or distillation [22]. Normally, this schedule type uses a standard dense training procedure followed by a retrained procedure after compression to reach higher accuracy. • During-training methods , which discover the sparse network during training without pre-trained network, can be divided into dense-to-sparse and sparse-to-sparse training strategy. Dense-to-sparse training strategy sparsifies the neural network during the training process through gradual magnitude pruning [17] or regularization techniques [38].
Sparseto-sparse training strategy initializes a random (topology and weights) sparse network, and dynamically optimizes the topology as well as the weights simultaneously during training [42, 12].
Our DSN model is based on the sparse-to-sparse training strategy.
• Before-training methods aim to obtain a trainable small or sparse neural network before the main training process.
Squeezenet [25] and MobileNets [23] are specifically engineered dense networks that are smaller than standard architectures and can run with limited resources on the edge device.
Furthermore, motivated by [41, 33], many works [53, 57] have shown that pruning at initialization without optimizing network topology during training can also perform equally well.
B More Experiments B.1 The Statistics of eNRF Sizes with Different Sparsity Ratios
The distributions of eNRF sizes in the first dynamic sparse CNN layer for HAR, Daily Sport and EEG2 datasets are shown in Figure 9, Figure 10 and Figure 11, respectively.
We can see that DSN model with kernel grouping can cover more diverse eNRF sizes than it does without kernel grouping under different sparsity ratios S .
With the decrease of S , the DSN model without kernel grouping tends to capture the large eNRF, while ignoring the small eNRF, which harms the performance of DSN model.
Under different sparsity ratios, the DSN model with kernel grouping can guarantee a satisfactory performance for different datasets by covering various eNRF. [Figure] [Figure] [Figure]
B.2 Better Kernel Sizes for EEG2 Dataset
We study the effect of kernel size k on the performance of spares/dense DSN for EEG2 dataset to verify our hypothesis that this dataset expects the local information.
The dense DSN model is defined exactly as our DSN model with dense connections along entire kernels.
In this way, the RF size in each layer equals to kernel size.
As shown in Figure 12, the dense DSN with a kernel size of 9 achieves the highest test accuracy.
However, with the increase of kernel size (RF size), the test accuracy drops dramatically.
It is obvious that the local context plays a significant role for more accurate classification for EEG2 dataset.
In contrast, our proposed sparse DSN model (sparsity ratio
S = 80% ) with a large kernel size can still achieve satisfactory performance, with the main reason that it can learn the adaptive eNRF for the dataset.
B.3 Can Various eNRF Outperform the Optimal NRF?
To demonstrate the advantage of various eNRF coverage, we compare the performances in terms of test accuracy between DSN models with various eNRF coverage and optimal NRF.
The dense
DSN models with different kernel sizes (from 5 to 40 with step 5) in each layer are used to search for optimal NRF in different datasets on the UCR 85 archive.
As shown in Figure 13, by capturing various eNRF, our DSN model can achieve similar performance to the optimal NRF model without the cumbersome hyper parameters.
[Figure]
B.4 The Effect of Kernel Size for DSN
From Figure we can find out how the kernel size k in the proposed DSN model affects the final test accuracy for UCR 85 archive.
We can see that the performance of DSN model with small kernel sizes (e.g. 15 and 21) is worse than that of it with large kernel sizes (e.g. 39 and 45).
It is because that the model with larger kernel sizes can cover larger eNRF which is useful to capture global information for some datasets.
However, when the kernel size is oversized (e.g. 45), some of the small eNRFs can be lost in a predefined sparsity ratio (e.g. 80% in here), which will slightly degrade the performance instead.
B.5 The Effect of Kernel Group for DSN
We study how the number of kernel groups N defined in Algorithm 1 affects the performance of proposed DSN model.
From Figure 15, we can see that DSN model with N = 2 outperforms other counterparts.
As discussed in 4.5 and B.1, grouping the kernels in each dynamic sparse CNN layer can help to achieve various NRF coverage especially with smaller sparsity ratio S .
However, when the number of kernel groups (e g. 4) is oversized, the distribution of eNRF sizes is nearly uniform, which will degrade the performance of DSN.
B.6 The Effect of Topology Initialization
As shown in Figure 16, the activated weights can be initialized randomly and sequentially before exploration, namely DSN initrand and DSN initseq respectively.
After initialization, the topologies initialized by DSN initrand and DSN initseq are different.
In detail, DSN initrand randomly activates the weights within the exploration regions, while DSN initseq activates the first i × kN S weights of kernels in i th group.
Figureexemplifies the case where k = 16 , N = 4 and S = 50% .
We further investigate which type of topology initialization are more suitable for the proposed DSN model.
According to the results shown in Figure 17 (left) and Figure 18, we can see that the proposed DSN model with sequential initialization performs better than that with random initialization.
Based on this, we adopt sequential initialization as the default setting for DSN model in all of the experiments.
Intuitively, the DSN method with random topology can also cover various eNRF.
We further study the different performances of NRF which is from dynamic sparse training and from random topology in DSN method.
We conduct ablation studies to compare DSN with its static variant (i.e. DSN fixrand ), which fixes its topology during training after activated weights random initialization.
According to the results shown in Figure 17 (right), we can find that the DSN model outperforms the DSN fixrand on
UCR 85 archive, indicating that what is learned by dynamic sparse training is the suitable activated weights together with their values.
[Figure]
[Figure] [Figure] [Figure]
[Figure]
B.7 Extremely Sparse DSN
We study how the performance of our DSN is affected under extremely sparse ratios (e [Table] g. S = 90% ,
S = 95% and S = 97% ).
As shown in Table 5, we can see that our DSN with sparsity ratio
S = 95% still performs better than OS-CNN while achieving more than 5 × resource cost reduction (e.g. Parameters and FLOPs).
When sparsity ratio S = 97% , which means that 97% weights in sparse CNN layer are set to zeros, the performance of our DSN model is only slightly worse than that of OS-CNN, but the resource cost is nearly 7 × less.
From Figure 19, we can see that most blue dots (corresponding to accuracies of OS-CNN) are below the orange dots (corresponding to accuracies of DSN with extremely sparsity ratio), which means that OS-CNN performs worse than extremely sparse DSN in most dataset.
What’s more, DSN is far superior to OS-CNN in some datasets (e [Table] g.
CinCECGtorso, MiddlePhalanxOutlineAgeGroup and Earthquakes).
[Table]
[Figure]
cy
[Table]
B.8 Learning Curve on Three Multivariate TS Dataset
Figure 20 shows the learning behaviors of our DSN model and OS-CNN.
We can see that our DSN model has a convergence speed similar to that of OS-CNN, while the resource requirements (memory and computational cost) are much less.
Furthermore, the performance of DSN is far superior to that of OS-CNN for EEG2 datasets .
B.9 Additional Experiments on other TSC datasets
In order to evaluate if the good learning capabilities of DSN can be generalized over other larger scale of univariate and multivariate time-series datasets we evaluate it on the following datasets:
• UCR 2018 archive [6].
This archive consists of 128 univariate TS datasets, which is the updated version of the UCR 85 archive.
Among them, 15 datasets are with unequal length and one (Fungi) has a single instance per class in the training files.
As shown in [40], most implementations are not set up to handle these kinds of data, we also only report our results on the remaining 112 problems in UCR 2018 archive.
• University of East Anglia (UEA) 30 archive [2].
It consists of 30 multivariate TS datasets with distinguishable characteristics, and various levels of complexity.
The class numbers vary from 2 to 39, and their numbers of variates vary from 2 to 963.
For UEA 30 archive, we add the resource awareness method Rocket [7] and the transformer based method TST [64] as our baselines. We summarize the performance on both additional univariate and multivariate TSC benchmarks in Table 6-7.
We can see that, in terms of test accuracy, our method can still match the state-of-the-art methods, while having much smaller computational cost (e.g. the number of parameters and FLOPs).
It is worth highlighting our proposed DSN method achieves a trade-off between the resource cost and performance in both univariate and multivariate
TSC benchmarks. [Table]
[Table]
C Detailed Results
For UCR 85 archive, the reported results of ResNet and InceptionTime are from [7], while that of
OS-CNN are from the official repository 5 .
For three multivariate datasets from UCI, the reported results of LSTM-FCN are from the original paper, and that of TapNet and OS-CNN are from the implementation of official repository with default settings.
For UEA 30 archive, the detailed results including test accuracy and resource cost on each dataset can be found in Table 9.
The detailed results for UCR 85 archive are given in Table 8.
We also visualize the detailed results in
Figure 21, we can see that the accuracies of DSN are nearly on the top in most of the datasets.
In addition, it is worth noting that DSN loses by a slight margin in some datasets, but wins by a big margin in others.
[Figure]
D More Implementation Details
In our dynamic sparse training algorithm 1, we choose the cosine annealing defined in Eq.
4 with ∆ T = 5 epochs and α = 0 .
For UEA 30 archive, in each dynamic sparse CNN layer, the output channel is set to 177 with sparsity ratio S = 90% .
For UCR 112 archive, except for each CNN layer is padded with zero-value, other hyper-parameters are the same as in UCR 85 archive.
We use
PyTorch 6 to implement our method and run our experiments on Nvidia Tesla P100.
[Table]
GunPoint
[Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
24 Ham
[Table] [Table]
[Table] [Table]
[Table] [Table]
HandOutlines 91.
[Table]
[Table] [Table]
[Table] [Table]
54 Haptics
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
82 Herring
[Table] [Table]
[Table] [Table]
[Table] [Table]
97 InlineSkate
[Table]
[Table] [Table]
[Table]
69 InsectWingbeatSound
[Table]
[Table] [Table]
[Table] [Table]
[Table]
77 ItalyPowerDemand
[Table] [Table]
01 LargeKitchenAppliances
[Table] [Table]
[Table] [Table]
56 Lighting
[Table] [Table]
[Table] [Table]
[Table] [Table]
95 Lighting
[Table] [Table]
[Table] [Table]
[Table] [Table]
27 MALLAT
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
05 Meat 96.
[Table] [Table]
[Table] [Table]
[Table] [Table]
13 MedicalImages
[Table] [Table]
[Table]
24.82 MiddlePhalanxOutlineAgeGroup
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
92 MiddlePhalanxOutlineCorrect
[Table]
81.33 ± 1.
[Table]
19.89 MiddlePhalanxTW 48.
61.45 ± 0.
20.00 MoteStrain 92.
90.73 ± 0.
20.88 NonInvasiveFatalECGThorax 94.
94.36 ± 0.
187.11 NonInvasiveFatalECGThorax 94.
94.72 ± 0.
187.11 OSULeaf 97.
98.84 ± 0.
106.01 OliveOil 83.
94.00 ± 3.
141.40 PhalangesOutlinesCorrect 83.
83.87 ± 0.
19.89 Phoneme 33.
33.70 ± 0.
254.93 Plane 100.
100.00 ± 0.
35.89 ProximalPhalanxOutlineAgeGroup 85.
86.05 ± 0.
19.92 ProximalPhalanxOutlineCorrect 92.
93.20 ± 0.
19.89 ProximalPhalanxTW 78.
81.95 ± 0.
20.00 RefrigerationDevices 52.
55.84 ± 1.
178.56 ScreenType 62.
63.95 ± 1.
178.56 ShapeletSim 77.
72.67 ± 4.
124.00 ShapesAll 92.
[Table] [Table]
[Table] [Table]
56.80 ± 3.
119.04 ERing 13.
92.22 ± 1.
18.55 EigenWorms 50.
39.08 ± 11.
5075.34 Epilepsy 76.
99.86 ± 0.
58.21 EthanolConcentration 37.
24.49 ± 0.
493.68 FaceDetection 54.
63.49 ± 0.
18.58 FingerMovements 58.
49.20 ± 1.
14.32 HandMovementDirection 36.
37.30 ± 2.
113.22 Handwriting 28.
33.65 ± 0.
43.78 Heartbeat 66.
78.34 ± 0.
117.00 InsectWingbeat 16.
38.62 ± 10.
7.07 JapaneseVowels 97.
98.86 ± 0.
8.53 LSST 37.
60.26 ± 4.
10.66 Libras 85.
96.44 ± 0.
13.22 MotorImagery 51.
57.40 ± 2.
867.23 NATOPS 88.
97.78 ± 0.
14.72 PEMS-SF 69.
80.12 ± 1.
57.15 PenDigits 97.
98.73 ± 0.
2.62 PhonemeSpectra 11.
31.97 ± 0.
62.77 RacketSports 80.
86.18 ± 1.
8.61 SelfRegulationSCP 87.
71.74 ± 0.
252.93 SelfRegulationSCP 47.
46.44 ± 4.
325.32 SpokenArabicDigits 99.
99.10 ± 0.
23.85 StandWalkJump 6.
38.67 ± 4.
705.04 UWaveGestureLibrary 89.
91.56 ± 0.