Dynamic Sparse Network for Time Series Classification: Learning What to “See”
Anonymous Author(s)
AffiliationAddress email
Abstract
The receptive field (RF), which determines the length of time series to be "seen"and used by a unit of feature extraction, is critical to improve the performance fortime series classification (TSC).
However, the variation of signal scales across andwithin time series data, makes it challenging to decide proper RF sizes for TSC.
Inthis paper, we propose a dynamic sparse network (DSN) with sparse connections for
TSC, which can learn to cover various RF without cumbersome hyper-parameterstuning.
The kernels in each sparse layer are sparse and can be explored under theconstraint regions by dynamic sparse training, which makes it possible to reducethe resource cost.
The experiment results show that the proposed DSN modelcan achieve the state-of-art performance on both univariate and multivariate TSCdatasets with less than 50% computational cost compared with recently baselinemethods, opening the path towards more accurate resource-aware methods for timeseries analyses.
Our code is provided in the supplementary material, and it will bemade available online.
Introduction
Time series classification (TSC) as an important research topic in the data mining communities [25],has a wide range of applications from health care [26, 13] and social security [58], to grid energy[41] and remote sensing [43].
In the last decade, neural networks, especially deep neural networks,have achieved competitive or even better performance than traditional TSC approaches (e g. DTW[46], BOSS [47], COTE [1]) in many cases [56, 25].
However, how to discover and exploit the various scaled signals hidden in time series (TS) isstill a significant challenge for TSC tasks [52, 38, 20].
One main reason lies in several variances,like sampling rate and record length, which are natural during time-series collection [47, 33, 5].
Furthermore, the amplitude offset, warping and occlusion of data points are unavoidable (see Figure 1).
So determining the optimal scales for feature extraction is difficult but significant in the TSC task.
One main solution is to cover as many receptive field (RF) scales as possible, so it does not ignoreany useful signals from the time series inputs [52, 6, 7, 27].
Grounding on the motivation that multiple kernels in combination can capture discriminative patternsdespite complex warping, Fawaz et al.
[12] aimed to ensemble convolutional neural networks witha large number and variety of kernels.
Tang et al.
[53] have stacked several OS-CNN blocks, eachof which consisted of a list of kernel sizes to cover all scales of RF and achieve better performanceacross several TSC tasks.
In Multi-scale Convolutional Neural Network (MCNN) [4], a grid searchwas applied to find suitable kernel sizes.
Without elaborate setting and fine-grid search, Dempster etal.
[6] transformed time series by using random convolutional kernels and then trained a time seriesclassifier.
Even though the use of kernels with various sizes can help to extract latent hierarchicalfeatures of multiple resolutions, it tends to cause a huge computational problem and overfits the datasets when a limited amount of training data is available [27].
To address this problem, dilatedconvolution [60] has been used to reduce model size while keeping the receptive field size.
Franceschiet al.
[14] presented an unsupervised learning model with convolutional kernels for time series featuretransformation, in which the dilation factors of kernels increased exponentially layer by layer.
To date,there is still an unresolved challenge with hyperparameter selections (e.g. kernel sizes and dilatedfactors) to capture patterns at different scales.
This raises a foundational question: Can we designa scalable method for TSC that achieves a trade-off between the computation and performancewithout cumbersome hyperparameter tuning?
[Figure]
Inspired by sparse neural network models which can match the performance of dense counterpartswith fewer connections [40, 36, 34, 61], we propose a dynamic sparse network (DSN) that is composedof CNN layers with large but dynamic sparse kernel for TSC tasks, which can automatically learnsparse connections in terms of covering various RF.
The proposed DSN is trained by a dynamic sparsetraining (DST) strategy, which makes it possible to reduce the computational cost (e.g. floating-pointoperations (FLOPs)) for training and inference as well.
Different from the traditional dynamic sparsetraining methods, which explore the connections in a layer-wise manner [40, 35, 11] and can hardlycover smaller RF, we propose a more fine-grained sparse training strategy for TSC.
Specifically, the
CNN kernels in each layer are split into several groups, each of which is sparse and can be exploredunder a constraint region during the course of sparse training.
In this way, each layer can extractfeatures in a more scalable and diverse way, and then by cascading, the model can cover more variousscales of RF.
To sum up, our main contributions are:
• We propose a novel dynamic sparse CNN layer with various effective RF, which can learn
to “see” selectively and adaptively for more accurate TSC.
• By stacking the above dynamic sparse CNN layers, we introduce a novel DSN model
natively for TSC, which is trained by DST to reduce computational and memory cost.
• To bridge gap between resource awareness and accuracy, we propose a more fine-grainedtraining strategy for DSN, which can more easily cover diverse effective RF for TSC.•
We perform extensive empirical evaluation on both univariate and multivariate TSC tasks,showing that DSN achieves superior performance in terms of accuracy and resource cost.
Related Work2.1 Time Series Classification
In the last decade, the success of deep learning encourages researchers to explore and extend itsapplication on TSC tasks [30, 25, 56, 23, 31].
For univariate TSC, deep learning-based modelsattempt to directly transform raw time series data into low-dimensional feature representation viaconvolutional or recurrent networks [27, 45, 42].
For multivariate TSC, LSTM and attention layersare often stacked with CNN layers to extract complementary features from TS data [28, 62].
Recently, many works attempt to extract features in a more scalable manner as TS data is naturallycomposed of multiple signals on different scales [47, 5, 52].
One main solution is to configure suitable kernels of various sizes such that they increase the probability to capture the proper scales[52, 53, 27].
Rocket-based methods [6, 7, 8] aim to use random kernels with several sizes and dilationfactors to cover diverse RF for univariate TSC.
Different from these works, our proposed dynamicsparse CNN layer can adaptively learn the various RF with deformable dilation factors and trade-offboth computation and performance without cumbersome hyperparameter tuning.
2.2 Sparse Training
Recently, the lottery ticket hypothesis has shown the possibility of training a sparse sub-network tomatch the performance of the dense network with less computational cost [15].
Rather than iterativelypruning from a dense network [17, 63, 15], recent works try to find an initial mask with one-shotpruning based on gradient information during training [32, 55].
After pruning, the topology of theneural network will be fixed during training.
However, this kind of models can hardly match theaccuracy achieved by their dense counterparts [39, 55, 51].
Introduced as a new training paradigm before the lottery ticket hypothesis, DST starts from a sparseneural network and allows the sparse connectivity to evolve dynamically with a fixed number ofparameters throughout the training [40, 11, 36, 61].
Nowadays, DST attracts increasing attentionsfrom other research fields such as reinforcement learning [49] and continual learning [48], due to itspotential of outperforming dense neural networks training [40, 61, 36].
Different from the traditional
DST methods, our proposed DSN is trained with a fine-grained sparse training strategy rather than atraditional layer-wise manner to capture more diverse RF for TSC.
2.3 Adaptive Receptive Field
The RF, which can be adaptively changed during the training, has proven to be effective in manydomains [54, 44, 57].
The adaptive RF can be usually captured by learning the optimal kernel sizes orthe masks of kernels during the training stage [50, 18, 57].
However, neither the kernels nor the masksare sparse, which may cause the huge computational problem when larger RF needed.
Different fromthese methods, our dynamic sparse CNN layers in DSN can be trained with DST, which can learn tocapture variable RF with deformable dilation factors.
What’s more, the kernels are always sparseduring training and inference, which makes it possible to reduce the computational cost.
The Proposed Model
3.1 Problem Definition
Definition 1.
( Time Series Classification (TSC) ).
Given a TS instance X
= { X 1 , . .
X n } ∈ R n × m ,where m denotes the number of variates and n denotes the number of time steps, TSC aims toaccurately predict the class label y ∈ { 1 , . .
. , c } from c classes.
When m equals 1, TSC is univariateand otherwise it is multivariate.
Definition 2.
( Time Series (TS) Training Set ).
A training set D
= { (cid:0) X (1) , y (1) (cid:1) , . . .
,(cid:0) X ( N ) , y ( N ) (cid:1) } consists of N time series instances,
where X ( i ) ∈ R n × m could either be a univariate or multivariate time series instance with its corresponding label y
( i ) ∈ { 1 , . .
, c } .
Note that, in our case, all instances have the same number of time steps in a TS dataset.
Without theloss of generality, given a training set, we aim to train a CNN classifier with adaptive and various RFwith a low resource cost (e.g. memory and computation) for the TSC task.
3.2 Dynamic Sparse CNN Layer with Adaptive Receptive Field
A straightforward strategy to cover various RF is to apply multi-sized kernels in each CNN layer, butit has several limitations.
Firstly, TS instances from different TS datasets do not have the same lengthand cycles 1 in most cases, making it difficult to set a fixed kernel configuration for all the datasetseven with prior knowledge.
Secondly, obtaining a large receptive field commonly requires a largekernel or stacking more layers, which introduces more parameters and thus increases storage andcomputational costs.
To tackle those challenges, the proposed dynamic sparse CNN layer possesses large but sparse kernels,which are learnable to capture adaptive RF.
Specifically, given the input feature map x l ∈ R c l − 1 × h × win the l th layer ( h equals 1 for univariate TSC and c l − 1 is the number of input channels) , andkernel weights Θ l ∈ R c l − 1 × c l × 1 × k ( k is the kernel size and c l is the number of output channels), theconvolution with stride 1 and padding in our proposed dynamic sparse CNN layer is formulated as
[Equation]
where O j ∈ R h × w denotes the output feature representation at the j -th output channel, Z denotesthe set of integers, I l ( · ) : R c l − 1 × c l × 1 × k
→ { 0 , 1 } c l − 1 × c l × 1 × k is an indicator function,
I l
(Θ l ) i,jindicates activated weights for Θ li,j ∈ R 1 × k
which is the kernel in the ( i, j ) -th channel, ⊙ denotesthe element-wise product, and · denotes the convolution operator.
The indicator function I l ( · ) , whichis learned during the training of the proposed DSN, satisfies that ∥
I l ( · ) ∥ 0 ≤ (1 − S ) N l where0 ≤ S < 1 is the sparsity ratio, ∥ .
∥ 0 denotes the L 0 norm and N l = c l − 1 ∗ c l ∗ 1 ∗ k .
When S > 0 ,the kernel is sparse, and we can use a large k in the dynamic sparse CNN layer to obtain a large RFwith reduced computational cost.
Remark 1.
( Effective Neighbour Receptive Field (eNRF) size in the dynamic sparse CNN layer ).
The receptive field is defined as the region in the input space that the feature of a CNN model islooking at.
We defined that Neighbour Receptive Field (NRF) is the feature region that used forgenerating every pixel feature of the successive layer.
Thus, the NRF size is equivalent to the kernelsize in the standard CNN layer (Considering the case that stride equals to 1).
Differently, the NRFsize of the proposed dynamic sparse CNN layer is smaller than the kernel size, when the first orlast weight in a kernel is not activated, e.g., ∃ i ∈ { 1 , . . .
, c l − 1 } and j ∈ { 1 , . .
c l } , such that
I l
(Θ l ) i,j, 1 , 1 = 0 or I l (Θ l ) i,j, 1 ,k = 0 .
As illustrated in Figure 2, we defined the eNRF size f li,j of thekernel Θ li,j as the distance between the first activated weight and last activated weight in the l -th
CNN layer as
[Equation]
otherwise
[Equation]
where Ind li,j denotes the set of indices corresponding to non-zero weights in the kernel Θ li,j .
Obviouslywe have 0 ≤ f li,j ≤ k .......
=
= =
= =
= =
= 3 =
The eNRF size set of l -th dynamic sparse CNN layeris denoted by F ( l ) , which satisfies that 0 ≤ min( F ( l ) )and max( F ( l ) ) ≤ k .
Take the case in the Figure 2as an simple example for l -th layer, then F ( l ) ={ 0 , 1 , 2 , 3 , 4 , 5 , 6 } .
Each dynamic sparse CNN layeris likely to cover various eRF sizes ranging from 1 tok .
When global information is expected, I l ( · ) couldactivate weights more dispersed to get a larger eNRFand exploit the input features selectively.
To capturethe local context, a smaller eNRF is expected, thusthe activated weights tend to be concentrated.
Bytaking k
= 5 as an example, I l (Θ l ) i,j may be [1, 0, 1, 0, 1] for global context, while it may be [0, 0,1, 1, 0] for local context.
In this way, the eNRF can be adaptively adjusted.
3.3 Architecture in DSN
The proposed DSN model consists of three sparse CNN modules, each of which composes of adynamic sparse CNN layer and a 1 × 1 CNN layer.
Following the stacked sparse CNN modules,there is an additional dynamic sparse CNN layer, two adaptive average pooling layers, and one 1 × 1convolution layer acting as the classifier in the DSN model.
The overall architecture is shown in
Figure 3.
The set of eNRF sizes S ( l ) in the l th sparse CNN module is equal to that in the dynamic sparse
CNN layer, since the eNRF size of 1 × 1 convolution is constantly equal to 1 .
S ( l ) satisfies that
R eL U R eL U
A dap t i v e A v g P oo l 1d
R eL U
C on v 1 x
B N
A dap t i v e A v g P oo l 1d
S pa r s e C
N N
M odu l e
S pa r s e C
N N
M odu l e
S pa r s e
CNN La y e r
S pa r s e
CNN La y e r
C on v 1 x
Sparse CNN Module
R eL U S pa r s e
C N N
M odu l e
0 ≤ min( S ( l ) ) and max( S ( l ) ) ≤ k .
Then, the set of eNRF sizes for the three successively stackedsparse CNN modules can be described by RF as
[Equation]
[Figure]
According to Eq.
(3), we can see that stacking multiple sparse CNN modules can increase the size of 168 RF linearly, where each dynamic sparse CNN layer increases by the size of S ( l ) .
For simplicity, thekernel size in each dynamic sparse CNN layer is consistently set to k in our study.
Then, RF satisfiesthat max( RF ) ≤ 3 k − 2 and 0 ≤ min( RF ) .
Thus, a large k in sparse CNN modules can increase therange of the eRF sizes for stacked sparse CNN modules.
3.4 Dynamic Sparse Training for DSN Model
Input: Dataset D , Network f Θ , Sparsity ratio S Exploration regions in layer l : {R l 1 , ..., R lN }
Exploration schedule: T , ∆ T , α , f decay 1: θ li ← initialize activated weights in Θ li using S and R li 2: for t = 1 to T do 3:
Sample a batch B t ∼ D L t = (cid:80) i ∼ B t L (( f θ ( x i ) , y i ) 4: if ( t mod ∆ T ) =
=
then 5: for each layer l = 1 to L do 6: for each group i = 1 to N do 7: u = n ( R li )
f decay ( t ; α, T ) (1 − S ) 8:
I prune = ArgTopK (cid:0) − (cid:12)(cid:12)
θ li (cid:12)(cid:12) , u (cid:1) 9:
I grow =
RandomK (cid:0) R li \ θ li , u (cid:1) 10:
I l ( . )
← Update I l ( . )
using I prune
and I grow 11: θ l i ← Update activated weights
I l (Θ l ) i ⊙ Θ l i 12: end for 13: end for 14: else θ = θ − α ∇ θ L t 15: end if 16: end for
In this section, we present the trainingstrategy to discovery the weights requiredto be activated in order to ensure wellperforming eNRF.
That is, we need to studyhow to update the indicator function
I l ( · )during the training of the proposed DSN.
We sparsely train the proposed DSN modelfrom scratch to keep kernel sparse following the main idea of DST methods [40].
Bydesign, during the training phase, the total number of activated weights must notexceed N l (1 − S ) .
However, we observethat the tiny eNRF is hard to capture by
DST methods with a layer-wise explorationmanner, namely activated weights are discovered layer by layer, especially whenthe sparsity ratio S is undersized (moreanalysis in Section 4.5).
Motivated by thisobservation, the kernels in each dynamicsparse CNN layer are divided into differentgroups, whose corresponding exploration regions are of different sizes as shown in
Figure 4.
Contrary to the DST methods, in the DSN the exploration of activated weights is separatelyperformed in different kernel groups, which is a more fine-grained strategy.
Specifically, the kernelweights Θ l ∈ R c l − 1 × c l × 1 × k in the l th layer are split into N groups along the output channel, that is,Θ l 1 , . . .
, Θ lN ∈ R c
l − 1 × clN × 1 × k with corresponding exploration regions R l 1 , . . .
, R lN .
The explorationregion R li in the i th group of the l th layer is defined as the first i × kN positions of each kernel in thisgroup.
By taking k = 6 and N = 3 for example, activated weights in the first group are only withinthe first two positions of each kernel in Θ l 1 , while they are within the whole kernel for the last group(as shown in Figure 4).
In this way, the various eNRF can be covered, and exploration space can bereduced (more details can be found in Appendix A.1) to improve exploration efficiency.
Given the weight exploration regions R l 1 , . . .
, R lN and the sparsity ratio S , we train
the proposed
DSN model as shown in Algorithm 1.
The activated weights are explored within the explorationregions and updated every ∆ t iterations.
The fraction of updated weight decays over time accordingthe function f decay ( t ; α, T ) , which follows the cosine annealing [9] as
[Equation]
where α is the initial fraction of activated weights updated, t is the current training iteration, and Tis the number of training iterations.
Thus, during the t th iteration, the number of updated activatedweights in the i th group of the l th layer is n ( R li ) f decay ( t ; α, T ) (1 − S ) , where n ( R li ) is the numberof weights can be explored in region R li .
During the update of activated weights, we firstly prunethe activated weights determined by ArgTopK (cid:0) − (cid:12)(cid:12) θ li (cid:12)(cid:12) , u (cid:1) and then randomly grow new activatedweights by RandomK (cid:0) R li \ θ li , u (cid:1) , where ArgTopK( v , u ) gives the indices of top u elements withlarger values in a vector v , RandomK( v , u ) outputs the indices of random u elements in a vector v ,and R li \ θ li denotes the weights within R li but except θ li .
The exploration for activated weights is straightforward.
Firstly, pruning weights with small magnitudes is intuitive, because the contribution of weights with smaller magnitudes is insignificant oreven negligible.
Considering the pruning recoverability, we randomly regrow new weights with thesame number as the pruned weights to achieve better activated weight exploration.
In this way, theexploration of weights is dynamic and of plasticity.
Experiments4.1 Datasets and Baselines
We evaluate the proposed method on both univariate and multivariate TSC datasets.
The details ofeach kind of dataset are as follows.
• Univariate TS dataset from UCR 85 archive [3] : This archive consists of 85 univariate TSdatasets, which are collected from various domains (e
g. health care and remote sensing),have distinguishable characteristics, and various levels of complexity.
The number of
instances in the training set varies from 16 to 8926, while their time step resolutions rangefrom 24 to 2709.
• Three multivariate TS datasets from UCI [10] : The EEG2 dataset contains 1200 instancesof 2 categories and 64 variates.
The Human Activity Recognition (HAR) dataset consists of10,299 instances with 6 categories, and the number of variates is 9.
The Daily Sport datasetcontains 9,120 instances of 19 categories, and its number of variates is 45.
The processingof these dataset is the same as in [28].
Univariate TSC baselines: The performance of the following three baseline methods are reported,
OS-CNN [52], InceptionTime [27], and ResNet[56], as they are widely used for univariate TSC.
Multivariate TSC baselines: Similar with [62], the following three multivariate baselines areselected: OS-CNN, TapNet [62], MLSTM-FCN [28].
To avoid an unfair comparison, we let outsidethe ensemble methods.
However, based on our best knowledge, our performance analysis accountsfor most of state-of-the-art methods.
4.2 Experimental Settings and Implementation Details
For optimization, we use the Adam optimizer with an initial learning rate of 3 × 10 − 4 , cosine decayedto 10 − 4 .
Our model is trained for 1,000 epochs with a mini-batch size of 16.
As in [27, 28], the bestmodel, which corresponds to the minimum training loss, is used to evaluate performance over thetesting sets.
Inspired by [27], the kernel sizes k in each dynamic sparse CNN layer are set to 39.
Thenumber of kernel groups N in Algorithm 1 is set to 3, which helps to cover the small, medium andlarge eRF.
Each setting is repeated for five times and the average results are reported.
4.3 Results and Analysis
We show the performance on both univariate and multivariate TSC benchmarks in Table 1-2.
Forunivariate TSC, we can see that our method outperforms the baseline methods in most cases with asmaller number of parameters (e.g. Params 2 ) and less computation cost (e.g. FLOPs) on UCR 85archive datasets.
For multivariate TSC, our proposed DSN method achieves better performance on
EEG2 and HAR datasets, while for Daily Sport, MLSTM-FCN was with 0.46% more accurate.
Itis worth highlighting that the resource cost (i.e. Params and FLOPs) of our proposed DSN is muchsmaller than that of baseline methods.
Due to the page limitation, the detailed results including testaccuracy and resource cost on each dataset of UCR can be found in Appendix C.
[Table] [Table]
4.4 Sensitivity Analysis
Effect of Sparsity Ratio: From Table
[Table]
we can find out how the sparsity ratio of dynamic sparse
CNN layer affects the final test accuracy and the resource cost in multivariate TS datasets.
Here,we analyze the trade-off between test accuracy and the resource cost under various sparsity ratios(i
[Table]
e. S ∈ [50% , 80% , 90%] ) and a dense DSN model.
Note that the dense model is defined exactlyas our DSN model with dense connections along entire kernels.
We can see that it is not that themore parameters of the model, the better performance it will achieve.
This is mainly because moreparameters will lead to overfitting and the proper sparse ratio will easily cover the desired receptivefield (more demonstrations can be seen in Appendix B.1).
Taking EEG2 data set as an example,with the decrease of sparse rate, the performance on it drops sharply.
We hypothesise that this dataset has more expectations for features from small receptive fields (the validation can be found in
Appendix B.2).
According to the experiments, a 80% sparsity ratio reflects a good trade-off betweenaccuracy and resource cost, which is the default setting for the dynamic sparse CNN layer in our
DSN model.
[Table]
Effect of Architecture: To show how the architecture of DSN affects the final performance, weadopt the critical difference diagram for a detailed comparison, which is usually used for evalution in
TSC [52, 27].
From Figure 5, we can find that the DSN model with 4 dynamic sparse CNN layersand 177 output channels in each layer performs consistently better than the other structures.
Theproposed model with more output channels seems to be crucial for preserving the capacity of thenetwork, while requires more parameters and FLOPs compared to other counterparts.
This highlightsan interesting trade-off between accuracy and computational efficiency.
According to the experiments,the DSN model with 4 dynamic sparse CNN layers and 141 output channels in each layer is thetrade-off between the performance and resource cost, which is the default setting for our DSN model.
better performance of the model.
[Figure]
4.5 Ablation Study
[Table]
Effect of Kernel Group:
As shownin Figure 6, training DSN withoutkernel group hampers the capture ofsmall eNRF, which can not guarantee satisfactory performance on thedatasets such as EEG2 expecting morelocal information.
With the decreasein sparsity ratio S , the phenomenonthat large RF occupies the majoritybecomes more severe (more demonstrations see in Appendix B.1), which increases the performance gap between training with group and without it as shown in Table 4.
Grouping the kernels can achieve various RF coverage and adorable accuracy for all datasets.
Effect of Dynamic Sparse Training:
Furthermore, we study the effect of the dynamic sparse trainingin our DSN method.
We conduct ablation studies to compare DSN with its static variant (i.e. DSN fix ),which fixes its topology during training after activated weights initialization.
In other words, theindicator function
I l ( · ) in Eq.
(1) will not update during the training process.
According to the resultsshown in Figure 7 (a), we can find that the DSN model mostly outperforms the DSN fix on UCR 85archive, indicating that what are learned by dynamic sparse training is the suitable activated weightstogether with their values.
[Figure]
4.6 Case Study on other DST Method
[Figure]
Likewise, the effectiveness of the proposed DSN method is also analyzedin the presence of other DST methods,such as RigL [11].
Rather than growing weights randomly (as in the line9 of Algorithm 1), RigL grows themwith the highest magnitude gradients.
From Figure 7 (c), we can see thatgrowing weights according to gradients (i.e. DSN grad ) can outperformsthe OS-CNN method in most cases on
UCR 85 archive.
Furthermore, as shown in Figure 7 (b), we can find that DSN grad has a similarperformance as growing weights randomly (i.e. DSN rand ).
Note that, DSN rand is the main settingof the proposed method in this paper with the reason that we don’t need to compute the full gradientsat any moment in time.
In summary, the proposed DSN with fine-grained sparse strategy can beeasily combined with the existing DST techniques, showing the potential for further improvementsby merging it with other advanced DST methods in the future.
4.7 Visualization of Activated Weights
Figure 8 shows the normalized activated weights of the DSN model after sparse training on HARdataset.
(a), (b) and (c) correspond to the activated weights of the three kernel groups in the firstdynamic sparse CNN layer, and each row of the figure represents one kernel.
We can see thatthe weights in each kernel are sparse, and can be activated within the constraint region defined in
Section 3.4.
Conclusions
This paper proposes dynamic sparse network (DSN) to cover diverse effective receptive field (eRF)for time series classification (TSC) without the cumbersome hyper-parameters tuning.
The proposed
DSN can achieve about 2 × reduction in memory and computational cost, while achieving superiorperformance in terms of accuracy on both univariate and multivariate TSC datasets.
Furthermore,
DSN explores the activated weights in a more fine-grained strategy, which can be easily merged with existing DST techniques, showing a potential for further improvements when combined withadvanced technology.
DSN provides a feasible solution to bridge gap between resource awareness andvarious eRF coverage for TSC, hoping to inspire other researches in different fields.
It is worthwhileextending DSN to other research domains (e g. time series forecasting and computer vision).
Our
DSN model has not yet been executed on real edge devices and the performance in terms of accuracyand memory footprint on edge devices is not investigated.
In the future, we hope to extend our workto other fields and decentralise training on edge devices
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix D
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets..
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]
A Exploration Spaces and Calculation of Model ParametersA.1 The Comparison of Exploration Space
The size of exploration space for the layer-wise manner in SET algorithm is:
[Equation]
If we divide the kernels in one layer into N groups and explore N l (1 − S ) N activated weights in eachgroup, then the size of the exploration space becomes:
[Equation]
When the corresponding exploration region of each group is defined as in Section 3.4, the explorationspace can be further reduced to:
[Equation]
A.2 The Calculation of Parameters
The summation of the number of non-zero weights (activated weights in sparse CNN layer + weightsin other layers) in the network is used to estimate the size of network as follows:
[Equation]
where L contains a set of sparse CNN layer indexes, and θ is the parameter of our DSN whichincludes an amounts of zero values.
∥
I l (Θ l ) i ⊙ Θ li ∥ 0 is controlled by the sparsity ratio S and thecorresponding exploration regions R li .
B More ExperimentsB.1 The Statistics of eRF Sizes with Different Sparsity Ratios
The distributions of eRF sizes in the first dynamic sparse CNN layer for HAR, Daily Sport and EEG2datasets are shown in Figure 9, Figure 10 and Figure 11, respectively.
We can see that DSN modelwith kernel grouping can cover more diverse eRF sizes than it does without kernel grouping underdifferent sparsity ratios S .
With the decrease of S , the DSN model without kernel grouping tends tocapture the large eRF, while ignoring the small eRF, which harms the performance of DSN model.
Under different sparsity ratios, the DSN model with kernel grouping can guarantee a satisfactoryperformance for different datasets by covering various eRF.
B.2 Better Kernel Sizes for EEG2 Dataset
We study the effect of kernel size k on the performance of spares/dense DSN for EEG2 dataset toverify our hypothesis that this dataset expects the local information.
The dense DSN model is definedexactly as our DSN model with dense connections along entire kernels.
In this way, the RF sizein each layer equals to kernel size.
As shown in Figure 12, the dense DSN with a kernel size of9 achieves the highest test accuracy.
However, with the increase of kernel size (RF size), the testaccuracy drops dramatically.
It is obvious that the local context plays a significant role for moreaccurate classification for EEG2 dataset.
In contrast, our proposed sparse DSN model (sparsity ratio
S = 80% ) with a large kernel size can still achieve satisfactory performance, with the main reasonthat it can learn the adaptive eRF for the dataset.
B.3 Can Various eRF Outperform the Optimal RF?
To demonstrate the advantage of various eRF coverage, we compare the performances in terms of testaccuracy between DSN models with various eRF coverage and optimal RF.
The dense DSN modelswith different kernel sizes (from 5 to 40 with step 5) in each layer are used to search for optimal RFin different datasets on the UCR 85 archive.
As shown in Figure 13, by capturing various eRF,
our
DSN model can achieve similar performance to the optimal RF model without the cumbersome hyperparameters.
[Figure]
B.4 The Effect of Kernel Size for DSN
From Figure 14, we can find out how the kernel size k in the proposed DSN model affects the finaltest accuracy for UCR 85 archive.
We can see that the performance of DSN model with small kernelsizes (e.g. 15 and 21) is worse than that of it with large kernel sizes (e.g. 39 and 45).
It is because thatthe model with larger kernel sizes can cover larger eRF which is useful to capture global informationfor some datasets.
However, when the kernel size is oversized (e.g. 45), some of the small eRFs canbe lost in a predefined sparsity ratio (e.g. 80% in here), which will slightly degrade the performanceinstead.
B.5 The Effect of Kernel Group for DSN
We study how the number of kernel groups N defined in Algorithm 1 affects the performance ofproposed DSN model.
From Figure 15, we can see that DSN model with N = 2 outperforms othercounterparts.
As discussed in 4.5 and B.1, grouping the kernels in each dynamic sparse CNN layercan help to achieve various RF coverage especially with smaller sparsity ratio S .
However, when thenumber of kernel groups (e.g. 4) is oversized, the distribution of eRF sizes is nearly uniform, whichwill degrade the performance of DSN.
B.6 The Effect of Topology Initialization
As shown in Figure 16, the activated weights can be initialized randomly and sequentially beforeexploration, namely DSN initrand and DSN initseq respectively.
After initialization, the topologies initializedby DSN initrand and DSN initseq are different.
In detail, DSN initrand randomly activates the weights within theexploration regions, while DSN initseq activates the first i × kN S weights of kernels in i th group.
Figure 16exemplifies
the case where k = 16 , N = 4 and S = 50% .
We further investigate which type oftopology initialization are more suitable for the proposed DSN model.
According to the resultsshown in Figure 17 (left) and Figure 18, we can see that the proposed DSN model with sequentialinitialization performs better than that with random initialization.
Based on this, we adopt sequentialinitialization as the default setting for DSN model in all of the experiments.
Intuitively, the DSN method with random topology can also cover various eRF.
We further study thedifferent performances of RF which is from dynamic sparse training and from random topology in
DSN method.
We conduct ablation studies to compare DSN with its static variant (i.e. DSN fixrand ),which fixes its topology during training after activated weights random initialization.
According tothe results shown in Figure 17 (right), we can find that the DSN model outperforms the DSN fixrand on
UCR 85 archive, indicating that what is learned by dynamic sparse training is the suitable activatedweights together with their values.
[Figure]
is better here is better here is better here is better here init init init
[Figure]
A cc u r a cy init init
B.7 Extremely Sparse DSN
[Figure]
We study how the performance of our DSN is affected under extremely sparse ratios (e.g. S = 90% ,
S = 95% and S = 97% ).
As shown in Table 5, we can see that our DSN with sparsity ratio
S = 95% still performs better than OS-CNN while achieving more than 5 × resource cost reduction(e.g.
Parameters and FLOPs).
When sparsity ratio S = 97% , which means that 97% weights insparse CNN layer are set to zeros, the performance of our DSN model is only slightly worse than thatof OS-CNN, but the resource cost is nearly 7 × less.
From Figure 19, we can see that most blue dots(corresponding to accuracies of OS-CNN) are below the orange dots (corresponding to accuraciesof DSN with extremely sparsity ratio), which means that OS-CNN performs worse than extremelysparse DSN in most dataset.
What’s more, DSN is far superior to OS-CNN in some datasets (e.g.
CinCECGtorso, MiddlePhalanxOutlineAgeGroup and Earthquakes).
[Table]
DSN( S = 80% )   126.
DSN( S = 85% )   100. DSN( S = 90% )   74.
DSN( S =
[Table]
DSN( S = 97% )   37.
[Figure]
[Figure]
Figure 20 shows the learning behaviors of our DSN model and OS-CNN.
We can see that our DSNmodel has a convergence speed similar to that of OS-CNN, while the resource requirements (memoryand computational cost) are much less.
Furthermore, the performance of DSN is far superior to thatof OS-CNN for EEG2 datasets .
B.9 Additional Experiments for Multivariate TSC
[Figure]
In order to understand if the good learning capabilities of DSN can be generalized over a larger rangeof multivariate time-series datasets we evaluate it on the following datasets:
• University of East Anglia (UEA) 30 archive [2].
It consists of 30 multivariate TS datasetswith distinguishable characteristics, and various levels of complexity.
The class numbersvary from 2 to 39, and their numbers of variates vary from 2 to 963.
We summarize the performance on UEA 30 archive in Table 6.
We can see that, in terms of testaccuracy, our method outperforms the baseline methods in most cases, while for OS-CNN, ourperformance can still mostly match with more than 2 × resource cost reduction (e.g. Parameters and
FLOPs).
It is worth highlighting our proposed DSN method achieves a trade-off between the resourcecost and performance.
The detailed results including test accuracy and resource cost on each datasetof UEA 30 archive can be found in Table 8.
[Table]
C Detailed Results
For UCR 85 archive, the reported results of ResNet and InceptionTime are from [6], while that of
OS-CNN are from the official repository 3 .
For three multivariate datasets from UCI, the reportedresults of LSTM-FCN are from the original paper, and that of TapNet and OS-CNN are from theimplementation of official repository with default settings.
The detailed results for UCR 85 archive are given in Table 7.
We also visualize the detailed results in
Figure 21, we can see that the accuracies of DSN are nearly on the top in most of the datasets.
Inaddition, it is worth noting that DSN loses by a slight margin in some datasets, but wins by a bigmargin in others.
[Figure]
D More Implementation Details
In our dynamic sparse training algorithm 1, we choose the cosine annealing defined in Eq.
4 with∆ T = 5 epochs and α = 0 .
We use PyTorch 4 to implement our method and run our experimentson Nvidia Tesla P100. E Preliminaries on Existing Compression Solutions
The existing compression solutions can be categorized into after-training, during-
training and before- training methods. • After-training methods obtain a sparse or small neural network from a pre-trained network by pruning the unimportant weights [19], quantifying [59] the weights or distillation [21].
• During-training methods , which can be divided into dense-to-sparse training strategyand sparse-to-sparse training strategy, discover the sparse network during training without 3 https://github.com/Wensi-Tang/OS-CNN.https //pytorch.org pre-trained network.
Dense-to-sparse training strategy sparsifies the neural network duringthe training process through gradual magnitude pruning [16] or regularization techniques[37].
Sparse-to-sparse training strategy initializes a random (topology and weights) sparsenetwork, and dynamically optimizes the topology as well as the weights simultaneouslyduring training [40, 11].
Our DSN model is based on the sparse-to-sparse strategy.•
Before-training methods aim to obtain a trainable small or sparse neural network beforethe main training process.
Squeezenet [24] and MobileNets [22] are specifically engineered
[Table]
dense networks that are smaller than standard architectures and can run with limited
re-
[Table]
sources on the edge device.
Motivated by SNIP [32], many works [51, 55] have shown thatpruning at initialization without optimizing weights performs equally well.
[Table] [Table]
GunPoint
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
24 Ham
[Table] [Table]
[Table] [Table]
[Table] [Table]
HandOutlines 91.
[Table]
[Table] [Table]
[Table] [Table]
54 Haptics
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
82 Herring
[Table] [Table]
[Table] [Table]
[Table] [Table]
97 InlineSkate
[Table]
[Table] [Table]
[Table]
69 InsectWingbeatSound
[Table]
[Table] [Table]
[Table] [Table]
[Table]
77 ItalyPowerDemand
[Table] [Table]
01 LargeKitchenAppliances
[Table] [Table]
[Table] [Table]
56 Lighting
[Table] [Table]
[Table] [Table]
[Table] [Table]
95 Lighting
[Table] [Table]
[Table] [Table]
[Table] [Table]
27 MALLAT
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
05 Meat 96.
[Table] [Table]
[Table] [Table]
[Table] [Table]
13 MedicalImages
[Table] [Table]
[Table]
24.82 MiddlePhalanxOutlineAgeGroup
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
92 MiddlePhalanxOutlineCorrect
[Table]
81.33 ± 1.
[Table]
19.89 MiddlePhalanxTW 48.
61.45 ± 0.
20.00 MoteStrain 92.
90.73 ± 0.
20.88 NonInvasiveFatalECGThorax 94.
94.36 ± 0.
187.11 NonInvasiveFatalECGThorax 94.
94.72 ± 0.
187.11 OSULeaf 97.
98.84 ± 0.
106.01 OliveOil 83.
94.00 ± 3.
141.40 PhalangesOutlinesCorrect 83.
83.87 ± 0.
19.89 Phoneme 33.
33.70 ± 0.
254.93 Plane 100.
100.00 ± 0.
35.89 ProximalPhalanxOutlineAgeGroup 85.
86.05 ± 0.
19.92 ProximalPhalanxOutlineCorrect 92.
93.20 ± 0.
19.89 ProximalPhalanxTW 78.
81.95 ± 0.
20.00 RefrigerationDevices 52.
55.84 ± 1.
178.56 ScreenType 62.
63.95 ± 1.
178.56 ShapeletSim 77.
72.67 ± 4.
124.00 ShapesAll 92.
[Table] [Table]
[Table] [Table]
± 3.
119.04 ERing 13.
92.22 ± 1.
18.55 EigenWorms 50.
39.08 ± 11.
5075.34 Epilepsy 76.
99.86 ± 0.
58.21 EthanolConcentration 37.
24.49 ± 0.
493.68 FaceDetection 54.
63.49 ± 0.
18.58 FingerMovements 58.
49.20 ± 1.
14.32 HandMovementDirection 36.
37.30 ± 2.
113.22 Handwriting 28.
33.65 ± 0.
43.78 Heartbeat 66.
78.34 ± 0.
117.00 InsectWingbeat 16.
38.62 ± 10.
7.07 JapaneseVowels 97.
98.86 ± 0.
8.53 LSST 37.
60.26 ± 4.
10.66 Libras 85.
96.44 ± 0.
13.22 MotorImagery 51.
57.40 ± 2.
867.23 NATOPS 88.
97.78 ± 0.
14.72 PEMS-SF 69.
80.12 ± 1.
57.15 PenDigits 97.
98.73 ± 0.
2.62 PhonemeSpectra 11.
31.97 ± 0.
62.77 RacketSports 80.
86.18 ± 1.
8.61 SelfRegulationSCP 87.
71.74 ± 0.
252.93 SelfRegulationSCP 47.
46.44 ± 4.
325.32 SpokenArabicDigits 99.
99.10 ± 0.
23.85 StandWalkJump 6.
38.67 ± 4.
705.04 UWaveGestureLibrary 89.
91.56 ± 0.