Dynamic Sparse Network for Time Series Classification: Learning What to “See”
Abstract
The receptive field (RF), which determines the region of time series to be “seen” and used, is critical to improve the performance for time series classification (TSC).
However, the variation of signal scales across and within time series data, makes it challenging to decide on proper RF sizes for TSC.
In this paper, we propose a dynamic sparse network (DSN) with sparse connections for TSC, which can learn to cover various RF without cumbersome hyper-parameters tuning.
The kernels in each sparse layer are sparse and can be explored under the constraint regions by dynamic sparse training, which makes it possible to reduce the resource cost.
The experimental results show that the proposed DSN model can achieve state-of-art performance on both univariate and multivariate TSC datasets with less than 50% computational cost compared with recent baseline methods, opening the path towards more accurate resource-aware methods for time series analyses.
Our code is publicly available at: https://github.com/QiaoXiao7282/DSN .
Introduction
Time series classification (TSC) as an important research topic in the data mining communities [14], has a wide range of applications from health monitoring [24, 16] and public security [58], to grid energy [41] and remote sensing [43].
In the last decade, neural networks, especially deep neural networks, have achieved competitive or even better performance than traditional TSC approaches (e.g. DTW [46], BOSS [47], COTE [1]) in many cases [56, 14].
However, how to discover and exploit the various scaled signals hidden in time series (TS) is still a significant challenge for TSC tasks [52, 36, 21].
One main reason lies in several variances, like sampling rate and record length, which are natural during time-series collection [47, 31, 5].
Furthermore, the amplitude offset, warping and occlusion of data points are unavoidable (see Figure 1).
So determining the optimal scales for feature extraction is difficult but significant in the TSC task.
One main solution is to cover as many receptive fields (RF) scales as possible, so it does not ignore any useful signals from the time series inputs [52, 7, 8, 25].
Grounding on the motivation that multiple kernels in combination can capture discriminative patterns despite complex warping, Fawaz et al.
[15] tried to ensemble convolutional neural networks with a large number and variety of kernels.
Tang et al.
[53] have stacked several OS-CNN blocks, each of ∗ Equal contribution.
which consisted of a list of kernel sizes to cover all scales of RF and achieve better performance on several TSC tasks.
In Multi-scale Convolutional Neural Network (MCNN) [4], a grid search was applied to find suitable kernel sizes.
Without elaborate setting and fine-grid search, Dempster et al.
[7] transformed time series by using random convolutional kernels and then trained a time series classifier.
Even though the use of kernels with various sizes can help to extract latent hierarchical features of multiple resolutions, it tends to cause a huge computational problem and overfits the datasets when a limited amount of training data is available [25].
To address this problem, dilated convolution [59] has been used to reduce the computational cost while keeping the receptive field size.
Franceschi et al.
[17] presented an unsupervised learning model with convolutional kernels for time series feature transformation, in which the dilation factors of kernels increased exponentially layer by layer.
To date, there is still an unresolved challenge with hyperparameter selections (e.g. kernel sizes and dilated factors) to capture patterns at different scales.
This raises a foundational question: Can we design a scalable method for TSC that achieves a trade-off between computation and performance without cumbersome hyperparameter tuning?
[Figure]
does not need to be configured with a series of kernel sizes k .
Inspired by sparse neural network models which can match the performance of dense counterparts with fewer connections [40, 35, 33, 60], we propose a dynamic sparse network (DSN) that is composed of CNN layers with large but dynamic sparse kernel for TSC tasks, which can automatically learn sparse connections in terms of covering various RF.
The proposed DSN is trained by a dynamic sparse training (DST) strategy, which makes it possible to reduce the computational cost (e.g. floating-point operations (FLOPs)) for training and inference as well.
Different from the traditional dynamic sparse training methods, which explore the connections in a layer-wise manner [40, 34, 13] and can hardly cover smaller RF, we propose a more fine-grained sparse training strategy for TSC.
Specifically, the
CNN kernels in each layer are split into several groups, each of which is sparse and can be explored under a constraint region during sparse training.
In this way, each layer can extract features in a more scalable and diverse way, and then by cascading, the model can cover more various scales of RF.
To sum up, our main contributions are:
• We propose a novel dynamic sparse CNN layer with various effective neighbour RF, which can learn to “see” selectively and adaptively for more accurate TSC.
• By stacking the above dynamic sparse CNN layers, we introduce a novel DSN model natively for TSC, which is trained by DST to reduce computational and memory cost.
• To bridge the gap between resource awareness and accuracy, we propose a more fine-grained training strategy for DSN, which can more easily cover diverse effective RF for TSC.
• We perform extensive empirical evaluation on both univariate and multivariate TSC tasks, showing that DSN achieves superior performance in terms of accuracy and resource cost.
Related Work 2.1 Time Series Classification
In the last decade, the success of deep learning encourages researchers to explore and extend its application on TSC tasks [28, 14, 56, 23, 29].
For univariate TSC, deep learning-based models attempt to directly transform raw time series data into low-dimensional feature representation via convolutional or recurrent networks [25, 45, 42].
For multivariate TSC, LSTM and attention layers are often stacked with CNN layers to extract complementary features from TS data [26, 62].
Recently, many works have attempted to extract features in a more scalable manner as TS data is unavoidably composed of signals on various scales [47, 5, 52].
One main solution is to configure suitable kernels of various sizes such that they increase the probability of capturing the proper scales [52, 53, 25].
Rocket-based methods [7, 8, 9] aim to use random kernels with several sizes and dilation factors to cover diverse RF for TSC.
Different from these works, our proposed dynamic sparse
CNN layer can adaptively learn the various RF with deformable dilation factors and trade-off both computation and performance without cumbersome hyperparameter tuning.
2.2 Sparse Training
Recently, the lottery ticket hypothesis has shown that it is possible to train a sparse sub-network to match the performance of its dense counterpart with less computational cost [18].
Rather than iteratively pruning from a dense network [19, 63, 18], recent works try to find an initial mask with one-shot pruning based on gradient information during training [30, 55].
After pruning, the topology of the neural network will be fixed during training.
However, this kind of models can hardly match the accuracy achieved by their dense counterparts [38, 55, 51].
Introduced as a new training paradigm before the lottery ticket hypothesis, DST starts from a sparse network and allows the sparse connectivity to be evolved dynamically with a fixed number of parameters throughout training [40, 13, 35, 60, 10].
Nowadays, DST attracts increasing attentions from other research fields such as reinforcement learning [49] and continual learning [48], due to its potential of outperforming dense neural networks training [40, 60, 35].
Different from the traditional
DST methods, our proposed DSN is trained with a fine-grained sparse training strategy rather than a traditional layer-wise manner to capture more diverse RF for TSC.
2.3 Adaptive Receptive Field
The RF, which can be adaptively changed during the training, has proven to be effective in many domains [54, 44, 57].
The adaptive RF can usually be captured by learning the optimal kernel sizes or the masks of kernels during the course of training [50, 20, 57].
However, neither the kernels nor the masks are sparse, which may cause the huge computational problem when larger RF is needed.
Different from these methods, our dynamic sparse CNN layers in DSN can be trained with DST, which can learn to capture variable RF with deformable dilation factors.
What’s more, the kernels are always sparse during training and inference, which makes it possible to reduce the computational cost.
The Proposed Model 3.1 Problem Definition
Definition 1.
( Time Series Classification (TSC) ).
Given a TS instance X
= { X 1 , . .
X n } ∈ R n × m , where m denotes the number of variates and n denotes the number of time steps, TSC aims to accurately predict the class label y ∈ { 1 , . .
. , c } from c classes.
When m equals 1, TSC is univariate and otherwise it is multivariate.
Definition 2.
( Time Series (TS) Training Set ).
A training set D
= { (cid:0) X (1) , y (1) (cid:1) , . . .
, (cid:0) X ( N ) , y ( N ) (cid:1) } consists of N time series instances,
where X ( i ) ∈ R n × m could either be a univariate or multivariate time series instance with its corresponding label y
( i ) ∈ { 1 , . .
, c } .
Note that, in our case, all instances have the same number of time steps in a TS dataset.
Without the loss of generality, given a training set, we aim to train a CNN classifier with adaptive and various RF with a low resource cost (e.g. memory and computation) for the TSC task.
3.2 Dynamic Sparse CNN Layer with Adaptive Receptive Field
A straightforward strategy to cover various RF is to apply multi-sized kernels in each CNN layer, but it has several limitations.
Firstly, TS instances from different TS datasets do not have the same length and cycles 2 in most cases, making it difficult to set a fixed kernel configuration for all the datasets even with prior knowledge.
Secondly, obtaining a large receptive field commonly requires a large kernel or stacking more layers, which introduces more parameters and thus increases storage and computational costs.
To tackle those challenges, the proposed dynamic sparse CNN layer possesses large but sparse kernels, which are learnable to capture adaptive RF.
Specifically, given the input feature map x l ∈ R c l − 1 × h × w in the l th layer ( h equals 1 for univariate TSC and c l − 1 is the number of input channels) , and kernel weights Θ
[Equation]
( k is the kernel size and c l is the number of output channels), the convolution with stride 1 and padding in our proposed dynamic sparse CNN layer is formulated as
[Equation]
where O j ∈ R h × w denotes the output feature representation at the j -th output channel, Z denotes the set of integers, I l ( · ) :
R c l − 1 × c l × 1 × k → { 0 , 1 } c l − 1 × c l × 1 × k is an indicator function,
I l
(Θ l ) i,j indicates activated weights for Θ li,j ∈ R 1 × k which is the kernel in the ( i, j ) -th channel, ⊙ denotes the element-wise product, and · denotes the convolution operator.
The indicator function I l ( · ) , which is learned during the training of the proposed DSN, satisfies that ∥
I l ( · ) ∥ 0 ≤ (1 − S ) N l where 0 ≤ S < 1 is the sparsity ratio, ∥ .
∥ 0 denotes the L 0 norm and N l = c l − 1 × c l × 1 × k .
When S > 0 , the kernel is sparse, and we can use a large k in the dynamic sparse CNN layer to obtain a large RF with reduced computational cost.
Remark.
( Effective Neighbour Receptive Field (eNRF) size in the dynamic sparse CNN layer ).
The receptive field is defined as the region in the input that the feature of a CNN model is looking at.
We defined the Neighbour Receptive Field (NRF) as the region considered by each feature in the successive layer.
Specifically, the NRF size is equivalent to the kernel size in the standard CNN layer (Considering the case that dilation equals to 1).
Differently, the NRF size of the proposed dynamic sparse CNN layer is smaller than the kernel size, when the first or last weight in a kernel is not activated, e.g., ∃ i ∈ { 1 , . . .
, c l − 1 } and j ∈ { 1 , . .
c l } , such that I l (Θ l ) i,j, 1 , 1 = 0 or
I l
(Θ l ) i,j, 1 ,k = 0 .
As illustrated in Figure 2, we defined the eNRF size f li,j of the kernel Θ li,j as the distance between the first activated weight and last activated weight in the l -th CNN layer as
[Equation]
otherwise
[Equation]
where Ind li,j denotes the set of indices corresponding to non-zero weights in the kernel Θ li,j .
Obviously we have 0 ≤ f li,j ≤ k .
= =
= = =
= =
= 3 =
The eNRF size set of l -th dynamic sparse CNN layer is denoted by F ( l ) , which satisfies that 0 ≤ min( F ( l ) ) and max( F ( l ) ) ≤ k .
Take the case in the Figure 2 as an simple example for l -th layer, then F ( l ) = { 0 , 1 , 2 , 3 , 4 , 5 , 6 } .
Each dynamic sparse CNN layer is likely to cover various eNRF sizes ranging from 1 to k .
When global information is expected, I l ( · ) could activate weights more dispersed to get a larger eNRF and exploit the input features selectively.
To capture the local context, a smaller eNRF is expected, thus the activated weights tend to be concentrated.
By taking k = 5 as an example, I l (Θ l ) i,j may be [1, 0, 1, 0, 1] for global context, while it may be [0, 0, 1, 1, 0] for local context.
In this way, the eNRF can be adaptively adjusted.
3.3 Architecture in DSN
The proposed DSN model consists of three sparse CNN modules, each of which composes of a dynamic sparse CNN layer and a 1 × 1 CNN layer.
Following the stacked sparse CNN modules, there is an additional dynamic sparse CNN layer, two adaptive average pooling layers, and one 1 ×convolution layer acting as the classifier in the DSN model.
The overall architecture is shown in
Figure 3.
[Figure]
The set of eNRF sizes S ( l ) in the l th sparse CNN module is equal to that in the dynamic sparse
CNN layer, since the eNRF size of 1 × 1 convolution is constantly equal to 1 .
S ( l ) satisfies that 0 ≤ min( S ( l ) ) and max( S ( l ) ) ≤ k .
Then, the set of eNRF sizes for the three successively stacked sparse CNN modules can be described by RF as
[Equation]
i ) ∈ S ( i ) , i ∈ { 1 , 2 , 3 } (cid:111) .
According to Eq.
(3), we can see that stacking multiple sparse CNN modules can increase the size of RF linearly, where l th dynamic sparse CNN layer increases by the size of S ( l ) .
For simplicity, the kernel size in each dynamic sparse CNN layer is consistently set to k in our study.
Then, RF satisfies that max( RF ) ≤ 3 k − 2 and 0 ≤ min( RF ) .
Thus, a large k in sparse CNN modules can increase the range of the eNRF sizes for stacked sparse CNN modules.
[Figure]
3.4 Dynamic Sparse Training for DSN Model
In this section, we present the training strategy to discover the weights required to be activated in order to ensure well-performing RF.
That is, we need to study how to update the indicator function
I l ( · ) during the training of the proposed DSN.
Input: Dataset D , Network f Θ , Sparsity ratio S Exploration regions in layer l : {R l 1 , ..., R lN }
Exploration schedule: T , ∆ T , α , f decay 1: θ li ← initialize activated weights in Θ li using S and R li 2: for t = 1 to T do 3:
Sample a batch B t ∼ D 4: L t = (cid:80) i ∼ B t L (( f θ ( x i ) , y i ) 5: if ( t mod ∆ T ) =
=
then 6: for each layer l = 1 to L do 7: for each group i = 1 to N do 8: u = n ( R li )
f decay ( t ; α, T ) (1 − S ) 9:
I prune = ArgTopK (cid:0) − (cid:12)(cid:12)
θ li (cid:12)(cid:12) , u (cid:1) 10:
I grow =
RandomK (cid:0) R li \ θ li , u (cid:1) 11:
I l ( . )
← Update I l ( . )
using I prune
and I grow 12: θ li ← Update activated weights I l (Θ li ) ⊙ Θ li 13: end for 14: end for 15: else 16: θ = θ − α ∇ θ L t 17: end if 18: end for
We sparsely train the proposed DSN model from scratch to keep kernel sparse following the main idea of DST methods [40].
By design, during the training phase, the total number of activated weights must not exceed N l (1 − S ) .
However, we observe that the tiny eNRF is hard to be captured by
DST methods with a layer-wise exploration manner, namely activated weights are discovered layer by layer, especially when the sparsity ratio S is undersized (more analysis in Section 4.5).
Motivated by this observation, the kernels in each dynamic sparse CNN layer are divided into different groups, whose corresponding exploration regions are of different sizes as shown in
Figure 4.
Contrary to the DST methods, in
DSN, the exploration of activated weights is separately performed in different kernel groups, which is a more fine-grained strategy.
Specifically, the kernel weights Θ l ∈ R c l − 1 × c l × 1 × k in the l th layer are split into N groups along the output channel, that is, Θ l 1 , . . .
, Θ lN ∈ R c
l − 1 × clN × 1 × k with corresponding exploration regions R l 1 , . . .
, R lN .
The exploration region R li in the i th group of the l th layer is defined as the first i × kN positions of each kernel in this group.
By taking k = 6 and N = 3 for example, activated weights in the first group are only within the first two positions of each kernel in Θ l 1 , while they are within the whole kernel for the last group (as shown in Figure 4).
In this way, the various eNRF can be covered, and exploration space can be reduced (more details can be found in Appendix A.1) to improve exploration efficiency.
Given the weight exploration regions R l 1 , . . .
, R lN and the sparsity ratio S , we train
the proposed
DSN model as shown in Algorithm 1.
The activated weights are explored within the exploration regions and updated every ∆ t iterations.
The fraction of updated weight decays over time according the function f decay ( t ; α, T ) , which follows the cosine annealing [11] as
[Equation]
where α is the initial fraction of activated weights updated, t is the current training iteration, and T is the number of training iterations.
Thus, during the t th iteration, the number of updated activated weights in the i th group of the l th layer is n ( R li ) f decay ( t ; α, T ) (1 − S ) , where n ( R li ) is the number of weights can be explored in region R li .
During the update of activated weights, we firstly prune the activated weights determined by ArgTopK (cid:0) − (cid:12) (cid:12) θ li (cid:12) (cid:12) , u (cid:1) and then randomly grow new activated weights by RandomK (cid:0) R li \ θ li , u (cid:1) , where ArgTopK( v , u ) gives the indices of top u elements with larger values in a vector v , RandomK( v , u ) outputs the indices of random u elements in a vector v , and R li \ θ li denotes the weights within R li but except θ li .
The exploration of activated weights is straightforward.
Firstly, pruning weights with small magnitudes is intuitive, because the contribution of weights with smaller magnitudes is insignificant or even negligible.
Considering the pruning recoverability, we randomly regrow new weights with the same number as the pruned weights to achieve better activated weight exploration.
In this way, the exploration of weights is dynamic and of plasticity, compared with the methods of pruning weights before and after training (more explanations about pruning and dynamic sparse training can be found in [22, 39, 32]).
Experiments
In this section, we evaluate the resource cost and accuracy of the proposed DSN model with recent baseline methods on both univariate and multivariate TSC data.
4.1 Datasets and Baselines
The details of each kind of dataset are as follows.
• Univariate TS dataset from UCR 85 archive [3] : This archive consists of 85 univariate TS datasets, which are collected from various domains (e.g. health monitoring and remote sensing), have distinguishable characteristics, and various levels of complexity.
The number of instances in the training set varies from 16 to 8926, while their time step resolutions range from 24 to 2709.
• Three multivariate TS datasets from UCI [12] : The EEG2 dataset contains 1200 instances of 2 categories and 64 variates.
The Human Activity Recognition (HAR) dataset consists of 10,299 instances with 6 categories, and the number of variates is 9.
The Daily Sport dataset contains 9,120 instances of 19 categories, and its number of variates is 45.
The processing of these datasets is the same as in [26].
Univariate TSC baselines: The performance of the following three baseline methods are reported,
OS-CNN [52], InceptionTime [25], and ResNet[56], as they are widely used for univariate TSC.
Multivariate TSC baselines: Similar with [62], the following three multivariate baselines are selected: OS-CNN, TapNet [62], MLSTM-FCN [26].
To avoid an unfair comparison, we let outside the ensemble methods.
However, based on our best knowledge, our performance analysis accounts for most of state-of-the-art methods.
4.2 Experimental Settings and Implementation Details
For optimization, we use the Adam optimizer with an initial learning rate of 3 × 10 − 4 , cosine decayed to 10 − 4 .
Our model is trained for 1,000 epochs with a mini-batch size of 16.
As in [25, 26], the best model, which corresponds to the minimum training loss, is used to evaluate performance over the testing sets.
Inspired by [25], the kernel sizes k in each dynamic sparse CNN layer are set to 39.
The number of kernel groups N in Algorithm 1 is set to 3, which helps to cover the small, medium, and large eNRF.
Each setting is repeated five times, and the average results are reported.
4.3 Results and Analysis
We show the performance on both univariate and multivariate TSC benchmarks in Table 1-2.
For univariate TSC, we can see that our method outperforms the baseline methods in most cases with a smaller number of parameters (e.g. Params 3 ) and less computation cost (e.g. FLOPs) on UCRarchive datasets.
For multivariate TSC, our proposed DSN method achieves better performance on
EEG2 and HAR datasets, while for Daily Sport, MLSTM-FCN was with 0.46% more accurate.
It is worth highlighting that the resource cost (i.e. Params and FLOPs) of our proposed DSN is much smaller than that of baseline methods.
Due to the page limitation, the detailed results, including test accuracy and resource cost on each dataset of UCR 85 archive, can be found in Appendix C. More results on UCR 128 [6] and UEA 30 archive [2] can be found in Appendix B.9.
[Table]
4.4 Sensitivity Analysis
Effect of Sparsity Ratio: From Table 3, we can find out how the sparsity ratio of dynamic sparse
CNN layer affects the final test accuracy and the resource cost in multivariate TS datasets.
Here,
[Table]
we analyze the trade-off between test accuracy and the resource cost under various sparsity ratios (i.e. S ∈ [50% , 80% , 90%] ) and a dense DSN model.
Note that the dense model is defined exactly as our DSN model with dense connections along entire kernels.
We can see that it is not that the more parameters of the model, the better performance it will achieve.
This is mainly because more parameters will lead to overfitting and the proper sparse ratio will easily cover the desired receptive field (more demonstrations can be seen in Appendix B.1).
Taking the EEG2 data set as an example, with the decrease of sparse rate, the performance on it drops sharply.
We hypothesise that this data set has more expectations for features from small receptive fields (the validation can be found in
Appendix B.2).
According to the experiments, a 80% sparsity ratio reflects a good trade-off between accuracy and resource cost, which is the default setting for the dynamic sparse CNN layer in our
DSN model.
and FLOPs ( M ) ↓ ) of the proposed DSN model with different sparsity ratios on three multivariate TS datasets from UCI.
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] Effect of Architecture: To show how the architecture of DSN affects the final performance, we adopt the critical difference diagram for a detailed comparison, which is usually used for evaluation in TSC [52, 25].
From Figure 5, we can find that the DSN model with 4 dynamic sparse CNN layers and 177 output channels in each layer performs consistently better than the other structures.
The proposed model with more output channels seems to be crucial for preserving the capacity of the network, while requires more parameters and FLOPs compared to other counterparts.
This highlights an interesting trade-off between accuracy and computational efficiency.
According to the experiments, the DSN model with 4 dynamic sparse CNN layers and 141 output channels in each layer is the trade-off between the performance and resource cost, which is the default setting for our DSN model.
Effect of Kernel Group: As shown in Figure 6, training DSN without kernel group hampers the capture of small eNRF, which can not guarantee satisfactory performance on the datasets such as
EEG2 expecting more local information.
With the decrease in sparsity ratio S , the phenomenon that large RF occupies the majority becomes more severe (more demonstrations see in Appendix B.1), which increases the performance gap between training with group and without it, as shown in Table 4.
Grouping the kernels can achieve various RF coverage and adorable accuracy for all datasets.
[Figure]
4.5 Ablation Study
[Table]
Effect of Dynamic Sparse Training:
Furthermore, we study the effect of the dynamic sparse training in our
DSN method.
We conduct ablation studies to compare DSN with its static variant (i.e. DSN fix ), which fixes its topology during training after activated weights initialization.
In other words, the indicator function I l ( · ) in
Eq.
(1) will not update during the training process.
According to the results shown in Figure 7 (a), we can find that the DSN model mostly outperforms the DSN fix on UCR 85 archive, indicating that what is learned by dynamic sparse training is the suitable activated weights together with their values is better here is better here is better here is better here (
[Figure]
[Figure]
is better here is better here (c)
4.6 Case Study on other DST Method
[Figure]
Likewise, the effectiveness of the proposed DSN method is also analyzed in the presence of other DST methods, such as RigL [13].
Rather than growing weights randomly (as in the line 9 of Algorithm 1), RigL grows them with the highest magnitude gradients.
From Figure 7 (c), we can see that growing weights according to gradients (i.e. DSN grad ) can outperforms the OS-CNN method in most cases on
UCR 85 archive.
Furthermore, as shown in Figure 7 (b), we can find that DSN grad has a similar performance as growing weights randomly (i.e. DSN rand ).
Note that, DSN rand is the main setting of the proposed method in this paper with the reason that we don’t need to compute the full gradients at any moment in time.
In summary, the proposed DSN with fine-grained sparse strategy can be easily combined with the existing DST techniques, showing the potential for further improvements by merging it with other advanced DST methods in the future.
4.7 Visualization of Activated Weights
Figure 8 shows the normalized activated weights of the DSN model after sparse training on HAR dataset.
(a), (b) and (c) correspond to the activated weights of the three kernel groups in the first dynamic sparse CNN layer, and each row of the figure represents one kernel.
We can see that the weights in each kernel are sparse, and can be activated within the constraint region defined in
Section 3.4.
Conclusions
This paper proposes dynamic sparse network (DSN) to cover diverse effective neighbour receptive field (eNRF) for time series classification (TSC) without the cumbersome hyper-parameters tuning.
The proposed DSN can achieve about 2 × reduction in memory and computational cost, while achieving superior performance in terms of accuracy on both univariate and multivariate TSC datasets.
Furthermore, DSN explores the activated weights in a more fine-grained strategy, which can be easily merged with existing DST techniques, showing a potential for further improvements when combined with advanced technology.
DSN provides a feasible solution to bridge the gap between resource awareness and various eNRF coverage for TSC, hoping to inspire other researchers in different fields.
It is worthwhile extending DSN to other research domains (e.g. time series forecasting and computer vision).
Our DSN model has not yet been executed on real edge devices, and the performance in terms of accuracy and memory footprint on edge devices is not investigated.
In the future, we hope to extend our work to other fields and decentralise training on edge devices.
Acknowledgements
This work has been partly funded by the NWO Perspectief project MEGAMIND.
We thank everyone who worked on making the datasets available and their contributions to the TSC community