O MNI -S CALE CNN S : A SIMPLE AND EFFECTIVE KERNEL SIZE CONFIGURATION FOR TIME SERIES CLASSIFICATION
A BSTRACT
The Receptive Field (RF) size has been one of the most important factors for One
Dimensional Convolutional Neural Networks (1D-CNNs) on time series classiﬁcation tasks.
Large efforts have been taken to choose the appropriate size because it has a huge inﬂuence on the performance and differs signiﬁcantly for each dataset.
In this paper, we propose an Omni-Scale block (OS-block) for 1D-CNNs, where the kernel sizes are decided by a simple and universal rule.
Particularly, it is a set of kernel sizes that can efﬁciently cover the best RF size across different datasets via consisting of multiple prime numbers according to the length of the time series.
The experiment result shows that models with the OS-block can achieve a similar performance as models with the searched optimal RF size and due to the strong optimal RF size capture ability, simple 1D-CNN models with
OS-block achieves the state-of-the-art performance on four time series benchmarks, including both univariate and multivariate data from multiple domains.
Comprehensive analysis and discussions shed light on why the OS-block can capture optimal RF sizes across different datasets.
I NTRODUCTION
One of the most challenging problems for Time Series Classiﬁcation (TSC) tasks is how to tell models in what time scales 1 to extract features.
Time series (TS) data is a series of data points ordered by time or other meaningful sequences such as frequency.
Due to the variety of information sources (e.g., medical sensors, economic indicators, and logs) and record settings (e.g., sampling rate, record length, and bandwidth), TS data is naturally composed of various types of signals on various time scales (Hills et al., 2014; Sch¨afer, 2015; Dau et al., 2018).
Thus, in what time scales can a model “see” from the TS input data has been a key for the performance of TS classiﬁcation.
Traditional machine learning methods have taken huge efforts to capture important time scales, and the computational resource consumption increase exponentially with the length of TS increase.
For example, for shapelet methods (Hills et al., 2014; Lines et al., 2012), whose discriminatory feature is obtained via ﬁnding sub-sequences from TS that can be representative of class membership, the time scale capture work is ﬁnding the proper sub-sequences length.
To obtain the proper length, even for a dataset with length 512, (Hills et al., 2014) has to try 71 different sub-sequence lengths.
For other methods, such as (Berndt & Clifford, 1994; Sch¨afer, 2015; Lucas et al., 2019), despite the time scale capture might be called by different names such as ﬁnding warping size or window length.
They all need searching works to identify those important time scales.
More recent deep learning based methods also showed that they had to pay a lot of attention to this time scale problem.
MCNN (Cui et al., 2016) searches the kernel size to ﬁnd the best RF of a 1D-CNN for every dataset.
Tapnet (Zhang et al., 2020) additionally considers the dilation steps.
Chen & Shi (2021) also take
[Figure]
the number of layers into considerations.
These are all important factors for the RF of CNNs and the performance for TSC.
Although a number of researchers have searched for the best RF of 1D-CNNs for TSC, there is still no agreed answer to 1) what size of the RF is the best?
And 2) how many different RFs should be used?
Models need to be equipped with different sizes and different numbers of RFs for a speciﬁc dataset.
Using the same setup for every dataset can lead to a signiﬁcant performance drop for some datasets.
For example, as shown by the statistics on the University of California Riverside (UCR)“bake off” datasets in Figure 1a, the accuracy of most datasets can have a variance of more than 5% just by changing the RF sizes of their model while keeping the rest of the conﬁgurations the same.
As also shown in Figure 1b, no RF can consistently perform the best over different datasets.
To avoid those complicated and resource-consuming searching work, we propose Omni-Scale block (OS-block), where the kernel choices for 1D-CNNs are automatically set through a simple and universal rule that can cover the RF of all scales.
The rule is inspired by Goldbach’s conjecture, where any positive even number can be written as the sum of two prime numbers.
Therefore, the
OS-block uses a set of prime numbers as the kernel sizes except for the last layer whose kernel sizes are 1 and 2.
In this way, a 1D-CNN with these kernel sizes can cover the RF of all scales by transforming TS through different combinations of these prime size kernels.
What’s more, the
OS-block is easy to implement to various TS datasets via selecting the maximum prime number according to the length of the TS.
In experiments, we show consistent state-of-the-art performance on four TSC benchmarks.
These benchmarks contain datasets from different domains, i.e., healthcare, human activity recognition, speech recognition, and spectrum analysis.
Despite the dynamic patterns of these datasets, 1DCNNs with our OS-block robustly outperform previous baselines with the uniﬁed training hyperparameters for all datasets such as learning rate, batch size, and iteration numbers.
We also did a comprehensive study to show our OS-block, the no time scale search solution, always matches the performance with the best RF size for different datasets.
M OTIVATIONS
Two phenomena of 1D-CNNs inspire the design of the OS-block.
In this section, we will introduce the two phenomena with examples in Figure 2 and more discussions can be found in Section 4.6.
Firstly, we found that, although the RF size is important, the 1D-CNNs are not sensitive to the speciﬁc kernel size conﬁgurations that we take to compose that RF size.
An example is given in the right image of the Figure
Secondly, the performance of 1D-CNNs is mainly determined by the best RF size it has.
To be speciﬁc, supposing we have multiple single-RF-size-models which are of similar model size and layer numbers, but each of them has a unique RF size.
Let’s denote the set of those RF sizes as S .
When testing those models on a dataset, we will have a set of accuracy results A .
Then, supposing we have a multi-kernel model which has multiple RF sizes 2 and set of those sizes is also S .
Then,
[Figure]
the accuracy of the multiple-RF-sizes-model will be similar to the highest value of A .
An example is given in the left image of Figure 2.
Speciﬁcally, when testing single-RF-size-models on the Google
Speechcommands dataset, the model’s performance is positive correlation with the model’s RF size.
For example, the light blue line whose set of RF size is { 99 } outperforms the light green line { 39 } and light red line { 9 } .
For those multiple-RF-sizes-models which has more than one element in their set of RF sizes, their performance are determined by the best (also the largest because of the positive correlation )
RF size it has.
Having more worse (smaller) RF sizes will not have much inﬂuence on the performance.
The second phenomenon means that, instead of searching for the best time scales, if the model covers all RF sizes, its performance will be similar to that of a model with the best RF size.
However, there are many designs that can cover all RF sizes.
Which one should be preferred?
Based on the ﬁrst phenomenon, from the performance perspective, we could choose any design that we want.
However, as we will show in Section 3.3, those candidate designs are not of the same characteristics such as the model size or the expandability for long TS data.
Therefore, the design of the OS-block that we propose aims at covering all RF sizes in an efﬁcient manner.
M ETHOD
The section is organized as follows:
Firstly, we give the problem deﬁnition in Section 3.1.
Then, we will explain how to construct the Omni-scale block (OS-block) which covers all receptive ﬁeld sizes in Section 3.2.
Section 3.3 will explain the reason why OS-block can cover RF of all sizes in an efﬁcient manner.
In Section 3.4, we will introduce how to apply the OS-block on TSC tasks.
3. P ROBLEM D EFINITION
TS data is denoted as X =
[ x 1 , x 2 , ..., x m ] , where m is the number of variates.
For univariate TS data, m = 1 and for m > 1 , the TS are multivariate.
Each variate is a vector of length l .
A TS dataset, which has n data and label pairs, can be denoted as: D
= { ( X 1 , y 1 ) , ( X 2 , y 2 ) , ..., ( X n , y n ) } , where ( X ∗ , y ∗ ) denotes the TS data X ∗ belongs to the class y ∗ .
The task of TSC is to predict the class label y ∗ when given a TS X ∗ .
3. A RCHITECTURE OF OS- BLOCK
The architecture of the OS-block is shown in Figure 3.
It is a three-layer multi-kernel structure, and each kernel does the same padding convolution with input.
For the kernel size conﬁguration, we use P ( i ) to denote the kernel size set of the i -th layer:
[Equation]
p k
Concatenation Concatenation
Concatenation Batchnorm+ReLU
Fully connected Global average pooling
OS-CNN
OS-block 1 2 3 5 7 11 ….
p k
OS-block
Input data
Output result
Batchnorm+ReLU Batchnorm+ReLU
Layer output
Layer output 1 2 3 5 7 11 13 17 19 … 1… … …8…12…16 18…20 24…28 30…32 36 38 … … … … … … … … … … … …
Green: prime number : OS to cover all sizes Red :
[Figure]
number : OS to cover all sizes in a range
P r
i m e nu m b e r li s t
Prime number list Sum
Layer Layer
Layer
Mathematical phenomenon 1D-CNN block design 1D-CNN classifier cover of scales
For
, since the TS length is l , we don’t need to cover RFs that are larger than l or smaller than 1 .
Therefore, the p k is the smallest prime number that can cover the RF size from 1 to l .
If we have prior knowledge, such as that we know there are cycles in the TS, or we know the length range of the hidden representative pattern.
We could change the RF size range of the OS-block by simply changing the prime number list.
An example is given in the left image in
Figure 3, which uses the prime number list in the blue block to cover the RF size range from 10 to 26.
RF sizes of the OS-block : The RF is deﬁned as the size of the region in the input that produces the feature.
Because each layer of the OS-block has more than one convolution kernel, there will be several different paths from the input signal to the ﬁnal output feature (Araujo et al., 2019;
Luo et al., 2016), and each path will have a RF size.
For the 3-layer OS-block, which has no pooling layer and the stride size is 1, the set of RF sizes S is the set of RF size of all paths, and it can be described as:
[Equation]
For the reasons that P ( i ) are prime number list when i ∈ { 1 , 2 } , the set { p
[Equation] [Equation]
p
[Equation] [Equation]
p ( i ) ∈
P ( i ) , i ∈ { 1 , 2 }} is the set of all even numbers E .
Thus, we have
[Equation]
With Equation 3 and Equation 1, we have
[Equation]
Where N + is the set of all integer numbers in the range.
Speciﬁcally, the S ≡ N + is because a real number must be an odd number or an even number, while E is the even number
[Figure]
the odd number set.
Therefore, with the proper selection of p k , we could cover any integer RF size in a range.
It should be noticed that, there might be many options to cover all RF sizes, we use the
Godlach’s conjecture to make sure that we could all scales.
3. OS- BLOCK COVER ALL SCALES IN AN EFFICIENT MANNER
From the model size perspective, using prime numbers is more efﬁcient than using even numbers or odd numbers.
To be speciﬁc, to cover receptive ﬁelds up to size r, the model size complexity of using prime size kernels is O ( r 2 /log
( r )) .
On the other hand, no matter we use even number pairs or odd number pairs, the model size complexity is O ( r 2 ) .
We also empirically show the advantage of our model on efﬁciency in the following table and this table has been added to Appendix A.7:
H OW TO APPLY OS- BLOCK ON TSC TASKS
Firstly, the OS-block could take both univariate and multivariate TS data by adjusting the input channel the same as the variate number of input TS data.
A simple example classiﬁer with OSblock, namely OS-CNN, is given in Figure 3.
The OS-CNN is composed of an OS-block with one global average pooling layer as the dimensional reduction module and one fully connected layer as the classiﬁcation module.
Other than OS-CNN, the OS-block is ﬂexible and easy to extend.
Speciﬁcally, convolution layers of OS-block can be calculated parallelly.
Thus, each layer can be viewed as one convolutional layer with zero masks.
Therefore, both the multi-kernel layers or the
OS-block itself are easy to extend with more complicated structures (such as dilation (Oord et al., 2016), attention or transformer (Shen et al., 2018a), and bottleneck) that are normally used in 1DCNN for performance gain.
In Figure 4, we give three examples which uses OS-block with other structures.
E XPERIMENT 4. B ENCHMARKS
We evaluate OS-block on 4 TSC benchmarks which include, in total, 159 datasets.
The details of each benchmark is as follows:
• Magnetoencephalography recording for Temporal Lobe Epilepsy diagnosis (MEGTLE) dataset (Gu et al., 2020):
The Magnetoencephalography dataset was recorded from epilepsy patients and was introduced to classify two subtypes (simple and complex) of temporal Lobe Epilepsy.
The dataset contains 2877 recordings which were obtained at the sampling frequency 1200 Hz.
Each recording is approximately 2 sec.
Therefore the length is about 2400.
• University of East Anglia (UEA)
30 archive (Bagnall et al., 2018):
This formulation of the archive was a collaborative effort between researchers at the University of East Anglia and the University of California, Riverside.
It is an archive of 30 multivariate TS datasets
[Table]
from
domains such as motion detection
physiological data, audio spectra classiﬁcation
Besides domains, those datasets also have various characteristics
For instance
those datasets,
number
• University of California, Riverside (UCR) 85 archive
(Chen et al., 2015): This is an archive of 85 univariate TS datasets from various domains such as speech reorganizations, health monitoring
[Table]
and spectrum analysis.
What’s more, those datasets also have different characteristics.
For instance, among those datasets, the class number varies from 2 to 60, the length of each dataset varies from 24 to 2709.
The number of training data varies from 16 to 8,926.
• University of California, Riverside (UCR)
128 archive (Dau et al., 2018):
This is an archive of 128 univariate TS datasets.
It is the updated version of the UCR 85 archive.
However, the new archive cannot be viewed as
a replacement for the former because they have different characteristics.
For example, for the UCR 85 archive, all TS data within a single dataset are of the same length, but that is not the same for the UCR 128 archive.
Besides that, in general, the added data in the UCR 128 archive, their default test set is bigger than the train set to reﬂect real-world scenarios.
4. E VALUATION CRITERIA
For all benchmarks, we follow the standard settings from previous literature.
Speciﬁcally, for the
MEG-TLE dataset, following Multi-Head Self-Attention Model (MSAM)
(Gu et al., 2020), models are evaluated by test accuracy and f1 score.
Besides using recommended metrics of each benchmark, we also compare the model size of OS-block with other deep learning methods.
For UEA 30, UCR 85 archives, and UCR 128 archives, following the evaluation advice from the archive (Dau et al., 2018; Bagnall et al., 2018), count of wins, and critical difference diagrams (cd-diagram) (Dau et al., 2018) were selected as the evaluation method.
Due to the page limitation, we list the average rank
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
i c k e t Y F a c e A
ll C
r
i c k e t Z B ee t l e F l y C r
i c k e t X
C h l o r i n e C o n c e n t r a t i o n W
i n e
P r o x i m a l P h a l a n x O u t li n e A g
e G r o u p
A rr o w H e a d F o r d B A
d
i a c O li v e O il
S h a p e l e t S
i m
C i n C E C G T
o r s o
P h a l a n g e s
O u t li n e s C o rr e c t B ee f
M i dd l e P h a l a n x O u t li n e
C o rr e c t
U W a
v e G e s t u r e L i b r a r y X
F i f t y W
o r d s
L i g h t n i n g
L i g h t n i n g 7 W o r m s
P r o x i m a l P h a l a n x T W
M e d i c a l I m a g e s
D i s t a l P h a l a n x O u t li n e
C o rr e c t
U W a
v e G e s t u r e L i b r a r y Z
D i s t a l P h a l a n x O u t li n e
A g e G r o u p
U W a
v e G e s t u r e L i b r a r y Y
W o r dS y n o n y m s
C o m p u t e r s
E l e c t r i
c
D e v
i c
e s
S m a ll K
i t c h e n A pp li a n c e s
E a r t
h q u a k e s H a m
W o r m s T w o
C l a ss
D i s t a l P h a l a n x T W H e rr i n g
I n s e c t W i n gb e a t S o un d
M i dd l e P h a l a n x O u t li n e
A g e G r o u p
S c r ee n T y p
e
M i dd l e P h a l a n x T W H a p t i c s
R e f r i g e r a t i
o n D e
v i
c e s
I n li n e S k a t e
P h
o n e m e
[Figure]
d a t
[Figure]
4. E XPERIMENT SETUP
For the MEG-TLE dataset, they were normalized by z-normalization (Chen et al., 2015).
For the other archives, we take the raw dataset without processing for datasets in those archives already normalized with z-normalization.
Following the setup of (Wang et al., 2017), we use the learning rate of 0.001, batch size of 16, and
Adam (Kingma & Ba, 2014) optimizer.
The baselines are chosen from the top seven methods from the leaderboard 4 of each benchmark.
For UCR archives, we ensemble ﬁve OS-CNNs, which stacks two OS-blocks with residential connections followed by the baseline IncpetionTime (Ismail Fawaz et al., 2019).
We use PyTorch 5 to implement our method and run our experiments on Nvidia Titan
XP.
4. S TATE - OF - THE - ART PERFORMANCE ON BENCHMARKS
We show consistent state-of-the-art performance on four benchmarks as in Table 1.
As we can see, for the MIT-TLE dataset, OS-CNN outperforms baselines in a ten times smaller model size.
For all dataset archives, the OS-block achieves the best average rank, which means that, in general, the
OS-block design can achieve better performance.
4. OS- BLOCK CAN CAPTURE THE BEST TIME SCALE
To demonstrate that the OS-block can capture the best time scale, we build 20 FCN models with different RF sizes (from 10 to 200 with step 10), and compare their performance with OS-CNN on the UCR 85 archive.
Speciﬁcally, the FCN (Wang et al., 2017) is selected as the backbone model for it has a similar structure as OS-CNN.
To obtain FCN with various RF sizes, we change the kernel size of each layer proportionally.
To be speciﬁc, the kernel sizes of the original three layer FCN are 8, 5, and 3, and the RF size is 14.
To obtain the RF size 30, we will set kernel sizes of each layer as 16,10, and 6.
To control variables, when the kernel size increases, we will reduce the channel number to keep the model size constant.
This will not inﬂuence the conclusion.
To check that, in Appendix A.2, we also provide the static result comparison between OS-CNN and FCNs with the ﬁxed channel number.
[Figure]
Due to the page limitation, the full result can be found in the supplementary material.
And in
Figure 5, a simple result comparison is given, and we could see that for most of the datasets, OSCNN can achieve a similar result as models with the best time scale.
4. D ISCUSSION ABOUT BEST TIME SCALE CAPTURE ABILITY
The result in Figure 5 empirically veriﬁes two phenomena that we mentioned in Section 2 with multiple datasets from multiple domains.
Firstly, the OS-block covers all scales.
Therefore, besides the important size, it also covers many redundant sizes, but those redundancies will not pull down the performance.
Secondly, OS-block composes the RF size via the prime design while the FCN uses a different design.
It means that the performances of 1D-CNNs are determined mainly by the
RF size instead of the kernel conﬁguration to compose that.
4. C ASE STUDY FOR THE BEST TIME SCALE CAPTURE ABILITY
To further demonstrate the RF size capture ability, we will give a case study that compares the class activation map (Zhou et al., 2016) of OS-CNN with that of models with the best RF size.
We select the ScreenType and InsectWingbeatSound datasets for the case study.
They were selected because they are of the largest and the smallest accuracy difference calculated by the accuracy of FCN with
RF size 10 (FCN(10))
minus accuracy of FCN with RF size 200 (FCN(200)).
Speciﬁcally, it can be seen as, among UCR 85 datasets, the ScreenType is the dataset which the FCN(10) outperform
FCN(200)
most, and InsectWingbeatSound is the dataset which the FCN(200) outperforms FCN(10)
most.
We visualize the class activation map of the ﬁrst instance in the two datasets, and the results are shown in Figure 6.
As we can see in Figure 6, the class activation map of the OS-block is similar to that of the model with the best RF size.
R ELATED WORKS
A TS data is a series of data points.
TSC aims at labeling unseen TS data via a model trained by labeled data (Dau et al., 2018; Chen et al., 2015).
One well-known challenge for TSC is telling the model in what time scale to extract features (Hills et al., 2014; Sch¨afer, 2015; Berndt & Clifford, 1994).
This is because TS data is naturally composed of multiple signals on different scales (Hills et al., 2014; Sch¨afer, 2015; Dau et al., 2018) but, without prior knowledge, it is hard to ﬁnd those scales directly.
The success of deep learning encourages researchers to explore its application on TS data (L¨angkvist et al., 2014;
Fawaz et al., 2019; Dong et al., 2021).
The Recurrent Neural Network (RNN) is designed for temporal sequence.
In general, it does not need extra hyper-parameters to identify information extraction scales.
However, RNN is rarely applied on TS classiﬁcation (Fawaz et al., 2019).
There are many reasons for this situation.
One widely accepted reason is that when faced with long
TS data, RNN models suffer from vanishing gradient and exploding gradient (Pascanu et al., 2013;
Fawaz et al., 2019; Bengio et al., 1994).
Nowadays, the most popular deep-learning method for TSC is 1D-CNN.
However, for 1D-CNNs, the feature extraction scale is still a problem.
For example, there is an unresolved challenge with kernel size selection where there exists different approaches but non consensus on which is best.
To date, the selection of feature extraction scales for 1D-CNN is regarded as a hyper-parameter selection problem e.g., (Cui et al., 2016) uses a grid search to ﬁnd kernel sizes, while the following methods tune it empirically (Zheng et al., 2014;
Wang et al., 2017; Rajpurkar et al., 2017; Serr`a et al., 2018; Ismail Fawaz et al., 2019; Kashiparekh et al., 2019).
Dilated convolution (Oord et al., 2016) is widely adopted in 1D-CNN to improve generalization ability for TS tasks (Oord et al., 2016;
Zhang et al., 2020; Li et al., 2021).
It takes a lower sampling frequency than the raw signal input thus can be viewed as a structure-based low bandpass ﬁlter.
Compared with the OS-block, the dilated convolution also needs prior knowledge or searching work to set the dilation size which will determine the threshold to ﬁlter out redundant information from
TS data.
Inception structure (Szegedy et al., 2015) is widely used in 1D-CNN for TSC tasks (Ismail Fawaz et al., 2019; Kashiparekh et al., 2019; Chen & Shi, 2021; Dong et al., 2021).
The design of the multi-kernel structure of OS-block is inspired from the inception structure (Szegedy et al., 2015).
Compared with existing works, the OS-block has two differences.
Firstly, the OS-block does not need to assign weight to important scales via complicated methods such as pre-train (Kashiparekh et al., 2019), attention (Chen & Shi, 2021;
Shen et al., 2018b), or a series of modiﬁcations such as bias removal and bottleneck for convolutions (Ismail Fawaz et al., 2019).
Secondly, OS-block does not need to search for candidate scales.
Speciﬁcally, those methods can only assign weight to a limited number of scales.
Thus, they still need searching works to answer a series of questions.
For example, Which sequence, such as geometric or arithmetic, should be preferred?
What’s the largest length to stop?
How do they select the depth of the neural network?
And how do they select the common difference or ratio for their sequence?
Adaptive receptive ﬁeld (Han et al., 2018; Tabernik et al., 2020;
Xiong et al., 2020; Pintea et al., 2021;
Liu et al., 2021; Tomen et al., 2021; Dong et al., 2021), has been proposed to learn the optimal kernel sizes during the training stage.
Generally, it can be viewed as learning a weight mask on kernels to control the receptive ﬁeld size.
The weight of the mask can be learned during the training step.
On the other hand, OS-block learns the linkage between kernels and uses kernels of different sizes to compose different receptive ﬁeld sizes.
In principle, the adaptive receptive ﬁeld can be used on the time series classiﬁcation tasks.
It improves the performance by enabling 1D-CNNs to have the best receptive ﬁeld size.
But the OS-block targets at covering all sizes of receptive ﬁled sizes and assign large weight on important sizes.
Mathematically, OS-block is a very general technique and can be extended to time series vision tasks by using the prime size design on the time dimension.
This is because the video classiﬁcation task and time series classiﬁcation task share the same challenge (Xie et al., 2018; Bian et al., 2017; Tan et al., 2021;
Liu et al., 2020; Li et al., 2020), which is the same region of interest might of different time scales for different data.
Thus, using the kernel of various sizes will increase the probability to catch proper scales.
However, in this paper, we mainly target the classic 1D time series classiﬁcation, which is an active research area with many open problems (Fawaz et al., 2019; Zhang et al., 2020;
Dempster et al., 2020) unsolven.
C ONCLUSION
The paper presents a simple 1D-CNN block, namely OS-block.
It does not need any feature extraction scale tuning and can achieve a similar performance as models with the best feature extraction scales.
The key idea is using prime number design to cover all RF sizes in an efﬁcient manner.
We conduct experiments to demonstrate that the OS-block can robustly capture the best time scale on datasets from multiple domains.
Due to the strong scale capture ability, it achieves a series SOTA performance on multiple TSC benchmarks.
Besides that, the OS-CNN results reveal two characteristics of 1D-CNN models, which will beneﬁt the development of the domain.
In the future, we could extend our work in the following aspects.
Firstly, other than the prime kernel size design, there might be a more efﬁcient design to cover all RF sizes.
Secondly, the OS-block can work with existing deep neural structures to achieve better performance, but there might be unique structures or variants of those existing structures that are more suitable for the OS-block.
Besides that, characteristics of OS-block are empirically analyzed via the way there must be a theoretical explanation of the characteristics.
A A PPENDIX A. S TATISTIC OF THE COMPARISON ( FIX MODEL SIZE )
More statistic results of the result in Figure 5 is shown in Figure 7, Figure 8 and Figure
A d i a c
A rr o w H e a d B ee
f
B ee t l e F l y
B i r d C h
i c k e n C a r C B F
C h l o r i n e C o n c e n t r a t i o n
C i n C E C G T
o r s o C o
ff ee
C o m p u t e r s C
r
i c k e t
X C r i c k e t
Y C
r
i c k e t Z
D i a t o m S
i
z
e R e d u c t
i
o n
D i s t a l P h a l a n x O u t li n e
A g e G r o u p
D i s t a l P h a l a n x O u t li n e
C o rr e c t
D i s t a l P h a l a n x T W
E a r t
h q u a k e s
E C G 200 E C G
E C G F i v e D a y s
E l e c t r i
c
D e v
i c
e s F a c e A ll F a c e F o u r
F a c e s U C R F i f t y W
o r d
s F
i s h F o r d
A F o r d B G un P o i n t H a m
H a n d O u t li n e s
H a p t i c
s H e rr i n g
I n li n e S k a t e
I n s e c t W i n gb e a t S o un d
I t a l y P o w e r D
e
m a n d
L a r g e K i t
c h e n A pp li a n c e s
L i g h t n i n g
L i g h t n i n g 7 M a ll a t M e a t
M e d i c a l I m a g e s
M i dd l e P h a l a n x O u t li n e
A g e G r o u p
M i dd l e P h a l a n x O u t li n e
C o rr e c t
M i dd l e P h a l a n x T W
M o t e S t r a i n
N o n I n v a s i v e F e t a l E C G
T h
o r a x
N o n I n v a s i v e F e t a l E C G
T h
o r a x 2 O li v e O il O S
U L e a f
P h a l a n g e s
O u t li n e s C o rr e c t
P h
o n e m e
P l a n e
P r o x i m a l P h a l a n x O u t li n e A g
e G r o u p
P r o x i m a l P h a l a n x O u t li n e C
o
rr
e c t
P r o x i m a l P h a l a n x T W
R e f r i g e r a t i
o n D e
v i
c e s
S c r ee n T y p
e
S h a p e l e t S
i m
S h a p e s A ll
S m a ll K
i t c h e n A pp li a n c e s
S o n y A
I B O
R o b o t S
u r f a c e
S o n y A
I B O
R o b o t S
u r f a c e
S t a r L i g h t
C u r v e s
S t r a w b e rr y
S w e
d
i s h
L e a f
S y m b o l s
S y n t
h e t
i c C o n t r o l
T o e S e g m e n t a t i o n
T o e S e g m e n t a t i o n 2 T r a c e
T w o L e a d E
C G
T w o P a tt e r n s
U W a
v e G e s t u r e L i b r a r y A ll
U W a
v e G e s t u r e L i b r a r y X
U W a
v e G e s t u r e L i b r a r y Y
U W a
v e G e s t u r e L i b r a r y Z W a f e r W i n e
W o r dS y n o n y m s W o r m s
W o r m s T
w o C l a ss Y
o g a
[Figure] [Figure]
[Figure]
A cc u r a c y
[Figure]
of
[Figure]
This means that, for an unknown dataset, using OS-block will have more than 56% chance to achieve a better result than grid search from 20 candidate scales.
When seeing the count of the number larger than 0.5 percentile, we could see that, for an unknown dataset, using OS-block will have more than 96% chance to achieve a better result than selecting a random scale.
D i a t o m S
i
z
e R e d u c t
i
o n O li v e O il C r
i c k e t X
U W a
v e G e s t u r e L i b r a r y Y
W o r dS y n o n y m s
U W a
v e G e s t u r e L i b r a r y Z
F i f t y W
o r d s
C r i c k e t
Y C
r
i c k e t Z
M e d i c a l I m a g e s
U W a
v e G e s t u r e L i b r a r y A ll
P r o x i m a l P h a l a n x O u t li n e C
o
rr
e c t Y
o g a
P h a l a n g e s
O u t li n e s C o rr e c t F a c e A ll
I n s e c t W i n gb e a t S o un d
U W a
v e G e s t u r e L i b r a r y X
C h l o r i n e C o n c e n t r a t i o n
N o n I n v a s i v e F e t a l E C G
T h
o r a x 2 F a c e s U C R
S w e
d
i s h
L e a f
T o e S e g m e n t a t i o n
D i s t a l P h a l a n x O u t li n e
A g e G r o u p
N o n I n v a s i v e F e t a l E C G
T h
o r a x 1 A d
i a c
S o n y A
I B O
R o b o t S
u r f a c e
I n li n e S k a t e
L a r g e K i t
c h e n A pp li a n c e s
G un P
o
i n t
E C G F i v e D a y s
S y n t
h e t
i c C o n t r o l E C G
S t r a w b e rr y C B F F o r d B C a r P l a n e T r a c e
T w o P a tt e r n s C o
ff ee
T w o L e a d E
C G W a f e r
M i dd l e P h a l a n x T W B ee f
S h a p e s A ll
S t a r L i g h t
C u r v e s
E l e c t r i
c
D e v
i c
e s
T o e S e g m e n t a t i o n 1 S y m b o l s E C G 5000 F
i s h
M o t e S t r a i n M a ll a t
S o n y A
I B O
R o b o t S
u r f a c e
H a n d O u t li n e s W i n e
D i s t a l P h a l a n x O u t li n e
C o
rr e c t F o r d
A
M i dd l e P h a l a n x O u t li n e
C o rr e c t
L i g h t n i n g
I t a l y P o w e r D
e
m a n d
P r o x i m a l P h a l a n x T W F a c e F o u r
A rr o w H e a d M e a t
P r o x i m a l P h a l a n x O u t li n e A g
e G r o u p
E a r t
h q u a k e s
R e f r i g e r a t i
o n D e
v i
c e s O S
U L e a f W o r m s
C i n C E C G T
o r s o
P h
o n e m e
M i dd l e P h a l a n x O u t li n e
A g e G r o u p L
i g h t n i n g 2 H a p t i c s
S m a ll K
i t c h e n A pp li a n c e s
D i s t a l P h a l a n x
T W H
e rr i n g B ee
t l e F l y
B i r d C h
i c k e n
S c r ee n T y p
e H a m
C o m p u t e r s
W o r m s T w o
C l a ss
S h a p e l e t S
i m
[Figure]
n
[Figure]
e
[Figure]
D i a t o m S
i
z
e R e d u c t
i
o n
W o r dS y n o n y m s
F i f t y W
o r d s
M e d i c a l I m a g e s
P r o x i m a l P h a l a n x O u t li n e C
o
rr
e c t Y
o g a
P h a l a n g e s
O u t li n e s C o rr e c t F a c e A ll
F a c e s U C R
S w e
d
i s h
L e a f
D i s t a l P h a l a n x O u t li n e
A g e G r o u p A d
i a c C a r P l a n e
M i dd l e P h a l a n x T W S h a p e s A ll S y m b o l s F
i s h
H a n d O u t li n e s
D i s t a l P h a l a n x O u t li n e
C o rr e c t
M i dd l e P h a l a n x O u t li n e
C o rr e c t
P r o x i m a l P h a l a n x T W F a c e F o u r
A rr o w H e
a
d
P r o x i m a l P h a l a n x O u t li n e A g
e G r o u p
O S U L e a f W o r m s
M i dd l e P h a l a n x O u t li n e
A g e G r o u p
D i s t a l P h a l a n x
T W H
e rr i n g B e e t
l e F l y
B i r d C h
i c k e n
W o r m s T w o
C l a ss C r
i c k e t X
U W a
v e G e s t u r e L i b r a r y Y
U W a
v e G e s t u r e L i b r a r y Z
C r i c k e t
Y C
r
i c k e t Z
U W a
v e G e s t u r e L i b r a r y A ll
U W a
v e G e s t u r e L i b r a r y X
T o e S e g m e n t a t i o n
I n li n e S k a t e
G un P
o
i n t
T o e S e g m e n t a t i o n 1 H a p t i c
s O li v e O il
I n s e c t W i n gb e a t S o un d
C h l o r i n e C o n c e n t r a t i o n
N o n I n v a s i v e F e t a l E C G
T h
o r a x
N o n I n v a s i v e F e t a l E C G
T h
o r a x
S o n y A
I B O
R o b o t S
u r f a c e
L a r g e K i t
c h e n A pp li a n c e s
E C G F i v e D a y s
E C G 200 S t r a w b e rr y F o r d B
T w o P a tt e r n s C o ff e e T r a c e W a f e r B e e f
S t a r L i g h t
C u r v e s
E l e c t r i
c
D e v
i c
e s
E C G 5000 M o t e S t r a i n
S o n y A
I B O
R o b o t S
u r f a c e 2 W i n e F
o r d
A
L i g h t n i n g
I t a l y P o w e r D
e
m a n d M e a t
E a r t
h q u a k e s
R e f r i g e r a t i
o n D e
v i
c e s
C i n C E C G T
o r s o
P h
o
n e m e L i g h t n i n g
S m a ll K
i t c h e n A pp li a n c e s
S c r ee n T y p
e H a m
C o m p u t e r s
S y n t
h e t
i c C o n t r o l C B F
T w o L e a d E
C G M a ll a t
S h a p e l e t S
i m
[Figure] [Figure]
[Figure]
A cc u r a c y r a n g ea cc u r a c y o f O S - C
NN
[Figure]
Synthetic
A. S TATISTIC OF THE COMPARISON ( FIX CHANNEL NUMBER )
When we keep the number of channels constant in FCN, the statistic result will be as this.
The
OS-CNN still achieves similar performance as the model with the best scales.
G
P o n t
C G F i y
S y n t
h e t
i c C o n t o l C B F T a c e C o ff
P l a n e
P tt r n s
T
L a
C
W
F h
[Table]
D t m S
i e R e d u c t
o n
S
[Table]
a r L
[Table]
h t C
[Table]
s
[Table] [Table]
[Table] [Table]
c
[Table]
[Table] [Table]
N
[Table]
n I n v a s i v
[Table]
F
[Table]
t a l E C G
T
[Table]
o r a x
N
[Table]
n I n a s i v
F t a l
C
T h
o r a x
T
S e g m n t a t i n
F a c
F o u
T
S e g m n t a t i o n 2 F o r d
A a n d t li n
I t l y P
r
D m n d
[Table]
[Table] [Table]
[Table]
P r o x i m a l P h a l a n x O u t li n e C
o
rr
e c t
L a r g e K i t
c h e n A pp li a n c e s
C r i c k e t Y F a c e A
ll C
r
i c k e t Z B ee t l e F l y C r
i c k e t X
C h l o r i n e C o n c e n t r a t i o n W
i n e
P r o x i m a l P h a l a n x O u t li n e A g
e G r o u p
A rr o w H e a d F o r d B A
d
i a c O li v e O il
S h a p e l e t S
i m
C i n C E C G T
o r s o
P h a l a n g e s
O u t li n e s C o rr e c t B ee f
M i dd l e P h a l a n x O u t li n e
C o rr e c t
U W a
v e G e s t u r e L i b r a r y X
F i f t y W
o r d s
L i g h t n i n g
L i g h t n i n g 7 W o r m s
P r o x i m a l P h a l a n x T W
M e d i c a l I m a g e s
D i s t a l P h a l a n x O u t li n e
C o rr e c t
U W a
v e G e s t u r e L i b r a r y Z
D i s t a l P h a l a n x O u t li n e
A g e G r o u p
U W a
v e G e s t u r e L i b r a r y Y
W o r dS y n o n y m s
C o m p u t e r s
E l e c t r i
c
D e v
i c
e s
S m a ll K
i t c h e n A pp li a n c e s
E a r t
h q u a k e s H a m
W o r m s T w o
C l a ss
D i s t a l P h a l a n x T W H e rr i n g
I n s e c t W i n gb e a t S o un d
M i dd l e P h a l a n x O u t li n e
A g e G r o u p
S c r ee n T y p
e
M i dd l e P h a l a n x T W H a p t i c s
R e f r i g e r a t i
o n D e
v i
c e s
I n li n e S k a t e
P h
o n e m e
[Figure]
[Figure] [Figure]
[Figure] [Figure]
c n F
[Figure]
U W a
v e G e s t u r e L i b r a r y Y
U W a
v e G e s t u r e L i b r a r y A ll
W o r dS y n o n y m s O li v e O il C r
i c k e t Y C r
i c k e t Z
U W a
v e G e s t u r e L i b r a r y X
U W a
v e G e s t u r e L i b r a r y Z
I n s e c t W i n gb e a t S o un d Y o g a
M e d i c a l I m a g e s
F a c e s U C R
P h a l a n g e s
O u t li n e s C o rr e c t
S h a p e s A ll
N o n I n v a s i v e F e t a l E C G
T h
o r a x
C h l o r i n e C o n c e n t r a t i o n
M o t e S t r a i n
N o n I n v a s i v e F e t a l E C G
T h
o r a x
H a n d O u t li n e s
G un P
o
i n t
E C G F i v e D a y s
S y n t
h e t
i c C o n t r o l
S o n y A
I B O
R o b o t S
u r f a c e
P r o x i m a l P h a l a n x O u t li n e C
o
rr
e c t
S t r a w b e rr y C B F C a r C o ff ee T
r
a c e
T w o P a tt e r n s P l a n e
T w o L e a d E
C G W a f e r
S y m b o l s
S t a r L i g h t
C u r v e s F o r d
A
T o e S e g m e n t a t i o n
S w e
d
i s h
L e a f
S o n y A
I B O
R o b o t S
u r f a c e
M i dd l e P h a l a n x O u t li n e
C o rr e c t
E l e c t r i
c
D e v
i c
e s F
i s h
E C G 5000 M a ll a t
A d i a c
L a r g e K i t
c h e n A pp li a n c e s
I n li n e S k a t e
T o e S e g m e n t a t i o n
D i s t a l P h a l a n x O u t li n e
C o rr e c t
M i dd l e P h a l a n x O u t li n e
A g e G r o u p F o r d B
D i s t a l P h a l a n x O u t li n e
A g e G r o u p E C G
P r o x i m a l P h a l a n x O u t li n e A g
e G r o u p
M i dd l e P h a l a n x T W
I t a l y P o w e r D
e
m a n d W o r m s M e a t F a c e A ll
L i g h t n i n g
D i s t a l P h a l a n x T W O S
U L e a f
A rr o w H e a d P h
o n e m e
P r o x i m a l P h a l a n x T W F a c e F o u r
E a r t
h q u a k e s
R e f r i g e r a t i
o n D e
v i
c
e s W i n e
L i g h t n i n g
C i n C E C G T
o r s o H a p t
i c s H e rr i n g
S m a ll K
i t c h e n A pp li a n c e s H a m B ee f
C o m p u t e r s
B i r d C h
i c k e n
S c r ee n T y p
e
W o r m s T w o
C l a ss
B ee t l e F l y
S h a p e l e t S
i m
Dataset name 0.8 0.
A cc u r a c y r a n g ea cc u r a c y o f O S - C
NN
[Figure]
D i a t o m S
i
z
e R e d u c t
i
o
n F i f t y W
o r d s
W o r dS y n o n y m s Y
o g a
M e d i c a l I m a g e s
F a c e s U C R
P h a l a n g e s
O u t li n e s C o rr e c t
S h a p e s A ll
H a n d O u t li n e s
P r o x i m a l P h a l a n x O u t li n e C
o
rr
e c t C a r P l a n e
S y m b o l s
S w e
d
i s h
L e a f
M i dd l e P h a l a n x O u t li n e
C o rr e c t F i s h
A d i a c
D i s t a l P h a l a n x O u t li n e
C o rr e c t
M i dd l e P h a l a n x O u t li n e
A g e G r o u p
D i s t a l P h a l a n x O u t li n e
A g e G r o u p
P r o x i m a l P h a l a n x O u t li n e A g
e G r o u p
M i d d l e P h a l a n x T W W o r m s F a c e A ll
D i s t a l P h a l a n x T W O S
U L e a f
A rr o w H e
a
d
P r o x i m a l P h a l a n x T W F a c e F o u r H
e rr i n g
B i r d C h
i c k e n
W o r m s T w o
C l a ss
B e e t l e F l y C r
i c k e t X
U W a
v e G e s t u r e L i b r a r y Y
U W a
v e G e s t u r e L i b r a r y A ll
C r i c k e t
Y C
r
i c k e t Z
U W a
v e G e s t u r e L i b r a r y X
U W a
v e G e s t u r e L i b r a r y Z
G un P
o
i n t
T o e S e g m e n t a t i o n
I n li n e S k a t e
T o e S e g m e n t a t i o n 1 H a p t i c
s O li v e O il
I n s e c t W i n gb e a t S o un d
N o n I n v a s i v e F e t a l E C G
T h
o r a x
C h l o r i n e C o n c e n t r a t i o n
M o t e S t r a i n
N o n I n v a s i v e F e t a l E C G
T h
o r a x
E C G F i v e D a y s
S o n y A
I B O
R o b o t S
u r f a c e
S t r a w b e rr y
T w o P a tt e r n s C o ff e e T r a c e W a f e r
S t a r L i g h t
C u r v e s F o r d
A
S o n y A
I B O
R o b o t S
u r f a c e
E l e c t r i
c
D e v
i c
e s
E C G
L a r g e K i t
c h e n A pp li a n c e s F o r d B E C G
I t a l y P o w e r D
e
m a n d M e a t
L i g h t n i n g
P h
o n e m e
E a r t
h q u a k e s
R e f r i g e r a t i
o n D e
v i
c
e s W i n e
L i g h t n i n g
C i n C E C G T
o r s o
S m a ll K
i t c h e n A pp li a n c e s H a m B ee f
C o m p u t e r s
S c r ee n T y p
e
S y n t
h e t
i c C o n t r o l C B F
T w o L e a d E
C G M a ll a t
S h a p e l e t S
i m
Dataset name 0.8 0.
A cc u r a c y r a n g ea cc u r a c y o f O S - C
NN
[Figure]
A. T HE CD - DIAGRAM RESULT
The critical difference diagram shows the average rank of each method with Wilcoxon-Holm posthoc analysis between each series.
A. E XAMPLES OF THE TWO PHENOMENA
In the Figure 2, the Google speechcommands dataset is selected as the dataset to show the example.
This is because, for this dataset, the relationship between performance and receptive ﬁeld size is proportional (As it is shown in Figure 16).
Thus, it is easy to control variables.
What’s more, in
Figure 18 and Figure 17, we show those two phenomena with more train and test split.
A. E XTEND OS- BLOCK WITH OTHER STRUCTURES
Layers in the OS-block and the OS-block itself are easy to extend with other complicated structures.
Figure 19 gives an explanation about the how to view the multi-kernel layers in OS-block as a single layer, and gives an example that how to combine the layer with dilation.
The Figure 4 shows that how to view the OS-block as a layer, and gives another two examples rather than OS-CNN.
[Figure] [Figure]
[Figure] [Figure]
A. E XPERIMENT RESULT OF OS- BLOCK WITH OTHER STRUCTURES
The Figure 20 and Figure 21 show that applied OS-block with residual connection, ensemble, and multi-channel architectures (individually or together) could further improve the performance.
The evaluation was on both UCR 85 and UEA 30 archives which contain datasets from different domains such as electrical devices analysis, Spectrum analysis, trafﬁc analysis, EEG analysis.
C OMPARE THE OS- BLOCK WITH OTHER
Mathematically, ﬁnding the optimal kernel conﬁguration is challenging, for it is a constrained combinatorial optimization searching for the best conﬁguration among an exponential number of candidates.
Our contribution is a simple and effective model design that does not need to solve the complex optimization problems and achieves state-of-the-art performance on several benchmarks.
From the model size perspective, using prime numbers is more efﬁcient than using even numbers or odd numbers.
To be speciﬁc, to cover RF of range r, the model size complexity of using prime size kernels is O ( r 2 /log
( r )) .
On the other hand, no matter we use even number pairs or odd number pairs, kernel sizes in each layer, the model size complexity of using the sequence is O ( r 2 ) .
As
Table 2 shows, compared with using odd number pairs or even numbers pairs prime numbers can achieve similar performance in a smaller model size.
[Table]