Paper: CNN Kernels Can Be the Best Shapelets (ICLR-2024)

Reviewer comment (Experiments):
The paper lacks a comprehensive review of related work, and the selection of competitor approaches for comparison is odd. The authors focus exclusively on optimization-based shapelet approaches. While space is limited, notable methods like Random Shapelet Forest and standard shapelet transform should not be omitted. Dictionary-based and interval-based approaches are also relevant and have achieved state-of-the-art performance in time series classification, yet they are not mentioned. Furthermore, the competitor models used in the experimental section are largely transformer-based or rely on embeddings, making for an unusual selection. It is recommended that the authors thoroughly review relevant literature on time series classification, such as the paper by Ruiz et al. (2021) and models like ROCKET by Dempster et al. (2020).

Author response:
Thank you for the suggestions. 

First, one kind of baseline models we have included in the Experiment section are *shapelet related methods, which includes not only optimization-based shapelet approaches*, but also the selection and generative based methods. The standard shapelet transform is denoted IGSVM in the table (Table F.1) and CD plot (Figure 3) according to the name in its paper. Besides, standard shapelet transform and its variants have also been mentioned in our related work like that: *Traditional practice is to search the raw datasets with some speed-up strategies, like paralleling computing (Chang et al., 2012), SAX transformation(Rakthanmanon & Keogh, 2013) and procedure simplification through newly designed measurements (Lines et al., 2012; Guillaume et al., 2022; Zakaria et al., 2012).*

Second, other two kinds of baselines are common deep learning methods (MLP, CNN, RNN) and state-of-the-art methods (TS2Vec, TST, DTW, etc), which covers various types of neural networks.

Third, we have added ROCKET as a baseline in the UCR dataset comparison upon your request. However, we also want to mention that ROCKET is already compared with some of our baselines, like TST and OS-CNN, in their papers. Since we are comparing with the state-of-the-art methods in recent two years, and many previous methods like HIVE-COTE and BOSS are already compared in our baselines' papers, we believe that superior performance over these state-of-the-art methods can well demonstrate the effectiveness of our model.

Finally, we have also updated our related work section by introducing non-deep-learning approaches like the dictionary-based method MUSE and the ensemble method HIVE-COTE.

Reviewer comment (Experiments):
Parts of the experimental section are unclear and require further clarification. Specific questions include: Is the "Initialization" phase's cost included in the runtime? In Table 1, why do the methods differ with respect to the cd plots? Why is the evaluation limited to 25 UCR datasets, and what was the criteria for selection? Several state-of-the-art methods like Rocket, CIF, ShapeletTransform, and MUSE are absent from the comparison.

Author response:
Yes, it is included. We have clarified this in Table 1.

In the CD plot (Figure 3), ShapeConv is compared with shapelet-based methods on 25 UCR datasets, and full results here are shown in Table F.1 in Appendix. In Table 1, ShapeConv is compared with state-of-the-art time-series classification models on 125 UCR and 29 UEA datasets, and the full results are shown in Table F.3 and F.4. Methods are different since they are two different experiments.

We use the subsets (25 UCR datasets) in the CD plot because the baseline (ADSN) only reported results on 25 UCR dataset, and the results of baselines are taken directly from the respective papers. Full results of ShapeConv and state-of-the-art classification methods on all 128 UCR datasets are shown in Table F.3 in Appendix.

We have added Rocket as a baseline in Table 1 and F.3 upon your request, and the ShapeletTransform (namely IGSVM) is already compared in the paper. We found the results of CIF and MUSE in [1], and the average accuracy on the 109 subset of UCR is as follows:

|  | ShapeConv | CIF | MUSE |
| - | - | - | - |
| Average Accuracy  | 0.856 | 0.846 | 0.845 |

All these results well demonstrate the effectiveness of ShapeConv.

Besides, we would like to emphasize that ShapeConv not targets on beating all state-of-the-art methods. It offers great benefits on interpretability and is able to seamlessly integrate with other deep learning models.

[1] Tan, C.W., et al. MultiRocket: multiple pooling operators and transformations for fast and effective time series classification. Data Min Knowl Disc 36, 1623–1646 (2022).

Reviewer comment (Evaluation):
The XAI evaluation is restricted to examples selected by the authors. While the author-selected examples support the paper's claims, they do not suffice to demonstrate the superiority of ShapeConv in terms of shapelet quality. Additionally, pairing ShapeConv with MLP or SVM models does not provide sufficient interpretability. It is suggested to test the approach with tree-based or linear models, or employ explainers such as SHAP to determine the importance of shapelets, especially in supervised tasks.

Author response:
We thank the reviewer for you insightful comments. The quality of shapelet, as well the quality of all the other interpretable methods, should be judge from two different perspectives: the effectiveness and the interpretability.

As for the effectiveness, we tested our model on a bunch of datasets compared to several baselines, and the experiment results have demonstrated the superiority in Figure 3, Table 1, Table D.1, Table F.1~F.6.

As for the interpretability, we have to admit that the evalution itself is a very tough research question in the interpretable time-series modeling task. However, we still designed the following approaches to verify that our methods are truly interpretable:
1. Figure 4, 5, 6, C.1 compared the shapelets learned by ShapeConv and the state-of-the-art learning-based shapelet baseline, LTS, and our shapelets are apparently fit better with the data. 
2. Appendix A provides a detailed explanation of the learned shapelets in a human-perceivable way on a real-world dataset, GunPoint, to demonstrate how these shapelets can be interpreted.

And we argue that pairing ShapeConv with MLP or SVM, and even deep models demonstrated in Appendix D, is designed to obtain better effectiveness and shouldn't harm the interpretabilty. This is because ShapeConv serves as a feature extractor conducting shapelet transform which directly engaged with the raw time series, and is sensitive to the difference of shapes. This idea first comes as shapelet transform [2] and has become a common practice followed by the following research works. As a matter of fact, all the shapelets visualized and analyzed are trained in an end-to-end manner and we didn't see any negative impact for interpretability.

Moreover, we added the following contents to the revised manuscript:
1. In Appendix D., we further investigate the interpretability of shapelets learned by ShapeConv for the real-world EEG classification task, and we excitedly found that some shapelets are very similar to textbook waveforms which have already been recognized in clinical practice. 
2. In Appendix E.1, we add more visualization examples of learned shapelets.
3. In Appendix E.2, we add visualizations of the SHAP values for ShapeConv and its variant without the shape regularizer (essentially a CNN with our initializations and diversity loss). The significant regions labeled by the SHAP value in the ablation overlaps with the determining regions of the ShapeConv, verifying the quality of the shapelets. Furthermore, ShapeConv excels in not only identifying the significant regions of the input, but also captures the shape of the discriminative subsequence. This capability sets it apart from traditional explainability methods like SHAP, which primarily focus on pinpointing the locations of these areas.

Please kindly refer to the revised version of our paper for details and figures.

[2] Jason L., et al. A shapelet transform for time series classification. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 289–297, 2012.

Reviewer comment (Reproducibility):
There is no discussion or citation concerning code implementation.

Author response:
The code implementation is included in the supplementary material.

Reviewer comment (Theory):
In the abstract (also reiterated in the introduction in other words), the authors state: "In this paper, we demonstrate that shapelets are essentially equivalent to a specific type of CNN kernel with a squared norm and pooling." This demonstration is not novel, as it is for example stated in Lods et al. 2017 (which is cited in the paper). While the proof here seems to aim for more rigor, Theorem 3.1 is not successful in this regard since it completely disregards the fact that the bias term in convolution is independent of the input, which is not the case in the $-\mathcal{N}(s_i, X_{j:j+l_s-1})$ term.

Author response:
We would like to emphasize that the bias term in convolution has no relation with the $-\mathcal{N}(\mathbf{s\_i}, \mathbf{X}\_{j:j+l\_s-1})$ term. The bias term, as we have explained in the end of Sec. 3.1, can be optionally added after the features are extrated with the shapelets (or CNN kernels). And the $-\mathcal{N}(\mathbf{s_i}, \mathbf{X}_{j:j+l_s-1})$ term is an additional squared L2 norm term added in ShapeConv besides the cross-correlation operator in CNN. The equivalence cannot be realized without this additional squared L2 norm. Therefore, the bias term do not influence the proof of equivalence, so that Theorem 3.1 is successful in the rigor equivalence between CNNs and shapelets.

Reviewer comment (Theory):
In Theorem 3.1, Squared Euclidean distance is used, not Euclidean distance as stated.

Author response:
Thanks for pointing out. We have revised our manuscript.

Reviewer comment (Novelty):
The review of the Related Work is very succinct, and a more thorough presentation of competing interpretable Shapelet-based methods would have been a plus.

Author response:
For the visualizations, we did not choose "toy examples" - the ShapeConv in Fig 4 could achieve 0.75 accuracy on the Herring dataset with 2 shapelets. When the number of shapelets are large, the model tends to overfit for these simple datasets. As for examples with large number of shapelets, we provided a real-world application of ShapeConv in EEG (Appendix D) where 2,688 shapelets are trained. Two of them are visualized in Figure D.1, and they matched nicely with the EEG examples from textbook.

We also provided more visualizations in the appendix E, showing learnt shapelets in 4 additional datasets from UCR: DodgerLoopWeekend, SonyAIBORobotSurface1, ItalyPowerDemand, BME. ShapeConv captures the determinative regions in all of them.

In terms of interpretability, ShapeConv is generally shapelet-based method, so its interpretability inherits from shapelet. We have compared the interpretation results of ShapeConv with the other shapelet-based method (i.e., LTS) via visualizations, and results shows that ShapeConv is better at capturing discriminative sub-sequences, thus having better interpretability.

Reviewer comment (Evaluation):
A more detailed comparison of the interpretability of the ShapeConv model with the baselines is required to fully assess interpretability.

Author response:
For the visualizations, we did not choose "toy examples" - the ShapeConv in Fig 4 could achieve 0.75 accuracy on the Herring dataset with 2 shapelets. When the number of shapelets are large, the model tends to overfit for these simple datasets. As for examples with large number of shapelets, we provided a real-world application of ShapeConv in EEG (Appendix D) where 2,688 shapelets are trained. Two of them are visualized in Figure D.1, and they matched nicely with the EEG examples from textbook.

We also provided more visualizations in the appendix E, showing learnt shapelets in 4 additional datasets from UCR: DodgerLoopWeekend, SonyAIBORobotSurface1, ItalyPowerDemand, BME. ShapeConv captures the determinative regions in all of them.

In terms of interpretability, ShapeConv is generally shapelet-based method, so its interpretability inherits from shapelet. We have compared the interpretation results of ShapeConv with the other shapelet-based method (i.e., LTS) via visualizations, and results shows that ShapeConv is better at capturing discriminative sub-sequences, thus having better interpretability.

Reviewer comment (Experiments):
Only toy examples are presented (e.g., Fig 4: 2 shapelets), and it is unclear what results would be when training with a large number of shapelets.

Author response:
For the visualizations, we did not choose "toy examples" - the ShapeConv in Fig 4 could achieve 0.75 accuracy on the Herring dataset with 2 shapelets. When the number of shapelets are large, the model tends to overfit for these simple datasets. As for examples with large number of shapelets, we provided a real-world application of ShapeConv in EEG (Appendix D) where 2,688 shapelets are trained. Two of them are visualized in Figure D.1, and they matched nicely with the EEG examples from textbook.

We also provided more visualizations in the appendix E, showing learnt shapelets in 4 additional datasets from UCR: DodgerLoopWeekend, SonyAIBORobotSurface1, ItalyPowerDemand, BME. ShapeConv captures the determinative regions in all of them.

In terms of interpretability, ShapeConv is generally shapelet-based method, so its interpretability inherits from shapelet. We have compared the interpretation results of ShapeConv with the other shapelet-based method (i.e., LTS) via visualizations, and results shows that ShapeConv is better at capturing discriminative sub-sequences, thus having better interpretability.

Reviewer comment (Experiments):
Providing visualization for a large number of datasets instead of only GunPoint, Herring, and ECG200 would be a real plus.

Author response:
For the visualizations, we did not choose "toy examples" - the ShapeConv in Fig 4 could achieve 0.75 accuracy on the Herring dataset with 2 shapelets. When the number of shapelets are large, the model tends to overfit for these simple datasets. As for examples with large number of shapelets, we provided a real-world application of ShapeConv in EEG (Appendix D) where 2,688 shapelets are trained. Two of them are visualized in Figure D.1, and they matched nicely with the EEG examples from textbook.

We also provided more visualizations in the appendix E, showing learnt shapelets in 4 additional datasets from UCR: DodgerLoopWeekend, SonyAIBORobotSurface1, ItalyPowerDemand, BME. ShapeConv captures the determinative regions in all of them.

In terms of interpretability, ShapeConv is generally shapelet-based method, so its interpretability inherits from shapelet. We have compared the interpretation results of ShapeConv with the other shapelet-based method (i.e., LTS) via visualizations, and results shows that ShapeConv is better at capturing discriminative sub-sequences, thus having better interpretability.

Reviewer comment (Experiments):
If the ShapeConv model (with the exact same initialization, regularization terms, etc.) were modified to replace the ShapeConv layer with a convolutional one, it is unclear what the performance would be. This experiment is required to fully assess if the norm terms are really helpful.

Author response:
We anticipate your proposed ablation is to drop the squared $L_2$ norm term $\mathcal{N}(\mathbf{s_i}, \mathbf{X'})$ from Equation (4) in the loss. However, then the shape regularizer in the loss would become 
$$
\mathcal{R}\_{shape}=\frac{1}{n_{out}}\sum_{i=1}^{n_{out}}  \min_{\mathbf{x}\in\hat{\mathbf{X}}_i} -2Y(\mathbf{s_i},\mathbf{x})
$$
where $Y(\mathbf{s_i},\mathbf{x})$ denotes the cross-correlation between kernel $\mathbf{s_i}$ and subsequence $\mathbf{x}$. Minimizing this regularizer would make the loss diverge and goes to negative infinity. This is because without the norm terms, the regularizer could not be interpreted as a distance and is not always nonnegative. 

If we do not use the shape regularizer, the formulation would become similar with the CNN baseline. We tested the ShapeConv without Shape Loss (by setting the $\lambda_{shape}$ in Equation (8) to zero). The results for the GunPoint dataset is illustrated in appendix E.2. We found that its kernel is not interpretable and it performs worse than ShapeConv (it constantly overfits the training set). The result performance is similar with the CNN baseline. We did not include the full results in the paper due to space constraints.

Reviewer comment (Reproducibility):
How are baseline model hyperparameters tuned, and which parameters are tuned?

Author response:
The results of the baseline models are taken directly from their papers. We have clarified this in the paper.

Reviewer comment (Experiments):
How are the datasets for the subsets (25 datasets for supervised learning and 36 datasets for unsupervised learning) selected?

Author response:
We choose these subsets following the baseline methods (ADSN for supervised and AutoShape for unsupervised) which only reported the results on these dataset. We have clarified this in the paper.

Reviewer comment (Experiments):
If the goal is to compare to state-of-the-art methods, other competitors such as ROCKET, COTE & variants should be included in the comparison.

Author response:
We have added comparison with ROCKET in Appendix Table F.3. COTE is already compared with the baseline method ADSN in their paper. We have not included these traditional methods in the main paper, but lean towards state-of-the-art deep-learning-based methods for the following reasons: (1) DL-based methods generally have more competitive performance; (2) ShapeConv can be generally regarded as DL-based methods, so both of them can be paralleled on GPU, achieving much faster runtime than traditional methods on CPU; (3) In terms of model expressiveness, comparing ShapeConv with DL-based methods is more reasonable, since both of them are neural network models. (3) Since our model is written in the PyTorch framework and could automatic differentiate, our method can act as a feature extractor and seamlessly integrate with other DL-based methods.

Reviewer comment (Writing):
In Section 1, the authors write: "they are more likely to overfit when the signal-to-noise ratio is relatively large." This seems like it should say "is relatively low."

Author response:
Thank you for pointing out. We have fixed the typo.

Reviewer comment (Experiments):
Have the authors assessed how important it is to use supervised information at initialization?

Author response:
Yes, we have done these two studies. First, we have the random initialization ablation in Figure 3 and Table F.1. Random initialization leads to large performance degeneration, indicating the importance of our designed initialization methods and supervised information at initialization.

Besides random initialization, we have also attempted many different ways. Your proposed method is similar with our unsupervised initialization approach - the only difference is that we first divide the input sequence into different regions and then do the sampling and k-means. Moreover, we have also tested your proposed method before and it was not as good as our current method. For supervised learning, such simpler approaches do not incorporate important supervised information. For unsupervised learning, pre-dividing according to regions makes clustering easier and contributes to better initialization. Therefore, we end up choosing the best tested approach as our current method.

Reviewer comment (Experiments):
Have the authors tried simpler initialization approaches such as kmeans++ on randomly selected subsequences of adequate length?

Author response:
Yes, we have done these two studies. First, we have the random initialization ablation in Figure 3 and Table F.1. Random initialization leads to large performance degeneration, indicating the importance of our designed initialization methods and supervised information at initialization.

Besides random initialization, we have also attempted many different ways. Your proposed method is similar with our unsupervised initialization approach - the only difference is that we first divide the input sequence into different regions and then do the sampling and k-means. Moreover, we have also tested your proposed method before and it was not as good as our current method. For supervised learning, such simpler approaches do not incorporate important supervised information. For unsupervised learning, pre-dividing according to regions makes clustering easier and contributes to better initialization. Therefore, we end up choosing the best tested approach as our current method.

Reviewer comment (Evaluation):
If ShapeConv is faster than LTS, it is probably more an artifact of the implementation since the overall complexity of ShapeConv is likely higher than that of LTS (similar local representation extracted, but ShapeConv has additional loss terms that induce more computations).

Author response:
Our method is faster than LTS is because our model is paralleled on GPU, while LTS is implemented on CPU (we tested against the LTS implementation in [tslearn](https://github.com/tslearn-team/tslearn/)). Additional loss terms are negligible in terms of computation during runtime compared to the convolution operation. While the theoretical computational complexity is almost the same, being able to compute forward pass and gradient in parallel makes ShapeConv much faster and better at learning shapelets.

Reviewer comment (Reproducibility):
Though the model's performance is promising, concerns may arise regarding the complexity of implementing ShapeConv compared to other traditional or deep learning models.

Author response:
ShapeConv is implemented as a Module in PyTorch similar to a common CNN layer, which can be easily integrated as a layer in any deep learning model. The implementation of ShapeConv is available at the supplementary material. Besides, the implementation of ShapeConv itself is simple and intuitive, which contains the convolution layer and some additional regularizers, aligning with Eq. (4) in Theorem 3.1.

Reviewer comment (Experiments):
This study opted for a combination of CNN and Shapelets to enhance interpretability while also boosting performance. For time series classification tasks, why not choose the stronger baseline models for research, such as RNN or Transformer?

Author response:
RNN and Transformer-based methods are already included in our baselines. Table F.2 shows results of ShapeConv and various RNN-based methods, and we also compared with Transformer-based model, TST [1], in Table 1 and appendix Table F.3. ShapeConv outperforms these methods on most of the tasks. These methods are not listed in the main paper since we only include most related and competitive methods due to the limited spaces.

[1] George Z., et al. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2114–2124, 2021.

Reviewer comment (Novelty):
The main contribution is based on Theorem 3.1, which shows that the shapelet transform is somewhat equivalent to a convolution layer followed by a max pooling operation. However, this fact has been observed previously to provide accelerated shapelet transform: the authors of [1] show that computing the distance profile ($dist(\mathbf{s}, \mathbf{x})$ for a given sequence $\mathbf{s}$ and all subsequences $\mathbf{x}$ of $\mathbf{X}$) is equivalent to a convolution.

Author response:
We argue that our work is definitely different from the work in [1] despite some faint connections. The authors of [1] discuss the connection between z-normalized Euclidean distance and dot products so as to utilize convolution techniques such as the fast Fourier transform (FFT) to achieve speed-ups in searching similar subsequences. The method are mentioned to "*suggest*" candidate shapelets as *indicators*, and barely limited to sub-sequences of the original time series which cannot meet the needs of deep learning. We, to be noticed, rigorously prove the equivalence between a certain neural convolutional layer (not just convolution) and the originally defined shapelet in order to propose a newly interpretable and learnable neural kernel for deep learning. 

In fact, we have properly cited the papers of [2] and [3] (also discuss the relationship between convolution and shapelet) which are more relevant and compared them with our work to highlight our contribution of demonstrating the strict equivalence between shapelet transform and a certain learnable neural operator. With GPUs’ built-in parallel computation of for deep learning, our ShapeConv is naturally high-speed and does not need extra algorithms specifically designed for acceleration. Besides, we also put forward the diversity regularization and incorporate human-knowledge-based initialization to enhance the interpretability of our proposed method as non-negligible contributions.

[1] Yeh, C. C. M., et al. (2016). Matrix Profile I: All Pairs Similarity Joins for Time Series: A Unifying View that Includes Motifs, Discords and Shapelets. Proceedings of the IEEE International Conference on Data Mining (ICDM), 1317–1322.

[2] Arnaud L., et al. Learning dtw-preserving shapelets. In Advances in Intelligent Data Analysis XVI: 16th International Symposium, IDA 2017, London, UK, October 26–28, 2017, Proceedings 16, pp. 198–209. Springer, 2017.

[3] Yichang W., et al. Learning interpretable shapelets for time series classification through adversarial regularization. arXiv preprint arXiv:1906.00917, 2019.

Reviewer comment (Experiments):
Convolutional layers are meant to be stacked. Unless I am mistaken, in the experiments, there is only one ShapeConv layer. Would the interpretability of ShapeConv remain if there are several layers?

Author response:
Thanks for your insightful comments! It's true that our ShapeConv can be stacked just like normal convolutional layers to support deep learning. We discussed how we treat ShapeConv as interpretable feature extractor and can be further combined with more sophiscated deep model in Appendix D, and demonstrate its superior performance compared to not using it on large-scale EEG dataset. 

As for the interpretability, we have added more investigations. Breifly speaking, we have excitedly discovered that a few shapelets are highly similar to waveforms in clinical textbooks. By aligning shapelets with data and conduct shapelet transform, we see clear separatable clusters. The overall experiments in Appendix D demonstrate the possibility of ShapeConv, when stacking with deep models, can benefit medical practitioners in practice by not only offering an accurate judgement, but also pointing out the area of interests with respect to their expert knowledge. **Please kindly refer to the revised version of our manuscript, Appendix D, for figures and more details.**