Paper: A Self-Supervised Pre-Training Model for Time Series Classification based on Data Pre-Processing (ICLR-2024)

Reviewer comment (Presentation):
The presentation of the paper is so unclear that one cannot fully grasp the main take-away message from the paper.

Author response:
Regarding the issues you raised about the expression of the article, I sincerely accept the criticism because my English is not good.

Reviewer comment (Writing):
The Introduction is close to non-informative.

Author response:
Regarding the issues you raised about the expression of the article, I sincerely accept the criticism because my English is not good.

Reviewer comment (Writing):
The notations in Section 3 are confusing. For example, the text discusses multivariate time series datasets (each dataset comes with a number of channels) and then suddenly states: "Data from multiple channels need to be analyzed together, we focus only on data from a single channel", yet in the experiments some datasets are multivariate.

Author response:
Lastly, regarding the issue of multivariate channels, we did mention in the article that it is applicable to multivariate sequences. However, we did not specifically mention methods for fusing multiple channels. We can use methods such as $\sqrt{x_{1i}^2+x_{2i}^2+\cdots+x_{ni}^2}$ to fuse data from multiple channels, etc.

Reviewer comment (Writing):
Algorithms 1 and 2 lack rigor. For example, in Algorithm 1, it is not clear whether \( i \) is the index of a time series (as it seems to be in \( x_i \)) or a time instant (as it seems to be in the slicing \( i:i+winsize \)).

Author response:
Secondly, you mentioned that the symbol $x_i$ is ambiguous in section 3 of the article. Here, $i$ represents the sample index. It is indeed a mistake on our part. In line 2 of Algorithm 1, it should be $winsize=ceil(max(x_i)-min(x_i))/2$ or $ceil(max(x'_i)-min(x'_i))/2$.

Reviewer comment (Theory):
The NT-Xent description is not clear enough. The authors write "inspired by the EMD method (Boudraa & Cexus, 2007), we sort the time series and calculate the Hausdorff distance between two time series, which preserves the time attribute while retaining the graphic features of the time series." However, upon reviewing the (Boudraa & Cexus, 2007) paper, it appears that it does not mention sorting the time series nor using the Hausdorff distance between them; instead, it focuses on decomposing the signal.

Author response:
Q1:
The similarity measurement is based on EMD for two main reasons. Firstly, in our actual projects, we use EMD decomposition to calculate the energy of waveforms. However, the decomposed waveforms are relatively smooth and lose some details, while real waveforms are generally complex in shape. Therefore, we thought of sorting time series and directly comparing their similarity. Secondly, considering that similar waveforms within the same category have higher similarity after sorting, we use the Hausdorff distance, which calculates the maximum distance between two waveforms. Sorting is equivalent to organizing the waveforms in advance, and the loss function in the fine-tuning stage requires refinement.

Reviewer comment (Novelty):
The novelty is quite unclear, since the paper relies on a well-established method (Zhang et al. 2022) and slightly modifies it by adding a pre-processing step and changing the similarity measure in the loss. Since these two points constitute the novelty of the paper, they should be motivated, discussed, and illustrated in many more details than what is done in the current version.

Author response:
Firstly, we humbly accept suggestions regarding the novelty of the article. However, the focus of the method proposed in this paper is on the data filtering part. Most existing articles mainly evaluate the generalization ability of models, while neglecting the weaknesses of the models themselves. If we only consider improving the models themselves, all current works have done very well. However, the field of time series analysis lacks an excellent data preprocessing method, which should be effective for the majority of data.

Reviewer comment (Presentation):
What representation is used in Figure 3? Is it tSNE? MultiDimensional Scaling? Something else?

Author response:
Q2:
We use the fit_transform function in the Sklearn library to convert the high-dimensional features of each waveform into a two-dimensional coordinate system, and plot the resulting graph, which is similar to the t-SNE method you mentioned.

Reviewer comment (Experiments):
Why do you evaluate on only 4 pairs of datasets while timeseriesclassification.com hosts many more datasets, arranged in categories?

Author response:
Q3:
We evaluated four sets of data from representative fields, and of course, other data can also be evaluated. For example, traffic data has a lower sampling rate and cannot reflect the advantages of our method in this paper. Therefore, we selected data with a higher sampling rate for evaluation.

Reviewer comment (Presentation):
In Figure 2, it would be better to highlight modified parts of Zhang et al's model. Particularly, it is unclear if CNN is part of the modification or the original model. If CNN is part of the modification, then its applicability on the transfer learning empirical study is incomplete.

Author response:
We apologize for the incomplete explanation in the article. The CNN module was added based on the research by Zhang et al., and it was introduced to mitigate the impact of abrupt changes in the waveform. If we consider it incomplete, we should include ablation experiments to verify the impact of adding or not adding the CNN module on the experimental results.

Reviewer comment (Experiments):
It is good that the authors use Hausdorff distance to preserve time attributes and retain graphical features. However, there are various other distance measures such as Bhattacharya distance, or EMD method, that could also be used instead. It would be good if the authors empirically show evidence why Hausdorff was chosen.

Author response:
Hausdorff distance is a method for calculating the maximum distance between two time series. Since this article uses the sorting similarity calculation method, using Hausdorff distance allows us to calculate the distance between time series with higher sorted similarity, which better reflects the distinguishing details between time series, rather than using cosine similarity or other distance calculation methods.

Reviewer comment (Experiments):
From Table 2, it is clear that DPTCS is better than the TF-C method that the authors have modified. However, the original Zhang et al., 2022 article evaluates one-to-many domain adaptation which brings out areas where other competing models perform better than TF-C. This key evaluation is missed by the authors in the current submission.

Author response:
I believe this evaluation is not necessary because we compared the accuracy of TF-C with our model in the experiment, and we also conducted comparative experiments in the ablation study section. If we compare the accuracy from the ablation study with TF-C, it can also demonstrate the advantages of our method.

Reviewer comment (Presentation):
In section 4.1, is Figure 5 incorrectly specified as algorithm 1?

Author response:
There is no error specification in Figure 5. The waveform in Figure 5 (a) and (b) does not have a platform part, so the waveform after processing is consistent with the original shape. The waveforms in (c), (d), (e), and (f) can demonstrate the effect of platform filtering.

Reviewer comment (Writing):
In algorithm 2, what is IQR?

Author response:
IQR (Interquartile range) refers to the concept of quartile range in statistics.

Reviewer comment (Writing):
In the introduction, what is NT-Xent loss?

Author response:
NT-Xent loss is a commonly used loss function in contrastive learning (from A Simple Framework for Contrastive Learning of Visual Representations by Ting Chen et al.). The formula is as follows.

$l_{i,j}=-log\frac{exp(sim(z_i,z_j)/\mathbb{T})}{\sum_{k=1}^{2N}[k\neq i]exp(sim(z_i,z_k)/\mathbb{T})}$

Here, the batch contains N samples, and there are $2N$ samples because there are two branches. Except for the augmented image corresponding to itself, the other $2N-2$ should be regarded as negative pairs. In the formula, i and j are positive pairs, and the denominator is negative pairs.

NT-Xent looks like a softmax function. This is because it adds vector similarity and temperature normalization factors. The similarity function is just cosine distance. Another difference is that the value in the denominator is the signed distance from positive examples to negative examples. It is not much different from CrossEntropyLoss. The intuition is that we want our similar vectors to be as close to 1 as possible because $-log(1) = 0$, which is the best loss. We want negative examples to be close to 0 because any non-zero value will reduce the value of similar vectors.

Reviewer comment (Novelty):
There are a few self-supervised contrastive learning-based models for time series classification as mentioned in Section 2.2. However, their limitations (and differences with the proposed work) have not been clearly discussed. There are several sentences at the end of Section 2.2 on this point but they are vague and difficult to follow. It is difficult to evaluate the novelty of the proposal.

Author response:
The method proposed in this article focuses on the data filtering part. Most existing articles mainly evaluate the model's generalization ability and ignore the weaknesses of the model itself. If we only consider improving the model itself, all current work has been done very well. However, the field of time series lacks an excellent data preprocessing method, and this method should be effective for most data.

Reviewer comment (Novelty):
"Our work follows the Zhang et al. (2022)’s work, We adopted his concept of time-frequency consistency, but made some modifications on the model. We embedded a CNN module behind the transformer": What exactly are the modifications? Just adding a CNN module after the transformer?

Author response:
Yes, we only added a 1convCNN module after the Transformer to weaken the abrupt changes in the time series. However, overall, the experimental results are still good.

Reviewer comment (Writing):
"Each sample consists of CT channels, and each channel contains |xi| data points:" What is a sample? A time series? What is a channel? What is a data point? And how are the |xi| data points of a channel formed?

Author response:
$C_\mathbb{T}$ represents all channels of a multivariate time series, where each channel is a time series $x_i$. The length of a time series (sample) is finite, and the number of data points is $|x_i|$. $\mathbb{T}$ represents a pre-training dataset, and $\mathbb{T'}$ represents a fine-tuning dataset. $\mathbb{T}$ does not have labels, while $\mathbb{T'}$ contains a small number of labels for fine-tuning the model.

Reviewer comment (Writing):
Where does "temp" in Algorithm 2 come from?

Author response:
In Algorithm 2, "temp" represents a sample extracted from $\mathbb{T}{step}$ and $\mathbb{T'}{step}$.

Reviewer comment (Experiments):
The experimental results are not too convincing. The URC time series repository has over 100 sets of time series. It is unclear how the 8 sets are chosen for the experiments.

Author response:
We evaluated 4 sets of representative data in different fields, and other data can also be evaluated, such as traffic data, which has a low sampling rate and cannot reflect the advantages of the method in this paper. Therefore, we selected data with a high sampling rate for evaluation.

Reviewer comment (Writing):
Statements like "The extensive use of time series in the industrial field (Bi et al., 2023)(Li et al., 2020)(Gupta et al., 2020) is beyond doubt." are too strong and may need to be tuned down.

Author response:
Modified to: Time series are widely used in the industrial field.

Reviewer comment (Writing):
The paper is difficult to read because of poor grammar. Too many sentences are hard to parse.

Author response:
We will improve the grammar and expression of the article, my English is really not good, thank you for your criticism!

Reviewer comment (Writing):
For the Methodology section 4.2 and 4.3, the data filtering algorithms are very briefly described, and some pseudocode is provided. It would be easier to understand the algorithms with some more details in the description. For instance, section 4.2 mentions “We use a FIR filter with a low-pass frequency designed to be adaptive. The value is set based on the maximum frequency of the current curve multiplied by sqrt(2)/2”. Although it is explained in the Appendix, a bit more explanation about what makes the filter adaptive and why the values sqrt(2)/2 is used would make this section in the main text more understandable.

Author response:
Setting the maximum frequency of $\sqrt{2}/2$ is mainly considered to be the golden ratio, using this value can remove most high-frequency noise, of course, in the actual situation, the use of other values is sometimes necessary.

Reviewer comment (Writing):
Some of the variables in the algorithms are not explained well. For example, section 4.3 mentions that “During the data platform filtering process, we need to make a reasonable division of the winscale value, otherwise overfiltering may occur”. An explanation of what the winscale value represents and a range of reasonable values would make this section stronger.

Author response:
Winscale is the size of a sliding window, the original text is on page 4 under Algo1.

“First, the algorithm calculates the maximum and minimum amplitudes in the data, and sets the sliding window size to half of the range. Based on the difference between the maximum and minimum amplitudes within the window, the algorithm determines whether to filter the current window.”

For the setting of winscale, the size of winscale is determined by half of the difference between the largest value and the smallest value of the time series, and half of the maximum and minimum value of the amplitude of the time series itself is mainly considered as the size of the sliding window. When the vibration amplitude of the time series is large, it indicates that there is not much of the waveform platform. At this time, we set a larger sliding window size. When the amplitude of the time series is small, it indicates that the probability of the waveform platform part being large becomes more and more, and then we set a smaller sliding window size.

Reviewer comment (Experiments):
The paper lists 8 datasets used in this work for evaluation and gives a description of each, but there is not much explanation for why these particular datasets were chosen. More detail about why these particular datasets were chosen would help this section. In the methodology, these 8 datasets are grouped into pairs where one dataset is used for pre-training and another is used for fine-tuning. After reading Table 1, it becomes clear that these datasets are paired by similar domains. However, it would be better if this was stated directly in the text instead of implied by Table 1.

Author response:
Yes, we quite agree with your suggestion and we will include a note in the text to show that the 8 sets of data are similar in the field.

Reviewer comment (Experiments):
The Ablation Study and its corresponding Figure 4 are not clear. It states that 7 combinations of DPTSC were compared, but it is not clear what these 7 combinations are, even after re-reading sections of the paper and appendix. The Figure 4 is also harder to read and understand compared to the other figures and tables in this work.

Author response:
Figure 4 shows the data preprocessing methods used in this paper, including platform filtering -D, FIR filtering -F and similarity modification -S, which were added or deleted for ablation experiments respectively. The original text reads as follows: “We compared 7 combinations of DPTSC, DPTSC-D, DPTSC-F, DPTSC-S, DPTSC-DF, DPTSCDS, and DPTSC-FS, and DPTSC-DFS by using data platform filtering as condition D, filtering as condition F, and sort similarity as condition S.”

The ablation experiment is a comparison of the previous data preprocessing methods. Finally, thank you again for your advice on our work.

Reviewer comment (Experiments):
The Ablation Study is vague, and it is unclear what it is comparing. This makes it difficult to see exactly which components in their pipeline are contributing to the improved results.

Author response:
Figure 4 shows the data preprocessing methods used in this paper, including platform filtering -D, FIR filtering -F and similarity modification -S, which were added or deleted for ablation experiments respectively. The original text reads as follows: “We compared 7 combinations of DPTSC, DPTSC-D, DPTSC-F, DPTSC-S, DPTSC-DF, DPTSCDS, and DPTSC-FS, and DPTSC-DFS by using data platform filtering as condition D, filtering as condition F, and sort similarity as condition S.”

The ablation experiment is a comparison of the previous data preprocessing methods. Finally, thank you again for your advice on our work.