Paper: Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approach (ICLR-2024)

Reviewer comment (Experiments):
The identification may not be accurate during the early stages of training when the model hasn't been well trained. It is possible that these errors propagate through the training process and accumulate.

Author response:
We prevent this problem by setting warm-up epochs. During the initial warm-up epochs, the model performs regular training and there is no downweighting for samples, ensuring stable model training. The downweighting is only activated after the initial warm-up epochs, thus minimizing the risk of error propagation due to imperfect identification at early stages.

Reviewer comment (Theory):
Since hard samples that are beneficial may also experience large losses, they could potentially be misidentified as faulty pairs. Could the authors explain how to avoid this problem?

Author response:
Avoiding potential misidentification of hard positive pairs as faulty pairs is an important point and we indeed take this challenge into account when designing DBPM. As discussed in Sec 4.2, hard positive pairs that are generated by strong augmentation while retaining their original identities can indeed benefit contrastive learning. 

Unlike faulty positive pairs, hard positive pairs are learnable, meaning that the model can still extract representative information shared between them. Therefore, hard positive pairs usually have relatively large losses at certain training iterations rather than suffering high losses over the entire training process. Pairs that constantly exhibit large contrastive losses during the entire training are more likely to be faulty positive pairs that are difficult for the model to learn. 

DBPM identifies bad positive pairs based on their historical training behaviors in a dynamic manner, which allows more reliable identification of potential bad positive pairs and reduces the chance of misidentification.

Reviewer comment (Experiments):
The identification of bad pairs is based on heuristically set thresholds. Could more principled statistical methods could be explored to automatically determine optimal thresholds?

Author response:
Thank you for your insightful comment. We totally agree that the principled statistical method in determining thresholds is important and offer a promising avenue for further investigations. At the current stage, DBPM is a simple and effective solution that aligns with our observations. More importantly, it provides a clear analysis of the newly recognized bad positive pair problem. By choosing proper thresholds, we have achieved meaningful improvements in the performance of contrastive learning models. 
	
Therefore, we plan to study more principled statistical methods in our future study. Besides statistical methods, another direction for future exploration is more sophisticated decision-making models (Reinforcement Learning) for automatically determining thresholds. 

We believe our work provides a strong baseline in the newly identified bad positive pair problem, and we anticipate that it will inspire more further research and explorations.

Reviewer comment (Experiments):
All evaluations were studied with the InfoNCE loss, does this approach generally work for other losses, for e.g., the supervised contrastive (SupCon) loss Khosla et. al, NeurIPS 2020.

Author response:
We would like to clarify that our experiments included evaluations on SimSiam, which employs the cosine similarity of positive pairs as its optimization objective, rather than InfoNCE loss. The results, as demonstrated in Tables 1 and 2, consistently indicate that our proposed DBPM enhances SimSiam's performance across various tasks.

In terms of supervised contrastive learning (SupCon), we did not include it in this study since the focus of the research is on self-supervised contrastive learning, where supervisory signals are derived from only contrasting views. The supervisory signals in supervised contrastive loss are derived from task-related labels, which may result in different observations and solutions than in self-supervised learning.

Reviewer comment (Experiments):
Did the authors conduct ablation experiments to distinguish the individual impacts of noisy versus faulty positive pairs on the model's performance? It would be insightful to determine if specific datasets are predominantly influenced by one over the other. Is it possible to isolate certain features and then forecast based on them?

Author response:
We thank the reviewer for the insightful comment. In Appendix A.8 Tables 7, 8, and 9, we provide an ablation analysis that includes the assessment of individual effects of specific types of bad positive pairs. The terms "w/o $t_{np}$" and "w/o $t_{fp}$" refer to models that exclude the handling of noisy positive pairs and faulty positive pairs, respectively.

It is possible that specific datasets are predominantly influenced by one over the other. For instance, in Table 7, the performance appears to be marginally affected by the choice of $\beta_{fp} \in [2,3]$ in scenarios where we do not address noisy positive pairs (w/o $t_{np}$). Conversely, when faulty positive pairs are not managed (w/o $t_{fp}$), a smaller $\beta_{np}$ is associated with improved performance, potentially indicating a higher prevalence of noise within the dataset.

Our DBPM can be used to determine such unbalanced influences by selectively isolating one type of bad positive pair, thereby enabling an insightful analysis of their respective influences on the dataset.

Reviewer comment (Novelty):
The paper zeroes in on contrastive learning techniques specific to time series data. A key issue it tackles is the identification of noisy and faulty positive pairs. However, the authors could further clarify why these types of pairs pose a unique challenge within the context of time series, distinguishing them from other data types.

Author response:
We thank the reviewer for your constructive suggestions. Here we would like to further clarify the uniqueness of the bad positive pair problem.

In image contrastive learning, the prior work [1] only identified the faulty view problem, as it is rare to see images containing overwhelming noise. Our findings reveal the presence of not only faulty positive pairs but also noisy positive pairs in time series contrastive learning. This can be attributed to the sensitive nature of time series data. In real-world applications, noise, data collection errors, and various distortions can lead to bad positive pairs. For example, in collecting electroencephalography (EEG) data, noise can occur due to ocular, muscular and cardiac activities [2]. 

Our paper presents both theoretical analysis and empirical evidence demonstrating that these types of pairs can negatively affect the representations learned from time series contrastive learning. 

The particular challenge of addressing two different types of bad positive pairs is unique to time series data and distinguishes them from image data. Our results reveal that methods directly adapted from image data, such as RINCE[1], fail to effectively tackle the issues these pairs present in the time series context. This underscores the necessity of understanding the unique characteristics of these pairs within time series data, highlighting a distinct challenge in this domain.

Following your suggestions, we will improve the clarity on the uniqueness of this problem by involving the above discussion in our revised manuscript.

[1] Chuang, Ching-Yao, et al. "Robust contrastive learning against noisy views." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Gandhi, Sunil, et al. "Denoising time series data using asymmetric generative adversarial networks." Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part III 22. Springer International Publishing, 2018.

Reviewer comment (Theory):
There is a potential need for the authors to address the assumptions underlying their model, particularly the noisy alignment assumption that $\xi_i \gg z_i$. This assumption might be overly stringent for real-world scenarios where noise typically represents a smaller fraction of the overall signal.

Author response:
We would like to clarify that, as we described in Sec 1 (Page 2), we assume that noisy pairs and faulty pairs are extreme cases that should represent a small portion of a normal dataset. However, the existence of such extreme cases can impair the quality of representations learned from the model. Our DBPM is designed to deal with extreme cases that adversely affect model performance in real-world datasets.

Reviewer comment (Experiments):
**Ablation Study Limitations:** The presented ablation studies are notably limited, which constrains the understanding of each component's unique effects on the overall system performance. Particularly, the roles of the memory module and the transformation module have not been analyzed independently. Distinct investigations into how each module contributes to the suppression of bad pairs, the efficiency of the learning process, and the final model accuracy would greatly enhance the transparency of the results.

Author response:
We thank the reviewer for the valuable suggestions. Following your suggestions, we conducted an additional ablation study on PTB-XL dataset in the following way: 

- **w/o M**: we directly identify bad positive pairs at each epoch without using the historical information.

- **w/o T**: we remove the transformation module and set the weight of identified bad pairs to zero.

The results are shown below:

|                                              | Rhythm  Classification  | | Form Classification |    | Diagnostic Classification | |
|--------|-------------|--------|-----|-------|------|---|
| SimCLR                                | $80.18\pm5.22$  | | $77.16\pm1.58$ |    | $80.50\pm2.16$
| SimCLR+DBPM (**w/o M**)  | $79.94\pm3.75$  | | $77.62\pm1.83$ |    | $81.54\pm1.82$
| SimCLR+DBPM (**w/o T**)   | $80.54\pm3.69$  | | $77.72\pm1.08$ |    | $81.21\pm2.05$
| SimCLR+DBPM                   | $81.56\pm1.84$  | | $78.07\pm1.01$ |    | $81.56\pm1.84$

We can see from the results that both the memory module **M** and transformation module **T** are important for ensuring consistent improvements of model performance. 

**M** plays a crucial role in reliably identifying potential bad positive pairs. For instance, if we do not use historical information in the Rhythm classification task, performance can be adversely affected. While **T** ensures a smooth weight reduction which contributes to a more stable training process. This can be seen from the suboptimal model performance when removing **T**.

We hope the results provide a more comprehensive understanding of the unique effects of each component on overall performance and address your concerns. 

Following your suggestion, we will add these new results to our revised manuscript.

Reviewer comment (Evaluation):
**Algorithmic Complexity:** The discussion on the algorithm’s additional computational complexity is mentioned but lacks depth. A detailed exploration of the computational costs, including an assessment of trade-offs, would be instructive. Further analysis on the scaling behavior, particularly concerning large datasets and the associated runtime and memory implications in comparison to baseline methods, would greatly benefit the paper.

Author response:
Thank you for your valuable suggestions. We conducted a detailed analysis on algorithmic complexity. Suppose we have $N$ instance:

**The parts of the algorithm that cause extra runtime**:

a. Calculation of the mean and standard deviation: 

* For each pair, its mean $m_{i,e}$ is updated based on the previous means, which is an $O(1)$ operation per instance since it is based on pre-computed values.

* The overall mean $\mu_e$ across all pairs at each epoch requires summing up $N$ mean values and dividing by $N$, which is an $O(N)$ operation.

* Once the overall mean $\mu_e$ is calculated, the $\sigma_e$ requires a pass over the $N$ mean values to calculate the sum of the squared difference from the $\mu_e$, which is an $O(N)$ operation. Then the calculation of actual $\sigma_e$ involves dividing by $N$ and taking the square root, both are $O(1)$ operations.

* Therefore, calculating the mean and standard deviation will result in a runtime complexity of $O(N)$.

b. Condition check:

* Each comparison is an $O(1)$ operation, so for all $N$ instances, the condition check is $O(N)$ per epoch.

c. Calculation of weights:

* Calculating the weights for each instance is a constant-time operation $O(1)$. Therefore, for all $N$ instances, the calculation of weights is $O(N)$.

In summary, the runtime complexity of DBPM is O(N).

**The parts of the algorithm that cause extra memory**:

In our algorithm, memory module **M** serves as a look-up table for storing the training loss for each sample along the training epochs, therefore original **M** is of the size $N \times E$. Here we show the **M** can be optimized to size $N \times 1$ in practical implementation: 

For sample $i$ at $e$-th epoch, its training behavior is calculated by averaging the training losses before $e$-th epoch:

$$
    m_{(i,e)} = \frac{1}{e-1} \cdot \sum_{e'=1}^{e-1} L_{(i,e')}
$$

The above equation can be rewritten as:

$$
    m_{(i,e)} = \\{[ \frac{1}{e-2} \cdot \sum_{e'=1}^{e-2} L_{(i,e')}] \cdot (e-2) + L_{(i,e)}\\} \cdot \frac{1}{e-1}
$$

$$
    m_{(i,e)} = [m_{(i,e-1)} \cdot (e-2) + L_{(i,e)}] \cdot \frac{1}{e-1}
$$

This means that, for calculating the mean training loss of sample $i$ at $e$-th epoch, we only need to store the mean training loss of sample $i$ at $(e-1)$-th epoch. Therefore, the memory size can be reduced to size $N \times 1$ in the practical implementation. Assuming a 16-bit floating-point number for each loss, the total extra memory required would be $2×N$ bytes. 

We will add these analysis to our revised manuscript.

Reviewer comment (Experiments):
**Parameter Sensitivity Analysis:** The method for selecting the hyperparameters critical for identifying bad positive pairs (specifically, the two $\beta$ parameters) seems to be heuristic-based. A thorough sensitivity analysis of these hyperparameters, with an emphasis on how they influence model performance, could substantiate the robustness of the paper. Although the preset heuristics for setting thresholds perform adequately, an adaptive strategy for threshold selection could potentially be more effective than manual tuning of the β values.

Author response:
The sensitivity analysis can be found in Appendix A.8 Tables 7, 8, and 9. The use of a manual threshold in our current design is a simple and effective solution that aligns with our observations. By choosing proper thresholds, we have achieved meaningful improvements in the performance of the contrastive learning models. 

We agree that relying on a manual threshold might not be the optimal solution for all datasets or models. Therefore, as we mentioned in Sec.5, we plan to explore more sophisticated techniques for adaptive thresholding in our future work. This could involve learning the threshold directly from the data or using decision-making methods (e.g., Reinforcement Learning) to adjust the threshold dynamically during training. 

We believe our work provides a strong baseline in the newly identified bad positive pair problem, and we anticipate that it will inspire more further research and explorations.

Reviewer comment (Experiments):
What are other potential applications (e.g. forecasting) where DBPM could be beneficial? Would be good to see evaluation beyond just classification.

Author response:
To provide a clearer analysis of the newly recognized bad positive pair problem, this study mainly focused on time series classification tasks. As we mentioned in Sec.5, we will apply DBPM to more applications such as forecasting and anomaly detection in our future works.