Paper: Soft Contrastive Learning for Time Series (ICLR-2024)

Reviewer comment (Novelty):
The related work is extensive but misses some seminal works regarding contrastive learning methods for time series tackling the challenges of inter/intra sample dependencies: "Subject-aware contrastive learning for biosignals" by Cheng et al. (2020) proposes to only use negative representations from the same time series to "promote subject-invariant" representation, what the authors would refer to as "temporal CL".

Author response:
**W1. Related works from medical domain**

In our initial submission, we cited papers about general TS rather than specific domains like the medical domain. However, we agree that a discussion on related works from specific domains would still be interesting and helpful to audiences. We have added a brief discussion on the papers that e9k3 suggested in **Section 2**. Below is a longer discussion that e9k3 might be specifically interested in:

- **Subject-aware contrastive learning for biosignals (Cheng et al, 2020)**:  Their proposed subject-aware CL contrasts multiple TS, which can be considered as **instance-wise CL** rather than temporal CL, as instance-wise CL contrasts the representations of **multiple TS**, while temporal CL contrasts the representations of multiple timestamps within a **single TS**, as illustrated in **Figure 1**. By architecture design, subject-aware CL cannot perform temporal CL, as they generate representations for the entire TS, such that the information along with timestamps are entangled.

- **CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients (Kiyasseh et al, 2021)**: Their proposed CLOCS performs CL over temporal and spatial dimensions, where the concept of spatial dimension is close to the channel (or multivariate) dimension in general TS. Hence, this idea might have a limited applicability, e.g., for univariate TS.

- **Neighborhood Contrastive Learning Applied to Online Patient Montoring (Yeche et al, 2021)**:  NCL performs CL with **multiple hard** assignments, where assignments are based on predefined attributes of instances. This assignment strategy of NCL is similar to NNCLR [A]. NCL jointly optimizes two conflicting losses with a trade-off: the neighbor alignment loss maximizing the similarity of neighbors as well as positive pairs, and the neighbor discriminative loss maximizing the similarity of positive pairs while minimizing the similarity of neighbors. However, different from two conflicting losses in NCL, our proposed two losses (soft instance-wise CL loss and soft temporal CL loss) do not contradict each other and operate on different dimensions: instance-wise and temporal, respectively.

Also, similar to subject-aware CL, **NCL does not perform temporal CL**; it contrasts representations of multiple TS, which corresponds to instance-wise CL. Note that, as illustrated in Figure 5 of the NCL paper, representations obtained from NCL disregard the temporal axis. Hence, by architecture design, subject-aware CL cannot perform temporal CL, as they generate representations for the entire TS, such that the information along with timestamps are entangled.

As we understand, NCL can be considered a variation of NNCLR [A], where it takes advantage of supervision from the medical domain to determine neighbors and jointly optimizes two conflicting losses. Hence, we categorize NCL as a soft contrastive learning and have added a discussion in **Section 2**.

[A] Dwibedi, Debidatta, et al. "With a little help from my friends: Nearest-neighbor contrastive learning of visual representations." ICCV (2021)

Reviewer comment (Novelty):
The related work is extensive but misses some seminal works regarding contrastive learning methods for time series tackling the challenges of inter/intra sample dependencies: "CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients" by Kiyasseh et al. (2021) proposes to, on the contrary, use representations from the same time series as positives, what the authors would refer to as "instance-wise CL". This is similar to TNC but with a neighborhood being defined as being from the same time series.

Author response:
**W1. Related works from medical domain**

In our initial submission, we cited papers about general TS rather than specific domains like the medical domain. However, we agree that a discussion on related works from specific domains would still be interesting and helpful to audiences. We have added a brief discussion on the papers that e9k3 suggested in **Section 2**. Below is a longer discussion that e9k3 might be specifically interested in:

- **Subject-aware contrastive learning for biosignals (Cheng et al, 2020)**:  Their proposed subject-aware CL contrasts multiple TS, which can be considered as **instance-wise CL** rather than temporal CL, as instance-wise CL contrasts the representations of **multiple TS**, while temporal CL contrasts the representations of multiple timestamps within a **single TS**, as illustrated in **Figure 1**. By architecture design, subject-aware CL cannot perform temporal CL, as they generate representations for the entire TS, such that the information along with timestamps are entangled.

- **CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients (Kiyasseh et al, 2021)**: Their proposed CLOCS performs CL over temporal and spatial dimensions, where the concept of spatial dimension is close to the channel (or multivariate) dimension in general TS. Hence, this idea might have a limited applicability, e.g., for univariate TS.

- **Neighborhood Contrastive Learning Applied to Online Patient Montoring (Yeche et al, 2021)**:  NCL performs CL with **multiple hard** assignments, where assignments are based on predefined attributes of instances. This assignment strategy of NCL is similar to NNCLR [A]. NCL jointly optimizes two conflicting losses with a trade-off: the neighbor alignment loss maximizing the similarity of neighbors as well as positive pairs, and the neighbor discriminative loss maximizing the similarity of positive pairs while minimizing the similarity of neighbors. However, different from two conflicting losses in NCL, our proposed two losses (soft instance-wise CL loss and soft temporal CL loss) do not contradict each other and operate on different dimensions: instance-wise and temporal, respectively.

Also, similar to subject-aware CL, **NCL does not perform temporal CL**; it contrasts representations of multiple TS, which corresponds to instance-wise CL. Note that, as illustrated in Figure 5 of the NCL paper, representations obtained from NCL disregard the temporal axis. Hence, by architecture design, subject-aware CL cannot perform temporal CL, as they generate representations for the entire TS, such that the information along with timestamps are entangled.

As we understand, NCL can be considered a variation of NNCLR [A], where it takes advantage of supervision from the medical domain to determine neighbors and jointly optimizes two conflicting losses. Hence, we categorize NCL as a soft contrastive learning and have added a discussion in **Section 2**.

[A] Dwibedi, Debidatta, et al. "With a little help from my friends: Nearest-neighbor contrastive learning of visual representations." ICCV (2021)

Reviewer comment (Novelty):
The related work is extensive but misses some seminal works regarding contrastive learning methods for time series tackling the challenges of inter/intra sample dependencies: "Neighborhood Contrastive Learning Applied to Online Patient Monitoring" by Yeche et al. (2021) introduces the trade-off used by the authors between instance-wise and temporal-wise CL controlled by $\lambda$. In particular, the objective proposed by Yeche et al., namely NCL, is similar to taking a hard assignment for instance-wise CL $w_I(i,j) = \mathbb{1}_{[i = j]}$ and a (discontinuous) uniform one over a window for temporal CL. This work should explicitly refer to these three works and in particular NCL, from which the SoftCLT is an extension to continuous neighborhood definitions, and ideally provide a comparison with it.

Author response:
**W1. Related works from medical domain**

In our initial submission, we cited papers about general TS rather than specific domains like the medical domain. However, we agree that a discussion on related works from specific domains would still be interesting and helpful to audiences. We have added a brief discussion on the papers that e9k3 suggested in **Section 2**. Below is a longer discussion that e9k3 might be specifically interested in:

- **Subject-aware contrastive learning for biosignals (Cheng et al, 2020)**:  Their proposed subject-aware CL contrasts multiple TS, which can be considered as **instance-wise CL** rather than temporal CL, as instance-wise CL contrasts the representations of **multiple TS**, while temporal CL contrasts the representations of multiple timestamps within a **single TS**, as illustrated in **Figure 1**. By architecture design, subject-aware CL cannot perform temporal CL, as they generate representations for the entire TS, such that the information along with timestamps are entangled.

- **CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients (Kiyasseh et al, 2021)**: Their proposed CLOCS performs CL over temporal and spatial dimensions, where the concept of spatial dimension is close to the channel (or multivariate) dimension in general TS. Hence, this idea might have a limited applicability, e.g., for univariate TS.

- **Neighborhood Contrastive Learning Applied to Online Patient Montoring (Yeche et al, 2021)**:  NCL performs CL with **multiple hard** assignments, where assignments are based on predefined attributes of instances. This assignment strategy of NCL is similar to NNCLR [A]. NCL jointly optimizes two conflicting losses with a trade-off: the neighbor alignment loss maximizing the similarity of neighbors as well as positive pairs, and the neighbor discriminative loss maximizing the similarity of positive pairs while minimizing the similarity of neighbors. However, different from two conflicting losses in NCL, our proposed two losses (soft instance-wise CL loss and soft temporal CL loss) do not contradict each other and operate on different dimensions: instance-wise and temporal, respectively.

Also, similar to subject-aware CL, **NCL does not perform temporal CL**; it contrasts representations of multiple TS, which corresponds to instance-wise CL. Note that, as illustrated in Figure 5 of the NCL paper, representations obtained from NCL disregard the temporal axis. Hence, by architecture design, subject-aware CL cannot perform temporal CL, as they generate representations for the entire TS, such that the information along with timestamps are entangled.

As we understand, NCL can be considered a variation of NNCLR [A], where it takes advantage of supervision from the medical domain to determine neighbors and jointly optimizes two conflicting losses. Hence, we categorize NCL as a soft contrastive learning and have added a discussion in **Section 2**.

[A] Dwibedi, Debidatta, et al. "With a little help from my friends: Nearest-neighbor contrastive learning of visual representations." ICCV (2021)


**W3-a. Laplacian kernel instead of sigmoid for instance-wise CL**

& **W3-b. Exploring different kernels for assignment function**

Indeed, we also conducted an ablation study on the choice of the assignment function for instance-wise CL in **Table F.1** (to make sure, the one in **Table 6(b)** is for temporal CL). However, as the Laplacian kernel was initially not our candidate, we additionally conducted experiments using the **Laplacian kernel** for instance-wise CL. As shown in the updated **Table F.1**, the performance of the Laplacian kernel (83.1%) is lower than the best performance of the sigmoid function (83.9%).

Though we have not tried generalized Gaussian variants with different $\beta$, we compared other types of choices like Neighbor or Linear for temporal CL (where the details can be found in **Appendix L**) instead, as different types of assignment functions would show more different results.

We note that the details of assignment functions can be found in below:
- Temporal CL: **Table 6(b)**, **Appendix  L**
- Instance-wise CL: **Appendix  F**

By the way, we note that the role of the assignment function is not mainly on squashing the range of D to [0,1] for instance-wise CL, as D is the min-max normalized distance in this case; we have updated the definition of D in **Section 3.2** accordingly. However, the assignment function is still useful in that it highlights a certain range of distances by nonlinear transformation, depending on the choice of the assignment function. To confirm this, we also added the case when we omit the assignment function (w/o kernel) in **Table F.1**, which shows worse performance (79.1%) than others.

**W3-c. Trade-off between temporal and instance-wise contrastive loss**

The hyperparameter controlling the trade-off between temporal and instance-wise contrastive losses is chosen by validation; in fact, $\lambda=0.5$ is mostly the best, and this observation is consistent with the original paper of methods that we apply SoftCLT on top of.

We note that, different from two conflicting losses in NCL, our proposed two losses (soft instance-wise CL loss and soft temporal CL loss) do not contradict each other and operate on different dimensions: instance-wise and temporal, respectively. As shown in the updated **Figure 1**, they construct negative pairs differently; temporal CL constructs negative pairs from the same TS but of different timestamps, while instance-wise CL constructs negative pairs from different TS but of the same timestamp.

Reviewer comment (Writing):
The authors refer multiple times to instance-wise and temporal CL before defining it properly in the method section. Pointing the reader to this section earlier or defining the terms in the introduction could improve clarity.

Author response:
**W2-a. Definition of instance-wise and temporal CL**

Thanks for the suggestion, we have added the definition of instance-wise and temporal CL in **Section 1** and **Figure 1**. We believe this will improve the clarity of the proposed method from the earlier part of our paper.

Reviewer comment (Theory):
The authors define their assignment function around a sigmoid function, which is defined over $\mathbb{R}$, whereas its input $D$ lies in $\mathbb{R}^+$. To address this, they tweak their sigmoid function to obtain a symmetric function $w(D) = \frac{2}{1+e^{Dt}}$. It is unclear why they did not instead rely on existing literature, typically using a Laplacian kernel $w(D) = e^{-\frac{D}{l}}$.

Author response:
**W3-a. Laplacian kernel instead of sigmoid for instance-wise CL**

& **W3-b. Exploring different kernels for assignment function**

Indeed, we also conducted an ablation study on the choice of the assignment function for instance-wise CL in **Table F.1** (to make sure, the one in **Table 6(b)** is for temporal CL). However, as the Laplacian kernel was initially not our candidate, we additionally conducted experiments using the **Laplacian kernel** for instance-wise CL. As shown in the updated **Table F.1**, the performance of the Laplacian kernel (83.1%) is lower than the best performance of the sigmoid function (83.9%).

Though we have not tried generalized Gaussian variants with different $\beta$, we compared other types of choices like Neighbor or Linear for temporal CL (where the details can be found in **Appendix L**) instead, as different types of assignment functions would show more different results.

We note that the details of assignment functions can be found in below:
- Temporal CL: **Table 6(b)**, **Appendix  L**
- Instance-wise CL: **Appendix  F**

By the way, we note that the role of the assignment function is not mainly on squashing the range of D to [0,1] for instance-wise CL, as D is the min-max normalized distance in this case; we have updated the definition of D in **Section 3.2** accordingly. However, the assignment function is still useful in that it highlights a certain range of distances by nonlinear transformation, depending on the choice of the assignment function. To confirm this, we also added the case when we omit the assignment function (w/o kernel) in **Table F.1**, which shows worse performance (79.1%) than others.

Reviewer comment (Experiments):
Exploring further the impact of the trade-off between local (temporal) and global (instance) feature learning, ruled by $\alpha$, would be a valuable addition to the ablations.

Author response:
**W3-c. Trade-off between temporal and instance-wise contrastive loss**

The hyperparameter controlling the trade-off between temporal and instance-wise contrastive losses is chosen by validation; in fact, $\lambda=0.5$ is mostly the best, and this observation is consistent with the original paper of methods that we apply SoftCLT on top of.

We note that, different from two conflicting losses in NCL, our proposed two losses (soft instance-wise CL loss and soft temporal CL loss) do not contradict each other and operate on different dimensions: instance-wise and temporal, respectively. As shown in the updated **Figure 1**, they construct negative pairs differently; temporal CL constructs negative pairs from the same TS but of different timestamps, while instance-wise CL constructs negative pairs from different TS but of the same timestamp.

Reviewer comment (Theory):
For instance-wise CL: the calculation of weight based on the distance in the data space makes the weighting process dependent on the scale of input data. Together with the wrapper of Sigmoid function, it might be saturated upon too large or too small input. This effect might make the weights not representative to use in instance-wise CL. While empirically it illustrates the effect over in latent space, more effort needs to be done to consider on which space one should rely on to calculate distance.

Author response:
**W1-b. Soft assignment's dependency on the scale of the input data**

Thank you for bringing this to our attention. In fact, we missed to talk about the detail that we **min-max normalized the pairwise distance matrix**, such that the minimum and maximum values become zero and one, respectively. We have updated the definition of D in **Section 3.2** accordingly.

We also note that the specific details that we could miss should be clear through the code we provided in the supplementary materials, and we will release the code upon acceptance for reproducibility.

Reviewer comment (Theory):
For temporal-wise CL: the current weight assignment implicitly assumes the data from neighbors’ timesteps should be weighted heavier than the data from far timesteps. However, that behavior might not always hold true, as illustrated in work of Tonekaboni (2021).

Author response:
**W2. Implicit assumption on soft assignments for temporal CL**

(Tonekaboni, 2021) considers CL for TS under non-stationarity. Indeed, non-stationarity can be addressed by other strategies like TNC [A] or CoST [B], and our method can be applied on top of them to improve the performance further. For example, CoST [B] is designed to learn disentangled seasonal-trend representations, and applying our SoftCLT on top of CoST improves the performance, which can be found in the new experiment in **Table H.3 in Appendix H**.

[A] Tonekaboni, Sana, Danny Eytan, and Anna Goldenberg. "Unsupervised representation learning for time series with temporal neighborhood coding." ICLR (2021)

[B] Woo, Gerald, et al. "CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting." ICLR (2022)


Nonetheless, we provide more discussion on our method with non-stationarity below.



&nbsp;


**W2-a. TS with non-stationarity - (1) Seasonality**

Soft temporal CL assigns more weight to closer timestamps, so one might be concerned if it fails when seasonality is present.
We argue that **instance-wise CL can indirectly capture the seasonality of TS**, as it contrasts by looking at the representations of the original TS, while **temporal CL** takes advantage of the non-seasonal portions. (In **Section 3.2**, we have clarified that the soft assignments for instance-wise CL are calculated using the original TS, not the augmented views.)
Following the terms from the review of e9K3, soft temporal CL and soft instance-wise CL captures the local and global features, respectively.

&nbsp;

To support our claim, we present both qualitative and quantitative analyses as follows.


**Quantitative analysis (Table 8)**

**Table 8** categorizes 128 UCR datasets by seasonality and shows that the performance gain by sot temporal CL is consistent regardless of the degree of seasonality.

&nbsp;

**Qualitative analysis (Section M - Figure M.1)**

We conducted additional analysis to show that our proposed **soft CL captures seasonality better than the hard CL**, as shown in **Figure M.1**. This figure depicts the t-SNE of representations for all time steps in a single TS **with seasonality**. The figure indicates that while hard CL fails to capture the seasonal patterns in the TS (as similar values, regardless of which phase they are in, are located closely in the embedding space), our proposed soft CL can grasp the global pattern, enabling it to capture the seasonal patterns (as similar values but with different seasonal phases are located differently in the embedding space).


&nbsp;


**W2-b. TS with non-stationarity: distribution shift**

Soft instance-wise CL captures the global view of the TS, and we believe this allows us to detect the **distribution shifts** in the TS. To support this claim, we present additional analysis in **Figure M.2 in Appendix M**.

**Figure M.2** shows the t-SNE of representations for all timestamps in a single TS **with distribution shifts**. The figure indicates that while hard CL fails to capture the sudden change in the TS (as all points are located gradually regardless of the sudden change in the TS), our proposed soft CL can detect such distribution shifts (as points before/after a certain change are clustered in groups).

Reviewer comment (Experiments):
The paper addresses a simplified scenario in which issues related to noise, seasonality, and non-stationarity are not considered, as there is no apparent mechanism in the approach to address these prevalent challenges found in real-world time series data.

Author response:
**W1. TS with noise, seasonality, and non-stationarity**

We found that concerns raised by EHZC are mostly about specific problems in TS, such as noise, seasonality, and non-stationarity. However, we note that **the primary focus on this work is on improving CL for TS** (which is SOTA in TS representation learning), **rather than addressing specific problems in TS**. Indeed, our proposed **SoftCLT is a simple** yet effective soft contrastive learning strategy for TS, so addressing such problems at the expense of simplicity might result in hampering our main contribution.

Rather, such specific problems can be addressed by other strategies, and our method can be applied on top of them to improve the performance further. For example, CoST [A] is designed to learn disentangled seasonal-trend representations, and applying our SoftCLT on top of CoST improves the performance, which can be found in the new experiment in **Table H.3 in Appendix H**.

Nonetheless, we discuss each problem below.

[A] Woo, Gerald, et al. "CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting." ICLR (2022)

&nbsp;



**W1-a. TS with noise**

We agree that some significant noises could pose challenges when calculating distances between TS for soft instance-wise CL. However, we think addressing this issue falls beyond our current scope and is more aptly addressed by metrics designed for measuring distances between TS, as we are not confined to a specific metric. Data preprocessing techniques such as smoothing may alleviate this concern.

&nbsp;



**W1-b. TS with non-stationarity - (1) Seasonality**

Soft temporal CL assigns more weight to closer timestamps, so one might be concerned if it fails when seasonality exists.

We argue that **instance-wise CL can indirectly capture the seasonality of TS**, as it contrasts instances by looking at the representations of their original TS, while **temporal CL** takes advantage of the non-seasonal portions. (In **Section 3.2**, we have clarified that the soft assignments for instance-wise CL are calculated using the original TS, not the augmented views.)
In other words, following the terms from the review of e9K3, soft temporal CL and soft instance-wise CL captures the local and global features, respectively.

&nbsp;

To support our claim, we present both qualitative and quantitative analyses as follows.

**Quantitative analysis (Table 8)**

**Table 8** categorizes 128 UCR datasets by seasonality and shows that the performance gain by soft temporal CL is consistent regardless of the degree of seasonality.

&nbsp;

**Qualitative analysis (Section M - Figure M.1)**

We conducted additional analysis to show that our proposed **soft CL captures seasonality better than the hard CL**, as shown in **Figure M.1**. This figure depicts the t-SNE of representations for all time steps in a single TS **with seasonality**. The figure indicates that while hard CL fails to capture the seasonal patterns in the TS (as similar values, regardless of which phase they are in, are located closely in the embedding space), our proposed soft CL can grasp the global pattern, enabling it to capture the seasonal patterns (as similar values but with different seasonal phases are located differently in the embedding space).


&nbsp;


**W1-c, Q3. TS with non-stationarity: distribution shift**

To our understanding, your concern around "non stationary in the time series, where the immediate next point might be the start of a different distribution" is the problem of distribution shift. Soft instance-wise CL captures the global view of the TS, and we believe this allows us to detect the **distribution shifts** in the TS. To support this claim, we present additional analysis in **Figure M.2 in Appendix M**.

**Figure M.2** shows the t-SNE of representations for all timestamps in a single TS **with distribution shifts**. The figure indicates that while hard CL fails to capture the sudden change in the TS (as all points are located gradually regardless of the sudden change in the TS), our proposed soft CL can detect such distribution shifts (as points before/after a certain change are clustered in groups).

Reviewer comment (Experiments):
Regarding robustness in the presence of noise and non-stationarity, there is no specific discussion or empirical evaluation.

Author response:
**W1-a. TS with noise**

We agree that some significant noises could pose challenges when calculating distances between TS for soft instance-wise CL. However, we think addressing this issue falls beyond our current scope and is more aptly addressed by metrics designed for measuring distances between TS, as we are not confined to a specific metric. Data preprocessing techniques such as smoothing may alleviate this concern.

&nbsp;



**W1-c, Q3. TS with non-stationarity: distribution shift**

To our understanding, your concern around "non stationary in the time series, where the immediate next point might be the start of a different distribution" is the problem of distribution shift. Soft instance-wise CL captures the global view of the TS, and we believe this allows us to detect the **distribution shifts** in the TS. To support this claim, we present additional analysis in **Figure M.2 in Appendix M**.

**Figure M.2** shows the t-SNE of representations for all timestamps in a single TS **with distribution shifts**. The figure indicates that while hard CL fails to capture the sudden change in the TS (as all points are located gradually regardless of the sudden change in the TS), our proposed soft CL can detect such distribution shifts (as points before/after a certain change are clustered in groups).

Reviewer comment (Theory):
Regarding seasonality, the authors mentioned "Our conjecture is that TS in the real world usually do not exhibit the perfect seasonality, as indicated by the ADF test result, such that SoftCLT takes advantage of the non-seasonal portions." While perfect seasonality may be absent in some datasets and may vary in intensity across different datasets, I believe completely disregarding it is not a practical approach.

Author response:
**W1-b. TS with non-stationarity - (1) Seasonality**

Soft temporal CL assigns more weight to closer timestamps, so one might be concerned if it fails when seasonality exists.

We argue that **instance-wise CL can indirectly capture the seasonality of TS**, as it contrasts instances by looking at the representations of their original TS, while **temporal CL** takes advantage of the non-seasonal portions. (In **Section 3.2**, we have clarified that the soft assignments for instance-wise CL are calculated using the original TS, not the augmented views.)
In other words, following the terms from the review of e9K3, soft temporal CL and soft instance-wise CL captures the local and global features, respectively.

&nbsp;

To support our claim, we present both qualitative and quantitative analyses as follows.

**Quantitative analysis (Table 8)**

**Table 8** categorizes 128 UCR datasets by seasonality and shows that the performance gain by soft temporal CL is consistent regardless of the degree of seasonality.

&nbsp;

**Qualitative analysis (Section M - Figure M.1)**

We conducted additional analysis to show that our proposed **soft CL captures seasonality better than the hard CL**, as shown in **Figure M.1**. This figure depicts the t-SNE of representations for all time steps in a single TS **with seasonality**. The figure indicates that while hard CL fails to capture the seasonal patterns in the TS (as similar values, regardless of which phase they are in, are located closely in the embedding space), our proposed soft CL can grasp the global pattern, enabling it to capture the seasonal patterns (as similar values but with different seasonal phases are located differently in the embedding space).

Reviewer comment (Theory):
Can this case be elaborated a bit further: “when α = 1, we give the assignment of one to the pairs with the distance of zero as well as the pairs of the same TS.” What if in the same TS we are experiencing two different patterns, shifts, or different distributions?

Author response:
**Q1. What if a pair of views from the same TS instance have two different patterns (shifts or different distribution)?**

We note that two views are generated by **data augmentation (DA)**, where DA is meant to generate multiple data with different raw values but same semantics in machine learning. Hence, DA should not destroy the semantics; in other words, **DA must not generate two different patterns for an instance if the patterns should be recognized as the characteristics of different classes**. In our case, we do not prescribe any specific DA techniques, but follow DA used in the baseline where our SoftCLT is applied.

If we misunderstood your question and/or our answer is insufficient, please ask us again.

Reviewer comment (Reproducibility):
In equation 3, augmentation for I and i+N, how is it performed? What if there is only a shift in the pattern in the instances; otherwise they are very similar — how do you address this in your computation? It would be great to include an illustration for this case to show your approach is robust to shift (or some noise), which is very common in real-world applications.

Author response:
**Q2. How is augmentation for i and i+N performed**

Note that i and i+N are indices of two augmented views from the same instance. As described in our answer for Q1, our proposed SoftCLT is not constrained to any specific architecture or any form of DA, but follows the settings of the target CL methods that we apply SoftCLT. Regarding the concern on shift or some noise potentially introduced by DA, such DA should not be used if they destroy the semantics.

As we applied our SoftCLT to TS2Vec [A], TS-TCC [B], and CA-TCC [C], we provide DA techniques used in these methods below:

- TS2Vec: It generates two views of a single TS by randomly sampling two segments ***with an overlap***, and it contrasts only the overlapped parts.
- TS-TCC and CA-TCC: They generate two views of a single TS by two different augmentations: strong augmentation (permutation-and-jitter) for one and weak augmentation (jitter-and-scale) for the other.

To clarify this, we have included additional details on the data augmentation techniques we employed in **Section 4**. Thank you for bringing this to our attention.


[A] Yue, Zhihan, et al. "Ts2vec: Towards universal representation of time series." AAAI (2022)

[B] Eldele, Emadeldeen, et al. "Time-series representation learning via temporal and contextual contrasting." IJCAI (2021)

[C] Eldele, Emadeldeen, et al. "Self-supervised contrastive representation learning for semi-supervised time-series classification." TPAMI (2023)

Reviewer comment (Experiments):
Problem with the comparisons. Entirely absent from the main paper is any comparison with CoST [1], or any more recent contrastive approach.

Author response:
**W1, Q1. Detailed comparison with CoST and an evaluation in the regression setting**

Thanks for bringing this to our attention. We applied our SoftCLT to CoST [A], and provide the results below and **Table H.3 in Appendix H** in the revision:

| | Multivariate || forecasting | | Univariate | | forecasting | |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | 
| | CoST | | CoST + Ours | | CoST | | CoST + Ours | |
| | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 | 0.650 | 0.585 | **0.619** | **0.570** | 0.094 | 0.230 | **0.091** | **0.227** |
| ETTh2 | 1.322 | 0.876 | **1.315** | **0.872** | 0.161 | 0.307 | **0.156** | **0.304** |
| ETTm1 | 0.409 | 0.439 | **0.407** | **0.436** | 0.054 | 0.164 | **0.051** | **0.159** |
| Weather | 0.432 | 0.465 | **0.430** | **0.463** | 0.189 | 0.310 | **0.181** | **0.307** |

We observed consistent improvements in performance for CoST in both multivariate and univariate forecasting tasks under various datasets. The results are average MSE/MAE above 5 horizons, and averaged over four runs with different random seeds, using the official code of CoST.

[A] Woo, Gerald, et al. "CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting." ICLR (2022)

Reviewer comment (Experiments):
Similarly, there is no comparison to recent approaches in the regression setting of TS2Vec (nothing prevents that comparison, the TS2Vec code works seamlessly for both approaches).

Author response:
**W2. Comparison with recent contrastive approaches**

We agree that comparing with recent CL approaches and figuring out the applicability of our SoftCLT to them should be interesting and strengthen our contribution. We are supposed to apply our SoftCLT to the SOTA methods per downstream tasks, e.g., TS2Vec [A] for classification/anomaly detection and TS-TCC/CA-TCC for semi-supervised learning. Thanks to bKG2, we have added CoST [B] for regression following their suggestion.

To be more specific, we provide how we choose the baseline that we apply SoftCLT to:

1. **Classification (TS2Vec [A])**: We chose the widely used UCR 128 and UEA 30 datasets for both univariate and multivariate TS classification, respectively, as these datasets cover TS from various domains and lengths, and TS2Vec has demonstrated SOTA performance for these tasks.

2. **Semi-supervised Classification (TS-TCC [B], CA-TCC [C], TS2Vec [A])**: We chose CA-TCC, as it is the CL method specifically designed for semi-supervised settings and has demonstrated SOTA performance. Given that CA-TCC is an extension of TS-TCC, we also included TS-TCC in our evaluation. Additionally, TS2Vec, renowned for its SOTA performance in standard classification, is also included.

3. **Transfer Learning (TS-TCC [B], CA-TCC [C])**: While there were other candidates (TS2Vec [A], TF-C [D], CoST [E], CLOCS [F]), we specifically chose TS-TCC and CA-TCC for the following reasons:

- TS2Vec [A]: Its performance, as indicated by experiments conducted by CA-TCC, is mostly worse than both TS-TCC and CA-TCC.

- TF-C [D]: Their result is not reproducible with their official code. This issue seems not unique to our experience by looking at several GitHub issues.

- Other CL methods (CoST [E], CLOCS [F]): These methods are specifically designed for TS forecasting, hence not considered for transfer learning in classification. Their performance is inferior to TS-TCC, as shown in the updated **Table 4(a)**.

4. **Anomaly Detection (TS2Vec [A])**: Given the scarcity of CL methods addressing anomaly detection, TS2Vec, a SOTA method in this domain, is chosen for our evaluation.


We have also added other recent methods (CoST [E], CLOCS [F], LaST [G]) to **Table 4** by comparing the transfer learning performance, as it would be beneficial to audiences.


[A] Yue, Zhihan, et al. "Ts2vec: Towards universal representation of time series." AAAI (2022)

[B] Eldele, Emadeldeen, et al. "Time-series representation learning via temporal and contextual contrasting." IJCAI (2021)

[C] Eldele, Emadeldeen, et al. "Self-supervised contrastive representation learning for semi-supervised time-series classification." TPAMI (2023)

[D] Zhang, Xiang, et al. "Self-supervised contrastive pre-training for time series via time-frequency consistency." NeurIPS (2022).

[E] Woo, Gerald, et al. "CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting." ICLR (2022)

[F] Kiyasseh, Dani, Tingting Zhu, and David A. Clifton. "Clocs: Contrastive learning of cardiac signals across space, time, and patients." ICML (2021).

[G] Wang, Zhiyuan, et al. "Learning latent seasonal-trend representations for time series forecasting." NeurIPS (2022)

Reviewer comment (Experiments):
The reviewer would like a detailed comparison with CoST and an evaluation in the regression setting, and if the authors provide this with warranting results, they will raise their score.

Author response:
**W1, Q1. Detailed comparison with CoST and an evaluation in the regression setting**

Thanks for bringing this to our attention. We applied our SoftCLT to CoST [A], and provide the results below and **Table H.3 in Appendix H** in the revision:

| | Multivariate || forecasting | | Univariate | | forecasting | |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | 
| | CoST | | CoST + Ours | | CoST | | CoST + Ours | |
| | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 | 0.650 | 0.585 | **0.619** | **0.570** | 0.094 | 0.230 | **0.091** | **0.227** |
| ETTh2 | 1.322 | 0.876 | **1.315** | **0.872** | 0.161 | 0.307 | **0.156** | **0.304** |
| ETTm1 | 0.409 | 0.439 | **0.407** | **0.436** | 0.054 | 0.164 | **0.051** | **0.159** |
| Weather | 0.432 | 0.465 | **0.430** | **0.463** | 0.189 | 0.310 | **0.181** | **0.307** |

We observed consistent improvements in performance for CoST in both multivariate and univariate forecasting tasks under various datasets. The results are average MSE/MAE above 5 horizons, and averaged over four runs with different random seeds, using the official code of CoST.

[A] Woo, Gerald, et al. "CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting." ICLR (2022)


**W2. Comparison with recent contrastive approaches**

We agree that comparing with recent CL approaches and figuring out the applicability of our SoftCLT to them should be interesting and strengthen our contribution. We are supposed to apply our SoftCLT to the SOTA methods per downstream tasks, e.g., TS2Vec [A] for classification/anomaly detection and TS-TCC/CA-TCC for semi-supervised learning. Thanks to bKG2, we have added CoST [B] for regression following their suggestion.

To be more specific, we provide how we choose the baseline that we apply SoftCLT to:

1. **Classification (TS2Vec [A])**: We chose the widely used UCR 128 and UEA 30 datasets for both univariate and multivariate TS classification, respectively, as these datasets cover TS from various domains and lengths, and TS2Vec has demonstrated SOTA performance for these tasks.

2. **Semi-supervised Classification (TS-TCC [B], CA-TCC [C], TS2Vec [A])**: We chose CA-TCC, as it is the CL method specifically designed for semi-supervised settings and has demonstrated SOTA performance. Given that CA-TCC is an extension of TS-TCC, we also included TS-TCC in our evaluation. Additionally, TS2Vec, renowned for its SOTA performance in standard classification, is also included.

3. **Transfer Learning (TS-TCC [B], CA-TCC [C])**: While there were other candidates (TS2Vec [A], TF-C [D], CoST [E], CLOCS [F]), we specifically chose TS-TCC and CA-TCC for the following reasons:

- TS2Vec [A]: Its performance, as indicated by experiments conducted by CA-TCC, is mostly worse than both TS-TCC and CA-TCC.

- TF-C [D]: Their result is not reproducible with their official code. This issue seems not unique to our experience by looking at several GitHub issues.

- Other CL methods (CoST [E], CLOCS [F]): These methods are specifically designed for TS forecasting, hence not considered for transfer learning in classification. Their performance is inferior to TS-TCC, as shown in the updated **Table 4(a)**.

4. **Anomaly Detection (TS2Vec [A])**: Given the scarcity of CL methods addressing anomaly detection, TS2Vec, a SOTA method in this domain, is chosen for our evaluation.


We have also added other recent methods (CoST [E], CLOCS [F], LaST [G]) to **Table 4** by comparing the transfer learning performance, as it would be beneficial to audiences.


[A] Yue, Zhihan, et al. "Ts2vec: Towards universal representation of time series." AAAI (2022)

[B] Eldele, Emadeldeen, et al. "Time-series representation learning via temporal and contextual contrasting." IJCAI (2021)

[C] Eldele, Emadeldeen, et al. "Self-supervised contrastive representation learning for semi-supervised time-series classification." TPAMI (2023)

[D] Zhang, Xiang, et al. "Self-supervised contrastive pre-training for time series via time-frequency consistency." NeurIPS (2022).

[E] Woo, Gerald, et al. "CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting." ICLR (2022)

[F] Kiyasseh, Dani, Tingting Zhu, and David A. Clifton. "Clocs: Contrastive learning of cardiac signals across space, time, and patients." ICML (2021).

[G] Wang, Zhiyuan, et al. "Learning latent seasonal-trend representations for time series forecasting." NeurIPS (2022)