Paper: Parametric Augmentation for Time Series Contrastive Learning (ICLR-2024)

Reviewer comment (Experiments):
My main concern is that the authors of this paper claim AutoTCL is a general augmentation method. While the method has been empirically proven effective in crossing different datasets, it would be very critical to test the methods on other contrastive learning frameworks.

Author response:
Thank you for the insightful comment.   We want to clarify that we propose a parametric data augmentation, based on which we propose a general time series learning framework AutoTCL.  We verify the "generalization" with multiple experiments. 
1.   We consider datasets from different domains, as recognized by the reviewer. 
2.   We consider two different tasks, forecasting and classification. For forecasting, we consider both univariate and multivariate settings.  Results in Table 1,2,3,8,10,11,12 verify the advantage of AutoTCL over other contrastive learning baselines.
3.   We consider two state-of-the-art methods, CoST and TS2vec as backbones.  For each method, we use it in AutoTCL and compare it with basic augmentation, random augmentation, and adversarial augmentations. Results in 4,6,7, verify the advantage of parametric data augmentation. 

[1] Chen, Xinlei, and Kaiming He. "Exploring simple siamese representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.  
[2]Chen, Ting, et al. "Big self-supervised models are strong semi-supervised learners." Advances in neural information processing systems 33 (2020): 22243-22255.

Reviewer comment (Novelty):
The authors have given sufficient discussion about augmentation techniques in the original data space. Recently, another branch of augmentations has stemmed from the feature space. I'd also suggested authors to discussion them for a more comprehensive review. [1,2,3] 

[1] Towards domain-agnostic contrastive learning

[2] Metaug: Contrastive learning via meta feature augmentation

[3] Hallucination Improves the Performance of Unsupervised Visual Representation Learning

Author response:
Thanks for your suggestion. We considered these papers and admitted their contribution to data augmentation. We added the discussion of these papers in the related work section.

Reviewer comment (Evaluation):
The performance of Multivariate time series forecasting results seems to be sub-optimal compared with univariate time. Could the authors provide some insights?

Author response:
In a multivariate forecasting setting, we decrease the average MSE by 2.9% and the average MAE by 1.2% than CoST. In Table 2,10, our method gained comparable results and the most time of first/second place. Although the improvement is lower than that in the univariate setting, it is still significant. We think the potential reason is that feature-level correlation is crucial in multivariate forecasting, however, this is not the research focus of this paper.  We thank the reviewer for asking an interesting question.  We will explore this in future works, and study augmentations for multivariate settings, as the reviewer has suggested.

Reviewer comment (Experiments):
What role does g play specifically? Especially in the analysis of the ablation experiments, I only observed differences in the results.

Author response:
As we discussed in Section 3.2, $g$ is an invertible function to conduct data augmentation.  Intuitively, given an $x$, we use factorization function $h$ to detect the informative component, and then adopt $g$ to generate an augmented view for the following contrastive learning. We have added some discussion (Page 9) in the analysis of ablation experiments to address the reviewer's concern.

Reviewer comment (Experiments):
The two networks h and g use the same structure. How to ensure that they indeed perform the two different functions claimed by the author?

Author response:
The factorization function $h$ aims to detect the informative component and $g$ aims to generate an augmented view. We believe that these two tasks are relevant and rely on the quality of hidden features. So we use a shared input layer and 1DCNN for both $h$ and $g$. However, $h$ and $g$ process the encoded features differently, through the factorization head and the transformation head, to produce the desired binary and non-zero outputs. factorization and transformation heads don't share parameters and they adopt different activation functions. Moreover, The network $h$ is trained with a size constraint loss function and the principle of relevant information, which encourages that its output is a compact binary vector. This binary vector represents a mask that identifies the informative components of the input. The size constraint is critical as it imposes sparsity on the output, guiding $h$ to select only the most salient features of the data. Conversely, $g$ is trained only with the principle of relevant information, generating a non-zero vector that enhances the original input for augmented views. Without the size constraint, $g$ is free to explore a wider space of augmentation possibilities.

Reviewer comment (Experiments):
How is Δv set in the experiments?

Author response:
As we introduced in section 3.3, Inspired by Dropout and TS2Vec, we implement the function $\eta(v^*,\Delta v)$  by randomly masking the hidden representation. Specifically, given a latent vector of a view $v$, after the first hidden layer, we randomly mask it along the time dimension with a binary vector. Each element is sampled independently from a Bernoulli distribution, Bern$(0.5)$.

Reviewer comment (Evaluation):
The paper does not verify whether the positive view generated by the proposed method contains more information than the original sequence. Although the paper showcases the augmented instances through visualization in Figure 5, as the authors mentioned in the abstract, "it is impractical to visually inspect the temporal structures in time series." Providing explanations for the generated positive samples based on Property 3 or other relevant data would be beneficial.

Author response:
The intuitive understanding of property 3 is that if the map between $x$ and $v$ is one-to-many, the generated view will contain more information compared to the raw input $x$. In other words, a good augmentation should preserve the underlying semantics that two different $x$ cannot map the same $v$.  To empirically show this property, we conducted case studies in Appendix D 3.2. We observe that AutoTCL can generate diverse views.  At the same time, they are closer to the original input instance compared to other input instances. We have added some discussion in the Appendix to make it clear.

Reviewer comment (Evaluation):
From the visualization in Figure 5, it appears that the output mask of g is a binary mask, which does not match the description in the article. The article interprets g as a non-zero transformation mask. If the output of g is also a binary mask, does it mean that the optimal transformation method for sample enhancement that the model learns is simply to mask some observations of the original sequence?

Author response:
We are sorry for this typo in the previous caption. It should be $h(x)$.  The output mask of $h(x)$ is a binary mask. we have fixed this typo and some imprecise description in the caption  (now, it is Table 9).

Reviewer comment (Theory):
There might be some minor inconsistencies or gaps in the proof that require further attention to ensure its rigor. Such as, in the proof of Property 1, an invertible mapping is not necessarily a one-to-one mapping, which depends on the domain. Of course, this does not affect the subsequent proof.

Author response:
Dear reviewer 3ku1, thank you for recognizing our contributions.  However, we respectfully disagree with the comment. 
In fact, an invertible mapping, by definition, is necessarily a one-to-one (injective) mapping.

To clarify:

1. **Invertible Mapping**: A function `f: A -> B` is invertible if there exists a function `g: B -> A` such that `g(f(a)) = a` for all `a` in `A` and `f(g(b)) = b` for all `b` in `B`. For a function to be invertible, it must be both injective (one-to-one) and surjective (onto).

2. **One-to-One (Injective) Mapping**: A function is one-to-one if it never maps distinct elements of its domain to the same element of its codomain. In other words, if `f(a) = f(b)` implies `a = b` for any `a, b` in the domain.


3. **Onto (Surjective) Mapping**: A function is onto if every element of the codomain is the image of at least one element of the domain.

For a function to be invertible, it must be both one-to-one and onto. The one-to-one property ensures that the function doesn't map two different elements to the same value (necessary for the existence of a unique inverse mapping), and the onto property ensures that every element in the codomain has a pre-image in the domain (ensuring that the inverse mapping is well-defined for every element in the codomain).

Reviewer comment (Presentation):
Some errors on formatting: page 6 “as random timestamp masking“, it seems unnecessary to bold it.

Author response:
Thanks for the suggestions. We have revised our paper accordingly.

Reviewer comment (Experiments):
Most of the comparison in table 6/7 is less relevant than the ablation study. The reason is the following: the authors are comparing different architectures. Most of these perform less well than CoST, and they are building on top of CoST. Hence these results are quite expected and should in my opinion not be presented first.

Author response:
Thanks for the suggestion.  We revised our paper accordingly to move tables `Ablation studies using CoST backbone` and `Ablation studies using TS2Vec backbone` in front of comparison to other baselines in the Appendix.