Paper: Capturing Static, Short-Term, and Long-Term Dynamics Through Self-Supervised Time Series Learning: CHRONOS (ICLR-2024)

Reviewer comment (Novelty):
The literature review effort of the field from the authors is extremely small. Between their introduction and their related work, the authors only cited four works. However, the field of SSL for time series is much richer. For example:  
- Franchesci et al. "Unsupervised scalable representation learning for multivariate time series", NeurIPS 2019  
- Cheng et al. "Subject-aware contrastive learning for biosignals", 2020  
- Mohsenvand et al. "Contrastive representation learning for electroencephalogram classification", Machine Learning for Health 2020  
- Tonekaboni et al. "Unsupervised representation learning for time series with temporal neighborhood coding", ICLR 2020  
- Kiyasseh et al. "Clocs: Contrastive learning of cardiac signals across space, time, and patients", ICML 2021  
- Yeche et al. "Neighborhood contrastive learning applied to online patient monitoring", ICML 2021  
- Eldede et al. "Time-series representation learning via temporal and contextual contrasting", IJCAI 2021  
- Fan et al. "Semi-supervised time series classification by temporal relation prediction", ICASSP 2021

Author response:
The Related Work section has been updated. The new version of the manuscript contains not only a broader view of related work, but also points out what challenges CHRONOS faces and makes it stand out from the rest.

Reviewer comment (Writing):
The authors do not have a part where they introduce precise notations even though they use a lot of different notations. This makes the reading quite complicated and leaves some parts of the method unclear.

Author response:
In the new version of the manuscript, the first paragraph in page 3, the Figure 1 should give a clearer view of the proposed method, in addition to the notations added bellow each equation should be enough for going throgh the paper.

Reviewer comment (Presentation):
The figures do not have descriptive captions. Except for the colors, no description is given for the numerous components of Figure 1. Conversely, in Figures 2 and 3, no explanation is given regarding the coloring scheme.

Author response:
An extended version of the captions for each figure has been incorporated to the final version of the manuscript.

Reviewer comment (Experiments):
The authors use a covariance loss borrowed from VICReg but do not provide ablation for that component. Hence, it is impossible to know if the performance improvement comes from the disentanglement of the dynamic and static features proposed by the authors or from this additional regularization.

Author response:
1. The required ablation study as been included in the Section 5.

Reviewer comment (Experiments):
The authors propose to use two projection spaces instead of one because of the adversarial nature of their objectives, but they do not provide an experiment justifying this choice over a unique space.

Author response:
2. We inherit the proposed configuration from DEBS study.

Reviewer comment (Experiments):
The authors introduce a window parameter for their short-term loss but do not provide ablation for it.

Author response:
3. We inherit the proposed configuration from DEBS study.

Reviewer comment (Evaluation):
Why report specificity and sensitivity and not AUROC directly? Using AUROC prevents the performance from being biased by potential miscalibration of a model.

Author response:
At the time of evaluation we consider that the accuracy, specificity and sensitivity together demonstrate sufficient statistical value to compare different methods.

Reviewer comment (Presentation):
What is the meaning of the colored bars in Figures 2 and 3?

Author response:
They represent the features that are included in the set of the most important features for the respective tasks. It is reflected now in the captions of each figure.

Reviewer comment (Theory):
Why carry constants in your loss terms that will have no impact on gradient, such as "1 +" or "1 -"?

Author response:
The different loss terms consist of the Cosine Similarity between pairs of representations. This metric has values within a range of (-1, 1). These constants (1 + or 1 -) are added for keeping the loss values in a strictly positive range. It is a common practise when the Cosine Similarity is used.

Reviewer comment (Writing):
The writing of the paper is very confusing. The authors described two distinct spaces, static and dynamic. However, Eq. (5) involves the calculation of the inner product between vectors from the two spaces, which indicates there is in fact only one space. Also, based on the description, it seems that the authors attempted to differentiate three types of temporal dynamics: static, short-term, and long-term. However, the proposed model only learns two types of representations, static and short-term.

Author response:
1. The name of the Dynamic space has been changed to short-term dynamic space in the new version of the manuscript. It should give more insights about the method.
2. The inner product is not calculated between the projection in the two spaces but using the predictor of the student network instead. 	This predictor can be seen as a bridge between the two spaces.
3. We do not know what you mean by only learning two types of representations. 
- 3a  Although it has no projection of its own, Long-Term Dynamic Loss leads the model to represent such dynamics. This is calculated between the representation of two different inputs from two different records which are delayed one from the other with a sufficient temporal gap. This loss function is motivated precisely to drive the model to capture the Long Term Dynamics. 
- 3b If you mention it for the experiments developed in section 4.2, where we have not included any that directly concerns the long-term dynamics, we believe that the mere presence of the most important feature for AFib Classification, as part of the set of static features (Section 4.2) as long as the results in the comparison against the other SOTA methods is enough to assess that we are capturing the three natures, as detailed in the discussion of results on the same section.

Reviewer comment (Writing):
The description of the proposed method lacks clarity. For example, what is the relationship between projector and predictor and that between teacher and student networks. The proposed architecture includes encoder(s), projectors, and predictor. What exactly the representation used in downstream prediction task is not clear.

Author response:
When the original manuscript was written, it was thought that referring to the fact that the framework presented in BYOL is used as a base was enough to understand these issues. However, due to this feedback a new paragraph has been added, (last paragraph in section 3.1.1) in which it is detailed that both teacher network and the teacher projectors are updated, using EMA from their respective student pairs. It has also been added that, as is common in SSL methods, the projector and predictor are discarded after the training process. The encoder output is used during the downstream tasks.

Reviewer comment (Theory):
The loss functions are not clearly motivated, or least need more explanation. For example, why two different records are used and why projection and prediction are compared while not both projections in Eq. (2). What are i and j in Eq. (3)? Hyperparameters? How were they determined?

Author response:
1. A more detailed description of the motivation of the different loss functions has been provided in the first paragraph of the page 3. This paragraph, in addition with the Figure 1, should provide more insights about the motivation of each Loss Function. The motivation of the L_LTD, which is one of the cores of this paper, is detailed in depth in Section 3.2.1

2. The student network predictions are used instead the projectors in line with the BYOL framework. 

3. i and j is the time delay between the different inputs. A line has been added below the equation to reflect this. The three inputs are extracted in a two-minute window, in line with the DEBS modus operandi.

Reviewer comment (Theory):
It seems to me that the proposed set of loss functions is problematic, leading to representation that has no discriminative capability across recordings. Eq (2) and Eq (3) drive similarity only and Eq (5) only encourages difference in learned dynamic and static features. What really helping here may be the covariance loss function (Eq. (6)), which needs more explanation and deserves an ablation study.

Author response:
1. The required Ablation Study has been incorporated to the new version of the manuscript (Section 5). The covariance loss function by itself does not drives the model to learn useful representations.

2. As mentioned in Section 3.2.1, the Eq(5) (4 in the new version of the manuscripts), serves both for encouraging the two spaces for capturing different features, and at the same time, capturing the long-term dynamics, since it is calculated using representations from different recordings (As Illustrated in Figure 1). It enhances also differences between different recordings.

3. A new paragraph has been added in the Ablation Study explaining that the incorporation of this loss metric between different records improve the performance on the downstream tasks. Additionaly, using it also as a Contrasting Factor between the projectors improve the performance of the whole system.

Reviewer comment (Evaluation):
The empirical results are relatively weak across the board. Statistic tests are needed in Table 1 (small difference in mean but with large variance) to show whether the differences have statistic significance.

Author response:
There was an error in the STD given in the original manuscript for the MAE errors of the Age Regression task. They were 10x larger. We are very grateful that you brought this to our attention and it has been corrected in the original manuscript. We believe that in this way the results have statistical value.

Reviewer comment (Novelty):
The introduction lacks citations, which are crucial for situating the research within the existing body of knowledge.

Author response:
We appreciate this feedback and agree with it. In the new version of the manuscript, citations have been incorporated in the introduction. Hopefully it will help to contextualize this manuscript within the existing body of knowledge.

Reviewer comment (Novelty):
The related work section needs to include more details of the current state of the field, as it fails to discuss current challenges and the advantages and disadvantages of existing methods. Its brevity and lack of depth do not adequately show how this new approach contributes to or differs from the state-of-the-art.

Author response:
Again, we agree with this point. The Related Work section has been updated. The new version of the manuscript contains not only a broader view of related work, but also points out what challenges CHRONOS faces and makes it stand out from the rest.

Reviewer comment (Theory):
There is an absence of intuitive explanations for the proposed methodology's expected effectiveness, which is a significant omission.

Author response:
After rereading the original manuscript with fresh eyes, we agree on this point. The first paragraph of page 3 has been incorporated in the new version of the manuscript for the purpose of clarifying the explanation of the method. We trust that this paragraph, in aid of Figure 1, will improve the overall understanding of the manuscript.

Reviewer comment (Presentation):
The captions of the figures could be improved to be self-contained.

Author response:
The captionsof the new version of the manuscript have been expanded to provide more information about what they represent.

Reviewer comment (Novelty):
The manuscript has similarities with previous studies, especially the DEBS study. Clarification is needed on the novel aspects of CHRONOS that distinguish it from prior work and what specific innovations CHRONOS introduces.

Author response:
Section 3 has been divided into revisiting DEBS paper, and the new features CHRONOS introduces. While it is true that this first subsection is not new to the existing body of knowledge, as it is inherited from DEBS, we believe it is of crucial importance in understanding the paper. We hope that in this new version all this will be much clearer.

Reviewer comment (Experiments):
The authors frame the work as generalizing to other applications besides medical ECG however it seems most of the motivation for the design of CHRONOS is very grounded in the specifics of ECG datasets and the provided tasks with references to medical-specific features, model selection, and dataset characteristics. An expansion of the evaluation to standard baselines like UCR, or the FD-a and FD-b tasks used to evaluate TF-C, could help show the proposed generalization characteristics of this approach with more empirical validation.

Author response:
We believe that there has been a potential confusion in the statement that CHRONOS may be generalizable to other types of time series data, especially for other types of physiological signals. This statement is motivated by the fact that all these signals share the three dynamics detailed in this study, and no technique (i.e., data augmentation) specific to ECG signals is used. This is why we are invited to think that a model can be optimized using CHRONOS to understand other types of data, using another dataset. However, we do not assure that the model optimized using ECG signals will be able to perform on other types of signals directly. We are sorry for the confusion and sections 6 has been modified to avoid this problem of understanding.

Reviewer comment (Experiments):
There are a number of existing time-series specific transformer applications. FormerTime (Cheng, et. al., 2023), for example, also focuses on multi-scale time series dynamics for multivariate time series classification, albeit without self-supervised learning. The authors should consider evaluations against these time-series specific transformer models such as Formertime and Informer to see if their proposed method continues to add benefit or if simply updating the encoder they use could lead to improvements across all the tasks evaluated using standard self-supervised learning.

Author response:
This paper shows a new SSL method for optimizing a DL model. It means that what it is evaluated here is how SSL different methods make the same model to understand better a specific kind of data. SSL methods are evaluated on generic DL models, commonly using ResNet or ViT (as this paper). It can be supposed that the application of the presented method to a specific architecture tailored to a particular kind of data can lead to a better performance. However, the model architecture is not a topic for this paper.

Reviewer comment (Presentation):
The figures with feature importance and overlap of features for the static and dynamic cases could benefit from more explication of which features were included (e.g. by including the feature name instead of its number).

Author response:
The features you refer to compose a vector of dimension 128, which is the output of the encoder (the model being optimized). There are no names to provide further insight into what they represent. That is why their position in this representation is used during this study.

Reviewer comment (Presentation):
All the references are improperly included in the text. I believe you should change from using `citet` to `citep` in most cases. References should include the author names in the sentence (e.g. “Additionally a SHAP Analysis *as proposed by* Lundberg & Lee (2017) is carried out”) or by including the author names in the citation (e.g. “Additionally a SHAP Analysis (Lundberg & Lee, 2017)...”). Please read the “Citations in Text” part of the provided LaTeX template for more information.

Author response:
We agree on this point, the relevant changes have been made in the new version of the manuscript, which you can see since it has been uploaded.

Reviewer comment (Experiments):
Could the authors expand on the implications of the ablation study conducted in section 5? For example: it’s not clear why the removal of the long term loss was only evaluated when the time scale ratio was set to 25/50/25 (besides this being the ratio chosen during selective optimization). Would the long term loss add more benefit in the case where the long-term ratio is higher than 25?

Author response:
We would like to thank you for this point, as there was an error in Table 4 in the original manuscript. Obviously when L_LTD is not taken into account, the split ratio is 50/50 over the other two. Regarding the second point and increasing the L_LTD ratio, in this same table (last row) the experiment you request has been carried out, showing that it does not improve the proposed configuration.

Reviewer comment (Experiments):
As an additional ablation, have the authors considered removing the selective optimization and randomly selecting a loss function at each step of the pre-training process?

Author response:
We have not considered this option, but our thinking is that it would confuse rather than help the optimization process.

Reviewer comment (Experiments):
In the SSL time series literature, pre-training on the Sleep Heart Health Study is quite common. Why do the authors believe this dataset is more or less useful for pretraining given the tasks at hand? Have the authors tried pre-training on any other datasets?

Author response:
We use SHHS since it contains two records belonging to the same subject, and these records are delayed with a sufficient temporal gap (around 4 years) to let these long-term dynamics changes to appear. This makes this specific database special, and this is the reason that it has been the only database used during the optimization. This explanation has been added to the Appendix (See Appendix B)