Original: Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency
Revised: Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Original: Abstract
Revised: Abstract

Original: Pre-training on time series poses a unique challenge  due to the potential mismatchbetween pre-training and target domains, such as shifts in temporal dynamics,fast-evolving trends,and long-range and short cyclic effects, which can lead topoor downstream performance.
Revised: Time series datasets pose a unique challenge for pre-training due to various kinds of discrepancy between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, long-range and short cyclic effects.

Original: While domain adaptation methods can mitigatethese shifts, most methods need examples directly from the target domain, makingthem suboptimal for pre-training.
Revised: While domain adaptation methods can mitigate these shifts, most methods need examples di- rectly from the target domain, making them suboptimal for pre-training on time series.

Original: To address this challenge, methods need toaccommodate target domains with different temporal dynamics and be capableof doing so without seeing any target examples during pre-training.
Revised: To fill this gap, methods need to accommodate a set of target domains withdifferent temporal dynamics and be capable to do so without seeing any target examples during pre-training.

Original: Relativeto other modalities, in time series, we expect that time-based and frequencybased representations of the same example  are located close together in the timefrequency space.
Revised: Relative to other fields, in time series we expect that time-based and frequency-based representations of the same example and their local augmentations are located close together in the time-frequency space.

Original: To this end, we posit that time-frequency consistency (TF-C)— embedding a time-based neighborhood of a particular example close to itsfrequency-based neighborhood and back — is desirable for pre-training.
Revised: To this end, we posit that time-frequency consistency (TF-C) — embedding atime-based neighborhood of a particular example close to its frequency-basedneighborhood and back — is desirable for pre-training.

Original: Motivatedby TF-C, we define a decomposable pre-training model, where the self-supervisedsignal is provided by the distance between time and frequency components, eachindividually trained by contrastive estimation.
Revised: Motivated by TF-C, we optimize a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each indi- vidually trained by contrastive estimation.

Original: We evaluate the new method oneight datasets, including electrodiagnostic testing, human  activity recognition,mechanical fault detection, and physical status monitoring.
Revised: We evaluate the new method on eightdatasets including electrodiagnostic testing, human daily activity recognition, mechanical fault detection, and physical status monitoring.

Original: Experiments againsteight state-of-the-art methods show that TF-C outperforms  baselines by 15.4% (F1score) on average in one-to-one settings ( e.g. , fine-tuning an EEG-pretrained modelon EMG data) and by up to 8.4% (F1 score) in challenging one-to-many settings( e.g. , fine-tuning an EEG-pretrained model for either hand-gesture recognitionor mechanical fault prediction), reflecting the breadth of scenarios that arise inreal-world applications.
Revised: Experiments against eightstate-of-the-art methods show our method outperforms the baselines by 15.4% (F1score) on average in one-to-one settings ( e.g. , fine-tuning an EEG-pretrained modelon EMG data) and by up to 8.4% (F1 score) in challenging one-to-many settings( e.g. , fine-tuning an EEG-pretrained model for either hand-gesture recognitionor mechanical fault prediction), reflecting the breadth of scenarios that arise inreal-world applications.

Original: The source code and  datasets are available at https:anonymous
Revised: The source code and all datasets are available at https:

Original: .4open.science/r/TFC-pretraining-6B07 .

Original:  Introduction
Revised:  Introduction

Original: Time series plays important roles in many areas, including clinical diagnosis, traffic analysis, andclimate science [1, 2, 3, 4, 5, 6].
Revised: Time series plays important roles in many areas, including clinical diagnosis, traffic analysis, andclimate science [1–6].

Original: While representation learning has considerably advanced analysis oftime series [7, 8, 9] and sequences more broadly [10], learning broadly generalizable representationsfor time series remains a fundamentally challenging problem [8, 11].
Revised: While representation learning has considerably advanced analysis of timeseries [7, 8] and sequences more broadly [9], learning broadly generalizable representations for timeseries remains a fundamentally challenging problem [8, 10].

Original: There are numerous immediatebenefits from being able to generate such representations, of which pre-training capability is particularly desirable and of great practical importance [12, 13].
Revised: There are numerous immediate benefitsfrom being able to generate such representations, of which pre-training capability is particularlydesirable and of great practical importance [11, 12].

Original: Central to pre-training is a question of howto process time series in a diverse dataset so as to greatly improve generalization on new time seriescoming from different datasets [14, 15, 10].
Revised: Central to pre-training is a question of how toprocess time series in a diverse dataset so as to greatly improve generalization on new time seriescoming from different datasets [13, 14, 9].

Original: By training a neural network model on a diversifieddataset and transferring it to a new target dataset for fine-tuning, i.e. , without explicit retraining onthat target data, we expect the resulting performance be at least as good as that of state-of-the-artmodels tailored to the target dataset.
Revised: By training a neural network model on a diversifieddataset and transferring it to a new target dataset for fine-tuning, i.e. , without explicit retraining onthat target data, we expect the resulting performance be at least as good as that of state-of-the-artmodels tailored to the target dataset.

Original: [Figure]
Revised: [Figure]

Original: However, unfortunately, the expected performance gains are often not realized for a variety of reasons( e.g. , distribution shifts, properties of the target dataset unknown during pre-training) [16, 17] thatget compounded by the complexity of time series: large variations of temporal dynamics acrossdatasets, varying semantic meaning, irregular sampling, system factors ( e.g. , different devices orsubjects), etc.
Revised: However, unfortunately, the expected performance gains are often not realized for a variety of reasons( e.g. , distribution shifts, properties of the target dataset unknown during pre-training) [15, 16] thatget compounded by the complexity of time series: large variations of temporal dynamics acrossdatasets, varying semantic meaning, irregular sampling, system factors ( e.g. , different devices orsubjects), etc.

Original: This complexity of time series limits the utility of knowledge transfer forpre-training [19, 20].
Revised: This complexity of time series limits the utility of knowledge transfer forpre-training [18, 19].

Original: For example, pre-training a model on a diverse time series dataset with mostlylow-frequency components (smooth trends) may not lead to positive transfer on downstream taskswith high-frequency components (transient events)[17].
Revised: For example, pre-training a model on a diverse time series dataset with mostlylow-frequency components (smooth trends) may not lead to positive transfer on downstream taskswith high-frequency components (transient events)

Original: Examining these challenges can provideclues to what kind of inductive biases could facilitate generalizable representations of time series –this paper offers a strategy for that through a novel time-frequency consistency principle.
Revised: Examining these challenges can provideclues to what kind of inductive biases could facilitate generalizable representations of time series –this paper offers a strategy for that through a novel time-frequency consistency principle.

Original: In addition, target dataset are not available in pre-training phase (different from domain adaption [21];
Revised: In addition, target dataset are not available in pre-training phase (different from domain adaption [20];

Original: Appendix A), requiring that pre-training model captures some latent shared property in time seriesthat can apply to unseen datasets.
Revised: Appendix A), requiring that pre-training model captures some latent shared property in time seriesthat can apply to unseen datasets.

Original: The principle of a shared property is universal in different timeseries datasets and thus allows knowledge transfer from a pre-training dataset to a fine-tuning dataset.
Revised: The principle of a shared property is universal in different timeseries datasets and thus allows knowledge transfer from a pre-training dataset to a fine-tune dataset.

Original: To make it more clear, a key principle for pre-training in computer vision (CV) is that the priorneural network layers learn object edges and shapes regardless of image style or tasks [22].
Revised: To make it more clear, a key principle for pre-training in computer vision (CV) is that the prior neuralnetwork layers learn object edges and shapes regardless of image style or tasks [21].

Original: Theshared foundation for pre-training in natural language processing (NLP) is the linguistic assumptionof semantics and grammars that is retrained across different languages [23].
Revised: The sharedfoundation for pre-training in natural language processing (NLP) is the linguistic assumption ofsemantics and grammars that is retrained across different languages [22].

Original: Nevertheless, it ishard to put forward such a principle for time series due to aforementioned temporal complexity.
Revised: Nevertheless, it is hard toput forward such a principle for time series due to aforementioned temporal characteristics.

Original: Moreover, supervised pre-training requires large annotated datasets, creating an obstacle to deeplearning deployment [24, 25].
Revised: Moreover, supervised pre-training requires large annotated datasets, creating an obstacle to deeplearning deployment [23, 24].

Original: For example, in medical domains, experts often disagree on groundtruth labeling ( e.g. , ECG signals falling in normal or abnormal rhythms [26, 27]), which can introduceunintended biases [28, 29].
Revised: For example, in medical domains, experts often disagree on groundtruth labeling [25, 26], which can introduce unintended biases, e.g. , ECG signals falling outside ofnormal and abnormal rhythms [27, 28].

Original: Self-supervised contrastive learning can overstep the lack of annotationsby providing self-supervision signal through contrastive instance discrimination [30].
Revised: Self-supervised contrastive learning can overstep the lack ofannotations by providing self-supervision signal through contrastive instance discrimination [29].

Original: Present work.
Revised: Present work.

Original: We introduce a strategy for self-supervised pre-training in time series by modeling
Revised: We introduce a strategy for self-supervised pre-training in time series by modeling

Original: Time-Frequency Consistency (TF-C).
Revised: Time-Frequency Consistency (TF-C).

Original: TF-C means the time-based representation and the frequencybased representation, which are learned from the same time series sample, are closer to each other ina joint time-frequency space, and farther apart if the representations are associated with different timeseries samples.
Revised: TF-C means the time-based representation and the frequencybased representation, which are learned from the same time series sample, are closer to each other ina joint time-frequency space, and farther apart if the representations are associated with different timeseries samples.

Original: Specifically, we adopt contrastive learning in time space to generate a time-basedrepresentation.
Revised: Specifically, we adopt contrastive learning in time space to generate a time-basedrepresentation.

Original: In parallel, we propose a set of novel augmentations based on the characteristic offrequency spectrum and produce a frequency-based embedding through contrastive instance discrimination.
Revised: In parallel, we propose a set of novel augmentations based on the characteristicof frequency spectrum and produce a frequency-based embedding through contrastive instancediscrimination.

Original: To the best of our knowledge, this is the first work that develops contrastive augmentationin frequency domain to leverage rich spectral information, also the first work that exploits the property of time-frequency consistency in time series.
Revised: To the best of our knowledge, this is the first work that implements augmentation infrequency domain to leverage rich information in frequency.

Original: The pre-training objective is to minimize thedistance between the time-based and frequency-based embeddings through a dedicated consistencyloss (Figure 1 (a)).
Revised: The pre-training objective is to minimizethe distance between the time-based and frequency-based embeddings through a dedicated TF-Closs (Figure 1 (a)).

Original: The designed loss provides self-supervising signals to optimize the pre-trainingmodel and captures the latent relationships between time domain and frequency domain.
Revised: The TF-C loss provides self-supervising signals to optimize the pre-trainingmodel and captures the latent relationships between time domain and frequency domain.

Original: The learnedrelationship  encoded in model parameters are transferred to initialize the fine-tuning model andimprove performance in datasets of interest (Figure 1 (b)).
Revised: The learnedrelationship ( i.e. , TF-C) encoded in model parameters are transferred to initialize the fine-tuningmodel and improve performance in datasets of interest (Figure 1 (b)).

Original: We evaluate the proposed pre-training model on eight time series datasets under two evaluationsettings ( i.e. , one-to-one and one-to-many).
Revised: We evaluate the proposed pre-training model on eight time series datasets under two evaluationsettings (one-to-one and one-to-many).

Original: The eight datasets cover a large set of variations: differentnumber of channels (from univariate to 9-channel multivariate), varying time series length (from128 to 5,120), different sampling rate (from 16 Hz to 4,000 Hz), different scenarios (neurological healthcare, human activity recognition, mechanical fault detection, physical status monitoring, etc.)and diverse types of signals (EEG, EMG, ECG, acceleration, and vibration).
Revised: The eight datasets cover a large set of variations: differentnumber of channels (from univariate to 9-channel multivariate), varying time series length (from128 to 5,120), different sampling rate (from 16 Hz to 4,000 Hz), different scenarios (neurologicalhealthcare, human activity recognition, mechanical fault detection, physical status monitoring, etc.) and diverse types of signals (EEG, EMG, ECG, acceleration, and vibration).

Original: We compare TF-Capproach to eight state-of-the-art baselines.
Revised: We compare TF-Capproach to eight state-of-the-art baselines.

Original: Results show that TF-C achieves positive transfer,outperformingall baselines by a large margin of 15.4% (F1 score) on average.
Revised: Results show that TF-C achieves positive transfer,outperforming baselines by a large margin of 15.4% (F1 score) on average.

Original: Further, the approachoutperforms the strongest baselines with an improvement of up to 7.2% in F1 score.
Revised: Further, the approachoutperforms the strongest baselines with an improvement of up to 7.2% in F1 score.

Original: Finally, the
Revised: Finally, the

Original: TF-C approach improves prior work by 8.4% in precision (when pre-training the model on sleep
Revised: TF-C approach improves prior work by 8.4% in precision (when pre-training the model on sleep

Original: EEG signals and fine-tuning it on hand-gesture recognition) in a challenging one-to-many setup thatapplies the same pre-trained model to multiple independent fine-tuning datasets.
Revised: EEG signals and fine-tuning it on hand-gesture recognition) in a challenging one-to-many setup thatapplies the same pre-trained model to multiple independent fine-tuning datasets.

Original:  Related Work
Revised:  Related Work

Original: Pre-training for time series.
Revised: Pre-training for time series.

Original: Although there are  studies on self-supervised representation learning fortime series [7, 8, 31, 32] and  self-supervised pre-training for images [33, 34, 35, 24], the intersectionof these two areas, i.e. , self-supervised pre-training for time series, remains underexplored.
Revised: Although there are some studies about self-supervised representationlearning for time series [7, 8, 30, 31] and many works on self-supervised pre-training on imagedata [32–34, 23], their intersection, self-supervised pre-training on time series data, still remainsinsufficiently explored.

Original: In timeseries, it’s not obvious that what assumptions are reasonable that can bridge pre-training dataset andtarget dataset.
Revised: In time series, it’s not obvious that what assumptions are reasonable thatcan bridge pre-training and target datasets.

Original: Hence, pre-training models in CV [36, 37, 14] and NLP [10, 15, 38] are not directlyapplicable due to data modality mismatch, andthe results leave room for improvement [31, 39, 40].
Revised: Hence, pre-training models in CV [35, 36, 13] and NLP [9, 14, 37] are not directly applicable due to data modality mismatch, and results leave roomfor improvement [30, 38, 39].

Original:  Shi et al.
Revised: So far, only Shi et al.

Original: [12] developed the only modelto date that is explicitly designed for self-supervised timeseries pre-training.
Revised: proposed a model for self-supervised timeseries pre-training explicitly [11].

Original: The model captures the local and global temporal pattern but it’s not convincingwhy the designed pretext task can capture generalizable representations.
Revised: The model captures the local and global temporal pattern but it’snot convincing why the designed pretext task can capture generalizable representations.

Original: Although several studiesapplied transfer learning inthe context of time series [7, 8, 18, 41], there is no foundation yet ofwhich conceptual properties are most suitable for pre-training on time series and why.
Revised: Althoughseveral studies applies to transfer setting in time series [7, 8, 17, 40], there is no devoted conceptualassumptions or dedicated architectural modules to justify why their model empowers knowledgetransfer across datasets.

Original: Addressingthis gap, we show that TF-C, designed to be invariant to different time-series datasets, can producegeneralizable pre-training models.
Revised: Addressing this gap, we posit the assumption of TF-C which is invariant todifferent time series datasets and can guide the development of effective pre-training models.

Original: Unlike domain adaptation [21, 42] that requires access to target datasets during training, pre-trainingmodels have accessonly to  pre-training datasets. As a result, one needs to identify a generalizabletime-series property to benefit from pre-training.
Revised: Different from domain adaptation models [20, 41] that implies target datasets are required to in modeltraining, our task only have access to the pre-training dataset which demands more generalizeableproperty in time series.

Original: Further, self-supervised domain adaptation does notneed labels inthe target dataset but still requires labels for model training [43, 44]. In contrast, TF-Cdoes not need any labels during pre-training.
Revised: Self-supervised domain adaptation means no labels in target dataset but stillrequire labels in model training [42, 43] while our model does not rely on labels in model pre-training.

Original: Contrastive learning with time series.
Revised: Contrastive learning with time series.

Original: Contrastive learning, a popular type of self-supervisedlearning, aims to learn an encoder that maps inputs into an embedding space such that positive samplepairs (original augmentation and another alternative augmentation/view of the same input sample)are pulled closer and negative sample pairs (original augmentation and an alternative input sampleaugmentation) are pushed apart [30, 45].
Revised: Contrastive learning, a popular type of self-supervisedlearning, aims to learn an encoder that maps inputs into an embedding space such that positive samplepairs (original augmentation and another alternative augmentation/view of the same input sample)are pulled closer and negative sample pairs (original augmentation and an alternative input sampleaugmentation) are pushed apart [29, 44].

Original: Contrastive learning in time series is less investigated incomparison, partly due to the challenge of identifying augmentations that capture key invarianceproperties in time series data.
Revised: Contrastive learning in time series is less investigated incomparison, partly due to the challenge of identifying augmentations that capture key invarianceproperties in time series data.

Original: For example, CLOCS defines adjacent segments of a time series aspositive pairs [41], and TNC assumes overlapping neighborhoods of time series should have similarrepresentations [46].
Revised: For example, CLOCS defines adjacent segments of a time series aspositive pairs [40], and TNC assumes overlapping neighborhoods of time series should have similarrepresentations [45].

Original: These methods leverage temporal invariance to define positive pairs whichare used to calculate contrastive loss, but other invariances, such as transformation invariance ( e.g. , SimCLR [40]), contextual invariance ( e.g. , TS2vec [47] and TS-TCC [48]) and augmentations arepossible.
Revised: These methods leverage temporal invariance to define positive pairs whichare used to calculate contrastive loss, but other forms of invariance- based, such as transformation( e.g. , SimCLR [39]) or contextual invariance ( e.g. , TS2vec [46], TS-TCC [47] ), augmentationsare also possible.

Original: In this work, we propose an augmentation bank that exploits multiple invariances to generatediverse augmentations (Sec. 4.1), which adds richness into the pre-training model [48].
Revised: In this work, we propose an augmentation bank that exploits multiple invariancesto generate diverse augmentations (Sec. 4.1), which adds richness into the pre-training model [47].

Original: Importantly,we propose frequency-based augmentations by perturbing the frequency spectrum of time series ( e.g. ,adding or removing the frequency components and manipulating the their amplitude; more details in Sec. 4.2) to learn better representations by exposing the model to local range of frequency variations.
Revised: Importantly, we propose frequency-based augmentations by perturbing the frequency spectrum oftime series ( e.g. , adding or removing the frequency components and manipulating the their amplitude;more details in Sec. 4.2) to learn better representations by exposing the model to local range offrequency variations.

Original: A previous work CoST processes sequential signals through frequency domain but the augmentationsare still implemented in time space [49].
Revised: A previous work CoST processes sequential signals through frequency domainbut the augmentations are still implemented in time space [48].

Original: Similarly, although BTSF [50] involves frequency domain,its data transformation is solely implemented in time domain using instance-level dropout.

Original: To thebest of our knowledge, this is the first work that directly perturbs the frequency spectrum to leveragefrequency-invariance for contrastive learning.
Revised: To the best of our knowledge, thisis the first work that develops augmentations in frequency domain to leverage frequency-invariancefor contrastive learning.

Original: Further, we develop a pre-training model that subjectsto TF-C upon two individual contrastive encoders.
Revised: Further, we propose a pre-training model that subjects to TF-C upon twoindividual contrastive encoders.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: We are given a pre-training dataset D pret =
Revised: We are given a pre-training dataset D pret =

Original: { x pret i | i = 1 , . .
Revised: { x pret i | i = 1 , . .

Original: . , N } of unlabeled time series sampleswhere sample x pret i has K pret channels and L pret timestamps.
Revised: . , N } of unlabeled time series sampleswhere sample x pret i has K pret channels and L pret timestamps.

Original: Let D tune =
Revised: Let D tune =

Original: { ( x tune i , y i ) | i = 1 , . .
Revised: { ( x tune i , y i ) | i = 1 , . .

Original: , M }be a fine-tuning ( i.e. , target; target and fine-tuning are used interchangeably) dataset of labeled timeseries samples, each having K tune channels and L tune timestamps.
Revised: , M }be a fine-tuning ( i.e. , target; target and fine-tuning are used interchangeably) dataset of labeled timeseries samples, each having K tune channels and L tune timestamps.

Original: Further, every sample x tune i is associated with label y i ∈ { 1 , . ..
Revised: Further, every sample x tune i isassociated with label y i ∈ { 1 , . .

Original: , C } , where C is the number of classes.
Revised: , C } , where C is the number of classes.

Original: Without loss of generality,in the following, we describe the approach for univariate (single-channel) time series, noting that theapproach can accommodate multivariate time series, different numbers of channels across samples,and varying lengths (shown in experiments in Sec. 5.2).
Revised: Without loss of generality,in the following, we describe the approach for univariate (single-channel) time series, noting that the approach can accommodate multivariate time series, different numbers of channels across samples,and varying lengths (shown in experiments in Sec. 5.2).

Original: We use superscript symbol (cid:101) to denotecontrastive augmentations.
Revised: We use superscript symbol (cid:101) to denotecontrastive augmentations.

Original: We note that x T i ≡ x i denotes an input time series sample, and x F i denotesthe frequency spectrum of x i .
Revised: We note that x T i ≡ x i denotes an input time series sample, and x F i denotesthe frequency spectrum of x i .

Original: Problem (Self-Supervised Contrastive Pre-Training For Time Series).
Revised: Problem (Self-Supervised Contrastive Pre-Training For Time Series).

Original: Given are an unlabeledpre-training dataset D pret with N samples and a target dataset D tune with M samples ( M ≪ N ).
Revised: Given are an unlabeledpre-training dataset D pret with N samples and a target dataset D tune with M samples ( M ≪ N ).

Original: Wewant to use D pret to pre-train a neural model F so that by fine-tuning model parameters on D tune , thefine-tuned model can generate a generalizable representation z tune i =
Revised: Wewant to use D pret to pre-train a neural model F so that by fine-tuning model parameters on D tune , thefine-tuned model can generate a generalizable representation z tune i =

Original: F ( x tune i ) for every x tune i .
Revised: F ( x tune i ) for every x tune i .

Original: We follow established pre-training setup, e.g. , [41]; for pre-training, only unlabeled dataset D pretis available while, for fine-tuning, a small labeled dataset D tune can be used.
Revised: We follow established pre-training setup, e.g. , [40]; for pre-training, only unlabeled dataset D pretis available while, for fine-tuning, a small labeled dataset D tune can be used.

Original: In short, a model
Revised: In short, a model

Original: F is pre-trained on unlabeled time series dataset D pret and its optimized model parameters Θ arefine-tuned to go from F ( · , Θ) to F ( · , Φ) using dataset D tune .
Revised: F is pre-trained on unlabeled time series dataset D pret and its optimized model parameters Θ arefine-tuned to go from F ( · , Θ) to F ( · , Φ) using dataset D tune .

Original: The Φ denotes fine-tuned modelparameters.
Revised: The Φ denotes fine-tuned modelparameters.

Original: Note that this problem ( i.e. , D pret is independent of target dataset) is different thandomain adaptation because we don’t access the fine-tuning dataset D tune during pre-training, meaningthat a successful pre-trained model should be applicable to multiple independent fine-tuning datasetswithout necessitating re-training.
Revised: Note that this problem ( i.e. , D pret is independent of target dataset) is different thandomain adaptation because we don’t access the fine-tuning dataset D tune during pre-training, meaningthat a successful pre-trained model should be applicable to multiple independent fine-tuning datasetswithout necessitating re-training.

Original: Rationale for Time-Frequency Consistency (TF-C).
Revised: Rationale for Time-Frequency Consistency (TF-C).

Original: For effective per-training on time series,our central idea is to identify a broadly generalizable property that is preserved across differenttime series datasets.
Revised: For effective per-training on time series,our central idea is to identify a broadly generalizable property that is preserved across differenttime series datasets.

Original: In time series in particular, the time domain shows how readouts change withtime whereas frequency domain tells us how much of the signal lies within each frequency bandover a range of frequencies ( e.g. , frequency spectrum) [51].
Revised: In time series in particular, the time domain shows how readouts change withtime whereas frequency domain tells us how much of the signal lies within each frequency bandover a range of frequencies ( e.g. , frequency spectrum) [49].

Original: Explicitly considering the frequencydomain can provide a deeper understanding of time series behavior that can be challenging tocapture solely in time domain [52].
Revised: Explicitly considering the frequencydomain can provide a deeper understanding of time series behavior that can be challenging tocapture solely in time domain [50].

Original: However, most existing time-series methods focus exclusively onmodeling time domain, e.g. , [47, 48], and ignore frequency domain altogether.
Revised: However, most existing time-series methods focus exclusively onmodeling time domain, e.g. , [46, 47], and ignore frequency domain altogether.

Original: One could argue thatis sufficient for high-capacity methods as time and frequency domains describe the same data, justfrom different aspects [53], and time series can be converted between time and frequency domainsusing transformation, such as Fourier [54, 52].
Revised: One could argue thatis sufficient for high-capacity methods as time and frequency domains describe the same data, justfrom different aspects [51], and time series can be converted between time and frequency domainsusing transformation, such as Fourier [52, 50].

Original: While we are not interested in modeling the twodomains individually, the relationship between them, grounded in signal processing theory, providesan invariance that is valid regardless of time series distribution [55, 56] and thus can serve as aninductive bias for pre-training to promote generalizability.
Revised: While we are not interested in modeling the twodomains individually, the relationship between them, grounded in signal processing theory, providesan invariance that is valid regardless of time series distribution [53, 54] and thus can serve as aninductive bias for pre-training to promote generalizability.

Original: Approaching this invariance through thelens of representation learning, we formulate Time-Frequency Consistency (TF-C) by postulatingthat for every time
Revised: Approaching this invariance through thelens of representation learning, we formulate Time-Frequency Consistency (TF-C) by postulatingthat for every time

Original: series sample x i , there exists a latent time-frequency space in which time-basedrepresentation z T i and frequency-based representation z F i of the same sample, together with their localaugmentations (defined later) are close to each other.
Revised: series sample x i , there exists a latent time-frequency space in which time-basedrepresentation z T i and frequency-based representation z F i of the same sample, together with their localaugmentations (defined later) are close to each other.

Original: Representational Time-Frequency Consistency (TF-C).
Revised: Representational Time-Frequency Consistency (TF-C).

Original: Let be given a time series sample x i .
Revised: Let be given a time series sample x i .

Original: Then in a model F satisfying TF-C, time-based representation z T i and frequency-based representationz F i learned from x i by F , and representations learned from local augmentations of x i are closetogether in the latent time-frequency space.
Revised: Then in a model F satisfying TF-C, time-based representation z T i and frequency-based representationz F i learned from x i by F , and representations learned from local augmentations of x i are closetogether in the latent time-frequency space.

Original: Our pre-training strategy is to use a time series dataset D pret to induce TF-C in F ’s model parametersΘ , which, in turn, are used to initialize the target model on D tune and produce generalizable representations for downstream prediction.
Revised: Our pre-training strategy is to use a time series dataset D pret to induce TF-C in F ’s model parametersΘ , which, in turn, are used to initialize the target model on D pret and produce generalizable representations for downstream prediction.

Original: The invariant nature of TF-C means that our approach can bridge
Revised: The invariant nature of TF-C means that our approach can bridge

Original: D pret and D tune even when large discrepancies exist between them (in terms of temporal dynamics,semantic meaning, etc.), providing a vehicle for a generalizable pre-training strategy for time series.
Revised: D pret and D pret even when large discrepancies exist between them (in terms of temporal dynamics,semantic meaning, etc.), providing a vehicle for a generalizable pre-training strategy for time series.

Original: To realize TF-C, our model F has four components (Figure 2): a time encoder G T , a frequencyencoder G F , and two cross-space projectors R T and R F that map time-based and frequency-basedrepresentations, respectively, to the same time-frequency space.
Revised: To realize TF-C, our model F has four components (Figure 2): a time encoder G T , a frequencyencoder G F , and two cross-space projectors R T and R F that map time-based and frequency-basedrepresentations, respectively, to the same time-frequency space.

Original: Together, the four componentsprovide a way to embed x i to the latent time-frequency space such that time-based embeddingz T i
Revised: Together, the four componentsprovide a way to embed x i to the latent time-frequency space such that time-based embeddingz T i

Original: =
Revised: =

Original: R T ( G T ( x T i )) and frequency-based embedding z F i = R F ( G F ( x F i )) are close together.
Revised: R T ( G T ( x T i )) and frequency-based embedding z F i = R F ( G F ( x F i )) are close together.

Original:  Our Approach
Revised:  Our Approach

Original: Next, we present the architecture of the developed self-supervised contrastive pre-training model
Revised: Next, we present the architecture of the proposed self-supervised contrastive pre-training model

Original: F .
Revised: F .

Original: Without specification, the data mentioned in this section are from pre-training dataset and the [Figure] superscript pret is omitted for simplification.
Revised: Without specification, the data mentioned in this section are from pre-training dataset and thesuperscript pret is omitted for simplification.

Original: Here we describe the model using univariate time seriesas an example, but our model can be straightforwardly applied to multivariate time series (Sec 5). 
Revised: Here we describe the model using univariate time seriesas an example, but our model can be straightforwardly applied to multivariate time series (Sec 5). [Figure]

Original: 4.1 Time-based Contrastive Encoder
Revised: 4.1 Time-based contrastive encoder

Original: Fora given input time series sample x i , we generate an augmentation set X T i through a time-basedaugmentation bank B T : x T i → X T i .
Revised: For input time series sample x i , we generate an augmentation set X T i through an augmentationbank B T : x T i → X T i .

Original: Each element (cid:101) x T i ∈
Revised: Each element (cid:101) x T i ∈

Original: X T i is  augmented from x i based onthe temporal characteristics.
Revised: X T i is a sample that is augmented from x i based on thetemporal characteristics.

Original: Here, the time-based augmentation bank includes jittering, scaling,time-shifts, and neighborhood segments, all well-established in contrastive learning [40, 48, 41].
Revised: Augmentation bank includes jittering, scaling, time-shifts, and neighborhoodsegments, all established in contrastive learning [39, 47, 40].

Original: We develop the augmentation bank to produce diverse augmentations (rather than a single type ofaugmentation) and expose the model to complex temporal dynamics, which produces more robusttime-based embeddings [48].
Revised: We develop the augmentation bank toproduce diverse augmentations (rather than a single types of augmentations) and expose the model tocomplex temporal dynamics, producing robust time-based embeddings [47].

Original: For the input x i , we randomly select an augmented sample (cid:101) x T i ∈ X T i and feed into a contrastivetime encoder G Tthat maps samples to embeddings. We have h T 
Revised: We randomly select an element (cid:101) x T i as an augmented time series and use the time encoder G T to mapsamples to embeddings h T i =

Original: i = G T ( x T i )and (cid:101) h T i = G T ( (cid:101) x T i ) .
Revised: G T ( x T i ) .

Original: As(cid:101) x T i is generated based on x T i , after passing through G T , we assume the embedding of x T i is close tothe embedding of (cid:101) x T i but far away from the embedding of x T j and (cid:101) x
Revised: As (cid:101) x T i is generated based on x T i , after passing through G T , weassume the embedding of x T i is close to the embedding of (cid:101) x T i but far away from the embedding of x T jand (cid:101) x

Original: T j that are derived from anothersample x T j ∈ D pret [34, 47, 41].
Revised: T j that are derived from another sample x T j ∈ D pret .

Original: In specific, we select the positive pair as ( x T i , (cid:101) x T i ) and negative pairsas ( x T i , x T j ) and ( x T i , (cid:101) x T j ) [34].
Revised: In specific, we select the positive pair as( x T i , (cid:101) x T i ) and negative pairs as ( x T i , x T j ) and ( x T i , (cid:101) x T j ) .

Original: Contrastive time loss.
Revised: Contrastive time loss.

Original: To maximize the similarity within a positive pair and minimize the similaritywithin a negative pair, we adopt the NT-Xent (the normalized temperature-scaled cross entropy loss)as distance function d which is widely used in contrastive learning [34, 40].
Revised: To maximize the similarity within a positive pair and minimize the similaritywithin a negative pair, we adopt the NT-Xent (the normalized temperature-scaled cross entropy loss)as distance function d which is widely used in contrastive learning [33, 39].

Original: In specific, we measurethe loss of time-based contrastive encoder in terms of sample x T i by:
Revised: In specific, we measurethe loss of time-based contrastive encoder in terms of sample x T i by:

Original: [Equation]
Revised: [Equation]

Original: where sim ( u , v )
Revised: where sim ( u , v )

Original: =
Revised: =

Original: u T v / ∥ u ∥ ∥ v ∥ denotes the cosine similarity, the 1 i ̸ = j ∈ { 0 , 1 } is a binaryindicator that equals to 0 when i = j , and τ is a temporal parameter.
Revised: u T v / ∥ u ∥ ∥ v ∥ denotes the cosine similarity, the 1 i ̸ = j ∈ { 0 , 1 } is a binaryindicator that equals to 0 when i = j , and τ is a temporal parameter.

Original: The x j ∈ D pret refers toa different time series sample  and its augmented sample.
Revised: Here, the x j ∈ D pret refers toboth the original sample x T j ∈ D pret and the corresponding augmented sample (cid:101) x T j ∈ X T j .

Original: This loss urges the time encoder G T togenerate closertime-based embeddings for positive pairs, and push the embeddings for negative pairsapart from each other.
Revised: This lossurges the time encoder G T to generate closer embeddings for positive pairs, and push the embeddingsfor negative pairs apart from each other.

Original: 4.2 Frequency-based Contrastive Encoder
Revised: 4.2 Frequency-based contrastive encoder

Original: We generate the frequency spectrum x F i from time series x T i through a transform operator ( e.g. , Fourier Transformation [54]).
Revised: Next, we generate the frequency spectrum x F i from time series x T i through a transform operator,e.g. , [52, 55].

Original: The frequency information of time series is universal and plays keyrole in classic signal processing [57, 53, 55], but rarely investigated in self-supervised contrastive learning for time series [58].

Original: In this section, we develop augmentation methods to perturb x F i basedon characteristics of frequency spectra and generate frequency-based representations.
Revised: We consider augmentation methods to perturb x F i based on characteristics of frequencyspectra.

Original: As every frequency component in the frequency spectrum denotes a base function ( e.g. , sinusoidalfunction for Fourier transformation) with the corresponding frequency and amplitude, in our augmentation methods, we perturb the frequency spectrum by adding or removing frequency components.
Revised: As every frequency component in the frequency spectrum denotes a base function ( e.g. ,sinusoidal function for Fourier transformation) with the corresponding frequency and amplitude. Inour augmentation methods, we perturb the frequency spectrum by adding or removing frequency components.

Original: Asmall perturbation on frequency spectrum may cause large changes to the temporal patterns in timedomain [55].
Revised: The small change of frequency spectrum will largely affect the temporal patterns intime domain [53].

Original: To make sure the perturbed time series is still similar to the original sample (not only infrequency domain but also in time domain; Figure 6), we use a small budget E in the perturbationswhere E denotes the number of frequency components we manipulate.
Revised: To make sure the perturbed time series is still similar to the original sample, weuse a small budget ∆ in the perturbations where ∆ denotes the number of frequency components weperturb.

Original: While removing frequencycomponents, we randomly select E frequency components and set their amplitudes as 0 .
Revised: While removing frequency components, we randomly select ∆ frequency components andset their amplitudes as 0 .

Original: Whileadding frequency components, we randomly choose E frequency components from the ones havesmaller amplitude than α · A m , and increase their amplitude to α · A m .
Revised: While adding frequency components, we randomly choose ∆ frequencycomponents from the ones have smaller amplitude than α · A m , and increase their amplitude to α · A m .

Original: The A m is the maximumamplitude in the frequency spectrum and α is a pre-defined coefficient to adjust the scale of theperturbed frequency component ( α = 0 .
Revised: The A m is the maximum amplitude in the frequency spectrum and α is a pre-defined coefficientto adjust the sale of the perturbed frequency component ( α = 0 .

Original: 5 in this work).
Revised: 5 in this work).

Original: We produce an augmentation set X F i forx F i through frequency-augmentation bank B F : x F i → X F i .
Revised: We produce anaugmentation set X F i for x F i through frequency-augmentation bank B F : x F i → X F i .

Original: We have two augmentation methods ( i.e. ,removing or adding frequency components) in B F , |X F i | = 2 .
Revised: We have twoaugmentation methods ( i.e. , removing/adding frequency components) in B F , |X F i | = 2 .

Original: We utilize a frequency encoder G F to map the frequency spectrum ( e.g. , x F i ) to a frequency-basedembedding ( e.g. , h F i = G F ( x F i ) ).
Revised: We utilize a frequency encoder G F to map the frequency spectrum ( e.g. , x F i ) to a frequency-basedembedding ( e.g. , h F i = G F ( x F i ) ).

Original: We assume the frequency encoder G F can learn similar embeddingfor the original frequency spectrum x F i and a slightly perturbed frequency spectrum (cid:101)
Revised: We assume the frequency encoder G F can learn similar embeddingfor the original frequency spectrum x F i and a slightly perturbed frequency spectrum (cid:101)

Original: x F i ∈
Revised: x F i ∈

Original: X F i .
Revised: X F i .

Original: Thus,we set the positive pair as ( x F i , (cid:101) x F i ) and the negative pairs as ( x F i , x F j ) and ( x F i , (cid:101) x F [Equation]
Revised: Thus,we set the positive pair as ( x F i , (cid:101) x F i ) and the negative pairs as ( x F i , x F j ) and ( x F i , (cid:101) x F j ) .

Original: Contrastive frequency loss.
Revised: Contrastive frequency loss.

Original: We calculate frequency-based contrastive loss for sample x i as:
Revised: We calculate frequency-based contrastive loss for sample x i as:

Original: [Equation]
Revised: [Equation]

Original: In preliminary experiments, we find that the value of τ has little effect on performance and use thesame τ throughout all experiments.
Revised: In preliminary work, we find that the value of τ has little effect on performance and use the same τthroughout all experiments.

Original: The L F ,i yield a frequency encoder G F producing embeddingsinvariant to frequency spectrum perturbations.
Revised: The L F ,i yield a frequency encoder G F producing embeddings invariantto frequency spectrum perturbations.

Original: 4.3 Time-Frequency Consistency
Revised: 4.3 Time-frequency consistency

Original: We develop a consistency lossitem L C ,i to urge the learned embeddings to satisfy TF-C: for a givensample, its time-based and frequency-based embeddings (and their local neighborhoods) are supposedto be close to each other (see Sec.
Revised: Our consistency loss L C urges learned embeddings to satisfy TF-C: for a given sample, its time-basedand frequency-based embeddings and their local neighborhoods should be close to each other.

Original: 3 for justification).

Original: To make sure the distance between embeddingsis measurable, we map  h T i from  time space and  h F i from  frequency space to a joint time-frequencyspace through projectors R T and R F , respectively.
Revised: Wefirst map the h T i from latent time domain and the h F i from latent frequency domain to a time-frequencydomain through projector R T and R F , respectively.

Original: In specific, for every input sample x i , we havefour embeddings, which are z T 
Revised: Particularly, for every input sample x i , we havefour embeddings, which are z T i

Original: i =
Revised: =

Original: R T ( h T i ) , (cid:101)
Revised: R T ( h T i ) , (cid:101)

Original: z T i
Revised: z T i

Original: =
Revised: =

Original: R T ( (cid:101) h T i ) , z F i
Revised: R T ( (cid:101) h T i ) , z F i

Original: =
Revised: =

Original: R F ( h F i ) , and (cid:101)
Revised: R F ( h F i ) , and (cid:101)

Original: z F i
Revised: z F i

Original: =
Revised: =

Original: R F ( (cid:101) h F i ) .
Revised: R F ( (cid:101) h F i ) .

Original: The firsttwo embeddings are generated based on temporal characteristics and the latter two embeddings areproduced based on the properties of frequency spectrum. To enforce the embeddings inthe time-frequency space subject to TF-C, we design a consistency loss
Revised: To enforce the embeddings in time-frequency domain subject to TF-C, we design a novel TF-C lossthat measures the distance between a time-frequency embedding pair. The time-frequency embeddingpair refers to the pair of a time-based embedding and a frequency-based embedding. Thus, for asample x i , we have four time-frequency embedding pairs, which are ( z T i , z F i ) , ( z T i , (cid:101)

Original: L C ,i that measures the distance between a time-based embedding and a frequency-based embedding.
Revised: z F i ) , ( (cid:101) z T i , z F i ) , and( (cid:101) z T i , (cid:101) z F i ) .

Original: We use S T F i =
Revised: We use S T F i =

Original: d ( z T i , z F i , D pret ) to denote the distance between z T i and z F i ) .
Revised: d ( z T i , z F i , D pret ) to denote the distance for embedding pair ( z T i , z F i ) .

Original: Similarly, we define S T (cid:101) F i , S (cid:101) TF i , and S (cid:101) T (cid:101) F i .
Revised: Similarly,we define S T (cid:101) F i , S (cid:101) TF i , and S (cid:101) T (cid:101) F i .

Original: Note, in this time-frequency space, we don’t consider the distance between z T i and (cid:101)
Revised: Note, in the time-frequency domain, we don’t consider the pair ( z T i , (cid:101)

Original: z T iwhere the two embeddings are from  the same domain ( i.e. , time domain).
Revised: z T i )where the two embeddings are from from the same domain ( i.e. , time domain).

Original: The same applies to pairthe distance between z F i and (cid:101)
Revised: It’s the same for pair( z F i , (cid:101)

Original: z F i  .
Revised: z F i ) .

Original: We have already considered information of above two pairs in thecalculation of L T ,i and L F ,i .
Revised: We have already considered the above two pairs in the calculation L T ,i and L F ,i .

Original: Next, let’s closely observe S T F i and S T (cid:101)
Revised: Next, let’s closely observe S T F i and S T (cid:101)

Original: F i that involve three embeddings: z T i , z F i , and (cid:101)
Revised: F i that involve three embeddings: z T i , z F i , and (cid:101)

Original: z F i .
Revised: z F i .

Original: Here, z T i andz F i are learned from the original sample ( x T i and x F i ) while (cid:101)
Revised: Here, z T i andz F i are learned from the original sample ( x T i and x F i ) while (cid:101)

Original: z F i is learned from the augmented (cid:101) x
Revised: z F i is learned from the augmented (cid:101) x

Original: F i .
Revised: F i .

Original: Thus,intuitively, z T i should be closer to z F i in comparison to (cid:101)
Revised: Thus,intuitively, z T i should be closer to z F i in comparison to (cid:101)

Original: z F i .
Revised: z F i .

Original: Motivated by the relative relationship, weencourage the proposed model to learn a S T F i that is smaller than S T (cid:101) F i .
Revised: Motivated by the relative relationship,we encourage the proposed model to learn a S T F i that is smaller than S T (cid:101) F i .

Original: Inspired by the triplet loss [59],we design ( S T F i − S T (cid:101) F i + δ ) as a term of consistency lossL C ,i where δ is a given constant marginto keep negative samples far apart [60].
Revised: Inspired by the tripletloss [56, 57], we design ( S T F i − S T (cid:101) F i + δ ) as a term of TF-C loss where δ is a given constant margin.

Original: This term optimize the model towards a smaller S T F i andrelatively larger S T (cid:101) F i .
Revised: This term optimize the model towards a smaller S T F i and relatively larger S T (cid:101) F i .

Original: Similarly, S TF i is supposed to be smaller than S (cid:101) TF i and S (cid:101) T (cid:101) F i , respectively.
Revised: Similarly, S TF i is supposed to be smaller than S (cid:101) TF i and S (cid:101) T (cid:101) F i , respectively.

Original: Insummary, we calculate the consistency loss L C , i for sample x i by:
Revised: In summary, we calculate theconsistency loss L C ,i for sample x i by:

Original: [Equation]
Revised: [Equation]

Original: where S pair i denotes between a time-based embedding ( e.g. , z T i or (cid:101) z T i ) and a frequency-based embedding ( e.g. , z F i or (cid:101) z F i ). In whichpair, there is at least one embedding that is derived from augmentedsample instead of the original sample.
Revised: where S pair i denotes the time-frequency embedding pair in which at least one embedding that isderived from augmented sample in time domain or frequency domain and δ is a pre-defined constant.

Original: The δ is a pre-defined constant.

Original: By combining all the tripletloss items, L C encourages the pre-training model to capture the consistency between time-based andfrequency-based embeddings in model optimization.
Revised: By combining three triplet loss items, L C stimulates the pre-training model to consider the consistencybetween time-based and frequency-based embeddings in model optimization.

Original: Note, although the Eq.
Revised: Although the Eq.

Original: 3 does not explicitlymeasure the loss across different time series samples ( e.g. , x i and x j ), the cross-sample relationshipsare implicitly covered in the calculation of S TF i and S pair i .
Revised: 3 doesnot explicitly measure the loss across different time series samples ( e.g. , x i and x j ), the cross-samplerelationships are implicitly covered in the calculation of S TF i and S pair i .

Original: 4.4 Implementation and Technical Details
Revised: 4.4 Implementation and technical details

Original: The overall loss function in pre-training has three terms.
Revised: The overall loss function in pre-training has three terms.

Original: First, time-based contrastive loss L T urges themodel to learn embeddings invariant to temporal augmentations.
Revised: First, time-based contrastive loss L T urges themodel to learn embeddings invariant to temporal augmentations.

Original: Second, frequency-based contrastiveloss L F promotes learning of embeddings invariant to frequency spectrum-based augmentations.
Revised: Second, frequency-based contrastiveloss L F promotes learning of embeddings invariant to frequency spectrum-based augmentations.

Original: Third, the consistency loss L C guides the model to retain the consistency between time-based andfrequency-based embeddings.
Revised: Third, the consistency loss L C guides the model to retain the consistency between time-based andfrequency-based embeddings.

Original: In summary, pre-training loss is the following:
Revised: In summary, pre-training loss is the following:

Original: [Equation]
Revised: [Equation]

Original: ) where λ controls the relative importance of contrastive and consistency losses.
Revised: where λ controls the relative importance of contrastive and consistency losses.

Original: We calculate thetotal loss by summing L TF-C ,i across all pre-training samples.
Revised: We calculate thetotal loss by summing L TF-C ,i across all pre-training samples.

Original: In implementation, the contrastivelosses are calculated within the batch.
Revised: In implementation, the contrastivelosses are calculated within the batch.

Original: Recall our problem definition, the model F we want to learn isthe combination of neural networks G T , R T , G F , and R F .
Revised: Recall our problem definition, the model F we want to learn isthe combination of neural networks G T , R T , G F , and R F .

Original: When pre-training is completed, we storeparameters of entire model, and denote it as F ( · , Θ) where Θ represents all trainable parameters.
Revised: When pre-training is completed, we storeparameters of entire model, and denote it as F ( · , Θ) where Θ represents all trainable parameters.

Original: When a test sample x tune i is presented, fine-tuned model F ( · , Φ) generates embedding z tune i byconcatenating time- and frequency-based embeddings as: z tune
Revised: When a test sample x tune i is presented, fine-tuned model F ( · , Φ) generates embedding z tune i byconcatenating time- and frequency-based embeddings as: z tune

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ]where Φ denotes fine-tuned model parameters.
Revised: [ z tune [Equation]

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We evaluate the new approach on 8 diverse datasets.
Revised: We evaluate the new approach on 8 diverse datasets.

Original: (1) S LEEP E EG [61] has 371,055univariate brainwaves (EEG; 100 Hz) collected from 197 individuals.
Revised: (1) S LEEP E EG [58] has 371,055univariate brainwaves (EEG; 100 Hz) collected from 197 individuals.

Original: Each sample associates withone among five sleeping stages.
Revised: Each sample associates withone among five sleeping stages.

Original: (2) E PILEPSY [62] monitors the brain activities of 500 subjects withsingle-channel EEG sensor (174 Hz).
Revised: (2) E PILEPSY [59] monitors the brain activities of 500 subjects withsingle-channel EEG sensor (174 Hz).

Original: Sample is labelled in binary based on whether the subject hasepilepsy or not.
Revised: Sample is labelled in binary based on whether the subject haveepilepsy or not.

Original: (3) F D - A [63] gathers the vibration signals from rolling bearing from a mechanicalsystem aiming at fault detection.
Revised: (3) F D - A [60] gathers the vibration signals from rolling bearing from a mechanicalsystem aiming at fault detection.

Original: Every sample has 5,120 timestamps and an indicator for one outof three mechanical device states.
Revised: Every sample has 5,120 timestamps and an indicator for one outof three mechanical device states.

Original: (4) F D - B [63] has the same setting as the F D - A but the rollingbearings are performed in different working conditions ( e.g. , varying rotational speed).
Revised: (4) F D - B [60] has the same setting as the F D - A but the rollingbearings are performed in different working conditions ( e.g. , varying rotational speed).

Original: (5) H AR [64]has 10,299 9-dimension samples from 6 daily activities.
Revised: (5) H AR [61]has 10,299 9-dimension samples from 6 daily activities.

Original: (6) G ESTURE [65] includes 440 samplesthat are collected from 8 hand gestures recorded by accelerometer.
Revised: (6) G ESTURE [62] includes 640 samplesthat are collected from 8 hand gestures recorded by accelerometer.

Original: (7) E CG [26] contains 8,528single-sensor ECG recordings with sorted into four classes based on human physiology.
Revised: (7) E CG [27] contains 8,528single-sensor ECG recordings with sorted into four classes based on human physiology.

Original: (8) E MG[66] consists of 163 EMG samples with 3-class labels implying muscular diseases.
Revised: (8) E MG[63] consists of 163 EMG samples with 3-class labels implying muscular diseases.

Original: Dataset labels arenot used in pre-training.
Revised: Dataset labels arenot used in pre-training.

Original: Further dataset statistics are in Appendix B and Appendix Table 3.
Revised: Further dataset statistics are in Appendix B and Appendix Table 3.

Original: Baselines.
Revised: Baselines.

Original: We consider 8 baseline methods.
Revised: We consider 8 baseline methods.

Original: This includes 6 state-of-the-art methods: TS-SD [12],
Revised: This includes 6 state-of-the-art methods: TS-SD [11],

Original: TS2vec [47], CLOCS [41], Mixing-up [18], TS-TCC [48], and SimCLR [40].
Revised: TS2vec [46], CLOCS [40], Mixing-up [17], TS-TCC [47], and SimCLR [39].

Original: The TS2Vec, TS-TCC,and SimCLR are designed for representation learning on a single dataset (not across datasets), weapply them fit our settings to make the results comparable.
Revised: The TS2Vec, TS-TCC,and SimCLR are designed for representation learning on a single dataset (not across datasets), weapply them fit our settings to make the results comparable.

Original: To examine utility of pre-training, weconsider two approaches (non-deep learning KNN model and the Random Init.
Revised: To examine utility of pre-training, weconsider two approaches (non-deep learning KNN model and the Random Init.

Original: approach, whichrandomly initializes the fine-tuning model) that are applied directly to fine-tuning datasets withoutany pre-training.
Revised: approach, whichrandomly initializes the fine-tuning model) that are applied directly to fine-tuning datasets withoutany pre-training.

Original: The evaluation metrics are accuracy, precision (macro-averaged), recall, F1 score,
Revised: The evaluation metrics are accuracy, precision (macro-averaged), recall, F1 score,

Original: AUROC, and AUPRC.
Revised: AUROC, and AUPRC.

Original: Method details and hyper-parameter selection are in Appendix D.
Revised: Method details and hyper-parameter selection are in Appendix D.

Original: Implementation.
Revised: Implementation.

Original: We use two 3-layer 1-D ResNets [67] as backbones for G T and G F encoders.
Revised: We use two 3-layer 1-D ResNets [64] as backbones for G T and G F encoders.

Original: Datasets contain long series (samples in F D - A and F D - B have 5,120 observations), and preliminaryexperiments identified ResNet as a better option than a Transformer variant [68].
Revised: Datasets contain long series (samples in F D - A and F D - B have 5,120 observations), and preliminaryexperiments identified ResNet as a better option than a Transformer variant [65].

Original: We use 2 fullyconnected layers for R T and R F (no sharing of parameters).
Revised: We use 2 fullyconnected layers for R T and R F (no sharing of parameters).

Original: We set E = [-
-] and α = 0 .
Revised: We set ∆ = [-
-]1 and α = 0 .

Original: 5 in frequencyaugmentations and τ = 0 .
Revised: 5 in frequencyaugmentations and τ = 0 .

Original: 2 , δ = 1 , λ = 0 .
Revised: 2 , δ = 1 , λ = 0 .

Original: 5 in loss functions.
Revised: 5 in loss functions.

Original: Reported are mean and standarddeviation values across 5 independent runs (both pre-training and fine-tuning) on the same data split.
Revised: Reported are mean and standarddeviation values across 5 independent runs (both pre-training and fine-tuning) on the same data split.

Original: Results for KNN (K=2) do not change so standard deviation is zero.
Revised: Results for KNN (K=2) do not change so standard deviation is zero.

Original: Details are in Appendix E.
Revised: Details are in Appendix E.

Original: [Table]

Original: 5.1 Results: One-to-One Pre-Training Evaluation
Revised: .1 Results

Revised: : One-to-one pre-training evaluation

Original: Setup.
Revised: Setup.

Original: In one-to-one evaluation, we pre-train a model on one pre-training dataset and use it for fine-tuning on one target dataset only.[Table] [Table] [Table]
Revised: In one-to-one evaluation, we pre-train a model on one pre-training dataset and use it forfine-tuning on one target dataset only.

Original: [Table]

Original: [Table] [Table] [Table] 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 (S LEEP E EG → E PILEPSY ):
Revised: Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 (S LEEP E EG → E PILEPSY ):

Original: Pre-training is doneon S LEEP E EG and fine-tuning on E PILEPSY .
Revised: Pre-training is doneon S LEEP E EG and fine-tuning on E PILEPSY .

Original: While both datasets describe a single-channel EEG,the signals are from different channels/positions on scalp, monitor different physiology (sleepvs. epilepsy), and are collected from different patients.Scenario [Table] [Table]
Revised: While both datasets describe a single-channel EEG,the signals are from different channels/positions on scalp, monitor different physiology (sleepvs. epilepsy), and are collected from different patients.

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] 2 Scenario 2 (F D - A → F D - B ): Datasets describe mechanical devices that operate in different working conditions, including rotational speed,load torque, and radial force.
Revised: Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 (F D - A → F D - B ): Datasetsdescribe mechanical devices that operate in different working conditions, including rotational speed,load torque, and radial force.

Original: Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3  (H AR → G ESTURE ): Datasets record different activities (6types of human daily activities vs. 8 hand gestures).
Revised: Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 (H AR → G ESTURE ): Datasets record different activities (6types of human daily activities vs. 8 hand gestures).

Original: While both datasets contain acceleration signals,
Revised: While both datasets contain acceleration signals,

Original: H AR has 9 channels while G ESTURE has 1 channel.

Original: Scenario

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: H AR has 9 channels while G ESTURE has 1 channel.

Original: [Table] [Table] [Table] 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 (E CG → E MG ): While both arephysiological datasets, the E CG records the electrical signal from the heart whereas E MG measuresmuscle response in response to a nerve’s stimulation of the muscle.
Revised: Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 (E CG → E MG ): While both arephysiological datasets, the E CG records the electrical signal from the heart whereas E MG measuresmuscle response in response to a nerve’s stimulation of the muscle.

Original: We note that the discrepancybetween pre-training and fine-tuning datasets in the above four scenarios is substantial, coveringdiverse kinds of variation in time series datasets: varying semantic meaning, sampling frequency,time series length, number of classes, and system factors ( e.g. , number of devices or subjects).
Revised: We note that the discrepancybetween pre-training and fine-tuning datasets in the above four scenarios is substantial, coveringdiverse kinds of variation in time series datasets: varying semantic meaning, sampling frequency,time series length, number of classes, and system factors ( e.g. , number of devices or subjects).

Original: Thesetup is further challenged by relatively small number of samples available for fine-tuning (E PILEPSY :60; F D - B : 60; G ESTURE : 480; E MG : 122).
Revised: Thesetup is further challenged by relatively small number of samples available for fine-tuning (E PILEPSY :60; F D - B : 60; G ESTURE : 480; E MG : 122).

Original: Further details are in Appendix B-C.
Revised: Further details are in Appendix B-C.

Original: Results.
Revised: Results.

Original: The results for four scenarios are shown in Table 1 and Appendix Tables 4-6.
Revised: The results for four scenarios are shown in Table 1 and Appendix Tables 4-6.

Original: Overall,
Revised: Overall,

Original: TF-C model has won 16 out of 24 tests (6 metrics in 4 scenarios), and is the second best performerin only 8 other tests.
Revised: TF-C model has won 16 out of 24 tests (6 metrics in 4 scenarios), and is the second best performerin only 8 other tests.

Original: We report all metrics but discuss the F1 score in the following.
Revised: We report all metrics but discuss the F1 score in the following.

Original: On average,our TF-C model claims a large margin of 15.4% over all baselines.
Revised: On average,our TF-C model claims a large margin of 15.4% over all baselines.

Original: Although the strongest baselineis varying (such as TS-TCC in Scenario 2; Mixing-up in Scenario 3), our model outperforms thestrongest baselines by 1.5% across all scenarios.
Revised: Although the strongest baselineis varying (such as TS-TCC in Scenario 2; Mixing-up in Scenario 3), our model outperforms thestrongest baselines by 1.5% across all scenarios.

Original: Specifically, when using H AR -pre-trained model forfine-tuning on G ESTURE (Scenario 3) (Table 1), the TF-C approach achieves the highest performanceof 79.91% which yields a margin of 7.2% over the best baseline TS-TCC (74.57%).
Revised: Specifically, when using H AR -pre-trained model forfine-tuning on G ESTURE (Scenario 3) (Table 1), the TF-C approach achieves the highest performanceof 79.91% which yields a margin of 7.2% over the best baseline TS-TCC (74.57%).

Original: One potentialreason that Scenario 3 is the most complex dataset in terms of classes (H AR has 6 classes while
Revised: One potentialreason that Scenario 3 is the most complex dataset in terms of classes (H AR has 6 classes while

Original: G ESTURE has 8 classes) that is difficult to model.
Revised: G ESTURE has 8 classes) that is difficult to model.

Original: Thee assumption is proved from side by the relativelower performances in Scenario 3 ( < 80% ) than other scenarios ( > 90% ).
Revised: Thee assumption is proved from side by the relativelower performances in Scenario 3 ( < 80% ) than other scenarios ( > 90% ).

Original: 5.2 Results: One-to-Many Pre-Training Evaluation
Revised: 5.2 Results: One-to-many pre-training evaluation

Original: Setup.
Revised: Setup.

Original: In one-to-many evaluation, pre-training is done using one dataset followed by fine-tuningon multiple target datasets independently without starting pre-training from scratch.
Revised: In one-to-many evaluation, pre-training is done using one dataset followed by fine-tuningon multiple target datasets independently without starting pre-training from scratch.

Original: Out of eightdatasets, S LEEP E EG has most complex temporal dynamics [69] and is the largest (371,055 samples).
Revised: Out of eightdatasets, S LEEP E EG has most complex temporal dynamics [66] and is the largest (371,055 samples).

Original: For that reason, we pre-train a model on S LEEP E EG and separately fine-tune a well-pre-trained modelon E PILEPSY , F D - B , G ESTURE , and E MG .
Revised: For that reason, we pre-train a model on S LEEP E EG and separately fine-tune a well-pre-trained modelon E PILEPSY , F D - B , G ESTURE , and E MG .

Original: Results.
Revised: Results.

Original: Results are shown in Table 2.
Revised: Results are shown in Table 2.

Original: Intuitively, there are less common between EEG signals andvibration, acceleration and EMG, we expect that the transfer learning will be less effective comparedto the one-to-one evaluation.
Revised: Intuitively, there are less common between EEG signals andvibration, acceleration and EMG, we expect that the transfer learning will be less effective comparedto the one-to-one evaluation.

Original: The pre-training and fine-tuning datasets are largely different in thebottom three blocks (S LEEP E EG → {F D - B , G ESTURE , E MG }).
Revised: The pre-training and fine-tuning datasets are largely different in thebottom three blocks (S LEEP E EG → {F D - B , G ESTURE , E MG }).

Original: The large gap reasonably leadsto a deterioration in baseline performances, however, our model has obviously higher toleranceto knowledge transfer across datasets with large difference.
Revised: The large gap reasonably leadsto a deterioration in baseline performances, however, our model has obviously higher toleranceto knowledge transfer across datasets with large difference.

Original: Notably, We find that the proposedmodel with TF-C earned best performance in 14 out of 18 settings in the three challenging settings:indicating our TF-C assumption is universal in time series.
Revised: Notably, We find that the proposedmodel with TF-C earned best performance in 14 out of 16 settings in the three challenging settings:indicating our TF-C assumption is universal in time series.

Original: For example, our approach outperformsthe strongest baseline by 8.4% (in precision) when fine-tuning on G ESTURE .
Revised: For example, our approach outperformsthe strongest baseline by 8.4% (in precision) when fine-tuning on G ESTURE .

Original: Our model has largepotential to serve as a universal model when there is no large pre-training dataset that is similar to thesmall fine-tuning dataset.
Revised: Our model has largepotential to serve as a universal model when there is no large pre-training dataset that is similar to thesmall fine-tuning dataset.

Original: Further, the TF-C approach consistently outperforms KNN and Random
Revised: Further, the TF-C approach consistently outperforms KNN and Random

Original: Init.
Revised: Init.

Original: (which are not pre-trained) by a large margin of 42.8% and 25.1% (both in F1 score) on average.[Table] [Figure] [Table]
Revised: (which are not pre-trained) by a large margin of 42.8% and 25.1% (both in F1 score) on average.

Original: Ablation study.
Revised: Ablation study.

Original: We wanted to evaluate how relevant model component are for effective pre-training.
Revised: We wanted to evaluate how relevant model component are for effective pre-training.

Original: As shown in Appendix Table 7 (S LEEP E EG → G ESTURE ), removing L C , L T , and L F result inperformance degradation (precision) of 6.1%, 7.2%, and 6.7%, respectively.
Revised: As shown in Appendix Table 7 (S LEEP E EG → G ESTURE ), removing L C , L T , and L F result inperformance degradation (precision) of 6.1%, 7.2%, and 6.7%, respectively.

Original: To validate that theperformance increment is not solely brought by a third loss term no matter what consistency itmeasures, we replaced consistency loss L C with a loss term measuring the consistency within timespace (named L TT-C ) or within frequency space (named L FF-C ).
Revised: To validate that theperformance increment is not solely brought by a third loss term no matter what consistency itmeasures, we replaced consistency loss L C with a loss term measuring the consistency within timespace (named L TT-C ) or within frequency space (named L FF-C ).

Original: Results show our consistency lossoutperforms L TT-C and L FF-C by 5.3% and 7.2% (accuracy), respectively.
Revised: Results show our consistency lossoutperforms L TT-C and L FF-C by 5.3% and 7.2% (accuracy), respectively.

Original: We also find that fullfine-tuning (optimizing the whole model) performs slightly better than partial fine-tuning (updatingonly final layers in R T and R F ), by 5.4% in F1 score.
Revised: We also find that full [Table] fine-tuning (optimizing the whole model) performs slightly better than partial fine-tuning (updatingonly final layers in R T and R F ), by 5.4% in F1 score.

Original:  Conclusion
Revised:  Conclusion

Original: We develop a pre-training approach that introduces time-frequency consistency (TF-C) as a mechanism to support knowledge transfer between time-series datasets.
Revised: We introduce a pre-training model using the TF-C inductive bias in time series to support knowledgetransfer across time-series datasets.

Original: The approach uses self-supervisedcontrastive estimation and injects TF-C into a pre-training model, bringing the encoded time-basedand frequency-based representations and  their local neighborhoods close together in the latent space.
Revised: The approach uses self-supervised contrastive estimation to inject TF-C in a pre-training model, bringing the encoded time-based and frequency-based representationsand their local neighborhoods closer together in the latent space.

Original: Experiments on eight datasets and four pre-training scenarios show the TF-C approach can outperformeight state-of-the-art baselines.
Revised: Experiments on eight datasets andfour pre-training scenarios show the TF-C approach can outperform eight state-of-the-art baselines.

Original: Our findings have implications for building broadly generalizablepre-training modelsfor time series.
Revised: Our findings have implications for building broadly generalizable pre-training models

Original: Limitations and future directions.
Revised: Limitations and future directions.

Original: We posit that TF-Cproperty can serve as a generalizable propertyfor pre-training on diverse time series datasets.
Revised: We posit that TF-C can serve as a generalizable property tosupport pre-training on diverse time series datasets.

Original: We are aware that there might exist additionalgeneralizable properties, such as temporal autoregressive processes, that could also prove useful forpre-training on time series.
Revised: There might exist additional generalizableproperties, such as temporal autoregressive processes, that could also proved useful for pre-trainingon time series.

Original: Furthermore, our method expects the input time series to be regularlysampled. However, our model can handle irregularly sampled time series using an encoder (such as 
Revised: Our method expects the input time series to be regularly sampled, however, our modelcan handle irregularly sampled time series with an encoder (such as Raindrop [67] and SeFT

Original: Raindrop [70] and SeFT [71]) that can embed irregular time series.
Revised: [68])that can learn time-based representations from irregular time series.

Original: For frequency encoder inputsx F i , possible alternatives include resampling or interpolation to obtain regularly sampled signals andusing regular or non-uniform FFT operations.
Revised: For frequency encoder inputs include resampling or interpolation to obtain regularly sampled signals andusing regular or non-uniform FFT operations.

Original: Broader Impacts
Revised: Broader Impacts

Original: Our proposed novel framework for time series unsupervised pre-training improves classificationperformance on target datasets in different application scenarios.
Revised: Our proposed novel framework for time series unsupervised pre-training improves classificationperformance on target datasets in different application scenarios.

Original: The recognition of time-frequencyconsistency as a universal property specific to time series data is a weak assumption that enableseffective, task- and domain-agnostic transfer learning.
Revised: The recognition of time-frequencyconsistency as a universal property specific to time series data is a weak assumption that enableseffective, task- and domain-agnostic transfer learning.

Original: We believe our work will inspire the researchcommunity to uncover other universal properties for transfer learning.
Revised: We believe our work will inspire the researchcommunity to uncover other universal properties for transfer learning.

Original: We also hope our work willalso attract more researchers to the more general problem of time series representation learning whichis still underappreciated relative to problems from CV and NLP fields.
Revised: We also hope our work willalso attract more researchers to the more general problem of time series representation learning whichis still underappreciated relative to problems from CV and NLP fields.

Original: On the society level, our work, along the line of transfer learning, can facilitate more efficient use oftime series data in various settings.
Revised: On the society level, our work, along the line of transfer learning, can facilitate more efficient use oftime series data in various settings.

Original: For example, in medical settings, some diseases of clinical interestmay have very small labelled dataset.
Revised: For example, in medical settings, some diseases of clinical interestmay have very small labelled dataset.

Original: In this case, unlabelled data from patients of different diseasesbut with similar underlying physiological conditions can be used to pre-train the model.
Revised: In this case, unlabelled data from patients of different diseasesbut with similar underlying physiological conditions can be used to pre-train the model.

Original: However,practitioners need to be aware of the limitations of the model, including that it may make biasedpredictions.
Revised: However,practitioners need to be aware of the limitations of the model, including that it may make biasedpredictions.

Original: Specifically, bias may exist in the source dataset used for pre-training due to an imbalanceof samples from subjects of different demographic attributes.
Revised: Specifically, bias may exist in the source dataset used for pre-training due to an imbalanceof samples from subjects of different demographic attributes.

Original: Also, the standardized medical protocolsfor collecting these datasets might be unsuitable for subjects with certain physiological attributes,creating unforseen bias that may be transferred to the finetuning phase.
Revised: Also, the standardized medical protocolsfor collecting these datasets might be unsuitable for subjects with certain physiological attributes,creating unforseen bias that may be transferred to the finetuning phase.

Original: All datasets in this paper are publicly available and are not associated with any privacy or securityconcern.
Revised: All datasets in this paper are publicly available and are not associated with any privacy or securityconcern.

Original: Furthermore, we have followed guidelines on responsible use specified by primary authorsof the datasets used in the current work.
Revised: Furthermore, we have followed guidelines on responsible use specified by primary authorsof the datasets used in the current work.

Original: Checklist
Revised: Checklist

Original: For all authors...(
Revised: For all authors...

Original: a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] In abstract and introduction, we claim that TF-C is
Revised: [Yes] In abstract and introduction, we claim that TF-C is

Original: a generalizable property of time series that can support pre-training, which is well-
Revised: a generalizable property of time series that can support pre-training, which is well-

Original: justified in Sec. 3 and experimentally demonstrated in Sec.
Revised: justified in Sec. 3 and experimentally demonstrated in Sec.

Original: 5 (our model consistently
Revised: 5 (our model consistently

Original: performs comparatively to or above baseline methods
Revised: performs comparatively to or above baseline methods).

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 6.
Revised: [Yes] See Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Broader Impact on Page 10.
Revised: [Yes] SeeBroader Impact on Page 10.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Yes, we include
Revised: [Yes] Yes, we include

Original: an anonymous link (see Abstract) that provides the source codes with all implementa-
Revised: an anonymous link (see Abstract) that provides the source codes with all implementa-

Original: tion details, implementation of baselines, and eight datasets.
Revised: tion details, implementation of baselines, and eight datasets.

Original: The link will be updated
Revised: The link will be updated

Original: to an non-anonymous link after acceptance.
Revised: to an non-anonymous link after acceptance.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyper-parameters,
Revised: g., data splits, hyper-parameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See implementation details in Sec. 5.
Revised: [Yes] See implementation details in Sec. 5.

Original: See Appendix E for baselinearchitectures and hyper-parameter settings.
Revised: See Appendix E for baseline architectures and hyper-parameter settings.

Original: More details can be found in the included
Revised: More details can be found in the included

Original: URL.
Revised: URL.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] We run experiments for 5 times and report the averagevalue with standard deviation.
Revised: [Yes] We run experiments for 5 times and report the averagevalue with standard deviation.

Original: See Table 1, Table
Revised: See Table 1, Table

Original: and Appendix Tables 4-6.(d)
Revised: and Appendix Tables 4-6.

Original:  Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix E.
Revised: [Yes] See Appendix E.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We used eightexisting datasets and 6 state-of-the-art baselines in contrastive learning and pre-trainingfor time series.
Revised: [Yes] We used eightexisting datasets and 6 state-of-the-art baselines in contrastive learning and pre-trainingfor time series.

Original: We cited the creators for every exist asset we used.
Revised: We cited the creators for every exist asset we used.

Original: See Sec. 5.
Revised: See Sec. 5.

Original: [Table]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] All dataset licenses are mentioned inthe Appendix E.(c)
Revised: [Yes] All dataset licenses are mentioned inthe Appendix E.

Original:  Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]See the anonymous link in Abstract.
Revised: [Yes] See the anonymous link in Abstract.

Original: [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [No] All data we use is freely available for download, without anyrequirement to re-contact the data curator.[Table]
Revised: [No] All data we use is freely available for download, without anyrequirement to re-contact the data curator.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [No] Our datasets are public, well-established, and
Revised: [No] Our datasets are public, well-established, and

Original: [Table]
Revised: [Table]

Original: do not contain PII or offensive content
Revised: do not contain PII or offensive content

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A](b)
Revised: [N/A](b)

Original: Did you describe any potential participant risks, with links to Institutional Review
Revised: Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A] [Table]
Revised: [N/A](c)

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
Revised: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]