Original: Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency
Revised: Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency

Original: Abstract
Revised: Abstract

Original: Pre-training on time series poses a unique challenge due to the potential mismatchbetween pre-training and target domains, such as shifts in temporal dynamics,fast-evolving trends, and long-range and short cyclic effects, which can lead topoor downstream performance.
Revised: Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short cyclic effects, which can lead to poor downstream performance.

Original: While domain adaptation methods can mitigatethese shifts, most methods need examples directly from the target domain, makingthem suboptimal for pre-training.
Revised: While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training.

Original: To address this challenge, methods need toaccommodate target domains with different temporal dynamics and be capableof doing so without seeing any target examples during pre-training.
Revised: To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training.

Original: Relativeto other modalities, in time series, we expect that time-based and frequencybased representations of the same example are located close together in the timefrequency space.
Revised: Relative to other modalities, in time series, we expect that time-based and frequencybased representations of the same example are located close together in the timefrequency space.

Original: To this end, we posit that time-frequency consistency (TF-C)— embedding a time-based neighborhood of a particular example close to itsfrequency-based neighborhood and back — is desirable for pre-training.
Revised: To this end, we posit that time-frequency consistency (TF-C) — embedding a time-based neighborhood of a particular example close to its frequency-based neighborhood and back — is desirable for pre-training.

Original: Motivatedby TF-C, we define a decomposable pre-training model, where the self-supervisedsignal is provided by the distance between time and frequency components, eachindividually trained by contrastive estimation.
Revised: Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation.

Original: We evaluate the new method oneight datasets, including electrodiagnostic testing, human activity recognition,mechanical fault detection, and physical status monitoring.
Revised: We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring.

Original: Experiments againsteight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1score) on average in one-to-one settings ( e.g. , fine-tuning an EEG-pretrained modelon EMG data) and by up to 8.4% (F1 score) in challenging one-to-many settings( e.g. , fine-tuning an EEG-pretrained model for either hand-gesture recognitionor mechanical fault prediction), reflecting the breadth of scenarios that arise inreal-world applications.
Revised: Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F score) on average in one-to-one settings ( e.g. , fine-tuning an EEG-pretrained model on EMG data) and by up to 8.4% (F1 score) in challenging one-to-many settings ( e.g. , fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction ), reflecting the breadth of scenarios that arise in real-world applications.

Original: The source code and datasets are available at https: anonymous
Revised: The source code and datasets are available at https: //github.com/

Original: .4open.science/r/TFC-pretraining-6B07 .
Revised: mims-harvard/TFC-pretraining .

Original:  Introduction
Revised:  Introduction

Original: Time series plays important roles in many areas, including clinical diagnosis, traffic analysis, andclimate science [1, 2, 3, 4, 5, 6].
Revised: Time series plays important roles in many areas, including clinical diagnosis, traffic analysis, and climate science [1, 2, 3, 4, 5, 6].

Original: While representation learning has considerably advanced analysis oftime series [7, 8, 9] and sequences more broadly [10], learning broadly generalizable representationsfor time series remains a fundamentally challenging problem [8, 11].
Revised: While representation learning has considerably advanced analysis of time series [7, 8, 9] and sequences more broadly [10], learning broadly generalizable representations for time series remains a fundamentally challenging problem [8, 11].

Original: There are numerous immediatebenefits from being able to generate such representations, of which pre-training capability is particularly desirable and of great practical importance [12, 13].
Revised: There are numerous immediate [Figure] benefits from being able to generate such representations, of which pre-training capability is particularly desirable and of great practical importance [12, 13].

Original: Central to pre-training is a question of howto process time series in a diverse dataset so as to greatly improve generalization on new time seriescoming from different datasets [14, 15, 10].
Revised: Central to pre-training is a question of how to process time series in a diverse dataset so as to greatly improve generalization on new time series coming from different datasets [14, 15, 10].

Original: By training a neural network model on a diversifieddataset and transferring it to a new target dataset for fine-tuning, i.e. , without explicit retraining onthat target data, we expect the resulting performance be at least as good as that of state-of-the-artmodels tailored to the target dataset.
Revised: By training a neural network model on a diversified dataset and transferring it to a new target dataset for fine-tuning, i.e. , without explicit retraining on that target data, we expect the resulting performance be at least as good as that of state-of-the-art models tailored to the target dataset.

Original: [Figure] However, unfortunately, the expected performance gains are often not realized for a variety of reasons( e.g. , distribution shifts, properties of the target dataset unknown during pre-training) [16, 17] thatget compounded by the complexity of time series: large variations of temporal dynamics acrossdatasets, varying semantic meaning, irregular sampling, system factors ( e.g. , different devices orsubjects), etc.
Revised: However, unfortunately, the expected performance gains are often not realized for a variety of reasons ( e.g. , distribution shifts, properties of the target dataset unknown during pre-training) [16, 17] that get compounded by the complexity of time series: large variations of temporal dynamics across datasets, varying semantic meaning, irregular sampling, system factors ( e.g. , different devices or subjects), etc.

Original: This complexity of time series limits the utility of knowledge transfer forpre-training [19, 20].
Revised: This complexity of time series limits the utility of knowledge transfer for pre-training [19, 20].

Original: For example, pre-training a model on a diverse time series dataset with mostlylow-frequency components (smooth trends) may not lead to positive transfer on downstream taskswith high-frequency components (transient events) [17].
Revised: For example, pre-training a model on a diverse time series dataset with mostly low-frequency components (smooth trends) may not lead to positive transfer on downstream tasks with high-frequency components (transient events) [17].

Original: Examining these challenges can provideclues to what kind of inductive biases could facilitate generalizable representations of time series –this paper offers a strategy for that through a novel time-frequency consistency principle.
Revised: Examining these challenges can provide clues to what kind of inductive biases could facilitate generalizable representations of time series – this paper offers a strategy for that through a novel time-frequency consistency principle.

Original: In addition, target dataset are not available in pre-training phase (different from domain adaption [21];
Revised: In addition, target dataset are not available in pre-training phase (different from domain adaption [21];

Original: Appendix A), requiring that pre-training model captures some latent shared property in time seriesthat can apply to unseen datasets.
Revised: Appendix A), requiring that pre-training model captures some latent shared property in time series that can apply to unseen datasets.

Original: The principle of a shared property is universal in different timeseries datasets and thus allows knowledge transfer from a pre-training dataset to a fine-tuning dataset.
Revised: The principle of a shared property is universal in different time series datasets and thus allows knowledge transfer from a pre-training dataset to a fine-tuning dataset.

Original: To make it more clear, a key principle for pre-training in computer vision (CV) is that the priorneural network layers learn object edges and shapes regardless of image style or tasks [22].
Revised: To make it more clear, a key principle for pre-training in computer vision (CV) is that the prior neural network layers learn object edges and shapes regardless of image style or tasks [22].

Original: Theshared foundation for pre-training in natural language processing (NLP) is the linguistic assumptionof semantics and grammars that is retrained across different languages [23].
Revised: The shared foundation for pre-training in natural language processing (NLP) is the linguistic assumption of semantics and grammars that is retrained across different languages [23].

Original: Nevertheless, it ishard to put forward such a principle for time series due to aforementioned temporal complexity.
Revised: Nevertheless, it is hard to put forward such a principle for time series due to aforementioned temporal complexity.

Original: Moreover, supervised pre-training requires large annotated datasets, creating an obstacle to deeplearning deployment [24, 25].
Revised: Moreover, supervised pre-training requires large annotated datasets, creating an obstacle to deep learning deployment [24, 25].

Original: For example, in medical domains, experts often disagree on groundtruth labeling ( e.g. , ECG signals falling in normal or abnormal rhythms [26, 27]), which can introduceunintended biases [28, 29].
Revised: For example, in medical domains, experts often disagree on groundtruth labeling ( e.g. , ECG signals falling in normal or abnormal rhythms [26, 27]), which can introduce unintended biases [28, 29].

Original: Self-supervised contrastive learning can overstep the lack of annotationsby providing self-supervision signal through contrastive instance discrimination [30].
Revised: Self-supervised contrastive learning can overstep the lack of annotations by providing self-supervision signal through contrastive instance discrimination [30].

Original: Present work.
Revised: Present work.

Original: We introduce a strategy for self-supervised pre-training in time series by modeling
Revised: We introduce a strategy for self-supervised pre-training in time series by modeling

Original: Time-Frequency Consistency (TF-C).
Revised: Time-Frequency Consistency (TF-C).

Original: TF-C means the time-based representation and the frequencybased representation, which are learned from the same time series sample, are closer to each other ina joint time-frequency space, and farther apart if the representations are associated with different timeseries samples.
Revised: TF-C means the time-based representation and the frequencybased representation, which are learned from the same time series sample, are closer to each other in a joint time-frequency space, and farther apart if the representations are associated with different time series samples.

Original: Specifically, we adopt contrastive learning in time space to generate a time-basedrepresentation.
Revised: Specifically, we adopt contrastive learning in time space to generate a time-based representation.

Original: In parallel, we propose a set of novel augmentations based on the characteristic offrequency spectrum and produce a frequency-based embedding through contrastive instance discrimination.
Revised: In parallel, we propose a set of novel augmentations based on the characteristic of frequency spectrum and produce a frequency-based embedding through contrastive instance discrimination.

Original: To the best of our knowledge, this is the first work that develops contrastive augmentationin frequency domain to leverage rich spectral information, also the first work that exploits the property of time-frequency consistency in time series.
Revised: To the best of our knowledge, this is the first work that develops contrastive augmentation in frequency domain to leverage rich spectral information, also the first work that exploits the property of time-frequency consistency in time series.

Original: The pre-training objective is to minimize thedistance between the time-based and frequency-based embeddings through a dedicated consistencyloss (Figure 1 (a)).
Revised: The pre-training objective is to minimize the distance between the time-based and frequency-based embeddings through a dedicated consistency loss (Figure 1 (a)).

Original: The designed loss provides self-supervising signals to optimize the pre-trainingmodel and captures the latent relationships between time domain and frequency domain.
Revised: The designed loss provides self-supervising signals to optimize the pre-training model and captures the latent relationships between time domain and frequency domain.

Original: The learnedrelationship encoded in model parameters are transferred to initialize the fine-tuning model andimprove performance in datasets of interest (Figure 1 (b)).
Revised: The learned relationship encoded in model parameters are transferred to initialize the fine-tuning model and improve performance in datasets of interest (Figure 1 (b)).

Original: We evaluate the proposed pre-training model on eight time series datasets under two evaluationsettings ( i.e. , one-to-one and one-to-many).
Revised: We evaluate the proposed pre-training model on eight time series datasets under two evaluation settings ( i.e. , one-to-one and one-to-many).

Original: The eight datasets cover a large set of variations: differentnumber of channels (from univariate to 9-channel multivariate), varying time series length (from128 to 5,120), different sampling rate (from 16 Hz to 4,000 Hz), different scenarios (neurological healthcare, human activity recognition, mechanical fault detection, physical status monitoring, etc.)and diverse types of signals (EEG, EMG, ECG, acceleration, and vibration).
Revised: The eight datasets cover a large set of variations: different number of channels (from univariate to 9-channel multivariate), varying time series length (from 128 to 5,120), different sampling rate (from 16 Hz to 4,000 Hz), different scenarios (neurological healthcare, human activity recognition, mechanical fault detection, physical status monitoring, etc.) and diverse types of signals (EEG, EMG, ECG, acceleration, and vibration).

Original: We compare TF-Capproach to eight state-of-the-art baselines.
Revised: We compare TF-C approach to eight state-of-the-art baselines.

Original: Results show that TF-C achieves positive transfer,outperforming all baselines by a large margin of 15.4% (F1 score) on average.
Revised: Results show that TF-C achieves positive transfer, outperforming all baselines by a large margin of 15.4% (F1 score) on average.

Original: Further, the approachoutperforms the strongest baselines with an improvement of up to 7.2% in F1 score.
Revised: Further, the approach outperforms the strongest baselines with an improvement of up to 7.2% in F1 score.

Original: Finally, the
Revised: Finally, the

Original: TF-C approach improves prior work by 8.4% in precision (when pre-training the model on sleep
Revised: TF-C approach improves prior work by 8.4% in precision (when pre-training the model on sleep

Original: EEG signals and fine-tuning it on hand-gesture recognition) in a challenging one-to-many setup thatapplies the same pre-trained model to multiple independent fine-tuning datasets.
Revised: EEG signals and fine-tuning it on hand-gesture recognition) in a challenging one-to-many setup that applies the same pre-trained model to multiple independent fine-tuning datasets.

Original:  Related Work
Revised:  Related Work

Original: Pre-training for time series.
Revised: Pre-training for time series.

Original: Although there are studies on self-supervised representation learning fortime series [7, 8, 31, 32] and self-supervised pre-training for images [33, 34, 35, 24], the intersectionof these two areas, i.e. , self-supervised pre-training for time series, remains underexplored.
Revised: Although there are studies on self-supervised representation learning for time series [7, 8, 31, 32] and self-supervised pre-training for images [33, 34, 35, 24], the intersection of these two areas, i.e. , self-supervised pre-training for time series, remains underexplored.

Original: In timeseries, it’s not obvious that what assumptions are reasonable that can bridge pre-training dataset andtarget dataset.
Revised: In time series, it’s not obvious that what assumptions are reasonable that can bridge pre-training dataset and target dataset.

Original: Hence, pre-training models in CV [36, 37, 14] and NLP [10, 15, 38] are not directlyapplicable due to data modality mismatch, and the results leave room for improvement [31, 39, 40].
Revised: Hence, pre-training models in CV [36, 37, 14] and NLP [10, 15, 38] are not directly applicable due to data modality mismatch, and the results leave room for improvement [31, 39, 40].

Original: Shi et al.
Revised: Shi et al.

Original: [12] developed the only model to date that is explicitly designed for self-supervised timeseries pre-training.
Revised: [12] developed the only model to date that is explicitly designed for self-supervised time series pre-training.

Original: The model captures the local and global temporal pattern but it’s not convincingwhy the designed pretext task can capture generalizable representations.
Revised: The model captures the local and global temporal pattern but it’s not convincing why the designed pretext task can capture generalizable representations.

Original: Although several studiesapplied transfer learning in the context of time series [7, 8, 18, 41], there is no foundation yet ofwhich conceptual properties are most suitable for pre-training on time series and why.
Revised: Although several studies applied transfer learning in the context of time series [7, 8, 18, 41], there is no foundation yet of which conceptual properties are most suitable for pre-training on time series and why.

Original: Addressingthis gap, we show that TF-C, designed to be invariant to different time-series datasets, can producegeneralizable pre-training models.
Revised: Addressing this gap, we show that TF-C, designed to be invariant to different time-series datasets, can produce generalizable pre-training models.

Original: Unlike domain adaptation [21, 42] that requires access to target datasets during training, pre-trainingmodels have access only to pre-training datasets.
Revised: Unlike domain adaptation [21, 42] that requires access to target datasets during training, pre-training models have access only to pre-training datasets.

Original: As a result, one needs to identify a generalizabletime-series property to benefit from pre-training.
Revised: As a result, one needs to identify a generalizable time-series property to benefit from pre-training.

Original: Further, self-supervised domain adaptation does notneed labels in the target dataset but still requires labels for model training [43, 44].
Revised: Further, self-supervised domain adaptation does not need labels in the target dataset but still requires labels for model training [43, 44].

Original: In contrast, TF-Cdoes not need any labels during pre-training.
Revised: In contrast, TF-C does not need any labels during pre-training.

Original: Contrastive learning with time series.
Revised: Contrastive learning with time series.

Original: Contrastive learning, a popular type of self-supervisedlearning, aims to learn an encoder that maps inputs into an embedding space such that positive samplepairs (original augmentation and another alternative augmentation/view of the same input sample)are pulled closer and negative sample pairs (original augmentation and an alternative input sampleaugmentation) are pushed apart [30, 45].
Revised: Contrastive learning, a popular type of self-supervised learning, aims to learn an encoder that maps inputs into an embedding space such that positive sample pairs (original augmentation and another alternative augmentation/view of the same input sample) are pulled closer and negative sample pairs (original augmentation and an alternative input sample augmentation) are pushed apart [30, 45].

Original: Contrastive learning in time series is less investigated incomparison, partly due to the challenge of identifying augmentations that capture key invarianceproperties in time series data.
Revised: Contrastive learning in time series is less investigated in comparison, partly due to the challenge of identifying augmentations that capture key invariance properties in time series data.

Original: For example, CLOCS defines adjacent segments of a time series aspositive pairs [41], and TNC assumes overlapping neighborhoods of time series should have similarrepresentations [46].
Revised: For example, CLOCS defines adjacent segments of a time series as positive pairs [41], and TNC assumes overlapping neighborhoods of time series should have similar representations [46].

Original: These methods leverage temporal invariance to define positive pairs whichare used to calculate contrastive loss, but other invariances, such as transformation invariance ( e.g. ,
Revised: These methods leverage temporal invariance to define positive pairs which are used to calculate contrastive loss, but other invariances, such as transformation invariance ( e.g. ,

Original: SimCLR [40]), contextual invariance ( e.g. , TS2vec [47] and TS-TCC [48]) and augmentations arepossible.
Revised: SimCLR [40]), contextual invariance ( e.g. , TS2vec [47] and TS-TCC [48]) and augmentations are possible.

Original: In this work, we propose an augmentation bank that exploits multiple invariances to generatediverse augmentations (Sec. 4.1), which adds richness into the pre-training model [48].
Revised: In this work, we propose an augmentation bank that exploits multiple invariances to generate diverse augmentations (Sec. 4.1), which adds richness into the pre-training model [48].

Original: Importantly,we propose frequency-based augmentations by perturbing the frequency spectrum of time series ( e.g. ,adding or removing the frequency components and manipulating the their amplitude; more details in Sec. 4.2) to learn better representations by exposing the model to local range of frequency variations.
Revised: Importantly, we propose frequency-based augmentations by perturbing the frequency spectrum of time series ( e.g. , adding or removing the frequency components and manipulating the their amplitude; more details in Sec. 4.2) to learn better representations by exposing the model to local range of frequency variations.

Original: A previous work CoST processes sequential signals through frequency domain but the augmentationsare still implemented in time space [49].
Revised: A previous work CoST processes sequential signals through frequency domain but the augmentations are still implemented in time space [49].

Original: Similarly, although BTSF [50] involves frequency domain,its data transformation is solely implemented in time domain using instance-level dropout.
Revised: Similarly, although BTSF [50] involves frequency domain, its data transformation is solely implemented in time domain using instance-level dropout.

Original:  To thebest of our knowledge, this is the first work that directly perturbs the frequency spectrum to leveragefrequency-invariance for contrastive learning.
Revised: More differences with CoST and BTSF in Appendix B. To the best of our knowledge, this is the first work that directly perturbs the frequency spectrum to leverage frequency-invariance for contrastive learning.

Original: Further, we develop a pre-training model that subjectsto TF-C upon two individual contrastive encoders.
Revised: Further, we develop a pre-training model that subjects to TF-C upon two individual contrastive encoders.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: We are given a pre-training dataset D pret =
Revised: We are given a pre-training dataset D pret =

Original: { x pret i | i = 1 , . .
Revised: { x pret i | i = 1 , . .

Original: . , N } of unlabeled time series sampleswhere sample x pret i has K pret channels and L pret timestamps.
Revised: . , N } of unlabeled time series samples where sample x pret i has K pret channels and L pret timestamps.

Original: Let D tune =
Revised: Let D tune =

Original: { ( x tune i , y i ) | i = 1 , . .
Revised: { ( x tune i , y i ) | i = 1 , . .

Original: , M }be a fine-tuning ( i.e. , target; target and fine-tuning are used interchangeably) dataset of labeled timeseries samples, each having K tune channels and L tune timestamps.
Revised: , M } be a fine-tuning ( i.e. , target; target and fine-tuning are used interchangeably) dataset of labeled time series samples, each having K tune channels and L tune timestamps.

Original: Further, every sample x tune i is associated with label y i ∈ { 1 , . . .
Revised: Further, every sample x tune i is associated with label y i ∈ { 1 , . . .

Original: , C } , where C is the number of classes.
Revised: , C } , where C is the number of classes.

Original: Without loss of generality,in the following, we describe the approach for univariate (single-channel) time series, noting that theapproach can accommodate multivariate time series, different numbers of channels across samples,and varying lengths (shown in experiments in Sec. 5.2).
Revised: Without loss of generality, in the following, we describe the approach for univariate (single-channel) time series, noting that the approach can accommodate multivariate time series, different numbers of channels across samples, and varying lengths (shown in experiments in Sec. 5.2).

Original: We use superscript symbol (cid:101) to denotecontrastive augmentations.
Revised: We use superscript symbol (cid:101) to denote contrastive augmentations.

Original: We note that x T i ≡ x i denotes an input time series sample, and x F i denotesthe frequency spectrum of x i .
Revised: We note that x T i ≡ x i denotes an input time series sample, and x F i denotes the frequency spectrum of x i .

Original: Problem (Self-Supervised Contrastive Pre-Training For Time Series).
Revised: Problem (Self-Supervised Contrastive Pre-Training For Time Series).

Original: Given are an unlabeledpre-training dataset D pret with N samples and a target dataset D tune with M samples ( M ≪ N ).
Revised: Given are an unlabeled pre-training dataset D pret with N samples and a target dataset D tune with M samples ( M ≪ N ).

Original: Wewant to use D pret to pre-train a neural model F so that by fine-tuning model parameters on D tune , thefine-tuned model can generate a generalizable representation z tune i =
Revised: We want to use D pret to pre-train a neural model F so that by fine-tuning model parameters on D tune , the fine-tuned model can generate a generalizable representation z tune i =

Original: F ( x tune i ) for every x tune i .
Revised: F ( x tune i ) for every x tune i .

Original: We follow established pre-training setup, e.g. , [41]; for pre-training, only unlabeled dataset D pretis available while, for fine-tuning, a small labeled dataset D tune can be used.
Revised: We follow established pre-training setup, e.g. , [41]; for pre-training, only unlabeled dataset D pret is available while, for fine-tuning, a small labeled dataset D tune can be used.

Original: In short, a model
Revised: In short, a model

Original: F is pre-trained on unlabeled time series dataset D pret and its optimized model parameters Θ arefine-tuned to go from F ( · , Θ) to F ( · , Φ) using dataset D tune .
Revised: F is pre-trained on unlabeled time series dataset D pret and its optimized model parameters Θ are fine-tuned to go from F ( · , Θ) to F ( · , Φ) using dataset D tune .

Original: The Φ denotes fine-tuned modelparameters.
Revised: The Φ denotes fine-tuned model parameters.

Original: Note that this problem ( i.e. , D pret is independent of target dataset) is different thandomain adaptation because we don’t access the fine-tuning dataset D tune during pre-training, meaningthat a successful pre-trained model should be applicable to multiple independent fine-tuning datasetswithout necessitating re-training.
Revised: Note that this problem ( i.e. , D pret is independent of target dataset) is different than domain adaptation because we don’t access the fine-tuning dataset D tune during pre-training, meaning that a successful pre-trained model should be applicable to multiple independent fine-tuning datasets without necessitating re-training.

Original: Rationale for Time-Frequency Consistency (TF-C).
Revised: Rationale for Time-Frequency Consistency (TF-C).

Original: For effective per-training on time series,our central idea is to identify a broadly generalizable property that is preserved across differenttime series datasets.
Revised: For effective per-training on time series, our central idea is to identify a broadly generalizable property that is preserved across different time series datasets.

Original: In time series in particular, the time domain shows how readouts change withtime whereas frequency domain tells us how much of the signal lies within each frequency bandover a range of frequencies ( e.g. , frequency spectrum) [51].
Revised: In time series in particular, the time domain shows how readouts change with time whereas frequency domain tells us how much of the signal lies within each frequency band over a range of frequencies ( e.g. , frequency spectrum) [51].

Original: Explicitly considering the frequencydomain can provide a deeper understanding of time series behavior that can be challenging tocapture solely in time domain [52].
Revised: Explicitly considering the frequency domain can provide a deeper understanding of time series behavior that can be challenging to capture solely in time domain [52].

Original: However, most existing time-series methods focus exclusively onmodeling time domain, e.g. , [47, 48], and ignore frequency domain altogether.
Revised: However, most existing time-series methods focus exclusively on modeling time domain, e.g. , [47, 48], and ignore frequency domain altogether.

Original: One could argue thatis sufficient for high-capacity methods as time and frequency domains describe the same data, justfrom different aspects [53], and time series can be converted between time and frequency domainsusing transformation, such as Fourier [54, 52].
Revised: One could argue that is sufficient for high-capacity methods as time and frequency domains describe the same data, just from different aspects [53], and time series can be converted between time and frequency domains using transformation, such as Fourier [54, 52].

Original: While we are not interested in modeling the twodomains individually, the relationship between them, grounded in signal processing theory, providesan invariance that is valid regardless of time series distribution [55, 56] and thus can serve as aninductive bias for pre-training to promote generalizability.
Revised: While we are not interested in modeling the two domains individually, the relationship between them, grounded in signal processing theory, provides an invariance that is valid regardless of time series distribution [55, 56] and thus can serve as an inductive bias for pre-training to promote generalizability.

Original:  Approaching this invariance through thelens of representation learning, we formulate Time-Frequency Consistency (TF-C) by postulatingthat for every time
Revised: We provide an analogy, from images to time series, regarding ‘invariance’ in Appendix C. Approaching this invariance through the lens of representation learning, we formulate Time-Frequency Consistency (TF-C) by postulating that for every time

Original: series sample x i , there exists a latent time-frequency space in which time-basedrepresentation z T i and frequency-based representation z F i of the same sample, together with their localaugmentations (defined later) are close to each other.
Revised: series sample x i , there exists a latent time-frequency space in which time-based representation z T i and frequency-based representation z F i of the same sample, together with their local augmentations (defined later) are close to each other.

Original: Representational Time-Frequency Consistency (TF-C).
Revised: Representational Time-Frequency Consistency (TF-C).

Original: Let be given a time series sample x i .
Revised: Let be given a time series sample x i .

Original: Then in a model F satisfying TF-C, time-based representation z T i and frequency-based representationz F i learned from x i by F , and representations learned from local augmentations of x i are closetogether in the latent time-frequency space.
Revised: Then in a model F satisfying TF-C, time-based representation z T i and frequency-based representation z F i learned from x i by F , and representations learned from local augmentations of x i are close together in the latent time-frequency space.

Original: Our pre-training strategy is to use a time series dataset D pret to induce TF-C in F ’s model parametersΘ , which, in turn, are used to initialize the target model on D tune and produce generalizable representations for downstream prediction.
Revised: Our pre-training strategy is to use a time series dataset D pret to induce TF-C in F ’s model parameters Θ , which, in turn, are used to initialize the target model on D tune and produce generalizable representations for downstream prediction.

Original: The invariant nature of TF-C means that our approach can bridge
Revised: The invariant nature of TF-C means that our approach can bridge

Original: D pret and D tune even when large discrepancies exist between them (in terms of temporal dynamics,semantic meaning, etc.), providing a vehicle for a generalizable pre-training strategy for time series.
Revised: D pret and D tune even when large discrepancies exist between them (in terms of temporal dynamics, semantic meaning, etc.), providing a vehicle for a generalizable pre-training strategy for time series.

Original: To realize TF-C, our model F has four components (Figure 2): a time encoder G T , a frequencyencoder G F , and two cross-space projectors R T and R F that map time-based and frequency-basedrepresentations, respectively, to the same time-frequency space.
Revised: To realize TF-C, our model F has four components (Figure 2): a time encoder G T , a frequency encoder G F , and two cross-space projectors R T and R F that map time-based and frequency-based representations, respectively, to the same time-frequency space.

Original: Together, the four componentsprovide a way to embed x i to the latent time-frequency space such that time-based embeddingz T i
Revised: Together, the four components [Figure] provide a way to embed x i to the latent time-frequency space such that time-based embedding z T i

Original: =
Revised: =

Original: R T ( G T ( x T i )) and frequency-based embedding z F i = R F ( G F ( x F i )) are close together.
Revised: R T ( G T ( x T i )) and frequency-based embedding z F i = R F ( G F ( x F i )) are close together.

Original:  Our Approach
Revised:  Our Approach

Original: Next, we present the architecture of the developed self-supervised contrastive pre-training model
Revised: Next, we present the architecture of the developed self-supervised contrastive pre-training model

Original: F .
Revised: F .

Original: Without specification, the data mentioned in this section are from pre-training dataset and the 
Revised: Without specification, the data mentioned in this section are from pre-training dataset and the superscript pret is omitted for simplification.

Original: [Figure] superscript pret is omitted for simplification. Here we describe the model using univariate time seriesas an example, but our model can be straightforwardly applied to multivariate time series (Sec 5).
Revised: Here we describe the model using univariate time series as an example, but our model can be straightforwardly applied to multivariate time series (Sec 5).

Original: 4.1 Time-based Contrastive Encoder
Revised: 4.1 Time-based Contrastive Encoder

Original: For a given input time series sample x i , we generate an augmentation set X T i through a time-basedaugmentation bank B T : x T i → X T i .
Revised: For a given input time series sample x i , we generate an augmentation set X T i through a time-based augmentation bank B T : x T i → X T i .

Original: Each element (cid:101) x T i ∈
Revised: Each element (cid:101) x T i ∈

Original: X T i is augmented from x i based onthe temporal characteristics.
Revised: X T i is augmented from x i based on the temporal characteristics.

Original: Here, the time-based augmentation bank includes jittering, scaling,time-shifts, and neighborhood segments, all well-established in contrastive learning [40, 48, 41].
Revised: Here, the time-based augmentation bank includes jittering, scaling, time-shifts, and neighborhood segments, all well-established in contrastive learning [40, 48, 41].

Original: We develop the augmentation bank to produce diverse augmentations (rather than a single type ofaugmentation) and expose the model to complex temporal dynamics, which produces more robusttime-based embeddings [48].
Revised: We develop the augmentation bank to produce diverse augmentations (rather than a single type of augmentation) and expose the model to complex temporal dynamics, which produces more robust time-based embeddings [48].

Original: For the input x i , we randomly select an augmented sample (cid:101) x T
Revised: For the input x i , we randomly select an augmented sample (cid:101) x T

Original: i ∈
Revised: i ∈

Original: X T i and feed into a contrastivetime encoder G T that maps samples to embeddings.
Revised: X T i and feed into a contrastive time encoder G T that maps samples to embeddings.

Original: We have h T
Revised: We have h T

Original: i =
Revised: i =

Original: G T ( x T i ) and (cid:101) h T i =
Revised: G T ( x T i ) and (cid:101) h T i =

Original: G T ( (cid:101) x T i ) .
Revised: G T ( (cid:101) x T i ) .

Original: As(cid:101) x T i is generated based on x T i , after passing through G T , we assume the embedding of x T i is close tothe embedding of (cid:101)
Revised: As (cid:101) x T i is generated based on x T i , after passing through G T , we assume the embedding of x T i is close to the embedding of (cid:101)

Original: x T i but far away from the embedding of x T j and (cid:101) x
Revised: x T i but far away from the embedding of x T j and (cid:101) x

Original: T j that are derived from anothersample x T j ∈ D pret [34, 47, 41].
Revised: T j that are derived from another sample x T j ∈ D pret [34, 47, 41].

Original: In specific, we select the positive pair as ( x T i , (cid:101) x T i ) and negative pairsas ( x T i , x T j ) and ( x T i , (cid:101) x T j ) [34].
Revised: In specific, we select the positive pair as ( x T i , (cid:101) x T i ) and negative pairs as ( x T i , x T j ) and ( x T i , (cid:101) x T j ) [34].

Original: Contrastive time loss.
Revised: Contrastive time loss.

Original: To maximize the similarity within a positive pair and minimize the similaritywithin a negative pair, we adopt the NT-Xent (the normalized temperature-scaled cross entropy loss)as distance function d which is widely used in contrastive learning [34, 40].
Revised: To maximize the similarity within a positive pair and minimize the similarity within a negative pair, we adopt the NT-Xent (the normalized temperature-scaled cross entropy loss) as distance function d which is widely used in contrastive learning [34, 40].

Original: In specific, we measurethe loss of time-based contrastive encoder in terms of sample x T i by:
Revised: In specific, we measure the loss of time-based contrastive encoder in terms of sample x T i by:

Original: [Equation]
Revised: [Equation]

Original: where sim ( u , v )
Revised: where sim ( u , v )

Original: =
Revised: =

Original: u T v / ∥ u ∥ ∥ v ∥ denotes the cosine similarity, the 1 i ̸ = j ∈ { 0 , 1 } is a binaryindicator that equals to 0 when i = j , and τ is a temporal parameter.
Revised: u T v / ∥ u ∥ ∥ v ∥ denotes the cosine similarity, the 1 i ̸ = j ∈ { 0 , 1 } is a binary indicator that equals to 0 when i = j , and τ is a temporal parameter.

Original: The x j ∈ D pret refers toa different time series sample and its augmented sample.
Revised: The x j ∈ D pret refers to a different time series sample and its augmented sample.

Original: This loss urges the time encoder G T togenerate closer time-based embeddings for positive pairs, and push the embeddings for negative pairsapart from each other.
Revised: This loss urges the time encoder G T to generate closer time-based embeddings for positive pairs, and push the embeddings for negative pairs apart from each other.

Original: 4.2 Frequency-based Contrastive Encoder
Revised: 4.2 Frequency-based Contrastive Encoder

Original: We generate the frequency spectrum x F i from time series x T i through a transform operator ( e.g. ,
Revised: We generate the frequency spectrum x F i from time series x T i through a transform operator ( e.g. ,

Original: Fourier Transformation [54]).
Revised: Fourier Transformation [54]).

Original: The frequency information of time series is universal and plays keyrole in classic signal processing [57, 53, 55], but rarely investigated in self-supervised contrastive learning for time series [58].
Revised: The frequency information of time series is universal and plays key role in classic signal processing [57, 53, 55], but rarely investigated in self-supervised contrastive learning for time series [58].

Original: In this section, we develop augmentation methods to perturb x F i basedon characteristics of frequency spectra and generate frequency-based representations.
Revised: In this section, we develop augmentation methods to perturb x F i based on characteristics of frequency spectra and generate frequency-based representations.

Original: As every frequency component in the frequency spectrum denotes a base function ( e.g. , sinusoidalfunction for Fourier transformation) with the corresponding frequency and amplitude, in our augmentation methods, we perturb the frequency spectrum by adding or removing frequency components.
Revised: As every frequency component in the frequency spectrum denotes a base function ( e.g. , sinusoidal function for Fourier transformation) with the corresponding frequency and amplitude, in our augmentation methods, we perturb the frequency spectrum by adding or removing frequency components.

Original: Asmall perturbation on frequency spectrum may cause large changes to the temporal patterns in timedomain [55].
Revised: A small perturbation on frequency spectrum may cause large changes to the temporal patterns in time domain [55].

Original: To make sure the perturbed time series is still similar to the original sample (not only infrequency domain but also in time domain; Figure 6), we use a small budget E in the perturbationswhere E denotes the number of frequency components we manipulate.
Revised: To make sure the perturbed time series is still similar to the original sample (not only in frequency domain but also in time domain; Figure 6), we use a small budget E in the perturbations where E denotes the number of frequency components we manipulate.

Original: While removing frequencycomponents, we randomly select E frequency components and set their amplitudes as 0 .
Revised: While removing frequency components, we randomly select E frequency components and set their amplitudes as 0 .

Original: Whileadding frequency components, we randomly choose E frequency components from the ones havesmaller amplitude than α · A m , and increase their amplitude to α · A m .
Revised: While adding frequency components, we randomly choose E frequency components from the ones have smaller amplitude than α · A m , and increase their amplitude to α · A m .

Original: The A m is the maximumamplitude in the frequency spectrum and α is a pre-defined coefficient to adjust the scale of theperturbed frequency component ( α = 0 .
Revised: The A m is the maximum amplitude in the frequency spectrum and α is a pre-defined coefficient to adjust the scale of the perturbed frequency component ( α = 0 .

Original: 5 in this work).
Revised: 5 in this work).

Original: We produce an augmentation set X F i forx F i through frequency-augmentation bank B F : x F i → X F i .
Revised: We produce an augmentation set X F i for x F i through frequency-augmentation bank B F : x F i → X F i .

Original: We have two augmentation methods ( i.e. ,removing or adding frequency components) in B F , |X F i | = 2 . 
Revised: We have two augmentation methods ( i.e. , removing or adding frequency components) in B F , |X F i | = 2 . More details on the exploration of frequency augmentation strategies in Appendix J.

Original: We utilize a frequency encoder G F to map the frequency spectrum ( e.g. , x F i ) to a frequency-basedembedding ( e.g. , h F i = G F ( x F i ) ).
Revised: We utilize a frequency encoder G F to map the frequency spectrum ( e.g. , x F i ) to a frequency-based embedding ( e.g. , h F i = G F ( x F i ) ).

Original: We assume the frequency encoder G F can learn similar embeddingfor the original frequency spectrum x F i and a slightly perturbed frequency spectrum (cid:101)
Revised: We assume the frequency encoder G F can learn similar embedding for the original frequency spectrum x F i and a slightly perturbed frequency spectrum (cid:101)

Original: x F i ∈
Revised: x F i ∈

Original: X F i .
Revised: X F i .

Original: Thus,we set the positive pair as ( x F i , (cid:101) x F i ) and the negative pairs as ( x F i , x F j ) and ( x F i , (cid:101) x F [Equation]
Revised: Thus, we set the positive pair as ( x F i , (cid:101) x F i ) and the negative pairs as ( x F i , x F j ) and ( x F i , (cid:101) x F j ) .

Original: Contrastive frequency loss.
Revised: Contrastive frequency loss.

Original: We calculate frequency-based contrastive loss for sample x i as:
Revised: We calculate frequency-based contrastive loss for sample x i as:

Original: [Equation]
Revised: [Equation]

Original: In preliminary experiments, we find that the value of τ has little effect on performance and use thesame τ throughout all experiments.
Revised: In preliminary experiments, we find that the value of τ has little effect on performance and use the same τ throughout all experiments.

Original: The L F ,i yield a frequency encoder G F producing embeddingsinvariant to frequency spectrum perturbations.
Revised: The L F ,i yield a frequency encoder G F producing embeddings invariant to frequency spectrum perturbations.

Original: 4.3 Time-Frequency Consistency
Revised: 4.3 Time-Frequency Consistency

Original: We develop a consistency loss item L C ,i to urge the learned embeddings to satisfy TF-C: for a givensample, its time-based and frequency-based embeddings (and their local neighborhoods) are supposedto be close to each other (see Sec. 3 for justification).
Revised: We develop a consistency loss item L C ,i to urge the learned embeddings to satisfy TF-C: for a given sample, its time-based and frequency-based embeddings (and their local neighborhoods) are supposed to be close to each other (see Sec. 3 for justification).

Original: To make sure the distance between embeddingsis measurable, we map h T i from time space and h F i from frequency space to a joint time-frequencyspace through projectors R T and R F , respectively.
Revised: To make sure the distance between embeddings is measurable, we map h T i from time space and h F i from frequency space to a joint time-frequency space through projectors R T and R F , respectively.

Original: In specific, for every input sample x i , we havefour embeddings, which are z T 
Revised: In specific, for every input sample x i , we have four embeddings, which are z T i

Original: i =
Revised: =

Original: R T ( h T i ) , (cid:101)
Revised: R T ( h T i ) , (cid:101)

Original: z T i
Revised: z T i

Original: =
Revised: =

Original: R T ( (cid:101) h T i ) , z F i
Revised: R T ( (cid:101) h T i ) , z F i

Original: =
Revised: =

Original: R F ( h F i ) , and (cid:101)
Revised: R F ( h F i ) , and (cid:101)

Original: z F i
Revised: z F i

Original: =
Revised: =

Original: R F ( (cid:101) h F i ) .
Revised: R F ( (cid:101) h F i ) .

Original: The firsttwo embeddings are generated based on temporal characteristics and the latter two embeddings areproduced based on the properties of frequency spectrum.
Revised: The first two embeddings are generated based on temporal characteristics and the latter two embeddings are produced based on the properties of frequency spectrum.

Original: To enforce the embeddings in the time-frequency space subject to TF-C, we design a consistency loss
Revised: To enforce the embeddings in the time-frequency space subject to TF-C, we design a consistency loss

Original: L C ,i that measures the distance between a time-based embedding and a frequency-based embedding.
Revised: L C ,i that measures the distance between a time-based embedding and a frequency-based embedding.

Original: We use S T F i =
Revised: We use S T F i =

Original: d ( z T i , z F i , D pret ) to denote the distance between z T i and z F i ) .
Revised: d ( z T i , z F i , D pret ) to denote the distance between z T i and z F i ) .

Original: Similarly, we define S T (cid:101) F i ,
Revised: Similarly, we define S T (cid:101) F i ,

Original: S (cid:101) TF i , and S (cid:101) T (cid:101) F i .
Revised: S (cid:101) TF i , and S (cid:101) T (cid:101) F i .

Original: Note, in this time-frequency space, we don’t consider the distance between z T i and (cid:101)
Revised: Note, in this time-frequency space, we don’t consider the distance between z T i and (cid:101)

Original: z T iwhere the two embeddings are from the same domain ( i.e. , time domain).
Revised: z T i where the two embeddings are from the same domain ( i.e. , time domain).

Original: The same applies to pairthe distance between z F i and (cid:101)
Revised: The same applies to pair the distance between z F i and (cid:101)

Original: z F i .
Revised: z F i .

Original: We have already considered information of above two pairs in thecalculation of L T ,i and L F ,i .
Revised: We have already considered information of above two pairs in the calculation of L T ,i and L F ,i .

Original: Next, let’s closely observe S T F i and S T (cid:101)
Revised: Next, let’s closely observe S T F i and S T (cid:101)

Original: F i that involve three embeddings: z T i , z F i , and (cid:101)
Revised: F i that involve three embeddings: z T i , z F i , and (cid:101)

Original: z F i .
Revised: z F i .

Original: Here, z T i andz F i are learned from the original sample ( x T i and x F i ) while (cid:101)
Revised: Here, z T i and z F i are learned from the original sample ( x T i and x F i ) while (cid:101)

Original: z F i is learned from the augmented (cid:101) x
Revised: z F i is learned from the augmented (cid:101) x

Original: F i .
Revised: F i .

Original: Thus,intuitively, z T i should be closer to z F i in comparison to (cid:101)
Revised: Thus, intuitively, z T i should be closer to z F i in comparison to (cid:101)

Original: z F i .
Revised: z F i .

Original: Motivated by the relative relationship, weencourage the proposed model to learn a S T F i that is smaller than S T (cid:101) F i .
Revised: Motivated by the relative relationship, we encourage the proposed model to learn a S T F i that is smaller than S T (cid:101) F i .

Original: Inspired by the triplet loss [59],we design ( S T F i − S T (cid:101) F i + δ ) as a term of consistency loss L C ,i where δ is a given constant marginto keep negative samples far apart [60].
Revised: Inspired by the triplet loss [59], we design ( S T F i − S T (cid:101) F i + δ ) as a term of consistency loss L C ,i where δ is a given constant margin to keep negative samples far apart [60].

Original: This term optimize the model towards a smaller S T F i andrelatively larger S T (cid:101) F i .
Revised: This term optimize the model towards a smaller S T F i and relatively larger S T (cid:101) F i .

Original: Similarly, S TF i is supposed to be smaller than S (cid:101) TF i and S (cid:101) T (cid:101) F i , respectively.
Revised: Similarly, S TF i is supposed to be smaller than S (cid:101) TF i and S (cid:101) T (cid:101) F i , respectively.

Original: Insummary, we calculate the consistency loss L C , i for sample x i by:
Revised: In summary, we calculate the consistency loss L C ,i for sample x i by:

Original: [Equation]
Revised: [Equation]

Original: where S pair i denotes between a time-based embedding ( e.g. , z T i or (cid:101)
Revised: where S pair i denotes between a time-based embedding ( e.g. , z T i or (cid:101)

Original: z T i ) and a frequency-based embedding ( e.g. , z F i or (cid:101)
Revised: z T i ) and a frequency-based embedding ( e.g. , z F i or (cid:101)

Original: z F i ).
Revised: z F i ).

Original: In which pair, there is at least one embedding that is derived from augmentedsample instead of the original sample.
Revised: In which pair, there is at least one embedding that is derived from augmented sample instead of the original sample.

Original: The δ is a pre-defined constant.
Revised: The δ is a pre-defined constant.

Original: By combining all the tripletloss items, L C encourages the pre-training model to capture the consistency between time-based andfrequency-based embeddings in model optimization.
Revised: By combining all the triplet loss items, L C encourages the pre-training model to capture the consistency between time-based and frequency-based embeddings in model optimization.

Original: Note, although the Eq.
Revised: Note, although the Eq.

Original: 3 does not explicitlymeasure the loss across different time series samples ( e.g. , x i and x j ), the cross-sample relationshipsare implicitly covered in the calculation of S TF i and S pair i .
Revised: 3 does not explicitly measure the loss across different time series samples ( e.g. , x i and x j ), the cross-sample relationships are implicitly covered in the calculation of S TF i and S pair i .

Original: 4.4 Implementation and Technical Details
Revised: 4.4 Implementation and Technical Details

Original: The overall loss function in pre-training has three terms.
Revised: The overall loss function in pre-training has three terms.

Original: First, time-based contrastive loss L T urges themodel to learn embeddings invariant to temporal augmentations.
Revised: First, time-based contrastive loss L T urges the model to learn embeddings invariant to temporal augmentations.

Original: Second, frequency-based contrastiveloss L F promotes learning of embeddings invariant to frequency spectrum-based augmentations.
Revised: Second, frequency-based contrastive loss L F promotes learning of embeddings invariant to frequency spectrum-based augmentations.

Original: Third, the consistency loss L C guides the model to retain the consistency between time-based andfrequency-based embeddings.
Revised: Third, the consistency loss L C guides the model to retain the consistency between time-based and frequency-based embeddings.

Original: In summary, pre-training loss is the following:
Revised: In summary, pre-training loss is the following:

Original: [Equation]
Revised: [Equation]

Original: ) where λ controls the relative importance of contrastive and consistency losses.
Revised: where λ controls the relative importance of contrastive and consistency losses.

Original: We calculate thetotal loss by summing L TF-C ,i across all pre-training samples.
Revised: We calculate the total loss by summing L TF-C ,i across all pre-training samples.

Original: In implementation, the contrastivelosses are calculated within the batch.
Revised: In implementation, the contrastive losses are calculated within the batch.

Original: Recall our problem definition, the model F we want to learn isthe combination of neural networks G T , R T , G F , and R F .
Revised: Recall our problem definition, the model F we want to learn is the combination of neural networks G T , R T , G F , and R F .

Original: When pre-training is completed, we storeparameters of entire model, and denote it as F ( · , Θ) where Θ represents all trainable parameters.
Revised: When pre-training is completed, we store parameters of entire model, and denote it as F ( · , Θ) where Θ represents all trainable parameters.

Original: When a test sample x tune i is presented, fine-tuned model F ( · , Φ) generates embedding z tune i byconcatenating time- and frequency-based embeddings as: z tunei
Revised: When a test sample x tune i is presented, fine-tuned model F ( · , Φ) generates embedding z tune i by concatenating time- and frequency-based embeddings as: z tune

Original: [Equation]
Revised: [Equation]

Original: ]where Φ denotes fine-tuned model parameters.
Revised: where Φ denotes fine-tuned model parameters.

Original:  Experiments
Revised:  Experiments

Original: Datasets. We evaluate the new approach on 8 diverse datasets. 
Revised: We compare the TF-C with 10 baselines on 8 diverse datasets. We investigate the time series classification tasks in the context of one-to-one and one-to-many transfer learning setups (the manyto-one setting is fundamentally different as discussed in Appendix K).

Original: (1) S LEEP E EG [61] has 371,055univariate brainwaves (EEG; 100 Hz) collected from 197 individuals.
Revised: We also assess TF-C in extensive downstream tasks including clustering and anomaly detection. Datasets [Equation] E EG [61] has 371,055 univariate brainwaves (EEG; 100 Hz) collected from 197 individuals.

Original: Each sample associates withone among five sleeping stages.
Revised: Each sample associates with one among five sleeping stages.

Original: (2) E PILEPSY [62] monitors the brain activities of 500 subjects withsingle-channel EEG sensor (174 Hz).
Revised: (2) E PILEPSY [62] monitors the brain activities of 500 subjects with single-channel EEG sensor (174 Hz).

Original: Sample is labelled in binary based on whether the subject hasepilepsy or not.
Revised: Sample is labelled in binary based on whether the subject has epilepsy or not.

Original: (3) F D - A [63] gathers the vibration signals from rolling bearing from a mechanicalsystem aiming at fault detection.
Revised: (3) F D - A [63] gathers the vibration signals from rolling bearing from a mechanical system aiming at fault detection.

Original: Every sample has 5,120 timestamps and an indicator for one outof three mechanical device states.
Revised: Every sample has 5,120 timestamps and an indicator for one out of three mechanical device states.

Original: (4) F D - B [63] has the same setting as the F D - A but the rollingbearings are performed in different working conditions ( e.g. , varying rotational speed).
Revised: (4) F D - B [63] has the same setting as the F D - A but the rolling bearings are performed in different working conditions ( e.g. , varying rotational speed).

Original: (5) H AR [64]has 10,299 9-dimension samples from 6 daily activities.
Revised: (5) H AR [64] has 10,299 9-dimension samples fromdaily activities.

Original: (6) G ESTURE [65] includes 440 samplesthat are collected from 8 hand gestures recorded by accelerometer.
Revised: (6) G ESTURE [65] includes 440 samples that are collected from 8 hand gestures recorded by accelerometer.

Original: (7) E CG [26] contains 8,528single-sensor ECG recordings with sorted into four classes based on human physiology.
Revised: (7) E CG [26] contains 8,528 single-sensor ECG recordings with sorted into four classes based on human physiology.

Original: (8) E MG[66] consists of 163 EMG samples with 3-class labels implying muscular diseases.
Revised: (8) E MG [66] consists of 163 EMG samples with 3-class labels implying muscular diseases.

Original: Dataset labels arenot used in pre-training.
Revised: Dataset labels are not used in pre-training.

Original: Further dataset statistics are in Appendix B andAppendix Table 3. 
Revised: Further dataset statistics are in Appendix D and Table 3. [Table]

Original: Baselines.
Revised: Baselines.

Original: We consider 8 baseline methods.
Revised: We consider 10 baseline methods.

Original: This includes 6 state-of-the-art methods: TS-SD [12],
Revised: This includes 8 state-of-the-art methods: TS-SD [12],

Original: TS2vec [47], CLOCS [41], Mixing-up [18], TS-TCC [48],and SimCLR [40].
Revised: TS2vec [47], CLOCS [41], Mixing-up [18], TS-TCC [48], SimCLR [40], TNC [46], and CPC [30].

Original: The TS2Vec, TS-TCC,and SimCLR are designed for representation learning on a single dataset (not across datasets), weapply them fit our settings to make the results comparable. 
Revised: The TS2Vec, TS-TCC, SimCLR, TNC, and CPC are designed for representation learning on a single dataset (not across datasets), we apply them fit our settings to make the results comparable. As the training of TNC and CPC are very time-consuming, we only compare with them in the setting of one-to-one setting (scenario 1).

Original:  To examine utility of pre-training, weconsider two approaches (non-deep learning KNN model and the Random Init.
Revised: TNC and CPC are relatively less competitive (Table 4), thus they are not included in other experiments. To examine utility of pre-training, we consider two approaches (non-deep learning KNN model and the Random Init.

Original: approach, whichrandomly initializes the fine-tuning model) that are applied directly to fine-tuning datasets withoutany pre-training.
Revised: approach, which randomly initializes the fine-tuning model) that are applied directly to fine-tuning datasets without any pre-training.

Original: The evaluation metrics are accuracy, precision (macro-averaged), recall, F1 score, AUROC, and AUPRC.
Revised: The evaluation metrics are accuracy, precision (macro-averaged), recall, F1 score, AUROC, and AUPRC.

Original: Method details and hyper-parameter selection are in Appendix D.

Original: Implementation.
Revised: Implementation.

Original: We use two 3-layer 1-D ResNets [67] as backbones for G T and G F encoders.
Revised: We use two 3-layer 1-D ResNets [67] as backbones for G T and G F encoders.

Original: Datasets contain long series (samples in F D - A and F D - B have 5,120 observations), and preliminaryexperiments identified ResNet as a better option than a Transformer variant [68].
Revised: Datasets contain long series (samples in F D - A and F D - B have 5,120 observations), and preliminary experiments identified ResNet as a better option than a Transformer variant [68].

Original: We use 2 fullyconnected layers for R T and R F (no sharing of parameters).
Revised: We use 2 fullyconnected layers for R T and R F (no sharing of parameters).

Original: We set E = 

Revised: We set E =

Original: and α = 0 .
Revised: and α = 0 .

Original: 5 in frequencyaugmentations and τ = 0 .
Revised: 5 in frequency augmentations and τ = 0 .

Original: 2 , δ = 1 , λ = 0 .
Revised: 2 , δ = 1 , λ = 0 .

Original: 5 in loss functions.
Revised: 5 in loss functions.

Original: Reported are mean and standarddeviation values across 5 independent runs (both pre-training and fine-tuning) on the same data split.
Revised: Reported are mean and standard deviation values across 5 independent runs (both pre-training and fine-tuning) on the same data split.

Original: Results for KNN (K=2) do not change so standard deviation is zero.
Revised: Results for KNN (K=2) do not change so standard deviation is zero.

Original: Details are in Appendix E.[Table]
Revised: Method details and hyper-parameter selection are in Appendix E.

Original: 5.1 Results: One-to-One Pre-Training Evaluation
Revised: 5.1 Results: One-to-One Pre-Training Evaluation

Original: Setup.
Revised: Setup.

Original: In one-to-one evaluation, we pre-train a model on one pre-training dataset and use it for fine-tuning on one target dataset only.[Table] [Table] [Table]
Revised: In one-to-one evaluation, we pre-train a model on one pre-training dataset and use it for fine-tuning on one target dataset only.

Original: [Table]

Original: [Table] [Table] [Table] 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 (S LEEP E EG → E PILEPSY ):
Revised: Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 (S LEEP E EG → E PILEPSY ):

Original: Pre-training is doneon S LEEP E EG and fine-tuning on E PILEPSY .
Revised: Pre-training is done on S LEEP E EG and fine-tuning on E PILEPSY .

Original: While both datasets describe a single-channel EEG,the signals are from different channels/positions on scalp, monitor different physiology (sleepvs. epilepsy), and are collected from different patients.Scenario [Table] [Table]
Revised: While both datasets describe a single-channel EEG, the signals are from different channels/positions on scalp, monitor different physiology (sleep vs. epilepsy), and are collected from different patients.

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] 2 Scenario 2 (F D - A → F D - B ): Datasets describe mechanical devices that operate in different working conditions, including rotational speed,load torque, and radial force.
Revised: Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 (F D - A → F D - B ): Datasets describe mechanical devices that operate in different working conditions, including rotational speed, load torque, and radial force.

Original: Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3  (H AR → G ESTURE ): Datasets record different activities (6types of human daily activities vs. 8 hand gestures).
Revised: Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 (H AR → G ESTURE ): Datasets record different activities ( types of human daily activities vs. 8 hand gestures).

Original: While both datasets contain acceleration signals,
Revised: While both datasets contain acceleration signals,

Original: H AR has 9 channels while G ESTURE has 1 channel.
Revised: H AR has 9 channels while G ESTURE has 1 channel.

Original: Scenario 
Revised: Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 (E CG → E MG ): While both are physiological datasets, the E CG records the electrical signal from the heart whereas E MG measures muscle response in response to a nerve’s stimulation of the muscle. We note that the discrepancy between pre-training and fine-tuning datasets in the above four scenarios is substantial, covering diverse kinds of variation in time series datasets: varying semantic meaning, sampling frequency, time series length, number of classes, and system factors ( e.g. , number of devices or subjects). The setup is further challenged by relatively small number of samples available for fine-tuning (E PILEPSY : 60; F D - B : 60; G ESTURE : 480; E MG : 122).

Original: [Table] [Table] [Table]
Revised: Further details are in AppendixF.

Original: [Table]
Revised: Results.

Original: [Table]
Revised: The results for four scenarios are shown in Table 1 and Tables 4-6. Overall, TF-C model has won 16 out of 24 tests (6 metrics in 4 scenarios), and is the second best performer in only 8 other tests. We report all metrics but discuss the F1 score in the following.

Original: [Table]
Revised: On average, our TF-C model claims a large margin of 15.4% over all baselines. Although the strongest baseline is varying (such as TS-TCC in Scenario 2; Mixing-up in Scenario 3), our model outperforms the strongest baselines by 1.5% across all scenarios. Specifically, when using H AR -pre-trained model for fine-tuning on G ESTURE (Scenario 3) (Table 1), the TF-C approach achieves the highest performance of 79.91% which yields a margin of 7.2% over the best baseline TS-TCC (74.57%). One potential reason that Scenariois the most complex dataset in terms of classes (H AR has 6 classes while G ESTURE has 8 classes)

Original: [Table] [Table]
Revised: [Table] that is difficult to model.

Original: 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 (E CG → E MG ): While both arephysiological datasets, the E CG records the electrical signal from  the heart whereas E MG measuresmuscle response in response to a nerve’s stimulation of the muscle.
Revised: Thee assumption is proved from side by the relative lower performances in Scenario 3 ( < 80% ) than other scenarios ( > 90% ). We visualize the learned representations in time-frequency space (Appendix I) which strongly support our assumption on TF-C. 5.2 Results: One-to-Many Pre Training Evaluation

Original: We note that the discrepancybetween pre-training and fine-tuning datasets in the above four scenarios is substantial, coveringdiverse kinds of variation in time series datasets: varying semantic meaning, sampling frequency,time series length, number of classes, and system factors ( e.g. , number of devices or subjects).
Revised: Setup.

Original: Thesetup is further challenged byrelatively small number of samples available for fine-tuning (E PILEPSY :60; F D- B : 60; G ESTURE : 480; E MG : 122). Further details are in Appendix B-C.
Revised: In one-to-many evaluation, pre-training is done using one dataset followed by fine-tuning on multiple target datasets independently without starting pre-training from scratch. Out of eight datasets, S LEEP E EG has most complex temporal dynamics [69] and is the largest (371,055 samples). For that reason, we pre-train a model on S LEEP E EG and separately fine-tune a well-pre-trained model on E PILEPSY , F D

Original: Results.
Revised: Results.

Original: The results for four scenarios are shown in Table 1 and Appendix Tables 4-6.
Revised: Results are shown in Table 10. Intuitively, there are less common between EEG signals and vibration, acceleration and EMG, we expect that the transfer learning will be less effective compared to the one-to-one evaluation. The pre-training and fine-tuning datasets are largely different in the bottom three blocks (S LEEP E EG → {F D - B , G ESTURE , E MG }).

Original: Overall,
Revised: The large gap reasonably leads to a deterioration in baseline performances, however, our model has obviously higher tolerance to knowledge transfer across datasets with large difference. Notably, We find that the proposed model with TF-C earned best performance in 14 out of 18 settings in the three challenging settings: indicating our TF-C assumption is universal in time series. For example, our approach outperforms the strongest baseline by 8.4% (in precision) when fine-tuning on G ESTURE . Our model has large potential to serve as a universal model when there is no large pre-training dataset that is similar to the small fine-tuning dataset.

Original:  TF-C model has won 16 out of 24 tests (6 metrics in 4 scenarios), and is the second best performerin only 8 other tests.
Revised: Further, the TF-C approach consistently outperforms KNN and Random Init. (which are not pre-trained) by a large margin of 42.8% and 25.1% (both in F1 score) on average.

Original: We report all metrics but discuss the F1 score in  the following.
Revised: Ablation study. In the context of one-to-one classification setting, we wanted to evaluate how relevant model component are for effective pre-training. As shown in Table 9 (S LEEP E EG → G ESTURE ), removing L C , L T , and L F result in performance degradation (precision) of 6.1%, 7.2%, and 6.7%, respectively. To validate that the performance increment is not solely brought by a third loss term no matter what consistency it measures, we replaced consistency loss L C with a loss term measuring the consistency within time space (named L TT-C ) or within frequency space (named L FF-C ).

Original: On average,our TF-C model claims a large margin of 15.4% over all baselines.
Revised: Results show our consistency loss outperforms L TT-C and L FF-C by 5.3% and 7.2% (accuracy), respectively. We also find that full fine-tuning (optimizing the whole model) performs slightly better than partial fine-tuning (updating only final layers in R T and R F ), by 5.4% in F1 score. More details in Appendix H.

Original: Although the strongest baselineis varying (such as TS-TCC in Scenario 2; Mixing-up in Scenario 3), our model outperforms thestrongest baselines by 1.5% across all scenarios.
Revised: Clustering Task. We evaluate the clustering performance of TF-C taking S LEEP E EG → E PILEPSY as an example. Specifically, we added a K-means (K=2), as Epilepsy has 2 classes, on top of z tune i in fine-tuning.

Original: Specifically, when using H AR -pre-trained model forfine-tuning on G ESTURE (Scenario 3) (Table 1), the TF-C approach achieves the highest performanceof 79.91% which yields a  margin of 7.2% over the best baseline TS-TCC (74.57%).
Revised: We adopt commonly used evaluation metrics: Silhouette score, Adjusted Rand Index (ARI), and Normalized Mutual Information (NMI). Table 7 shows our TF-C obtains the best clustering surpassing the strongest baseline (TS-TCC) by a large margin (5.4% in Silhouette score). It conveys that TF-C can capture more distinctive representations with the knowledge transferred from pre-training, which is consistent with the superiority of TF-C in above classification tasks.

Original: One potentialreason that Scenario 3 is the most complex dataset in terms of classes (H AR has 6 classes while
Revised: Anomaly Detection Task. Moreover, we evaluate how TF-C performs for anomaly detection. Note we work on the sample-level rather than observation-level anomaly detection. Based on global patterns, the former aims to detect abnormal time series samples instead of outlier observations in the samples (as in BTSF [50] and USAD [70]), which emphasizes the local context.

Original: G ESTURE has 8 classes) that is difficult to model.
Revised: To examine our TF-C for anomaly detection, we chose a different setup (F D - A → F D - B ) different from the setup (S LEEP E EG → E PILEPSY ) used for clustering above. Specifically, we built a small subset of F D - B with 1,000 samples, of which 900 are from undamaged bearings, and the remainingare from bearings with inner or outer damage. Undamaged samples are considered “normal,” and inner/outer damaged samples are “outliers.”

Original: Thee assumption is proved from side by the relativelower performances in Scenario 3 ( < 80% ) than other scenarios ( > 90% ). 5.2 Results: One-to-Many Pre-Training Evaluation
Revised: In fine-tuning, we used one-class SVM on top of learned representations z tune i . The experimental results (Table 8) show that our TF-C outperforms five competitive baselines with up to 4.5% in F-1 Score. Results show that the proposed TF-C is more sensitive to anomalous samples and can effectively detect the abnormal status in mechanical devices.

Original:  Setup.
Revised:  Conclusion

Original: In one-to-many evaluation, pre-training is done using one dataset followed by fine-tuningon multiple target datasets independently without starting pre-training from scratch.
Revised: We develop a pre-training approach that introduces time-frequency consistency (TF-C) as a mechanism to support knowledge transfer between time-series datasets. The approach uses self-supervised contrastive estimation and injects TF-C into a pre-training model, bringing the encoded time-based and frequency-based representations and their local neighborhoods close together in the latent space.

Original: Out of eightdatasets, S LEEP E EG has most complex temporal dynamics [69] and is the largest (371,055 samples).
Revised: Limitations and future directions.

Original: For that reason, we pre-train a model on S LEEP E EG and separately fine-tune a well-pre-trained modelon E PILEPSY , F D - B , G ESTURE , and E MG . Results.
Revised: We posit that TF-C property can serve as a generalizable property for pre-training on diverse time series datasets. We are aware that there might exist additional generalizable properties, such as temporal autoregressive processes, that could also prove useful for pre-training on time series.

Original: Results are shown in Table 2.
Revised: Moreover, our method expects the input time series to be regularly sampled. However, our model can handle irregularly sampled time series using an encoder (such as Raindrop [71] and SeFT [72]) that can embed irregular time series. For frequency encoder inputs x F i , possible alternatives include resampling or interpolation to obtain regularly sampled signals and using regular or non-uniform FFT operations.

Revised: Furthermore, we agree that the current embedding scheme and loss functions are favorable for classification tasks that leverage global information over tasks that use local context (e.g., forecasting).

Revised: Even though, we demonstrate that our model achieves superior performance in broad downstream tasks such as clustering and anomaly detection (Sec. 5.3).

Revised: Acknowledgments and Disclosure of Funding

Revised: M.Z. and X.Z. are supported, in part, by NSF under Nos.

Revised: IIS-2030459 and IIS-2033384, US Air Force

Revised: Contract No. FA8702-15-D-0001, Harvard Data Science Initiative, Amazon Research Award, Bayer

Revised: Early Excellence in Science Award, AstraZeneca Research, and Roche Alliance with Distinguished

Revised: Scientists Award.

Revised: T.T. is supported by the Under Secretary of Defense for Research and Engineering under US Air Force Contract No. FA8702-15-D-0001.

Revised: Broader Impacts

Revised: Our proposed novel framework for time series unsupervised pre-training improves classification performance on target datasets in different application scenarios.

Revised: The recognition of time-frequency consistency as a universal property specific to time series data is a weak assumption that enables effective, task- and domain-agnostic transfer learning.

Revised: We believe our work will inspire the research community to uncover other universal properties for transfer learning.

Revised: We also hope our work will also attract more researchers to the more general problem of time series representation learning which is still underappreciated relative to problems from CV and NLP fields.

Revised: On the society level, our work, along the line of transfer learning, can facilitate more efficient use of time series data in various settings.

Revised: For example, in medical settings, some diseases of clinical interest may have very small labelled dataset.

Revised: In this case, unlabelled data from patients of different diseases but with similar underlying physiological conditions can be used to pre-train the model.

Revised: However, practitioners need to be aware of the limitations of the model, including that it may make biased predictions.

Revised: Specifically, bias may exist in the source dataset used for pre-training due to an imbalance of samples from subjects of different demographic attributes.

Revised: Also, the standardized medical protocols for collecting these datasets might be unsuitable for subjects with certain physiological attributes, creating unforseen bias that may be transferred to the finetuning phase.

Revised: All datasets in this paper are publicly available and are not associated with any privacy or security concern.

Revised: Furthermore, we have followed guidelines on responsible use specified by primary authors of the datasets used in the current work.

Revised: Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Revised: [Yes] In abstract and introduction, we claim that TF-C is a generalizable property of time series that can support pre-training, which is welljustified in Sec.

Revised: 3 and experimentally demonstrated in Sec. 5 (our model consistently performs comparatively to or above baseline methods).

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes] See Section 6.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] See Broader Impact on Page 10.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Revised: [N/A] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] Yes, we include an anonymous link (see Abstract) that provides the source codes with all implementation details, implementation of baselines, and eight datasets.

Revised: The link will be updated to an non-anonymous link after acceptance.

Revised: (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?

Revised: [Yes] See implementation details in Sec. 5.

Revised: See Appendix E for baseline architectures and hyper-parameter settings.

Revised: More details can be found in the included URL.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] We run experiments for 5 times and report the average value with standard deviation.

Revised: See Table 1, Tables 4-6, and Table 10.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See Appendix E. 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] We used eight existing datasets and 6 state-of-the-art baselines in contrastive learning and pre-training for time series.

Revised: We cited the creators for every exist asset we used.

Revised: See Sec.

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] All dataset licenses are mentioned in the Appendix D. (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes] See the anonymous link in Abstract.

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Revised: [No] All data we use is freely available for download, without any requirement to re-contact the data curator.

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [No] Our datasets are public, well-established, and do not contain PII or offensive content 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: Intuitively, there are less common between EEG signals andvibration, acceleration andEMG, we expect that the transfer learning will be less effective comparedto the one-to-one evaluation.
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Appendix A Further information on the relationship between our pre-training approach and domain adaptation Here we note our problem definition of pre-training is fundamentally different from domain adaptation [S1, S2, S3, S4, S5, S6] 1 in order to prevent any confusion between this work and domain adaptation methods.

Original: The pre-training and fine-tuning datasets are largely different in thebottom three blocks (S LEEP E EG → {F D - B , G ESTURE , E MG }).
Revised: DA applies a model trained on a pre-training dataset ( i.e. , source dataset) to a different target dataset [21, 42]. In contrast, self-supervised pre-training has four key differences with domain adaptation. (1) First, our model only requires the pre-training dataset while domain adaptation techniques generally require access to the target dataset [S7, S8, S9, S10, S11, S12, S13, S14, S15, S16]. (2) Second, our model can be applied to multiple unseen target datasets (without re-training the pre-trained model for every target dataset) while domain adaptation approaches use the target dataset during model training, e.g. , [S17, S18, S19, S20, S12, S14, S21] (see also Sec.

Original: The large gap reasonably leadsto a deterioration in baseline performances, however, our model has obviously higher toleranceto knowledge transfer across datasets with large difference.
Revised: 5.2 for experimental results).

Original: Notably, We find that the proposedmodel with TF-C earned best performance in 14 out of 18 settings in  the three challenging settings:indicating our TF-C assumption is universal in time series.
Revised: (3) Third, our approach can be used in scenarios where the feature space in pre-training is different from that in the target dataset (see Scenarios 1, 3, and 4 in Sec. 5.1 for experimental results). In contrast, domain adaptation methods usually restrict pre-training and target datasets to have the same feature space (but possible different distributions), e.g. , [S22, S18, S19, S20, S13].

Original: For example, our approach outperformsthe strongest baseline by 8.4% (in precision) when fine-tuning on G ESTURE . Our model has largepotential to serve as a universal model when there is no large pre-training dataset that is similar to thesmall fine-tuning dataset.
Revised: In summary, to support transfer learning across different time series datasets, a pre-training approach needs a capability to capture a generalizable property of time series, one that is shared across different time series datasets regardless of the specific semantic meaning of a time series signal ( e.g. , ECG, EMG, acceleration, vibration), conditions of data acquisition ( e.g. , variation across subjects and devices), sampling frequencies, etc. This work develops a self-supervised contrastive pre-training strategy that fulfills these requirements by injecting an appropriate inductive bias (called

Original:  Further,  the TF-C approach consistently outperforms KNN and Random
Revised: Time-Frequency Consistency, TF-C, into the model (Sec. 3). Further, we clarify that the term ‘self-supervised’ has different meanings in DA and in pretraining [S23, S24, S25, S26].

Original: Init. (which are not pre-trained) by a large margin of 42.8% and 25.1% (both in F1 score) on average. [Table]
Revised: The ‘self-supervised domain adaptation’ [S27, S16, S21, S15] or ‘unsupervised domain adaptation’ [S1, S22, S28, S11, S14] means that there are no labels in the target dataset, however that still requires labels in the pre-training dataset.

Original: [Figure]
Revised: In contrast, ‘self-supervised pretraining’

Original: [Table] Ablation study. We wanted to evaluate how relevant model component are for effective pre-training.
Revised: [S29, S30, S31] ( i.e. , the problem studied here, in line with a breadth of existing literature on pre-training) indicates the setting where no labels are available in pre-training.

Original: As shown in Appendix Table 7 (S LEEP E EG → G ESTURE ), removing L C , L T , and L F result inperformance degradation (precision) of 6.1%, 7.2%, and 6.7%, respectively.
Revised: Appendix B Detailed differences with CoST and BTSF

Original: To validate that theperformance increment is not solely brought by a third loss term no matter what consistency itmeasures, we replaced consistency loss L C with a loss term measuring the consistency within timespace (named L TT-C ) or within frequency space (named L FF-C ).
Revised: Up to the submission of this manuscript, there is no existing contrastive augmentations in time series’ frequency domain. There are two models, CoST [49] and BTSF [50], that involved frequency domain in contrastive learning, however, the proposed TF-C is fundamentally different with them in the following aspects. We take BTSF as an example while the differences also apply to CoST.

Original: Results show our consistency lossoutperforms L TT-C and L FF-C by 5.3% and 7.2% (accuracy), respectively. We also find that fullfine-tuning (optimizing the whole model) performs slightly better than partial fine-tuning (updatingonly final layers in R T and R F ), by 5.4% in F1 score.  Conclusion
Revised: • Problem definitions for both papers are different.

Original: We develop a pre-training approach that introduces time-frequency consistency (TF-C) as a mechanism to support knowledge transfer between time-series datasets.
Revised: Our method is designed to produce generalizable representations that can transfer to a different time series dataset (going from pre-training to a fine-tuning dataset) for the purpose of transfer learning.

Original: The approach uses self-supervisedcontrastive estimation and injects TF-C into a pre-training model, bringing the encoded time-basedand frequency-based representations and their local neighborhoods close together in the latent space. Experiments on eight datasets and four pre-training scenarios show the TF-C approach can outperformeight state-of-the-art baselines. Our findings have implications for building broadly generalizablepre-training models for time series. Limitations and future directions.
Revised: In contrast, BTSF attempts to learn embeddings within the same dataset for the purpose of representation learning.

Original: We posit that TF-C property  can  serve as a generalizable propertyfor pre-training on diverse time series datasets.
Revised: Our model captures the TF-C property invariant to different time series (in terms of various temporal dynamics, semantic meaning, etc.) and can thus serve as a vehicle for transfer learning. In contrast, BTSF learns embeddings invariant to perturbations (i

Original:  We are aware that there might exist additionalgeneralizable properties, such as temporal autoregressive processes, that could also prove useful forpre-training on  time series.
Revised: e., instance-level dropout) of the same time series. • The modeling of the frequency domain is different in both papers. We developed augmentations in the frequency domain based on the spectral properties of time series.

Original: Furthermore, our method expects theinput time series to be regularlysampled. However, our model can handle irregularly sampled time series using an encoder (such as
Revised: In contrast, although BTSF involves a frequency domain, its data transformation is solely implemented in the time domain (using instance-level dropout; Sec. 3.1 in BTSF). That is, the BTSF method applies the FFT after augmenting samples in the time domain which can lead to information loss.

Original: Raindrop [70] and SeFT [71]) that can embed irregular time series.
Revised: • BTSF emphasizes fusing temporal and spectral features to generate discriminative embeddings.

Original: For frequency encoder inputsx F i , possible alternatives include resampling or interpolation to obtain regularly sampled signals andusing regular or non-uniform FFT operations.
Revised: Unlike BTSF, our model leverages the consistency between time-based and frequency-based embeddings to produce generalizable time series representations. Our model maps every sample to a time-frequency embedding space and constrains the relative relationships between embeddings (through triplet loss) according to the TF-C property.

Original: Broader Impacts
Revised: The underlying consistency allows TF-C to realize transfer learning across time series datasets, which is TF-C’s unique advantage In summary, TF-C introduces frequency domain augmentations in the sense that it directly perturbs the frequency spectrum. TF-C is the first method that uses frequency domain augmentations to enable transfer learning in time series.

Original: Our proposed novel framework for time series unsupervised pre-training improves classificationperformance on target datasets in different application scenarios. The recognition of time-frequencyconsistency as a universal property specific to time series data  is a weak assumption that enableseffective, task- and domain-agnostic transfer learning.
Revised: Here, we provide an analogy, from images to time series, to aid understanding of the ‘invariance’ property of data representations. In computer vision, it is well known that an image and its augmented views obtained by simple transformations (e.g., rotation, translation, scaling, etc.) can be used in different frameworks such as transformation prediction and contrastive instance discrimination to obtain invariant representations.

Original: We believe our work will inspire the researchcommunity to uncover other universal properties for transfer learning.
Revised: These transformations are used as augmentations to guide the learning of self-supervised representations. For transformation prediction, the model learns representations that are equivariant to the selected transformation as the information embedded in the transformation needs to be embedded in the representation for the final layer to solve the pretext task. For contrastive instance discrimination, the representations are sensitive to the instances while learning invariances to transformations or views. The intuition behind such transformations is that the underlying object in the image is the same no matter how the image is rotated, translated, re-scaled, etc.

Original: We also hope our work willalso attract more researchers to themore general problem of time series representation learning whichis still underappreciated relative to problems from CV and NLP fields.
Revised: In other words, the information carried by the original image and the transformed image is the same. Similarly, the time domain and frequency domain representations carry the same information in time series. Thus it is reasonable to expect that by exploring local neighborhoods in the time domain and frequency domain and enforcing consistency of feature representations inter- and intra- domains, invariance properties are captured in our model via self-supervised learning.

Original: On the society level, our work, along the line of transfer learning, can facilitate more efficient use oftime series data in various settings.
Revised: Thus, information carried in time and frequency domains of the same or similar time series sample should be the same. This invariance, formalized as ‘Time-Frequency Consistency’, is helpful for self-supervised pre-training. We will include the above discussion in the camera-ready version.

Original: For example, in medical settings, some diseases of clinical interestmay have very small labelled dataset.
Revised: Appendix D Additional information on datasets and pre-training evaluation D.1 Datasets

Original: In this case, unlabelled data from patients of different diseasesbut with similar underlying physiological conditions can be used to pre-train the model. 
Revised: We use eight diverse time series datasets to evaluate our model. The datasets used in one-to-one and one-to-many pre-training evaluations are the same. The dataset statistics are shown in Table 3. Processed model-ready datasets are in our GitHub Repository ( https://anonymous.4open.

Original: However,practitioners need to be aware of the limitations of the model, including that it may make biasedpredictions.
Revised: science/r/TFC-pre-training-6B07 ).

Original: Specifically, bias may exist in the source dataset used for pre-training due to an imbalanceof samples from subjects of different demographic attributes.
Revised: Following is a detailed description of datasets. S LEEP E EG [61]. The dataset contains 153 whole-night sleeping electroencephalography (EEG) recordings produced by a sleep cassette. Data are collected from 82 healthy subjects.

Original: Also, the standardized medical protocolsfor collecting these datasets might be unsuitable for subjects with certain physiological attributes,creating unforseen bias that may be transferred to the finetuning phase.
Revised: The 1-lead

Original: All datasets in this paper are publicly available and are not associated with any privacy or securityconcern.
Revised: EEG signal is sampled at 100 Hz. We segment the EEG signals into segments (window size is 200) without overlapping, and each segment forms a sample. Every sample is associated with one of the five sleeping patterns/stages: Wake (W), Non-rapid eye movement (N1, N2, N3), and

Original: Furthermore, we have followed guidelines on responsible use specified by primary authorsof the datasets used in the current work.
Revised: Rapid Eye Movement (REM). After segmentation, we have 371,055 EEG samples. The raw dataset ( https://www.physionet.org/content/sleep-edfx/1.0.0/ ) is distributed under the Open Data Commons Attribution License v1.0.

Original: Checklist
Revised: E PILEPSY [62]. The dataset contains single-channel EEG measurements from 500 subjects.

Original: For all authors...(
Revised: For every subject, the brain activity was recorded for 23.6 seconds.

Original: a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: The dataset was then divided and shuffled (to mitigate sample-subject association) into 11,500 samples of 1 second each, sampled at 178 Hz. The raw dataset features five classification labels corresponding to different states of subjects or measurement locations — eyes open, eyes closed, EEG measured in the healthy brain region, EEG measured in the tumor region, and whether the subject has a seizure episode.

Original: [Yes] In abstract and introduction, we claim that TF-C is
Revised: To emphasize the distinction between positive and negative samples in terms of epilepsy, We merge the first four classes into one, and each time series sample has a binary label describing if the [Table]

Original: a generalizable property of time series that can support pre-training, which is well-
Revised: associated subject is experiencing a seizure or not. There are 11,500 EEG samples in total.

Original: justified in Sec. 3 and experimentally demonstrated in Sec. 5 (our model consistently
Revised: To evaluate the performance of a pre-trained model on a small fine-tuning dataset, we choose a small set (60 samples; 30 samples for each class) for fine-tuning and assess the model with a validation set (20 samples; 10 samples for each class).

Original: performs comparatively to or above baseline methods (b) Did you describe the limitations of your work?
Revised: Finally, the model with the best validation performance is used to make predictions on the test set ( i.e. , the remaining 11,420 samples). Statistics of finetuning, validation

Original: [Yes] See Section 6.
Revised: [Table] and test sets in the other three target datasets are in Appendix 3.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: The raw dataset ( https://repositori.upf.edu/handle/10230/42894 ) is distributed under the Creative Commons License (CC-BY) 4.0.

Original: [Yes] See
Revised: F D - A and F D - B [63].

Original: Broader Impact on Page 10.
Revised: The dataset is generated by an electromechanical drive system that monitors the condition of rolling bearings and detects their failures. Four subsets of data are collected under various conditions, whose parameters include rotational speed, load torque, and radial force. Each rolling bearing can be undamaged, inner damaged, and outer damaged, which leads to three classes in total.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to 
Revised: We denote the subsets corresponding to condition A and condition B as Faulty Detection Condition A ( F D - A ) and Faulty Detection Condition B ( F D - B ), respectively. Each original recording has a single channel with a sampling frequency of 64k

Original: them?
Revised: Hz and lasts 4 seconds. To deal with lengthy recordings, we follow the procedure described by Eldele et al. Specifically, we use a sliding window length of 5,120 observations and a shifting length of 1,024 or 4,096 to ensure that samples are relatively balanced between classes. The raw dataset ( https://mb.uni-paderborn.de/en/kat/main-research/ datacenter/bearing-datacenter/data-sets-and-download ) is distributed under the Creative Commons Attribution-Non Commercial 4.0 International License.

Original: [Yes]
Revised: H AR [64].

Original: If you are including theoretical results...
Revised: This dataset contains recordings of 30 health volunteers performing daily activities, including walking, walking upstairs, walking downstairs, sitting, standing, and lying. Prediction labels are the six activities. The wearable sensors on a smartphone measure triaxial linear acceleration and triaxial angular velocity at 50 Hz. After preprocessing and isolating gravitational acceleration from body acceleration, there are nine channels ( i.e. , 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer) in total.

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: The raw dataset ( https://archive.ics.uci.edu/ml/ datasets/Human+Activity+Recognition+Using+Smartphones ) is distributed as-is. Any commercial use is not allowed.

Original: [N/A]
Revised: G ESTURE [65].

Original: (b) Did you include complete proofs of all theoretical results?
Revised: The dataset contains accelerometer

Original: [N/A]
Revised: [Table]

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: and fine-tuning datasets) ; otherwise, if the model ( i.e. , KNN, TS-SD, Mixing-up, and TS2vec) don’t have channel generalization ability, we only use three acceleration channels in H AR for pre-training.

Original: mental results (either in the supplemental material or as a URL)?
Revised: The raw dataset is accessible through http://www.timeseriesclassification.com/ description.php?Dataset=UWaveGestureLibrary .

Original: [Yes] Yes, we include an anonymous link (see Abstract) that provides the source codes with all implementa-
Revised: While the distribution license is not explicitly mentioned, the dataset is a public resource based on [65]. E CG [26].

Original: tion details, implementation of baselines, and eight datasets.
Revised: This is the 2017 PhysioNet Challenge focusing on classifying ECG recordings.

Original: The link will be updated
Revised: The singlelead ECG measures four different underlying conditions of cardiac arrhythmias. More specifically, these classes correspond to the recordings of normal sinus rhythm, atrial fibrillation (AF), alternative rhythm, or others (too noisy to be classified). The recordings are sampled at 300 Hz. Furthermore, the dataset is imbalanced, with much fewer samples from the atrial fibrillation and noisy classes out of all four.

Original:  to an non-anonymous link after acceptance.
Revised: To preprocess the dataset, we use the code from the CLOCS paper, which applied a fixed-length window of 1,500 observations to divide long recordings into short samples of 5 seconds that are physiologically meaningful. Because of the imbalanced dataset, we report AUROC and AUPRC (insensitive to label distribution) in our results. The raw dataset ( https://physionet.

Original: (b) Did you specify all the training details (e
Revised: org/content/challenge-2017/1.0.0/ ) is distributed under the Open Data Commons Attribution License v1.0. E MG [66]. Electromyograms (EMGs) measure muscle responses as electrical activity in response to neural stimulation, and they can be used to diagnose certain muscular dystrophies and neuropathies.

Original: g., data splits, hyper-parameters,
Revised: The dataset consists of single-channel EMG recording from the tibialis anterior muscle of three healthy volunteers suffering from neuropathy and myopathy. The recordings are sampled at a frequency of 4K Hz. Each patient ( i.e. , disorder) is a classification category. Then the recordings are split into time-series samples using a fixed-length window of 1,500 observations.

Original:  how they
Revised: The raw dataset ( https://physionet.org/content/emgdb/1.0.0/ ) is distributed under the Open Data Commons Attribution License v1.0. D.2 Varying lengths and dimensionality of time series in pre-training vs. target datasets We use established strategies to address the challenging question of how to process time series with the variable number of dimensions (channels) and measurements (note that this study focuses on designing a pre-training model that can transfer knowledge across disparate time-series datasets).

Original: were chosen)?
Revised: When the dimensionality or length of time series is different between pre-training and target datasets, we view the pre-training dataset as an anchor according to which we adjust (pre-process) the target dataset. For length adjustments, we use zero padding to increase the number of observations or use downsampling to reduce the number of observations. For dimensionality adjustments, our model applies to time series at the level of a single lead, enabling the ability to generalize to multivariate time series when we make different channels share the model ( i.e. , time encoder, frequency encoder, and two projectors) parameters. In this way, we have a z TUNE i for each channel of the time series (Sec. 4) and concatenate the learned representations across all channels to form the final representation of the input time series sample.

Original: [Yes] See implementation detailsin Sec. 5.
Revised: [Table] [Table] Appendix E TF-C and baseline architectures and implementation details

Original: See Appendix E for baselinearchitectures and hyper-parameter settings.
Revised: We compare our TF-C model against eight state-of-the-art baselines and two additional methods (Sec. 5): (1) directly fit a KNN (K=2) classifier with the target (fine-tuning) dataset; (2) randomly initialize a model (the model structure and experimental settings are the same with our TF-C model), ignore pre-training, and directly train from scratch with the target dataset. We implement the baselines follow the corresponding papers including TS-SD [12], TS2vec [47], CLOCS [41], Mixing-up [18], TS-TCC [48], SimCLR [40], TNC [46] [Table]

Original: More details can be found in the included
Revised: and CPC [30]. We use default settings for hyper-parameters as reported in the original works unless noted below. All pre-training and fine-tuning with baselines are done with a single Tesla V100 GPU with 32 Gb of allocated memory provided by Harvard Medical School’s O2 High Performance Computing platform. TF-C (our model)

Revised: Our time-based contrastive encoder F T adopts 1-D ResNet backbone similar to

Original: URL.
Revised: SimCLR.

Revised: Specifically, after hyper-parameter tuning, the encoder consists of three layers convolutional blocks: the kernel sizes in all layers are 8; the strides are 8, 1, and 1, respectively; the the depths are 32, 64, and 128, respectively.

Revised: We use max pooling after each convolutional layer and all the pooling kernel sizes and strides are set as 2.

Original: (c) Did you report error bars (e
Revised: The cross-space projector R T contains two fully-connected layers with hidden dimensions 256 and 128, respectively. In the transformation from time space to frequency space, we use the full spectrum (symmetrical) thus x T and x F have the same dimension. For simplify, the frequency encoder F F and projector R F have same structure (but different parameters) with their counterparts in time space. Our preliminary experiments show that the change of model structure has relatively small change on performance.

Revised: For example, replacing F T by 2-layer LSTM or 3-layer

Revised: Transformer only cause a slight drop of F1 score ( ± 1.2%; one-to-one setting; Scenario 1).

Revised: In pre-training, we use Adam optimizer with learning rate of 0.0003 and 2-norm penalty coefficient of 0.0005.

Revised: We use batch size of 64 and training epoch of 40.

Revised: We record the best performance in terms of F1 score and save the parameters in the corresponding epoch as Θ .

Revised: In fine-tuning, we initialize the model parameters with Θ and optimize the model on fine-tuning set.

Revised: The hyper-parameters are the same as in pre-training.

Revised: For the classification task in fine-tuning stage, we adopt a 2-layer fully-connected layer as a classifier.

Revised: The hidden dimensions are 64 and the number classes in target dataset, respectively.

Revised: We measure the classification loss via the cross entropy function and the loss is jointly optimized with the fine-tuning loss ( i.e. , L TF-C ).

Revised: TS-SD [12] uses a modified attention mechanism to encode latent features and use denoising or DTWsimilarity prediction as pretext-tasks.

Revised: The encoder network is based on self-attention mechanism [68] but instead of the linear layer that produce the K, Q, V matrices from the input sequence, TS-SD uses a single convolutional layer to replace the linear layer.

Revised: With multihead attention, TS-SD sets a different kernel size for each head to attempt capture short- and long-range temporal patterns.

Revised: Although we found that 12 heads was relatively redundant and may be replaced by fewer heads, we still use 12 heads to keep the same with the original work in [12].

Revised: For self-supervised pre-training, we adopt the denoising pretext task that attempts to minimize the mean square distance between the original input time series and the output of the encoder, for which the input is an augmented time series sample with noise added to a subseries.

Revised: To cope with small numerical values, we use a learning rate of 3e-7 in pre-training and 3e-4 during fine-tuning.

Revised: While training on pre-training dataset for the four scenarios, the batch sizes varied from 4 to 128 and were manually chosen to achieve reasonable performance for each scenario.

Revised: During fine-tuning, we use # epochs ranging from 20 to 80 and a batch size of 16.

Revised: Although the source code of TS-SD is not released, we implement the TS-SD model to our best understanding and tune the model to achieve the best performance as a strong baseline.

Revised: We public the implemented TS-SD in our online repository.

Revised: TS2vec [47] introduces the notion of contextual consistency and uses a hierarchical loss function to capture long-range structure in time series.

Revised: TS2vec is a powerful representative learning method and have a specially-designed architecture.

Revised: The encoder network consists of three components.

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: First, the input time series is augmented by selecting overlapping subseries. They are projected into a higher dimensional latent space. Then, latent vectors for input time series are masked at randomly chosen positions. Finally, a dilated CNN with residual blocks produce the contextual representations.

Revised: To compute the loss, the representations are gradually pooled along the time dimension and at each step a loss function based on contextual consistency is applied.

Revised: For the baseline experiment, we found that the original 10 layers of ResNet blocks is redundant, and we reduce residual blocks in the encoder from 10-layer to 2-layer without compromising model performances.

Revised: To make the model comparable with other baselines, we also restrict the hidden dimension to 1 and use a batch size of 64, except in the second scenario we reduce it to 16 given the length of samples in the F D - A / F D - B datasets.

Revised: CLOCS [41] assumes that samples from different subjects are negative pairs in contrastive learning and applies it to classify ECG signals.

Revised: CLOCS makes the basic assumption that time series samples from different patients (the application scenario is in learning cardiac signals) should have dissimilar representations.

Revised: Based on that, they devised three slightly different architectures - CMSC, CMLC, and CMSMLC, where ECG recordings from non-overlapping temporal segments, different ECG leads, and both of these are treated as positive pairs in case if they come from the same patients.

Revised: For this model, the encoder consists of alternating convolutional and pooling layers.

Revised: So given the different input lengths, the hyper-parameters for these layers have to be modified accordingly.

Revised: For example, we use kernel sizes of 7 , 7 , 7 , stride of 3 , 3 , 3 , pooling layer size of 2 for the last scenario of E CG to E MG but increased stride to 4 and increased representation size for a time series sample from 64 to 128 in case of F D - A to F D - B due to much longer time series samples.

Revised: Mixing-up [18] proposes new mixing-up augmentation and pretext tasks that aim to correctly predict the mixing proportion of two time series samples.

Revised: In Mixing-up, the augmentation is chosen as the convex combination of two randomly drawn time series from the dataset, where the mixing parameter is random drawn from a beta distribution.

Revised: The contrastive loss is then computed between the two inputs and the augmented time series.

Revised: The loss is a minor modification of NT-Xent loss and is designed to encourage the correct prediction of the amount of mixing.

Revised: We use the same beta distribution as reported in the original Mixing-up model.

Revised: TS-TCC [48] leverages contextual information with a transformer-based autoregressive model and ensures transferability by using both strong and weak augmentations.

Revised: TS-TCC proposed a challenging pretext task.

Revised: An input time series sample is first augmented by adding noise, scaling, and permuting time series.

Revised: The views are then passed through an encoder consisting of three convolutional layers before processed by the temporal contrasting module.

Revised: During temporal contrasting, for each view, a transformer architecture is used to learn a contextual representation.

Revised: The learned representation is then used to predict latent observation of the other augmented view at a future time.

Revised: The contextual representations are then projected and maximized similarity using NT-Xent loss.

Revised: For this baseline, we mostly adopted the hyper-parameters presented in the original paper.

Revised: We use a learning rate of 3e-4 throughout, and a batch size of 128 during pre-training and 16 during fine-tuning.

Revised: SimCLR [40] is a state-of-the-art model in self-supervised representation learning of images.

Revised: It utilizes deep learning architectures to generate augmentation-based embeddings and optimize the model parameters by minimizing NT-Xent loss in the embedding space.

Original: [Yes] We run experiments for 5 times and report the averagevalue with standard deviation.
Revised: [40] applies the original SimCLR model for time series data. In this work, we compare with the modified SimCLR as in [40]. The SimCLR contrastive learning framework consists of four major components.

Original: See Table 1, Table
Revised: Although initially proposed for image data, it is readily adapted to time series, as shown in [40]. An input time series sample is first stochastically augmented into two related views. Then a base encoder extracts representation vectors. ResNet is used as encoder backbone for simplicity.

Revised: Then the projection head transforms representations into a latent space where the NT-Xent loss is applied.

Original:  and Appendix Tables 4-6.(d)
Revised: For time series, SimCLR investigated different augmentations, including adding noise, scaling, rotation, negation, flipping in time, permutation of subseries, time warping, and channel shuffling. All the unmentioned hyper-parameters are kept the same with the original model. TNC [46] focuses on learning representations that encode the underlying state of a non-stationary time series.

Revised: This is achieved by ensuring that the distribution of observations in the latent space is different from the distribution of temporally separated observations.

Revised: For each time point, the

Revised: TNC model calculates a neighborhood size using the Augmented Dickey–Fuller test.

Revised: The encoded representations of two different windows will be passed to a discriminator to predict the probability that they belong to the same temporal neighborhood.

Revised: We implement the TFC model following the public GitHub repository provided by the authors.

Revised: We observed that the ADF tests are extremely slow, so TNC is only used as a baseline in scenario 1 in one-to-one setting (classification task).

Revised: CPC [30] aims to learn the representations that encode the global information shared across different segments of a time series signal by the strategy of predictive coding.

Revised: Instead of modelling the conditional distribution at future time points for prediction, the authors choose to encode the context and observation into a vector representation in a way that maximally preserves their mutual information.

Revised: For the model architecture, a nonlinear encoder first maps the sequence of observations to latent representations.

Revised: Then an autoregressive model summarizes past latent representations to produce a context representation at every time point.

Revised: The two components are trained to jointly optimize a loss function which is inspired by noise contrastive estimation.

Revised: We used the implementation provided by the authors of the TNC paper [46] and kept their default choices of parameters whenever reasonable.

Revised: The TS-SD is the only method, to our knowledge, that explicitly aims at time series pre-training but the underlying assumption that can support knowledge transfer across different time series datasets is not clearly provided.

Revised: The TS2vec, CLOCS, Mixing-up, TS-TCC, TNC, and CPC are designed for representation learning (instead of pre-training) but involved transfer learning setting in their experiments.

Revised: We modify them to perform pre-training task and make them comparable to our model.

Revised: The SimCLR is a widely use baseline in computer vision and simply adopted to deal with time series by [40].

Revised: We compare our model with SimCLR to further show our superiority.

Revised: Appendix F Additional results on one-to-one evaluation

Revised: In one-to-one pre-training evaluation, we consider four different scenarios with paired pre-training and fine-tuning datasets from different fields (Sec. 5.1).

Original: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: In each scenario, the pre-training and fine-tuning datasets share same semantic meanings ( e.g. , both EEG signals; Scenario 1) or similar semantic meanings ( e.g. , ECG and EMG are both physiological signals; Scenario 4). For example, in the first scenario, S LEEP E EG and E PILEPSY both involve EEG measurements but the recordings are taken under different physical conditions. For each scenario, we first train each baseline model using the pre-training dataset and record the converged parameters. The parameters are then taken to initialize the model on the fine-tuning phase which uses the target dataset.

Original: [Yes] See Appendix E. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: In fine-tuning, we allowed all parameters in the model to be optimized (no freezing of earlier layers; full fine-tuning).

Original: (a) If your work uses existing assets, did you cite the creators? [Yes] We used eightexisting datasets and 6 state-of-the-art baselines in contrastive learning and pre-trainingfor time series. We cited the creators for every exist asset we used.
Revised: Our preliminary results show that the performance of full fine-tuning is slightly better than partial fine-tuning by 5.4% in F1 score (one-to-one setting;

Original: See Sec. 5.
Revised: Scenario 1).

Original: [Table] (b) Did you mention the license of the assets? [Yes] All dataset licenses are mentioned inthe Appendix E.(c) Did you include any new assets either in the supplemental material or as a URL?
Revised: After fine-tuning, we evaluate the final model on the test set and report metrics including accuracy, precision (macro-averaged), recall, F1-score, AUROC (one-versus-rest), and AUPRC.

Original: [Yes]See the anonymous link in Abstract. [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [No] All data we use is freely available for download, without anyrequirement to re-contact the data curator.
Revised: The evaluation results of four scenarios are shown in Table 1 (main paper) and Tables 4-6.

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table] 
Revised: [Table] [Table]

Original: information or offensive content? [No] Our datasets are public, well-established, and 
Revised: Appendix G Results on downstream clustering and anomaly detection tasks

Original: [Table]
Revised: Beyond the common downstream classification tasks, we conducted extensive experiments to examine the proposed TF-C in diverse downstream tasks.

Original: do not contain PII or offensive content
Revised: The comparison results with state-of-the-art baselines are shown in Table 7 for clustering and in Table 8 for anomaly detection.

Original: [Table] If you used crowdsourcing or conducted research with human subjects...
Revised: The experimental setups and results analysis are shown in Section 5. Appendix H Further results on ablation study We conduct ablation studies (Sec. 5) to evaluate the importance of every component in the developed

Revised: TF-C model.

Revised: For the example of one-to-one setting ( S LEEP E EG → G ESTURE ) when pre-training the model using S LEEP E EG dataset and fine-tuning on G ESTURE dataset, ablation study results are shown in Table 9.

Revised: Through comparison, we observe that the full TF-C model achieves the highest performance in every evaluation metrics, indicating every component, especially the novel consistency loss L C ), contributes to the model’s performance.

Revised: Appendix I Visualization of embeddings in time-frequency space

Revised: To better demonstrate the effectiveness of the developed consistency loss L C , we visualize the learned embeddings in time-frequency space.

Revised: In one-to-one setting, we pre-train the TF-C model on

Revised: S LEEP E EG and fine-tune on E PILEPSY (Sec. 5.1).

Revised: We visualize the testing samples of E PILEPSY in fine-tuning stage.

Revised: The learned time-based embeddings z T i and frequency-based embedding z F i have 128 dimensions.

Revised: We map the embeddings to a 2-dimensional space through UMAP (20 neighbors; minimum distance as 0.2; cosine distance) for visualization.

Revised: In this work, we develop a novel consistency loss L C to induce the model learn closer time-based and frequency-based embeddings in a time-frequency space (Sec. 4.3).

Revised: As our problem setting is that the fine-tuning dataset has small scale ( M ≪ N ; Sec. 3), we randomly select 100 samples from the testing set ( E PILEPSY ) for visualization.

Revised: The visualization of the learned embeddings in time-frequency space is shown in Figure 3 (without L C v.s. with L C ).

Revised: We can observe that: (1) when without L C (panel a), the time-based and frequency-based embeddings are clustered separately; the embeddings are largely clustered together when having L C (panel b).

Revised: The observation demonstrates that the proposed L C has the ability to push time-based and frequency-based embeddings closer to each other.

Revised: (2) for a specific sample x i (marked as dashed line), we can find that its time-based

Original: [Table] 
Revised: [Table] [Table]

Revised: embedding (colored as light blue) and frequency-based (colored as dark blue) embedding are closer to each other in panel (b) than in panel (a).

Revised: In particular, the cosine distance, between z T i and z F i of this sample, has reduced from 0.89 to 0.71 when using L C in our model.

Revised: (3) the frequency-based embeddings of ‘epilepsy’ and ‘health’ samples are largely overlapped when without L C but clustered well when considering L C , indicating the learned embeddings (with L C ) can potentially enhance downstream classification performance.

Revised: Furthermore, we attempt to quantify the impact of our consistency loss item on the quality of learned embeddings, using the one-to-many setting as shown in Figure 4.

Revised: We pretrain the TF-C model on S LEEP E EG dataset and fine-tuning on E PILEPSY , F D -B , G ESTURE , and E MG , respectively.

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: Next, we explain how the cosine distance is calculated taking E MG dataset as an example. For [Figure] D i m e n s i o n

Revised: Epilepsy

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: D i m e n s i o n

Original: [N/A](b)
Revised: [Figure] a b [Figure] [Figure]

Original: Did you describe any potential participant risks, with links to Institutional Review
Revised: distance averaged across all testing samples We observe that the cosine distance is larger when removing our proposed consistency loss item L C indicating our model indeed pushes the time-based and frequency-based embeddings closer to each other.

Revised: Epilepsy

Revised: FD-B

Revised: Gesture

Revised: EMG

Revised: [Figure]

Revised: [Figure]

Revised: I n t e r - c l a ss C o s i n e d i s t a n c e every sample x i in E MG , we measure the cosine distance between its time-based embedding z T i and frequency-based embedding z F i in the 128-dimension time-frequency space.

Revised: Then, we take the average of the distance across all testing samples and report in Figure 4.

Revised: The number of testing samples of each dataset can be found in Table 3.

Revised: We find that the cosine distance is larger when removing the proposed consistency loss item L C , which means our model, as designed and expected, indeed pushes the time-based and frequency-based embeddings closer to each other.

Revised: For example, in

Revised: E PILEPSY , the averaged cosine distance dropped 22.2% (from 1.32 to 1.08) when using consistency loss (Sec. 4.3).

Revised: At the same time, the embeddings of samples drawn from different classes are more easily distinguished from each other, as shown in Figure 5: we observe that our TF-C model (with consistency loss L T increase the inter-class cosine distance from 0.88 to 1.42 with a remarkable margin of 61.4% (one-to-many setting; testing set of E PILEPSY ), indicating our model indeed has the ability to increase the distinctiveness of learned embeddings and bring forward better classification performance on the fine-tuning dataset of interest.

Revised: The results of visualization increase our confidence that the developed TF-C pre-training model can capture generalizable properties and boost knowledge transfer across different time series datasets.

Revised: Appendix J Additional information on domain augmentations

Revised: To the best of our knowledge, this is the first work that direct perturbs the frequency domain of time series in contrastive learning.

Revised: Here, we thoroughly explore various frequency augmentations and discuss the design of manipulation strategy in frequency domain.

Revised: In particular, we analyze eight frequency augmentation policies and discuss their performance.

Revised: We consider three types of frequency augmentations:

Revised: • Low- vs. High-band perturbations : High-frequency components contribute to the fast varying parts of the time series (e

Revised: g., sharp transitions).

Revised: In contrast, low-frequency components contribute to slow signal variations in the time domain.

Revised: Low band perturbations correspond to the first half of the frequency spectrum.

Original:  [Table]
Revised: [Figure] [Table]

Revised: • Single- vs. Multi-component perturbations (aka varying budget size): Perturbing several frequency components (i.e., larger budget) leads to more substantial changes in the time domain.

Revised: In this analysis, we set the budget to 5 for multi-component perturbations and 1 for single-component perturbations.

Revised: • Random vs. Distributional perturbations : The selection of components in the spectrum that get perturbed is important because it influences the resulting time-based representation of the augmented sample.

Original: Board (IRB) approvals, if applicable?
Revised: Frequency components can be sampled uniformly at random or according to a user-specified distribution (i [Table] e., Gaussian in the following analysis). Based on the above considerations we developed eight frequency augmentations.

Original: [N/A]
Revised: In Fig. 6 (E PILEPSY dataset)) we visualize a series of perturbations in the frequency domain and their effects in the time domain (through the inverse FFT). In addition

Original: [Table]
Revised: [Table]

Revised: we explore how different spectral perturbations affect the performance of TF-C. Tableshows results for the one-to-one setup (S LEEP E EG → E PILEPSY ).

Revised: We find the following:

Revised: • Perturbing low-band components performs better than augmenting high-band elements of the spectrum (0.8% change in F1 performance on average across the four setups).

Revised: This can be explained by the nature of the underlying time series: in physiological signals, most information is carried in a low band (e.g., EEG signals are most informative in 0.5Hz while > 70Hz is considered noise).

Revised: This means that the choice of low- vs. highband components depends on the signal context.

Revised: For example, high-band components are informative for mechanical signals.

Revised: We further verified this by external experiments in the F D - A → F D - B setting, where perturbing high-band components outperform low-band augmentations by 0.8% in F1.

Revised: Thus, we consider perturbations across the entire spectral band in the paper to achieve broad generalization.

Revised: • In general, perturbing many frequency components degrades model performance.

Revised: One reason for worse performance is that too big a budget incurs too significant changes in the time-based representation of augmented samples.

Revised: In that case, the augmented samples would be easily distinguished by a contrastive model, leading to poor contrastive encoders.

Revised: For this reason, we use single-component perturbations in the paper.

Revised: • We cannot draw a clear conclusion on whether the random selection of components is better than Gaussian distribution-based as both F1 scores are very similar.

Original: (c) Did you include the estimated hourly wage paid to participants and  the total amount
Revised: The optimal choice of sampling strategy likely depends on both the time-series signal and prediction task. We use random sampling in the paper. Appendix K Discussion on many-to-one setting We probed the one-to-one setting and one-to-many setting in knowledge transfer of time series (Sec. 5).

Revised: Here, we discuss the many-to-one set up: pre-train on a mixture of multiple datasets and fine-tune on a single pure dataset.

Revised: We notice that the many-to-one setup fundamentally differs from the one-to-one and also from oneto-many configurations.

Revised: Notably, in the one-to-one and one-to-many setups, the model is pre-trained on a single pre-training dataset presumed to originate from a data source exhibiting homogeneity.

Original: [Table]
Revised: For example, samples in the pre-training dataset could describe similar devices (e.g., different kinds of boring machines) with similar sampling rates, lengths, semantic meanings, and/or relevant temporal patterns that occur on comparable time scales, etc. When some level of homogeneity exists in a pre-training dataset, the method can learn patterns and apply them to different fine-tuning datasets. In contrast, in many-to-one setups, pre-training datasets can be incredibly heterogeneous. For example, a pre-training dataset might contain samples describing boring machines’ behavior and recordings of EEG medical devices.

Original: spent on participant compensation?
Revised: For this reason, pre-training can be severely hampered by diversity and misalignments across samples in the pre-training dataset. When we merge many heterogeneous samples into a single pre-training dataset, the model might struggle to find patterns underlying the combined dataset. While models in computer vision have been successfully pre-trained on large, diverse datasets, such as ImageNet, the literature on pre-training for time series is much sparser. Further, heterogeneous pre-training datasets pose additional challenges due to the unique properties of time series.

Revised: Learning universal representations from multi-sourced time series datasets (i.e., many-to-one setups) remains an open challenge.

Original: [N/A]
Revised: It is an interesting future direction, and we hope this study can facilitate future research. We have explored this and provide preliminary experimental results in four many-to-one setups (named N-to-one for N = 1,2, 3, and 4). Take four-to-one as an example. We merge four pre-training datasets (S LEEP E EG ,F D - A , H AR , and E CG ) into one pre-training dataset.

Revised: We had to preprocess the datasets to make a many-to-one setup feasible.

Revised: To address the mismatch between the number of channels, for the multivariate dataset (H AR ), we take a single channel (i.e., x-axis acceleration) to make sure all samples have the same number of channels.

Revised: To align the length across datasets, we make all samples have 1,500 observations by zero-padding (for the ones shorter than 1,500) and cutting-off (for the ones longer than 1,500).

Original: [Table]
Revised: Further, we take 25% samples (instead of 100%; for computational reasons only) from each of the four datasets to create a pre-training dataset with 108,300 samples that belong to 18 classes in total. Similarly, we built pre-training for three-to-one (merge S LEEP E EG ,F D - A , and H AR ) and two-to-one (merge S LEEP E EG and F D - A ). We take the above N-to-one pre-training datasets and repeat the following analysis. First, we pre-train our TF-C and then fine-tune it on E PILEPSY for classification.

Revised: The AUROC for one-to-one, twoto-one, three-to-one, and four-to-one settings are 0.9819, 0.9525, 0.8607, and 0.7253, respectively.

Revised: We observe AUROC decreases with the increasing heterogeneity of pre-training datasets, which is consistent with our analysis in the first part of this answer.

Revised: Nevertheless, whether there exists a framework that overcomes the problems and enables many-to-one pre-training for time series remains to be seen, and we believe it’s an exciting direction for future research.