Original: Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency
Revised: Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency

Original: Abstract
Revised: Abstract

Original: Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short cyclic effects, which can lead to poor downstream performance.
Revised: Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short-cyclic effects, which can lead to poor downstream performance.

Original: While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training.
Revised: While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training.

Original: To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training.
Revised: To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training.

Original: Relative to other modalities, in time series, we expect that time-based and frequencybased representations of the same example are located close together in the timefrequency space.
Revised: Relative to other modalities, in time series, we expect that time-based and frequencybased representations of the same example are located close together in the timefrequency space.

Original: To this end, we posit that time-frequency consistency (TF-C) — embedding a time-based neighborhood of a particular example close to its frequency-based neighborhood and back — is desirable for pre-training.
Revised: To this end, we posit that time-frequency consistency (TF-C) — embedding a time-based neighborhood of a particular example close to its frequency-based neighborhood and back — is desirable for pre-training.

Original: Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation.
Revised: Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation.

Original: We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring.
Revised: We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring.

Original: Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F score) on average in one-to-one settings ( e.g. , fine-tuning an EEG-pretrained model on EMG data) and byup to 8.4% (F1 score) in challenging one-to-many settings ( e.g. , fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction ), reflecting the breadth of scenarios that arise in real-world applications.
Revised: Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1 score) on average in one-to-one settings ( e.g. , fine-tuning an EEG-pretrained model on EMG data) and by 8.4% (precision) in challenging one-to-many settings ( e.g. , fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real-world applications.

Original: The source code and datasets are available at https: //github.com/
Revised: The source code and datasets are available at https: //github.com/

Original: mims-harvard/TFC-pretraining .
Revised: mims-harvard/TFC-pretraining .

Original:  Introduction
Revised:  Introduction

Original: Time series plays important roles in many areas, including clinical diagnosis, traffic analysis, and climate science [1, 2, 3, 4, 5, 6].
Revised: Time series plays important roles in many areas, including clinical diagnosis, traffic analysis, and climate science [1, 2, 3, 4, 5, 6].

Original: While representation learning has considerably advanced analysis of time series [7, 8, 9] and sequences more broadly [10], learning broadly generalizable representations for time series remains a fundamentally challenging problem [8, 11].
Revised: While representation learning has considerably advanced analysis of time series [7, 8, 9] and sequences more broadly [10], learning broadly generalizable representations for time series remains a fundamentally challenging problem [8, 11].

Original: There are numerous immediate
Revised: There are numerous immediate

Original: [Figure]
Revised: [Figure]

Original: benefits from being able to generate such representations, of which pre-training capability is particularly desirable and of great practical importance [12, 13].
Revised: benefits from being able to generate such representations, of which pre-training capability is particularly desirable and of great practical importance [12, 13].

Original: Central to pre-training is a question of how to process time series in a diverse dataset so as to greatly improve generalization on new time series coming from different datasets [14, 15, 10].
Revised: Central to pre-training is a question of how to process time series in a diverse dataset so as to greatly improve generalization on new time series coming from different datasets [14, 15, 10].

Original: By training a neural network model on a diversified dataset and transferring it to a new target dataset for fine-tuning, i.e. , without explicit retraining on that target data, we expect the resulting performance be at least as good as that of state-of-the-art models tailored to the target dataset.
Revised: By training a neural network model on a diversified dataset and transferring it to a new target dataset for fine-tuning, i.e. , without explicit retraining on that target data, we expect the resulting performance be at least as good as that of state-of-the-art models tailored to the target dataset.

Original: However, unfortunately, the expected performance gains are often not realized for a variety of reasons ( e.g. , distribution shifts, properties of the target dataset unknown during pre-training) [16, 17] that get compounded by the complexity of time series: large variations of temporal dynamics across datasets, varying semantic meaning, irregular sampling, system factors ( e.g. , different devices or subjects), etc.
Revised: However, unfortunately, the expected performance gains are often not realized for a variety of reasons ( e.g. , distribution shifts, properties of the target dataset unknown during pre-training) [16, 17] that get compounded by the complexity of time series: large variations of temporal dynamics across datasets, varying semantic meaning, irregular sampling, system factors ( e.g. , different devices or subjects), etc.

Original: This complexity of time series limits the utility of knowledge transfer for pre-training [19, 20].
Revised: This complexity of time series limits the utility of knowledge transfer for pre-training [19, 20].

Original: For example, pre-training a model on a diverse time series dataset with mostly low-frequency components (smooth trends) may not lead to positive transfer on downstream tasks with high-frequency components (transient events) [17].
Revised: For example, pre-training a model on a diverse time series dataset with mostly low-frequency components (smooth trends) may not lead to positive transfer on downstream tasks with high-frequency components (transient events) [17].

Original: Examining these challenges can provide clues to what kind of inductive biases could facilitate generalizable representations of time series – this paper offers a strategy for that through a novel time-frequency consistency principle.
Revised: Examining these challenges can provide clues to what kind of inductive biases could facilitate generalizable representations of time series – this paper offers a strategy for that through a novel time-frequency consistency principle.

Original: In addition, target dataset are not available in pre-training phase (different from domain adaption [21];
Revised: In addition, target dataset are not available in pre-training phase (different from domain adaption [21];

Original: Appendix A), requiring that pre-training model captures some latent shared property in time series that can apply to unseen datasets.
Revised: Appendix A), requiring that pre-training model captures some latent shared property in time series that can apply to unseen datasets.

Original: The principle of a shared property is universal in different time series datasets and thus allows knowledge transfer from a pre-training dataset to a fine-tuning dataset.
Revised: The principle of a shared property is universal in different time series datasets and thus allows knowledge transfer from a pre-training dataset to a fine-tuning dataset.

Original: To make it more clear, a key principle for pre-training in computer vision (CV) is that the prior neural network layers learn object edges and shapes regardless of image style or tasks [22].
Revised: To make it more clear, a key principle for pre-training in computer vision (CV) is that the prior neural network layers learn object edges and shapes regardless of image style or tasks [22].

Original: The shared foundation for pre-training in natural language processing (NLP) is the linguistic assumption of semantics and grammars that is retrained across different languages [23].
Revised: The shared foundation for pre-training in natural language processing (NLP) is the linguistic assumption of semantics and grammars that is retrained across different languages [23].

Original: Nevertheless, it is hard to put forward such a principle for time series due to aforementioned temporal complexity.
Revised: Nevertheless, it is hard to put forward such a principle for time series due to aforementioned temporal complexity.

Original: Moreover, supervised pre-training requires large annotated datasets, creating an obstacle to deep learning deployment [24, 25].
Revised: Moreover, supervised pre-training requires large annotated datasets, creating an obstacle to deep learning deployment [24, 25].

Original: For example, in medical domains, experts often disagree on groundtruth labeling ( e.g. , ECG signals falling in normal or abnormal rhythms [26, 27]), which can introduce unintended biases [28, 29].
Revised: For example, in medical domains, experts often disagree on groundtruth labeling ( e.g. , ECG signals falling in normal or abnormal rhythms [26, 27]), which can introduce unintended biases [28, 29].

Original: Self-supervised contrastive learning can overstep the lack of annotations by providing self-supervision signal through contrastive instance discrimination [30].
Revised: Self-supervised contrastive learning can overstep the lack of annotations by providing self-supervision signal through contrastive instance discrimination [30].

Original: Present work.
Revised: Present work.

Original: We introduce a strategy for self-supervised pre-training in time series by modeling
Revised: We introduce a strategy for self-supervised pre-training in time series by modeling

Original: Time-Frequency Consistency (TF-C).
Revised: Time-Frequency Consistency (TF-C).

Original: TF-C means the time-based representation and the frequencybased representation, which are learned from the same time series sample, are closer to each other in a joint time-frequency space, and farther apart if the representations are associated with different time series samples.
Revised: TF-C means the time-based representation and the frequencybased representation, which are learned from the same time series sample, are closer to each other in a joint time-frequency space, and farther apart if the representations are associated with different time series samples.

Original: Specifically, we adopt contrastive learning in time space to generate a time-based representation.
Revised: Specifically, we adopt contrastive learning in time space to generate a time-based representation.

Original: In parallel, we propose a set of novel augmentations based on the characteristic of frequency spectrum and produce a frequency-based embedding through contrastive instance discrimination.
Revised: In parallel, we propose a set of novel augmentations based on the characteristic of frequency spectrum and produce a frequency-based embedding through contrastive instance discrimination.

Original: To the best of our knowledge, this is the first work that develops contrastive augmentation in frequency domain to leverage rich spectral information, also the first work that exploits the property of time-frequency consistency in time series.
Revised: To the best of our knowledge, this is the first work that develops contrastive augmentation in frequency domain to leverage rich spectral information, also the first work that exploits the property of time-frequency consistency in time series.

Original: The pre-training objective is to minimize the distance between the time-based and frequency-based embeddings through a dedicated consistency loss (Figure 1 (a)).
Revised: The pre-training objective is to minimize the distance between the time-based and frequency-based embeddings through a dedicated consistency loss (Figure 1 (a)).

Original: The designed loss provides self-supervising signals to optimize the pre-training model and captures the latent relationships between time domain and frequency domain.
Revised: The designed loss provides self-supervising signals to optimize the pre-training model and captures the latent relationships between time domain and frequency domain.

Original: The learned relationship encoded in model parameters are transferred to initialize the fine-tuning model and improve performance in datasets of interest (Figure 1 (b)).
Revised: The learned relationship encoded in model parameters are transferred to initialize the fine-tuning model and improve performance in datasets of interest (Figure 1 (b)).

Original: We evaluate the proposed pre-training model on eight time series datasets under two evaluation settings ( i.e. , one-to-one and one-to-many).
Revised: We evaluate the proposed pre-training model on eight time series datasets under two evaluation settings ( i.e. , one-to-one and one-to-many).

Original: The eight datasets cover a large set of variations: different number of channels (from univariate to 9-channel multivariate), varying time series length (from 128 to 5,120), different sampling rate (from 16 Hz to 4,000 Hz), different scenarios (neurological healthcare, human activity recognition, mechanical fault detection, physical status monitoring, etc.) and diverse types of signals (EEG, EMG, ECG, acceleration, and vibration).
Revised: The eight datasets cover a large set of variations: different number of channels (from univariate to 9-channel multivariate), varying time series length (from 128 to 5,120), different sampling rate (from 16 Hz to 4,000 Hz), different scenarios (neurological healthcare, human activity recognition, mechanical fault detection, physical status monitoring, etc.) and diverse types of signals (EEG, EMG, ECG, acceleration, and vibration).

Original: We compare TF-C approach to eight state-of-the-art baselines.
Revised: We compare TF-C approach to eight state-of-the-art baselines.

Original: Results show that TF-C achieves positive transfer, outperforming all baselines by a large margin of 15.4% (F1 score) on average.
Revised: Results show that TF-C achieves positive transfer, outperforming all baselines by a large margin of 15.4% (F1 score) on average.

Original: Further, the approach outperforms the strongest baselines with an improvement of up to 7.2% in F1 score.
Revised: Further, the approach outperforms the strongest baselines with an improvement of up to 7.2% in F1 score.

Original: Finally, the
Revised: Finally, the

Original: TF-C approach improves prior work by 8.4% in precision (when pre-training the model on sleep
Revised: TF-C approach improves prior work by 8.4% in precision (when pre-training the model on sleep

Original: EEG signals and fine-tuning it on hand-gesture recognition) in a challenging one-to-many setup that applies the same pre-trained model to multiple independent fine-tuning datasets.
Revised: EEG signals and fine-tuning it on hand-gesture recognition) in a challenging one-to-many setup that applies the same pre-trained model to multiple independent fine-tuning datasets.

Original:  Related Work
Revised:  Related Work

Original: Pre-training for time series.
Revised: Pre-training for time series.

Original: Although there are studies on self-supervised representation learning for time series [7, 8, 31, 32] and self-supervised pre-training for images [33, 34, 35, 24], the intersection of these two areas, i.e. , self-supervised pre-training for time series, remains underexplored.
Revised: Although there are studies on self-supervised representation learning for time series [7, 8, 31, 32] and self-supervised pre-training for images [33, 34, 35, 24], the intersection of these two areas, i.e. , self-supervised pre-training for time series, remains underexplored.

Original: In time series, it’s not obvious that what assumptions are reasonable that can bridge pre-training dataset and target dataset.
Revised: In time series, it’s not obvious that what assumptions are reasonable that can bridge pre-training dataset and target dataset.

Original: Hence, pre-training models in CV [36, 37, 14] and NLP [10, 15, 38] are not directly applicable due to data modality mismatch, and the results leave room for improvement [31, 39, 40].
Revised: Hence, pre-training models in CV [36, 37, 14] and NLP [10, 15, 38] are not directly applicable due to data modality mismatch, and the results leave room for improvement [31, 39, 40].

Original: Shi et al.
Revised: Shi et al.

Original: [12] developed the only model to date that is explicitly designed for self-supervised time series pre-training.
Revised: [12] developed the only model to date that is explicitly designed for self-supervised time series pre-training.

Original: The model captures the local and global temporal pattern but it’s not convincing why the designed pretext task can capture generalizable representations.
Revised: The model captures the local and global temporal pattern but it’s not convincing why the designed pretext task can capture generalizable representations.

Original: Although several studies applied transfer learning in the context of time series [7, 8, 18, 41], there is no foundation yet of which conceptual properties are most suitable for pre-training on time series and why.
Revised: Although several studies applied transfer learning in the context of time series [7, 8, 18, 41], there is no foundation yet of which conceptual properties are most suitable for pre-training on time series and why.

Original: Addressing this gap, we show that TF-C, designed to be invariant to different time-series datasets, can produce generalizable pre-training models.
Revised: Addressing this gap, we show that TF-C, designed to be invariant to different time-series datasets, can produce generalizable pre-training models.

Original: Unlike domain adaptation [21, 42] that requires access to target datasets during training, pre-training models have access only to pre-training datasets.
Revised: Unlike domain adaptation [21, 42] that requires access to target datasets during training, pre-training models have access only to pre-training datasets.

Original: As a result, one needs to identify a generalizable time-series property to benefit from pre-training.
Revised: As a result, one needs to identify a generalizable time-series property to benefit from pre-training.

Original: Further, self-supervised domain adaptation does not need labels in the target dataset but still requires labels for model training [43, 44].
Revised: Further, self-supervised domain adaptation does not need labels in the target dataset but still requires labels for model training [43, 44].

Original: In contrast, TF-C does not need any labels during pre-training.
Revised: In contrast, TF-C does not need any labels during pre-training.

Original: Contrastive learning with time series.
Revised: Contrastive learning with time series.

Original: Contrastive learning, a popular type of self-supervised learning, aims to learn an encoder that maps inputs into an embedding space such that positive sample pairs (original augmentation and another alternative augmentation/view of the same input sample) are pulled closer and negative sample pairs (original augmentation and an alternative input sample augmentation) are pushed apart [30, 45].
Revised: Contrastive learning, a popular type of self-supervised learning, aims to learn an encoder that maps inputs into an embedding space such that positive sample pairs (original augmentation and another alternative augmentation/view of the same input sample) are pulled closer and negative sample pairs (original augmentation and an alternative input sample augmentation) are pushed apart [30, 45].

Original: Contrastive learning in time series is less investigated in comparison, partly due to the challenge of identifying augmentations that capture key invariance properties in time series data.
Revised: Contrastive learning in time series is less investigated in comparison, partly due to the challenge of identifying augmentations that capture key invariance properties in time series data.

Original: For example, CLOCS defines adjacent segments of a time series as positive pairs [41], and TNC assumes overlapping neighborhoods of time series should have similar representations [46].
Revised: For example, CLOCS defines adjacent segments of a time series as positive pairs [41], and TNC assumes overlapping neighborhoods of time series should have similar representations [46].

Original: These methods leverage temporal invariance to define positive pairs which are used to calculate contrastive loss, but other invariances, such as transformation invariance ( e.g. ,
Revised: These methods leverage temporal invariance to define positive pairs which are used to calculate contrastive loss, but other invariances, such as transformation invariance ( e.g. ,

Original: SimCLR [40]), contextual invariance ( e.g. , TS2vec [47] and TS-TCC [48]) and augmentations are possible.
Revised: SimCLR [40]), contextual invariance ( e.g. , TS2vec [47] and TS-TCC [48]) and augmentations are possible.

Original: In this work, we propose an augmentation bank that exploits multiple invariances to generate diverse augmentations (Sec. 4.1), which adds richness into the pre-training model [48].
Revised: In this work, we propose an augmentation bank that exploits multiple invariances to generate diverse augmentations (Sec. 4.1), which adds richness into the pre-training model [48].

Original: Importantly, we propose frequency-based augmentations by perturbing the frequency spectrum of time series ( e.g. , adding or removing the frequency components and manipulating the their amplitude; more details in Sec. 4.2) to learn better representations by exposing the model to local range of frequency variations.
Revised: Importantly, we propose frequency-based augmentations by perturbing the frequency spectrum of time series ( e.g. , adding or removing the frequency components and manipulating the their amplitude; more details in Sec. 4.2) to learn better representations by exposing the model to local range of frequency variations.

Original: A previous work CoST processes sequential signals through frequency domain but the augmentations are still implemented in time space [49].
Revised: A previous work CoST processes sequential signals through frequency domain but the augmentations are still implemented in time space [49].

Original: Similarly, although BTSF [50] involves frequency domain, its data transformation is solely implemented in time domain using instance-level dropout.
Revised: Similarly, although BTSF [50] involves frequency domain, its data transformation is solely implemented in time domain using instance-level dropout.

Original: More differences with CoST and BTSF in Appendix B. To the best of our knowledge, this is the first work that directly perturbs the frequency spectrum to leverage frequency-invariance for contrastive learning.
Revised: More differences with CoST and BTSF in Appendix B. To the best of our knowledge, this is the first work that directly perturbs the frequency spectrum to leverage frequency-invariance for contrastive learning.

Original: Further, we develop a pre-training model that subjects to TF-C upon two individual contrastive encoders.
Revised: Further, we develop a pre-training model that subjects to TF-C upon two individual contrastive encoders.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: We are given a pre-training dataset D pret =
Revised: We are given a pre-training dataset D pret =

Original: { x pret i | i = 1 , . .
Revised: { x pret i | i = 1 , . .

Original: . , N } of unlabeled time series samples where sample x pret i has K pret channels and L pret timestamps.
Revised: . , N } of unlabeled time series samples where sample x pret i has K pret channels and L pret timestamps.

Original: Let D tune =
Revised: Let D tune =

Original: { ( x tune i , y i ) | i = 1 , . .
Revised: { ( x tune i , y i ) | i = 1 , . .

Original: , M } be a fine-tuning ( i.e. , target; target and fine-tuning are used interchangeably) dataset of labeled time series samples, each having K tune channels and L tune timestamps.
Revised: , M } be a fine-tuning ( i.e. , target; target and fine-tuning are used interchangeably) dataset of labeled time series samples, each having K tune channels and L tune timestamps.

Original: Further, every sample x tune i is associated with label y i ∈ { 1 , . . .
Revised: Further, every sample x tune i is associated with label y i ∈ { 1 , . . .

Original: , C } , where C is the number of classes.
Revised: , C } , where C is the number of classes.

Original: Without loss of generality, in the following, we describe the approach for univariate (single-channel) time series, noting that the approach can accommodate multivariate time series, different numbers of channels across samples, and varying lengths (shown in experiments in Sec. 5.2).
Revised: Without loss of generality, in the following, we describe the approach for univariate (single-channel) time series, noting that the approach can accommodate multivariate time series, different numbers of channels across samples, and varying lengths (shown in experiments in Sec. 5.2).

Original: We use superscript symbol (cid:101) to denote contrastive augmentations.
Revised: We use superscript symbol (cid:101) to denote contrastive augmentations.

Original: We note that x T i ≡ x i denotes an input time series sample, and x F i denotes the frequency spectrum of x i .
Revised: We note that x T i ≡ x i denotes an input time series sample, and x F i denotes the frequency spectrum of x i .

Original: Problem (Self-Supervised Contrastive Pre-Training For Time Series).
Revised: Problem (Self-Supervised Contrastive Pre-Training For Time Series).

Original: Given are an unlabeled pre-training dataset D pret with N samples and a target dataset D tune with M samples ( M ≪ N ).
Revised: Given are an unlabeled pre-training dataset D pret with N samples and a target dataset D tune with M samples ( M ≪ N ).

Original: We want to use D pret to pre-train a neural model F so that by fine-tuning model parameters on D tune , the fine-tuned model can generate a generalizable representation z tune i =
Revised: We want to use D pret to pre-train a neural model F so that by fine-tuning model parameters on D tune , the fine-tuned model can generate a generalizable representation z tune i =

Original: F ( x tune i ) for every x tune i .
Revised: F ( x tune i ) for every x tune i .

Original: We follow established pre-training setup, e.g. , [41]; for pre-training, only unlabeled dataset D pret is available while, for fine-tuning, a small labeled dataset D tune can be used.
Revised: We follow established pre-training setup, e.g. , [41]; for pre-training, only unlabeled dataset D pret is available while, for fine-tuning, a small labeled dataset D tune can be used.

Original: In short, a model
Revised: In short, a model

Original: F is pre-trained on unlabeled time series dataset D pret and its optimized model parameters
Revised: F is pre-trained on unlabeled time series dataset D pret and its optimized model parameters

Original: Θ are fine-tuned to go from F ( · , Θ) to F ( · , Φ) using dataset D tune .
Revised: Θ are fine-tuned to go from F ( · , Θ) to F ( · , Φ) using dataset D tune .

Original: The Φ denotes fine-tuned model parameters.
Revised: The Φ denotes fine-tuned model parameters.

Original: Note that this problem ( i.e. , D pret is independent of target dataset) is different than domain adaptation because we don’t access the fine-tuning dataset D tune during pre-training, meaning that a successful pre-trained model should be applicable to multiple independent fine-tuning datasets without necessitating re-training.
Revised: Note that this problem ( i.e. , D pret is independent of target dataset) is different than domain adaptation because we don’t access the fine-tuning dataset D tune during pre-training, meaning that a successful pre-trained model should be applicable to multiple independent fine-tuning datasets without necessitating re-training.

Original: Rationale for Time-Frequency Consistency (TF-C).
Revised: Rationale for Time-Frequency Consistency (TF-C).

Original: For effective per-training on time series, our central idea is to identify a broadly generalizable property that is preserved across different time series datasets.
Revised: For effective per-training on time series, our central idea is to identify a broadly generalizable property that is preserved across different time series datasets.

Original: In time series in particular, the time domain shows how readouts change with time whereas frequency domain tells us how much of the signal lies within each frequency band over a range of frequencies ( e.g. , frequency spectrum) [51].
Revised: In time series in particular, the time domain shows how readouts change with time whereas frequency domain tells us how much of the signal lies within each frequency band over a range of frequencies ( e.g. , frequency spectrum) [51].

Original: Explicitly considering the frequency domain can provide a deeper understanding of time series behavior that can be challenging to capture solely in time domain [52].
Revised: Explicitly considering the frequency domain can provide a deeper understanding of time series behavior that can be challenging to capture solely in time domain [52].

Original: However, most existing time-series methods focus exclusively on modeling time domain, e.g. , [47, 48], and ignore frequency domain altogether.
Revised: However, most existing time-series methods focus exclusively on modeling time domain, e.g. , [47, 48], and ignore frequency domain altogether.

Original: One could argue that is sufficient for high-capacity methods as time and frequency domains describe the same data, just from different aspects [53], and time series can be converted between time and frequency domains using transformation, such as Fourier [54, 52].
Revised: One could argue that is sufficient for high-capacity methods as time and frequency domains describe the same data, just from different aspects [53], and time series can be converted between time and frequency domains using transformation, such as Fourier [54, 52].

Original: While we are not interested in modeling the two domains individually, the relationship between them, grounded in signal processing theory, provides an invariance that is valid regardless of time series distribution [55, 56] and thus can serve as an inductive bias for pre-training to promote generalizability.
Revised: While we are not interested in modeling the two domains individually, the relationship between them, grounded in signal processing theory, provides an invariance that is valid regardless of time series distribution [55, 56] and thus can serve as an inductive bias for pre-training to promote generalizability.

Original: We provide an analogy, from images to time series, regarding ‘invariance’ in Appendix C. Approaching this invariance through the lens of representation learning, we formulate Time-Frequency Consistency (TF-C) by postulating that for every time
Revised: We provide an analogy, from images to time series, regarding ‘invariance’ in Appendix C. Approaching this invariance through the lens of representation learning, we formulate Time-Frequency Consistency (TF-C) by postulating that for every time

Original: series sample x i , there exists a latent time-frequency space in which time-based representation z T i and frequency-based representation z F i of the same sample, together with their local augmentations (defined later) are close to each other.
Revised: series sample x i , there exists a latent time-frequency space in which time-based representation z T i and frequency-based representation z F i of the same sample, together with their local augmentations (defined later) are close to each other.

Original: Representational Time-Frequency Consistency (TF-C).
Revised: Representational Time-Frequency Consistency (TF-C).

Original: Let be given a time series sample x i .
Revised: Let be given a time series sample x i .

Original: Then in a model F satisfying TF-C, time-based representation z T i and frequency-based representation z F i learned from x i by F , and representations learned from local augmentations of x i are close together in the latent time-frequency space.
Revised: Then in a model F satisfying TF-C, time-based representation z T i and frequency-based representation z F i learned from x i by F , and representations learned from local augmentations of x i are close together in the latent time-frequency space.

Original: Our pre-training strategy is to use a time series dataset D pret to induce TF-C in F ’s model parameters Θ , which, in turn, are used to initialize the target model on D tune and produce generalizable representations for downstream prediction.
Revised: Our pre-training strategy is to use a time series dataset D pret to induce TF-C in F ’s model parameters Θ , which, in turn, are used to initialize the target model on D tune and produce generalizable representations for downstream prediction.

Original: The invariant nature of TF-C means that our approach can bridge
Revised: The invariant nature of TF-C means that our approach can bridge

Original: D pret and D tune even when large discrepancies exist between them (in terms of temporal dynamics, semantic meaning, etc.), providing a vehicle for a generalizable pre-training strategy for time series.
Revised: D pret and D tune even when large discrepancies exist between them (in terms of temporal dynamics, semantic meaning, etc.), providing a vehicle for a generalizable pre-training strategy for time series.

Original: To realize TF-C, our model F has four components (Figure 2): a time encoder G T , a frequency encoder G F , and two cross-space projectors R T and R F that map time-based and frequency-based representations, respectively, to the same time-frequency space.
Revised: To realize TF-C, our model F has four components (Figure 2): a time encoder G T , a frequency encoder G F , and two cross-space projectors R T and R F that map time-based and frequency-based representations, respectively, to the same time-frequency space.

Original: Together, the four components
Revised: Together, the four components

Original: [Figure]
Revised: [Figure]

Original: provide a way to embed x i to the latent time-frequency space such that time-based embedding z T i
Revised: provide a way to embed x i to the latent time-frequency space such that time-based embedding z T i

Original: =
Revised: =

Original: R T ( G T ( x T i )) and frequency-based embedding z F i = R F ( G F ( x F i )) are close together.
Revised: R T ( G T ( x T i )) and frequency-based embedding z F i = R F ( G F ( x F i )) are close together.

Original:  Our Approach
Revised:  Our Approach

Original: Next, we present the architecture of the developed self-supervised contrastive pre-training model
Revised: Next, we present the architecture of the developed self-supervised contrastive pre-training model

Original: F .
Revised: F .

Original: Without specification, the data mentioned in this section are from pre-training dataset and the superscript pret is omitted for simplification.
Revised: Without specification, the data mentioned in this section are from pre-training dataset and the superscript pret is omitted for simplification.

Original: Here we describe the model using univariate time series as an example, but our model can be straightforwardly applied to multivariate time series (Sec 5).
Revised: Here we describe the model using univariate time series as an example, but our model can be straightforwardly applied to multivariate time series (Sec 5).

Original: 4.1 Time-based Contrastive Encoder
Revised: 4.1 Time-based Contrastive Encoder

Original: For a given input time series sample x i , we generate an augmentation set X T i through a time-based augmentation bank B T : x T i → X T i .
Revised: For a given input time series sample x i , we generate an augmentation set X T i through a time-based augmentation bank B T : x T i → X T i .

Original: Each element (cid:101) x T i ∈
Revised: Each element (cid:101) x T i ∈

Original: X T i is augmented from x i based on the temporal characteristics.
Revised: X T i is augmented from x i based on the temporal characteristics.

Original: Here, the time-based augmentation bank includes jittering, scaling, time-shifts, and neighborhood segments, all well-established in contrastive learning [40, 48, 41].
Revised: Here, the time-based augmentation bank includes jittering, scaling, time-shifts, and neighborhood segments, all well-established in contrastive learning [40, 48, 41].

Original: We develop the augmentation bank to produce diverse augmentations (rather than a single type of augmentation) and expose the model to complex temporal dynamics, which produces more robust time-based embeddings [48].
Revised: We develop the augmentation bank to produce diverse augmentations (rather than a single type of augmentation) and expose the model to complex temporal dynamics, which produces more robust time-based embeddings [48].

Original: For the input x i , we randomly select an augmented sample (cid:101) x T
Revised: For the input x i , we randomly select an augmented sample (cid:101) x T

Original: i ∈
Revised: i ∈

Original: X T i and feed into a contrastive time encoder G T that maps samples to embeddings.
Revised: X T i and feed into a contrastive time encoder G T that maps samples to embeddings.

Original: We have h T
Revised: We have h T

Original: i =
Revised: i =

Original: G T ( x T i ) and (cid:101) h T i =
Revised: G T ( x T i ) and (cid:101) h T i =

Original: G T ( (cid:101) x T i ) .
Revised: G T ( (cid:101) x T i ) .

Original: As (cid:101) x T i is generated based on x T i , after passing through G T , we assume the embedding of x T i is close to the embedding of (cid:101)
Revised: As (cid:101) x T i is generated based on x T i , after passing through G T , we assume the embedding of x T i is close to the embedding of (cid:101)

Original: x T i but far away from the embedding of x T j and (cid:101) x
Revised: x T i but far away from the embedding of x T j and (cid:101) x

Original: T j that are derived from another sample x T j ∈ D pret [34, 47, 41].
Revised: T j that are derived from another sample x T j ∈ D pret [34, 47, 41].

Original: In specific, we select the positive pair as ( x T i , (cid:101) x T i ) and negative pairs as ( x T i , x T j ) and ( x T i , (cid:101) x T j ) [34].
Revised: In specific, we select the positive pair as ( x T i , (cid:101) x T i ) and negative pairs as ( x T i , x T j ) and ( x T i , (cid:101) x T j ) [34].

Original: Contrastive time loss.
Revised: Contrastive time loss.

Original: To maximize the similarity within a positive pair and minimize the similarity within a negative pair, we adopt the NT-Xent (the normalized temperature-scaled cross entropy loss) as distance function d which is widely used in contrastive learning [34, 40].
Revised: To maximize the similarity within a positive pair and minimize the similarity within a negative pair, we adopt the NT-Xent (the normalized temperature-scaled cross entropy loss) as distance function d which is widely used in contrastive learning [34, 40].

Original: In specific, we measure the loss of time-based contrastive encoder in terms of sample x T i by:
Revised: In specific, we measure the loss of time-based contrastive encoder in terms of sample x T i by:

Original: [Equation]
Revised: [Equation]

Original: where sim ( u , v )
Revised: where sim ( u , v )

Original: =
Revised: =

Original: u T v / ∥ u ∥ ∥ v ∥ denotes the cosine similarity, the 1 i ̸ = j ∈ { 0 , 1 } is a binary indicator that equals to 0 when i = j , and τ is a temporal parameter.
Revised: u T v / ∥ u ∥ ∥ v ∥ denotes the cosine similarity, the 1 i ̸ = j ∈ { 0 , 1 } is a binary indicator that equals to 0 when i = j , and τ is a temporal parameter.

Original: The x j ∈ D pret refers to a different time series sample and its augmented sample.
Revised: The x j ∈ D pret refers to a different time series sample and its augmented sample.

Original: This loss urges the time encoder G T to generate closer time-based embeddings for positive pairs, and push the embeddings for negative pairs apart from each other.
Revised: This loss urges the time encoder G T to generate closer time-based embeddings for positive pairs, and push the embeddings for negative pairs apart from each other.

Original: 4.2 Frequency-based Contrastive Encoder
Revised: 4.2 Frequency-based Contrastive Encoder

Original: We generate the frequency spectrum x F i from time series x T i through a transform operator ( e.g. ,
Revised: We generate the frequency spectrum x F i from time series x T i through a transform operator ( e.g. ,

Original: Fourier Transformation [54]).
Revised: Fourier Transformation [54]).

Original: The frequency information of time series is universal and plays key role in classic signal processing [57, 53, 55], but rarely investigated in self-supervised contrastive learning for time series [58].
Revised: The frequency information of time series is universal and plays key role in classic signal processing [57, 53, 55], but rarely investigated in self-supervised contrastive learning for time series [58].

Original: In this section, we develop augmentation methods to perturb x F i based on characteristics of frequency spectra and generate frequency-based representations.
Revised: In this section, we develop augmentation methods to perturb x F i based on characteristics of frequency spectra and generate frequency-based representations.

Original: As every frequency component in the frequency spectrum denotes a base function ( e.g. , sinusoidal function for Fourier transformation) with the corresponding frequency and amplitude, in our augmentation methods, we perturb the frequency spectrum by adding or removing frequency components.
Revised: As every frequency component in the frequency spectrum denotes a base function ( e.g. , sinusoidal function for Fourier transformation) with the corresponding frequency and amplitude, in our augmentation methods, we perturb the frequency spectrum by adding or removing frequency components.

Original: A small perturbation on frequency spectrum may cause large changes to the temporal patterns in time domain [55].
Revised: A small perturbation on frequency spectrum may cause large changes to the temporal patterns in time domain [55].

Original: To make sure the perturbed time series is still similar to the original sample (not only in frequency domain but also in time domain; Figure 6), we use a small budget E in the perturbations where E denotes the number of frequency components we manipulate.
Revised: To make sure the perturbed time series is still similar to the original sample (not only in frequency domain but also in time domain; Figure 6), we use a small budget E in the perturbations where E denotes the number of frequency components we manipulate.

Original: While removing frequency components, we randomly select E frequency components and set their amplitudes as 0 .
Revised: While removing frequency components, we randomly select E frequency components and set their amplitudes as 0 .

Original: While adding frequency components, we randomly choose E frequency components from the ones have smaller amplitude than α · A m , and increase their amplitude to α · A m .
Revised: While adding frequency components, we randomly choose E frequency components from the ones have smaller amplitude than α · A m , and increase their amplitude to α · A m .

Original: The A m is the maximum amplitude in the frequency spectrum and α is a pre-defined coefficient to adjust the scale of the perturbed frequency component ( α = 0 .
Revised: The A m is the maximum amplitude in the frequency spectrum and α is a pre-defined coefficient to adjust the scale of the perturbed frequency component ( α = 0 .

Original: 5 in this work).
Revised: 5 in this work).

Original: We produce an augmentation set X F i for x F i through frequency-augmentation bank B F : x F i → X F i .
Revised: We produce an augmentation set X F i for x F i through frequency-augmentation bank B F : x F i → X F i .

Original: We have two augmentation methods ( i.e. , removing or adding frequency components) in B F , |X F i | = 2 .
Revised: We have two augmentation methods ( i.e. , removing or adding frequency components) in B F , |X F i | = 2 .

Original: More details on the exploration of frequency augmentation strategies in Appendix J.
Revised: More details on the exploration of frequency augmentation strategies in Appendix J.

Original: We utilize a frequency encoder G F to map the frequency spectrum ( e.g. , x F i ) to a frequency-based embedding ( e.g. , h F i = G F ( x F i ) ).
Revised: We utilize a frequency encoder G F to map the frequency spectrum ( e.g. , x F i ) to a frequency-based embedding ( e.g. , h F i = G F ( x F i ) ).

Original: We assume the frequency encoder G F can learn similar embedding for the original frequency spectrum x F i and a slightly perturbed frequency spectrum (cid:101)
Revised: We assume the frequency encoder G F can learn similar embedding for the original frequency spectrum x F i and a slightly perturbed frequency spectrum (cid:101)

Original: x F i ∈
Revised: x F i ∈

Original: X F i .
Revised: X F i .

Original: Thus, we set the positive pair as ( x F i , (cid:101) x F i ) and the negative pairs as ( x F i , x F j ) and ( x F i , (cid:101) x F j ) .
Revised: Thus, we set the positive pair as ( x F i , (cid:101) x F i ) and the negative pairs as ( x F i , x F j ) and ( x F i , (cid:101) x F j ) .

Original: Contrastive frequency loss.
Revised: Contrastive frequency loss.

Original: We calculate frequency-based contrastive loss for sample x i as:
Revised: We calculate frequency-based contrastive loss for sample x i as:

Original: [Equation]
Revised: [Equation]

Original: In preliminary experiments, we find that the value of τ has little effect on performance and use the same τ throughout all experiments.
Revised: In preliminary experiments, we find that the value of τ has little effect on performance and use the same τ throughout all experiments.

Original: The L F ,i yield a frequency encoder
Revised: The L F ,i yield a frequency encoder

Original: G F producing embeddings invariant to frequency spectrum perturbations.
Revised: G F producing embeddings invariant to frequency spectrum perturbations.

Original: 4.3 Time-Frequency Consistency
Revised: 4.3 Time-Frequency Consistency

Original: We develop a consistency loss item L C ,i to urge the learned embeddings to satisfy TF-C: for a given sample, its time-based and frequency-based embeddings (and their local neighborhoods) are supposed to be close to each other (see Sec. 3 for justification).
Revised: We develop a consistency loss item L C ,i to urge the learned embeddings to satisfy TF-C: for a given sample, its time-based and frequency-based embeddings (and their local neighborhoods) are supposed to be close to each other (see Sec. 3 for justification).

Original: To make sure the distance between embeddings is measurable, we map h T i from time space and h F i from frequency space to a joint time-frequency space through projectors R T and R F , respectively.
Revised: To make sure the distance between embeddings is measurable, we map h T i from time space and h F i from frequency space to a joint time-frequency space through projectors R T and R F , respectively.

Original: In specific, for every input sample x i , we have four embeddings, which are z T i
Revised: In specific, for every input sample x i , we have four embeddings, which are z T i

Original: =
Revised: =

Original: R T ( h T i ) , (cid:101)
Revised: R T ( h T i ) , (cid:101)

Original: z T i
Revised: z T i

Original: =
Revised: =

Original: R T ( (cid:101) h T i ) , z F i
Revised: R T ( (cid:101) h T i ) , z F i

Original: =
Revised: =

Original: R F ( h F i ) , and (cid:101)
Revised: R F ( h F i ) , and (cid:101)

Original: z F i
Revised: z F i

Original: =
Revised: =

Original: R F ( (cid:101) h F i ) .
Revised: R F ( (cid:101) h F i ) .

Original: The first two embeddings are generated based on temporal characteristics and the latter two embeddings are produced based on the properties of frequency spectrum.
Revised: The first two embeddings are generated based on temporal characteristics and the latter two embeddings are produced based on the properties of frequency spectrum.

Original: To enforce the embeddings in the time-frequency space subject to TF-C, we design a consistency loss
Revised: To enforce the embeddings in the time-frequency space subject to TF-C, we design a consistency loss

Original: L C ,i that measures the distance between a time-based embedding and a frequency-based embedding.
Revised: L C ,i that measures the distance between a time-based embedding and a frequency-based embedding.

Original: We use S T F i =
Revised: We use S T F i =

Original: d ( z T i , z F i , D pret ) to denote the distance between z T i and z F i ) .
Revised: d ( z T i , z F i , D pret ) to denote the distance between z T i and z F i ) .

Original: Similarly, we define S T (cid:101) F i ,
Revised: Similarly, we define S T (cid:101) F i ,

Original: S (cid:101) TF i , and S (cid:101) T (cid:101) F i .
Revised: S (cid:101) TF i , and S (cid:101) T (cid:101) F i .

Original: Note, in this time-frequency space, we don’t consider the distance between z T i and (cid:101)
Revised: Note, in this time-frequency space, we don’t consider the distance between z T i and (cid:101)

Original: z T i where the two embeddings are from the same domain ( i.e. , time domain).
Revised: z T i where the two embeddings are from the same domain ( i.e. , time domain).

Original: The same applies to pair the distance between z F i and (cid:101)
Revised: The same applies to pair the distance between z F i and (cid:101)

Original: z F i .
Revised: z F i .

Original: We have already considered information of above two pairs in the calculation of L T ,i and L F ,i .
Revised: We have already considered information of above two pairs in the calculation of L T ,i and L F ,i .

Original: Next, let’s closely observe S T F i and S T (cid:101)
Revised: Next, let’s closely observe S T F i and S T (cid:101)

Original: F i that involve three embeddings: z T i , z F i , and (cid:101)
Revised: F i that involve three embeddings: z T i , z F i , and (cid:101)

Original: z F i .
Revised: z F i .

Original: Here, z T i and z F i are learned from the original sample ( x T i and x F i ) while (cid:101)
Revised: Here, z T i and z F i are learned from the original sample ( x T i and x F i ) while (cid:101)

Original: z F i is learned from the augmented (cid:101) x
Revised: z F i is learned from the augmented (cid:101) x

Original: F i .
Revised: F i .

Original: Thus, intuitively, z T i should be closer to z F i in comparison to (cid:101)
Revised: Thus, intuitively, z T i should be closer to z F i in comparison to (cid:101)

Original: z F i .
Revised: z F i .

Original: Motivated by the relative relationship, we encourage the proposed model to learn a S T F i that is smaller than S T (cid:101) F i .
Revised: Motivated by the relative relationship, we encourage the proposed model to learn a S T F i that is smaller than S T (cid:101) F i .

Original: Inspired by the triplet loss [59], we design ( S T F i − S T (cid:101) F i + δ ) as a term of consistency loss L C ,i where δ is a given constant margin to keep negative samples far apart [60].
Revised: Inspired by the triplet loss [59], we design ( S T F i − S T (cid:101) F i + δ ) as a term of consistency loss L C ,i where δ is a given constant margin to keep negative samples far apart [60].

Original: This term optimize the model towards a smaller S T F i and relatively larger S T (cid:101) F i .
Revised: This term optimize the model towards a smaller S T F i and relatively larger S T (cid:101) F i .

Original: Similarly, S TF i is supposed to be smaller than S (cid:101) TF i and S (cid:101) T (cid:101) F i , respectively.
Revised: Similarly, S TF i is supposed to be smaller than S (cid:101) TF i and S (cid:101) T (cid:101) F i , respectively.

Original: In summary, we calculate the consistency loss L C ,i for sample x i by:
Revised: In summary, we calculate the consistency loss L C ,i for sample x i by:

Original: [Equation]
Revised: [Equation]

Original: where S pair i denotes between a time-based embedding ( e.g. , z T i or (cid:101)
Revised: where S pair i denotes between a time-based embedding ( e.g. , z T i or (cid:101)

Original: z T i ) and a frequency-based embedding ( e.g. , z F i or (cid:101)
Revised: z T i ) and a frequency-based embedding ( e.g. , z F i or (cid:101)

Original: z F i ).
Revised: z F i ).

Original: In which pair, there is at least one embedding that is derived from augmented sample instead of the original sample.
Revised: In which pair, there is at least one embedding that is derived from augmented sample instead of the original sample.

Original: The δ is a pre-defined constant.
Revised: The δ is a pre-defined constant.

Original: By combining all the triplet loss items, L C encourages the pre-training model to capture the consistency between time-based and frequency-based embeddings in model optimization.
Revised: By combining all the triplet loss items, L C encourages the pre-training model to capture the consistency between time-based and frequency-based embeddings in model optimization.

Original: Note, although the Eq.
Revised: Note, although the Eq.

Original: 3 does not explicitly measure the loss across different time series samples ( e.g. , x i and x j ), the cross-sample relationships are implicitly covered in the calculation of S TF i and S pair i .
Revised: 3 does not explicitly measure the loss across different time series samples ( e.g. , x i and x j ), the cross-sample relationships are implicitly covered in the calculation of S TF i and S pair i .

Original: 4.4 Implementation and Technical Details
Revised: 4.4 Implementation and Technical Details

Original: The overall loss function in pre-training has three terms.
Revised: The overall loss function in pre-training has three terms.

Original: First, time-based contrastive loss L T urges the model to learn embeddings invariant to temporal augmentations.
Revised: First, time-based contrastive loss L T urges the model to learn embeddings invariant to temporal augmentations.

Original: Second, frequency-based contrastive loss L F promotes learning of embeddings invariant to frequency spectrum-based augmentations.
Revised: Second, frequency-based contrastive loss L F promotes learning of embeddings invariant to frequency spectrum-based augmentations.

Original: Third, the consistency loss L C guides the model to retain the consistency between time-based and frequency-based embeddings.
Revised: Third, the consistency loss L C guides the model to retain the consistency between time-based and frequency-based embeddings.

Original: In summary, pre-training loss is the following:
Revised: In summary, pre-training loss is the following:

Original: [Equation]
Revised: [Equation]

Original: where λ controls the relative importance of contrastive and consistency losses.
Revised: where λ controls the relative importance of contrastive and consistency losses.

Original: We calculate the total loss by summing L TF-C ,i across all pre-training samples.
Revised: We calculate the total loss by summing L TF-C ,i across all pre-training samples.

Original: In implementation, the contrastive losses are calculated within the batch.
Revised: In implementation, the contrastive losses are calculated within the batch.

Original: Recall our problem definition, the model F we want to learn is the combination of neural networks G T , R T , G F , and R F .
Revised: Recall our problem definition, the model F we want to learn is the combination of neural networks G T , R T , G F , and R F .

Original: When pre-training is completed, we store parameters of entire model, and denote it as F ( · , Θ) where Θ represents all trainable parameters.
Revised: When pre-training is completed, we store parameters of entire model, and denote it as F ( · , Θ) where Θ represents all trainable parameters.

Original: When a test sample x tune i is presented, fine-tuned model F ( · , Φ) generates embedding z tune i by concatenating time- and frequency-based embeddings as: z tune
Revised: When a test sample x tune i is presented, fine-tuned model F ( · , Φ) generates embedding z tune i by concatenating time- and frequency-based embeddings as: z tune

Original: [Equation]
Revised: [Equation]

Original: where Φ denotes fine-tuned model parameters.
Revised: where Φ denotes fine-tuned model parameters.

Original:  Experiments
Revised:  Experiments

Original: We compare the TF-C with 10 baselines on 8 diverse datasets.
Revised: We compare the TF-C with 10 baselines on 8 diverse datasets.

Original: We investigate the time series classification tasks in the context of one-to-one and one-to-many transfer learning setups (the manyto-one setting is fundamentally different as discussed in Appendix K).
Revised: We investigate the time series classification tasks in the context of one-to-one and one-to-many transfer learning setups (the manyto-one setting is fundamentally different as discussed in Appendix K).

Original: We also assess TF-C in extensive downstream tasks including clustering and anomaly detection.
Revised: We also assess TF-C in extensive downstream tasks including clustering and anomaly detection.

Original: Datasets
Revised: Datasets

Original: [Equation]
Revised: [Equation]

Original: E EG [61] has 371,055 univariate brainwaves (EEG; 100 Hz) collected from 197 individuals.
Revised: E EG [61] has 371,055 univariate brainwaves (EEG; 100 Hz) collected from 197 individuals.

Original: Each sample associates with one among five sleeping stages.
Revised: Each sample associates with one among five sleeping stages.

Original: (2) E PILEPSY [62] monitors the brain activities of 500 subjects with single-channel EEG sensor (174 Hz).
Revised: (2) E PILEPSY [62] monitors the brain activities of 500 subjects with single-channel EEG sensor (174 Hz).

Original: Sample is labelled in binary based on whether the subject has epilepsy or not.
Revised: Sample is labelled in binary based on whether the subject has epilepsy or not.

Original: (3) F D - A [63] gathers the vibration signals from rolling bearing from a mechanical system aiming at fault detection.
Revised: (3) F D - A [63] gathers the vibration signals from rolling bearing from a mechanical system aiming at fault detection.

Original: Every sample has 5,120 timestamps and an indicator for one out of three mechanical device states.
Revised: Every sample has 5,120 timestamps and an indicator for one out of three mechanical device states.

Original: (4) F D - B [63] has the same setting as the F D - A but the rolling bearings are performed in different working conditions ( e.g. , varying rotational speed).
Revised: (4) F D - B [63] has the same setting as the F D - A but the rolling bearings are performed in different working conditions ( e.g. , varying rotational speed).

Original: (5) H AR [64] has 10,299 9-dimension samples fromdaily activities.
Revised: (5) H AR [64] has 10,299 9-dimension samples fromdaily activities.

Original: (6) G ESTURE [65] includes 440 samples that are collected from 8 hand gestures recorded by accelerometer.
Revised: (6) G ESTURE [65] includes 440 samples that are collected from 8 hand gestures recorded by accelerometer.

Original: (7) E CG [26] contains 8,528 single-sensor ECG recordings with sorted into four classes based on human physiology.
Revised: (7) E CG [26] contains 8,528 single-sensor ECG recordings with sorted into four classes based on human physiology.

Original: (8) E MG [66] consists of 163 EMG samples with 3-class labels implying muscular diseases.
Revised: (8) E MG [66] consists of 163 EMG samples with 3-class labels implying muscular diseases.

Original: Dataset labels are not used in pre-training.
Revised: Dataset labels are not used in pre-training.

Original: Further dataset statistics are in Appendix D and Table 3.
Revised: Further dataset statistics are in Appendix D and Table 3.

Original: [Table]
Revised: [Table]

Original: Baselines.
Revised: Baselines.

Original: We consider 10 baseline methods.
Revised: We consider 10 baseline methods.

Original: This includes 8 state-of-the-art methods: TS-SD [12],
Revised: This includes 8 state-of-the-art methods: TS-SD [12],

Original: TS2vec [47], CLOCS [41], Mixing-up [18], TS-TCC [48], SimCLR [40], TNC [46], and CPC [30].
Revised: TS2vec [47], CLOCS [41], Mixing-up [18], TS-TCC [48], SimCLR [40], TNC [46], and CPC [30].

Original: The TS2Vec, TS-TCC, SimCLR, TNC, and CPC are designed for representation learning on a single dataset (not across datasets), we apply them fit our settings to make the results comparable.
Revised: The TS2Vec, TS-TCC, SimCLR, TNC, and CPC are designed for representation learning on a single dataset (not across datasets), we apply them fit our settings to make the results comparable.

Original: As the training of TNC and CPC are very time-consuming, we only compare with them in the setting of one-to-one setting (scenario 1).
Revised: As the training of TNC and CPC are very time-consuming, we only compare with them in the setting of one-to-one setting (scenario 1).

Original: TNC and CPC are relatively less competitive (Table 4), thus they are not included in other experiments.
Revised: TNC and CPC are relatively less competitive (Table 4), thus they are not included in other experiments.

Original: To examine utility of pre-training, we consider two approaches (non-deep learning KNN model and the Random Init.
Revised: To examine utility of pre-training, we consider two approaches (non-deep learning KNN model and the Random Init.

Original: approach, which randomly initializes the fine-tuning model) that are applied directly to fine-tuning datasets without any pre-training.
Revised: approach, which randomly initializes the fine-tuning model) that are applied directly to fine-tuning datasets without any pre-training.

Original: The evaluation metrics are accuracy, precision (macro-averaged), recall, F1 score, AUROC, and AUPRC.
Revised: The evaluation metrics are accuracy, precision (macro-averaged), recall, F1 score, AUROC, and AUPRC.

Original: Implementation.
Revised: Implementation.

Original: We use two 3-layer 1-D ResNets [67] as backbones for G T and G F encoders.
Revised: We use two 3-layer 1-D ResNets [67] as backbones for G T and G F encoders.

Original: Datasets contain long series (samples in F D - A and F D - B have 5,120 observations), and preliminary experiments identified ResNet as a better option than a Transformer variant [68].
Revised: Datasets contain long series (samples in F D - A and F D - B have 5,120 observations), and preliminary experiments identified ResNet as a better option than a Transformer variant [68].

Original: We use 2 fullyconnected layers for R T and R F (no sharing of parameters).
Revised: We use 2 fullyconnected layers for R T and R F (no sharing of parameters).

Original: We set E = 

Revised: We set E =

Original: and α = 0 .
Revised: and α = 0 .

Original: 5 in frequency augmentations and τ = 0 .
Revised: 5 in frequency augmentations and τ = 0 .

Original: 2 , δ = 1 , λ = 0 .
Revised: 2 , δ = 1 , λ = 0 .

Original: 5 in loss functions.
Revised: 5 in loss functions.

Original: Reported are mean and standard deviation values across 5 independent runs (both pre-training and fine-tuning) on the same data split.
Revised: Reported are mean and standard deviation values across 5 independent runs (both pre-training and fine-tuning) on the same data split.

Original: Results for KNN (K=2) do not change so standard deviation is zero.
Revised: Results for KNN (K=2) do not change so standard deviation is zero.

Original: Method details and hyper-parameter selection are in Appendix E.
Revised: Method details and hyper-parameter selection are in Appendix E.

Original: 5.1 Results: One-to-One Pre-Training Evaluation
Revised: 5.1 Results: One-to-One Pre-Training Evaluation

Original: Setup.
Revised: Setup.

Original: In one-to-one evaluation, we pre-train a model on one pre-training dataset and use it for fine-tuning on one target dataset only.
Revised: In one-to-one evaluation, we pre-train a model on one pre-training dataset and use it for fine-tuning on one target dataset only.

Original: Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 (S LEEP E EG → E PILEPSY ):
Revised: Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 Scenario 1 (S LEEP E EG → E PILEPSY ):

Original: Pre-training is done on S LEEP E EG and fine-tuning on E PILEPSY .
Revised: Pre-training is done on S LEEP E EG and fine-tuning on E PILEPSY .

Original: While both datasets describe a single-channel EEG, the signals are from different channels/positions on scalp, monitor different physiology (sleep vs. epilepsy), and are collected from different patients.
Revised: While both datasets describe a single-channel EEG, the signals are from different channels/positions on scalp, monitor different physiology (sleep vs. epilepsy), and are collected from different patients.

Original: Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 (F D - A → F D - B ): Datasets describe mechanical devices that operate in different working conditions, including rotational speed, load torque, and radial force.
Revised: Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 Scenario 2 (F D - A → F D - B ): Datasets describe mechanical devices that operate in different working conditions, including rotational speed, load torque, and radial force.

Original: Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 (H AR → G ESTURE ): Datasets record different activities ( types of human daily activities vs. 8 hand gestures).
Revised: Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 Scenario 3 (H AR → G ESTURE ): Datasets record different activities ( types of human daily activities vs. 8 hand gestures).

Original: While both datasets contain acceleration signals,
Revised: While both datasets contain acceleration signals,

Original: H AR has 9 channels while G ESTURE has 1 channel.
Revised: H AR has 9 channels while G ESTURE has 1 channel.

Original: Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 (E CG → E MG ): While both are physiological datasets, the E CG records the electrical signal from the heart whereas E MG measures muscle response in response to a nerve’s stimulation of the muscle.
Revised: Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 Scenario 4 (E CG → E MG ): While both are physiological datasets, the E CG records the electrical signal from the heart whereas E MG measures muscle response in response to a nerve’s stimulation of the muscle.

Original: We note that the discrepancy between pre-training and fine-tuning datasets in the above four scenarios is substantial, covering diverse kinds of variation in time series datasets: varying semantic meaning, sampling frequency, time series length, number of classes, and system factors ( e.g. , number of devices or subjects).
Revised: We note that the discrepancy between pre-training and fine-tuning datasets in the above four scenarios is substantial, covering diverse kinds of variation in time series datasets: varying semantic meaning, sampling frequency, time series length, number of classes, and system factors ( e.g. , number of devices or subjects).

Original: The setup is further challenged by relatively small number of samples available for fine-tuning (E PILEPSY : 60; F D - B : 60; G ESTURE : 480; E MG : 122).
Revised: The setup is further challenged by relatively small number of samples available for fine-tuning (E PILEPSY : 60; F D - B : 60; G ESTURE : 480; E MG : 122).

Original: Further details are in AppendixF.
Revised: Further details are in AppendixF.

Original: Results.
Revised: Results.

Original: The results for four scenarios are shown in Table 1 and Tables 4-6.
Revised: The results for four scenarios are shown in Table 1 and Tables 4-6.

Original: Overall, TF-C model has won 16 out of 24 tests (6 metrics in 4 scenarios), and is the second best performer in only 8 other tests.
Revised: Overall, TF-C model has won 16 out of 24 tests (6 metrics in 4 scenarios), and is the second best performer in only 8 other tests.

Original: We report all metrics but discuss the F1 score in the following.
Revised: We report all metrics but discuss the F1 score in the following.

Original: On average, our TF-C model claims a large margin of 15.4% over all baselines.
Revised: On average, our TF-C model claims a large margin of 15.4% over all baselines.

Original: Although the strongest baseline is varying (such as TS-TCC in Scenario 2; Mixing-up in Scenario 3), our model outperforms the strongest baselines by 1.5% across all scenarios.
Revised: Although the strongest baseline is varying (such as TS-TCC in Scenario 2; Mixing-up in Scenario 3), our model outperforms the strongest baselines by 1.5% across all scenarios.

Original: Specifically, when using H AR -pre-trained model for fine-tuning on G ESTURE (Scenario 3) (Table 1), the TF-C approach achieves the highest performance of 79.91% which yields a margin of 7.2% over the best baseline TS-TCC (74.57%).
Revised: Specifically, when using H AR -pre-trained model for fine-tuning on G ESTURE (Scenario 3) (Table 1), the TF-C approach achieves the highest performance of 79.91% which yields a margin of 7.2% over the best baseline TS-TCC (74.57%).

Original: One potential reason that Scenariois the most complex dataset in terms of classes (H AR has 6 classes while G ESTURE has 8 classes)
Revised: One potential reason that Scenariois the most complex dataset in terms of classes (H AR has 6 classes while G ESTURE has 8 classes)

Original: [Table]
Revised: [Table]

Original: that is difficult to model.
Revised: that is difficult to model.

Original: Thee assumption is proved from side by the relative lower performances in Scenario 3 ( < 80% ) than other scenarios ( > 90% ).
Revised: Thee assumption is proved from side by the relative lower performances in Scenario 3 ( < 80% ) than other scenarios ( > 90% ).

Original: We visualize the learned representations in time-frequency space (Appendix I) which strongly support our assumption on TF-C.
Revised: We visualize the learned representations in time-frequency space (Appendix I) which strongly support our assumption on TF-C.

Original: 5.2 Results: One-to-Many Pre
Revised: 5.2 Results: One-to-Many Pre

Original: Training Evaluation
Revised: Training Evaluation

Original: Setup.
Revised: Setup.

Original: In one-to-many evaluation, pre-training is done using one dataset followed by fine-tuning on multiple target datasets independently without starting pre-training from scratch.
Revised: In one-to-many evaluation, pre-training is done using one dataset followed by fine-tuning on multiple target datasets independently without starting pre-training from scratch.

Original: Out of eight datasets, S LEEP E EG has most complex temporal dynamics [69] and is the largest (371,055 samples).
Revised: Out of eight datasets, S LEEP E EG has most complex temporal dynamics [69] and is the largest (371,055 samples).

Original: For that reason, we pre-train a model on S LEEP E EG and separately fine-tune a well-pre-trained model on E PILEPSY , F D
Revised: For that reason, we pre-train a model on S LEEP E EG and separately fine-tune a well-pre-trained model on E PILEPSY , F D

Original: Results.
Revised: Results.

Original: Results are shown in Table 10.
Revised: Results are shown in Table 10.

Original: Intuitively, there are less common between EEG signals and vibration, acceleration and EMG, we expect that the transfer learning will be less effective compared to the one-to-one evaluation.
Revised: Intuitively, there are less common between EEG signals and vibration, acceleration and EMG, we expect that the transfer learning will be less effective compared to the one-to-one evaluation.

Original: The pre-training and fine-tuning datasets are largely different in the bottom three blocks (S LEEP E EG → {F D - B , G ESTURE , E MG }).
Revised: The pre-training and fine-tuning datasets are largely different in the bottom three blocks (S LEEP E EG → {F D - B , G ESTURE , E MG }).

Original: The large gap reasonably leads to a deterioration in baseline performances, however, our model has obviously higher tolerance to knowledge transfer across datasets with large difference.
Revised: The large gap reasonably leads to a deterioration in baseline performances, however, our model has obviously higher tolerance to knowledge transfer across datasets with large difference.

Original: Notably, We find that the proposed model with TF-C earned best performance in 14 out of 18 settings in the three challenging settings: indicating our TF-C assumption is universal in time series.
Revised: Notably, We find that the proposed model with TF-C earned best performance in 14 out of 18 settings in the three challenging settings: indicating our TF-C assumption is universal in time series.

Original: For example, our approach outperforms the strongest baseline by 8.4% (in precision) when fine-tuning on G ESTURE .
Revised: For example, our approach outperforms the strongest baseline by 8.4% (in precision) when fine-tuning on G ESTURE .

Original: Our model has large potential to serve as a universal model when there is no large pre-training dataset that is similar to the small fine-tuning dataset.
Revised: Our model has large potential to serve as a universal model when there is no large pre-training dataset that is similar to the small fine-tuning dataset.

Original: Further, the TF-C approach consistently outperforms KNN and Random
Revised: Further, the TF-C approach consistently outperforms KNN and Random

Original: Init.
Revised: Init.

Original: (which are not pre-trained) by a large margin of 42.8% and 25.1% (both in F1 score) on average.
Revised: (which are not pre-trained) by a large margin of 42.8% and 25.1% (both in F1 score) on average.

Original: Ablation study.
Revised: Ablation study.

Original: In the context of one-to-one classification setting, we wanted to evaluate how relevant model component are for effective pre-training.
Revised: In the context of one-to-one classification setting, we wanted to evaluate how relevant model component are for effective pre-training.

Original: As shown in Table 9 (S LEEP E EG → G ESTURE ), removing L C , L T , and L F result in performance degradation (precision) of 6.1%, 7.2%, and 6.7%, respectively.
Revised: As shown in Table 9 (S LEEP E EG → G ESTURE ), removing L C , L T , and L F result in performance degradation (precision) of 6.1%, 7.2%, and 6.7%, respectively.

Original: To validate that the performance increment is not solely brought by a third loss term no matter what consistency it measures, we replaced consistency loss L C with a loss term measuring the consistency within time space (named L TT-C ) or within frequency space (named L FF-C ).
Revised: To validate that the performance increment is not solely brought by a third loss term no matter what consistency it measures, we replaced consistency loss L C with a loss term measuring the consistency within time space (named L TT-C ) or within frequency space (named L FF-C ).

Original: Results show our consistency loss outperforms L TT-C and L FF-C by 5.3% and 7.2% (accuracy), respectively.
Revised: Results show our consistency loss outperforms L TT-C and L FF-C by 5.3% and 7.2% (accuracy), respectively.

Original: We also find that full fine-tuning (optimizing the whole model) performs slightly better than partial fine-tuning (updating only final layers in R T and R F ), by 5.4% in F1 score.
Revised: We also find that full fine-tuning (optimizing the whole model) performs slightly better than partial fine-tuning (updating only final layers in R T and R F ), by 5.4% in F1 score.

Original: More details in Appendix H.
Revised: More details in Appendix H.

Original: 5.3 Extensive Downstream Tasks: Clustering and Anomaly Detection
Revised: 5.3 Extensive Downstream Tasks: Clustering and Anomaly Detection

Original: Clustering Task.
Revised: Clustering Task.

Original: We evaluate the clustering performance of TF-C taking S LEEP E EG → E PILEPSY as an example.
Revised: We evaluate the clustering performance of TF-C taking S LEEP E EG → E PILEPSY as an example.

Original: Specifically, we added a K-means (K=2), as Epilepsy has 2 classes, on top of z tune i in fine-tuning.
Revised: Specifically, we added a K-means (K=2), as Epilepsy has 2 classes, on top of z tune i in fine-tuning.

Original: We adopt commonly used evaluation metrics: Silhouette score, Adjusted Rand
Revised: We adopt commonly used evaluation metrics: Silhouette score, Adjusted Rand

Original: Index (ARI), and Normalized Mutual Information (NMI).
Revised: Index (ARI), and Normalized Mutual Information (NMI).

Original: Table 7 shows our TF-C obtains the best clustering surpassing the strongest baseline (TS-TCC) by a large margin (5.4% in Silhouette score).
Revised: Table 7 shows our TF-C obtains the best clustering surpassing the strongest baseline (TS-TCC) by a large margin (5.4% in Silhouette score).

Original: It conveys that TF-C can capture more distinctive representations with the knowledge transferred from pre-training, which is consistent with the superiority of TF-C in above classification tasks.
Revised: It conveys that TF-C can capture more distinctive representations with the knowledge transferred from pre-training, which is consistent with the superiority of TF-C in above classification tasks.

Original: Anomaly Detection Task.
Revised: Anomaly Detection Task.

Original: Moreover, we evaluate how TF-C performs for anomaly detection.
Revised: Moreover, we evaluate how TF-C performs for anomaly detection.

Original: Note we work on the sample-level rather than observation-level anomaly detection.
Revised: Note we work on the sample-level rather than observation-level anomaly detection.

Original: Based on global patterns, the former aims to detect abnormal time series samples instead of outlier observations in the samples (as in BTSF [50] and USAD [70]), which emphasizes the local context.
Revised: Based on global patterns, the former aims to detect abnormal time series samples instead of outlier observations in the samples (as in BTSF [50] and USAD [70]), which emphasizes the local context.

Original: To examine our TF-C for anomaly detection, we chose a different setup (F D - A → F D - B ) different from the setup (S LEEP E EG → E PILEPSY ) used for clustering above.
Revised: To examine our TF-C for anomaly detection, we chose a different setup (F D - A → F D - B ) different from the setup (S LEEP E EG → E PILEPSY ) used for clustering above.

Original: Specifically, we built a small subset of F D - B with 1,000 samples, of which 900 are from undamaged bearings, and the remainingare from bearings with inner or outer damage.
Revised: Specifically, we built a small subset of F D - B with 1,000 samples, of which 900 are from undamaged bearings, and the remainingare from bearings with inner or outer damage.

Original: Undamaged samples are considered “normal,” and inner/outer damaged samples are “outliers.”
Revised: Undamaged samples are considered “normal,” and inner/outer damaged samples are “outliers.”

Original: In fine-tuning, we used one-class SVM on top of learned representations z tune i .
Revised: In fine-tuning, we used one-class SVM on top of learned representations z tune i .

Original: The experimental results (Table 8) show that our TF-C outperforms five competitive baselines with up to 4.5% in F-1 Score.
Revised: The experimental results (Table 8) show that our TF-C outperforms five competitive baselines with up to 4.5% in F-1 Score.

Original: Results show that the proposed TF-C is more sensitive to anomalous samples and can effectively detect the abnormal status in mechanical devices.
Revised: Results show that the proposed TF-C is more sensitive to anomalous samples and can effectively detect the abnormal status in mechanical devices.

Original:  Conclusion
Revised:  Conclusion

Original: We develop a pre-training approach that introduces time-frequency consistency (TF-C) as a mechanism to support knowledge transfer between time-series datasets.
Revised: We develop a pre-training approach that introduces time-frequency consistency (TF-C) as a mechanism to support knowledge transfer between time-series datasets.

Original: The approach uses self-supervised contrastive estimation and injects TF-C into a pre-training model, bringing the encoded time-based and frequency-based representations and their local neighborhoods close together in the latent space.
Revised: The approach uses self-supervised contrastive estimation and injects TF-C into a pre-training model, bringing the encoded time-based and frequency-based representations and their local neighborhoods close together in the latent space.

Original: Limitations and future directions.
Revised: Limitations and future directions.

Original: We posit that TF-C property can serve as a generalizable property for pre-training on diverse time series datasets.
Revised: We posit that TF-C property can serve as a generalizable property for pre-training on diverse time series datasets.

Original: We are aware that there might exist additional generalizable properties, such as temporal autoregressive processes, that could also prove useful for pre-training on time series.
Revised: We are aware that there might exist additional generalizable properties, such as temporal autoregressive processes, that could also prove useful for pre-training on time series.

Original: Moreover, our method expects the input time series to be regularly sampled.
Revised: Moreover, our method expects the input time series to be regularly sampled.

Original: However, our model can handle irregularly sampled time series using an encoder (such as
Revised: However, our model can handle irregularly sampled time series using an encoder (such as

Original: Raindrop [71] and SeFT [72]) that can embed irregular time series.
Revised: Raindrop [71] and SeFT [72]) that can embed irregular time series.

Original: For frequency encoder inputs x F i , possible alternatives include resampling or interpolation to obtain regularly sampled signals and using regular or non-uniform FFT operations.
Revised: For frequency encoder inputs x F i , possible alternatives include resampling or interpolation to obtain regularly sampled signals and using regular or non-uniform FFT operations.

Original: Furthermore, we agree that the current embedding scheme and loss functions are favorable for classification tasks that leverage global information over tasks that use local context (e.g., forecasting).
Revised: Furthermore, we agree that the current embedding scheme and loss functions are favorable for classification tasks that leverage global information over tasks that use local context (e.g., forecasting).

Original: Even though, we demonstrate that our model achieves superior performance in broad downstream tasks such as clustering and anomaly detection (Sec. 5.3).
Revised: Even though, we demonstrate that our model achieves superior performance in broad downstream tasks such as clustering and anomaly detection (Sec. 5.3).

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: M.Z. and X.Z. are supported, in part, by NSF under Nos.
Revised: M.Z. and X.Z. are supported, in part, by NSF under Nos.

Original: IIS-2030459 and IIS-2033384, US Air Force
Revised: IIS-2030459 and IIS-2033384, US Air Force

Original: Contract No. FA8702-15-D-0001, Harvard Data Science Initiative, Amazon Research Award, Bayer
Revised: Contract No. FA8702-15-D-0001, Harvard Data Science Initiative, Amazon Research Award, Bayer

Original: Early Excellence in Science Award, AstraZeneca Research, and Roche Alliance with Distinguished
Revised: Early Excellence in Science Award, AstraZeneca Research, and Roche Alliance with Distinguished

Original: Scientists Award.
Revised: Scientists Award.

Original: T.T. is supported by the Under Secretary of Defense for Research and Engineering under US Air Force Contract No. FA8702-15-D-0001.
Revised: T.T. is supported by the Under Secretary of Defense for Research and Engineering under US Air Force Contract No. FA8702-15-D-0001.

Original: Broader Impacts
Revised: Broader Impacts

Original: Our proposed novel framework for time series unsupervised pre-training improves classification performance on target datasets in different application scenarios.
Revised: Our proposed novel framework for time series unsupervised pre-training improves classification performance on target datasets in different application scenarios.

Original: The recognition of time-frequency consistency as a universal property specific to time series data is a weak assumption that enables effective, task- and domain-agnostic transfer learning.
Revised: The recognition of time-frequency consistency as a universal property specific to time series data is a weak assumption that enables effective, task- and domain-agnostic transfer learning.

Original: We believe our work will inspire the research community to uncover other universal properties for transfer learning.
Revised: We believe our work will inspire the research community to uncover other universal properties for transfer learning.

Original: We also hope our work will also attract more researchers to the more general problem of time series representation learning which is still underappreciated relative to problems from CV and NLP fields.
Revised: We also hope our work will also attract more researchers to the more general problem of time series representation learning which is still underappreciated relative to problems from CV and NLP fields.

Original: On the society level, our work, along the line of transfer learning, can facilitate more efficient use of time series data in various settings.
Revised: On the society level, our work, along the line of transfer learning, can facilitate more efficient use of time series data in various settings.

Original: For example, in medical settings, some diseases of clinical interest may have very small labelled dataset.
Revised: For example, in medical settings, some diseases of clinical interest may have very small labelled dataset.

Original: In this case, unlabelled data from patients of different diseases but with similar underlying physiological conditions can be used to pre-train the model.
Revised: In this case, unlabelled data from patients of different diseases but with similar underlying physiological conditions can be used to pre-train the model.

Original: However, practitioners need to be aware of the limitations of the model, including that it may make biased predictions.
Revised: However, practitioners need to be aware of the limitations of the model, including that it may make biased predictions.

Original: Specifically, bias may exist in the source dataset used for pre-training due to an imbalance of samples from subjects of different demographic attributes.
Revised: Specifically, bias may exist in the source dataset used for pre-training due to an imbalance of samples from subjects of different demographic attributes.

Original: Also, the standardized medical protocols for collecting these datasets might be unsuitable for subjects with certain physiological attributes, creating unforseen bias that may be transferred to the finetuning phase.
Revised: Also, the standardized medical protocols for collecting these datasets might be unsuitable for subjects with certain physiological attributes, creating unforseen bias that may be transferred to the finetuning phase.

Original: All datasets in this paper are publicly available and are not associated with any privacy or security concern.
Revised: All datasets in this paper are publicly available and are not associated with any privacy or security concern.

Original: Furthermore, we have followed guidelines on responsible use specified by primary authors of the datasets used in the current work.
Revised: Furthermore, we have followed guidelines on responsible use specified by primary authors of the datasets used in the current work.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] In abstract and introduction, we claim that TF-C is a generalizable property of time series that can support pre-training, which is welljustified in Sec.
Revised: [Yes] In abstract and introduction, we claim that TF-C is a generalizable property of time series that can support pre-training, which is welljustified in Sec.

Original: 3 and experimentally demonstrated in Sec. 5 (our model consistently performs comparatively to or above baseline methods).
Revised: 3 and experimentally demonstrated in Sec. 5 (our model consistently performs comparatively to or above baseline methods).

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 6.
Revised: [Yes] See Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Broader Impact on Page 10.
Revised: [Yes] See Broader Impact on Page 10.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Yes, we include an anonymous link (see Abstract) that provides the source codes with all implementation details, implementation of baselines, and eight datasets.
Revised: [Yes] Yes, we include an anonymous link (see Abstract) that provides the source codes with all implementation details, implementation of baselines, and eight datasets.

Original: The link will be updated to an non-anonymous link after acceptance.
Revised: The link will be updated to an non-anonymous link after acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?

Original: [Yes] See implementation details in Sec. 5.
Revised: [Yes] See implementation details in Sec. 5.

Original: See Appendix E for baseline architectures and hyper-parameter settings.
Revised: See Appendix E for baseline architectures and hyper-parameter settings.

Original: More details can be found in the included URL.
Revised: More details can be found in the included URL.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We run experiments for 5 times and report the average value with standard deviation.
Revised: [Yes] We run experiments for 5 times and report the average value with standard deviation.

Original: See Table 1, Tables 4-6, and Table 10.
Revised: See Table 1, Tables 4-6, and Table 10.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix E. 4.
Revised: [Yes] See Appendix E. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We used eight existing datasets and 6 state-of-the-art baselines in contrastive learning and pre-training for time series.
Revised: [Yes] We used eight existing datasets and 6 state-of-the-art baselines in contrastive learning and pre-training for time series.

Original: We cited the creators for every exist asset we used.
Revised: We cited the creators for every exist asset we used.

Original: See Sec.
Revised: See Sec.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] All dataset licenses are mentioned in the Appendix D. (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] All dataset licenses are mentioned in the Appendix D. (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] See the anonymous link in Abstract.
Revised: [Yes] See the anonymous link in Abstract.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [No] All data we use is freely available for download, without any requirement to re-contact the data curator.
Revised: [No] All data we use is freely available for download, without any requirement to re-contact the data curator.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No] Our datasets are public, well-established, and do not contain PII or offensive content 5.
Revised: [No] Our datasets are public, well-established, and do not contain PII or offensive content 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: Appendix A Further information on the relationship between our pre-training approach and domain adaptation
Revised: Appendix A Further information on the relationship between our pre-training approach and domain adaptation

Original: Here we note our problem definition of pre-training is fundamentally different from domain adaptation [S1, S2, S3, S4, S5, S6] 1 in order to prevent any confusion between this work and domain adaptation methods.
Revised: Here we note our problem definition of pre-training is fundamentally different from domain adaptation [S1, S2, S3, S4, S5, S6] 1 in order to prevent any confusion between this work and domain adaptation methods.

Original: DA applies a model trained on a pre-training dataset ( i.e. , source dataset) to a different target dataset [21, 42].
Revised: DA applies a model trained on a pre-training dataset ( i.e. , source dataset) to a different target dataset [21, 42].

Original: In contrast, self-supervised pre-training has four key differences with domain adaptation.
Revised: In contrast, self-supervised pre-training has four key differences with domain adaptation.

Original: (1) First, our model only requires the pre-training dataset while domain adaptation techniques generally require access to the target dataset [S7, S8, S9, S10, S11, S12, S13, S14, S15, S16].
Revised: (1) First, our model only requires the pre-training dataset while domain adaptation techniques generally require access to the target dataset [S7, S8, S9, S10, S11, S12, S13, S14, S15, S16].

Original: (2) Second, our model can be applied to multiple unseen target datasets (without re-training the pre-trained model for every target dataset) while domain adaptation approaches use the target dataset during model training, e.g. , [S17, S18, S19, S20, S12, S14, S21] (see also Sec.
Revised: (2) Second, our model can be applied to multiple unseen target datasets (without re-training the pre-trained model for every target dataset) while domain adaptation approaches use the target dataset during model training, e.g. , [S17, S18, S19, S20, S12, S14, S21] (see also Sec.

Original: 5.2 for experimental results).
Revised: 5.2 for experimental results).

Original: (3) Third, our approach can be used in scenarios where the feature space in pre-training is different from that in the target dataset (see Scenarios 1, 3, and 4 in Sec.
Revised: (3) Third, our approach can be used in scenarios where the feature space in pre-training is different from that in the target dataset (see Scenarios 1, 3, and 4 in Sec.

Original: 5.1 for experimental results).
Revised: 5.1 for experimental results).

Original: In contrast, domain adaptation methods usually restrict pre-training and target datasets to have the same feature space (but possible different distributions), e.g. , [S22, S18, S19, S20, S13].
Revised: In contrast, domain adaptation methods usually restrict pre-training and target datasets to have the same feature space (but possible different distributions), e.g. , [S22, S18, S19, S20, S13].

Original: In summary, to support transfer learning across different time series datasets, a pre-training approach needs a capability to capture a generalizable property of time series, one that is shared across different time series datasets regardless of the specific semantic meaning of a time series signal ( e.g. , ECG, EMG, acceleration, vibration), conditions of data acquisition ( e.g. , variation across subjects and devices), sampling frequencies, etc.
Revised: In summary, to support transfer learning across different time series datasets, a pre-training approach needs a capability to capture a generalizable property of time series, one that is shared across different time series datasets regardless of the specific semantic meaning of a time series signal ( e.g. , ECG, EMG, acceleration, vibration), conditions of data acquisition ( e.g. , variation across subjects and devices), sampling frequencies, etc.

Original: This work develops a self-supervised contrastive pre-training strategy that fulfills these requirements by injecting an appropriate inductive bias (called
Revised: This work develops a self-supervised contrastive pre-training strategy that fulfills these requirements by injecting an appropriate inductive bias (called

Original: Time-Frequency Consistency, TF-C, into the model (Sec. 3).
Revised: Time-Frequency Consistency, TF-C, into the model (Sec. 3).

Original: Further, we clarify that the term ‘self-supervised’ has different meanings in DA and in pretraining [S23, S24, S25, S26].
Revised: Further, we clarify that the term ‘self-supervised’ has different meanings in DA and in pretraining [S23, S24, S25, S26].

Original: The ‘self-supervised domain adaptation’ [S27, S16, S21, S15] or ‘unsupervised domain adaptation’ [S1, S22, S28, S11, S14] means that there are no labels in the target dataset, however that still requires labels in the pre-training dataset.
Revised: The ‘self-supervised domain adaptation’ [S27, S16, S21, S15] or ‘unsupervised domain adaptation’ [S1, S22, S28, S11, S14] means that there are no labels in the target dataset, however that still requires labels in the pre-training dataset.

Original: In contrast, ‘self-supervised pretraining’
Revised: In contrast, ‘self-supervised pretraining’

Original: [S29, S30, S31] ( i.e. , the problem studied here, in line with a breadth of existing literature on pre-training) indicates the setting where no labels are available in pre-training.
Revised: [S29, S30, S31] ( i.e. , the problem studied here, in line with a breadth of existing literature on pre-training) indicates the setting where no labels are available in pre-training.

Original: Appendix B Detailed differences with CoST and BTSF
Revised: Appendix B Detailed differences with CoST and BTSF

Original: Up to the submission of this manuscript, there is no existing contrastive augmentations in time series’ frequency domain.
Revised: Up to the submission of this manuscript, there is no existing contrastive augmentations in time series’ frequency domain.

Original: There are two models, CoST [49] and BTSF [50], that involved frequency domain in contrastive learning, however, the proposed TF-C is fundamentally different with them in the following aspects.
Revised: There are two models, CoST [49] and BTSF [50], that involved frequency domain in contrastive learning, however, the proposed TF-C is fundamentally different with them in the following aspects.

Original: We take BTSF as an example while the differences also apply to CoST.
Revised: We take BTSF as an example while the differences also apply to CoST.

Original: • Problem definitions for both papers are different.
Revised: • Problem definitions for both papers are different.

Original: Our method is designed to produce generalizable representations that can transfer to a different time series dataset (going from pre-training to a fine-tuning dataset) for the purpose of transfer learning.
Revised: Our method is designed to produce generalizable representations that can transfer to a different time series dataset (going from pre-training to a fine-tuning dataset) for the purpose of transfer learning.

Original: In contrast, BTSF attempts to learn embeddings within the same dataset for the purpose of representation learning.
Revised: In contrast, BTSF attempts to learn embeddings within the same dataset for the purpose of representation learning.

Original: Our model captures the TF-C property invariant to different time series (in terms of various temporal dynamics, semantic meaning, etc.) and can thus serve as a vehicle for transfer learning.
Revised: Our model captures the TF-C property invariant to different time series (in terms of various temporal dynamics, semantic meaning, etc.) and can thus serve as a vehicle for transfer learning.

Original: In contrast, BTSF learns embeddings invariant to perturbations (i
Revised: In contrast, BTSF learns embeddings invariant to perturbations (i

Original: e., instance-level dropout) of the same time series.
Revised: e., instance-level dropout) of the same time series.

Original: • The modeling of the frequency domain is different in both papers.
Revised: • The modeling of the frequency domain is different in both papers.

Original: We developed augmentations in the frequency domain based on the spectral properties of time series.
Revised: We developed augmentations in the frequency domain based on the spectral properties of time series.

Original: In contrast, although BTSF involves a frequency domain, its data transformation is solely implemented in the time domain (using instance-level dropout; Sec. 3.1 in BTSF).
Revised: In contrast, although BTSF involves a frequency domain, its data transformation is solely implemented in the time domain (using instance-level dropout; Sec. 3.1 in BTSF).

Original: That is, the BTSF method applies the FFT after augmenting samples in the time domain which can lead to information loss.
Revised: That is, the BTSF method applies the FFT after augmenting samples in the time domain which can lead to information loss.

Original: • BTSF emphasizes fusing temporal and spectral features to generate discriminative embeddings.
Revised: • BTSF emphasizes fusing temporal and spectral features to generate discriminative embeddings.

Original: Unlike BTSF, our model leverages the consistency between time-based and frequency-based embeddings to produce generalizable time series representations.
Revised: Unlike BTSF, our model leverages the consistency between time-based and frequency-based embeddings to produce generalizable time series representations.

Original: Our
Revised: Our

Original: model maps every sample to a time-frequency embedding space and constrains the relative relationships between embeddings (through triplet loss) according to the TF-C property.
Revised: model maps every sample to a time-frequency embedding space and constrains the relative relationships between embeddings (through triplet loss) according to the TF-C property.

Original: The underlying consistency allows TF-C to realize transfer learning across time series datasets, which is TF-C’s unique advantage
Revised: The underlying consistency allows TF-C to realize transfer learning across time series datasets, which is TF-C’s unique advantage

Original: In summary, TF-C introduces frequency domain augmentations in the sense that it directly perturbs the frequency spectrum.
Revised: In summary, TF-C introduces frequency domain augmentations in the sense that it directly perturbs the frequency spectrum.

Original: TF-C is the first method that uses frequency domain augmentations to enable transfer learning in time series.
Revised: TF-C is the first method that uses frequency domain augmentations to enable transfer learning in time series.

Original: Appendix C Time series invariance between time and frequency domains
Revised: Appendix C Time series invariance between time and frequency domains

Original: Here, we provide an analogy, from images to time series, to aid understanding of the ‘invariance’ property of data representations.
Revised: Here, we provide an analogy, from images to time series, to aid understanding of the ‘invariance’ property of data representations.

Original: In computer vision, it is well known that an image and its augmented views obtained by simple transformations (e.g., rotation, translation, scaling, etc.) can be used in different frameworks such as transformation prediction and contrastive instance discrimination to obtain invariant representations.
Revised: In computer vision, it is well known that an image and its augmented views obtained by simple transformations (e.g., rotation, translation, scaling, etc.) can be used in different frameworks such as transformation prediction and contrastive instance discrimination to obtain invariant representations.

Original: These transformations are used as augmentations to guide the learning of self-supervised representations.
Revised: These transformations are used as augmentations to guide the learning of self-supervised representations.

Original: For transformation prediction, the model learns representations that are equivariant to the selected transformation as the information embedded in the transformation needs to be embedded in the representation for the final layer to solve the pretext task.
Revised: For transformation prediction, the model learns representations that are equivariant to the selected transformation as the information embedded in the transformation needs to be embedded in the representation for the final layer to solve the pretext task.

Original: For contrastive instance discrimination, the representations are sensitive to the instances while learning invariances to transformations or views.
Revised: For contrastive instance discrimination, the representations are sensitive to the instances while learning invariances to transformations or views.

Original: The intuition behind such transformations is that the underlying object in the image is the same no matter how the image is rotated, translated, re-scaled, etc.
Revised: The intuition behind such transformations is that the underlying object in the image is the same no matter how the image is rotated, translated, re-scaled, etc.

Original: In other words, the information carried by the original image and the transformed image is the same.
Revised: In other words, the information carried by the original image and the transformed image is the same.

Original: Similarly, the time domain and frequency domain representations carry the same information in time series.
Revised: Similarly, the time domain and frequency domain representations carry the same information in time series.

Original: Thus it is reasonable to expect that by exploring local neighborhoods in the time domain and frequency domain and enforcing consistency of feature representations inter- and intra- domains, invariance properties are captured in our model via self-supervised learning.
Revised: Thus it is reasonable to expect that by exploring local neighborhoods in the time domain and frequency domain and enforcing consistency of feature representations inter- and intra- domains, invariance properties are captured in our model via self-supervised learning.

Original: Thus, information carried in time and frequency domains of the same or similar time series sample should be the same.
Revised: Thus, information carried in time and frequency domains of the same or similar time series sample should be the same.

Original: This invariance, formalized as ‘Time-Frequency Consistency’, is helpful for self-supervised pre-training.
Revised: This invariance, formalized as ‘Time-Frequency Consistency’, is helpful for self-supervised pre-training.

Original: We will include the above discussion in the camera-ready version.
Revised: We will include the above discussion in the camera-ready version.

Original: Appendix D Additional information on datasets and pre-training evaluation D.1 Datasets
Revised: Appendix D Additional information on datasets and pre-training evaluation D.1 Datasets

Original: We use eight diverse time series datasets to evaluate our model.
Revised: We use eight diverse time series datasets to evaluate our model.

Original: The datasets used in one-to-one and one-to-many pre-training evaluations are the same.
Revised: The datasets used in one-to-one and one-to-many pre-training evaluations are the same.

Original: The dataset statistics are shown in Table 3.
Revised: The dataset statistics are shown in Table 3.

Original: Processed model-ready datasets are in our GitHub Repository ( https://anonymous.4open.
Revised: Processed model-ready datasets are in our GitHub Repository ( https://anonymous.4open.

Original: science/r/TFC-pre-training-6B07 ).
Revised: science/r/TFC-pre-training-6B07 ).

Original: Following is a detailed description of datasets.
Revised: Following is a detailed description of datasets.

Original: S LEEP E EG [61].
Revised: S LEEP E EG [61].

Original: The dataset contains 153 whole-night sleeping electroencephalography (EEG) recordings produced by a sleep cassette.
Revised: The dataset contains 153 whole-night sleeping electroencephalography (EEG) recordings produced by a sleep cassette.

Original: Data are collected from 82 healthy subjects.
Revised: Data are collected from 82 healthy subjects.

Original: The 1-lead
Revised: The 1-lead

Original: EEG signal is sampled at 100 Hz.
Revised: EEG signal is sampled at 100 Hz.

Original: We segment the EEG signals into segments (window size is 200) without overlapping, and each segment forms a sample.
Revised: We segment the EEG signals into segments (window size is 200) without overlapping, and each segment forms a sample.

Original: Every sample is associated with one of the five sleeping patterns/stages:
Revised: Every sample is associated with one of the five sleeping patterns/stages:

Original: Wake (W), Non-rapid eye movement (N1, N2, N3), and
Revised: Wake (W), Non-rapid eye movement (N1, N2, N3), and

Original: Rapid Eye Movement (REM).
Revised: Rapid Eye Movement (REM).

Original: After segmentation, we have 371,055 EEG samples.
Revised: After segmentation, we have 371,055 EEG samples.

Original: The raw dataset ( https://www.physionet.org/content/sleep-edfx/1.0.0/ ) is distributed under the Open Data Commons Attribution License v1.0.
Revised: The raw dataset ( https://www.physionet.org/content/sleep-edfx/1.0.0/ ) is distributed under the Open Data Commons Attribution License v1.0.

Original: E PILEPSY [62].
Revised: E PILEPSY [62].

Original: The dataset contains single-channel EEG measurements from 500 subjects.
Revised: The dataset contains single-channel EEG measurements from 500 subjects.

Original: For every subject, the brain activity was recorded for 23.6 seconds.
Revised: For every subject, the brain activity was recorded for 23.6 seconds.

Original: The dataset was then divided and shuffled (to mitigate sample-subject association) into 11,500 samples of 1 second each, sampled at 178 Hz.
Revised: The dataset was then divided and shuffled (to mitigate sample-subject association) into 11,500 samples of 1 second each, sampled at 178 Hz.

Original: The raw dataset features five classification labels corresponding to different states of subjects or measurement locations — eyes open, eyes closed, EEG measured in the healthy brain region, EEG measured in the tumor region, and whether the subject has a seizure episode.
Revised: The raw dataset features five classification labels corresponding to different states of subjects or measurement locations — eyes open, eyes closed, EEG measured in the healthy brain region, EEG measured in the tumor region, and whether the subject has a seizure episode.

Original: To emphasize the distinction between positive and negative samples in terms of epilepsy, We merge the first four classes into one, and each time series sample has a binary label describing if the
Revised: To emphasize the distinction between positive and negative samples in terms of epilepsy, We merge the first four classes into one, and each time series sample has a binary label describing if the

Original: [Table]
Revised: [Table]

Original: associated subject is experiencing a seizure or not.
Revised: associated subject is experiencing a seizure or not.

Original: There are 11,500 EEG samples in total.
Revised: There are 11,500 EEG samples in total.

Original: To evaluate the performance of a pre-trained model on a small fine-tuning dataset, we choose a small set (60 samples; 30 samples for each class) for fine-tuning and assess the model with a validation set (20 samples; 10 samples for each class).
Revised: To evaluate the performance of a pre-trained model on a small fine-tuning dataset, we choose a small set (60 samples; 30 samples for each class) for fine-tuning and assess the model with a validation set (20 samples; 10 samples for each class).

Original: Finally, the model with the best validation performance is used to make predictions on the test set ( i.e. , the remaining 11,420 samples).
Revised: Finally, the model with the best validation performance is used to make predictions on the test set ( i.e. , the remaining 11,420 samples).

Original: Statistics of finetuning, validation
Revised: Statistics of finetuning, validation

Original: [Table]
Revised: [Table]

Original: and test sets in the other three target datasets are in Appendix 3.
Revised: and test sets in the other three target datasets are in Appendix 3.

Original: The raw dataset ( https://repositori.upf.edu/handle/10230/42894 ) is distributed under the
Revised: The raw dataset ( https://repositori.upf.edu/handle/10230/42894 ) is distributed under the

Original: Creative Commons License (CC-BY) 4.0.
Revised: Creative Commons License (CC-BY) 4.0.

Original: F D - A and F D - B [63].
Revised: F D - A and F D - B [63].

Original: The dataset is generated by an electromechanical drive system that monitors the condition of rolling bearings and detects their failures.
Revised: The dataset is generated by an electromechanical drive system that monitors the condition of rolling bearings and detects their failures.

Original: Four subsets of data are collected under various conditions, whose parameters include rotational speed, load torque, and radial force.
Revised: Four subsets of data are collected under various conditions, whose parameters include rotational speed, load torque, and radial force.

Original: Each rolling bearing can be undamaged, inner damaged, and outer damaged, which leads to three classes in total.
Revised: Each rolling bearing can be undamaged, inner damaged, and outer damaged, which leads to three classes in total.

Original: We denote the subsets corresponding to condition A and condition B as Faulty Detection Condition A ( F D - A ) and Faulty Detection Condition B ( F D - B ), respectively.
Revised: We denote the subsets corresponding to condition A and condition B as Faulty Detection Condition A ( F D - A ) and Faulty Detection Condition B ( F D - B ), respectively.

Original: Each original recording has a single channel with a sampling frequency of 64k
Revised: Each original recording has a single channel with a sampling frequency of 64k

Original: Hz and lasts 4 seconds.
Revised: Hz and lasts 4 seconds.

Original: To deal with lengthy recordings, we follow the procedure described by Eldele et al.
Revised: To deal with lengthy recordings, we follow the procedure described by Eldele et al.

Original: Specifically, we use a sliding window length of 5,120 observations and a shifting length of 1,024 or 4,096 to ensure that samples are relatively balanced between classes.
Revised: Specifically, we use a sliding window length of 5,120 observations and a shifting length of 1,024 or 4,096 to ensure that samples are relatively balanced between classes.

Original: The raw dataset ( https://mb.uni-paderborn.de/en/kat/main-research/ datacenter/bearing-datacenter/data-sets-and-download ) is distributed under the Creative Commons Attribution-Non Commercial 4.0 International License.
Revised: The raw dataset ( https://mb.uni-paderborn.de/en/kat/main-research/ datacenter/bearing-datacenter/data-sets-and-download ) is distributed under the Creative Commons Attribution-Non Commercial 4.0 International License.

Original: H AR [64].
Revised: H AR [64].

Original: This dataset contains recordings of 30 health volunteers performing daily activities, including walking, walking upstairs, walking downstairs, sitting, standing, and lying.
Revised: This dataset contains recordings of 30 health volunteers performing daily activities, including walking, walking upstairs, walking downstairs, sitting, standing, and lying.

Original: Prediction labels are the six activities.
Revised: Prediction labels are the six activities.

Original: The wearable sensors on a smartphone measure triaxial linear acceleration and triaxial angular velocity at 50 Hz.
Revised: The wearable sensors on a smartphone measure triaxial linear acceleration and triaxial angular velocity at 50 Hz.

Original: After preprocessing and isolating gravitational acceleration from body acceleration, there are nine channels ( i.e. , 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer) in total.
Revised: After preprocessing and isolating gravitational acceleration from body acceleration, there are nine channels ( i.e. , 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer) in total.

Original: The raw dataset ( https://archive.ics.uci.edu/ml/ datasets/Human+Activity+Recognition+Using+Smartphones ) is distributed as-is.
Revised: The raw dataset ( https://archive.ics.uci.edu/ml/ datasets/Human+Activity+Recognition+Using+Smartphones ) is distributed as-is.

Original: Any commercial use is not allowed.
Revised: Any commercial use is not allowed.

Original: G ESTURE [65].
Revised: G ESTURE [65].

Original: The dataset contains accelerometer
Revised: The dataset contains accelerometer

Original: [Table]
Revised: [Table]

Original: and fine-tuning datasets)
Revised: and fine-tuning datasets)

Original: ; otherwise, if the model ( i.e. , KNN, TS-SD, Mixing-up, and TS2vec) don’t have channel generalization ability, we only use three acceleration channels in H AR for pre-training.
Revised: ; otherwise, if the model ( i.e. , KNN, TS-SD, Mixing-up, and TS2vec) don’t have channel generalization ability, we only use three acceleration channels in H AR for pre-training.

Original: The raw dataset is accessible through http://www.timeseriesclassification.com/ description.php?Dataset=UWaveGestureLibrary .
Revised: The raw dataset is accessible through http://www.timeseriesclassification.com/ description.php?Dataset=UWaveGestureLibrary .

Original: While the distribution license is not explicitly mentioned, the dataset is a public resource based on [65].
Revised: While the distribution license is not explicitly mentioned, the dataset is a public resource based on [65].

Original: E CG [26].
Revised: E CG [26].

Original: This is the 2017 PhysioNet Challenge focusing on classifying ECG recordings.
Revised: This is the 2017 PhysioNet Challenge focusing on classifying ECG recordings.

Original: The singlelead ECG measures four different underlying conditions of cardiac arrhythmias.
Revised: The singlelead ECG measures four different underlying conditions of cardiac arrhythmias.

Original: More specifically, these classes correspond to the recordings of normal sinus rhythm, atrial fibrillation (AF), alternative rhythm, or others (too noisy to be classified).
Revised: More specifically, these classes correspond to the recordings of normal sinus rhythm, atrial fibrillation (AF), alternative rhythm, or others (too noisy to be classified).

Original: The recordings are sampled at 300 Hz.
Revised: The recordings are sampled at 300 Hz.

Original: Furthermore, the dataset is imbalanced, with much fewer samples from the atrial fibrillation and noisy classes out of all four.
Revised: Furthermore, the dataset is imbalanced, with much fewer samples from the atrial fibrillation and noisy classes out of all four.

Original: To preprocess the dataset, we use the code from the CLOCS paper, which applied a fixed-length window of 1,500 observations to divide long recordings into short samples of 5 seconds that are physiologically meaningful.
Revised: To preprocess the dataset, we use the code from the CLOCS paper, which applied a fixed-length window of 1,500 observations to divide long recordings into short samples of 5 seconds that are physiologically meaningful.

Original: Because of the imbalanced dataset, we report AUROC and
Revised: Because of the imbalanced dataset, we report AUROC and

Original: AUPRC (insensitive to label distribution) in our results.
Revised: AUPRC (insensitive to label distribution) in our results.

Original: The raw dataset ( https://physionet.
Revised: The raw dataset ( https://physionet.

Original: org/content/challenge-2017/1.0.0/ ) is distributed under the Open Data Commons
Revised: org/content/challenge-2017/1.0.0/ ) is distributed under the Open Data Commons

Original: Attribution License v1.0.
Revised: Attribution License v1.0.

Original: E MG [66].
Revised: E MG [66].

Original: Electromyograms (EMGs) measure muscle responses as electrical activity in response to neural stimulation, and they can be used to diagnose certain muscular dystrophies and neuropathies.
Revised: Electromyograms (EMGs) measure muscle responses as electrical activity in response to neural stimulation, and they can be used to diagnose certain muscular dystrophies and neuropathies.

Original: The dataset consists of single-channel EMG recording from the tibialis anterior muscle of three healthy volunteers suffering from neuropathy and myopathy.
Revised: The dataset consists of single-channel EMG recording from the tibialis anterior muscle of three healthy volunteers suffering from neuropathy and myopathy.

Original: The recordings are sampled at a frequency of 4K Hz.
Revised: The recordings are sampled at a frequency of 4K Hz.

Original: Each patient ( i.e. , disorder) is a classification category.
Revised: Each patient ( i.e. , disorder) is a classification category.

Original: Then the recordings are split into time-series samples using a fixed-length window of 1,500 observations.
Revised: Then the recordings are split into time-series samples using a fixed-length window of 1,500 observations.

Original: The raw dataset ( https://physionet.org/content/emgdb/1.0.0/ ) is distributed under the Open Data
Revised: The raw dataset ( https://physionet.org/content/emgdb/1.0.0/ ) is distributed under the Open Data

Original: Commons Attribution License v1.0.
Revised: Commons Attribution License v1.0.

Original: D.2 Varying lengths and dimensionality of time series in pre-training vs. target datasets
Revised: D.2 Varying lengths and dimensionality of time series in pre-training vs. target datasets

Original: We use established strategies to address the challenging question of how to process time series with the variable number of dimensions (channels) and measurements (note that this study focuses on designing a pre-training model that can transfer knowledge across disparate time-series datasets).
Revised: We use established strategies to address the challenging question of how to process time series with the variable number of dimensions (channels) and measurements (note that this study focuses on designing a pre-training model that can transfer knowledge across disparate time-series datasets).

Original: When the dimensionality or length of time series is different between pre-training and target datasets, we view the pre-training dataset as an anchor according to which we adjust (pre-process) the target dataset.
Revised: When the dimensionality or length of time series is different between pre-training and target datasets, we view the pre-training dataset as an anchor according to which we adjust (pre-process) the target dataset.

Original: For length adjustments, we use zero padding to increase the number of observations or use downsampling to reduce the number of observations.
Revised: For length adjustments, we use zero padding to increase the number of observations or use downsampling to reduce the number of observations.

Original: For dimensionality adjustments, our model applies to time series at the level of a single lead, enabling the ability to generalize to multivariate time series when we make different channels share the model ( i.e. , time encoder, frequency encoder, and two projectors) parameters.
Revised: For dimensionality adjustments, our model applies to time series at the level of a single lead, enabling the ability to generalize to multivariate time series when we make different channels share the model ( i.e. , time encoder, frequency encoder, and two projectors) parameters.

Original: In this way, we have a z TUNE i for each channel of the time series (Sec. 4) and concatenate the learned representations across all channels to form the final representation of the input time series sample.
Revised: In this way, we have a z TUNE i for each channel of the time series (Sec. 4) and concatenate the learned representations across all channels to form the final representation of the input time series sample.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Appendix E TF-C and baseline architectures and implementation details
Revised: Appendix E TF-C and baseline architectures and implementation details

Original: We compare our TF-C model against eight state-of-the-art baselines and two additional methods (Sec. 5): (1) directly fit a KNN (K=2) classifier with the target (fine-tuning) dataset; (2) randomly initialize a model (the model structure and experimental settings are the same with our TF-C model), ignore pre-training, and directly train from scratch with the target dataset.
Revised: We compare our TF-C model against eight state-of-the-art baselines and two additional methods (Sec. 5): (1) directly fit a KNN (K=2) classifier with the target (fine-tuning) dataset; (2) randomly initialize a model (the model structure and experimental settings are the same with our TF-C model), ignore pre-training, and directly train from scratch with the target dataset.

Original: We implement the baselines follow the corresponding papers including TS-SD [12], TS2vec [47],
Revised: We implement the baselines follow the corresponding papers including TS-SD [12], TS2vec [47],

Original: CLOCS [41], Mixing-up [18], TS-TCC [48], SimCLR [40], TNC [46]
Revised: CLOCS [41], Mixing-up [18], TS-TCC [48], SimCLR [40], TNC [46]

Original: [Table]
Revised: [Table]

Original: and CPC [30].
Revised: and CPC [30].

Original: We use default settings for hyper-parameters as reported in the original works unless noted below.
Revised: We use default settings for hyper-parameters as reported in the original works unless noted below.

Original: All pre-training and fine-tuning with baselines are done with a single Tesla V100 GPU with 32 Gb of allocated memory provided by Harvard Medical School’s O2 High Performance Computing platform.
Revised: All pre-training and fine-tuning with baselines are done with a single Tesla V100 GPU with 32 Gb of allocated memory provided by Harvard Medical School’s O2 High Performance Computing platform.

Original: TF-C (our model)
Revised: TF-C (our model)

Original: Our time-based contrastive encoder F T adopts 1-D ResNet backbone similar to
Revised: Our time-based contrastive encoder F T adopts 1-D ResNet backbone similar to

Original: SimCLR.
Revised: SimCLR.

Original: Specifically, after hyper-parameter tuning, the encoder consists of three layers convolutional blocks: the kernel sizes in all layers are 8; the strides are 8, 1, and 1, respectively; the the depths are 32, 64, and 128, respectively.
Revised: Specifically, after hyper-parameter tuning, the encoder consists of three layers convolutional blocks: the kernel sizes in all layers are 8; the strides are 8, 1, and 1, respectively; the the depths are 32, 64, and 128, respectively.

Original: We use max pooling after each convolutional layer and all the pooling kernel sizes and strides are set as 2.
Revised: We use max pooling after each convolutional layer and all the pooling kernel sizes and strides are set as 2.

Original: The cross-space projector R T contains two fully-connected layers with hidden dimensions 256 and 128, respectively.
Revised: The cross-space projector R T contains two fully-connected layers with hidden dimensions 256 and 128, respectively.

Original: In the transformation from time space to frequency space, we use the full spectrum (symmetrical) thus x T and x F have the same dimension.
Revised: In the transformation from time space to frequency space, we use the full spectrum (symmetrical) thus x T and x F have the same dimension.

Original: For simplify, the frequency encoder F F and projector R F have same structure (but different parameters) with their counterparts in time space.
Revised: For simplify, the frequency encoder F F and projector R F have same structure (but different parameters) with their counterparts in time space.

Original: Our preliminary experiments show that the change of model structure has relatively small change on performance.
Revised: Our preliminary experiments show that the change of model structure has relatively small change on performance.

Original: For example, replacing F T by 2-layer LSTM or 3-layer
Revised: For example, replacing F T by 2-layer LSTM or 3-layer

Original: Transformer only cause a slight drop of F1 score ( ± 1.2%; one-to-one setting; Scenario 1).
Revised: Transformer only cause a slight drop of F1 score ( ± 1.2%; one-to-one setting; Scenario 1).

Original: In pre-training, we use Adam optimizer with learning rate of 0.0003 and 2-norm penalty coefficient of 0.0005.
Revised: In pre-training, we use Adam optimizer with learning rate of 0.0003 and 2-norm penalty coefficient of 0.0005.

Original: We use batch size of 64 and training epoch of 40.
Revised: We use batch size of 64 and training epoch of 40.

Original: We record the best performance in terms of F1 score and save the parameters in the corresponding epoch as Θ .
Revised: We record the best performance in terms of F1 score and save the parameters in the corresponding epoch as Θ .

Original: In fine-tuning, we initialize the model parameters with Θ and optimize the model on fine-tuning set.
Revised: In fine-tuning, we initialize the model parameters with Θ and optimize the model on fine-tuning set.

Original: The hyper-parameters are the same as in pre-training.
Revised: The hyper-parameters are the same as in pre-training.

Original: For the classification task in fine-tuning stage, we adopt a 2-layer fully-connected layer as a classifier.
Revised: For the classification task in fine-tuning stage, we adopt a 2-layer fully-connected layer as a classifier.

Original: The hidden dimensions are 64 and the number classes in target dataset, respectively.
Revised: The hidden dimensions are 64 and the number classes in target dataset, respectively.

Original: We measure the classification loss via the cross entropy function and the loss is jointly optimized with the fine-tuning loss ( i.e. , L TF-C ).
Revised: We measure the classification loss via the cross entropy function and the loss is jointly optimized with the fine-tuning loss ( i.e. , L TF-C ).

Original: TS-SD [12] uses a modified attention mechanism to encode latent features and use denoising or DTWsimilarity prediction as pretext-tasks.
Revised: TS-SD [12] uses a modified attention mechanism to encode latent features and use denoising or DTWsimilarity prediction as pretext-tasks.

Original: The encoder network is based on self-attention mechanism [68] but instead of the linear layer that produce the K, Q, V matrices from the input sequence, TS-SD uses a single convolutional layer to replace the linear layer.
Revised: The encoder network is based on self-attention mechanism [68] but instead of the linear layer that produce the K, Q, V matrices from the input sequence, TS-SD uses a single convolutional layer to replace the linear layer.

Original: With multihead attention, TS-SD sets a different kernel size for each head to attempt capture short- and long-range temporal patterns.
Revised: With multihead attention, TS-SD sets a different kernel size for each head to attempt capture short- and long-range temporal patterns.

Original: Although we found that 12 heads was relatively redundant and may be replaced by fewer heads, we still use 12 heads to keep the same with the original work in [12].
Revised: Although we found that 12 heads was relatively redundant and may be replaced by fewer heads, we still use 12 heads to keep the same with the original work in [12].

Original: For self-supervised pre-training, we adopt the denoising pretext task that attempts to minimize the mean square distance between the original input time series and the output of the encoder, for which the input is an augmented time series sample with noise added to a subseries.
Revised: For self-supervised pre-training, we adopt the denoising pretext task that attempts to minimize the mean square distance between the original input time series and the output of the encoder, for which the input is an augmented time series sample with noise added to a subseries.

Original: To cope with small numerical values, we use a learning rate of 3e-7 in pre-training and 3e-4 during fine-tuning.
Revised: To cope with small numerical values, we use a learning rate of 3e-7 in pre-training and 3e-4 during fine-tuning.

Original: While training on pre-training dataset for the four scenarios, the batch sizes varied from 4 to 128 and were manually chosen to achieve reasonable performance for each scenario.
Revised: While training on pre-training dataset for the four scenarios, the batch sizes varied from 4 to 128 and were manually chosen to achieve reasonable performance for each scenario.

Original: During fine-tuning, we use # epochs ranging from 20 to 80 and a batch size of 16.
Revised: During fine-tuning, we use # epochs ranging from 20 to 80 and a batch size of 16.

Original: Although the source code of TS-SD is not released, we implement the TS-SD model to our best understanding and tune the model to achieve the best performance as a strong baseline.
Revised: Although the source code of TS-SD is not released, we implement the TS-SD model to our best understanding and tune the model to achieve the best performance as a strong baseline.

Original: We public the implemented TS-SD in our online repository.
Revised: We public the implemented TS-SD in our online repository.

Original: TS2vec [47] introduces the notion of contextual consistency and uses a hierarchical loss function to capture long-range structure in time series.
Revised: TS2vec [47] introduces the notion of contextual consistency and uses a hierarchical loss function to capture long-range structure in time series.

Original: TS2vec is a powerful representative learning method and have a specially-designed architecture.
Revised: TS2vec is a powerful representative learning method and have a specially-designed architecture.

Original: The encoder network consists of three components.
Revised: The encoder network consists of three components.

Original: First, the input time series is augmented by selecting overlapping subseries.
Revised: First, the input time series is augmented by selecting overlapping subseries.

Original: They are projected into a higher dimensional latent space.
Revised: They are projected into a higher dimensional latent space.

Original: Then, latent vectors for input time series are masked at randomly chosen positions.
Revised: Then, latent vectors for input time series are masked at randomly chosen positions.

Original: Finally, a dilated CNN with residual blocks produce the contextual representations.
Revised: Finally, a dilated CNN with residual blocks produce the contextual representations.

Original: To compute the loss, the representations are gradually pooled along the time dimension and at each step a loss function based on contextual consistency is applied.
Revised: To compute the loss, the representations are gradually pooled along the time dimension and at each step a loss function based on contextual consistency is applied.

Original: For the baseline experiment, we found that the original 10 layers of ResNet blocks is redundant, and we reduce residual blocks in the encoder from 10-layer to 2-layer without compromising model performances.
Revised: For the baseline experiment, we found that the original 10 layers of ResNet blocks is redundant, and we reduce residual blocks in the encoder from 10-layer to 2-layer without compromising model performances.

Original: To make the model comparable with other baselines, we also restrict the hidden dimension to 1 and use a batch size of 64, except in the second scenario we reduce it to 16 given the length of samples in the F D - A / F D - B datasets.
Revised: To make the model comparable with other baselines, we also restrict the hidden dimension to 1 and use a batch size of 64, except in the second scenario we reduce it to 16 given the length of samples in the F D - A / F D - B datasets.

Original: CLOCS [41] assumes that samples from different subjects are negative pairs in contrastive learning and applies it to classify ECG signals.
Revised: CLOCS [41] assumes that samples from different subjects are negative pairs in contrastive learning and applies it to classify ECG signals.

Original: CLOCS makes the basic assumption that time series samples from different patients (the application scenario is in learning cardiac signals) should have dissimilar representations.
Revised: CLOCS makes the basic assumption that time series samples from different patients (the application scenario is in learning cardiac signals) should have dissimilar representations.

Original: Based on that, they devised three slightly different architectures - CMSC, CMLC, and CMSMLC, where ECG recordings from non-overlapping temporal segments, different ECG leads, and both of these are treated as positive pairs in case if they come from the same patients.
Revised: Based on that, they devised three slightly different architectures - CMSC, CMLC, and CMSMLC, where ECG recordings from non-overlapping temporal segments, different ECG leads, and both of these are treated as positive pairs in case if they come from the same patients.

Original: For this model, the encoder consists of alternating convolutional and pooling layers.
Revised: For this model, the encoder consists of alternating convolutional and pooling layers.

Original: So given the different input lengths, the hyper-parameters for these layers have to be modified accordingly.
Revised: So given the different input lengths, the hyper-parameters for these layers have to be modified accordingly.

Original: For example, we use kernel sizes of 7 , 7 , 7 , stride of 3 , 3 , 3 , pooling layer size of 2 for the last scenario of E CG to E MG but increased stride to 4 and increased representation size for a time series sample from 64 to 128 in case of F D - A to F D - B due to much longer time series samples.
Revised: For example, we use kernel sizes of 7 , 7 , 7 , stride of 3 , 3 , 3 , pooling layer size of 2 for the last scenario of E CG to E MG but increased stride to 4 and increased representation size for a time series sample from 64 to 128 in case of F D - A to F D - B due to much longer time series samples.

Original: Mixing-up [18] proposes new mixing-up augmentation and pretext tasks that aim to correctly predict the mixing proportion of two time series samples.
Revised: Mixing-up [18] proposes new mixing-up augmentation and pretext tasks that aim to correctly predict the mixing proportion of two time series samples.

Original: In Mixing-up, the augmentation is chosen as the convex combination of two randomly drawn time series from the dataset, where the mixing parameter is random drawn from a beta distribution.
Revised: In Mixing-up, the augmentation is chosen as the convex combination of two randomly drawn time series from the dataset, where the mixing parameter is random drawn from a beta distribution.

Original: The contrastive loss is then computed between the two inputs and the augmented time series.
Revised: The contrastive loss is then computed between the two inputs and the augmented time series.

Original: The loss is a minor modification of NT-Xent loss and is designed to encourage the correct prediction of the amount of mixing.
Revised: The loss is a minor modification of NT-Xent loss and is designed to encourage the correct prediction of the amount of mixing.

Original: We use the same beta distribution as reported in the original Mixing-up model.
Revised: We use the same beta distribution as reported in the original Mixing-up model.

Original: TS-TCC [48] leverages contextual information with a transformer-based autoregressive model and ensures transferability by using both strong and weak augmentations.
Revised: TS-TCC [48] leverages contextual information with a transformer-based autoregressive model and ensures transferability by using both strong and weak augmentations.

Original: TS-TCC proposed a challenging pretext task.
Revised: TS-TCC proposed a challenging pretext task.

Original: An input time series sample is first augmented by adding noise, scaling, and permuting time series.
Revised: An input time series sample is first augmented by adding noise, scaling, and permuting time series.

Original: The views are then passed through an encoder consisting of three convolutional layers before processed by the temporal contrasting module.
Revised: The views are then passed through an encoder consisting of three convolutional layers before processed by the temporal contrasting module.

Original: During temporal contrasting, for each view, a transformer architecture is used to learn a contextual representation.
Revised: During temporal contrasting, for each view, a transformer architecture is used to learn a contextual representation.

Original: The learned representation is then used to predict latent observation of the other augmented view at a future time.
Revised: The learned representation is then used to predict latent observation of the other augmented view at a future time.

Original: The contextual representations are then projected and maximized similarity using NT-Xent loss.
Revised: The contextual representations are then projected and maximized similarity using NT-Xent loss.

Original: For this baseline, we mostly adopted the hyper-parameters presented in the original paper.
Revised: For this baseline, we mostly adopted the hyper-parameters presented in the original paper.

Original: We use a learning rate of 3e-4 throughout, and a batch size of 128 during pre-training and 16 during fine-tuning.
Revised: We use a learning rate of 3e-4 throughout, and a batch size of 128 during pre-training and 16 during fine-tuning.

Original: SimCLR [40] is a state-of-the-art model in self-supervised representation learning of images.
Revised: SimCLR [40] is a state-of-the-art model in self-supervised representation learning of images.

Original: It utilizes deep learning architectures to generate augmentation-based embeddings and optimize the model parameters by minimizing NT-Xent loss in the embedding space.
Revised: It utilizes deep learning architectures to generate augmentation-based embeddings and optimize the model parameters by minimizing NT-Xent loss in the embedding space.

Original: [40] applies the original
Revised: [40] applies the original

Original: SimCLR model for time series data.
Revised: SimCLR model for time series data.

Original: In this work, we compare with the modified SimCLR as in [40].
Revised: In this work, we compare with the modified SimCLR as in [40].

Original: The SimCLR contrastive learning framework consists of four major components.
Revised: The SimCLR contrastive learning framework consists of four major components.

Original: Although initially proposed for image data, it is readily adapted to time series, as shown in [40].
Revised: Although initially proposed for image data, it is readily adapted to time series, as shown in [40].

Original: An input time series sample is first stochastically augmented into two related views.
Revised: An input time series sample is first stochastically augmented into two related views.

Original: Then a base encoder extracts representation vectors.
Revised: Then a base encoder extracts representation vectors.

Original: ResNet is used as encoder backbone for simplicity.
Revised: ResNet is used as encoder backbone for simplicity.

Original: Then the projection head transforms representations into a latent space where the NT-Xent loss is applied.
Revised: Then the projection head transforms representations into a latent space where the NT-Xent loss is applied.

Original: For time series,
Revised: For time series,

Original: SimCLR investigated different augmentations, including adding noise, scaling, rotation, negation, flipping in time, permutation of subseries, time warping, and channel shuffling.
Revised: SimCLR investigated different augmentations, including adding noise, scaling, rotation, negation, flipping in time, permutation of subseries, time warping, and channel shuffling.

Original: All the unmentioned hyper-parameters are kept the same with the original model.
Revised: All the unmentioned hyper-parameters are kept the same with the original model.

Original: TNC [46] focuses on learning representations that encode the underlying state of a non-stationary time series.
Revised: TNC [46] focuses on learning representations that encode the underlying state of a non-stationary time series.

Original: This is achieved by ensuring that the distribution of observations in the latent space is different from the distribution of temporally separated observations.
Revised: This is achieved by ensuring that the distribution of observations in the latent space is different from the distribution of temporally separated observations.

Original: For each time point, the
Revised: For each time point, the

Original: TNC model calculates a neighborhood size using the Augmented Dickey–Fuller test.
Revised: TNC model calculates a neighborhood size using the Augmented Dickey–Fuller test.

Original: The encoded representations of two different windows will be passed to a discriminator to predict the probability that they belong to the same temporal neighborhood.
Revised: The encoded representations of two different windows will be passed to a discriminator to predict the probability that they belong to the same temporal neighborhood.

Original: We implement the TFC model following the public GitHub repository provided by the authors.
Revised: We implement the TFC model following the public GitHub repository provided by the authors.

Original: We observed that the ADF tests are extremely slow, so TNC is only used as a baseline in scenario 1 in one-to-one setting (classification task).
Revised: We observed that the ADF tests are extremely slow, so TNC is only used as a baseline in scenario 1 in one-to-one setting (classification task).

Original: CPC [30] aims to learn the representations that encode the global information shared across different segments of a time series signal by the strategy of predictive coding.
Revised: CPC [30] aims to learn the representations that encode the global information shared across different segments of a time series signal by the strategy of predictive coding.

Original: Instead of modelling the conditional distribution at future time points for prediction, the authors choose to encode the context and observation into a vector representation in a way that maximally preserves their mutual information.
Revised: Instead of modelling the conditional distribution at future time points for prediction, the authors choose to encode the context and observation into a vector representation in a way that maximally preserves their mutual information.

Original: For the model architecture, a nonlinear encoder first maps the sequence of observations to latent representations.
Revised: For the model architecture, a nonlinear encoder first maps the sequence of observations to latent representations.

Original: Then an autoregressive model summarizes past latent representations to produce a context representation at every time point.
Revised: Then an autoregressive model summarizes past latent representations to produce a context representation at every time point.

Original: The two components are trained to jointly optimize a loss function which is inspired by noise contrastive estimation.
Revised: The two components are trained to jointly optimize a loss function which is inspired by noise contrastive estimation.

Original: We used the implementation provided by the authors of the TNC paper [46] and kept their default choices of parameters whenever reasonable.
Revised: We used the implementation provided by the authors of the TNC paper [46] and kept their default choices of parameters whenever reasonable.

Original: The TS-SD is the only method, to our knowledge, that explicitly aims at time series pre-training but the underlying assumption that can support knowledge transfer across different time series datasets is not clearly provided.
Revised: The TS-SD is the only method, to our knowledge, that explicitly aims at time series pre-training but the underlying assumption that can support knowledge transfer across different time series datasets is not clearly provided.

Original: The TS2vec, CLOCS, Mixing-up, TS-TCC, TNC, and CPC are designed for representation learning (instead of pre-training) but involved transfer learning setting in their experiments.
Revised: The TS2vec, CLOCS, Mixing-up, TS-TCC, TNC, and CPC are designed for representation learning (instead of pre-training) but involved transfer learning setting in their experiments.

Original: We modify them to perform pre-training task and make them comparable to our model.
Revised: We modify them to perform pre-training task and make them comparable to our model.

Original: The SimCLR is a widely use baseline in computer vision and simply adopted to deal with time series by [40].
Revised: The SimCLR is a widely use baseline in computer vision and simply adopted to deal with time series by [40].

Original: We compare our model with SimCLR to further show our superiority.
Revised: We compare our model with SimCLR to further show our superiority.

Original: Appendix F Additional results on one-to-one evaluation
Revised: Appendix F Additional results on one-to-one evaluation

Original: In one-to-one pre-training evaluation, we consider four different scenarios with paired pre-training and fine-tuning datasets from different fields (Sec. 5.1).
Revised: In one-to-one pre-training evaluation, we consider four different scenarios with paired pre-training and fine-tuning datasets from different fields (Sec. 5.1).

Original: In each scenario, the pre-training and fine-tuning datasets share same semantic meanings ( e.g. , both EEG signals; Scenario 1) or similar semantic meanings ( e.g. , ECG and EMG are both physiological signals; Scenario 4).
Revised: In each scenario, the pre-training and fine-tuning datasets share same semantic meanings ( e.g. , both EEG signals; Scenario 1) or similar semantic meanings ( e.g. , ECG and EMG are both physiological signals; Scenario 4).

Original: For example, in the first scenario, S LEEP E EG and E PILEPSY both involve EEG measurements but the recordings are taken under different physical conditions.
Revised: For example, in the first scenario, S LEEP E EG and E PILEPSY both involve EEG measurements but the recordings are taken under different physical conditions.

Original: For each scenario, we first train each baseline model using the pre-training dataset and record the converged parameters.
Revised: For each scenario, we first train each baseline model using the pre-training dataset and record the converged parameters.

Original: The parameters are then taken to initialize the model on the fine-tuning phase which uses the target dataset.
Revised: The parameters are then taken to initialize the model on the fine-tuning phase which uses the target dataset.

Original: In fine-tuning, we allowed all parameters in the model to be optimized (no freezing of earlier layers; full fine-tuning).
Revised: In fine-tuning, we allowed all parameters in the model to be optimized (no freezing of earlier layers; full fine-tuning).

Original: Our preliminary results show that the performance of full fine-tuning is slightly better than partial fine-tuning by 5.4% in F1 score (one-to-one setting;
Revised: Our preliminary results show that the performance of full fine-tuning is slightly better than partial fine-tuning by 5.4% in F1 score (one-to-one setting;

Original: Scenario 1).
Revised: Scenario 1).

Original: After fine-tuning, we evaluate the final model on the test set and report metrics including accuracy, precision (macro-averaged), recall, F1-score, AUROC (one-versus-rest), and AUPRC.
Revised: After fine-tuning, we evaluate the final model on the test set and report metrics including accuracy, precision (macro-averaged), recall, F1-score, AUROC (one-versus-rest), and AUPRC.

Original: The evaluation results of four scenarios are shown in Table 1 (main paper) and Tables 4-6.
Revised: The evaluation results of four scenarios are shown in Table 1 (main paper) and Tables 4-6.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Appendix G Results on downstream clustering and anomaly detection tasks
Revised: Appendix G Results on downstream clustering and anomaly detection tasks

Original: Beyond the common downstream classification tasks, we conducted extensive experiments to examine the proposed TF-C in diverse downstream tasks.
Revised: Beyond the common downstream classification tasks, we conducted extensive experiments to examine the proposed TF-C in diverse downstream tasks.

Original: The comparison results with state-of-the-art baselines are shown in Table 7 for clustering and in Table 8 for anomaly detection.
Revised: The comparison results with state-of-the-art baselines are shown in Table 7 for clustering and in Table 8 for anomaly detection.

Original: The experimental setups and results analysis are shown in Section 5.
Revised: The experimental setups and results analysis are shown in Section 5.

Original: Appendix H Further results on ablation study
Revised: Appendix H Further results on ablation study

Original: We conduct ablation studies (Sec. 5) to evaluate the importance of every component in the developed
Revised: We conduct ablation studies (Sec. 5) to evaluate the importance of every component in the developed

Original: TF-C model.
Revised: TF-C model.

Original: For the example of one-to-one setting ( S LEEP E EG → G ESTURE ) when pre-training the model using S LEEP E EG dataset and fine-tuning on G ESTURE dataset, ablation study results are shown in Table 9.
Revised: For the example of one-to-one setting ( S LEEP E EG → G ESTURE ) when pre-training the model using S LEEP E EG dataset and fine-tuning on G ESTURE dataset, ablation study results are shown in Table 9.

Original: Through comparison, we observe that the full TF-C model achieves the highest performance in every evaluation metrics, indicating every component, especially the novel consistency loss L C ), contributes to the model’s performance.
Revised: Through comparison, we observe that the full TF-C model achieves the highest performance in every evaluation metrics, indicating every component, especially the novel consistency loss L C ), contributes to the model’s performance.

Original: Appendix I Visualization of embeddings in time-frequency space
Revised: Appendix I Visualization of embeddings in time-frequency space

Original: To better demonstrate the effectiveness of the developed consistency loss L C , we visualize the learned embeddings in time-frequency space.
Revised: To better demonstrate the effectiveness of the developed consistency loss L C , we visualize the learned embeddings in time-frequency space.

Original: In one-to-one setting, we pre-train the TF-C model on
Revised: In one-to-one setting, we pre-train the TF-C model on

Original: S LEEP E EG and fine-tune on E PILEPSY (Sec. 5.1).
Revised: S LEEP E EG and fine-tune on E PILEPSY (Sec. 5.1).

Original: We visualize the testing samples of E PILEPSY in fine-tuning stage.
Revised: We visualize the testing samples of E PILEPSY in fine-tuning stage.

Original: The learned time-based embeddings z T i and frequency-based embedding z F i have 128 dimensions.
Revised: The learned time-based embeddings z T i and frequency-based embedding z F i have 128 dimensions.

Original: We map the embeddings to a 2-dimensional space through UMAP (20 neighbors; minimum distance as 0.2; cosine distance) for visualization.
Revised: We map the embeddings to a 2-dimensional space through UMAP (20 neighbors; minimum distance as 0.2; cosine distance) for visualization.

Original: In this work, we develop a novel consistency loss L C to induce the model learn closer time-based and frequency-based embeddings in a time-frequency space (Sec. 4.3).
Revised: In this work, we develop a novel consistency loss L C to induce the model learn closer time-based and frequency-based embeddings in a time-frequency space (Sec. 4.3).

Original: As our problem setting is that the fine-tuning dataset has small scale ( M ≪ N ; Sec. 3), we randomly select 100 samples from the testing set ( E PILEPSY ) for visualization.
Revised: As our problem setting is that the fine-tuning dataset has small scale ( M ≪ N ; Sec. 3), we randomly select 100 samples from the testing set ( E PILEPSY ) for visualization.

Original: The visualization of the learned embeddings in time-frequency space is shown in Figure 3 (without L C v.s. with L C ).
Revised: The visualization of the learned embeddings in time-frequency space is shown in Figure 3 (without L C v.s. with L C ).

Original: We can observe that: (1) when without L C (panel a), the time-based and frequency-based embeddings are clustered separately; the embeddings are largely clustered together when having L C (panel b).
Revised: We can observe that: (1) when without L C (panel a), the time-based and frequency-based embeddings are clustered separately; the embeddings are largely clustered together when having L C (panel b).

Original: The observation demonstrates that the proposed L C has the ability to push time-based and frequency-based embeddings closer to each other.
Revised: The observation demonstrates that the proposed L C has the ability to push time-based and frequency-based embeddings closer to each other.

Original: (2) for a specific sample x i (marked as dashed line), we can find that its time-based
Revised: (2) for a specific sample x i (marked as dashed line), we can find that its time-based

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: embedding (colored as light blue) and frequency-based (colored as dark blue) embedding are closer to each other in panel (b) than in panel (a).
Revised: embedding (colored as light blue) and frequency-based (colored as dark blue) embedding are closer to each other in panel (b) than in panel (a).

Original: In particular, the cosine distance, between z T i and z F i of this sample, has reduced from 0.89 to 0.71 when using L C in our model.
Revised: In particular, the cosine distance, between z T i and z F i of this sample, has reduced from 0.89 to 0.71 when using L C in our model.

Original: (3) the frequency-based embeddings of ‘epilepsy’ and ‘health’ samples are largely overlapped when without L C but clustered well when considering L C , indicating the learned embeddings (with L C ) can potentially enhance downstream classification performance.
Revised: (3) the frequency-based embeddings of ‘epilepsy’ and ‘health’ samples are largely overlapped when without L C but clustered well when considering L C , indicating the learned embeddings (with L C ) can potentially enhance downstream classification performance.

Original: Furthermore, we attempt to quantify the impact of our consistency loss item on the quality of learned embeddings, using the one-to-many setting as shown in Figure 4.
Revised: Furthermore, we attempt to quantify the impact of our consistency loss item on the quality of learned embeddings, using the one-to-many setting as shown in Figure 4.

Original: We pretrain the TF-C model on S LEEP E EG dataset and fine-tuning on E PILEPSY , F D -B , G ESTURE , and E MG , respectively.
Revised: We pretrain the TF-C model on S LEEP E EG dataset and fine-tuning on E PILEPSY , F D -B , G ESTURE , and E MG , respectively.

Original: Next, we explain how the cosine distance is calculated taking E MG dataset as an example.
Revised: Next, we explain how the cosine distance is calculated taking E MG dataset as an example.

Original: For
Revised: For

Original: [Figure]
Revised: [Figure]

Original: D i m e n s i o n
Revised: D i m e n s i o n

Original: Epilepsy
Revised: Epilepsy

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: D i m e n s i o n
Revised: D i m e n s i o n

Original: [Figure]
Revised: [Figure]

Original: a b
Revised: a b

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: distance averaged across all testing samples
Revised: distance averaged across all testing samples

Original: We observe that the cosine distance is larger when removing our proposed consistency loss item
Revised: We observe that the cosine distance is larger when removing our proposed consistency loss item

Original: L
Revised: L

Original: C indicating our model indeed pushes the time-based and frequency-based embeddings closer to each other.
Revised: C indicating our model indeed pushes the time-based and frequency-based embeddings closer to each other.

Original: Epilepsy
Revised: Epilepsy

Original: FD-B
Revised: FD-B

Original: Gesture
Revised: Gesture

Original: EMG
Revised: EMG

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: I n t e r - c l a ss C o s i n e d i s t a n c e every sample x i in E MG , we measure the cosine distance between its time-based embedding z T i and frequency-based embedding z F i in the 128-dimension time-frequency space.
Revised: I n t e r - c l a ss C o s i n e d i s t a n c e every sample x i in E MG , we measure the cosine distance between its time-based embedding z T i and frequency-based embedding z F i in the 128-dimension time-frequency space.

Original: Then, we take the average of the distance across all testing samples and report in Figure 4.
Revised: Then, we take the average of the distance across all testing samples and report in Figure 4.

Original: The number of testing samples of each dataset can be found in Table 3.
Revised: The number of testing samples of each dataset can be found in Table 3.

Original: We find that the cosine distance is larger when removing the proposed consistency loss item L C , which means our model, as designed and expected, indeed pushes the time-based and frequency-based embeddings closer to each other.
Revised: We find that the cosine distance is larger when removing the proposed consistency loss item L C , which means our model, as designed and expected, indeed pushes the time-based and frequency-based embeddings closer to each other.

Original: For example, in
Revised: For example, in

Original: E PILEPSY , the averaged cosine distance dropped 22.2% (from 1.32 to 1.08) when using consistency loss (Sec. 4.3).
Revised: E PILEPSY , the averaged cosine distance dropped 22.2% (from 1.32 to 1.08) when using consistency loss (Sec. 4.3).

Original: At the same time, the embeddings of samples drawn from different classes are more easily distinguished from each other, as shown in Figure 5: we observe that our TF-C model (with consistency loss L T increase the inter-class cosine distance from 0.88 to 1.42 with a remarkable margin of 61.4% (one-to-many setting; testing set of E PILEPSY ), indicating our model indeed has the ability to increase the distinctiveness of learned embeddings and bring forward better classification performance on the fine-tuning dataset of interest.
Revised: At the same time, the embeddings of samples drawn from different classes are more easily distinguished from each other, as shown in Figure 5: we observe that our TF-C model (with consistency loss L T increase the inter-class cosine distance from 0.88 to 1.42 with a remarkable margin of 61.4% (one-to-many setting; testing set of E PILEPSY ), indicating our model indeed has the ability to increase the distinctiveness of learned embeddings and bring forward better classification performance on the fine-tuning dataset of interest.

Original: The results of visualization increase our confidence that the developed TF-C pre-training model can capture generalizable properties and boost knowledge transfer across different time series datasets.
Revised: The results of visualization increase our confidence that the developed TF-C pre-training model can capture generalizable properties and boost knowledge transfer across different time series datasets.

Original: Appendix J Additional information on domain augmentations
Revised: Appendix J Additional information on domain augmentations

Original: To the best of our knowledge, this is the first work that direct perturbs the frequency domain of time series in contrastive learning.
Revised: To the best of our knowledge, this is the first work that direct perturbs the frequency domain of time series in contrastive learning.

Original: Here, we thoroughly explore various frequency augmentations and discuss the design of manipulation strategy in frequency domain.
Revised: Here, we thoroughly explore various frequency augmentations and discuss the design of manipulation strategy in frequency domain.

Original: In particular, we analyze eight frequency augmentation policies and discuss their performance.
Revised: In particular, we analyze eight frequency augmentation policies and discuss their performance.

Original: We consider three types of frequency augmentations:
Revised: We consider three types of frequency augmentations:

Original: • Low- vs. High-band perturbations : High-frequency components contribute to the fast varying parts of the time series (e
Revised: • Low- vs. High-band perturbations : High-frequency components contribute to the fast varying parts of the time series (e

Original: g., sharp transitions).
Revised: g., sharp transitions).

Original: In contrast, low-frequency components contribute to slow signal variations in the time domain.
Revised: In contrast, low-frequency components contribute to slow signal variations in the time domain.

Original: Low band perturbations correspond to the first half of the frequency spectrum.
Revised: Low band perturbations correspond to the first half of the frequency spectrum.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: • Single- vs. Multi-component perturbations (aka varying budget size): Perturbing several frequency components (i.e., larger budget) leads to more substantial changes in the time domain.
Revised: • Single- vs. Multi-component perturbations (aka varying budget size): Perturbing several frequency components (i.e., larger budget) leads to more substantial changes in the time domain.

Original: In this analysis, we set the budget to 5 for multi-component perturbations and 1 for single-component perturbations.
Revised: In this analysis, we set the budget to 5 for multi-component perturbations and 1 for single-component perturbations.

Original: • Random vs. Distributional perturbations : The selection of components in the spectrum that get perturbed is important because it influences the resulting time-based representation of the augmented sample.
Revised: • Random vs. Distributional perturbations : The selection of components in the spectrum that get perturbed is important because it influences the resulting time-based representation of the augmented sample.

Original: Frequency components can be sampled uniformly at random or according to a user-specified distribution (i
Revised: Frequency components can be sampled uniformly at random or according to a user-specified distribution (i

Original: [Table]
Revised: [Table]

Original: e., Gaussian in the following analysis).
Revised: e., Gaussian in the following analysis).

Original: Based on the above considerations we developed eight frequency augmentations.
Revised: Based on the above considerations we developed eight frequency augmentations.

Original: In Fig.
Revised: In Fig.

Original: 6 (E PILEPSY dataset))
Revised: 6 (E PILEPSY dataset))

Original: we visualize a series of perturbations in the frequency domain and their effects in the time domain (through the inverse FFT).
Revised: we visualize a series of perturbations in the frequency domain and their effects in the time domain (through the inverse FFT).

Original: In addition
Revised: In addition

Original: [Table]
Revised: [Table]

Original: we explore how different spectral perturbations affect the performance of TF-C. Tableshows results for the one-to-one setup (S LEEP E EG → E PILEPSY ).
Revised: we explore how different spectral perturbations affect the performance of TF-C. Tableshows results for the one-to-one setup (S LEEP E EG → E PILEPSY ).

Original: We find the following:
Revised: We find the following:

Original: • Perturbing low-band components performs better than augmenting high-band elements of the spectrum (0.8% change in F1 performance on average across the four setups).
Revised: • Perturbing low-band components performs better than augmenting high-band elements of the spectrum (0.8% change in F1 performance on average across the four setups).

Original: This can be explained by the nature of the underlying time series: in physiological signals, most information is carried in a low band (e.g., EEG signals are most informative in 0.5Hz while > 70Hz is considered noise).
Revised: This can be explained by the nature of the underlying time series: in physiological signals, most information is carried in a low band (e.g., EEG signals are most informative in 0.5Hz while > 70Hz is considered noise).

Original: This means that the choice of low- vs. highband components depends on the signal context.
Revised: This means that the choice of low- vs. highband components depends on the signal context.

Original: For example, high-band components are informative for mechanical signals.
Revised: For example, high-band components are informative for mechanical signals.

Original: We further verified this by external experiments in the F D - A → F D - B setting, where perturbing high-band components outperform low-band augmentations by 0.8% in F1.
Revised: We further verified this by external experiments in the F D - A → F D - B setting, where perturbing high-band components outperform low-band augmentations by 0.8% in F1.

Original: Thus, we consider perturbations across the entire spectral band in the paper to achieve broad generalization.
Revised: Thus, we consider perturbations across the entire spectral band in the paper to achieve broad generalization.

Original: • In general, perturbing many frequency components degrades model performance.
Revised: • In general, perturbing many frequency components degrades model performance.

Original: One reason for worse performance is that too big a budget incurs too significant changes in the time-based representation of augmented samples.
Revised: One reason for worse performance is that too big a budget incurs too significant changes in the time-based representation of augmented samples.

Original: In that case, the augmented samples would be easily distinguished by a contrastive model, leading to poor contrastive encoders.
Revised: In that case, the augmented samples would be easily distinguished by a contrastive model, leading to poor contrastive encoders.

Original: For this reason, we use single-component perturbations in the paper.
Revised: For this reason, we use single-component perturbations in the paper.

Original: • We cannot draw a clear conclusion on whether the random selection of components is better than Gaussian distribution-based as both F1 scores are very similar.
Revised: • We cannot draw a clear conclusion on whether the random selection of components is better than Gaussian distribution-based as both F1 scores are very similar.

Original: The optimal choice of sampling strategy likely depends on both the time-series signal and prediction task.
Revised: The optimal choice of sampling strategy likely depends on both the time-series signal and prediction task.

Original: We use random sampling in the paper.
Revised: We use random sampling in the paper.

Original: Appendix K Discussion on many-to-one setting
Revised: Appendix K Discussion on many-to-one setting

Original: We probed the one-to-one setting and one-to-many setting in knowledge transfer of time series (Sec. 5).
Revised: We probed the one-to-one setting and one-to-many setting in knowledge transfer of time series (Sec. 5).

Original: Here, we discuss the many-to-one set up: pre-train on a mixture of multiple datasets and fine-tune on a single pure dataset.
Revised: Here, we discuss the many-to-one set up: pre-train on a mixture of multiple datasets and fine-tune on a single pure dataset.

Original: We notice that the many-to-one setup fundamentally differs from the one-to-one and also from oneto-many configurations.
Revised: We notice that the many-to-one setup fundamentally differs from the one-to-one and also from oneto-many configurations.

Original: Notably, in the one-to-one and one-to-many setups, the model is pre-trained on a single pre-training dataset presumed to originate from a data source exhibiting homogeneity.
Revised: Notably, in the one-to-one and one-to-many setups, the model is pre-trained on a single pre-training dataset presumed to originate from a data source exhibiting homogeneity.

Original: For example, samples in the pre-training dataset could describe similar devices (e.g., different kinds of boring machines) with similar sampling rates, lengths, semantic meanings, and/or relevant temporal patterns that occur on comparable time scales, etc.
Revised: For example, samples in the pre-training dataset could describe similar devices (e.g., different kinds of boring machines) with similar sampling rates, lengths, semantic meanings, and/or relevant temporal patterns that occur on comparable time scales, etc.

Original: When some level of homogeneity exists in a pre-training dataset, the method can learn patterns and apply them to different fine-tuning datasets.
Revised: When some level of homogeneity exists in a pre-training dataset, the method can learn patterns and apply them to different fine-tuning datasets.

Original: In contrast, in many-to-one setups, pre-training datasets can be incredibly heterogeneous.
Revised: In contrast, in many-to-one setups, pre-training datasets can be incredibly heterogeneous.

Original: For example, a pre-training dataset might contain samples describing boring machines’ behavior and recordings of EEG medical devices.
Revised: For example, a pre-training dataset might contain samples describing boring machines’ behavior and recordings of EEG medical devices.

Original: For this reason, pre-training can be severely hampered by diversity and misalignments across samples in the pre-training dataset.
Revised: For this reason, pre-training can be severely hampered by diversity and misalignments across samples in the pre-training dataset.

Original: When we merge many heterogeneous samples into a single pre-training dataset, the model might struggle to find patterns underlying the combined dataset.
Revised: When we merge many heterogeneous samples into a single pre-training dataset, the model might struggle to find patterns underlying the combined dataset.

Original: While models in computer vision have been successfully pre-trained on large, diverse datasets, such as ImageNet, the literature on pre-training for time series is much sparser.
Revised: While models in computer vision have been successfully pre-trained on large, diverse datasets, such as ImageNet, the literature on pre-training for time series is much sparser.

Original: Further, heterogeneous pre-training datasets pose additional challenges due to the unique properties of time series.
Revised: Further, heterogeneous pre-training datasets pose additional challenges due to the unique properties of time series.

Original: Learning universal representations from multi-sourced time series datasets (i.e., many-to-one setups) remains an open challenge.
Revised: Learning universal representations from multi-sourced time series datasets (i.e., many-to-one setups) remains an open challenge.

Original: It is an interesting future direction, and we hope this study can facilitate future research.
Revised: It is an interesting future direction, and we hope this study can facilitate future research.

Original: We have explored this and provide preliminary experimental results in four many-to-one setups (named N-to-one for N = 1,2, 3, and 4).
Revised: We have explored this and provide preliminary experimental results in four many-to-one setups (named N-to-one for N = 1,2, 3, and 4).

Original: Take four-to-one as an example.
Revised: Take four-to-one as an example.

Original: We merge four pre-training datasets (S LEEP E EG ,F D - A , H AR , and E CG ) into one pre-training dataset.
Revised: We merge four pre-training datasets (S LEEP E EG ,F D - A , H AR , and E CG ) into one pre-training dataset.

Original: We had to preprocess the datasets to make a many-to-one setup feasible.
Revised: We had to preprocess the datasets to make a many-to-one setup feasible.

Original: To address the mismatch between the number of channels, for the multivariate dataset (H AR ), we take a single channel (i.e., x-axis acceleration) to make sure all samples have the same number of channels.
Revised: To address the mismatch between the number of channels, for the multivariate dataset (H AR ), we take a single channel (i.e., x-axis acceleration) to make sure all samples have the same number of channels.

Original: To align the length across datasets, we make all samples have 1,500 observations by zero-padding (for the ones shorter than 1,500) and cutting-off (for the ones longer than 1,500).
Revised: To align the length across datasets, we make all samples have 1,500 observations by zero-padding (for the ones shorter than 1,500) and cutting-off (for the ones longer than 1,500).

Original: Further, we take 25% samples (instead of 100%; for computational reasons only) from each of the four datasets to create a pre-training dataset with 108,300 samples that belong to 18 classes in total.
Revised: Further, we take 25% samples (instead of 100%; for computational reasons only) from each of the four datasets to create a pre-training dataset with 108,300 samples that belong to 18 classes in total.

Original: Similarly, we built pre-training for three-to-one (merge S LEEP E EG ,F D - A , and H AR ) and two-to-one (merge S LEEP E EG and F D - A ).
Revised: Similarly, we built pre-training for three-to-one (merge S LEEP E EG ,F D - A , and H AR ) and two-to-one (merge S LEEP E EG and F D - A ).

Original: We take the above N-to-one pre-training datasets and repeat the following analysis.
Revised: We take the above N-to-one pre-training datasets and repeat the following analysis.

Original: First, we pre-train our TF-C and then fine-tune it on E PILEPSY for classification.
Revised: First, we pre-train our TF-C and then fine-tune it on E PILEPSY for classification.

Original: The AUROC for one-to-one, twoto-one, three-to-one, and four-to-one settings are 0.9819, 0.9525, 0.8607, and 0.7253, respectively.
Revised: The AUROC for one-to-one, twoto-one, three-to-one, and four-to-one settings are 0.9819, 0.9525, 0.8607, and 0.7253, respectively.

Original: We observe AUROC decreases with the increasing heterogeneity of pre-training datasets, which is consistent with our analysis in the first part of this answer.
Revised: We observe AUROC decreases with the increasing heterogeneity of pre-training datasets, which is consistent with our analysis in the first part of this answer.

Original: Nevertheless, whether there exists a framework that overcomes the problems and enables many-to-one pre-training for time series remains to be seen, and we believe it’s an exciting direction for future research.
Revised: Nevertheless, whether there exists a framework that overcomes the problems and enables many-to-one pre-training for time series remains to be seen, and we believe it’s an exciting direction for future research.