O UT - OF - DISTRIBUTION R EPRESENTATION L EARNING FOR T IME S ERIES C LASSIFICATION
A BSTRACT
Time series classiﬁcation is an important problem in the real world.
Due to its nonstationary property that the distribution changes over time, it remains challenging to build models for generalization to unseen distributions.
In this paper, we propose to view time series classiﬁcation from the distribution perspective.
We argue that the temporal complexity of a time series dataset could attribute to unknown latent distributions that need characterize.
To this end, we propose D IVERSIFY for outof-distribution (OOD) representation learning on dynamic distributions of times series.
D IVERSIFY takes an iterative process: it ﬁrst obtains the ‘worst-case’ latent distribution scenario via adversarial training, then reduces the gap between these latent distributions.
We then show that such an algorithm is theoretically supported.
Extensive experiments are conducted on seven datasets with different OOD settings across gesture recognition, speech commands recognition, wearable stress and affect detection, and sensor-based human activity recognition.
Qualitative and quantitative results demonstrate that D IVERSIFY signiﬁcantly outperforms other baselines and effectively characterizes the latent distributions.
Code is available at https://github.com/microsoft/robustlearn .
I NTRODUCTION
Time series classiﬁcation is one of the most challenging problems in the machine learning and statistics community (Fawaz et al., 2019; Du et al., 2021).
One important nature of time series is the non-stationary property, indicating that its statistical features are changing over time.
For years, there have been tremendous efforts for time series classiﬁcation, such as hidden Markov models (Fulcher & Jones, 2014), RNN-based methods (H ¨ usken & Stagge, 2003), and Transformer-based approaches (Li et al., 2019; Drouin et al., 2022).
We propose to model time series from the distribution perspective to handle its dynamically changing distributions; more precisely, to learn out-of-distribution (OOD) representations for time series that generalize to unseen distributions.
The general OOD/domain generalization problem has been extensively studied (Wang et al., 2022; Lu et al., 2022; Krueger et al., 2021; Rame et al., 2022), where the key is to bridge the gap between known and unknown distributions.
Despite existing efforts, OOD in time series remains less studied and more challenging.
Compared to image classiﬁcation, the dynamic distribution of time series data keeps changing over time, containing diverse distribution information that should be harnessed for better generalization.
Figure 1 shows an illustrative example.
OOD generalization in image classiﬁcation often involves several domains whose domain labels are static and known (subﬁgure (a)), which can be employed to build OOD models.
However, Figure 1 (b) shows that in EMG time series data (Lobov et al., 2018), the distribution is changing dynamically over time and its domain information is unavailable .
If no attention is paid to exploring its latent distributions (i.e., sub-domains), predictions may fail in face of diverse sub-domain distributions (subﬁgure (c)).
This will dramatically impede existing OOD algorithms due to their reliance on domain information.
In this work, we propose D IVERSIFY , an OOD representation learning algorithm for time series classiﬁcation by characterizing the latent distributions inside the data.
Concretely speaking, D I -
[Figure]
VERSIFY consists of a min-max adversarial game: on one hand, it learns to segment the time series data into several latent sub-domains by maximizing the segment-wise distribution gap to preserve diversities, i.e., the ‘worst-case’ distribution scenario; on the other hand, it learns domain-invariant representations by reducing the distribution divergence between the obtained latent domains.
Such latent distributions naturally exist in time series, e.g., the activity data from multiple people follow different distributions.
Additionally, our experiments show that even the data of one person still has such diversity: it can also be split into several latent distributions.
Figure 1 (d) shows that D IVERSIFY can effectively characterize the latent distributions (more results are in Sec. 3.5).
To summarize, our contributions are four-fold:
Novel perspective: We propose to view time series classiﬁcation from the distribution perspective to learn OOD representation, which is more challenging than the traditional image classiﬁcation due to the existence of unidentiﬁed latent distributions.
Novel methodology:
D IVERSIFY is a novel framework to identify the latent distributions and learn generalized representations.
Technically, we propose pseudo domain-class labels and adversarial self-supervised pseudo labeling to obtain the pseudo domain labels.
Theoretical insights: We provide the theoretical insights behind D IVERSIFY to analyze its design philosophy and conduct experiments to prove the insights.
Superior performance and insightful results: Qualitative and quantitative results using various backbones demonstrate the superiority of D IVERSIFY in several challenging scenarios: difﬁcult tasks, signiﬁcantly diverse datasets, and limited data.
More importantly, D IVERSIFY can successfully characterize the latent distributions within a time series dataset.
M ETHODOLOGY
A time-series training dataset D tr can be often pre-processed using sliding window 1 to N inputs:
D tr = { ( x i , y i ) } Ni =1 , where x i ∈ X ⊂ R p is the p -dimensional instance and
y i ∈ Y = { 1 , . . .
, C } is its label.
We use P tr ( x , y ) on X × Y to denote the joint distribution of the training dataset.
Our goal is to learn a generalized model from D tr to predict well on an unseen target dataset, D te , which is inaccessible in training.
In our problem, the training and test datasets have the same input and output spaces but different distributions, i.e., X tr = X te , Y tr = Y te , but P tr ( x , y ) (cid:54) =
P te ( x , y ) .
We aim to train a model h from D tr to achieve minimum error on D te .
2. M OTIVATION
What are domain and distribution shift in time series?
Time series may consist of several unknown latent distributions (domains), even if the dataset is fully labeled.
For instance, data collected by sensors of three persons may belong to two different distributions due to their dissimilarities.
This can be termed as spatial distribution shift.
Surprisingly, we even ﬁnd temporal distribution shifts in [Figure] experiments (Figure 6) that distributions of one person can also change at different time.
Those shifts widely exist in time series, as suggested by (Zhang et al., 2021;
Ragab et al., 2022).
OOD generalization requires latent domain characterization.
Due to the non-stationary property, naive approaches that treat time series as one distribution fail to capture domain-invariant (OOD) features since they ignore the diversities inside the dataset.
In Figure 1 (c), we assume the training domain contains two sub-domains (circle and plus points).
Directly treating it as one distribution via existing OOD approaches may generate the black margin.
Red star points are misclassiﬁed to the green class when predicting on the OOD domain (star points) with the learned model.
Thus, multiple diverse latent distributions in time series should be characterized to learn better OOD features.
A brief formulation of latent domain characterization.
Following above discussions, a time series may consist of K unknown latent domains 23 rather than a ﬁxed one, i.e., P tr ( x , y )
=
(cid:80)
Ki =1 π i P i ( x , y ) , where P i ( x , y ) is the distribution of the i -th latent one with weight
π i , (cid:80) Ki =
1 π i = 1 .
4 There could be inﬁnite ways to obtain P
i s and our goal is to learn the ‘worst-case’ distribution scenario where the distribution divergence between each P i and P j is maximized.
Why the ‘worstcase’ scenario?
It will maximally preserve the diverse information of each latent distribution, thus beneﬁting generalization.
For an illustration, these obtained latent distributions are shown in Sec. 3.5.
2. D IVERSIFY
In this paper, we propose D IVERSIFY to learn OOD representations for time series classiﬁcation.
The core of D IVERSIFY is to characterize the latent distributions and then minimize the distribution divergence between each two.
D IVERSIFY utilizes an iterative process: it ﬁrst obtains the ’worst-case’ distribution scenario from a given dataset, then bridges the distribution gaps between each pair of latent distributions.
Figure 2 describes its main procedures, where steps 2 ∼ 4 are iterative:
Pre-processing: this step adopts the sliding window to split the entire training dataset into ﬁxed-size windows.
We
argue that the data from one window is the smallest domain unit.
Fine-grained feature update
: this step updates the feature extractor using the proposed pseudo domain-class labels as the supervision.
Latent distribution characterization: it aims to identify the domain label for each instance to obtain the latent distribution information.
It maximizes the different distribution gaps to enlarge diversity.
Domain-invariant representation learning: this step utilizes pseudo domain labels from the last step to learn domain-invariant representations and train a generalizable model.
Fine-grained Feature Update. Before characterizing the latent distributions, we perform ﬁnegrained feature updates to obtain ﬁne-grained representation.
As shown in Figure 2 (blue), we propose a new concept: pseudo domain-class label to fully utilize the knowledge contained in domains and classes, which serves as the supervision for feature extractor.
Features are more ﬁne-grained w.r.t.
domains and labels, instead of only attached to domains or labels.
At the ﬁrst iteration, there is no domain label d (cid:48) and we simply initialize d (cid:48)
=
for all samples.
We treat per category per domain as a new class with label s ∈ { 1 , 2 , · · · , S } .
We have S = K × C where K is the pre-deﬁned number of latent distributions that can be tuned in experiments.
We perform pseudo domain-class label assignment to get discrete values for supervision: s = d (cid:48)
× C
+ y.
Let h (2) f , h (2) b , h (2) c be feature extractor, bottleneck, and classiﬁer, respectively (we use superscripts to denote step number).
Then, the supervised loss is computed using the cross-entropy loss (cid:96) :
[Equation]
Latent Distribution Characterization. This step characterizes the latent distributions contained in one dataset.
As shown in Figure 2 (green), we propose an adapted version of adversarial training to disentangle the domain labels from the class labels.
However, there are no actual domain labels provided, which hinders such disentanglement.
Inspired by (Caron et al., 2018), we employ a self-supervised pseudo-labeling strategy to obtain domain labels.
First, we attain the centroid for each domain with class-invariant features:
[Equation]
where h (3) f , h (3) b , h (3) c are feature extractor, bottleneck, and classiﬁer, respectively.
˜ µ k is the initial centroid of the k th latent domain while δ k is the k th element of the logit soft-max output.
Then, we obtain the pseudo domain labels via the nearest centroid classiﬁer using a distance function D :
[Equation]
Then, we compute the centroids and obtain the updated pseudo domain labels:
[Equation]
where I ( a ) = 1 when a is true, otherwise 0 .
After obtaining d (cid:48) , we can compute the loss of step 2:
[Equation]
[Equation]
where h (3) adv is the discriminator for step 3 that contains several linear layers and one classiﬁcation layer.
R λ 1 is the gradient reverse layer with hyperparameter λ 1 (Ganin et al., 2016).
After this step, we can obtain pseudo domain label d (cid:48) for x .
Domain-invariant Representation Learning. After obtaining the latent distributions, we learn domain-invariant representations for generalization.
In fact, this step (purple in Figure 2) is simple: we borrow the idea from DANN (Ganin et al., 2016) and directly use adversarial training to update the classiﬁcation loss L cls and domain classiﬁer loss L dom using gradient reversal layer (GRL) (a common technique that facilitates adversarial training via reversing gradients)
(Ganin et al., 2016):
[Equation]
where (cid:96) is the cross-entropy loss and R λ 2 is the gradient reverse layer with hyperparameter λ 2 (Ganin et al., 2016).
We will omit the details of GRL and adversarial training here since they are common techniques in deep learning.
More details are presented in Appendix B.2.
Training, Inference, and Complexity.
We repeat these steps until convergence or max epochs.
Different from existing methods, the last two steps only optimize the last few independent layers but not the feature extractor.
We perform inference with the modules from the last step.
Most of the trainable parameters are shared between modules, indicating that D IVERSIFY has the same model size as existing methods and can reach quick convergence in experiments (Figure F.5).
2. T HEORETICAL I NSIGHTS
We present some theoretical insights to show that our approach is well motivated in theory.
Proofs can be found in Appendix A.
Proposition 2.1.
Let X be a space and H be a class of hypotheses corresponding to this space.
Let Q and the collection { P i } Ki =1 be distributions over X and let { ϕ i } Ki =1 be a collection of non-negative coefﬁcient with (cid:80) i ϕ i
= 1 .
Let O be a set of distributions s.t.
∀ S ∈ O , the following holds
[Equation]
Then, for any h ∈ H ,
[Equation]
where λ (cid:48) is the error of an ideal joint hypothesis.
ε P ( h ) is the error for a hypothesis h on a distribution P .
d H ∆ H ( P , Q ) is H -divergence which measures differences in distribution (Ben-David et al., 2010).
The ﬁrst item in Eq.
(8), λ (cid:48) , is often neglected since it is small in reality.
The second item, (cid:80) i ϕ i
ε P
i ( h ) , exists in almost all methods and can be minimized via supervision from class labels with cross-entropy loss in Eq.
Our main purpose is to minimize the last two items in Eq.
Here Q corresponds to the unseen out-of-distribution target domain.
The last term 12 max i,j d H ∆ H
( P i , P j ) is common in OOD theory which measures the maximum differences among source domains.
This corresponds to step 4 in our approach.
Finally, the third item, 12 min S ∈O d H ∆ H ( S , Q ) , explains why we exploit sub-domains in step 3.
Since our goal is to learn a model which can perform well on an unseen target domain, we cannot obtain Q .
To minimize 12 min S ∈O d H ∆ H ( S , Q ) , we can only enlarge the range of O .
We have to max i,j d H ∆ H
( P i , P j ) according to Eq.
(7), corresponding to step 3 in our method which tries to segment the time series data into several latent sub-domains by maximizing the segment-wise distribution gap to preserve diversities, i.e., the ‘worst-case’ distribution scenario.
E XPERIMENTS
We perform evaluations on four diverse time series classiﬁcation tasks: gesture recognition, speech commands recognition, wearable stress&affect detection, and sensor-based activity recognition.
Time series OOD algorithms are currently less studied and there are only two recent strong approaches for comparison: GILE (Qian et al., 2021) and AdaRNN (Du et al., 2021).
5 We further compare with 7 general OOD methods 6 from DomainBed (Gulrajani & Lopez-Paz, 2021): ERM, DANN (Ganin et al., 2016), CORAL (Sun & Saenko, 2016), Mixup (Zhang et al., 2018), GroupDRO (Sagawa et al., 2020), RSC (Huang et al., 2020), and ANDMask (Parascandolo et al., 2021).
More details of these methods are in Sec.
B.2 and B.3.
For fairness, all methods (except GILE and AdaRNN) use a feature net with two blocks and each block has one convolution layer, one pooling layer, and one batch normalization layer, following (Wang et al., 2019).
We also use Transformers (Vaswani et al., 2017) for backbone.
Detailed data pre-processing, architecture, and hyperparameters are in Appendix C.5 and D. Ablations with various backbones are in Figure 8 and Appendix F.4.
Most OOD methods require the domain labels known in training while ours does not, which is more challenging and practical.
We conduct the training-domain-validation strategy and the training data are split by 8 : 2 for training and validation.
We tune all methods to report the average best performance of three trials for fairness.
Note that the “target” in experiments is unseen and only used for testing.
K in D IVERSIFY is treated as a hyperparameter and we tune it to record the best OOD performance.
7 Per-segment accuracy is the evaluation metric.
Time complexity and convergence are in Sec.
F.5 , showing its quick convergence.
3. G ESTURE R ECOGNITION
[Table]
First, we evaluate D IVERSIFY on EMG for gestures Data
Set (Lobov et al., 2018).
It contains data of 36 subjects with 7 classes and we select 6 common classes for our experiments.
We randomly divide 36 subjects into four domains (i.e., 0 , 1 , 2 , 3 ).
More details on EMG and domain splits can be found in Sec.
C.2 and C.6 respectively.
EMG data is affected by many factors since it comes from bioelectric signals.
EMG data are scene and device-dependent, which means the same person may generate different data when performing the same activity with the same device at a different time (i.e., distribution shift across time (Wilson et al., 2020; Purushotham et al., 2016)) or with the different devices at the same time.
Thus, the EMG benchmark is challenging.
Table 1 shows that with the same backbone, our method achieves the best average performance and is 4 .
3 % better than the second-best method.
D IVERSIFY even outperforms AdaRNN which has a stronger backbone.
3. S PEECH C OMMANDS
[Table]
[Figure]
Then, we adopt a regular speech recognition task, the Speech Commands dataset (Warden, 2018).
It consists of one-second audio recordings of both background noise and spoken words such as ‘left’ and ‘right’.
It is collected from more than 2,000 persons, thus is more complicated.
Following (Kidger et al., 2020), we use 34, time series corresponding to ten spoken words to produce a balanced classiﬁcation problem.
Since this dataset is collected from multiple persons, the training and test distributions are different, which is also an OOD problem with one training domain.
There are many subjects and each subject only records a few audios.
Thus, we do not split each sample.
Figure 3 shows the results on two different backbones.
Compared with GroupDRO, D IVERSIFY has over 1 % improvement with a basic CNN backbone and over 0 .
6 % improvement with a strong backbone MatchBoxNet3-1-64 (Majumdar & Ginsburg, 2020).
It demonstrates the superiority of our method on a regular time-series benchmark containing massive distributions.
3. W EARABLE S TRESS AND A FFECT D ETECTION
[Figure]
Here, 0 ∼ 4 in x-axis denotes the unseen test dataset.
We further evaluate D IVERSIFY on a larger dataset, Wearable Stress and Affect Detection (WESAD)
(Schmidt et al., 2018).
WESAD is a public dataset that contains physiological and motion data ofsubjects with 63 , 000 , 000 instances.
We utilize sensor modalities of chest-worn devices including electrocardiogram, electrodermal activity, electromyogram, respiration, body temperature, and three axis acceleration.
We split 15 subjects into four domains (details are in Sec.
C.6).
Results Figure 4 showed that our method achieves the best performance compared to other state-of-the-art methods with an improvement of over 8 % on this larger dataset.
3. S ENSOR - BASED H UMAN A CTIVITY R ECOGNITION
Finally, we construct four diverse OOD settings by leveraging four sensor-based human activity recognition datasets: DSADS (Barshan & Y¨uksek, 2014), USC-HAD (Zhang & Sawchuk, 2012),
UCI-HAR (Anguita et al., 2012), and PAMAP (Reiss & Stricker, 2012).
These datasets are collected from different people and positions using accelerometer and gyroscope, with 11 , 741 , 000 instances in total.
(1) Cross-person generalization aims to learn generalized models for different persons.
(2) Cross-position generalization aims to learn generalized models for different sensor positions.
(3) Cross-dataset generalization aims to learn generalized models for different datasets.
(4) OnePerson-To-Another aims to learn generalized models for different persons from data of a single
[Table] [Table]
person.
8 For simplicity, we use 0 , 1 , · · · to denote different domains.
More details on datasets information, setting construction, and domain splits can be found in Sec.
C.3, C.4, and C.6.
Table 2 and Table 3 show the results on four settings for HAR, where our method signiﬁcantly outperforms the second-best baseline by 2 . 4 % , 1 . 4 % , 9 .
9 % , and 5 . 8 % respectively.
All results demonstrate the superiority of D IVERSIFY .
More results using Transformer can be found in F.4.
We observe more insightful conclusions.
(1) When the task is difﬁcult:
In the Cross-Person setting,
USC-HAD may be the most difﬁcult task.
Although it has more samples, it contains 14 subjects with only two sensors on one position, which may bring more difﬁculty in learning.
The results prove the above argument that all methods perform terribly on this benchmark while ours has the largest improvement.
(2) When datasets are signiﬁcantly more diverse:
Compared to Cross-Person and
Cross-Position settings, Cross-Dataset may be more difﬁcult since all datasets are totally different and samples are inﬂuenced by subjects, devices, sensor positions, and some other factors.
In this setting, our method is substantially better than others.
(3) Limited data: Compared with Cross-Person setting,
One-Person-To-Another is more difﬁcult since it has fewer data samples.
In this case, enhancing diversity can bring a remarkable improvement and our method can boost the performance.
3. A NALYSIS
Ablation study We present ablation study to answer the following three questions.
(1) Why obtaining pseudo domain labels with class-invariant features in step 3?
If we obtain pseudo domain labels with common features, domain labels may have correlations with class labels, which may introduce 8 In One-Person-To-Another setting, we only report average accuracy of four tasks on each dataset.
Since only one domain exists in training dataset for this setting, DANN and CORAL cannot be implemented here.
[Figure] [Figure]
[Figure]
contradictions when learning domain-invariant representations and lead to common performance.
This is certiﬁed by the results in Figure 5(a).
(2) Why using ﬁne-grained domain-class labels in step 2?
If we utilize pseudo domain labels to update the feature net, it may make the representations seriously biased towards domain-related features and thereby leads to terrible performance on classiﬁcation, which is proved in Figure 5(b).
If we only utilize class labels to update the feature net, it may make representations biased to class-related features, thus D IVERSIFY is unable to obtain true latent sub-domains, as shown in Figure 5(c).
Hence, we should employ ﬁne-grained domain-class labels to obtain representations with both domain and class information.
(3) The more latent domains, the better?
More latent domains may not bring better results (Figure 5(d)) since a dataset may only have a few latent domains and introducing more may contradict its intrinsic data property.
Plus, more latent domains also make it harder to obtain pseudo domain labels and learn domain-invariant features.
Existence of latent distributions What exactly can our D IVERSIFY learn?
In Figure 6(a), for a subject in USC-HAD, there is more than one latent distribution for his walking activities, showing the existence of temporal distribution shift: the distribution of the same activity could change.
For spatial distribution shift, Figure 6(b) on EMG dataset shows that our algorithm found three latent distributions from the EMG data of multiple people.
These results indicate the existence of latent distributions with both temporal and spatial distribution shifts.
9 More results are in Appendix F.1.
Quantitative analysis for ‘worst-case’ distributions We present quantitative analysis by computing the H -divergence (Ben-David et al., 2010) to show the effectiveness of our ‘worst-case distribution’.
As shown in Figure 6(c) and 6(d), compared to initial domain splits, latent sub-domains generated by our method have larger H -divergence among each other.
According to Prop. 2.1, larger H -divergence among domains brings better generalization.
This again shows the efﬁcacy of
D IVERSIFY in computing the ’worst-case’ distribution scenario.
More results are in Appendix F.3.
Visualization study We present some visualizations to show the rationales of D IVERSIFY .
Data points with different initial domain labels are mixed together in Figure 7(a) while D IVERSIFY can characterize different latent distributions and separate them well in Figure 7(b).
Figure 7(d) and 7(c) show that D IVERSIFY can learn better domain-invariant representations compared to the latest method ANDMask.
To sum up, D IVERSIFY can ﬁnd better latent domains to enhance generalization.
More results are in Appendix F.1.
Varying backbones Figure 8 shows the results using small, medium, and large backbones, respectively (we implement them with different numbers of layers.).
Results indicate that larger models tend and the experimental results above prove that paying attention to shifts comprehensively can bring larger divergence and better results.
to achieve better OOD generalization performance.
Our method outperforms others in all backbones, showing that D IVERSIFY presents consistently strong OOD performance in different architectures.
More results with Transformer are in Appendix F.
Parameter sensitivity is in Appendix F.2.
R ELATED W ORK
[Figure]
Time series classiﬁcation is a challenging problem.
Researches mainly focus on temporal relation modeling via specially-designed methods (Dempster et al., 2021), RNN-based networks (Dennis et al., 2019), or Transformer architecture (Drouin et al., 2022).
To our best knowledge, there is only one recent work (Du et al., 2021) that studied time series from the distribution level.
However, AdaRNN is a two-stage non-differential method that is tailored for RNN.
Domain / OOD generalization (Wang et al., 2022; Lu et al., 2022) typically assumes the availability of domain labels for training.
Speciﬁcally, Matsuura & Harada (2020) also studied DG without domain labels by clustering with the style features for images, which is not applied to time series and is not end-to-end trainable.
Disentanglement (Peng et al., 2019; Zhang et al., 2022b) tries to disentangle the domain and label information, but they also assume access to domain information.
Single domain generalization is similar to our problem setting which also involves one training domain (Fan et al., 2021;
Li et al., 2021; Wang et al., 2021; Zhu & Li, 2022).
However, they treated the single domain as one distribution and did not explore latent distributions.
Multi-domain learning is similar to DG which also trains on multiple domains, but also tests on training distributions. Deecke et al. (2022) proposed sparse latent adapters to learn from unknown domain labels, but their work does not consider the min-max worst-case distribution scenario and optimization.
In domain adaptation , Wang et al. (2020) proposed the notion of domain index and further used variational models to learn them (Xu et al., 2023), but took a different modeling methodology since they did not consider min-max optimization.
Mixture models (Rasmussen et al., 1999) are models representing the presence of subpopulations within an overall population, e.g., Gaussian mixture models.
Our approach has a similar formulation but does not use generative models.
Subpopulation shift is a new setting (Koh et al., 2021) that refers to the case where the training and test domains overlap, but their relative proportions differ.
Our problem does not belong to this setting since we assume that these distributions do not overlap.
Distributionally robust optimization (Delage & Ye, 2010) shares a similar paradigm with our work, whose paradigm is also to seek a distribution that has the worst performance within a range of the raw distribution.
GroupDRO (Sagawa et al., 2020) studied DRO at a group level.
However, we study the internal distribution shift instead of seeking a global distribution close to the original one.
L IMITATION AND D ISCUSSION
D IVERSIFY could be more perfect by pursuing the following avenues.
1) Estimate the number of latent distributions K automatically: we currently treat it as a hyperparameter.
Seek the semantics behind latent distributions (e.g., Figure 6(a)): can adding more human knowledge obtain better latent distributions? 3)
Extend D IVERSIFY beyond classiﬁcation, but for forecasting problems.
Moreover, we argue that dynamic distributions not only exist in time series, but also in general machine learning data such as images and text (Deecke et al., 2022; Xu et al., 2023).
Thus, it is of great interest to apply our approach to these domains to further improve their performance.
C ONCLUSION
We proposed D IVERSIFY to learn generalized representation for time series classiﬁcation.
D IVERSIFY employs an adversarial game that maximizes the ‘worst-case’ distribution scenario while minimizing their distribution divergence.
We demonstrated its effectiveness in different applications.
We are surprised that not only a mixed dataset, but one dataset from a single person can also contain several latent distributions.
Characterizing such latent distributions will greatly improve the generalization performance on unseen datasets.
ACKNOWLEDGEMENT
This work is supported by National Key Research & Development Program of China (No.
2020YFC2007104), Natural Science Foundation of China (No. 61972383), the Strategic Priority Research Program of Chinese Academy of Sciences ( No. XDA28040500), Science Research Foundation of the Joint Laboratory Project on Digital Ophthalmology and Vision Science
(No.SZYK202201)
C ONTENTS A Theoretical Insights
A.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Proof of Proposition 2.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Method Details
B.1 Domain-invariant Representation Learning . .
B.2 Method Formulation and Implementation . . . . . . . . . . . . . . . . . . . . . .
B.3 Comparisons to Other Latest Methods . . . . . . . . . . . . . . . . . . . . . . . .
C Dataset
C.1 Datasets Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 More details on EMG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Details on Sensor-based Human Activity Recognition Dataset . . . . . . . . . . .
C.4 Details on Different Settings for Human Activity Recognition . . . . . . . . . . . .
C.5 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.6 Details on Domain Splits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Network Architecture and Hyperparameters
E Evaluation Metrics
F More Experimental Results
F.
Visualization Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Parameter Sensitivity . .
F.
H -divergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
The Inﬂuence of Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Time Complexity and Convergence Analysis . . . . . . . . . . . . . . . . . . . . .
A T HEORETICAL I NSIGHTS A. B ACKGROUND
For a distribution P with an ideal binary labeling function h ∗ and a hypothesis h , we deﬁne the error ε P ( h ) in accordance with (Ben-David et al., 2010) as:
[Equation]
We also give the deﬁnition of H -divergence according with (Ben-David et al., 2010).
Given two distributions P , Q over a space X and a hypothesis class H ,
[Equation]
(10) where I h
= { x ∈ X| h ( x ) = 1 } .
We often consider the H ∆ H -divergence in (Ben-David et al., 2010) where the symmetric difference hypothesis class H ∆ H is the set of functions characteristic to disagreements between hypotheses.
Theorem A.1.
(Theorem 2.1 in (Sicilia et al., 2021), modiﬁed from Theorem 2 in (Ben-David et al., 2010)).
Let X be a space and H be a class of hypotheses corresponding to this space.
Suppose P and Q are distributions over X .
Then for any h ∈ H , the following holds
[Equation]
with λ (cid:48)(cid:48) the error of an ideal joint hypothesis for Q , P .
Theorem A.1 provides an upper bound on the target-error.
λ (cid:48)(cid:48) is a property of the dataset and hypothesis class and is often ignored.
Theorem A.1 demonstrates the necessity to learn domain invariant features.
A. P ROOF OF P ROPOSITION 2.1.
Proposition 2.1.
Let X be a space and H be a class of hypotheses corresponding to this space.
Let Q and the collection { P i } Ki =1 be distributions over X and let { ϕ i } Ki =1 be a collection of non-negative coefﬁcient with (cid:80) i ϕ i
= 1 .
Let the object O be a set of distributions such that for every S ∈ O the following holds
[Equation]
Then, for any h ∈ H ,
[Equation]
where λ (cid:48) is the error of an ideal joint hypothesis.
Proof.
On one hand, with Theorem A.1, we have
[Equation] [Equation]
On the other hand, with Theorem A.1, we have
[Equation] [Equation]
Since ε (cid:80) i ϕ i
P i ( h ) =
(cid:80) i ϕ i
ε P
i ( h ) , and d H ∆ H ( (cid:80) i ϕ i
P i , S ) ≤ max
i,j d H ∆ H
( P i , P j ) , we have ε Q ( h ) ≤ λ (cid:48)
+ (cid:88)
[Equation] [Equation]
where λ (cid:48) =
λ (cid:48)
+ λ (cid:48) 2 .
Equation 16 for all S ∈ O holds.
Therefore, we complete the proof.
B M ETHOD D ETAILS B. D OMAIN - INVARIANT R EPRESENTATION L EARNING
Domain-invariant representation learning utilizes adversarial training which contains a feature network, a domain discriminator, and a classiﬁcation network.
The domain discriminator tries its best to discriminate domain labels of data while the feature network tries its best to generate features to confuse the domain discriminator, which thereby obtains domain-invariant representation.
Therefore, it is an adversarial process, and in our setting, it can be expressed as follows,
[Equation] [Equation]
To optimize Eq.
(17), we need an iterative process to optimize h (4) b , h (4) c and h (4) adv iteratively, which is cumbersome.
It is better to optimize h (4) b , h (4) c and h (4) adv at the same time.
It is obvious that the key is to solve the problems caused by the negative sign in Eq.
Therefore, a special gradient reversal layer (GRL), a popular implementation of the adversarial training in training several domains as suggested by (Ganin et al., 2016), came.
GRL acts as an identity transformation during the forward propagation while it takes the gradient from the subsequent level and changes its sign before passing it to the preceding layer during the backpropagation.
During the forward propagation, the GRL can be ignored.
During the backpropagation, the GRL makes the sign of gradient on h (4) b reverse, which solves the problems caused by the negative sign in Eq.
B. M ETHOD F ORMULATION AND I MPLEMENTATION
While it is common to use some probability or Bayesian approaches for formulation when one mentions distributions, we actually do not formulate the latent distributions: we are not a generative or parametric method.
In fact, the concept of latent distribution is just a notion to help understand our method.
Our ultimate goal is to infer which distribution a segment belongs to for best OOD performance.
Thus, we do not care what a distribution exactly looks like or even parameterize it since it is not our focus.
As long as we can obtain diverse latent distributions, things are all done.
In real implementation, the latent distributions are just represented as domain labels: latent distribution i is also a domain i that certain time series segments belong to, as we stated in the introduction part.
Additionally, we also acknowledge that parameterizing the latent distributions may help to get better performance, which can be left for future research.
B. C OMPARISONS TO O THER L ATEST M ETHODS
Here, we offer more details on comparisons to other latest methods utilized in the main paper.
DANN (Ganin et al., 2016) is a method that utilizes the adversarial training to force the discriminator unable to classify domains for better domain-invariant features.
It requires domain labels and splits data in advance while ours is a universal method.
CORAL (Sun & Saenko, 2016) is a method that utilizes the covariance alignment in feature layers for better domain-invariant features.
It also requires domain labels and splits data in advance.
Mixup (Zhang et al., 2018) is a method that utilizes interpolation to generate more data for better generalization.
Ours mainly focuses on generalized representation learning.
GroupDRO (Sagawa et al., 2020) is a method that seeks a global distribution with the worst performance within a range of the raw distribution for better generalization.
Ours study the internal distribution shift instead of seeking a global distribution close to the original one.
RSC (Huang et al., 2020) is a self-challenging training algorithm that forces the network to activate features as much as possible by manipulating gradients.
It belongs to gradient operation-based DG while ours is to learn generalized features.
ANDMask (Parascandolo et al., 2021) is another gradient-based optimization method that belongs to special learning strategies.
Ours focuses on representation learning.
GILE (Qian et al., 2021) is a disentanglement method designed for cross-person human activity recognition.
It is based on VAEs and requires domain labels.
AdaRNN (Du et al., 2021) is a method with a two-stage that is non-differential and it is tailored for RNN.
A speciﬁc algorithm is designed for splitting.
Ours is universal and is differential with better performance.
C D ATASET
C. D ATASETS I NFORMATION
Table 4 shows the statistical information on each dataset.
[Table]
M ORE DETAILS ON EMG
Electromyography (EMG) is a typical time-series data that is based on bioelectric signals.
We use
EMG for gestures Data Set (Lobov et al., 2018) that contains raw EMG data recorded by MYO
Thalmic bracelet.
The bracelet is equipped with eight sensors equally spaced around the forearm that simultaneously acquire myographic signals.
Data of 36 subjects are collected while they performed series of static hand gestures and the number of instances is 40 , 000 − 50 , 000 recordings in each column.
It contains 7 classes and we select 6 common classes for our experiments.
We randomly divide 36 subjects into four domains (i.e., 0 , 1 , 2 , 3 ) without overlapping and each domain contains data of 9 persons.
C. D ETAILS ON S ENSOR - BASED H UMAN A CTIVITY R ECOGNITION D ATASET
UCI daily and sports dataset ( DSADS ) (Barshan & Y¨uksek, 2014) consists of 19 activities collected from 8 subjects wearing body-worn sensors on 5 body parts.
USC-SIPI human activity dataset ( USC-HAD ) (Zhang & Sawchuk, 2012) is composed of 14 subjects (7 male, 7 female, aged 21 to 49) executing 12 activities with a sensor tied on the front right hip.
UCI-HAR (Anguita et al., 2012) is collected by 30 subjects performing 6 daily living activities with a waist-mounted smartphone.
PAMAP (Reiss & Stricker, 2012) contains data of 18 activities, performed by 9 subjects wearingsensors.
C. D ETAILS ON D IFFERENT S ETTINGS FOR H UMAN A CTIVITY R ECOGNITION
We construct four different settings representing different degrees of generalization: (1) Crossperson generalization : This setting utilizes DSADS, USC-HAD, PAMAP 10 datasets to construct three benchmarks.
Within each dataset, we randomly split the data into four groups and then use three groups as training data to learn a generalized model for the last group.
(2) Cross-position generalization : this setting uses DSADS dataset and data from each position denotes a different domain.
Each sample contains three sensors with nine dimensions.
We treat one position as the test domain while the others are for training.
(3) Cross-dataset generalization : this setting uses all four datasets, and each dataset corresponds to a different domain.
Six common classes are selected.
Two sensors from each dataset that belong to the same position are selected and data is down-sampled to have the same dimension.
(4) One-Person-To-Another .
This setting adopts DSADS, USC-HAD, and PAMAP datasets.
In each dataset, we randomly select four pairs of persons where one is the training and the other is the test.
C. D ATA P REPROCESSING
We will introduce how we preprocess data and the ﬁnal dimension of data for experiments here.
We mainly utilize the sliding window technique, a common technique in time-series classiﬁcation, to split data.
As its name suggests, this technique involves taking a subset of data from a given array or sequence.
Two main parameters of the sliding window technique are the window size, describing a subset length, and the step size, describing moving forward distance each time.
For EMG, we set the window size 200 and the step size 100, which means there exist 50% overlaps between two adjacent samples.
We normalize each sample with ˜ x
= x − min X max X − min X .
X contains all x .
The ﬁnal dimension is 8 × 1 × 200 .
For Speech Commands, we follow (Kidger et al., 2020).
For WESAD, we utilize the same preprocessing as EMG.
Now we give details on all datasets in Cross-person setting.
For DSADS, we directly utilize data split by the providers.
The ﬁnal dimension shape is 45 × 1 × 125 .
45 = 5 × 3 × 3 where 5 means ﬁve positions, the ﬁrst 3 means three sensors, and the second 3 means each sensor has three axes.
For USC-HAD, the window size is 200 and the step size is 100.
The ﬁnal dimension shape is 6 × 1 × 200 .
For PAMAP, the window size is 200 and the step size is 100.
The ﬁnal dimension shape is 27 × 1 × 200 .
For UCI-HAR, we directly utilize data split by the providers.
The ﬁnal dimension shape is 6 × 1 × 128 .
For Cross-position, we directly utilize samples obtained from DSADS in Cross-person setting.
Since each position corresponds to one domain, a sample is split into ﬁve samples in the ﬁrst dimension.
And the ﬁnal dimension shape is 9 × 1 × 125 .
For Cross-dataset, we directly utilize samples obtained in Cross-person setting.
To make all datasets share the same label space and input space, we select six common classes, including WALKING,
WALKING UPSTAIRS, WALKING DOWNSTAIRS, SITTING, STANDING, LAYING.
In addition, we down-sample data and select two sensors from each dataset that belong to the same position.
The ﬁnal dimension shape is 6 × 1 × 50 .
For One-Person-To-Another, we randomly select four pairs of persons from DSADS, USC-HAD, and
PAMAP respectively.
Four tasks are 1 → 0 , 3 → 2 , 5 → 4 , and 7 → 6 .
Each number corresponds to one subject.
And the ﬁnal dimension shape is 45 × 1 × 125 , 6 × 1 × 200 , and 27 × 1 × 200 for
DSADS, USC-HAD, and PAMAP respectively.
As we can see, samples in EMG, WESAD, and HAR all have more than one channel (the ﬁrst dimension shape), which means they are all multivariate.
C. D ETAILS ON D OMAIN S
[Table]
We introduce how we split data here.
Since Speech Commands is a regular task, we just randomly split the entire dataset into a training dataset, a validation dataset, and a testing dataset.
We mainly focus on EMG, WESAD, and HAR, and we construct domains for OOD tasks.
We denote subjects of a dataset with 0 − s n , where s n is the number of subjects in the dataset.
For example, there are 36 subjects in EMG and we utilize 0 , 1 , 2 , · · · , 35 to denote data of them respectively.
Table 5 shows the initial domain splits of EMG, WESAD, and all datasets for HAR in Cross-person setting.
We just want to make each domain has a similar number of samples in one dataset.
As noted in the main paper, we also utilize 0, 1, 2, and 3 to represent different domains but they have different meanings with subjects.
When conducting experiments, we take one domain as the testing data and the others as the training data.
Our method is not inﬂuenced by the splits of the training data since we do not need the domain labels.
[Table]
D N ETWORK A RCHITECTURE AND H YPERPARAMETERS
For the architecture, the model contains two blocks, and each has one convolution layer, one pooling layer, and one batch normalization layer.
A single-fully-connected layer is used as the bottleneck layer while another fully-connected layer serves as the classiﬁer.
All methods are implemented with
PyTorch (Paszke et al., 2019).
The maximum training epoch is set to 150.
The Adam optimizer with weight decay 5 × 10 − 4 is used.
The learning rate for GILE is 10 − 4 .
The learning rate for the rest methods is 10 − 2 or 10 − 3 .
(For Speech Commands with MatchBoxNet3-1-64, we also try the learning rate, 10 − 4 .)
We tune hyperparameters for each method.
For the pooling layer, we utilize MaxPool2d in PyTorch.
The kernel size is (1 , 2) ad the stride is 2.
For the convolution layer, we utilize Conv2d in PyTorch.
Different tasks have different kernel sizes and Table 6 shows the kernel sizes.
[Table]
E E VALUATION M ETRICS
We utilize average accuracy on the testing dataset as our evaluation metrics for all benchmarks.
Average accuracy is the most common metric for DG and it can be computed as the following,
[Equation]
I y ( y ∗ ) is an indicator function.
If y = y ∗ , it equals 1, otherwise it equals 0.
# | · | represents the number of the set.
h is the model to learn.
Please note that X te has a different distribution from X tr for EMG and HAR.
And x has been preprocessed and each sample is a segment.
[Figure]
on EMG.
Different colors correspond to different classes while different shapes correspond to different domains.
[Figure]
F M ORE E XPERIMENTAL R ESULTS F. V ISUALIZATION S TUDY
We show more visualization study in this part.
As shown in Figure 9(a) and Figure 9(b), both ERM and RSC also cannot obtain ﬁne domain-invariant representations and our method still achieves the best domain-invariant representations.
As shown in Figure 10, compared with initial domain splits, latent sub-domains generated by our method are better separated.
F. P ARAMETER S ENSITIVITY
[Figure]
There are mainly four hyperparameters in our method: K which is the number of latent sub-domains, λ 1 for the adversarial part in step 3, λ 2 for the adversarial part in step 4, and local epochs and total rounds.
For fairness, the product of local epochs and total rounds is the same value.
We evaluate the parameter sensitivity of our method in Figure 11 where we change one parameter and ﬁx the other to record the results.
From these results, we can see that our method achieves better performance in a wide range, demonstrating that our method is robust.
F. H - DIVERGENCE
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
Figure 12 shows H -divergence among domains with initial splits and our splits on EMG, which demonstrates our splits have larger H -divergence and thereby can bring better generalization.
F. T HE I NFLUENCE OF A RCHITECTURES
To ensure that our method can work with different sizes of models, we add some more experiments with more complex or simpler architectures.
As shown in Table 7, where small, medium, and large indicate the different model sizes (our paper uses the medium), we see a clear picture that model sizes inﬂuence the results, and our method also achieves the best performance.
Small corresponds to the model with one convolutional layer, Medium corresponds to the model with two convolutional layers, and Large corresponds to the model with four convolutional layers.
For most methods, more complex models bring better results.
[Table] [Table]
DANN 72.
GroupDRO 72.
RSC 72.
Ours 75.
ERM
DANN
RSC
Ours
CNN
Transformer
We also try Transformer (Vaswani et al., 2017) as the backbone for comparisons.
As shown in (Zhang et al., 2022a), Transformer often has a better generalization ability compared to CNN, which implies improving with Transformer is more difﬁcult.
From Table 8, we can see that each method with Transformer has a remarkable improvement on EMG.
Compared to ERM, DANN almost has no improvement but ours still has further improvements and achieves the best performance.
To further validate the advantage of our method, we perform the experiments on a more difﬁcult task, i.e. the ﬁrst task of cross-dataset where distribution gaps are larger.
As shown in Figure F.4, our method still achieves the best performance in this more difﬁcult situation while DANN even performs worse than ERM, which demonstrates the importance of more accurate sub-domain labels.
Overall, for all architectures, our method achieves the best performance.
method
ERM DANN
CORAL
Mixup
GroupDRO
RSC
ANDMask
Ours time 305.
F.
T IME C OMPLEXITY AND C ONVERGENCE
A NALYSIS
[Figure]
results on EMG.
We also provide some analysis on time complexity and convergence.
Since we only optimize the feature extractor in Step 2, our method does not cost too much time.
And the results in Table 9 prove this argument empirically.
The convergence results are shown in Figure F.5.
Our method is convergent.
Although there are some little ﬂuctuations, these ﬂuctuations exist widely in all domain generalization methods due to different distributions of different samples.