D IVERSIFY TO G ENERALIZE : L EARNING G ENERALIZED R EPRESENTATIONS FOR T IME S ERIES C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Time series classiﬁcation is an important problem in real world.
Due to its nonstationary property that the distribution changes over time, it remains challenging to build models for generalization to unseen distributions.
In this paper, we propose to view the time series classiﬁcation problem from the distribution perspective.
We argue that the temporal complexity attributes to the unknown latent distributions within.
To this end, we propose D IVERSIFY to learn generalized representations for time series classiﬁcation.
D IVERSIFY takes an iterative process: it ﬁrst obtains the worst-case distribution scenario via adversarial training, then matches the distributions between all segments.
We also present some theoretical insights.
Extensive experiments on gesture recognition, speech commands recognition, and sensor-based human activity recognition demonstrate that D IVERSIFY signiﬁcantly outperforms other baselines and effectively characterizes the latent distributions by qualitative and quantitative analysis.
I NTRODUCTION
Time series classiﬁcation is one of the most challenging problems in machine learning and statistics community.
Example applications include sensor-based human activity recognition, Parkinson’s disease diagnosis, and electronic power consumption (Fawaz et al., 2019).
One important nature of time series is non-stationary property, which means that its statistical features are changing over time.
For years, there have been tremendous efforts to tackle the time series classiﬁcation problem, such as hidden Markov models (Fulcher & Jones, 2014), RNN-based methods (H ¨ usken & Stagge, 2003), and Transformer-based approaches (Li et al., 2019).
In this paper, we are speciﬁcally interested in modeling time series from the distribution perspective.
More precisely, we aim to learn representations for time series that can generalize to unseen distributions.
Note that this scenario has been extensively studied in existing literature of domain generalization (Muandet et al., 2013; Wang et al., 2021a) and out-of-distribution generalization (Krueger et al., 2021), where researchers are keen to bridge the gap between known and unknown distributions, thus generalize well.
While most of the efforts are done in the image context, few of them focus on the time series domain, which is more challenging.
Although time series share a similar goal as image data in domain generalization, it naturally brings more challenges due to its nonstationary property: the distribution keeps changing over time, which contains diverse distribution information that should be harnessed well for better generalization.
We show an illustrative example in Figure 1.
Domain generalization in image data often involves several domains and the domain information is known (subﬁgure (a)).
Thus, we can leverage such domain information to build generalization models.
However, in Figure 1 (b), we see that in time series data, although the distribution is changing dynamically over time, its domain information is not available.
This will dramatically impede the modeling of existing domain generalization algorithms as they typically assume access to domain information (subﬁgure (c)).
In order to learn a generalized time series model, we propose D IVERSIFY , a domain generalization algorithm to characterize the latent distributions inside the time series data.
Concretely speaking, our method consists of a min-max adversarial game that: on one hand, it learns to segment the time series data into several latent sub-domains by maximizing the segment-wise distribution gap to preserve
[Figure]
diversities, i.e., the worst-case distribution scenario; on the other hand, it learns domain-invariant representations by reducing the distribution divergence for the worst-case scenario.
Such diversiﬁcation naturally exists in a non-stationary dataset where the data from multiple people naturally follow several latent distributions.
Moreover, it is also surprising to ﬁnd that even the data of one person still has such diversiﬁcation: it can also be split into several latent distributions.
Obviously, D IVERSIFY can effectively characterize the latent distributions (Figure 1 (d)). To summarize, our contributions are three-fold:
• Novel problem: For the ﬁrst time, we identify the generalized representation problem of time series data, which is challenging than traditional image classiﬁcation problem due to the existence of unidentiﬁed latent distributions.
• New methodology:
We propose D IVERSIFY , a theoretically-motivated solution to solve the generalized representation learning problem to identify the latent distributions.
• Good performance: Our approach is extensively evaluated in three types of tasks: gesture recognition, speech command recognition, and sensor-based activity recognition.
By qualitative and quantitative analysis, we demonstrate the superiority of D IVERSIFY on several challenging scenarios: on difﬁcult tasks, signiﬁcantly diverse datasets, and limited data.
M ETHODOLOGY 2.
P ROBLEM F ORMULATION
We are given a time-series dataset D tr = { ( x i , y i ) } Ni =1 as the training dataset, where N is the number of samples, x i ∈
X ⊂ R p is the p -dimensional instance (sliding window) and y i ∈ Y = { 1 , . . .
, C } is its label.
We use P tr ( x , y ) on X × Y to denote the joint distribution of the training dataset.
Our goal is to learn a generalized model from D tr to predict well on an unseen target dataset,
D te , which is inaccessible in training.
Like D tr , time series in D te also be split into short series.
In our problem, the training and test datasets have the same input and output spaces but different distributions, i.e., X tr = X te , Y te =
Y te , but P tr ( x , y )
(cid:54) =
P te ( x , y ) .
We aim to train a model h from D tr to minimize the risk on D te : min h E ( x ,y ) ∼ P te
[ h ( x ) (cid:54) =
y ] .
Note that due to the non-stationary property, the training dataset may be composed of several unknown latent distributions, instead of one ﬁxed distribution, i.e., P tr ( x , y )
= (cid:80)
Ki =
1 π i P i ( x , y ) , where P i ( x , y ) is the distribution of the i -th sub-domains of the training data and
π
i is its weight.
K is the number of sub-domains that is unknown, and (cid:80) K i =
1 π i = 1 .
2. M OTIVATION
The labeled time series data can be composed of several latent distributions (domains) that are challenging to characterize, even if the dataset is fully labeled.
For instance, data collected by sensors of three persons may belong to two different distributions when considering their similarities.
Moreover, even for data from one single person, different segments of one sequence may follow different distributions.
In a nutshell, in reality, there often exist several sub-domains in one time series dataset.
To ensure good generalization performance on the test dataset, it is important to learn distributioninvariant , or domain-invariant representations from the training dataset by characterizing its latent distributions.
These latent distributions may contain both benign and malignant knowledge that inﬂuences generalization on the target dataset.
In Figure 1 (c), we assume the source domain contains two sub-domains (circle and plus points).
Directly learning from the entire source domain by treating it as one distribution may generate the black margin.
Although green star data points and red star points can be classiﬁed easily, red star data points are misclassiﬁed to the green class when predicting on the out-of-distribution domain (star points) with the learned model.
While other methods fail, our method can characterize the latent distributions, which will be introduced later.
2. D IVERSIFY
In this paper, we propose D IVERSIFY to learn generalized representations for time series classiﬁcation.
The core of D IVERSIFY is to characterize the latent distributions in a time series dataset and then to minimize the distribution divergence between each two.
To characterize the diverse information for better generalization, D IVERSIFY utilizes an iterative process: it ﬁrst obtains the worst-case distribution scenario from a given dataset, then bridges the distribution gaps between each pair of latent distributions.
Why the worst-case scenario?
We argue that the worst-case scenario will maximally preserve the diverse information of each latent distribution, thus beneﬁting generalization.
[Figure] Figure 2 describes the main procedures of D IVERSIFY :
Pre-processing: this step adopts the sliding window to split the entire training dataset into ﬁxed-size windows.
We believe that the data from one window is the smallest domain unit.
Fine-grained feature update
: this step updates the feature extractor using the proposed pseudo domain-class labels as the supervision.
Latent distribution characterization: this step aims to identify the domain label for each instance to acquire the latent distribution information.
It tries to maximize the different distribution gaps to enlarge diversity.
Domain-invariant representation learning: this step utilizes pseudo domain labels from the last step to learn domain-invariant representations and train a generalizable model.
Note that steps 2 ∼ 4 are iterative as shown in Figure 2.
Now we elaborate their details.
Fine-grained Feature Update Before we characterize the latent distributions, we perform ﬁnegrained feature update to obtain useful representations.
As shown in Figure 2 (blue), in order to fully utilize the knowledge contained in the domains and classes, we propose a new concept: pseudo domain-class label , which serves as the supervision for feature extractor.
This will ensure that the feature extractor can learn ﬁne-grained information which beneﬁts the later steps.
At the ﬁrst iteration, there is no domain label d (cid:48) and we simply initialize d (cid:48)
=
for all samples.
We treat per class per domain as a new class, and set the label to s ∈ { 1 , 2 , · · · , S } .
We have S = K × C where K is the pre-deﬁned number of latent distributions that can be tuned in experiments.
We perform the following pseudo domain-class label assignment to get discrete values for supervision: s = d (cid:48)
× C + y.
(Response to reviewer kiW8: this equation is put inline.) Let h (2) f , h (2) b , h (2) c be feature extractor, bottleneck, and classiﬁer, respectively (we use superscript to denote step number), then, the supervised loss is computed using the cross-entropy loss (cid:96) :
[Equation]
Latent Distribution Characterization This step characterizes the latent distributions contained in one dataset.
As shown in Figure 2 (green), we employ an adversarial training strategy to disentangle the domain labels from the class labels.
However, there are no actual domain labels provided, which hinders such disentanglement.
Inspired by (Liang et al., 2020), we adopt a self-supervised pseudolabeling strategy to obtain domain labels.
Firstly, we attain the centroid for each domain with class-invariant features:
[Equation]
where h (3) f , h (3) b , h (3) c are feature extractor, bottleneck, and classiﬁer, respectively.
˜ µ k is the initial centroid of the k th latent sub-domains while δ k is the k th element of the logit soft-max output.
Then, we obtain the pseudo domain labels via the nearest centroid classiﬁer using a distance function D :
[Equation]
Then, we compute the centroids based on the new pseudo labels and obtain the updated pseudo domain labels:
[Equation]
where I ( a ) = 1 when a is true, otherwise 0 .
After we obtain d (cid:48)
[Equation]
we can compute the goal of step 2:
[Equation]
where h (3) adv is the discriminator for step 3 that contains several linear layers and one classiﬁcation layer.
R λ 1 is the gradient reverse layer with hyperparameter λ 1 (Ganin et al., 2016).
After this step ends, we can obtain pseudo domain label d (cid:48) for x .
Domain-invariant Representation Learning After obtaining the latent distributions, we learn domain-invariant representations for generalization.
In fact, this step (Figure 2 purple) is simple: we borrow the idea from DANN (Ganin et al., 2016) and directly use adversarial training to update the classiﬁcation loss L cls and domain classiﬁer loss L dom using GRL:
[Equation]
where (cid:96) is the cross-entropy loss and R λ 2 is the gradient reverse layer with hyperparameter λ 2 (Ganin et al., 2016). More details of GRL and adv.
training are presented in appendix A.1.
We repeat the above three steps until convergence or max epochs.
The ﬁnal model is selected via a validation dataset split from the source domain (Gulrajani & Lopez-Paz, 2021).
As for inference, we predict the labels with the modules from the last step.
2. T HEORETICAL I NSIGHTS
We present some theoretical insights to show that our approach is well motivated in theory.
Proposition 2.1.
(Proposition 3.1 in (Sicilia et al., 2021))
Let X be a space and let H be a class of hypotheses corresponding to this space.
Let Q and the collection { P i } ki =1 be distributions over X and let { ϕ i } ki =1 be a collection of non-negative coefﬁcients with (cid:80) i ϕ i
= 1 .
Let the object O be a set of distributions such that for every S ∈ O the following holds
[Equation]
Then, for any h ∈ H , the following equality holds:
[Equation]
where λ ϕ = (cid:80) i ϕ i
λ
i and each λ
i is the error of an ideal joint hypothesis for Q and P i .
ε P ( h ) is the error for a hypothesis h on a distribution P .
d H ∆ H ( P , Q ) is H -divergence which measure differences in distribution (Ben-David et al., 2010).
The ﬁrst item in Eq. 8, λ ϕ , is often neglected since it is small in reality.
The second item, (cid:80) i ϕ i
ε P
i ( h ) , exists in almost all methods and can be minimized via supervision from class labels with cross-entropy loss in Eq.
Our main purpose is to minimize the last two items in Eq. 8.
Here Q corresponds to the unseen out-of-distribution target domain.
The last term 12 max i,j d H ∆ H
( P i , P j ) is common in DA and DG which measures the maximum differences among source domains.
This corresponds to step 4 in our approach.
Finally, the third item, 12 min S ∈O d H ∆ H ( S , Q ) , explains why we exploit sub-domains in step 3.
Since our goal is to learn a model which can perform well on an unseen target domain, we cannot obtain Q .
To minimize 12 min S ∈O d H ∆ H ( S , Q ) , we can only enlarge the range of O .
We have to max i,j d H ∆ H
( P i , P j ) according to Eq. 7, corresponding to step 3 in our method which tries to segment the time series data into several latent sub-domains by maximizing the segment-wise distribution gap to preserve diversities, i.e., the worst-case distribution scenario.
E XPERIMENTS
We extensively evaluate our approach in three time series classiﬁcation tasks: gesture recognition, speech commands recognition, and sensor-based human activity recognition.
These applications represent diverse situations of time series, thus can thoroughly reﬂect the advantage of our method.
We will introduce their detail later.
Per-segment accuracy is adopted as the evaluation metric. For all experiments, we conduct the training-domain-validation strategy following DomainBed (Gulrajani & Lopez-Paz, 2021).
The training data are split by 8 : 2 for training and validation.
For comparison, we re-implement eight recent strong comparison methods following DomainBed: ERM,
DANN (Ganin et al., 2016), CORAL (Sun & Saenko, 2016), Mixup (Zhang et al., 2018), GroupDRO (Sagawa et al., 2020), RSC (Huang et al., 2020), ANDMask (Parascandolo et al., 2021), and
GILE (Qian et al., 2021).
For fairness, all methods (except GILE) utilize a feature net with two blocks and each block has one convolution layer, one pooling layer, and one batch normalization layer.
Moreover, the bottleneck and classiﬁcation layer adopts a FC layer, respectively.
See appendix C for more details. Note that most methods require the domain labels known in training while our does not, which is more challenging and practical.
We tune all methods to report the averaged best performance of three trials.
3. G ESTURE RECOGNITION
Electromyography (EMG) is a common type of time-series data that is based on bioelectric signals.
We use EMG for gestures Data Set (Lobov et al., 2018) that contains raw EMG data recorded by MYO Thalmic bracelet.
The bracelet is equipped with eight sensors equally spaced around the forearm that simultaneously acquire myographic signals. Data of 36 subjects are collected while they performed series of static hand gestures and the number of instances is 40 , 000 − 50 , 000 recordings in each column. It contains 7 classes and we select 6 common classes for our experiments. We randomly divide 36 subjects into four domains without overlapping and each domain contains data of 9 persons.
Experiments are done in three random trials, thus mitigating inﬂuence of unfair splits.
[Table]
Since EMG data comes from bioelectric signals, it is affected by many factors.
EMG data are scene and devicedependent, which means the same person may generate different data when performing the same activity with the same device at a different time or with the different devices at the same time.
Therefore, the benchmark of EMG is challenging.
Table 1 shows that our method achieves the best average performance and is 4 .
3 % better than the second-best method.
Table 7 in appendix shows the advantage of our method in different model sizes.
3. S PEECH C OMMANDS
[Figure]
Then, we adopt a regular speech recognition task, the Speech Commands dataset (Warden, 2018).
The dataset consists of one-second audio recordings of both background noise and spoken words such as ’left’, ’right’, etc.
It is collected from more than 2,000 persons, thus is more complicated.
Following (Kidger et al., 2020), we use 34,975 time series corresponding to ten spoken words to produce a balanced classiﬁcation problem.
Since this dataset is collected from multiple persons, thus the training and test distributions are different, which is also an OOD problem with one training domain.
There are many subjects and each subject only records few audios.
Thus, an audio recording is short so we do not split each sample.
Figure 3 shows the recognition results on two backbones. Compared with GroupDRO, D IVERSIFY has over 1 % improvement based on a basic CNN architecture and over 0 .
6 % improvement based on MatchBoxNet3-1-64 (Majumdar & Ginsburg, 2020)
(its baseline is already strong). The results demonstrate the superiority of our method on a regular time-series benchmark composed of massive distributions.
3. S ENSOR - BASED H UMAN A CTIVITY R ECOGNITION
Finally, we employ several popular sensor-based human activity recognition datasets for more detailed evaluation.
UCI daily and sports dataset ( DSADS ) (Barshan & Y ¨ uksek, 2014) consists of 19 activities collected from 8 subjects wearing body-worn sensors on 5 body parts.
USC-SIPI human activity dataset ( USC-HAD ) (Zhang & Sawchuk, 2012) is composed of 14 subjects (7 male,female, aged from 21 to 49) executing 12 activities with a sensor tied on the front right hip.
UCIHAR (Anguita et al., 2012) is collected by 30 subjects performing 6 daily living activities with a waist-mounted smartphone.
PAMAP physical activity monitoring dataset (Reiss & Stricker, 2012) contains data of 18 activities, performed by 9 subjects wearing 3 sensors.
We construct four different settings representing different degrees of generalization: (1) Crossperson generalization : This setting utilizes DSADS, USC-HAD, PAMAP 1 datasets to construct three benchmarks.
Within each dataset, we randomly split the data into four groups 2 and then use three groups as training data to learn a generalized model for the last group.
(2) Cross-position generalization : this setting uses DSADS dataset and data from each position denotes a different domain.
Each sample contains three sensors with nine dimensions.
We treat one position as the test domain while the others are for training.
(3) Cross-dataset generalization : this setting uses all four datasets, and each dataset corresponds to a different domain.
Six common classes are selected.
Two sensors from each dataset that belong to the same position are selected and data is down-sampled to ensure the same dimension of data.
(4) One-Person-To-Another .
This setting adopts DSADS,
HAR
[Table] [Table] USC-HAD, and PAMAP datasets. In each dataset, we randomly select four pairs of persons where one is the training and the other is the test.
For simplicity, we use 0 , 1 , · · · to denote different domains.
All experiments are done in three random trials, mitigating the inﬂuence of unfair splits.
Table 2 and Table 3 show the results on four settings for HAR 3 .
For all settings for HAR, our method achieves the best average performances.
For Cross-person, Cross-position, Cross-dataset, and One-Person-To-Another settings, our method signiﬁcantly outperforms the second-best baseline by 2 . 4 % , 1 . 4 % , 9 .
9 % , and 5 . 8 % respectively.
All results demonstrate the superiority of D IVERSIFY in different generalization settings.
We observe more insightful conclusions.
(1) When the task is difﬁcult: In the Cross-Person setting, USC-HAD may be the most difﬁcult task.
Although it includes more samples, it contains 14 subjects with only two sensors on one position, which may bring more difﬁculty in this situation.
The experimental results prove the above argument that all methods perform terribly on this benchmark while ours has the largest improvement.
(2) When datasets are signiﬁcantly more diverse: Compared to Cross-Person and Cross-Position settings, Cross-Dataset may be more difﬁcult since all datasets are totally different that samples are inﬂuenced by subjects, devices, sensor positions, and some other factors.
In this setting, our method is substantially better than others.
(3) Limited data: Compared with Cross-Person setting, One-Person-To-Another is more difﬁcult since it has fewer data samples.
Therefore, in this situation, enhancing diversity can bring a remarkable improvement and our method can boost the performance.
3. A NALYSIS
Ablation Study We present ablation study to answer the following three questions: (1) Why obtaining pseudo domain labels with class-invariant features in step 3?
If we obtain pseudo domain labels with common features, domain labels may have correlations with class labels, which may introduce contradictions when learning domain-invariant representations and lead common performance.
This is certiﬁed by the results in Figure 4(a).
(2) Why using ﬁne-grained domain-class labels in step 2?
If we utilize pseudo domain labels to update the feature net, it may make the representations seriously biased towards domain-related features and thereby leads to terrible performance on classiﬁcation, which is proved in Figure 4(b).
If we only utilize class labels to update the feature net, it may make representations biased to class-related features, make D IVERSIFY unable to obtain true latent sub-domains, and thereby has unremarkable performance, which is proved in Figure 4(c). [Figure] [Figure]
Therefore, we should utilize ﬁne-grained domain-class labels to obtain representations with both domain and class information.
(3) The more latent sub-domains, the better?
More latent sub-domains may not bring better results as shown in Figure 4(d).
A dataset may only contain a few sub-domains and introducing more may have a conﬂict with the intrinsic data property.
Plus, more latent subdomains also brings difﬁculty to obtain pseudo domain labels and learn domain-invariant features, which may have negative effects. Visualization Study We present some visualizations to show the rationales of our method and more results are in appendix.
Data points with different initial domain labels are mixed together in
Figure 5(a) while our D IVERSIFY can characterize different latent distributions to separate them well in Figure 5(b). It illustrates that our method can ﬁnd better latent sub-domains. From Figure 5(d) and Figure 5(c), we can see that our method can learn better domain-invariant representations compared to the latest method ANDMask.
Therefore, D IVERSIFY can ﬁnd better latent sub-domains and then ﬁnd domain-invariant representation from split domains, which can ﬁnally enhance generalization.
Existence of latent distributions Latent sub-domains widely exist in time series, even in data collected from one person when performing the same activity.
In Figure 6(a), for a subject in USC-HAD dataset, there is more than one latent distribution for his walking activities.
Figure 6(b) demonstrates similar phenomena exist in EMG dataset where our algorithm found three latent distributions.
Quantitative analysis for worst-case distributions We present quantitative analysis by computing the H -divergence (Ben-David et al., 2010) to show the effectiveness of our ‘worst-case distribution’.
As shown in Figure 6(c) and Figure 6(d), compared with initial domain splits, latent sub-domains generated by our method have larger H -divergence among each other.
According to
Prop. 2.1, larger H -divergence among domains can bring better generalization.
R ELATED W ORK
Time series classiﬁcation is a challenging problem for years.
Existing researches mainly focus on the modeling of temporal relations using either RNN-based methods or the recently-proposed
Transformer architecture.
To our best knowledge, there is only one recent work (Du et al., 2021) that studied time series from the distribution level.
However, their method is not end-to-end trainable since they developed a two-stage method that is non-differential.
[Figure] Domain / OOD generalization (Wang et al., 2021a) mainly consists of data augmentation (Zhang et al., 2018;
Yue et al., 2019), domain-invariant feature learning (Muandet et al., 2013; Ganin et al., 2016), and meta-learning (Li et al., 2018) techniques.
The work of (Matsuura & Harada, 2020) also studied DG without knowing the domain labels by clustering with the style features for image data.
However, that work is not applied to time series and it is not end-to-end trainable due to clustering.
Disentanglement (Peng et al., 2019; Wang et al., 2021b) tries to disentangle the domain and label information, but they assume access to domain information.
Single domain generalization is similar to our problem setting that also involves one training domain (Fan et al., 2021;
Li et al., 2021; Qiao et al., 2020; Wang et al., 2021c).
Most work adopted generative models and data augmentation strategies.
However, they treated the single domain as one distribution and did not explore the latent sub-distributions.
Speciﬁcally, Wang et al.
(2021c) proposed a ‘learning-to-diversify’ algorithm that has a similar name to ours, but focused on mutual information maximization.
Mixture models (Rasmussen et al., 1999) are models representing the presence of subpopulations within an overall population, e.g., Gaussian mixture models.
Our approach belongs to this category in general with focus on OOD.
Moreover, we do not rely on generative models as most mixture models.
Subpopulation shift is a new setting introduced in (Koh et al., 2021), which refers to the situation that the training and test domains overlap, but their relative proportions differ.
Our problem does not belong to this setting since we assume that the training and test distributions do not overlap.
Distributionally robust optimization (Delage & Ye, 2010) shares a similar paradigm with our work, whose paradigm is also to seek a distribution that has the worst performance within a range of the raw distribution.
However, we study the internal distribution shift instead of seeking a global distribution close to the original one.
GroupDRO (Sagawa et al., 2020) studied the DRO at a group level, which is signiﬁcantly different from our idea.
Pseudo-labeling is a common strategy in semi-supervised and unsupervised learning. SHOT (Liang et al., 2020) combined the pseudo-labeling strategy and IM (Information Maximization) to learn target adaptation features for source free domain adaptation. However, we mainly focus on domain generalization and thereby have different goals and updating styles.
DANN (Ganin et al., 2016) has become the fundamental technology in domain adaptation and domain generalization, using adversarial training to obtain domain-invariant representations. In our method, we utilize it to obtain class-invariant and domain-invariant representations based on all features.  C ONCLUSIONS AND F UTURE W ORK We proposed D IVERSIFY to learn generalized representations for time series classiﬁcation.
D IVERSIFY employs an adversarial game that maximizes the worst-case distribution scenario while minimizing their distribution divergence.
We demonstrated its effectiveness in different types of applications via qualitative and quantitative analysis.
We are surprised that not only a mixed dataset, but one dataset from a single person can also contain several latent distributions.
Characterizing such latent distributions will greatly improve the generalization performance on unseen datasets.
In the future, we plan to extend our algorithm by seeking new architectures and developing techniques to estimate the value of domain numbers K .
Moreover, we plan to apply our algorithm to other time series-related applications such as forecasting and video classiﬁcation. A M ETHOD D ETAILS A. D OMAIN - INVARIANT R EPRESENTATION L EARNING
Domain-invariant representation learning utilizes adversarial training which contains a feature network, a domain discriminator, and a classiﬁcation network.
The domain discriminator tries its best to discriminate domain labels of data while the feature network tries its best to generate features to confuse the domain discriminator, which thereby obtains domain-invariant representation.
Therefore, it is an adversarial process, and in our setting, it can be expressed as following,
[Equation] To optimize Eq. 9, we need an iterative process to optimize h (4) b , h (4) c and h (4) adv iteratively, which is cumbersome. It is better to optimize h (4) b , h (4) c and h (4) adv at the same time.
It is obvious that the key is to solve the problems caused by the negative sign in Eq.
Therefore, a special gradient reversal layer (GRL), a popular implementation of the adversarial training in training several domains as suggested by (Ganin et al, 2016), came.
GRL acts as an identity transformation during the forward propagation while it takes the gradient from the subsequent level and changes its sign before passing it to the preceding layer during the backpropagation.
During the forward propagation, the GRL can be ignored.
During the backpropagation, the GRL makes the sign of gradient on h (4) b reverse, which solves the problems caused by the negative sign in Eq. 9.
B D ATASET B. D ATASETS I NFORMATION
Table 4 shows the statistical information on each dataset.
[Table]
B. D ATA P REPROCESSING
We will introduce how we preprocess data and the ﬁnal dimension of data for experiments here.
We mainly utilize the sliding window technique, a common technique in time-series classiﬁcation, to split data.
As its name suggests, this technique involves taking a subset of data from a given array or sequence.
Two main parameters of the sliding window technique are the window size, describing a subset length, and the step size, describing moving forward distance each time.
For EMG, we set the window size 200 and the step size 100, which means there exist 50% overlaps between two adjacent samples.
We normalize each sample with ˜ x
= x − min X max X − min X .
X contains all x .
The ﬁnal dimension is 8 × 1 × 200 .
For Speech Commands, we follow (Kidger et al., 2020).
Now we give details on all datasets in Cross-person setting.
For DSADS, we directly utilize data split by the providers.
The ﬁnal dimension shape is 45 × 1 × 125 .
45 = 5 × 3 × 3 wheremeans ﬁve positions, the ﬁrst 3 means three sensors, and the second 3 means each sensor has three axes.
For USC-HAD, the window size is 200 and the step size is 100.
The ﬁnal dimension shape is
6 × 1 × 200 .
For PAMAP, the window size is 200 and the step size is 100.
The ﬁnal dimension shape is 27 × 1 × 200 .
For UCI-HAR, we directly utilize data split by the providers.
The ﬁnal dimension shape is 6 × 1 × 128 .
For Cross-position, we directly utilize samples obtained from DSADS in Cross-person setting.
Since each position corresponds to one domain, a sample is split into ﬁve samples in the ﬁrst dimension.
And the ﬁnal dimension shape is 9 × 1 × 125 .
For Cross-dataset, we directly utilize samples obtained in Cross-person setting.
To make all datasets share the same label space and input space, we select six common classes, including WALKING,
WALKING UPSTAIRS, WALKING DOWNSTAIRS, SITTING, STANDING, LAYING.
In addition, we down-sample data and select two sensors from each dataset that belong to the same position.
The ﬁnal dimension shape is 6 × 1 × 50 .
For One-Person-To-Another, we randomly select four pairs of persons from DSADS, USC-HAD, and PAMAP respectively.
Four tasks are 1 → 0 , 3 → 2 , 5 → 4 , and 7 → 6 .
Each number corresponds to one subject.
And the ﬁnal dimension shape is 45 × 1 × 125 , 6 × 1 × 200 , and 27 × 1 × 200 for DSADS, USC-HAD, and PAMAP respectively.
As we can see, samples in EMG and HAR are all have more than one channels (the ﬁrst dimension shape), which means they are all multivariate.
B. I NITIAL D OMAIN S PLITS
We introduce how we split data here.
Since Speech Commands is a regular task, we just randomly split the entire dataset to a training dataset, a validation dataset, and a testing dataset.
We mainly focus on EMG and HAR, and we construct domains for OOD tasks.
We denote subjects of a dataset with 0 − s n , where s n is the number of subjects in the dataset.
For example, there are 36 subjects in EMG and we utilize 0 , 1 , 2 , · · · , 35 to denote data of them respectively.
Table 5 shows the initial domain splits of EMG and all datasets for HAR in Cross-person setting.
We just want make each domain has the similar number of samples in one dataset.
As noted in the main paper, we also utilize 0, 1, 2, and 3 to represent different domains but they have different meanings with subjects.
When conducting experiments, we take one domain as the testing data and the others as the training data.
Our method is not inﬂuenced by the splits of the training data since we do not need the domain labels.
[Table]
C N ETWORK A RCHITECTURE AND H YPERPARAMETERS
For the architecture, the model contains two blocks, and each has one convolution layer, one pooling layer, and one batch normalization layer.
A single-fully-connected layer is used as the bottleneck layer while another fully-connected layer serves as the classiﬁer.
All methods are implemented with
PyTorch (Paszke et al., 2019).
The maximum training epoch is set to 150.
The Adam optimizer with weight decay 5 × 10 − 4 is used.
The learning rate for GILE is 10 − 4 .
The learning rate for the rest methods is 10 − 2 or 10 − 3 . (For new added Speech Commands with MatchBoxNet3-1-64, we also try the learning rate, 10 − 4 .)
We tune hyperparameters for each method.
For the pooling layer, we utilize MaxPool2d in PyTorch.
The kernel size is (1 , 2) ad the stride is 2.
For the convolution layer, we utilize Conv2d in PyTorch.
Different tasks have different kernel sizes and Table 6 shows the kernel sizes.
[Table]
D E VALUATION M ETRICS
We utilize average accuracy on the testing dataset as our evaluation metrics for all benchmarks.
Average accuracy is the most common metric for DG and it can be computed as the following,
[Equation] I y ( y ∗ ) is an indicator function. If y = y ∗ , it equals 1, otherwise it equals 0. # | · | represent the number of the set.
h is the model to learn.
Please note that X te has a different distribution from X tr for EMG and HAR.
And x has been preprocessed and each sample is a segment.
[Figure] [Figure]
E M ORE E XPERIMENTAL R ESULTS E. V ISUALIZATION S TUDY
We show more visualization study in this part.
As shown in Figure 7(a) and Figure 7(b), both ERM and RSC also cannot obtain ﬁne domain-invariant representations and our method still achieves the best domain-invariant representations. As shown in Figure 8, compare with initial domain splits, latent sub-domains generated by our method are better separated.
E. P ARAMETER S ENSITIVITY
There are mainly four hyperparameters in our method: K which is the number of latent sub-domains, λ 1 for the adversarial part in step 3, λ 2 for the adversarial part in step 4, and local epochs and total [Figure] rounds.
For fairness, the product of local epochs and total rounds is the same value.
We evaluate the parameter sensitivity of our method in Figure 9 where we change one parameter and ﬁx the other to record the results.
From these results, we can see that our method achieves better performance in a wide range, demonstrating that our method is robust. E. H - DIVERGENCE
[Figure]
Figure 10 shows H -divergence among domains with initial splits and our splits on EMG, which demonstrates our splits have larger H -divergence and thereby can bring better generalization. E. T HE I NFLUENCE OF A RCHITECTURES
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
To ensure that our method can work with different sizes of models
[Table]
we add some more experiments with more complex or simpler architectures.
As shown in Table
[Table]
where small, medium, large indicates the different model sizes (our paper uses the medium)
[Table]
we see a clear picture that model sizes inﬂuence the results and our method also achieves the best performance. Small corresponds to the model with one convolutional layer, Medium corresponds to the model with two convolutional layers, and Large corresponds to the model with four convolutional layers. For most methods, more complex models bring better results. For all architectures, our method achieves the best performance.
E. R ESULTS ON WESAD
We show results on a larger dataset, WESAD, here.
Wesad (Schmidt et al., 2018) is a publicly available dataset for wearable stress and affect detection, which contains physiological and motion data of 15 subjects.
Wesad contains 63,000,000 instances.
And we utilize techniques mentioned above to preprocess it.
We utilize sensor modalities of chestworn device including electrocardiogram, electrodermal activity, electromyogram, respiration, body temperature, and three axis acceleration.
We split 15 subjects into four domains, [(0-3), (4-7), (811), (12-14)].
The results are shown in Table 8, and we can see that our method achieves the best performance compared to other state-of-the-art methods with an improvement over 8 % .
[Table]