D OMAIN G ENERALIZATION VIA I NVARIANT R EPRESENTATION UNDER D OMAIN -C LASS D EPENDENCY
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning domain-invariant representation is a dominant approach for domain generalization , where we need to build a classiﬁer that is robust toward domain shifts induced by change of users, acoustic or lighting conditions, etc.
However, prior domain-invariance-based methods overlooked the underlying dependency of classes (target variable) on source domains during optimization, which causes the trade-off between classiﬁcation accuracy and domain-invariance, and often interferes with the domain generalization performance.
This study ﬁrst provides the notion of domain generalization under domain-class dependency and elaborates on the importance of considering the dependency by expanding the analysis of
Xie et al.
We then propose a method, invariant feature learning under optimal classiﬁer constrains (IFLOC) , which explicitly considers the dependency and maintains accuracy while improving domain-invariance.
Speciﬁcally, the proposed method regularizes the representation so that it has as much domain information as the class labels, unlike prior methods that remove all domain information.
Empirical validations show the superior performance of IFLOC to baseline methods, supporting the importance of the domain-class dependency in domain generalization and the efﬁcacy of the proposed method for overcoming the issue.
I NTRODUCTION
In supervised learning problems we typically assume that samples are obtained from the same distribution in training and testing; however, such an assumption does not hold in many practical situations, depressing the classiﬁcation accuracy for the test data (Torralba & Efros (2011)).
One typical situation is domain generalization (e.g., Blanchard et al.
(2011)): we have labeled data from several source domains and collectively exploit them so that the trained system generalizes to other, unseen but somewhat similar, target domains.
Such challenges arise in many applications, e.g., hand-writing recognition (Shankar et al.
robust speech recognition (Sriram et al.
and sensor data interpretation (Erfani et al.
To address domain generalization, many methods take advantage of invariant feature learning (Muandet et al.
; Erfani et al.
; Ghifary et al.
; Xie et al. (2017)).
Such methods assume that learning the representation ( h ) that is invariant to domains ( d ) from input data ( x ) prevents h to overﬁt to source domains and leads to higher classiﬁcation accuracy for unseen domains.
To obtain such h , we used various methods to measure the invariance of h to d and imposed some regularization on the measurement.
For example, domain adversarial networks (DAN) (Ganin et al.
; Xie et al.
(2017)) measure the invariance using a domain classiﬁer (also called a discriminator) parameterized by deep neural networks and impose regularization by deceiving it.
Most prior works, however, overlooked the underlying dependency of classes on source domains, which we refer to as domain-class dependency .
More speciﬁcally, we deﬁne domain-class dependency as the situation where domain and class labels are statistically dependent due to some common latent factor ( z ) of y and d (Figure 1-right).
Under the domain-class dependency, merely forcing the optimal domain-invariance harms the classiﬁcation accuracy, as shown in Figure 1-(c).
Intuitively speaking, since y contains information about d under domain-class dependency, h must keep at least as much domain information as y to achieve the optimal classiﬁcation accuracy; however, invariant feature learning attempts to remove all domain information from h , which causes the trade-off. It might be similar to the situation where p ( y j x ) and p ( x ) change across domains due to the causal ) When domain and class are independent, domain structure y ! x (Zhang et al.
; Gong et al.
(2016) in domain adaptation and Li et al. (2018c) in domain generalization), which we call conditional probability shift . However, the shift does not cause the trade-off as long as y and d are independent (Figure 1-left), so it is necessary to focus on the relationship between y and d . Unfortunately, domain-class dependency is common in real-world datasets as shown in Zhang et al.
The dependency can be caused by both the characteristics of data and errors in collecting data.
For example, the WISDM Activity Prediction dataset (Kwapisz et al.
(2011)), where classes and domains correspond to activities and users, exhibits the dependency because (1) some activities (jogging and climbing stairs) are strenuous ( data characteristics ) and (2) other activities (sitting and standing) and some users were added only after the study began ( data-collection errors ).
In this paper, we address domain generalization under domain-class dependency.
We ﬁrst expand the analysis about DAN by Xie et al.
(2017), show that domain-class dependency causes the tradeoff problem, and then derive a way to evade the trade-off.
Speciﬁcally, we investigate the condition where the domain-invariance is maximized under the constraint that it does not interfere with classiﬁcation accuracy (Figure 1 (b)), because the primary purpose of domain generalization is classiﬁcation rather than domain-invariance itself.
We then propose a novel method invariant feature learning under optimal classiﬁer constraint (IFLOC) , modifying DAN’s regularization term to make the learned representation have as much domain information as the class labels, i.e., H ( d j h )
=
H ( d j y ) holds (here H denotes entropy).
Like DAN, IFLOC has an encoder, classiﬁer, and domain discriminator, and also takes over the good properties of DAN: it does not depend on pre-deﬁned metrics (e.g., maximum mean discrepancy (Tzeng et al.
and it can be trained in an end-to-end manner.
Empirical validations show the superior performance of IFLOC to baseline methods, supporting the importance of considering domain-class dependency in domain generalization tasks and the efﬁcacy of the proposed approach for overcoming the issue.
The main contributions of this paper can be summarized as follows.
Firstly, we elaborate on the trade-off problem under domain-class dependency, both theoretically and experimentally, for the ﬁrst time in domain generalization context.
Secondly, to address the issue we provide theoretical analysis, which shows to what extent latent representations can become invariant to domains without interfering with classiﬁcation accuracy.
Finally, based on the analysis we propose a novel method IFLOC, and validated its efﬁcacy by the experiments on both synthetic and real world datasets.
R ELATED W ORKS
Invariant feature learning is a general-purpose method applicable to domain generalization as well as to domain adaptation (e.g., Tzeng et al.
; Ganin et al.
style transfer (e.g.,
Lample et al.
; Chou et al.
(2018)), and fairness-aware classiﬁcation
(e.g., Zemel et al.
Louizos et al.
; Madras et al.
However, it is likely that adjusting it to each speciﬁc task can improve performance.
For example, in the fairness-aware classiﬁcation task Madras et al.
(2018) proposed to optimize the fairness criterion directly instead of applying invariance to sensitive variables.
By analogy, we adapted invariant feature learning for domain generalization so as to address the domain-class dependency problem.
Domain generalization has been attracting considerable attention in recent years (Blanchard et al.
; Muandet et al. ; Shankar et al. Note that it is different from domain adaptation in that we cannot obtain input and label data from target domain(s).
Although the efﬁcacy of domain-invariance-based methods had been known, Li et al.
(2017) showed that non end-to-end methods such as DICA (Muandet et al.
and MTAE (Ghifary et al.
(2015)) do not tend to outperform even vanilla CNN.
Thus, end-to-end methods are desirable and can be divided into two categories: adversarial-learning-based methods such as DAN (Ganin et al.
; Xie et al. (2017)) and pre-deﬁned-metric-based methods (e.g., Ghifary et al.
; Li et al.
(2018b)).
In particular, IFLOC closely relates to DAN.
Although DAN was originally invented for domain adaptation, Xie et al.
(2017) showed its efﬁcacy in domain generalization.
Also, Xie et al.
(2017) provided the intuitive explanation of the trade-off between classiﬁcation accuracy and domain invariance.
However, they did not provide any way to deal with the problem because its focus is invariant feature learning itself.
Louppe et al.
(2017) provided the similar analysis with Xie et al.
(2017), but differs in that they focused on the relation between nuisance parameters (domains) and output distribution of a domain classiﬁer.
IFLOC also relates to domain confusion loss (Tzeng et al.
in that their encoders attempt to directly minimize Kullback-Leibler divergence (KLD) between output distribution of the discriminators and some domain distribution ( p ( d j y ) in IFLOC and uniform distribution in domain confusion loss), rather than deceive the discriminator as DAN.
There are several studies that address domain generalization without utilizing invariant feature learning.
For example, Motiian et al.
; Li et al.
(2018c) proposed to make use of semantic alignment, which attempts to make latent representation given class label ( p ( h j y ) )
identical within source domains.
This approach was originally proposed in Gong et al. (2016) in domain adaptation context, but its efﬁcacy for domain-class dependency is not obvious because it focuses on conditional probability shift.
CrossGrad (Shankar et al.
(2018)) is one of the recent state-of-the-art domain generalization methods, which utilizes data augmentation with adversarial examples.
However, since the method relies on the assumption that y and d are independent, it might not be directly applicable to our setting.
MLDG (Li et al.
(2018a)), also one of the state-of-the-art methods, utilizes metalearning.
Since it makes no assumption about the relation between y and d , it could be combined with our proposed method, though we have not experimentally conﬁrmed it.
There are several kinds of distributional shifts other than domain-class dependency, such as conditional probability shift.
Although the distinction between that shift and domain-class dependency is important, it has been received less attention.
For example, Li et al.
(2018c) claimed that conditional probability shift might harm the performance of domain-invariance-based methods, but our analysis in Sec.4.1.1 suggests that the root cause of the performance degradation is not it but domain-class dependency. They also proposed to correct the shift of p ( y ) across source domains by aligning sampling frequency of each class across domains. However, this approach is not applicable to when some classes are rarely or never appear in some domains (e.g., as in WISDM dataset). In domain adaptation, Zhang et al. (2013); Gong et al.
(2016) address the situation where p ( y ) changes across source and target domains by estimating p ( y ) change using unlabeled target data.
However, this approach is not applicable (or necessary) to domain generalization because our problem setting is different from theirs in that we are agnostic on target domain and aim to care about p ( y ) change within source domains instead.
P RELIMINARIES 3. P ROBLEM S TATEMENT OF D OMAIN G ENERALIZATION
Denote X ; Y , and D as the input feature, label, and domain spaces, respectively.
With random variables x 2 X ; y 2 Y , and d 2 D ， we can deﬁne the probability distribution for each domain d as p ( x; y j d ) .
Here, we assume that y and d are discrete variables for simplicity.
In domain generalization, we are given a training dataset consisting of D s = f x s i ; y s i g n s i =1 for all s 2 f 1 ; 2 ; :::; m g .
Here, each D s corresponds to samples drawn from the source domain p ( x; y j d = s ) .
Using the training dataset, we train a classiﬁer f : X !
Y , and use the classiﬁer to predict labels of samples drawn from the unknown target domain p ( x; y j d = t ) .
3. D OMAIN A DVERSARIAL N ETWORKS FOR D OMAIN G ENERALIZATION
In this section, we give a brief overview of DAN (Ganin et al.
given that our proposed method is an extension of it.
DAN trains a domain discriminator that attempts to predict domains from latent representations encoded by an encoder, while simultaneously trains the encoder to remove domain information by deceiving the discriminator.
This procedure ensures that there is no or little domain information in the representations, so a label classiﬁer attached to the encoder can make robust predictions regarding unseen target domains.
Formally, we denote f E ( x ) ; q M ( y j h ) ; and q D ( d j h ) ( E; M , and D are the parameters) as deterministic encoder, probabilistic model of label classiﬁer, and that of domain discriminator, respectively.
Then, the objective function of DAN is described as follows:
[Equation]
[Equation] Here, the second term in Eq.1 simply maximizes the log likelihood of q M as well as in standard classiﬁcation problems.
On the other hand, the ﬁrst term corresponds to a minimax game between the encoder and discriminator, where the decoder q D ( d j h ) tries to predict d from h and the encoder f E ( x ) tries to fool q D ( d j h ) .
As Xie et al.
(2017) originally showed, the minmax game ensures that the learned representation has no or little domain information, i.e., the representation becomes domain-invariant.
Such invariance makes a prediction from h to y independent from d , and therefore hopefully helps to build a classiﬁer that correctly handle samples drawn from unknown domains.
Below is a brief explanation.
Since h is a deterministic mapping of x , the joint probability distribution of h; d and y can be deﬁned as follows:
[Equation]
Here, we use the notation of ~ p E for the true probability distribution that depends on the encoder’s parameter E .
Using Eq.2, Eq.1 can be replaced as follows:
[Equation]
Assuming E is ﬁxed, the solutions M (cid:3) and D (cid:3) to Eq.3 obviously satisfy q M (cid:3)
( y j h ) =
~ p E ( y j h ) and q D (cid:3) ( d j h ) =
~ p E ( d j h ) .
Then, substituting q M (cid:3) and q D (cid:3) into Eq.3, we can obtain the following optimization problem depending only on E :
[Equation]
Solving Eq.4, we can obtain the solutions M (cid:3) , D (cid:3) , and E (cid:3) , which are in Nash equilibrium.
Here,
H ~ p E ( d j h ) means conditional entropy with joint probability distribution ~ p E ( d; h ) .
Thus, minimizing the second term in Eq.4 intuitively means learning (the mapping function f E to)
the latent representation h which contains as much information about y as possible.
On the other hand, the ﬁrst term can be regarded as a regularizer that attempts to learn h which is invariant to d .
O UR APPROACH 4. A NALYSIS OF D OMAIN -C LASS D EPENDENCY
We address domain generalization under domain-class dependency, i.e., the situation where p ( y j d ) ̸ = p ( y ) holds.
Although the issue had been overlooked, it is common in real-world datasets given that they can have the dependency in nature, e.g., nocturnal annimals (class) do not tend to appear in daylight (domain), and the dependency in such datasets is often not corrected unlike in standard benchmark datasets.
To address the problem, we expand the analysis of Xie et al.
(2017) to theoretically show that domain-class dependency causes the trade-off between accuracy and invariance, and to consider to what extent the latent representation should become invariant.
4.1. T RADE -O FF C AUSED BY D OMAIN -C LASS D EPENDENCY
We ﬁrst show that the performance of DAN explained in the previous section suffers from the existence of domain-class dependency.
The following analysis also suggests that all of the methods that utilize domain-invariant representation suffer from the dependency.
Concretely, we show that the domain-class dependency causes the trade-off between classiﬁcation accuracy and domain invariance: when d and y are not statistically independent, any E cannot optimize the ﬁrst and second term in Eq.4 at the same time.
In this analysis, for simplicity, we assume that we can obtain any ~ p E ( y j h ) ; ~ p E ( d j h ) , i.e., the models have enough capacity and there are no optimization difﬁculties.
To begin with, we consider only the ﬁrst term in Eq.4 and address the optimization problem: [Equation]
[Equation]
Using the property of entropy, H ~ p E ( d j h ) is bounded as follows:
[Equation]
Here, H ~ p E ( d j h )
=
H ( d ) holds only if h and d are independent.
Thus, Eq.5 has the solution E 1 (cid:3) , which satisﬁes the following condition:
[Equation]
Eq.7 suggests that the regularizer in DAN is intended to remove all information about domains from latent variables, thereby making domains and latent variables independent.
Next, we analogically consider only the second term in Eq.4, thereby addressing the following optimization problem:
[Equation]
Since conditional entropy H ( a j b ) has a minimum value when b contains all information about a ,
Eq.8 has the solution E 2 (cid:3) , which satisﬁes the following equation:
[Equation]
Using Eq.9 and the property of entropy: H ( a j b; c ) (cid:20) H ( a j b ) , we can obtain the following condition:
[Equation]
Eq.10 implies that h has at least as much information about d as y does.
Now, we assume that y and d are not independent, i.e., domain-class dependency exists, and obtain the following condition:
[Equation]
Considering Eq.7 and Eq.11, E 1 (cid:3) ̸
= E 2 (cid:3) holds.
This means that when y and d are not independent, there is no solution E that optimizes Eq.5 and Eq.8 at the same time, i.e., there is a trade-off between classiﬁcation accuracy and domain invariance. It is worth noting that although Li et al.
(2018c) claimed that conditional probability shift (the causal structure y !
x ) could harm the domain generalization performance of invariance-based methods, this analysis suggests that it does not harm DAN as long as domain and class are independent. It can be conﬁrmed by considering Eq.7 and Eq.10; even when the shift occurs, i.e., H ( y j x; d ) < H ( y j x ) holds and then H ~ p E ( y j h; d ) (cid:20) H ~ p E ( y j h ) holds, it does not conﬂict with H ~ p E (cid:3) ( d j h )
=
H ( d j y ) =
H ( d ) as long as H ( d j y ) =
H ( d ) holds.
In other words, we only need to infer latent variable h that satisﬁes the causal structure y !
h !
x to avoid the trade-off.
Although Gong et al.
(2016) showed the similar result in domain adaptation context, it has been overlooked in domain generalization.
4.1. O PTIMAL D OMAIN -I NVARIANCE UNDER D OMAIN -C LASS D EPENDENCY
If we cannot avoid the trade-off, the next question is how to deal with it, i.e., to what extent the representation should become domain-invariant for domain generalization tasks.
We propose to maximize domain-invariance within a range that does not interfere with classiﬁcation accuracy, rather than merely enforcing domain-invariance without any constraint.
The reason for the constraint is that the primary purpose of domain generalization is classiﬁcation for unseen domains rather than domain-invariance itself, and the improvement of the invariance could harm the classiﬁcation performance for them.
For example, in WISDM, if we know the target activity (class) was performed by not an old but yound man (domain), we can predict it was jogging with higher probability, so we should avoid removing such domain information that is useful in the classiﬁcation task.
As another example, if the target domain has the similar characteristics as a certain source domain (or as an extreme case, p ( x; y j d = s ) =
p ( x; y j d = t ) holds), giving priority to domain-invariance obviously interferes with the domain generalization performance.
Given that Eq.10 is the necessary condition where we can build an optimal classiﬁer, we can write the optimization problem of maximizing domain-invariance within a range that does not interfere with classiﬁcation accuracy as follows:
[Equation]
12) subject to [Equation] [Equation] Continuing, we can obtain the solution E (cid:3) , which obviously satisﬁes H ~ p E (cid:3) ( d j h ) =
H ( d j y ) .
More speciﬁcally, when we want to maximize domain-invariance (Eq.12) within the range that does not interfere with accuracy (Eq.13), the solution satisﬁes H ~ p E (cid:3) ( d j h ) =
H ( d j y ) .
So without interfering with classiﬁcation accuracy we can remove domain information from h to the extent that
H ~ p E ( d j h )
=
H ( d j y ) holds , i.e., h has as much information about d as y does.
4. P ROPOSED M ETHOD
Based on the above analysis, the remaining challenge is how to impose such regularization that makes H ~ p E ( d j h )
=
H ( d j y ) hold.
Although DAN might be able to achive that condition by carefully tuning the regularizer ( (cid:13) in Eq.1), such tuning is time-consuming and impracticable as suggested in our experiments.
Alternatively, we propose a novel method called IFLOC, modifying DAN’s regularization term: while the encoder of DAN attempts to fool a discriminator, that of IFLOC attempts to directly minimize KLD between p ( d j y ) and q D ( d j h ) .
Formally, IFLOC solves the following joint optimization problem by alternating gradient descent.
[Equation]
The second term in Eq.14 and Eq.15 respectively means maximization of log-likelihood of q M and q D as well as DAN.
However, the ﬁrst term in Eq.14 differs from DAN in that it is intended to satisfy q D ( d j h )
=
p ( d j y ) for almost every ( y; h ) pair.
Next we show that the regularization of IFLOC is intended to achieve H ~ p E ( d j h )
=
H ( d j y ) .
Similarly to Section 3.2, D (cid:3) and M (cid:3) , which are the solutions to Eq.14 and Eq.15 with ﬁxed E , obviously satisfy q (cid:3) D = ~ p E ( d j h ) ; q (cid:3) M = ~ p E ( y j h ) .
Thus Eq.14 can be written as follows:
[Equation] [Equation]
Since the minimization of the KLD term does not interfere with the second term optimization,
E (cid:3) , which is the solution to Eq.16 and in Nash equilibrium, satisﬁes E h;y (cid:24) ~ p E (cid:3) ( h;y )
[ D KL [ p ( d j y )
j ~ p E (cid:3) ( d j h )]] = 0 .
Then, H ~ p E (cid:3) ( d j h ) =
H ( d j y ) obviously holds.
Note that we cannot obtain true p ( d j y ) , but we can use a maximum likelihood or maximum a posteriori estimator for it.
Also, we could use some divergences other than D KL [ p ( d j y )
j q D ( d j h )] in
Eq.14, e.g., D KL [ q D ( d j h )
j p ( d j y )] , but in doing so, we could not observe performance gain, so we discontinued testing them. [Table]
E XPERIMENTS 5. D ATASETS
BMNISTR We created the Biased Rotated MNIST dataset (BMNISTR) by modifying the sample size of MNISTR (Ghifary et al.
so that class distribution differs among the domains.
Specifically, we created four variants of MNISTR that have different types of domain-class dependency, referred to as BMNISTR-1 through BMNISTR-4.
As shown in Table 1, BMNISTR-1, -2, andhave similar trends but different degrees of dependency; BMNISTR-1 and BMNISTR-4 differ in trends.
In MNISTR, each class is represented by 10 digits.
Each domain was created by rotating images by 15 degree increments: 0, 15, 30, 45, [Table] and 75 (referred to as M0, ..., M75).
Each image is cropped to 16 x 16 in accordance with Ghifary et al.
In training, we employed one-domainleave-out setting: trained on ﬁve of the six domains and then tested using the remaining one.
We used two convolution layers and two fully-connected (FC) layers (with nonlinear activations) as the encoder, three FC layers as the classiﬁer, and two FC layers as the discriminator.
PACS The PACS dataset (Li et al.
(2017)) has 9991 images across 7 categories (dog, elephant, giraffe, guitar, house, horse, and person) and 4 domains comprising different stylistic depictions (Photo, Art painting, Cartoon, and Sketch).
It has domain-class dependency probably because samples in some < domain, class > pairs are difﬁcult to obtain.
For example, p ( y = person j d = Phot) is much higher than p ( y = person j d = Sketch) , which indicates that photos of person are easier to obtain than those of animals, but sketches of persons are more difﬁcult to obtain than those of animals in the wild.
The concrete sample sizes for each category and style is shown in Table 4 in appendix.
In training, we employed one-domain-leave-out setting as well as in BMNISTR, and used the ImageNet pre-trained AlexNet CNN (Krizhevsky et al.
as the base network, following previous studies (Li et al.
(2017; 2018a)).
The two-FC-layer discriminator was connected to the last FC layer, following Ganin et al.
WISDM The WISDM Activity Prediction dataset contains sensor data of accelerometers for six human activities (walking, jogging, upstairs, downstairs, sitting, and standing) performed byusers (domains).
WISDM suffers from the dependency due to the reason noted in Sec.1.
The concrete sample sizes for each user and activity is shown in Table 5 in appendix.
Referring to Andrey (2017), we use the sliding-window procedure with 60 frames (=3 seconds) and 20-frame overlap.
The total number of samples was 54455.
In training, we used randomly chosen < 10 / 26 > , < 16 / 20 > , and < 26 / 10 > users as < source / target > domains.
We parameterized the encoder using three convolution layers followed by one FC layer and the classiﬁer by logistic regression, following previous studies (Yang et al.
; Iwasawa et al.
The two-FC-layer discriminator was connected to the output of the encoder.
5. B ASELINES
To demonstrate the efﬁcacy of the proposed method IFLOC, we compared it with the following methods.
(1) CNN is a vanilla convolutional networks trained on the aggregation of data from all source domains.
Although CNN has no special treatments for domain generalization, Li et al.
(2017) reports that it outperforms many traditional domain generalization methods.
(2) DAN (Xie et al.
[Table] (2017)) is expected to generalize across domains via invariant feature learning [Table] but it has the tradeoff between domain invariance and classiﬁcation accuracy as explained in Section 4.1.1.
We trained
DAN with a gradient reverse layer following Ganin et al.
; Xie et al. (2017).
Also, we used (3) IFLOC-Abl , which is a version of IFLOC modiﬁed for ablation studies.
IFLOC-Abl replaces
D KL [ p ( d j y )
j ~ p E ( d j h )] in Eq.14 of D KL [ p ( d ) j ~ p E ( d j h )] , so it attempts to learn the representation that is completely invariant to domains or make H ( d j h )
=
H ( d ) hold as well as DAN.
Comparing
IFLOC and IFLOC-Abl, we measured the genuine effect of taking domain-class dependency into account.
In training IFLOC and IFLOC-Abl, we cannot obtain true p ( d j y ) and p ( d ) , so we used maximum likelihood estimators of them for calculating the KLD terms.
5. E XPERIMENTAL S ETTINGS
For all the datasets and methods, we used RMSprop for optimization.
And we set the learning rate, batch size, and the number of iterations as 5e-4, 128, and 10k for BMNISTR; 5e-5, 64, and 10k for PACS; 1e-4, 128, and 30k for WISDM, respectively.
For DAN, IFLOC-Abl, and IFLOC we optimized the weighting parameter (cid:13) from f 0 : 0001 ; 0 : 001 ; 0 : 01 ; 0 : 1 ; 1 ; 10 g , and used the (cid:13) annealing following Ganin et al. (2016).
In all the experiments, we split source data into 80% of training data and 20% of validation data, assuming that target data are not absolutely available in the training phase.
We conducted experiments multiple times with different seeds.
Speciﬁcally, we trained on 10 and 25 seeds in BMNISTR and WISDM, chose the best hyperparameter that achieved the highest validation accuracies measured in each epoch, and reported the mean scores (accuracies and f-values) for the hyperparameter.
In PACS, since it requires a long time to train on, we chose the best (cid:13) from f 0 : 0001 ; 0 : 001 ; 0 : 01 ; 0 : 1 g with three experiment, and reported the mean scores in experiments with 20 seeds in total.
Also, we empirically measured the level of domain-invariance by training a post-hoc classiﬁer that is intended to predict d over learned representation, following previous studies (Xie et al.
; Iwasawa et al.
; Moyer et al.
Speciﬁcally, we trained the classiﬁer with 400 hidden units on 10k iterations (by RMSprop optimizer with a 0.001 learning rate and 128 batch size) with the data that is used for training the models.
We then evaluated the domain classiﬁcation accuracy (referred to as D-Acc) 10 times at equal intervals during training, and reported D-Acc in the nearest time when the validation accuracy is maximized.
5. R ESULTS
We ﬁrst investigated how domain-class dependency affects the performance of domain-invariancebased methods.
In Table 2, we compared mean f-scores for the classes 0 through 4 and classesthrough 9 in BMNISTR with the target domain M0.
Recall that sample sizes for the classes 0 (cid:24)are variable across domains, whereas the classes 5 (cid:24) 9 has identical sample sizes across domains (Table 1).
The f-scores show that IFLOC outperformed DAN and IFLOC-Abl in most datasetclass pairs, which supports that domain-class dependency depresses the performance of domaininvariance-based methods and that IFLOC can mitigate the problem.
Futher, relative improvement of
IFLOC to IFLOC-Abl is more signiﬁcant for the classes 0 (cid:24) 4 than 5 (cid:24) 9 in BMNISTR-1, BMNISTR2, and BMNISTR-4, suggesting that IFLOC tends to increase performance more signiﬁcantly for classes where the domain-class dependency occurs.
Also, the improvement is more signiﬁcant in
[Figure] [Figure] BMNISTR-1 than in BMNISTR-3, suggesting that the stronger the domain-class dependency is, the lower the performance of domain-invariance-based methods becomes.
Finally, although the dependencies of BMNISTR-1 and BMNISTR-4 have different trends as described in Table 1, IFLOC improved f-scores in both datasets.
Next we investigated the relationship between the strength of regularization and performance.
Figures 2 and 3 show the hyperparameter sensitivity of class accuracies (Y-Acc) and domain accuracies (D-Acc) for DAN, IFLOC-Abl, and IFLOC.
Note that the gray line in Figures 2-(c) and 3-(c) shows the trivial baseline predicting the majority label.
From these ﬁgures, we can make the following observations.
(1) All the methods including IFLOC could improve the invariance by using stronger regularizer.
Concretely, Figures 2-(c) and 3-(c) show D-Acc tends to become low (invariance becomes high) for all the models when the regularizer becomes strong (such as (cid:13) = 1 or 10) except that
IFLOC-Abl has high D-Acc with (cid:13)
= 10 in Figure 3-(c).
That high D-Acc might be because the validation accuracy achieved the highest value before the domain-invariance matured.
(Recall that the more the representation becomes invariant, the lower the accurary becomes under the trade-off).
(2) The training of IFLOC tends to be more stable than that of DAN when the regularizer becomes strong.
Figures 2-(a,b) and 3-(a,b) show that IFLOC and IFLOC-Abl could achieve higher Y-Acc than DAN when (cid:13) = 1 or 10 , i.e., the regularization is strong, except for IFLOC-Abl with (cid:13) = 1 in Figure 3-(b).
This tendency might be because the regularizer of IFLOC is KLD and thus bounded by 0 , in contrast to that of DAN that can increase to inﬁnity and destabilize the traininig.
(3) IFLOC, as it was designed, does not tend to decrease classiﬁcation accuracy with strong regularizer, and thus IFLOC is robust toward hyperparameter choice.
Figures 2-(b) and 3-(a,b) show that while Y-Acc of IFLOC-Abl decreases with strong regularization (such as when (cid:13) = 1 or 10 ), that of IFLOC does not decrease as much.
Finally, we compared mean accuracies (with standard errors) in both synthetic (BMNISTR) and standard benchmark (PACS and WISDM) datasets (Table 3).
Note that the H ( d ) =
H ( d j y ) column is estimated from source data, which indicates the strength of domain-class dependency.
IFLOC outperformed IFLOC-Abl in BMNISTR with all the target domains; PACS with photo, art painting, and sketch target domains; and WISDM with 26- and 20-target-user domains.
Also, IFLOC outperformed DAN in BMNISTR with all the target domains; PACS with photo and art painting target [Table] domains; and WISDM with 26- and 10-target-user domains.
This supports the importance of considering domain-class dependency in real-world datasets and the efﬁcacy of the proposed model.
Table 3 also shows that when the number of source domains increased from 10 to 26 in WISDM, the improvement of IFLOC from IFLOC-Abl became insigniﬁcant.
One possible reason is that
WISDM with 10 target users has low domain-class dependency than with 26 target users as shown in the H ( d ) =
H ( d j y ) column.
Another possible reason is the optimization difﬁculty.
As Moyer et al.
(2018) reported, in adversarial invariant feature learning, an encoder often overﬁts to the discriminator trained alongside that encoder, and does not provide truly invariant representation (the same problem can be observed in Figures 2-(c) and 3-(c)).
We suspect that when the number of source domains increases, the optimization of the domain discriminator becomes difﬁcult, which makes the encoder overﬁt to that poor discriminator and worsen the problem.
Also, the improvement of IFLOC from IFLOC-Abl is less signiﬁcant in WISDM than that in BMNISTR and PACS, which could be related to the same problem since the number of source domains for BMNISTR and PACS is smaller than that for WISDM.
If the optimization difﬁculty prevents IFLOC from working properly, we might be able to mitigate it by using ideas from the studies that investigate the convergence and optimization difﬁculty in adversarial training (e.g., Nagarajan & Kolter (2017);
Heusel et al.
Balduzzi et al.
C ONCLUSION
In this paper, we addressed domain generalization under domain-class dependency , which was overlooked by most prior domain generalization methods relying on domain-invariant representation.
We theoretically showed the importance of considering the dependency and the way to overcome the problem by expanding the analysis of Xie et al.
We then proposed a novel method IFLOC, which maximizes domain-invariance within a range that does not interfere with classiﬁcation accuracy.
Empirical validations show the superior performance of IFLOC to the baseline methods, supporting the importance of the domain-class dependency in domain generalization tasks and the efﬁcacy of the proposed method for overcoming the issue.
Future work includes applying the reguralization idea of making H ( d j h )
=
H ( d j y ) to other methods, e.g., Conditional VAE (Louizos et al. (2016)) or CrossGrad (Shankar et al. because they have clear and tractable data generating process but assume the independence of y and d .
Also intended is to use it for transfer learning tasks in a few-shot setting (e.g., life-long learning) where domain-class dependency is likely to occur due to scarce sample size. A D OMAIN - CLASS DEPENDENCY IN PACS AND WISDM
[Table]
[Table]
User
User
User
User
User
User
User
User
User
User
User
User
User
User
User
User