A DVERSARIAL F EATURE L EARNING UNDER A CCURACY C ONSTRAINT FOR D OMAIN G ENERALIZATION
A BSTRACT
Learning domain-invariant representation is a dominant approach for domain generalization.
However, previous methods based on domain invariance overlooked the underlying dependency of classes on domains, which is responsible for the trade-off between classiﬁcation accuracy and the invariance.
This study proposes a novel method adversarial feature learning under accuracy constraint (AFLAC) , which maximizes domain invariance within a range that does not interfere with accuracy.
Empirical validations show that the performance of AFLAC is superior to that of baseline methods, supporting the importance of considering the dependency and the efﬁcacy of the proposed method to overcome the problem
I NTRODUCTION
In supervised learning we typically assume that samples are obtained from the same distribution in training and testing; however, because this assumption does not hold in many practical situations it reduces the classiﬁcation accuracy for the test data (Torralba & Efros, 2011).
One typical situation is domain generalization (DG) (Blanchard et al., 2011; Shankar et al., 2018; Sriram et al., 2018;
Erfani et al., 2016): we have labeled data from several source domains and collectively exploit them such that the trained system generalizes to other unseen, but somewhat similar, target domain(s).
This paper considers DG under the situation where domain d and class y labels are statistically dependent owing to some common latent factor z (Figure 1-(c)), which we referred to as domainclass dependency .
For example, the WISDM Activity Prediction dataset (WISDM, Kwapisz et al.
(2011)), where y and d correspond to activities and wearable device users, exhibits this dependency because (1) some activities (e.g., jogging) are strenuous to the extent that some unathletic subjects avoided them ( data characteristics ), or (2) other activities were added only after the study began and the initial subjects could not perform them ( data-collection errors ).
The dependency is common in real-world datasets (Zhang et al., 2013) and a similar setting has been investigated in domain adaptation (DA) studies, but most prior DG studies overlooked the dependency.
Most prior DG methods utilize invariant feature learning (IFL)
(e.g., Muandet et al.
IFL attempts to learn feature representation h from input data x which is invariant to d .
When source and target domains have some common structure (see, Muandet et al.
, IFL prevents the classiﬁer from overﬁtting to source domains (Figure 1-(b)).
However, under the dependency, merely imposing the domain invariance can adversely affect the classiﬁcation accuracy as pointed out by
Xie et al.
and illustrated in Figure 1.
Although that trade-off occurs in source domains (because DG uses only source data during optimization), it can also negatively affect the classiﬁcation performance for target domain(s).
For example, if the target domain has characteristics similar (or same as an extreme case) to those of a certain source domain, giving priority to domain invariance obviously interferes with the DG performance (Figure 1-(d)).
In this paper, considering that prioritizing domain invariance under the trade-off can negatively affect the DG performance, we propose a novel method adversarial feature learning under accuracy constraint (AFLAC) , which maximizes domain invariance within a range that does not interfere with the classiﬁcation accuracy (Figure 1-(e)) on adversarial training.
Speciﬁcally, AFLAC is intended to achieve accuracy-constrained domain invariance , which we deﬁne as the maximum H ( d j h ) ( H denotes entropy) value under the condition H ( y j x ) =
H ( y j h ) ( h has as much y information as x ).
Empirical validations show that the performance of AFLAC is superior to that of baseline methods, supporting the importance of considering domain-class dependency and the efﬁcacy of the proposed approach for overcoming the issue.
R ELATED W ORK
DG has been attracting considerable attention in recent years, and most prior DG methods utilize
IFL (Muandet et al., 2013; Erfani et al., 2016; Ghifary et al., 2017).
In particular, our proposed method is based on Domain Adversarial Nets (DAN), which was originally invented for DA (Ganin et al., 2016) and Xie et al.
(2017) demonstrated its efﬁcacy in DG.
In addition, Xie et al.
(2017) intuitively explained the trade-off between classiﬁcation accuracy and domain invariance, but they did not suggest any solution to the problem except for carefully tuning a weighting parameter.
Several studies that address DG without utilizing IFL have been conducted.
For example, CCSA (Motiian et al., 2017), CIDG (Li et al., 2018b), and CIDDG (Li et al., 2018c) proposed to make use of semantic alignment , which attempts to make latent representation given class label ( p ( h j y ) )
identical within source domains.
This approach was originally proposed by Gong et al.
(2016) in the
DA context, but its efﬁcacy to overcome the trade-off problem is not obvious.
CrossGrad (Shankar et al., 2018), which is one of the recent state-of-the-art DG methods, utilizes data augmentation with adversarial examples.
However, because the method relies on the assumption that y and d are independent, it might not be directly applicable to our setting.
In DA, Zhang et al.
; Gong et al.
(2016) address the situation where p ( y ) changes across the source and target domains by correcting the change of p ( y ) using unlabeled target data, which is often accomplished at the cost of classiﬁcation accuracy for the source domain.
However, this approach is not applicable (or necessary) to DG because we are agnostic on target domain(s), and this paper is concerned with the change of p ( y ) within source domains.
Instead, we propose to maximize the classiﬁcation accuracy for source domains while improving the domain invariance.
P ROPOSED M ETHOD 3. A CCURACY -C ONSTRAINED D OMAIN I NVARIANCE
Here we provide the notion of accuracy-constrained domain invariance, which is the maximum domain invariance within a range that does not interfere with the classiﬁcation accuracy.
The reason for the constraint is that the primary purpose of DG is the classiﬁcation for unseen domains rather than domain itself, and the improvement of the invariance could detrimentally affect the performance.
Theorem 1 Let h
= f ( x ) , i.e., h is a deterministic mapping of x with a function f .
We deﬁne accuracy-constrained domain invariance as the maximum H ( d j h ) value under the constraint that
H ( y j x ) = 0 , i.e., there is no labeling error, and h has as much y information as x , i.e., H ( y j h ) =
H ( y j x ) .
Accuracy-constrained domain invariance equals H ( d j y ) .
Proof 1 Using the properties of entropy, the following inequation holds:
[Equation]
By assumption, H ( y j x ) =
H ( y j h ) = 0 , and thus the following inequation holds:
[Equation]
Thus, the maximum H ( d j h ) value under the constraints is H ( d j y ) .
3. AFLAC
We propose a novel method named AFLAC, which is designed to achieve accuracy-constraind domain invariance.
Formally, we denote f E ( x ) ; q M ( y j h ) ; and q D ( d j h ) ( E; M , and D are the parameters) as the deterministic encoder, probabilistic model of the label classiﬁer, and that of domain discriminator, respectively.
Then, the objective function of AFLAC is described as follows:
[Equation]
[Equation]
Here (cid:13) denotes a weighting parameter.
Note that, although we cannot obtain true distribution p ( d j y ) , we can use the maximum likelihood estimator of it when y and d are discrete, as is usual with DG.
Here we formally show that AFLAC is intended to achieve H ( d j h )
=
H ( d j y ) (accuracy-constrained domain invariance) by a Nash equilibrium analysis similar to Goodfellow et al.
; Xie et al.
We deﬁne D (cid:3) and M (cid:3) as the solutions to Eq.
3 and Eq. 4 with ﬁxed E .
They obviously satisfy q (cid:3) D = p ( d j h ) ; q (cid:3) M = p ( y j h ) , respectively.
Thus, V in Eq.
3 can be written as follows:
[Equation]
E (cid:3) , which we deﬁne as the solution to Eq. 5 and in Nash equilibrium, satisﬁes not only H ( y j h ) =
H ( y j x ) (optimal classiﬁcation accuracy) but also E h;y (cid:24) p ( h;y )
[ D KL [ p ( d j y )
j p ( d j h )]] = 0 , which is a sufﬁcient condition for H ( d j h ) = H ( d j y ) by the deﬁnition of the conditional entropy.
E XPERIMENTS 4. D ATASETS
BMNISTR We created the Biased Rotated MNIST dataset (BMNISTR) by modifying the sample size of the popular benchmark dataset MNISTR (Ghifary et al., 2015), such that the class distribution differed among the domains.
In MNISTR, each domain was created by rotating images by 15 degree increments: 0, 15, ..., 75 (referred to as M0, ..., M75).
We created four variants of MNISTR that have different types of domain-class dependency, referred to as BMNISTR-1 through BMNISTR-3.
As shown in Table 1-left, BMNISTR-1, -2 have similar trends but different degrees of dependency, whereas BMNISTR-1 and BMNISTR-3 differ in terms of their trends.
In training, we employed a leave-one-domain-out setting (Ghifary et al., 2015): we trained the models on ﬁve of the six domains and tested them using the remaining one.
WISDM WISDM contains sensor data of accelerometers of six human activities (walking, jogging, upstairs, downstairs, sitting, and standing) performed by 36 users (domains).
WISDM has the dependency for the reason noted in Sec.
we randomly selected < 10 / 26 > and < 26 / 10 > users as < source / target > domains, and split the source data into training and validation data.
4. B ASELINES
We compared AFLAC with the following methods.
(1) CNN is a vanilla convolutional networks trained on the aggregation of data from all source domains.
(2) DAN (Xie et al., 2017) is expected
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Figure]
to generalize across domains utilizing domain-invariant representation, but it can be affected by the trade-off as pointed out by Xie et al.
(3) CIDDG is our re-implementation of Li et al.
(2018c), which is designed to achieve semantic alignment on adversarial training.
Additionally, we used (4) AFLAC-Abl , which is a version of AFLAC modiﬁed for ablation studies.
AFLACAbl replaces D KL [ p ( d j y )
j q D ( d j h )] in Eq.
3 of D KL [ p ( d ) j q D ( d j h )] , thus it attempts to learn the representation that is completely invariant to domains or make H ( d j h )
=
H ( d ) hold as well as
DAN.
Comparing AFLAC and AFLAC-Abl, we measured the genuine effect of taking domain-class dependency into account.
When training AFLAC and AFLAC-Abl, we cannot obtain true p ( d j y ) and p ( d ) , hence we used their maximum likelihood estimators for calculating the KLD terms.
4. R ESULTS
We ﬁrst investigated the extent to which domain-class dependency affects the performance of domain-invariance-based methods.
In Table 1-right, we compared the mean F-measures for the classes 0 through 4 and classes 5 through 9 in BMNISTR with the target domain M0.
Recall that the sample sizes for the classes 0 (cid:24) 4 are variable across domains, whereas the classes 5 (cid:24) 9 have identical sample sizes across domains.
The F-measures show that AFLAC outperformed baselines in most dataset-class pairs, which supports that the dependency reduces the performance of IFL methods and that AFLAC can mitigate the problem.
Further, the relative improvement of AFLAC to AFLAC-Abl is more signiﬁcant for the classes 0 (cid:24) 4 than for 5 (cid:24) 9 in BMNISTR-1 and BMNISTR-3, suggesting that AFLAC tends to increase performance more signiﬁcantly for classes in which the dependency occurs.
Moreover, the improvement is more signiﬁcant in BMNISTR-1 than in BMNISTR-2, suggesting that the stronger the dependency is, the lower the performance of domain-invariance-based methods becomes.
Finally, although the dependencies of BMNISTR-1 and BMNISTR-3 have different trends, AFLAC improved the F-measures in both datasets.
Next we investigated the relationship between the strength of regularization and performance.
Figures 2-(b, c, d) show that the accuracy gaps of AFLAC-Abl and AFLAC increase with strong regularization (such as when (cid:13) = 10 ), suggesting that AFLAC, as it was designed, does not tend to reduce accuracy with strong regularizer, and thus AFLAC is robust toward hyperparameter choice.
C ONCLUSION
In this paper, we proposed a novel method AFLAC, which maximizes domain invariance within a range that does not interfere with classiﬁcation accuracy on adversarial training.
Empirical validations show the superior DG performance of AFLAC to the baseline methods, supporting the importance of the domain-class dependency in domain generalization tasks and the efﬁcacy of the proposed method for overcoming the issue.