D OMAIN -F REE A DVERSARIAL S PLITTING FOR D OMAIN G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Domain generalization is an approach that utilizes several source domains to train the learner to be generalizable to unseen target domain to tackle domain shift issue.
It has drawn much attention in machine learning community.
This paper aims to learn to generalize well to unseen target domain without relying on the knowledge of the number of source domains and domain labels.
We unify adversarial training and meta-learning in a novel proposed Domain-Free Adversarial Splitting (DFAS) framework.
In this framework, we model the domain generalization as a learning problem that enforces the learner to be able to generalize well for any train/val subsets splitting of the training dataset.
To achieve this goal, we propose a min-max optimization problem which can be solved by an iterative adversarial training process.
In each iteration, it adversarially splits the training dataset into train/val subsets to maximize domain shift between them using current learner, and then updates the learner on this splitting to be able to generalize well from train-subset to val-subset using meta-learning approach.
Extensive experiments on three benchmark datasets under three different settings on the source and target domains show that our method achieves state-of-the-art results and conﬁrm the effectiveness of our method by ablation study.
We also derive a generalization error bound for theoretical understanding of our method.
I NTRODUCTION
Deep learning approach has achieved great success in image recognition
(He et al., 2016; Krizhevsky et al., 2012; Simonyan & Zisserman, 2014).
However, deep learning methods mostly succeed in the case that the training and test data are sampled from the same distribution ( i.e. , the i.i.d. assumption).
However, this assumption is often violated in real-world applications since the equipments/environments that generate data are often different in training and test datasets.
When there exists distribution difference ( domain shift (Torralba & Efros, 2011) ) between training and test datasets, the performance of trained model, i.e., learner , will signiﬁcantly degrade.
To tackle the domain shift issue, domain adaptation approach (Pan & Yang, 2010; Daume III &
Marcu, 2006; Huang et al., 2007) learns a transferable learner from source domain to target domain.
Domain adaptation methods align distributions of different domains either in feature space (Long et al., 2015; Ganin et al., 2016) or in raw pixel space (Hoffman et al., 2018), which relies on unlabeled data from target domain at training time.
However, in many applications, it is unrealistic to access the unlabeled target data, therefore this prevents us to use domain adaptation approach in this setting, and motivates the research on the learning problem of domain generalization.
Domain generalization (DG) approach (Blanchard et al., 2011; Muandet et al., 2013) commonly uses several source domains to train a learner that can generalize to an unseen target domain.
The underlying assumption is that there exists a latent domain invariant feature space across source domains and unseen target domain.
To learn the domain invariant features, (Muandet et al., 2013;
Ghifary et al., 2015; Li et al., 2018b) explicitly align distributions of different source domains in feature space.
(Balaji et al., 2018; Li et al., 2019b; 2018a; Dou et al., 2019) split source domains into meta-train and meta-test to simulate domain shift and train learner in a meta-learning approach.
(Shankar et al., 2018; Carlucci et al., 2019;
Zhou et al., 2020; Ryu et al., 2020) augment images or features to train learner to enhance generalization capability.
Conventional domain generalization methods assume that the domain labels are available.
But in a more realistic scenario, the domain labels may be unknown (Wang et al., 2019).
To handle this domain-free setting, Carlucci et al.
(2019) combines supervised learning and self-supervised learning to solve jigsaw puzzles of the training images.
Matsuura & Harada (2020) divides samples into several latent domains via clustering and trains a domain invariant feature extractor via adversarial training.
Huang et al.
(2020) discards the dominant activated features, forcing the learner to activate remaining features that correlate with labels.
Another line of works (Volpi et al., 2018; Qiao et al., 2020) tackle the single source setting that the training set comprises a single domain, and the train and test data are from different domains.
In this work, we focus on a general learning scenario of domain generalization as follows.
First, we do not know the domain label of each data and do not assume that there are several domains in the training dataset.
Second, we do not assume that the training and test data are from different domains ( e.g. , styles).
However, the previous domain-free DG methods (Matsuura & Harada, 2020) commonly evaluate on the datasets ( e.g. , PACS) composed of several domains though they do not use domain labels in training.
In our domain-free setting, we do not assume and know the domains in the training dataset, we therefore model domain generalization as a learning problem that the learner should be able to generalize well for any splitting of train/val subsets, i.e. , synthetic source/target domains, over the training dataset.
This explicitly enforces that the trained learner should be generalizable for any possible domain shifts within the training dataset.
To achieve this goal, we propose an adversarial splitting model that is a min-max optimization problem, due to the difﬁculty of enumerating all splittings.
In this min-max problem, we adversarially split training dataset to train/val subsets by maximizing the domain shift between them based on the given learner, and then update learner by minimizing the prediction error on val-subset using meta-learning approach given the splitting.
By optimizing this min-max problem, we enforce the learner to generalize well even in the worst-case splitting.
We also investigate L 2 -normalization of features in our domain generalization method.
It is surprisingly found that L 2 -normalization can improve performance of learner and mitigate gradient explosion in the meta-learning process of DG.
We further theorectically analyze the underlying reasons for this ﬁnding.
This proposed domain generalization approach is dubbed Domain-Free Adversarial Splitting , i.e., DFAS .
To verify the effectiveness of our method, we conduct extensive experiments on benchmark datasets of PACS, Ofﬁce-Home and CIFAR-10 under different settings with multiple/single source domains.
In experiments that the training data are from several source domains, our method achieves state-ofthe-art results on both PACS and Ofﬁce-Home datasets.
We also ﬁnd that our method signiﬁcantly outperforms baselines in experiments that the training data are from a single source domain on PACS and CIFAR-10.
We also conﬁrm the effectiveness of our method by ablation study.
Based on domain adaptation theory, we also derive an upper bound of the generalization error on unseen target domain.
We analyze that the terms in this upper bound are implicitly minimized by our method.
This theoretical analysis partially explains the success of our method.
R ELATED W ORKS
We summarize and compare with related domain generalization (DG) methods in two perspectives, i.e. , DG with domain labels and DG without domain labels.
DG with domain labels.
When the domain labels are available, there are three categories of methods for DG.
First, (Muandet et al., 2013; Ghifary et al., 2015; Li et al., 2018b; Piratla et al., 2020) learn domain invariant features by aligning feature distributions or by common/speciﬁc feature decomposition.
Second, (Li et al., 2019a; Balaji et al., 2018; Li et al., 2019b; 2018a; Dou et al., 2019; Du et al., 2020a;b) are based on meta-learning approach that splits given source domains into meta-train and meta-test domains and trains learner in an episodic training paradigm.
Third, (Shankar et al., 2018; Carlucci et al., 2019;
Zhou et al., 2020; Wang et al., 2020) augment fake domain data to train learner for enhancing generalization capability of learner.
Our method may mostly relate to the above second category of methods.
But differently, we consider the DG problem in domain-free setting and adversarially split training dataset to synthesize domain shift in a principled min-max optimization method, instead of using leave-one-domain-out splitting in these methods.
DG without domain labels.
When the domain label is unavailable, to enhance generalization ability of learner, Wang et al.
(2019) extracts robust feature representation by projecting out superﬁcial patterns like color and texture.
Carlucci et al.
(2019) proposes to solve jigsaw puzzles of the training images.
Matsuura & Harada (2020) divides samples into several latent domains via clustering and learns domain invariant features via adversarial training of feature extractor and domain discriminator.
Huang et al.
(2020) discards the dominant activated features, forcing the learner to activate remaining features that correlate with labels.
Volpi et al.
(2018) and Qiao et al.
(2020) propose adversarial data augmentation to tackle the setting that the training set comprises a single domain.
In methodology, these methods either explicitly force the learner to extract robust features (Wang et al., 2019; Matsuura & Harada, 2020; Huang et al., 2020) or augment new data to increase training data (Carlucci et al., 2019; Qiao et al., 2020; Volpi et al., 2018).
While our method is a novel meta-learning approach for DG by introducing adversarial splitting of training dataset during training, without relying on data/domain augmentation.
Moreover, our approach is ﬂexible in handling the settings that the training data could be from a single or several domains.
M ETHOD
In our setting, since we do not assume and know the domains in the training dataset, the training data could be independently sampled from several underlying source domains or just from a single source domain.
We denote S
=
( x i , y i ) } Ni =1 as the training dataset.
Our goal is to train the learner with S that can generalize well on an unseen target domain.
In the following sections, we introduce details of our proposed model in Sect.
3.1, followed by its optimization method in Sect.
We also investigate L 2 -normalization for domain generalization in Sect. 3.3.
Theoretical analysis for our method is presented in Sect.
Experimental results are reported in Sect.
Sect.
6 concludes this paper.
3. D OMAIN -F REE A DVERSARIAL S PLITTING M ODEL
As mentioned in Sect.
1, we model DG as a learning problem that enforces the learner to be able to generalize well for any train/val subsets splitting of the training dataset.
The learner is trained using meta-learning approach (Finn et al., 2017).
To formulate our idea mathematically, we ﬁrst introduce some notations.
We denote f as a function/learner ( f could be a deep neural network, e.g. ,
ResNet (He et al., 2016)) that outputs classiﬁcation score of the input image, l as the loss such as cross-entropy, S t and S v as the train-subset and val-subset respectively such that S =
S t ∪ S v
and
S t ∩ S v = ∅ .
The formulated optimization problem for domain generalization is
[Equation]
S
[Equation]
In Eq.
(1), Γ ξ = { S v : S v ⊂ S, | S v | = ξ } is the set of all possible val-subsets of S with length of ξ , S t =
S − S v is train-subset paired with each S v , L ( θ ( w ); S v ) =
1 | S v | (cid:80) ( x,y ) ∈ S v l ( f θ ( w ) ( x ) , y ) is the loss on S v , where θ ( w ) is the parameters of f , L ( θ ; S t , w ) is L ( θ ; S t ) with θ initialized by w and R ( w ) is regularization term.
In the optimization model of Eq.
(1), the parameter θ ( w ) of learner trained on S t is treated as a function of the initial parameter w .
To force the learner trained on S t to generalize well on S v , we directly minimize the loss L ( θ ( w ); S v ) , dubbed generalization loss , on val-subset S v , w.r.t.
the parameter θ ( w ) trained on S t .
Solving Eq.
(1) will force the learner to be able to generalize well from any train-subset to corresponding val-subset.
Since | Γ ξ | may be extremely large, it is infeasible to enumerate all possible train/val splittings.
Thus, we propose the following adversarial splitting model instead,
[Equation]
In the min-max problem of Eq.
(2), the train/val ( S t / S v ) splitting is optimized to maximize the generalization loss to increase the domain shift between train and val subsets by ﬁnding the hardest splitting to the learner.
While w is optimized by minimizing the generalization loss of learner over the splitting.
Solving this adversarial splitting optimization model in Eq.
(2) enforces the learner to be generalizable even for the worst-case splitting.
We therefore expect that the trained learner is robust to the domain shifts within the training dataset.
For the regularization term R ( w ) , we set it to be the training loss on S t ( i.e. , R ( w ) =
L ( w ; S t ) ), which additionally constrains that the learner with parameter w should be effective on S t (Li et al., 2018a).
The effect of the hyper-parameters ξ will be discussed in Appendix B.3.
In conventional adversarial machine learning, adversarial training is imposed on adversarial samples and learner to increase robustness of the learner to adversarial corruption (Goodfellow et al., 2015).
While in our optimization model of Eq.
(2), adversarial training is conducted on data splitting and learner to force the learner to be robust to domain shift between train/val subsets.
Our model bridges adversarial training and meta-learning.
It is a general learning framework for domain generalization and is a complement to adversarial machine learning.
3. O PTIMIZATION
This section focuses on the optimization of Eq.
Since Eq.
(2) is a min-max optimization problem, we alternately update S v and w by ﬁxing the other one as known.
We should also consider the inner loop for optimization of θ ( w ) in the bi-layer optimization problem of Eq.
We next discuss these updating steps in details.
The convergence and computational cost of this algorithm will be also discussed in Appendix B.4 and B.5 respectively.
Inner loop for optimization of θ ( w ) .
We adopt ﬁnite steps of gradient descent to approximate the minimizer θ ( w ) of the inner objective L ( θ ; S t ) with initial value w .
This approximation technique has been introduced in machine learning community several years ago (Sun & Tappen, 2011; Finn et al., 2017; Fan et al., 2018).
For convenience of computation, following (Li et al., 2018a; Dou et al., 2019), we only conduct gradient descent by one step as
[Equation]
where α is the step size of inner optimization and its effect will be discussed in Appendix B.3.
Optimizing w with ﬁxed S v .
For convenience, we denote g tw = ∇ θ L ( θ ; S t ) | θ = w .
Fixing S v ( S t is then ﬁxed), w can also be updated by gradient descent, i.e. ,
[Equation]
where η is the step size of outer optimization.
Finding the hardest splitting S v with ﬁxed w .
Fixing w , to ﬁnd S v ∈ Γ ξ to maximize L ( w − αg tw ; S v ) , we do ﬁrst order Taylor expansion for L ( w − αg tw ; S v ) by L ( w − αg tw ; S v ) ≈ L ( w ; S v )
− α (cid:104)
g tw , g vw (cid:105) , where g vw = ∇ θ
L
( θ ; S v ) | θ = w and (cid:104)· , ·(cid:105) denotes the inner product.
From the deﬁnition of L , g tw and g vw , the optimization problem of max S v ∈ Γ ξ {L ( w ; S v ) − α (cid:104)
g tw , g vw (cid:105)} can be written as max S v ∈ Γ ξ { 1 | S v | (cid:80) ( x
,y ) ∈ S v l ( f w ( x ) , y ) − α (cid:104)∇ w l ( f w ( x ) , y ) , g t w (cid:105)} .
This problem is equivalent to the following splitting formulation:
[Equation]
where we introduced an auxiliary variable A .
Eq.
(5) can be solved by alternatively updating S v and A .
Given A , we compute and rank the values of l ( f w ( x ) , y ) − α (cid:104)∇ w l ( f w ( x ) , y ) , A (cid:105) for all ( x, y ) ∈ S and select the largest ξ samples to constitute the S v .
Given S v ( S t is then given), we update A by A = g tw = 1 | S t | (cid:80) ( x,y
) ∈
S t ∇ w l ( f w ( x ) , y ) .
We also discuss details and convergence of this alternative iteration in Appendix D. Since computing gradient w.r.t.
all parameters is time and memory consuming, we only compute gradient w.r.t.
parameters of the ﬁnal layer of learner f .
3. L 2 -N ORMALIZATION FOR E XTRACTED F EATURE
L 2 -normalization has been used in face recognition (Liu et al., 2017; Wang et al., 2018) and domain adaptation (Saito et al., 2019; Gu et al., 2020), but is rarely investigated in domain generalization.
We investigate L 2 -normalization in domain generalization in this paper.
It is found surprisingly in experiments that L 2 -normalization not only improves the performance of learner (see Sect. 5.4 and Appendix A), but also mitigates gradient explosion (see Appendix B.1) that occurs frequently during the training of meta-learning for DG (Finn et al., 2017; Dou et al., 2019).
We next discuss details of L 2 -normalization in our method and analyze why L 2 -normalization mitigates gradient explosion.
Feature L 2 -normalization.
The L 2 -normalization is used as a component of our learner f .
Specifically, we decompose f into feature extractor f e
( e.g. , the convolutional layers of ResNet), the transform f n representing L 2 -normalization and classiﬁer f c , i.e. , f
= f
c ◦ f n ◦ f e .
The feature of input image x extracted by f e is fed to f n to output an unit vector z , i.e. , z = f n ( f e ( x ))
=
f e
[Equation]
x ) [Equation] f e
[Equation]
x ) (cid:107) .
The classiﬁer f c consists of unit weight vectors W =
[ w 1 , w 2 , · · · , w K ] , where K is the number of classes and (cid:107) w k (cid:107)
= 1 , ∀ k .
f c takes z as an input and outputs the classiﬁcation score vector σ m,s ( W T z ) .
σ m,s ( · ) is the marginal softmax function deﬁned by
[Equation]
where y is the label of x , [ · ] k indicates the k -th element, I { a } is the indicator function that returns 1 if a is true, 0 otherwise, m and s are hyper-parameters indicating margin and radius respectively.
Analysis of mitigating gradient explosion.
We ﬁnd that L 2 -normalization mitigates gradient explosion in the training of meta-learning for domain generalization.
For the sake of simplicity, we analyze gradient norm of loss w.r.t.
parameters of f c in the meta-learning process of domain generalization, with f e as ﬁxed function.
Without loss of generality, we consider the case that K
=
( i.e. , binary classiﬁcation), s = 1 and m = 0 .
In this case, we have the following proposition.
Proposition 1.
Under the above setting, if the input feature of f c is L 2 -normalized, the gradient norm of loss w.r.t.
parameters of f c in the meta-learning process of DG is bounded.
Sketch of proof .
Given feature z , the loss of binary classiﬁcation is L ( w ; z ) =
− y log( σ ( w T z )) − (1 − y )
log(1 − σ ( w T z )) , where σ is the sigmoid function.
Let w (cid:48)
=
w − α ∇ w L ( w ; z ) , then ∇ w L ( w (cid:48) ; z ) =
( I − αH ) ∇ w (cid:48)
L ( w (cid:48) ; z ) , where H is the Hessian matrix.
The gradient norm (cid:107)∇ w L ( w (cid:48) ; z ) (cid:107) ≤ (cid:107)
I − αH (cid:107) (cid:107)∇ w (cid:48)
L ( w (cid:48) ; z ) (cid:107) ≤ (1 + | α | (cid:107) H (cid:107) ) (cid:107)∇ w (cid:48)
L ( w (cid:48) ; z ) (cid:107) .
Since ∇ w (cid:48)
L ( w (cid:48) ; z ) =
( p − y )
z and H = p (1 − p )
zz T where p = σ ( w T z ) , (cid:107) H (cid:107) =
sup u : (cid:107)
u (cid:107) =1 (cid:107) Hu (cid:107) ≤ sup u : (cid:107) u (cid:107) =1 (cid:13)(cid:13)
zz T u (cid:13)(cid:13) ≤ (cid:107)
z (cid:107) 2 and (cid:107)∇ w (cid:48)
L ( w (cid:48) ; z ) (cid:107) ≤ (cid:107)
z (cid:107) .
If (cid:107) z (cid:107)
= 1 , we have (cid:107)∇ w L ( w (cid:48) ; z ) (cid:107) ≤ 1 + | α | .
According to Proposition 1, L 2 -normalization can mitigate gradient explosion under the above setting.
The analysis of gradient norm of loss w.r.t.
parameters of both f c and f e in the meta-learning process is much more complex, left for our future work.
T HEORETICAL A NALYSIS
This section presents theoretical understanding of our method.
We ﬁrst derive a generalization error bound on target domain in theorem 1 for the general setting of meta-learning for DG.
Then, based on theorem 1, we theoretically explain the reason on the success of our method.
Without loss of generality, we consider binary classiﬁcation problem.
We denote H as the set of all possible f , i.e. , H = { f w : w ∈ R M } , where M is the number of parameters.
For any S v ∈ Γ ξ and S t = S − S v , we let H S t
= { f θ ( w ) : θ ( w ) =
arg min θ L ( θ ; S t , w ) , w ∈ R M } .
The metalearning approach for DG is to ﬁnd a function in H S t to minimize classiﬁcation loss on S v .
Note that, although the training samples in S may be sampled from several distributions, they can still be seen as being i.i.d.
sampled from a mixture of these distributions.
We next respectively denote
P = (cid:80) Dd =1 β d P d as the mixture distribution with β d representing the sampling ratio of the d -th source domain, (cid:15) Ψ Q ( f ) =
E ( x,y ) ∼Q
[ I { Ψ( f ( x )) (cid:54) =
y } ] as the generalization error on distribution Q of unseen target domain, ˆ (cid:15) Ψ S v ( f ) = 1 | S v | (cid:80) ( x
,y ) ∈ S v I { Ψ( f ( x ))
(cid:54) =
y } as the empirical error on S v , V C
( H )
as the VC-dimension of H , and Ψ( · ) as the prediction rule such as the Bayes Optimal Predictor.
Based on the domain adaptation theory (Ben-David et al., 2007; 2010) and inspired by the analysis in (Saito et al., 2019), we have the following theorem.
Theorem 1.
Let γ be a constant, assume E Q
[ I { l ( f ( x ) ,y ) >γ } ] ≥ E P [ I { l ( f ( x ) ,y ) >γ } ] , then given any
S v ∈ Γ ξ and S t = S − S v and for any δ ∈ (0 , 1) , with probability at least 1 − 2 δ , we have ∀ f ∈ H S t ,
[Table]
[Equation]
where
[Equation]
loss-related indicator deﬁned by
[Equation]
Proof is given in Appendix E. The assumption of E Q [ I { [Equation] ( f ( x
[Equation]
y ) >γ } ] ≥ E P [ I { l ( f ( x ) ,y ) >γ } ] in theo- rem 1 is realistic because the data of Q is not accessed at training time [Equation] and the learner trained on data of P should have smaller classiﬁcation loss on P than Q .
In theorem 1, C 1 , C 2 , C 3 are constants to f .
In Eq.
(7), the generalization error (cid:15)
Ψ l Q ( f ) on Q can be bounded by the empirical error ˆ (cid:15)
Ψ l S v ( f ) on S v , the term B ( S v ) that measures the discrepancy between P and Q , and the last two constant terms in Eq.
To obtain lower (cid:15) Ψ l Q ( f ) , we need to minimize ˆ (cid:15)
Ψ l S v ( f ) and B ( S v ) .
Minimizing B ( S v ) w.r.t.
S v is equivalent to
[Equation]
Intuitively, Eq.
(10) means to ﬁnd a S v ∈ Γ ξ such that the inﬁmum ratio of examples in S v having loss greater than γ is maximized.
This min-max problem of Eq.
(10) for computing the error bound bears the similar idea as our min-max problem.
Our adversarial splitting model in Eq.
(2) can implicitly realize the goal of Eq.
(10) and meanwhile ensure lower ˆ (cid:15)
Ψ l S v ( f ) for any S v .
The maximization in Eq.
(10) corresponds to our adversarial splitting that ﬁnds the hardest val-subset
S v for the learner in Eq.
The inﬁmum in Eq.
(10) corresponds to the minimization of the loss in Eq.
(2) on S v w.r.t.
the learner parameterized by θ ( w ) .
Instead of using indicator function I in
Eq.
(10), in our model of Eq.
(2), we choose differentiable classiﬁcation loss for easier optimization.
E XPERIMENTS
We verify the effectiveness of our method in three types of experimental settings: Multi Source with
Domain Shift (MSDS) that the training data are from several source domains and there exists domain
[Table]
[Table]
shift between training and test data, Single Source with Domain Shift (SSDS) that the training data are from a single source domain and there exists domain shift between training and test data, and Same
Source and Target Domain (SSTD) that the training and test data are from a same single domain.
The source codes will be released online.
We conduct experiments on three benchmark datasets.
PACS (Li et al., 2017) contains four domains, including art painting (A), cartoon (C), photo (P), sketch (S), sharing seven classes.
OfﬁceHome (Volpi et al., 2018), a dataset widely used in domain adaptation and recently utilized in domain generalization, consists of four domains: Art (Ar), Clipart (Cl), Product (Pr), Real World (Rw), sharing 65 classes.
Both of these two datasets are utilized to conduct experiments in settings of MSDS and SSDS .
CIFAR-10 (Krizhevsky et al., 2009) is taken for the experimental setting of SSTD .
5. T YPE I: M ULTI S OURCE WITH D OMAIN S HIFT (MSDS)
In the setting of MSDS , following (Carlucci et al., 2019), we use leave-one-domain-out crossvalidation, i.e. , training on three domains and testing on the remaining unseen domain, on PACS and
Ofﬁce-Home.
Note that the domain labels are not used during training.
We adopt ResNet18 and
ResNet50 (He et al., 2016)
pre-trained on ImageNet (Russakovsky et al., 2015).
For each of them, the last fully-connected layer is replaced by a bottleneck layer, then the corresponding network is taken as feature extractor f e .
Full implementation details are reported in Appendix C.
We compare our method with several state-of-the-art methods, including D-SAM (D’Innocente &
Caputo, 2018), JiGen (Carlucci et al., 2019), MASF (Dou et al., 2019), MMLD (Matsuura & Harada, 2020), MetaReg (Balaji et al., 2018), RSC (Huang et al., 2020).
The results on PACS and Ofﬁce-Home are reported in Table 1 and Table 2 respectively.
Our DFAS achieves state-of-the-art results based on both ResNet18 and ResNet50 on both PACS (85.4%, 89.0%) and Ofﬁce-Home (64.3%, 70.3%), outperforming RSC by 0.3% and 1.2% on PACS based on ResNet18 and ResNet50 respectively, and by 1.2% on Ofﬁce-Home based on ResNet18 (these methods do not conduct experiment on
Ofﬁce-Home using ResNet50).
Compared with Baseline that directly aggregates three source domains to train learner with standard fully-connected layer as classiﬁer f c , our method of DFAS improves its performance by 5.0% and 5.7% on PACS based on ResNet18 and ResNet50 respectively, and by 2.1% and 2.1% on Ofﬁce-Home based on ResNet18 and ResNet50 respectively.
In Table 1, on PACS,
DFAS signiﬁcantly outperforms Baseline in almost all tasks except when P is taken as target domain.
Note that, in the task that domain S, of which the style is extremely different from rest three domains, is target domain, our DFAS boosts the accuracy of Baseline by 12.0% and 15.3% based on ResNet and ResNet50 respectively.
This indicates that our method can generalize well when domain shift is large.
Ofﬁce-Home is challenging for DG since the number of classes is larger than other datasets.
As shown in Table 2, our DFAS outperforms Baseline stably in almost all tasks on Ofﬁce-Home.
These performance improvements demonstrate the effectiveness of our method in the case that training data are from multi-source domains and the unseen target domain is different from source domains.
5. T YPE II: S INGLE S OURCE WITH D OMAIN S HIFT (SSDS)
We conduct this type of experiment on PACS based on ResNet18.
In this experiment, we train learner on one domain and test on each of the rest three domains, resulting in total 12 tasks.
Implementation details are shown in Appendix C. Our method is compared with related methods, including Baseline that directly trains learner with standard fully-connected layer as classiﬁer f c on the source domain,
Jien (Carlucci et al., 2019) and SagNet (Nam et al., 2019).
Results are reported in Table 3.
Our method of DFAS outperforms Baseline and SagNet by 4.2% and 4.7% respectively.
We observe that our method outperforms Baseline in 10 tasks among all 12 tasks.
The performance boosts are large in tasks when domain S is set to be source domain.
These performance improvements demonstrate the effectiveness of our method in the case that training data are from single source domain and the unseen target domain is different from source domain.
5. T YPE III: S AME S OURCE AND T ARGET D OMAIN (SSTD)
[Table]
We also apply our DG method to the common recognition task that the training and test data are from a same domain, i.e. , SSTD , on CIFARdataset.
To investigate the effect of training size, we sample different sizes of training data from the provided training set ( i.e. , source domain).
Implementation details are in
Appendix C. As shown in Table 4, our DFAS outperforms Baseline, JiGen and MMLD by 2.4%, 3.6% and 4.3% respectively on average.
The results of JiGen and MMLD are obtained by running their codes on CIFAR-10.
We observe that DFAS outperforms Baseline and compared methods in all different numbers of training data.
In general, the performance boost is larger when the number of training data is smaller.
This may be because the learner is more possible to be overﬁtting when the training size is smaller and our DFAS is designed to extract better generalizable features.
5. A BLATION STUDY
[Table]
We examine each component of our method on PACS based on ResNet18 under MSDS setting.
Results are reported in Table 5, in which the ﬁrst row lists the test domain, and L 2 means L 2 -normalization as discussed in Sect.
The accuracy boost (2.4%) over Baseline achieved by Baseline (w/ L 2 ) shows that L 2 -normalization is beneﬁcial to performance improvement.
DFAS-1 denotes our method DFAS using gradient clip to mitigate gradient explosion instead of L 2 -normalization.
DFAS-1 outperforms Baseline by 3.2%, lower than DFAS using L 2 -normalization.
DFAS-2 denotes the DFAS using randomly selected train/val splitting, instead of using the adversarial splitting optimization in Eq.
DFAS-3 ﬁnds the hardest val-subset only based on loss by setting α = 0 in Eq.
(5), instead of using both loss and gradient.
DFAS-2 and DFAS-3 degrade the performance of DFAS by 1.1% and 0.4% respectively, indicating that our adversarial splitting model is effective.
Due to space limit, we add more ablation experiments in Appendix A and B.2 to further verify the effectiveness of the proposed method.
C ONCLUSION
In this paper, we unify adversarial training and meta-learning in a novel proposed Domain-Free Adversarial Splitting (DFAS) framework to tackle the general domain generalization problem.
Extensive experiments show the effectiveness of the proposed method.
We are interested in deeper theoretical understanding and more applications of our method in the future work.
A A DDITIONAL A BLATION S TUDY
To further verify the effectiveness of each component of our method we conduct additional ablation experiments on PACS dataset based on ResNet18 in both MSDS and SSDS setting. The results are reported in Table 6 and Table 7. [Table]
[Table]
In Table 6 and 7, L 2 -norm denotes feature L 2 -normalization deﬁned in Sect.3.3. Rand-split means the random splitting strategy that we randomly split the train/val subsets at each step of updating the Effectiveness of L 2 -normalization. In Table 6, L 2 -norm (82.7%) outperforms Baseline (80.4%) by 2.3% and L 2 -norm + Adv-split (i.e., DFAS) (85.4%) outperforms Adv-split (83.6%) by 1.8% in MSDS setting.
In Table 7, L 2 -norm (63.1%) outperforms Baseline (62.4%) by 0.7% and L 2norm + Adv-split (i.e., DFAS) (66.6%) outperforms Adv-split (65.1%) by 1.5% in SSDS setting. These performance improvements demonstrate that the feature L 2 -normalization is useful in domain generalization. Effectiveness of adversarial splitting model.
In Table 6, Adv-split (83.6%) outperforms Baseline (80.4%) by 3.2% and L 2 -norm + Adv-split (i.e., DFAS) (85.4%) outperforms L 2 -norm (82.7%) by 2.7% in MSDS setting. In Table 7, Adv-split (65.1%) outperforms Baseline (62.4%) by 2.7% and L 2 -norm + Adv-split (i.e., DFAS) (66.6%) outperforms L 2 -norm (63.1%) by 3.5% in SSDS setting. These results indicating that our proposed adversarial splitting model is effective.
Effectiveness of adversarial splitting over random splitting. In Table 6, L 2 -norm + Adv-split (85.4%) outperforms L 2 -norm + Rand-split (84.3%) by 1.1% and Adv-split (83.6%) outperforms Rand-split (82.8%) by 0.8% in MSDS setting. In Table 6, L 2 -norm + Adv-split (66.6%) outperforms
L 2 -norm + Rand-split (65.4%) by 1.2% and Adv-split (65.1%) outperforms Rand-split (64.1%) by 1.0% in SSDS setting. These results demonstrate that the adversarial splitting model outperforms the random splitting strategy in different experimental settings. B A NALYSIS B. M ITIGATING G RADIENT E XPLOSION BY L 2 - NORMALIZATION . To show that L 2 -normalization can mitigate gradient explosion, we conduct the same experiments independently for 50 times respectively with L 2 -normalization and without L 2 -normalization.
Then we count the numbers of occurrences of gradient explosion that are reported in Table 8.
From Table 8, we can observe that L 2 -normalization can mitigate gradient explosion.
[Table]
In the section, we compare our adversarial splitting with domain-label-based splitting that is commonly used in meta-learning-based DG methods. Due to variations of style, poses, sub-classes, etc., the internal inconsistency within dataset is complicated. Domain-label partially capture the inconsistency, while cannot cover all possible internal inconsistency.
Our adversarial splitting method does not rely on the domain label. It iteratively ﬁnds the hardest train/val splitting to the learner to maximize the inconsistency and train the learner to generalize well for the hardest splitting, in an adversarial training way. This strategy more ﬂexibly investigates the possible inconsistency within training dataset, adaptively to the learner, and can potentially enhance the generalization ability of learner. [Table]
We ﬁrst empirically show that the domain-label-based splitting (denoted as Label-split) is not as hard as our adversarial splitting (Adv-split) to the learner in Table 9.
In Table 9, we report the values of objective function in Eq.
(5) of Adv-split and Label-split by ﬁxing the learner with different network
[Figure]
parameters w i at different epoch (1th, 2th, 5th and 10th) in the training process. Larger values in the table indicates that the splitting is harder to the learner (i.e., network). It can be observed that the domain-label-based splitting (Label-split) is not as hard as Adv-split to learner. We also conduct experiments on PACS in MSDS setting to fairly compare different splittings, including adversarial splitting (Adv-split), domain-label-based splitting (Label-split) and random splitting (Rand-split).
The results are reported in Table 10. Table 10 shows that adversarial splitting outperforms random splitting and domain-label-based splitting when training data are from multiple domains. [Table] When the training data are from only a single domain, our adversarial splitting also performs well (as in Table 3).
However, domain-label-based splitting cannot be used in this setting, since there is no domain label available.
Effect of hyper-parameter ξ . In Fig. 1(a), we show the performance of our method when varying the hyper-parameter ξ , i.e., length of the val-subset S v in adversarial splitting of training dataset.
The best result is obtained when ξ = | S | 2 , and the results are similar when ξ | S | ranges from 0.3 to 0.7.
Effect of hyper-parameter α . We evaluate the effect of α in MSDS setting on PACS dataset in Fig. 1(b).
From Fig. 1(b), the ACC is stable to the values of α in large range of 1e-6 to 1e-4. Small α results in small step-size for parameter updating in meta-learning framework, and limits the beneﬁts from meta-learning and adversarial splitting. Larger α results in larger step-size for gradient descent based network updating, which may fail to decrease the training loss from the optimization perspective. Effect of hyper-parameter m .
The effect of m is evaluated in MSDS setting on PACS dataset in Fig. 1(c). Fig. 1(c) shows that the result is not sensitive to the value of m .
We testify the convergence of DFAS with errors and losses in different tasks in Fig. In Fig. 2(a) and Fig. 2(b)
, we show the classiﬁcation error curves on target domains (A and Ar respectively) in the setting of MSDS . In Fig. 2(c), we show the training losses in task A in MSDS setting. These training curves indicates that DFAS converges in the training process. We also observe that DFAS has better stability than Baseline, in Fig. 2(a) and Fig. 2(b).
[Figure]
We compare the computational cost of adversarial splitting and random splitting (its meaning is discussed in Appendix A) in this section.
Since we only update the worst-case splitting per epoch, instead of at each step of updating parameters, the computational cost is only slightly higher than that of random splitting. To show this, we compare the total training times of the adversarial splitting and random spitting in the same number of steps (20000), as Table 11. [Table] From Table 11, the training time of Adv-split is only 5.6% (0.33/5.90) higher than Rand-split
[Table]
We visualize the feature space learned by our method of DFAS and Baseline (shown in Fig. 3), by t-SNE (Maaten & Hinton, 2008).
It appears that DFAS yields better separation of classes and better alignment of distributions of source and unseen target domains, which possibly explains the accuracy improvements achieved by our DFAS.
C I MPLEMENTATION D ETAILS
For the setting of MSDS , we use ResNet18 and ResNet50 (He et al., 2016) pre-trained on ImageNet (Russakovsky et al., 2015).
For each of them, the last fully-connected layer is replaced by a bottleneck layer, then the corresponding network is taken as feature extractor f e .
The dimension of the bottleneck layer is set to be 512 when the backbone is ResNet18 as in (Saito et al., 2019), and 256 when the backbone is ResNet50 as in (Gu et al., 2020).
Following (Gu et al., 2020), s is set to 7.5 for PACS and 10.0 for Ofﬁce-Home.
m is set to 0.2 for PACS and 0.1 for Ofﬁce-Home.
ξ is set to | S | 2 .
SGD with momentum of 0.9 is utilized to update parameters of learner.
The learning rate of classiﬁer and bottleneck layer is 10 times of convolutional layers, which is widely adopted in domain adaptation (Long et al., 2015; Ganin et al., 2016).
Following (Ganin et al., 2016), the learning rate of convolutional layer is adjusted by η = 0 .
001 (1+10 p ) 0 .
75 , where p is the optimizing progress linearly changing from 0 to 1.
The learning rate α of inner loop optimization is set to 10 − 5 .
The parameters are updated for 20000 steps and the hardest val-subset is updated per 200 steps.
The batchsize is set to 64.
The running mean and running variance of Batch Normalization (BN) layers are ﬁxed as the pre-trained values on ImageNet during training, which is discussed in (Du et al., 2020a).
Due to memory limit, when implementing experiments based on ResNet50, we adopt the ﬁrst order approximation (Finn et al., 2017) that stops the gradient of g tw in Eq.
(4) for reducing memory and computational cost. [Figure]
For the setting of SSDS , we conduct experiment based on ResNet18 on PACS.
The implementation details are same as MSDS .
For the setting of SSTD , we conduct experiment based on ResNet18 on
CIFAR-10.
The hyper-parameters of s and m are set to 8.0 and 0.2 respectively.
Other implementation details are same as MSDS except that, in BN layers, the running mean and running variance are updated.
We implement experiments using Pytorch (Paszke et al., 2019) on a single NVIDIA Tesla P100 GPU.
D O PTIMIZATION A LGORITHM FOR F INDING THE H ARDEST S v D. O PTIMIZATION A LGORITHM
To solve the problem of
[Equation]
(11) we design an alternative iteration algorithm in Sect. 3.2.
Speciﬁcally, we initialize A with gradient of a sample randomly selected from S .
Then we alternatively update S v and A .
Given A , S v is updated by solving
[Equation]
where the constraints are derived from the deﬁnition of Γ ξ .
Equation (12) indicates that the optimal
S v consists of ξ samples that have the largest values of l ( f w ( x ) , y )
− α (cid:104)∇ w l ( f w ( x ) , y ) , A (cid:105) .
Thus we compute and rank the values of l ( f w ( x ) , y )
− α (cid:104)∇ w l ( f w ( x ) , y ) , A (cid:105) for all ( x, y ) ∈ S and select the largest ξ samples to constitute the S v .
Given S v ( S t is then given), we update A to satisfy the constraint A = g tw in Eq.
(11), then A is
[Equation]
D. C ONVERGENCE IN E XPERIMENTS
We show empirically the convergence of this alternative iteration algorithm in Fig.
4, with the values of objective function in Eq.
Figure 4 shows that the values of objective function converges after only a few iterations.
[Figure] show the values of objective function in Eq.
(11) in two different runs with different initializations.
We also check if the splitting changes when the value of objective function converges. To do this, we count the ratio of changed sample indexes in S v at each iteration, as in Table 12. Table 12 shows that the splitting is not changed when the value of objective function converges.
[Table]
E P ROOF OF T HEOREM
We ﬁrst introduce VC-dimension based generalization bound and domain adaptation theory in
Appendix E.1, then present two lemmas in Appendix E.2, and ﬁnally give the proof of Theorem 1 in
Appendix E.3.
E. P RELIMINARY
VC-dimension based generalization bound.
Theorem A-1.
(Abu-Mostafa et al., 2012) Let S be the set of training data i.i.d.
sampled for distribution P .
For any δ ∈ (0 , 1) , with probability at least 1 − δ , we have ∀ h
( h : X → { 0 , 1 } ) in hypothesis space H ,
[Equation]
where (cid:15) P ( h ) =
E ( x,y ) ∼P
[ I { ( h ( x )) (cid:54) =
y } ] and ˆ (cid:15) S ( h ) =
1 | S | (cid:80) (
x,y ) ∈ S I { ( h ( x )) (cid:54) = y } .
Domain adaptation theory.
Theorem A-2.
(Ben-David et al., 2007; 2010)
For any h in hypothesis space H , we have
[Equation]
(15) where λ ∗ ≥ inf h (cid:48) ∈H { (cid:15) P ( h (cid:48) )
+ (cid:15) Q ( h (cid:48) ) } and
[Equation]
(16) is H -divergence.
E. L EMMAS
Lemma A-1.
For any S v ∈ Γ ξ and S t = S − S v , ∀ δ ∈ (0 , 1) , with probability at least 1 − δ , we have ∀ f ∈ H S t ,
[Equation]
where (cid:15) Ψ P ( f ) =
E ( x,y ) ∼P
[ I { Ψ( f ( x )) (cid:54) =
y } ] is generalization error on distribution P , ˆ (cid:15)
Ψ S v ( f ) =|
S v | (cid:80) ( x,y ) ∈ S v I { Ψ( f ( x ))
(cid:54) =
y } is empirical error,
H Ψ S t
=
{ Ψ ◦ f : f ∈ H S t } , V C
( H Ψ S t ) is the
VC-dimension of H Ψ S t , and Ψ( · ) is the prediction rule such as the Bayes Optimal Predictor, i.e., Ψ( f ( x ))
=
I { f ( x ) ≥ 12 } .
Proof:
From the deﬁnition of H Ψ S t , for any f ∈ H S t , there exists a h f ∈ H Ψ S t such that h f = Ψ ◦ f .
Applying Theorem A-1, with probability at least 1 − δ , we have ∀ f ∈ H S t ,
[Equation]
Lemma A-2.
For any S v ∈ Γ ξ and S t = S − S v , let (cid:15) Ψ P ( g ) =
inf f ∈H St (cid:15) Ψ P ( f ) and ˆ (cid:15)
Ψ S v ( h ) =
inf f ∈H St ˆ (cid:15) Ψ S v ( f ) , then ∀ δ ∈ (0 , 1) , with probability at least 1 − δ , we have
[Equation]
Proof:
From the deﬁnition of g and h
[Equation]
we have ˆ (cid:15) Ψ S v ( g ) [Equation]
ˆ (cid:15) Ψ S v ( h ) .
∀ δ ∈ (0 , 1) , with probability at least 1 − δ , we have
[Equation]
Thus, Eq.
(19) holds.
E. P ROOF OF T HEOREMProof
We denote by H Ψ l the hypothesis space such that ∀ h ∈ H
Ψ l , [Equation]
otherwise , (21) for f ∈ H .
Then
[Equation]
In the fourth equation [Equation] we utilize the assumption that E Q
[ I { l ( f ( x
[Equation] ,y [Equation] >γ } ] ≥ E P [ I { l ( f ( x ) ,y ) >γ } ] .
Given any S v ∈ Γ ξ and S t = S − S v [Equation] we replace H by H S t , then
[Equation]
23) where C 1 = sup S (cid:48) v ∈ Γ ξ sup f ∈H S − S (cid:48) v E Q [ I { l ( f ( x ) ,y ) >γ } ] .
Applying Theorem A- [Equation] for any f ∈ H S t , we have
[Equation]
24) where λ ∗ ( S v )
[Equation]
Let C
[Equation]
sup S
[Equation]
∈ Γ ξ inf f [Equation] ∈H S
[Equation]
S (cid:48) v { (cid:15)
Ψ l P ( f (cid:48) )
+ (cid:15)
Ψ l Q ( f (cid:48) ) } , we have
[Equation]
Applying Lemma A-1 to the ﬁrst term of right side in Eq.
(25), ∀ δ ∈ (0 , 1) , with probability at least 1 − δ , we have ∀ f ∈ H S t , [Equation]
[Equation]
Applying Lemma A-2 to the third term of right side in Eq.
(25), ∀ δ ∈ (0 , 1) , with probability at least 1 − δ , we have
[Equation]
Combining Eq.
(25), (26), (27) and thanks to the union bound, for any δ ∈ (0 , 1) , with probability at least 1 − 2 δ , we have ∀ f ∈ H S t ,
[Equation]
where B ( S v ) =
C 1 − inf f (cid:48)
[Equation]
S v | [Equation] ( x,y ) ∈ S v I { l
[Equation] f (cid:48) ( x ) ,y ) >γ } .
Using the fact that | S v | = ξ and let
C 2 = sup S
[Equation]
C (
[Equation]
) , we have
[Equation]