A C LOSER L OOK AT S MOOTHNESS IN D OMAIN A DVERSARIAL T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Domain adversarial training has been ubiquitous for achieving invariant represen - tations and is used widely for various domain adaptation tasks.
In recent times, methods converging to smooth optima have shown improved generalization for supervised learning tasks like classiﬁcation.
In this work, we analyze the effect of smoothness enhancing formulations on domain adversarial training, the objective of which is a combination of task loss (eg. classiﬁcation, regression etc.) and adversarial terms.
In contrast to task loss, our analysis shows that converging to smooth minima w.r t. adversarial loss leads to sub-optimal generalization on the target domain .
Based on the analysis, we introduce the Smooth Domain Adversarial training (SDAT) procedure, which effectively enhances the performance of existing domain adversarial methods for both classiﬁcation and object detection tasks.
Our smoothness analysis also provides insight into the extensive usage of SGD over Adam in domain adversarial training
I NTRODUCTION
Domain Adversarial Training (Ganin & Lempitsky, 2015) (DAT) refers to adversarial learning of neural network based feature representations that are invariant to the domain.
For example, car images from the clipart domain have similar feature representations as car images from the web domain.
DAT has been widely useful in diverse areas (cited 3540 times) such as fairness (Adel et al., 2019), object detection (Saito et al., 2019), domain generalization (Li et al., 2018), imageto-image translation (Liu et al., 2017) etc.
The prime driver of research on DAT is its application in unsupervised Domain Adaptation (DA), which aims to learn a classiﬁer using labeled source data and unlabeled target data, such that it generalizes well on target data.
Various enhancements like superior objectives (Acuna et al., 2021; Zhang et al., 2019), architectures (Long et al., 2018) etc. have been proposed to improve its effectiveness.
However, as DAT objective is combination of
Generative Adversarial Network (GAN) (Goodfellow et al., 2014) and Empirical Risk Minimization (ERM) (Vapnik, 2013) objectives, there has not been much focus on explicitly analyzing the nature of optimization in DAT.
One direction of work aiming to improve generalization of ERM on unseen data focuses on developing algorithms that converge to a smooth (or a ﬂat) minima (Foret et al., 2021;
Keskar & Socher, 2017).
However, we ﬁnd that these techniques, when directly applied for
DAT, do not signiﬁcantly improve the generalization on the target domain (Sec. 4 and 7).
In this work, we analyze the loss landscape near the optimal point obtained by DAT, to gain insights into curvature.
We ﬁrst focus on the eigen-spectrum of Hessian of the task loss (ERM term for classiﬁcation) where we ﬁnd that using Stochastic Gradient Descent (SGD) as optimizer converges to a smoother minima in comparison to Adam (Kingma & Ba, 2014).
Further we ﬁnd that smoother minima w.r.t.task loss leads to better generalization on the target domain .
Contrary to task loss, we ﬁnd that smoothness enhancing formulation for adversarial components worsen performance, rendering ERM-based techniques which enhance smoothness for all loss components ineffective.
Hence we introduce Smooth Domain Adversarial Training (SDAT), which aims only to reach a smooth minima w.r.t.
task loss, and helps in generalizing better on the target domain.
SDAT requires an additional gradient computation step and can be combined with existing methods with a few lines of code.
We show the soundness of the SDAT method theoretically by proving a generalization bound (Sec. 4) on target error.
We extensively verify the empirical efﬁcacy of SDAT across various datasets for classiﬁcation (i.e., DomainNet, VisDA-2017 and Ofﬁce-Home), along with showing a prototypical application in DA for object detection, demonstrating it’s diverse applicability.
In summary, we make the following contributions:
[Figure]
• We analyze the optimization procedure of DAT, establishing the correlation between the smoothness near optima w.r.t.
task loss and generalization on the target domain.
• Contrary to ERM, we show through our theoretical and empirical analysis that smoothness enhancing adversarial formulation leads to sub-optimal performance.
• For enhancing the smoothness w.r.t.
task loss near optima in DAT, we propose a novel and theoretically motivated SDAT that improves the generalization on the target domain.
SDAT effectively increases the average performance of even state-of-the-art adversarial adaptation methods.
R ELATED W ORK
Unsupervised Domain Adaptation : It refers to a class of methods that aim to adapt models to work in a target domain distinct from what it was trained on.
One of the most prominent lines of work is based on DAT (Ganin & Lempitsky, 2015).
This involves using an additional discriminator to distinguish between samples of source and target domain.
The goal of the model is to learn features that can not be distinguished between source and target.
The follow-up works have improved this basic idea by introducing a class information based discriminator (CDAN (Long et al., 2018)), introducing a transferable normalization function (Wang et al., 2019) etc.
In this work, we focus on analyzing and improving such methods.
Another line of work involves DA by using self-training on target domain (Kundu et al., 2020b;a; Prabhu et al., 2020) which will not be the focus of this work.
Smoothness of Loss Landscape:
As neural networks operate in the regime of over parameterized models, low error on training data does not always lead to better generalization (Keskar et al., 2017).
Often it has been stated (He et al., 2019; Dziugaite & Roy, 2017) that smoother minima does generalize better on unseen data.
But until recently, this was practically expensive as smoothing required additional costly computations.
Recently, a method called Sharpness Aware Minimization (SAM) (Foret et al., 2021) has been proposed to ﬁnd a smoother minima with an additional gradient computation step. SAM also improves the ImageNet model performance (Chen et al., 2021) on ImageNet-C and ImageNet-R (which are out of distribution). It has also been observed that smoothness w.r.t.
input (image) is beneﬁcial for domain adaptation (Shu et al., 2018;
Cai et al., 2021), which motivates us to explore smoothness w.r.t weights (W) in case of DAT.
However, the earlier work has focused on achieving a smoother minima w.r.t. W for ERM.
Currently, no study has been done if the loss function is composed of both ERM and adversarial objectives (as present in DAT).
B ACKGROUND 3. P RELIMINARIES
We will primarily focus on Unsupervised DA where we have labeled source data S = { ( x si , y si ) } and unlabeled target data T = { ( x ti ) } .
The source samples are assumed to be sampled i.i.d.
from source distribution P S deﬁned on input space X , similarly target samples are sampled i.i.d.
from P T .
Y is used for denoting the label set which is { 1 , 2 , . . .
, k } in our case as we perform multiclass ( k ) classiﬁcation.
We denote y :
X → Y a mapping from images to labels.
Our task is to ﬁnd a hypothesis function h θ that has a low risk on the target distribution.
The source risk (a.k.a expected error) of the hypothesis h θ is deﬁned with respect to loss function l as: R lS ( h θ ) =
E x ∼ P S
[ l ( h θ ( x ) , y ( x ))] .
The target risk R lT ( h θ ) is deﬁned analogously.
The empirical versions of source and target risk will be denoted by ˆ R lS
( h θ ) and ˆ R lT ( h θ ) .
All notations used in paper are summarized in Table F.
In this work we build on the domain adaption theory of (Acuna et al., 2021) which is a generalization of Ben-David et al.
We ﬁrst deﬁne the discrepancy between the two domains.
Deﬁnition 3.1 ( D φh θ , H discrepancy) .
The discrepancy between two domains P S and P T is deﬁned as following:
[Equation]
Here φ ∗ is a frenchel conjugate of a lower semi-continuous convex function φ that satisﬁes φ (1) = 0 , and H is the set of all possible hypothesis (i.e. Hypothesis Space).
This discrepancy distance D φh θ , H is based on variational formulation of f-divergence (Nguyen et al., 2010) for the convex function φ .
The D φh θ
, H is the lower bound estimate of the f-divergence function
D φ ( P S || P T ) .
See Lemma 4 in (Acuna et al., 2021) for additional details.
We state a bound on target risk R lT ( h θ ) based on D φh θ , H discrepancy (Acuna et al., 2021):
Theorem 1 ( Generalization bound ) .
Suppose l : Y × Y → [0 , 1] ⊂ dom φ ∗ .
Let h ∗ be the ideal joint classiﬁer with least λ ∗ =
R lS ( h ∗ )
+ R lT ( h ∗ ) (i.e. joint risk) in H .
We have the following relation between source and target risk:
[Equation]
The above generalization bound shows that the target risk R lT ( h θ ) is upper bounded by the source risk R lS ( h θ ) and the discrepancy term D φh θ , H along with an irreducible constant error λ ∗ .
Hence, this infers that reducing source risk and discrepancy lead a to reduction in target risk.
Based on this, we concretely deﬁne the unsupervised adversarial adaptation procedure in the next section.
3. U NSUPERVISED D OMAIN A DAPTATION
In this section we ﬁrst deﬁne the components of the framework we use for our purpose: h θ =
f
Θ ◦ g ψ where g ψ is the feature extractor and f Θ is the classiﬁer.
The domain discriminator D Φ , used for estimating the discrepancy between P S and P T is a classiﬁer whose goal is to distinguish between the features of two domains.
For minimizing the target risk (Th. 1), the optimization problem can be written as:
[Equation]
The discrepancy term under some assumptions (refer App. B) can be upper bounded by a tractable term:
[Equation]
where d Φ S,T = E x ∼ P S [log( D Φ ( g ψ ( x )))]
+ E x ∼ P T log[1 − D Φ ( g ψ ( x ))] .
This leads to the ﬁnal optimization objective of:
[Equation]
The ﬁrst term in practice is empirically approximated by using ﬁnite samples
ˆ R lS
( h θ ) and used as task loss (classiﬁcation) for minimization.
The empirical estimate of the second term is adversarial loss which is optimized using a gradient reversal layer (GRL) as it has a min-max form.
(Overview in Fig.
The above procedure composes DAT, and we use CDAN (Long et al., 2018) as our default
DAT method.
[Figure]
A NALYSIS OF S MOOTHNESS
In this section, we analyze the curvature properties of the loss with respect to the parameters.
Specifically, we focus on analyzing the Hessian of empirical source risk H = ∇ 2 θ ˆ R lS ( h θ ) which is the Hessian of classiﬁcation (task) loss term.
For quantifying the smoothness, we measure the trace
Tr ( H ) and maximum eigenvalue of Hessian ( λ max ) as a proxy for quantifying smoothness.
This is motivated by analysis of which states that the high value of λ max and Tr ( H ) are indicative of low smoothness (Jastrzebski et al., 2020).
We articulate our conjecture informally below:
Conjecture 1.
Smoothing of empirical source risk (i.e. task loss) ˆ R lS ( h θ ) leads to efﬁcient DAT.
In other words, decreasing λ max of ∇ 2 θ ˆ R lS ( h θ ) leads to reduced error on target domain ˆ R lT ( h θ ) .
For verifying our conjecture, we analyze the eigen spectrum of the Hessian ˆ R lT ( h θ ) where we ﬁnd that in contrast to standard ERM (Ghorbani et al., 2019)
the negative eigenvalues do not disappear as the training progresses.
We show the λ max , Tr ( H ) and eigen spectrum for different algorithms, namely DAT w/ Adam, DAT w/ SGD and our proposed SDAT (which is described in detail in later sections) in Fig.
We ﬁnd that high smoothness leads to better generalization on the target domain .
We also provide additional results in Fig.
3 for empirical veriﬁcation of the conjecture.
Our conjecture also explains the reason for widespread usage of SGD for DAT as SGD converges to smoother minima (Ganin & Lempitsky, 2015;
Long et al., 2018;
Saito et al., 2018a) which leads to efﬁcient DAT, even though Adam has shown to be effective for min-max optimization (Gemp & McWilliams, 2019).
More details regarding the Hessian analysis are provided in the supplementary (refer App. D).
4. S MOOTHING L OSS L ANDSCAPE
In this section we ﬁrst introduce the losses which are based on Sharpness Aware Minimization (Foret et al., 2021)
(SAM).
The basic idea of SAM is to ﬁnd a smoother minima (i.e. low loss in (cid:15) neighborhood of θ ) by using the following objective given formally below:
[Equation]
Here L obj is any objective function to be minimized and ρ ≥ 0 is a hyperparameter which deﬁnes the maximum norm of the (cid:15) .
Since ﬁnding the exact solution of inner maximization is hard, SAM maximizes the ﬁrst order approximation:
[Equation]
The ˆ (cid:15) ( θ ) is added to the weights θ .
The gradient update for θ is then computed as ∇ θ L obj ( θ )
| θ +ˆ (cid:15) ( θ ) .
The above procedure can be seen as a generic smoothness enhancing formulation for any L obj .
We now analogously introduce the sharpness aware source risk for ﬁnding a smooth minima:
[Equation]
[Figure]
We also now deﬁne the sharpness aware discrepancy estimation objective below:
[Equation]
As d Φ S,T is to be maximized the sharpness aware objective will have min || (cid:15)
||≤ ρ instead of max || (cid:15)
||≤ ρ , as it needs to ﬁnd smoother maxima.
We now theoretically analyse the difference in discrepancy estimation for smooth version d Φ (cid:48)(cid:48) S,T (Eq. 9) in comparison to non-smooth version d Φ (cid:48) S,T (Eq. 4).
Assuming
D Φ is a L -smooth (common assumption for non-convex optimization (Carmon et al., 2020)), η is a small constant and d ∗ S,T the optimal discrepancy, the theorem states:
Theorem 2.
For a given classiﬁer h θ and one step of (steepest) gradient ascent i.e. Φ (cid:48) = Φ + η
[Equation] ∇ d [Equation] S
[Equation] T [Equation] ||∇ d
[Equation]
S,
[Equation]
|| ) and Φ (cid:48)(cid:48) =
Φ + [Equation] [Equation] where α is the angle between ∇ d Φ S,T and ∇ d Φ S,T | Φ+ˆ (cid:15) (Φ) .
The d Φ (cid:48) S,T (non-smooth version) can exceed d Φ (cid:48)(cid:48) S,T (smooth discrepancy) signiﬁcantly, as the term d ∗ S,T − d Φ S,T (cid:54)→ 0 , as the h θ objective is to oppose the convergence of d Φ S,T to optima d ∗ S,T (min-max training in Eq. 11).
Thus d Φ (cid:48) S,T can be a better estimate of discrepancy in comparison to d Φ (cid:48)(cid:48) S,T .
A better estimate of d Φ s,t helps in effectively reducing the discrepancy between P S and P T , hence leads to reduced R lT ( h θ ) .
This is also observed in practice that smoothing the discriminator (SDAT w/ adv in Fig. 3) leads to low domain classiﬁcation accuracy (proxy measure for d Φ s,t ) in comparison to
DAT.
Due to ineffective discrepancy estimation, SDAT w/ adv results in sub-optimal generalization on target domain i.e. high target error R lT ( h θ ) (Fig. 3).
For further establishing the generality of sub-optimality of smooth adversarial loss, we also perform experiments on Spectral Normalised
Generative Adversarial Networks (SNGAN) (Miyato et al., 2018).
In case of SNGAN we also ﬁnd that smoothing discriminator through SAM leads to suboptimal performance (higher FID) as in Fig.
The above evidences indicates that smoothing the adversarial loss leads to sub-optimality , hence it should not be done in practice.
The proof of the above theorem and additional experimental details are provided in the supplementary (refer App.
C and App. E).
4. S MOOTH D OMAIN A DVERSARIAL T RAINING (SDAT)
We propose smooth domain adversarial training which only focuses on converging to smooth minima w.r.t.
task loss (i.e. empirical source risk), whereas does no change for the discrepancy term.
We deﬁne the optimization objective of our smooth domain adversarial training below:
[Equation]
The ﬁrst term is the sharpness aware risk, and the second term is the discrepancy term which is not smooth in our procedure.
The term d Φ S,T estimates D φh θ ,H ( P S || P T ) discrepancy.
We empirically ﬁnd that this optimization procedure effectively reduces the generalization error on the target domain compared to all other alternatives.
We now show that optimizing Eq.
11 reduces R lT ( h θ ) by providing a generalization bound.
Theorem 3.
Suppose l is the loss function, we denote λ ∗ :
= R lS ( h ∗ ) + R lT ( h ∗ ) and let h ∗ be the ideal joint hypothesis:
[Equation]
where γ : R + → R + is a strictly increasing function.
The bound is similar to generalization bounds for domain adaptation (Ben-David et al., 2010; Acuna et al., 2021).
The main difference is the sharpness aware risk term max || (cid:15) ||≤
ρ ˆ R lS ( h θ ) in place of source risk R l S ( h θ ) , and an additional term that depends on the norm of the weights γ ( || θ
|| 2 2 /ρ 2 ) .
The ﬁrst is minimized by decreasing the empirical sharpness aware source risk by using SAM loss shown in Sec. 4.
The second term is reduced by decreasing the discrepancy between source and target domains.
The third term, as it is a function of norm of weights
||
θ
|| 22 , can be reduced by using either L2 regularization or weight decay.
Since we assume that the H hypothesis class we have is rich, the λ ∗ term is small.
We now show the improvements due to SDAT empirically in the following sections.
A DAPTATION FOR CLASSIFICATION
We evaluate our proposed method on three datasets: Ofﬁce-Home, VisDA-2017, and DomainNet, as well as by combining SDAT with two DAT based DA techniques: CDAN and CDAN+MCC.
5. D ATASETS
Ofﬁce-Home (Venkateswara et al., 2017): Ofﬁce-Home consists of around 15,500 images fromclasses and four distinct domains: Art (Ar), Clipart (Cl), Product (Pr) and Real World (Rw).
VisDA-2017 (Peng et al., 2017): VisDA is a dataset that focuses on the transition from simulation to real world and contains approximately 280K images across 12 classes.
DomainNet (Peng et al., 2019):
DomainNet consists of 0.6 million images across 345 classes belonging to six domains.
The domains are infograph (inf), clipart (clp), painting (pnt), sketch (skt), real and quickdraw.
5. D OMAIN A DAPTATION M ETHODS
CDAN (Long et al., 2018): Conditional Domain Adversarial network is a popular DA algorithm that improves the performance of the DANN algorithm.
CDAN introduces the idea of multi-linear conditioning to align the source and target distributions better.
CDAN* in Table 1 and 4 refers to our implementation of CDAN method.
CDAN + MCC (Jin et al., 2020):
In this method, the minimum class confusion loss term is added as a regularizer to CDAN.
Minimum class confusion is a non-adversarial term that minimizes the pairwise class confusion on the target domain.
This achieves state of the art accuracy among adversarial adaptation methods.
5. I MPLEMENTATION D ETAILS
We implement our proposed method in the Transfer-Learning-Library (Junguang Jiang & Long, 2020) toolkit developed in PyTorch (Paszke et al., 2019).
The main difference between the performance reported in the CDAN and our implementation (CDAN*) is the batch normalization layer in the domain classiﬁer, which enhances performance.
We use a ResNet-50 backbone for Ofﬁce-Home experiments and a ResNet-101 backbone for VisDA2017 and DomainNet experiments.
The backbone is initialized with ImageNet weights.
We use a [Table] learning rate of 0.01 with batch size 32 in all of our experiments.
We tune ρ value in SDAT for a particular dataset and use the same value across domains.
The ρ value is set to 0.02 for the Ofﬁce-Home experiments, 0.005 for the VisDA-2017 experiments and 0.05 for the DomainNet experiments.
More details are present in supplementary (refer App. F).
5. R ESULTS
[Table]
Table 2 shows the results on the large and challenging DomainNet dataset across ﬁve domains as done in (Junguang Jiang & Long, 2020).
The proposed method improves the performance of CDAN signiﬁcantly across all source-target pairs.
On speciﬁc source-target pairs like infograph (cid:41) real, the performance increase is 4.5%.
The overall performance of
CDAN is improved by nearly 1.8% which is signiﬁcant considering the large number of classes and images present in DomainNet.
For the Ofﬁce-Home dataset, we compare our methods with other domain adaptation algorithms including DANN, SRDC, MDD and f-DAL.
The results for the Ofﬁce-Home dataset are shown in Table 1.
We can see that adding SDAT improves the performance on both CDAN and CDAN+MCC across all the transfer tasks.
CDAN+MCC w/ SDAT achieves state-of-the-art adversarial adaptation performance on the
Ofﬁce-Home dataset.
The class-wise accuracy on VisDA-2017 are reported in Table 4.
CDAN w/ SDAT improves the overall performance of CDAN by more than 1.5%.
CDAN w/ SDAT improves the performance of underperforming minority classes like bicycle and car.
Additional baselines and results are reported in supplementary (refer App. G).
A DAPTATION FOR OBJECT DETECTION
To further validate our approach’s generality and extensibility, we did experiments on DA for object detection.
We use the same setting as proposed in DA-Faster (Chen et al., 2018) with all domain adaptation components and use it as our baseline.
We use the mean Average Precision at 0.5 IoU (mAP) as our evaluation metric.
In object detection, the smoothness enhancement can be achieved in two ways:
a) DA-Faster w/ SDAT-Classiﬁcation: Smoothness enhancement for classiﬁcation loss.
b) DA-Faster w/ SDAT:
Smoothness enhancment for the combined classiﬁcation and regression loss.
Comparison of both ways is present in Sec. 6.2.
[Table]
6. E XPERIMENTAL S ETUP
We evaluate our proposed approach on object detection on two different domain shifts:
Pascal to Clipart ( P → C ): Pascal (Everingham et al., 2010) is a real-world image dataset which consists images with 20 different object categories.
Clipart (Inoue et al., 2018) is a graphical image dataset with complex backgrounds and has the same 20 categories as Pascal.
We use Resnet-
(He et al., 2016) backbone for Faster R-CNN (Ren et al., 2015) following Saito et al.
Cityscapes to Foggy Cityscapes ( C → Fc ): Cityscapes (Cordts et al., 2016) is a street scene dataset for driving, whose images are collected in clear weather.
Foggy Cityscapes (Sakaridis et al., 2018) dataset is synthesized from Cityscapes for the foggy weather.
We use Resnet-50 (He et al., 2016) as the backbone for Faster R-CNN for experiments on this task.
Both domains have the same 8 object categories with instance labels.
The training is done via SGD with momentum 0.9 for 70k iterations with the learning rate of 10 − 3 , and then dropped to 10 − 4 after 50k iterations.
We split the target data into train and validation sets and report the best mAP on validation data.
Additional experimental and implementation details are present in supplementary (refer App. F).
6. R ESULTS
Table 3 shows the results on two domain shifts with varying batch size ( bs ) during training.
We ﬁnd that only smoothing w.r.t.
classiﬁcation loss is much more effective (SDAT-Classiﬁcation) than smoothing w.r.t.
combined classiﬁcation and regression loss (SDAT).
On average, SDATClassiﬁcation produces an mAP gain of 2.0% compared to SDAT, and 2.8% compared to DA-Faster baseline.
[Table]
The proposed SDAT-Classiﬁcation signiﬁcantly outperforms DA-Faster baseline and improves mAP by 1.3% on P → C and by 2.8% on C → Fc .
It is noteworthy that increase in performance of SDAT-Classiﬁcation is consistent even after training with higher batch size ( bs = 8) achieving improvement of 4.3% in mAP.
Table 3 also shows that even DA-Faster w/ SDAT (i.e. smoothing both classiﬁcation and regression) outperforms DA-Faster by 0.9 % on average across all experiments.
The improvement due to SDAT on adaptation for object detection shows the generality of SDAT across techniques that have some form of adversarial component present in the loss formulation.
D ISCUSSION
How much smoothing is optimal?
: Figure 4 (A) shows the ablation on ρ value (higher ρ value corresponds to more smoothing) on the Ar (cid:41) Cl and Cl (cid:41) Pr from Ofﬁce-Home dataset with CDAN backbone.
The performance of the different values of ρ is higher than the baseline with ρ = 0. It Our
CDAN Refer Section 5.3.
[Figure]
Which components beneﬁt from smooth optima?
: Figure 4 (C) shows the effect of introducing smoothness enhancement for different components in DAT.
For this we use SAM on a) classiﬁer (SDAT) b) discriminator (SDAT w/ adv) c) both classiﬁer and discriminator (SDAT-all).
It can be seen that smoothing the adversarial component (SDAT w/ adv) reduces the performance to 51.0%, which is signiﬁcantly lower than even the DAT baseline.
Is it Robust to Label Noise? :
In practical, real-world scenarios, the labeled datasets are often corrupted with some amount of label noise.
Due to this, performing domain adaptation with such data is challenging.
We ﬁnd that smoother minima through SDAT lead to robust models which generalize well on the target domain.
Figure 4 (B) provides the comparison of SGD vs. SDAT for different percentages of label noise injected into training data (by ﬂipping the labels).
[Table]
Is it better than other smoothing techniques?
To answer this question, we compare SDAT with different smoothing techniques originally proposed for ERM.
We speciﬁcally compare our method against DAT, Label Smoothing (LS) (Szegedy et al., 2016), and VAT (Miyato et al., 2019).
Stutz et al.
(2021) recently showed that these techniques produce a significantly smooth loss landscape in comparison to SGD.
We also compare with a very recent
SWAD (Cha et al., 2021) technique which is shown effective for domain generalization.
For this, we run our experiments on four different splits of the Ofﬁce-Home dataset and summarize our results in Table 5.
We ﬁnd that techniques for ERM (LS and VAT) fail to provide signiﬁcant consistent gain in performance which also conﬁrms the requirement of speciﬁc smoothing strategies for DAT.
We ﬁnd that SDAT even outperforms SWAD on average by a signiﬁcant margin of 1.2%.
Additional details regarding the speciﬁc methods are provided in the supplementary (refer App. H).
C ONCLUSION
In this work, we analyse the curvature of loss surface of DAT used extensively for Unsupervised
Domain Adaptation.
We ﬁnd that converging to a smooth minima w.r.t.
task loss (i.e., empirical source risk) leads to better generalization on the target domain.
We also theoretically and empirically show that smoothness enhancing for adversarial components of loss lead to sub-optimal results, hence should be avoided in practice.
We then introduce our practical and effective method, SDAT, which only increases the smoothness w.r.t.
task loss, leading to better generalization on the target domain.
SDAT leads to an effective increase even for the state of the art methods for adversarial domain adaptation and can be incorporated with just a few lines of code change.
One limitation of SDAT is presence of no automatic way of selecting ρ (determines extent of smoothness) which is a good future direction to explore.
A PPENDICES A N OTATION T ABLE
Table S1 contains all the notations used in the paper and the proofs of theorems.
[Table]
B C ONNECTION OF D ISCREPANCY TO d Φ S,T (E Q . 4) IN M AIN P APER
We refer reader to Appendix C.2 of Acuna et al.
(2021) for relation of d Φ S,T .
The d Φ S,T term deﬁned in Eq.
4 given as:
[Equation]
The above term is exactly the Eq.
C.1 in Acuna et al.
(2021) where they show that optimal d Φ S,
T i
[Table]
e.:
[Equation]
(S2)
Hence we can say from result in Eq.
4 is a consequence of Lemma 1 and Proposition 1 in (Acuna et al., 2021), assuming that D Φ satisﬁes the constraints in Proposition 1.
C P ROOF OF T HEOREMS
In this section we provide proofs for the theoretical results present in the paper:
Theorem 1 ( Generalization bound ) .
Suppose l : Y × Y → [0 , 1] ⊂ dom φ ∗ .
Let h ∗ be the ideal joint classiﬁer with error λ ∗ =
R lS ( h ∗ )
+ R lT ( h ∗ ) .
We have the following relation between source and target risk:
[Equation]
Proof.
We refer the reader to Theorem 2 in Appendix B of Acuna et al.
(2021) for the detailed proof the theorem.
We now introduce a Lemma for smooth functions which we will use in the proofs subsequently:
Lemma 1.
For an L-smooth function f ( w ) the following holds where w ∗ is the optimal minima:
[Equation]
Proof.
The L-smooth function by deﬁnition satisﬁes the following:
[Equation]
Now we minimize the upper bound wrt v to get a tight bound on f ( w ∗ ) .
[Equation]
after doing ∇ v D ( v ) =
we get:
[Equation]
By substituting the value of v in the upper bound we get:
[Equation]
Hence rearranging the above term gives the desired result:
[Equation]
Theorem 2.
For a given classiﬁer h θ and one step of (steepest) gradient ascent i.e. Φ (cid:48) =
Φ + η ( ∇ d Φ S,T / ||∇ d Φ S,T || ) and Φ (cid:48)(cid:48) =
Φ + η ( ∇ d Φ S,T | Φ+ˆ (cid:15) (Φ) / ||∇ d Φ S,T | Φ+ˆ (cid:15) (Φ) || ) for maximizing
[Equation]
where α is the angle between ∇ d Φ S,T and ∇ d Φ S,T | Φ+ˆ (cid:15) (Φ) .
Proof of Theorem 2.
We assume that the function is L -smooth (the assumption of L-smoothness is the basis of many results in non-convex optimization (Carmon et al., 2020)) in terms of input x .
As for a ﬁxed h θ as we use a reverse gradient procedure for measuring the discrepancy, only one step analysis is shown.
This is because only a single step of gradient is used for estimating discrepancy d Φ S,T i.e. one step of each min and max optimization is performed alternatively for optimization.
After this the h θ is updated to decrease the discrepancy.
Any differential function can be approximated by the linear approximation in case of small η :
[Equation]
The dot product between two vectors can be written as the following function of norms and angle θ between those:
[Equation]
The steepest value will be achieved when cos θ = 1 which is actually v = ∇ d Φ S,T
[Equation]
||
[Equation]
Φ S,
[Equation]
x ) || .
Now we compare the descent in another direction v 2 =
[Equation]
| from the gradient descent.
The difference in value can be characterized by:
[Equation]
As α is an angle between ∇ d Φ S,T | w + (cid:15) ( w ) ( v 2 ) and ∇ d Φ S,T ( X ) ( v ) .
The suboptimality is dependent on the gradient magnitude.
We use the following result to show that when optimality gap d ∗ S,T − d Φ S,T ( x ) is large the difference between two directions is also large.
For an L-smooth function the following holds according to Lemma 1:
[Equation]
As we are performing gradient ascent f ( w ) = − d Φ s,t , we get the following result:
[Equation]
This shows that difference in value of by taking a step in direction of gradient v vs taking the step in a different direction v 2 is upper bounded by the d ∗ S,T − d Φ S,T ( x ) , hence if we are far from minima the difference can be potentially large.
As we are only doing one step of gradient ascent d ∗ S,T − d Φ S,T will be potentially large, hence can lead to suboptimal measure of discrepancy.
Theorem 3.
Suppose l is the loss function, we denote λ ∗ :
= R lS ( h ∗ ) + R l T ( h ∗ ) and let h ∗ be the ideal joint hypothesis:
[Equation]
where γ : R + → R + is a strictly increasing function.
Proof of Theorem 3: In this case we make use of Theorem 2 in the paper sharpness aware minimization (Foret et al., 2021) which states the following: The source risk R S ( h ) is bounded using the following PAC-Bayes generalization bound for any ρ with probability 1 − δ :
[Equation]
here n is the training set size used for calculation of empirical risk ˆ R S ( h ) , k is the number of parameters and
||
θ
|| 2 is the norm of the weight parameters.
The second term in equation can be abbreviated as γ ( || θ || 2 ) .
Hence,
[Equation]
From the generalization bound for domain adaptation for any f-divergence (Acuna et al., 2021) (Theorem 2) we have the following result.
[Equation]
Combining the above two inequalities gives us the required result we wanted to prove i [Equation] e.
[Equation]
D H ESSIAN A NALYSIS
We use the PyHessian library (Yao et al., 2020) to calculate the Hessian eigenvalues and the Hessian
Eigen Spectral Density.
All the calculations are performed using 50% of the source data at the last checkpoint.
Only the source class loss is used for calculating to clearly illustrate our point.
The partition was selected randomly, and the same partition was used across all the runs.
We also made sure to use the same environment to run all the Hessian experiments.
A subset of the data was used for Hessian calculation mainly because the hessian calculation is computationally expensive (Yao et al., 2020). This is commonly done in hessian experiments. For example, (Chen et al., 2021) (refer Appendix D) uses 10% of training data for Hessian Eigenvalue calculation The PyHessian library uses Lanczos algorithm (Ghorbani et al., 2019) for calculating the Eigen Spectral density of the
Hessian and uses the Hutchinson method to calculate the trace of the Hessian efﬁciently.
[Table]
[Table]
E S MOOTHNESS OF D ISCRIMINATOR IN SNGAN
We also did the similar experiment of smoothing discriminator in DAT (Sec. 4.1) for SNGAN
Miyato et al.
as the adversarial objective in GAN is similar to DAT.
We use the same conﬁguration for SNGAN as described in PyTorchStudioGAN (Kang & Park, 2020) for both CIFAR (Krizhevsky et al., 2009) and TinyImageNet 3 with batch size of 256 in both cases.
We then smooth the discriminator while discriminator is trained by using the same formulation as in Eq. 9.
We ﬁnd that smoothing discriminator leads to higher (suboptimal)
Fr ´ echet Inception Distance in case of
GANs as well, shown in Fig.
F E XPERIMENTAL D ETAILS F. I MAGE C LASSIFICATION
Ofﬁce-Home : For CDAN methods, we train the models using mini-batch stochastic gradient descent (SGD) with a batch size of 32 and a learning rate of 0.01.
The learning rate schedule is the same as (Ganin et al., 2016).
We train it for a total of 30 epochs with 1000 iterations per epoch.
The momentum parameter in SGD is set to 0.9 and a weight decay of 0.001 is used.
For CDAN+MCC experiments, we use a temperature parameter (Jin et al., 2020) of 2.5.
The bottleneck dimension for the features is set to 2048.
VisDA-2017 : We use a ResNet-101 backbone initialized with ImageNet weights for VisDAexperiments.
Center Crop is also used as an augmentation during training.
We use a bottleneck dimension of 256 for both algorithms.
For CDAN runs, we train the model for 30 epochs with same optimizer setting as that of OfﬁceHome.
For CDAN+MCC runs, we use a temperature parameter of 3.0 and a learning rate of 0.002.
DomainNet : We use a ResNet-101 backbone initialized with ImageNet weights for DomainNet experiments.
We run all the experiments for 30 epochs with 2500 iterations per epoch.
The other parameters are the same as that of Ofﬁce-Home.
To show the effectiveness of SDAT fairly and promote reproducibility, we run with and without
SDAT on the same GPU and environment and with the same seed.
All the above experiments were run on Nvidia V100 and RTX 2080 GPUs.
We used Wandb (Biewald, 2020) to track our experiments.
We will be releasing the code to promote reproducible research.
F.1. A RCHITECTURE OF D OMAIN D ISCRIMINATOR
One of the major reasons for increased accuracy in Ofﬁce-Home baseline CDAN compared to reported numbers in the paper is the architecture of domain classiﬁer.
The main difference is the use
[Table]
of batch normalization layer in domain classiﬁer,
which was done in the library (Junguang Jiang &
Long, 2020).
Table S2 shows the architecture of the feature classiﬁer and domain classiﬁer.
F. A DDITIONAL I MPLEMENTATIONS D ETAILS FOR DA FOR O BJECT DETECTION
In SDAT, we modiﬁed the loss function present in Chen et al.
by adding classiﬁcation loss smoothing, i.e. smoothing classiﬁcation loss of RPN and ROI, used in Faster R-CNN (Ren et al., 2015), by training with source data.
Similarly, we applied smoothing to regression loss and found it to be less effective.
We implemented SDAT for object detection using Detectron2 (Wu et al., 2019).
We ﬁxed ρ to 0.15 for object detection experiments.
G A DDITIONAL R ESULTS
VisDA-2017 : Table S3 shows the overall accuracy on the VisDA-2017 with ResNet-101 backbone.
The accuracy reported in this table is the overall accuracy of the dataset, whereas the accuracy reported in the Table 5 of the main paper refers to the mean of the accuracy across classes.
CDAN w/ SDAT outperforms CDAN by 1.7%, showing the effectiveness of SDAT in large scale Synthetic → Real shifts.
With CDAN+MCC as the backbone, adding SDAT improves the performance of the method to 81.2%.
DomainNet : Table S5 shows the results of the proposed method on DomainNet across ﬁve domains.
We compare our results with ADDA and MCD and show that CDAN achieves much higher performance on DomainNet compared to other techniques.
It can be seen that CDAN w/ SDAT further improves the overall accuracy on DomainNet by 1.8%.
Results with DANN (Ganin & Lempitsky, 2015): Domain Adversarial neural networks introduced the concept of adversarial training in domain adaptation and is a seminal paper in the ﬁeld of domain adaptation.
Table S4 shows the results on some splits on Ofﬁce-Home with DANN and DANN w/
SDAT.
DANN w/ SDAT improves upon the performance on DANN speciﬁcally in challenging splits like Clipart → Art where DANN w/ SDAT gets a 1% increase over DANN.
We have shown results with three different domain adaptation algorithms namely DANN (Ganin &
Lempitsky, 2015), CDAN (Long et al., 2018) and CDAN+MCC (Jin et al., 2020).
SDAT has shown to improve the performance of all the three DA methods.
This shows that SDAT is a generic method that can applied on top of any domain adversarial training based method to get better performance.
[Table]
H D IFFERENT S MOOTHING T ECHNIQUES
Stochastic Weight Averaging (SWA) (Izmailov et al., 2018):
SWA is a widely popular technique to reach a ﬂatter minima.
The idea behind SWA is that averaging weights across epochs leads to better generalization because it reaches a wider optima.
The recently proposed SWA-Densely (SWAD) (Cha et al., 2021) takes this a step further and proposes to average the weights across iterations instead of epochs.
SWAD shows improved performance on domain generalization tasks.
We average every 400 iterations in the SWA instead of averaging per epochs.
We tried averaging across 800 iterations as well and the performance was comparable.
Virtual Adversarial Training (VAT) (Miyato et al., 2019): VAT is regularization technique which makes use of adversarial perturbations.
Adversarial perturbations are created using Algo.
1 present in (Miyato et al., 2019).
We added VAT by optimizing the following objective:
[Equation]
(S13)
This value acts as a negative measure of smoothness and minimizing this will make the model smooth.
For training, we set hyperparameters (cid:15) to 15.0, ξ to 1e-6, and α as 0.1.
Label Smoothing (LS) (Szegedy et al., 2016):
The idea behind label smoothing is to have a distribution over outputs instead of one hot vectors.
Assuming that there are k classes, the correct class gets a probability of 1 - α and the other classes gets a probability of α / (k-1).
(Stutz et al., 2021) mention that label smoothing tends to avoid sharper minima during training.
We use a smoothing parameter ( α ) of 0.1 in all the experiments in Table S6.
We also show results with smoothing parameter of 0.2 and observe comparable performance.
We observe that label smoothing slightly improves the performance over DAT
[Table]
I O PTIMUM R HO VALUE
Table S7 and S8 show that ρ = 0.02 works robustly across experiments providing an increase in performance (although it does not achieve the best result each time) and can be used as a rule of thumb.
[Table]
[Table]
J R ELIABILITY OF E MPIRICAL R ESULTS
To show the reliability and soundness of empirical results
[Table]
we run a few splits of DomainNet datasets across three different seeds.
Table S9 contains the results on CDAN and CDAN w/ SDAT on DomainNet dataset.
It can be seen there is trivial difference between the three runs and CDAN w/
SDAT outperforms CDAN
[Table]
[Table]