Domain Adaptation and Generalization: A Low-Complexity Approach
Abstract: Well-performing deep learning methods are essential in today’s perception of robotic systems such as autonomous driving vehicles.
Ongoing research is due to the real-life demands for robust deep learning models against numerous domain changes and cheap training processes to avoid costly manual-labeling efforts.
These requirements are addressed by unsupervised domain adaptation methods, in particular for synthetic to real-world domain changes.
Recent top-performing approaches are hybrids consisting of multiple adaptation technologies and complex training processes.
In contrast, this work proposes EasyAdap, a simple and easy-to-use unsupervised domain adaptation method achieving near state-of-the-art performance on the synthetic to real-world domain change.
Our evaluation consists of a comparison to numerous top-performing methods, and it shows the competitiveness and further potential of domain adaptation and domain generalization capabilities of our method.
We contribute and focus on an extensive discussion revealing possible reasons for domain generalization capabilities, which is necessary to satisfy real-life application’s demands.
Introduction
Numerous applications have become feasible due to deep learning methods for computer vision, including the environment perception of autonomous vehicles.
In particular, architectures emerged to deliver high-quality results for semantic segmentation tasks, i e., a pixel-wise classification of images.
Unfortunately, it is in the nature of deep learning models to be sensitive to domain changes, e. g., domain changes from day to night, from telescope to wide-angle lenses or from rural to urban areas.
An extensive training effort is necessary to keep the high-quality on the target domain, i. e., to adapt the model from the source to the target domain.
Moreover, supervised training methods for the target domains are infeasible due to the high costs of manually labeling data 1 and the virtually endless number of target domains 2 .
Several approaches exist, minimizing or even completely avoiding labeling costs for the targetdomain data.
For example, Active learning approaches implement a human-in-the-loop training process, which asks a human to provide labels for a selection of samples, in order to achieve highquality training results with minimal manual labeling effort [2].
While active learning for semantic segmentation tasks remains a huge challenge, it also still requires a certain amount of manual effort [3, 4, 5].
Unsupervised domain adaptation methods completely avoid the usage of manually labeled [Figure] S o u r c e D o m a i n D a t a
T a r g e t D o m a
i n D a t a [Figure] target-domain data. Instead, they align the distributions of source- and target-domain data in the input [6, 7, 8], feature [9, 10, 11, 12], output space [13, 14, 15], or a combination of those [16, 17, 18].
The latter is complex in design, and therefore, hard to reproduce.
The usage of synthetic datasets as source domain completely abandons the need for manual labeling effort [19, 20].
However, the large domain gap between synthetic and real-world domains makes an adaptation between these domains increasingly difficult.
Several aspects cause this domain gap, such as a decreased variety in synthetic data and the differences regarding geometry, texture, and lighting conditions.
Due to these impediments, models trained on synthetic datasets poorly perform on real-world data.
On the other hand, real-life applications demand models that are robust against a variety of domain changes.
But performing domain adaptation to a large number of domains is not feasible due to the immense training effort.
Desirable domain adaptation methods achieve a domain generalization.
These methods train models not only to improve on the source or target domain but also on unseen domains [21, 22, 23].
This work addresses the problem of unsupervised domain adaptation from synthetic to real-world domains.
We propose EasyAdap , an iterative extension of our work in [11], which conducts a distribution alignment in the feature- and output-space (see Figure 1 for an overview).
While achieving near state-of-the-art performance on the GTA5 [19] to Cityscapes [1] domain change, our method is still simple and easy-to-tune due to its simple architecture 3 .
Our feature-space distribution alignment is based on a self-supervision through a clustering loss of the target-domain features towards class-specific source-domain feature centroids.
A self-training addresses the alignment of the distribution in the output space.
We study the interplay between this self-supervision and self-training and show that a synergistic effect exists (see section 4.4).
We discuss the reasons for this effect in Section 4.5.
Overall, our work contributes EasyAdap, an easy-to-use unsupervised domain adaptation method that yields near state-of-the-art performance.
We contribute an extensive study of various unsupervised domain adaptation methods regarding domain generalization.
Section 2 discusses domain adaptation methods related to our work.
While Section 3 describes our method, Section 4 evaluates our approach and compares it to other domain adaptation methods w.r.t. the performance on the target domain and unseen domains.
This work concludes with Section 5.
Related Work
We categorize related unsupervised domain adaptation methods for semantic segmentation tasks regarding their adaptation in the input, feature, and output space and hybrid methods thereof.
The latter (hybrids) comprise methods from the different categories, making up the majority of the recent top-performing publications.
While domain adaptation is a broad field (e. g., which includes learning new classes), we concentrate on adaptation approaches that align the distributions in the input, feature, and output space.
The following subsections collect and briefly describe examples for these categories.
Input-Space Adaptation.
Aligning the data distribution in the input space usually resembles a style transfer from the source domain to the target domain.
Since a style transfer only changes the textures while keeping the geometry of the images, the ground truth is also valid for the styletransferred images.
Now, training a model on labeled style-transferred images from the source domain is close to training a model on the target domain.
The most successful approaches for style transfer implement Cycle-GANs [6, 7], Fourier transformation [8, 24] or global image operators [16].
Feature-Space Adaptation.
The adaptation in the feature space of a deep learning model usually relies on adversarial training or self-supervision and aims at distribution alignment.
When the prelogit feature distributions of the source and target domain match, the classifier in the last layer classifies the source- and target-domain images in the same way.
Adversarial training is based on a feature extractor and a domain discriminator network, classifying the feature space into source and target domains.
The optimization goal is to generate a feature space that is not discriminable into the source and target domain while containing relevant representations for the semantic segmentation [9, 25].
A recent approach implements a contrastive loss 4 on class-wise features [10] attracting features of the same class while repulsing features of different classes independent of their domain.
The approaches presented in [12] and our approach [11] apply semantic self-supervision to align the distribution in the feature space by attracting the target-domain features in the pre-logit feature space towards class-specific source-domain centroids.
In contrary to the contrastive self-supervision presented in [10] the attraction mechanism used in [11, 12] does not rely on pseudo-labels for the target domain.
Our work advances the approach of [11].
Output-Space Adaptation.
Distribution alignment approaches in the output space either implement an adversarial training or a self-training.
Adversarial approaches consider the structure of the output space, i. e., they discriminate images between the source and target domain based on the (spatial) distribution of the output of the segmentation model [13, 14].
Aiming at an easily reproducible approach, we avoid adversarial approaches in our work.
Most of the current top-performing methods implement a self-training on the target domain, which is a training of the model on self-inferred labels ( pseudo-labels ).
The effect of self-training depends on the quality of the pseudo-labels, i. e., these methods require strategies for the mitigation of false pseudo-labels.
Common approaches ignore low-confident pseudo-labels using a fixed confidence threshold [27, 28].
Some approaches filter low-confident pseudo-labels dynamically [15, 17, 29].
Hybrid Methods.
Most of the top performing approaches are hybrid methods combining adaptions methods working in the input, feature, and output space, notably [16, 17, 18].
The approach presented in [18] performs a distribution alignment in the feature (clustering) and output space (selftraining).
The approach in [16] even combines adaptation methods in the input, feature, and output space.
The currently best performing approach presented in [17] applies methods for distribution alignment in feature and output space and additionally performs three structurally different adaptation stages.
Compared to these currently top-performing methods, our approach comes with a much lower complexity regarding design and implementation effort.
Method
We introduce a nested iterative training approach for domain adaptation based on repeated application of self-training and semantic clustering (see Figure 1).
With this, we advance our approach in [11] from a two stage model to an iterative model.
We initialize the process with a training on the source domain (see Section 3.1).
The inner loop of the training process establishes a training on the source and target domain, stopping early after a short number of epochs.
The training on the target domain comprises a semantic clustering and a self-training which is implemented as described 4 Cf. [26] for an overview of contrastive learning. in [11].
Upon finishing such an inner iteration, the outer loop of the training process creates new pseudo-labels and repeats the process.
Section 3.2 describes the algorithm for the training process.
3.1 Building Blocks
Source Training Self-training approaches require models with a sufficient understanding of the data to provide valuable pseudo-labels.
To that, our approach initializes the model with a sophisticated supervised training method on the source domain, including a rich data augmentation and a class-uniform sampling strategy.
Zhu et al.
[30] showed significant improvements in supervised training when applying a class-uniform sampling strategy.
We adapt their class-uniform sampling in the following way.
We first gather a list of objects from the dataset, i.e., a list of polygons 5 , each of which has a class attribute and a centroid.
We sample from this list regarding uniformly distributed class attributes during the training.
Eventually, each training batch combines randomly chosen crops and crops that are centered around these polygons.
A parameter 0 ≤ γ ≤ 1 controls the ratio of random samples and class-uniformly selected samples.
We pair this uniform sampling with a strong data augmentation consisting of Gaussian blurring, color jittering, and random scaling 6 .
Semantic Clustering
When training on the source- and target-domain data, our approach for clustering the feature space is inspired by the image-classification method presented in [12] and advances our work in [11] 7 .
We compute the class centroids on the source-domain data as running averages of class-specific feature representations in the pre-logit feature space.
We then compute a similarity matrix between these class centroids and the feature representations of the target-domain data.
After weighing each entry of the similarity matrix with a constant that scales with the certainty of the classification on the target-domain data, our loss function computes the entropy of the weighted similarity matrix.
This process implements a clustering 8 algorithm that executes through the backpropagation.
For details, please refer to the description in [11] or the supplementary material.
The process of Semantic Clustering is a form of self-supervision.
Self-Training Our process applies a simple self-training approach.
Before training a model on the target domain, we create pseudo-labels for the target-domain images given the current model.
To mitigate the influence of false pseudo-labels, we exclude labels from the self-training that yield a prediction-vector entropy exceeding the threshold β · log K , where K is the number of classes and β > 0 .
Improving the quality and impact of the pseudo-labels for self-training requires implementing an iterative adaptation process.
I. e., each adaptation step first needs to create new pseudo-labels before resetting the model’s weights and (re-) starting the training.
3.2 EasyAdap: Assembling the Bricks
Figure 1 and Algorithm 1 in the supplementary material describe our training process.
First, we perform the supervised training on the source-domain data with class-uniform sampling and a strong data augmentation to gain an initial model M (0) .
Then, the process enters the domain adaptation loop.
Each adaption step first (re-) creates the pseudo-labels using the current available model M ( n ) (i. e., M (0) in the first iteration).
After resetting the model’s weights (to weights pre-trained on ImageNet[31]), the adaptation step starts training a new model.
We rebuild the list of samples regarding random and class-uniform samples in each epoch of the training.
While iterating over the smaller (source- or target-domain) dataset, we sample as many batches as the larger dataset allows, i. e., we restart sampling from the smaller dataset until we finish processing the larger dataset.
We compute the semantic segmentation loss for each training batch on the source- and target-domain data using the ground-truth and pseudo-labels, respectively.
We use the features of the sourcedomain data to update the running average of the class centroids.
The similarities between these centroids and the target-domain features yield the clustering loss.
We update the model’s weights using both loss functions.
Evaluation and Discussion
This section evaluates our approach, EasyAdap, regarding domain adaptation and domain generalization quality.
To that, we evaluate its performance on the domain shift from synthetic to real-world data via adaptations from GTA5 [19] and Synthia [20] to Cityscapes [1], respectively.
We compare our approach against numerous top-performing approaches from the literature in Section 4.1.
Since real applications require domain generalization instead of specific domain shifts, we discuss and compare our approach regarding domain generalization against existing domain adaptation approaches in Section 4.2.
Section 4.3 evaluates the impact of our data augmentation and the class-uniform sampling.
In Section 4.4, we show the effects of the self-training and self-supervision.
We conclude our evaluation with a discussion in Section [-4.
-]4.
All of our experiments are based on an implementation of DeepLabV3+ 9 with a WideResNet [32] backbone.
We scaled all target-domain training images to match the size of the source-domain images.
While we train our model on crops of 400 × 400 pixels in batches of 24 source- andtarget-domain images, we validate it on the original sized images.
The training process implements a stochastic gradient descent optimizer with momentum 0 .
9 and a weight decay of 10 − 4 .
While the source-only setups trained for 45 epochs, our iterative adaptation steps stopped early afterepochs.
We decay the initial learning rate of 0 .
007 with a factor 1 − epoch epoch max after each epoch.
We configure our domain adaptation approach as proposed in [11] except for the self-training threshold, which we now set to β = 116 (see Section 3.1).
4.1 Comparison to Existing Approaches
Table 1 shows the performance of state-of-the-art unsupervised domain adaptation methods from
GTA5 to the Cityscapes dataset.
This table includes methods aligning distributions on the input space [6, 16], feature space [33, 34, 35], output space (e [Table] g., via self-training [15, 27]), and hybrid methods [16, 17, 18] (see Section 2 for a description of this categorization)
[Table]
[Table] l [Table] t r a
[Table]
t r a [Table] fi
[Table]
s i
[Table]
t
[Table]
ti
[Table]
Our method EasyAdap achieves near state-of-the-art performance in terms of mIoU on the GTA to Cityscapes domain change: We achieve 56 .
32% mIoU, which is comparable to the currently best approach ProDA with 57 .
5% mIoU. Similar to ProDA, we gain our improvement over other approaches from classes like traffic light, traffic sign, fence, rider [Table] and motorbike , which are difficult to learn.
Our method outperforms approaches that have similarities to our method (see Section 2) on this domain change.
In particular, this includes CLST and CAG with 50 . 2% and 51 .
6% mIoU, respectively.
Except for ProDA, the same holds for all hybrid methods.
ProDA, however, is hard-totune due to its complex architecture (four structurally different training stages) and hyper-parametertuning.
Advancing our method from [11] into an iterative domain adaptation process improves the results from 46 .
4% mIoU to 56 .
32% due to synergistic effects between our self-supervision and self-training.
We further discuss these effects in Section 4.
[Table] [Table] [Table] Table 2 shows the performance of unsupervised domain adaptation methods from Synthia to the
Cityscapes dataset.
Without changing any hyper-parameters tuned for the GTA5 to Cityscapes domain change, our approach still performs well, ranking behind ProDA, SAC, and CLST, all of which are also well-performing in the adaptation from GTA5 to the Cityscapes dataset.
4.2 Domain Generalization
This section evaluates the domain generalization capabilities of our domain adaptation method and compares it with several other approaches.
To that, Table 3 shows the performance of several models on the datasets Cityscapes, BDD [39], and four different domains of the dataset ACDC [40].
No model has seen samples of BDD or ACDC during the training.
Furthermore, the source-only models did not see the Cityscapes samples during training.
BDD differs from the target domain of the adaptation (Cityscapes) by containing a diverse set of weather and lighting conditions and by being recorded across the USA instead of Germany.
ACDC, which was recorded in Europe, defines the specific partitions rain, fog, snow, and night .
Hence, regarding the models trained on GTA5 (and Cityscapes without ground-truth), these datasets include domain shifts regarding the environment, weather, and sensors.
The improvements of our source-only models due to data augmentation and additional class-uniform sampling also transfer to the unseen domains of BDD and ACDC.
Considering the DeepLabV [29, 13] and DeepLabV3+ source-only models, there is no clear superior model.
Note that the target model (trained on the Cityscapes dataset) yields similar results on BDD as the adaptation approaches ProDA and EasyAdap from GTA5 to Cityscapes.
Apart from that, the target model always outperforms the domain adaptation approaches significantly.
While ProDA leads the board of domain adaptation approaches, EasyAdap achieves very close results.
Furthermore, EasyAdap outperforms the other domain adaptation approaches (see table 3).
4.3 Source-Only Training
The quality of our source-only training is crucial for the later creation of the first pseudo-labels. To study the design decision of our source-only training, Table 1 and 2 show the performance of different source-only models: trained without any augmentation (source-only); trained with color jitter, Gaussian blurring, and random scaling (source-only aug); and trained with augmentation and class-uniform sampling (source-only uni 50% and source-only uni 100% ). On both datasets, GTA and Synthia, the augmentation improves the plain source-only model’s quality by over 10% mIoU. An additional class-uniform sampling improves the model’s quality further by over 5% mIoU. Sampling the complete batch via class-uniform sampling (source-only uni 100% ) outperforms a mixed batch of random and class-uniformly sampled images (source-only uni 50% ).
4.4 Impact of Adaptation Features
We conducted several experiments to study the impact of EasyAdap’s features on the model’s quality.
Figure 2 shows the best mIoU values on the validation set per adaptation step, which consists of N train = 20 training epochs (see Algorithm 1).
We successively enabled selftraining, class-uniform sampling 10 , and the semantic self-supervision on the target domain (see
Section 6.1 and 3.1).
Figure 2 shows that a self-training yields similar improvements as a combination with an additional class-uniform sampling on the target domain.
A further self-supervision shows a steeper learning curve in the first few adaptation steps and yields a more durable learning behavior.
I. e., the two learning processes without self-supervision stagnate after four adaptation steps, but the model trained with additional self-supervision still increases in quality.
4.5 Discussion and Limitation
Does domain adaptation transfer knowledge?
Table 3 shows that the target model trained on Cityscapes ( Cityscapes model ) achieves similar performance on BDD as the ProDA and EasyAdap models adapted to the Cityscapes dataset.
While the Cityscapes model has to overcome the domain change from sunny to rainy weather included in BDD, the adaptation methods need to overcome the domain change from synthetic to real-world.
The Cityscapes model also achieves similar performance on ACDC’s rain domain as the ProDA and EasyAdap models.
In this case, the target domain consists of rainy images only, which challenges the Cityscapes model even more, while the adapted models from GTA5 seem to play out their advantage of having seen synthetic rainy images during their training.
On the other hand, the Cityscapes model outperforms the adapted models on the foggy and snowy domains, which are unseen for all adapted models.
We think that the additional domain gap from the synthetic to the real world for the adapted models is the reason for the performance drop of the adapted models compared to the Cityscapes model.
Hence, the adapted models show the behavior of knowledge transfer (here, rainy images) from synthetic to real-world domains.
This observation supports approaches adapting from synthetic to real-world domains.
Moreover, it supports the demand for richer synthetic datasets regarding different domains, such as sensor, weather, lighting, and environmental domains.
How to support domain generalization through adaptation?
Since a domain adaptation to every possible encountered domain is infeasible in real-life applications, this work explicitly addresses and studies domain generalization effects of unsupervised domain adaptation methods, including our own method EasyAdap.
Section 4.2 and Table 3 show stronger generalization effects for some of the adaptation methods.
In particular, SAC, ProDA, and EasyAdap show significant improvements in the unseen domains of BDD and ACDC.
These methods apply self-training while AdapSeg does not, and Seg-Uncertainty only applies self-training without improving the pseudo-labels through recreation.
These observations open the question of whether self-training, which mimics a supervised training on both domains, yields these strong generalization effects.
What are the reasons for synergistic effects?
The experiments in Section 4.4 and Figure 2 show a synergistic effect since enabling self-training and self-supervision gains 15% mIoU while enabling only one of the mechanisms gains only 8% and 4% , respectively.
Hence, simultaneously enabling both mechanisms gains another 3% atop each single mechanism.
We argue that there is a synergy between self-training and semantic self-supervision, i. e., they support each other.
On the one hand, self-training aims to mimic a supervised training on the source and target domain, which helps the training process to generalize upon both domains.
This generalization effect yields a better alignment of the feature distribution in both domains, which in turn helps the self-supervision to attract the target-domain features to the correct source-domain class centroids.
On the other hand, the self-supervision reduces the number of correct but ignored pseudo-labels by sharpening the feature space and therefore reducing the uncertainty of correctly classified pixels.
How does the sampling strategy affect generalization?
Sections 4.2 and 4.3 show that our sourceonly models generalize well to unseen real-world data.
We argue that combining a strong data augmentation with a class-uniform sampling improves the domain generalization by avoiding overfitting.
On the one hand, seldom classes gain weight in the training by oversampling, but the strong augmentation avoids overfitting.
On the other hand, large-area classes such as vegetation and terrain lose weight compared to seldom classes, which again avoids overfitting the training data.
Conclusion
We propose the easy-to-use unsupervised domain adaptation method EasyAdap with near state-ofthe-art performance based on a self-supervision and self-training strategy.
Our evaluation shows that the quality of our method comes from a combination of improvements, including preliminary training on the source domain with increased domain generalization capabilities due to strong data augmentation and a class-uniform sampling; a self-training that yields strong domain generalization capabilities; and a synergy of our self-supervision and self-training.
Our evaluation includes an extensive discussion of domain adaptation methods at hand regarding knowledge transfer and domain generalization capabilities.
Our discussion includes several research questions regarding domain adaptation and domain generalization of training and adaptation methods. Acknowledgement
The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Climate Action" within the project
“KI Delta Learning“ (Förderkennzeichen 19A19013K).
The authors would like to thank the consortium for the successful cooperation