Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation
Abstract
In this paper, we consider the problem of domain generalization in semantic segmentation, which aims to learn a robust model using only labeled synthetic (source) data.
The model is expected to perform well on unseen real (target) domains.
Our study finds that the image style variation can largely influence the model’s performance and the style features can be well represented by the channel-wise mean and standard deviation of images.
Inspired by this, we propose a novel adversarial style augmentation ( AdvStyle ) approach, which can dynamically generate hard stylized images during training and thus can effectively prevent the model from overfitting on the source domain.
Specifically, AdvStyle regards the style feature as a learnable parameter and updates it by adversarial training.
The learned adversarial style feature is used to construct an adversarial image for robust model training.
AdvStyle is easy to implement and can be readily applied to different models.
Experiments on two synthetic-to-real semantic segmentation benchmarks demonstrate that AdvStyle can significantly improve the model performance on unseen real domains and show that we can achieve the state of the art.
Moreover,
AdvStyle can be employed to domain generalized image classification and produces a clear improvement on the considered datasets.
Introduction
Semantic segmentation plays a critical role in autonomous driving, which has achieved impressive improvements with the recent development of deep networks [20, 5, 1, 45].
However, these achievements have been mostly attributed to large-scale labeled segmentation datasets, in which annotating pixel-wise labels is very expensive and time-consuming.
Moreover, the model trained on one dataset commonly produces poor performance on unseen datasets captured in different conditions.
This degradation phenomenon is mainly caused by domain shifts [7], including differences in weather, season, light, etc.
For instance, the segmentation model trained on the dataset captured in sunny
London will have low accuracy when deployed on the streets of Zurich in rainy weather.
To address the cross-domain problem, domain adaptation methods [21, 41] are designed to transfer the knowledge of labeled source data to unlabeled target data.
However, one of their main drawbacks is that they require the use of target data during training, which cannot always be accessible in practice.
Another promising line is domain generalization (DG), which focuses on learning a generalizable model using only the labeled source domain.
To reduce the annotating cost and protect data privacy, the existing DG works [7, 40, 46] in semantic segmentation commonly choose to learn the robust model with synthetic data, e.g. , GTAV [28].
In this paper, we focus on this synthetic-to-real DG problem for semantic segmentation.
[Figure]
To eliminate the impact caused by the large domain gap between synthetic and real data, existing solutions mainly aim at augmenting the synthetic source data with extra real-world samples [40, 14] or learning domain-invariant features with carefully designed modules [24, 7].
The key idea behind them is to avoid the model overfitting on the source domain.
This work follows this idea and introduces a new augmenting approach in the perspective of image style for domain generalized semantic segmentation, which is motivated by the observations in Fig.
First , when visualizing the samples of different datasets in Fig.
1(a)
we observe that the image styles are quite different among them, e.g. , the road color.
The style distribution gap can also be observed in Fig. 1(b).
Second , the channel-wise mean and standard deviation of an image, which is called style feature in this paper, can well represent the image style.
When changing the style feature, the image style of an example varies while the semantic content is well maintained (see Fig. 1(c)).
Third , changing the style features of testing samples will largely deteriorate the model performance (see Fig. 1(d)).
Importantly, when replacing the style features with the ones of CityScapes, which has a large distribution distance to
GTAV (see red and green dots in Fig. 1(b)), the performance is lower than other style changing cases.
This indicates that the model performance on testing data is highly related to the style distribution gap between training and testing data.
Taking the above observations into consideration, we argue that the image style is an important factor that affects the model performance and propose the adversarial style augmentation ( AdvStyle ) for domain generalized semantic segmentation.
Specifically, AdvStyle contains two steps: adversarial style learning and robust model learning.
In adversarial style learning, we first decompose the training sample into style feature and normalized image.
The style feature is regarded as a learnable parameter, which is used to reconstruct a new training example together with the normalized image.
Then, we feed the reconstructed example into the segmentation model and optimize the style feature using the adversarial segmentation loss.
The updated style feature is called adversarial style feature and is used to produce hard example in the following step.
In robust model training, we first generate an adversarial example by de-normalizing the normalized image with the learned adversarial style feature.
The adversarial image and the original image are then used to train a robust model using the segmentation loss.
In AdvStyle, the adversarial image is dynamically generated based on the current model.
In this way, the model is always encouraged to update with difficult styles and thus will be more robust to style variations in unseen domains.
In Fig.
2, we show the comparison between
AdvStyle and traditional augmentation methods.
Our AdvStyle can significantly improve the model performance on unseen target domains and clearly outperforms the other augmentation methods.
To summarize, our contributions are threefold:
• We propose the novel adversarial style augmentation (AdvStyle) for domain generalized semantic segmentation, which can consistently improve the results on unseen real domains.
AdvStyle introduces very limited learnable parameters (6-dim feature for each example) and can be easily
implemented with different networks and DG methods.
[Figure]
• Experiments on two synthetic-to-real DG benchmarks demonstrate the effectiveness of the proposed AdvStyle and show that we achieve new state-of-the-art DG performance.
• We show that AdvStyle can also be applied to single DG in image classification and can produce state-of-the-art accuracy on two datasets.
Related Work
Domain Generalization (DG).
To tackle the deficiency of annotated data, DG [40, 14, 7, 33, 46, 4, 43] is introduced to learn a robust model with one or multiple source domains, where the model is expected to perform well on unseen domains.
Recent DG works in semantic segmentation mostly use synthetic data as the source domain, which can be automatically generated but have a large distribution gap to real-world datasets.
One main stream of DG methods [40, 14] focuses on augmenting training samples with extra real-world data from ImageNet [9].
Learning domain-invariant features [7, 24, 33] is another stream to narrow the domain gap.
[24] and [7] leverage instance normalization [34] and whitening transformation to remove the domain-specific information, respectively.
[33] exchanges style features of two samples and adjusts style features with the attention mechanism.
Different from the above methods, our AdvStyle generates new samples by learning adversarial styles using only the synthetic source data.
Adversarial Training in DG.
Adversarial training [12] is initially proposed to learn a robust model that can combat imperceptible perturbations.
In recent years, adversarial training is applied to single
DG in image classification [36, 27, 26, 11], by regarding adversarial samples as augmented unseen samples.
[36] is the first to introduce adversarial samples in DG by max-min iterative training procedure.
Later methods form novel domains with the generated adversarial samples and learn a domain-invariant representation by meta-learning [27, 26] or adaptive normalization [11].
Different from them, this paper adopts adversarial training for semantic segmentation and generates adversarial samples in the perspective of image style.
Style Variation.
Style features are widely studied in image translation [16, 10].
By varying the style features, the image style can be changed while semantic content will be maintained.
Inspired by this, recent works focus on generating data of novel distributions by modifying style features, which are used to train a more robust model.
One effective manner is to generate new styles by exchanging [33, 44] or mixing styles [47] between samples.
On the other hand, new styles can be generated by learnable modules [37].
Instead, we generate novel styles by adversarial training, which encourages the model to always optimize with hard stylized examples.
This work is also closely related to [2], which generates adversarial examples by colorizing.
However, it requires a pre-trained colorization model to change the image color, which is much more complex than our AdvStyle.
In addition, [2] aims to impair the performance of models by adversarial examples.
In contrast, our
AdvStyle leverages the adversarial examples to improve the generalization ability of the segmentation model.
This work also has a connection with “Learning-to-Simulate” [30].
However, [30] tries to learn good sets of parameters for an image rendering simulator in actual computer vision applications while we attempt to learn a generalized model for semantic segmentation.
Our work also shares a similar idea with AdvBN [32]: generating novel styles through adversarial learning. However, unlike AdvBN, which perturbs feature statistics in the Batch Normalization (BN) [18] layers, we directly generate stylized adversarial examples at the image level, which not only avoids the selection of BN layers, but also requires fewer learnable parameters (6-dim vector).
[Figure]
Method
Problem Definition.
Synthetic-to-real domain generalization (DG) focuses on training a robust model with one labeled synthetic domain S , where the model is expected to perform well on unseen domains {T 1 , T 2 , · · · } of different real-world distributions.
As stated by [36], the DG task can be formulated as solving the worst-case problem:
[Equation]
where θ is the model parameters and T is the target domains.
L task denotes the task-specific loss function, which is the pixel-wise cross-entropy loss in this paper.
D ( S , T ) denotes the distribution distance between the source domain and target domains in semantic space.
It is constrained to be lower than ρ for semantic consistency.
Inspired by Eq.
1, we propose a novel approach to generate a dynamic source domain S + , which can help us to reduce the domain shifts between the synthetic domain S and real-world domains T during training.
3.1 Overview
In the introduction, we show that the image style is an important factor that influences the DG performance.
In addition, the channel-wise mean and standard deviation of an image, which is called style feature, can well represent the image style.
Inspired by that, we propose the adversarial style augmentation approach ( AdvStyle ) for domain generalized semantic segmentation.
AdvStyle can dynamically generate images with new styles during training and effectively improve the generalization ability of the segmentation model.
AdvStyle includes two steps: adversarial style learning and robust model training.
In adversarial style learning, we first decompose the image into normalized image and style feature and then update the style feature by adversarial segmentation loss.
In robust model training, we compose the adversarial image by the normalized image and learned adversarial style feature.
The model is then optimized with both original and adversarial images.
These two steps are implemented in each training iteration, enabling us dynamically generate hard stylized samples for the current model.
Our overall framework is illustrated in Fig.
3.2 Adversarial Style Learning
Given an training image x at each iteration, we first compute the channel-wise mean µ and standard deviation σ of x and then obtain the normalized image ¯ x by normalization:
[Equation]
where H and W denote the spatial size of x .
After that, we initialize the adversarial style feature µ + and σ + by µ and σ , which are regarded as learnable parameters.
Then we reconstruct the image with µ + , σ + and ¯ x and forward it into the network for loss computation.
During the backward, the parameters of the network are fixed and the adversarial style feature is updated by:
[Equation]
where γ is the learning rate for adversarial style learning and L seg is the cross-entropy loss.
ˆ x = ¯ x · σ + + µ + is the reconstructed image.
Notice that the style feature is optimized by the adversarial gradient of L seg .
Indeed, the adversarial style learning process can be iterated multiple times.
In our experiment, we find that one-step adversarial style learning achieves similar results with multi-step ones but is much more efficient.
Therefore, we only update the style feature once.
3.3 Robust Model Training
Given the learned adversarial style feature ( µ + and σ + ), we use it to generate the adversarial sample x + with the corresponding normalized image:
[Equation]
Then the original image x and the generated adversarial image x
+ are forwarded to the model for optimization, which can be formulated by,
[Equation]
The detailed training procedure and Pytorch-like pseudo-code can be found in the Appendix.
During testing, we directly input the original samples into the network without implementing AdvStyle.
3.4 Discussion
Adversarial data augmentation have been studied by several works for DG in image classification.
Most of them [27, 36] generate pixel-wise perturbations on the training image x , which usually require additional constraint loss to guarantee the semantic consistency in Eq. 1.
In addition, these works focus on the image classification task where the recognition result is mostly related to global feature.
However, in semantic segmentation, the model needs to produce the per-pixel predictions so that it is more difficult to ensure the pixel-wise semantic consistency during adversarial learning.
Instead, our AdvStyle varies the style feature of the image, which will maintain the semantic content of most pixels and thus can well guarantee the pixel-wise semantic consistency for semantic segmentation.
We conduct experiments in Table 3, which demonstrate the superiority of the proposed AdvStyle over pixel-wise adversarial learning.
Experiments 4.1 Experimental Setup
Datasets and Implementation Details.
For the synthetic-to-real domain generalization (DG), we use one of the synthetic datasets (GTAV [28] or SYNTHIA [29]) as the source domain and evaluate the model performance on three real-world datasets (CityScapes [8], BDD-100K [39], and Mapillary [23]).
The details of the datasets can be found in the Appendix.
Following [7], we use DeepLabV3+ [6] as the segmentation model.
The segmentation model is constructed by three backbones, including MobileNetV2 [31], ResNet-50 [13] and ResNet-101.
We adopt SGD optimizer with an initial learning rate 0.01, momentum 0.9 and weight decay 5 × 10 − 4 to optimize the model.
The polynomial decay [19] with the power of 0.9 is used as the learning rate scheduler.
The learning rate of AdvStyle γ is set to 3.
All models are trained for 40K iterations with a batch size of 16.
Four widely used data augmentation techniques are used during training, including color jittering, Gaussian blur, random cropping and random flipping.
The input image is randomly cropped to 768 × 768 for training.
The original image size is used for testing.
Evaluation Metric.
Following [7], the model obtained by the last training iteration is used to evaluate the mIoU performance on the three real-world validation sets.
For each method, we report the [Table] [Table] result averaged on 3 runs.
When using GTAV as the source domain, we use the 19 shared semantic categories for training and evaluation.
When using SYNTHIA as the source domain, we useshared categories for training and evaluation, i.e., ignoring the train, truck, and terrain categories.
4.2 Evaluation
Effectiveness of AdvStyle on Different Models.
Our AdvStyle is a model-agnostic method, which can be directly applied to different models without modifying the models.
To verify the effectiveness of AdvStyle, we apply it to models with different backbones and normalization modules.
The backbones include MobileNetV2, ResNet-50 and ResNet-101.
The normalization modules include vanilla batch norm (baseline), instance-batch norm (IBN-Net [24]), and instance selective whitening (ISW [7]).
In Table 1, we show the results of using GTAV as the source domain.
We can make the following conclusions.
First, injecting instance-batch norm (IBN-Net) and instance selective whitening (ISW) modules can consistently improve the performance of the baseline.
Second, the proposed AdvStyle can significantly enhance the generalization performance of the baseline model for all backbones.
Specifically, the average mIoU is increased from 26.03%, 27.42% and 31.47% to 32.11%, 37.39% and 37.34% for MobileNetV2, ResNet-50 and ResNet-101, respectively.
In addition, the baseline with AdvStyle produces higher average mIoU than IBN-Net and ISW for all backbones.
Third, when adding AdvStyle, the results of IBN-Net and ISW can be further improved for all settings.
For example, when using ResNet-101 as the backbone, AdvStyle improves the average mIoU of
IBN-Net and ISW by 6.09% and 6.53%, respectively.
In Table 2, we show the results of using
SYNTHIA as the source domain and also observe clear improvements for AdvStyle.
These results verify the prominent advantage of our AdvStyle on different models.
Techniques.
We investigate the impact of different augmentation methods, including color jittering, Gaussian blur, AdvPixel [36] and the proposed
AdvStyle.
AdvPixel is a state-of-the-art method for domain generalized image classification.
The main difference between AdvPixel and AdvStyle is that AdvPixel learns pixel-wise adversarial example while AdvStyle learns style-wise adversarial example. We reproduce AdvPixel in our
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
setting and select the adversarial learning rate (=10) that achieves the best performance.
The random cropping and random flipping are used in default.
Results in Table 3 show that all four augmentation methods can improve the generalization performance.
Importantly, our AdvStyle produces significant improvement compared to other three methods.
Specifically, when using AdvStyle, the average mIoU is increased from 22.91% to 35.32%.
This improvement is about 8% higher than the other 3 methods.
Moreover, AdvStyle is well complementary to color jittering and Gaussian blur.
When combining these three methods, the mIoU is further improved in all target domains.
Compared to AdvPixel, our AdvStyle achieves clearly higher performance, no matter using color jittering and Gaussian blur.
This demonstrates the advantage of learning adversarial style in domain generalized semantic segmentation.
Comparison of Different Style-Aware Methods.
In Table 4, we compare AdvStyle with three styleaware augmentation methods, including MixStyle [47], CrossStyle [33] and RandStyle.
MixStyle mixes the styles of two samples with a convex weight while CrossStyle directly swaps the styles of two samples.
RandStyle can be regarded a reduction of our AdvStyle, which randomly adds
Gaussian noise into the style feature.
All style-aware methods are implemented on the image-level for fair comparison.
We can find that (1) all style-aware methods can consistently improve the performance on all target domains and (2) AdvStyle achieves the best results.
The first finding verifies the effectiveness of augmenting image styles and the second finding shows the benefit of learning adversarial styles over other style-aware methods for domain generalized semantic segmentation.
Parameter Analysis on the Adversarial Learning Rate.
The proposed AdvStyle has one important hyperparameter, i.e. , the adversarial learning rate γ .
To study the impact of γ , we vary it in the range of [0 . 1 , 40] .
Results in Fig.
4 show that AdvStyle can significantly improve the performance on all target domains even with a small value of γ .
The best results are achieved when γ is between 1 and 10.
Assigning a too large value to γ ( e.g. , 40) may produce unrealistic styles and thus hampers the model training.
4.3 Comparison with State-of-The-Art Methods
In Table 5, we compare our method with state-of-the-art DG methods in semantic segmentation, including IBN-Net [24], SW [25], IterNorm [15], ISW [7], DPRC [40] and FSDR [14].
The source domain is GTAV and the backbones are ResNet-50 and ResNet-101. Note that, since different [Table] different segmentation networks ( e
[Table]
g.
[Table] DeepLabV2 [5] [Table] DeepLabV3+ [6] and FCN [20]), different training sets ( e.g. , the whole GTAV and the training set of GTAV), different training strategies ( e.g. , learning rate and optimizer) , different auxiliary data ( e.g. , ImageNet samples) and different evaluation manners ( e.g. , the best model and the last model), it is hard to compare them in an absolutely fair way.
We show the results of each method as well as the absolute gain against the corresponding baseline.
From Table 5, we can make the following conclusions.
First , when using the same baseline model, adding AdvStyle can produce better results than IBN-Net, SW, IterNorm and ISW.
Moreover, when applying AdvStyle to IBN-Net or ISW, we achieve new state-of-the-art performance for both ResNet-50 and ResNet-101.
Second , when compared across baselines, “Baseline+AdvStyle” achieves the state-of-the-art mIoU for ResNet-50.
On the other hand, when using ResNet-101 as the backbone, “IBN-Net+AdvStyle” produces higher results than DRPC and comparable results with
FSDR.
Importantly, both FSDR and DRPC use extra ImageNet images and select the best training checkpoints for each target domain.
Instead, “IBN-Net+AdvStyle” only utilizes the source data and uses the last training checkpoint to evaluate all target domains.
Third , when using the best checkpoint for evaluation, we (“ISW+AdvStyle”) produce better performance than DRPC and FSDR, leading to the new state-of-the-art results under the “best checkpoint setting”.
Even so, we argue that it is more reasonable to use the last checkpoint for evaluating all target domains.
This is because we can not always have the right labeled validation sets to select the best model for unseen domains in practice. [Table]
[Table] [Table] [Table] To make more fair comparisons with the state-of-the-art methods, we also use the whole set of GTAV for training ISW and “ISW+AdvStyle”.
The results with ResNet-101 are reported in the last two rows of Table 5.
We can find that using the whole set of GTAV can produce higher results on the
CityScapes and the Mapillary datasets.
4.4 Performance under Multi-Source Setting
In Table 6, we evaluate the models under the multi-source domain generalization setting, where both
GTAV and SYNTHIA are used as the source domains.
The compared methods include baseline [7],
IBN-Net [24], ISW [7], and our AdvStyle.
Clearly, AdvStyle consistently improves the results of
ISW, further verifying the effectiveness of the proposed AdvStyle.
4.5 Visualization
Qualitative Comparison of Segmentation Results.
In Fig.
5, we compare the segmentation results for different methods on target domains.
It is clear that, the proposed AdvStyle can consistently improve the segmentation results for baseline, IBN and ISW models, especially for the easily-confused classes, e.g. , road vs sidewalk and building vs sky.
t-SNE of Styles.
In Fig.
6, we visualize the style features generated by AdvStyle during the training phase for the GTAV training set, where ResNet-50 is used as the backbone.
We can find that AdvStyle can continuously generate new style features that are different from the original distribution.
The new style features have the chance to be located at the distributions of other datasets during the training process.
Moreover, AdvStyle will also generate styles that are out of the distributions of the four datasets (G, C, B, M) and may appear in other unseen domains.
The visualization results further demonstrate that AdvStyle can encourage the model to meet more diverse and unseen styles during training, leading to a more robust model.
4.6 Evaluation on Image Classification Task
To verify the versatility of the proposed AdvStyle, we evaluate it on single DG in image classification.
Experiments are conducted on two popular DG datasets, i.e. , Digits and PACS.
The details of the datasets and implementation can be found in the Appendix.
Results on Digits.
In Table 7, we compare with the baseline (ERM [35]) and 6 state-of-the-art methods, including CCSA [22], d-SNE [38], JiGen [3], ADA [36], M-ADA [27] and ME-ADA [42].
For AdvStyle, we implement it with ERM and ME-ADA.
It is clear that, our AdvStyle can significantly improve the accuracy of ERM on all target domains. When applying AdvStyle to the state-of-the-art method (ME-ADA), the performance is improved by a large margin, i.e. , 8.0% in average accuracy.
[Figure]
[Table] [Table]
Results on PACS.
We compare with the baseline (ERM [35]) and three state-of-the-art DG methods, including JiGen [3], RSC [17] and L2D [37].
We reproduce JiGen, RSC and L2D with their official source codes.
All methods use the same baseline (ERM).
Results in Table 8 show that JiGen and
RSC produce limited improvements.
Instead, our AdvStyle can significantly increase the accuracy on all domains for both ERM and RSC.
Compared to the recent published work (L2D), our method (ERM+AdvStyle) outperforms it by 2.4% in average accuracy.
In addition, AdvStyle can also be applied to L2D and yields an improvement of 6.3% in average accuracy.
The results on Digits and PACS validate that AdvStyle can also be effectively applied to single domain generalized image classification and can achieve state-of-the-art accuracy.
Conclusion
In this paper, we propose a novel augmentation approach, called adversarial style augmentation ( AdvStyle ), for domain generalization (DG) in semantic segmentation.
AdvStyle dynamically generates hard stylized images by learning adversarial image-level style feature, which can encourage the model learning with more diverse samples.
With AdvStyle, the model can refrain from the problem of overfitting on the source domain and thus can be more robust to the style variations of unseen domains.
AdvStyle is easy to implement and can be directly integrated with different models without modifying the network structures and learning strategies.
Experiments on two synthetic-to-real settings show that AdvStyle can largely improve the generalization performance and achieve state-of-the-art performance.
In addition, AdvStyle can be employed to single DG in image classification and obtain significant improvement.
Acknowledgement
This work is supported by the EU H2020 project AI4Media (No. 951911) and the PRIN project
PREVUE (Prot.
2017N2RK7K).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re us- ing/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable
[Table]