T OWARDS R OBUST O BJECT D ETECTION I NVARIANT TO R EAL -W ORLD D OMAIN S HIFTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Safety-critical applications such as autonomous driving require robust object detection invariant to real-world domain shifts.
Such shifts can be regarded as different domain styles, which can vary substantially due to environment changes, but deep models only know the training domain style.
Such domain style gap impedes object detection generalization on diverse real-world domains.
Existing classification domain generalization (DG) methods cannot effectively solve the robust object detection problem, because they either rely on multiple source domains with large style variance or destroy the content structures of the original images.
In this paper, we analyze and investigate effective solutions to overcome domain style overfitting for robust object detection without the above shortcomings.
Our method, dubbed as Normalization Perturbation (NP), perturbs the channel statistics of source domain low-level features to synthesize various latent styles, so that the trained deep model can perceive diverse potential domains and generalizes well even without observations of target domain data in training.
This approach is motivated by the observation that feature channel statistics of the target domain images deviate around the source domain statistics.
We further explore the style-sensitive channels for effective style synthesis.
Normalization Perturbation only relies on a single source domain and is surprisingly simple and effective, contributing a practical solution by effectively adapting or generalizing classification DG methods to robust object detection.
Extensive experiments demonstrate the effectiveness of our method for generalizing object detectors under real-world domain shifts.
I NTRODUCTION
Object detection, a fundamental computer vision task, plays an important role in various safety-critical applications, including autonomous driving (Grigorescu et al., 2020), video surveillance (Raghunandan et al., 2018), and healthcare (Dusenberry et al., 2020).
Deep learning has made great progress on in-domain data (Ren et al., 2015; Bochkovskiy et al., 2020) for object detection, but its performance usually degrades under domain shifts (Sakaridis et al., 2018; Michaelis et al., 2019), where the testing (target) data differ from the training (source) data.
Real-world domain shifts are usually brought by environment changes, such as different weather and time conditions, attributed by diverse contrast, brightness, texture, etc.
Trained models usually overfit to the source domain style and generalize poorly in other domains, posing serious problems in challenging real-world usage such as autonomous driving.
Figure 1(b) shows a large gap of feature channel statistics between two distinct domains, Cityscapes (Cordts et al., 2016) and Foggy
Cityscapes (Sakaridis et al., 2018), especially in shallow CNN layers which preserve more style information (Zhou et al., 2020c; Pan et al., 2018).
Deep models trained on the source domain cannot generalize well on the target domain, due to the discrepancy in feature channel statistics caused by the domain style overfitting.
Domain generalization (DG) (Muandet et al., 2013; Ghifary et al., 2016; Mahajan et al., 2021; Li et al., 2020) aims to solve this hard and significant problem.
Major undertaking has been done on improving domain generalization of classification models, where multiple source domains with large inter-image style variance are available for model training.
However, less attention has been paid on robust object detection (Wang et al., 2021) which is of equal importance if not more in many visual perceptual systems.
The closely related unsupervised domain adaptation (UDA) object
[Figure]
After equipping Normalization Perturbation in shallow CNN layers our model can effectively blend distinct domain style distributions.
Thus our model generalizes much better on the target domain.
detection (Schneider et al., 2020; Nado et al., 2020) has been widely studied, but it requires target domain images for model training, which is often infeasible for some online object detection systems.
Synthesizing new domains have been demonstrated as an effective solution for domain generalization in the classification task (Nuriel et al., 2021;
Zhou et al., 2020c).
The rationale behind is that the model can learn domain-invariant representations and generalizes well by perceiving a large variety of synthesized domains during training.
But existing domain synthesis methods are all specifically designed for image classification and it is non-trivial to directly apply these methods for robust object detection because of the task gap between classification and detection.
Specifically, the feature-level synthesis approach (Zhou et al., 2020c; Li et al., 2022b) is effective for classification DG problem, but it requires multiple source domains with large style variance.
However, there is usually only one single source domain for robust object detection due to the expensive annotation cost, which means that only relatively small style variance can exist.
In this situation, previous feature-level synthesis approach cannot synthesize sufficient diverse domains.
The image generation based synthesis approach (Jackson et al., 2019; Geirhos et al., 2018) can effectively address the single-source domain problem by leveraging an extra large-scale style image dataset (Kaggle) to synthesize diverse domains.
But the image generation procedure may suffer from the potential destruction on image contents, which are essential for the hierachical object detection where large context diversity may be present.
In this paper, we perform in-depth problem analysis for the under-explored robust object detection, and propose a novel domain style synthesis approach.
Figure 1(a) shows our motivation.
Feature channel statistics of the target domain image deviate around the source domain statistics.
Thus by perturbing the feature channel statistics of source domain images in the shallow CNN layers, we can effectively synthesize new domains.
The perturbed feature statistics correspond to various latent domain styles, so that the trained model perceives diverse potential domains accordingly.
Such perturbation enables deep models to learn domain-invariant representations where distinct domains can be effectively blended together in the learned feature space.
To further boost the performance, we have also explored the style-sensitive channels for effective domain style synthesis.
Figure 1(b) 1 shows the distinct domains can be effectively blended by the perturbed channel statistics in shallow CNN layers.
The learned deep CNN representations are thus more robust to the variations of different domain styles and generalize well on the target domain.
Note that the model is only trained using a single source domain data without any access to the target domain style or data.
Our method normalizes features into different scales, thus called Normalization Perturbation (NP).
NP perturbs the feature channel statistics of source domain images to effectively blend distinct domain style distributions, which is not limited by the small source domain style variance inherent in while does not affect the input images.
Thus our model generalizes much better on robust object detection, which has small source domain style variance and large context diversity.
Our NP is simple and effective, generalizes well under various real-world domain shifts, and outperforms previous
DG and UDA methods on multiple dense prediction tasks.
Our object detection DG method design contributes a practical solution by effectively adapting or generalizing existing feature perturbation based classification DG methods to robust object detection.
In summary, our contributions are:
• We investigate the real-world domain shifts for robust object detection and highlight shortcomings of existing domain synthesis approach.
We contribute to researchers on adapting existing classification DG methods to effectively address the object detection DG problem.
• We propose to perturb the channel statistics of source domain features to synthesize various new domain styles, which enables a model to learn domain-variant representations for good domain generalization.
• Our method generalizes surprisingly well under various real-world domain shifts and is very simple, without any extra input, learnable parameters or loss.
Extensive experiments are conducted to verify the effectiveness of our method.
R ELATED W ORKS
Domain Generalization (DG) (Hendrycks & Dietterich, 2018; Venkateswara et al., 2017), which targets at generalizing models to unseen domains, relies on source data typically consisting of multiple distinct domains.
DG has been mainly studied in the context of object recognition task (Du et al., 2020a;b; Yue et al., 2019).
DG of dense prediction tasks (Seemakurthy et al., 2022) (Hasan et al., 2021; Lin et al., 2021) has attracted increasing interests because of its wide real-world applications.
The closely related unsupervised domain adaptation (UDA) (Long et al., 2016; Damodaran et al., 2018) has been widely studied on dense prediction tasks (VS et al., 2021;
Kim et al., 2019) for realworld applications, which aims to generalize model to the target domain by accessing its unlabeled images.
Both DG and UDA share significant overlap in technicalities, such as domain alignment (Jia et al., 2020; Zhou et al., 2021), self-supervised learning (Xie et al., 2020; Iqbal & Ali, 2020), feature disentanglement (Wang et al., 2020b; Yang et al., 2019), and data augmentation (Choi et al., 2019; Wang et al., 2022).
Our method closely relates to the following works.
Normalization-based Methods.
The normalization layers are leveraged to improve model generalization ability.
Various normalization variants have been proposed, such as domain-specific
BatchNorm (Liu et al., 2020; Segu et al., 2020), AdaBN (Li et al., 2016), PreciseBN (Wu & Johnson, 2021), Instance-Batch Normalization (Choi et al., 2021a; Pan et al., 2018), Adversarially Adaptive
Normalization (Fan et al., 2021), Switchable Normalization (Luo et al., 2019), and Semantic Aware
Normalization (Peng et al., 2022).
The test-time adaptation (Wang et al., 2020a; Zhang et al., 2021) attempts to estimate accurate normalization statistics for the target domain during testing.
These methods fit normalization layers to the specific target domains, while our method normalizes features into different scales to implicitly synthesize arbitrary new domains, and is optimization-free.
Synthesizing New Domains.
Data augmentation (Volpi & Murino, 2019; Otálora et al., 2019) has been widely used to synthesize new domains in DG and UDA.
Some methods synthesize new domain images using image-to-image translation models, such as the random (Xu et al., 2020) or learnable augmentation networks (Carlucci et al., 2019b; Zhou et al., 2020a), and style transfer models (Yue et al., 2019; Somavarapu et al., 2020).
Other works propose to perform implicit domain synthesis through the feature-level augmentation (Zhou et al., 2020c; Jin et al., 2021; Mancini et al., 2020;
[Table]
Tang et al., 2021; Nuriel et al., 2021;
Gong et al
[Table]
, 2019) to mix CNN feature statistics of distinct domains to significantly improve the domain synthesis efficiency.
The above methods rely on image generation or multiple source domains for new domain synthesis, and thus the efficiency or effectiveness is limited.
On the other hand, our method only relies on a single source domain to diversify by perturbing the feature channel statistics to produce various latent domain styles.
While SFA (Li et al., 2021) performs the activation-wise feature perturbation, which may destroy meaningful image contents, our method performs the channel-wise feature statistics perturbation and keeps the image contents unchanged.
Note that some methods also perform feature statistics perturbation, e .
g .,
Mixstyle (Zhou et al., 2020c), DSU (Li et al., 2022b), and pAdaIN (Nuriel et al., 2021).
But they all rely on multiple source domains with large style variance to synthesize diverse domains.
Their effectiveness is unknown for object detection under real-world domain shifts.
In contrast, our method is surprisingly effective for the real-world applications of robust object detection.
P ROBLEM A NALYSIS
We aim at out-of-domain generalization for robust object detection.
Object detectors are trained on source domains and tested on different target domains unknown during training.
In this section, we analyze the shortcomings of existing domain synthesis approach on robust object detection to motivate our method.
Robust object detection is much harder than the classification domain generalization problem.
The latter usually contains multiple source domains with large style variance for training, and the pertinent training images are usually dominated by one target object in a relatively simple background.
By contrast, with small domain style variance and large context diversity, robust object detection usually has only one single source domain for training due to the expensive annotation cost, where the training images usually contain multiple objects in various scales in front of complex background.
Thus, existing classification DG methods cannot handle well the robust object detection problem, especially for the domain synthesis based methods.
We conduct empirical studies to demonstrate the shortcomings of existing domain synthesis methods on robust object detection, so as to motivate our Normalization Perturbation method on the robust object detection task.
The model is trained on the source domain and evaluated on the unseen target domain, i .
e ., Cityscapes (C) (Cordts et al., 2016)
→ Foggy Cityscapes (F) (Sakaridis et al., 2018), and
Sim10k (S) (Johnson-Roberson et al., 2017)
→ Cityscapes (C).
We use the Faster R-CNN (Ren et al., 2015) model (with ImageNet (Deng et al., 2009)-pretrained ResNet-50 (He et al., 2016) backbone) as the baseline.
The detection performance is evaluated using the mean average precision (mAP) metric with the threshold of 0.5.
For better comparisons, we also conduct classification DG experiments following the experiment setups in Mixstyle (Zhou et al., 2020c) and DSU (Li et al., 2022b), which are closely related methods to our NP method.
Refer to the appendix for full experimental details.
Small Domain Style Variance Restricts Feature-level Domain Synthesis.
To facilitate the method illustration, we rewrite the feature-level domain synthesis methods formulation from y =
σ new x − µσ + µ new ( x is the original feature with its channel mean µ and standard deviation σ , and y is the output feature with the synthesized new channel mean µ new and standard deviation σ new .)
to
[Equation]
[Table]
This equation can be represented as y =
x + ε x , where ε x = ( σ new σ − 1) x − σ new σ µ + µ new denotes the perturbation (random noise) given the original input feature x .
The perturbation term ε x depends on the given x and how methods perturb the feature x .
We use Mixstyle and DSU as the example to illustrate these feature statistic perturbation methods are suboptimal for robust object detection, as shown in Table 1.
When ( (cid:80) σ ( x ) → 0 , (cid:80) µ
( x ) → 0 ), both Mixstyle and DSU have ( σ 2 ( ε x ) → 0 , µ ( ε x ) → 0) , and thus fail to perturb the feature x where the method formulation y = x
+ ε x collapses to y = x .
Mixstyle and DSU rely on large variance in inter-image feature statistics ( (cid:80) σ ( x ) , (cid:80) µ ( x )) to produce sufficient feature noise σ 2 ( ε x ) .
Thus it works well on the classification DG datasets, e .
g ., PACS, which contains multiple source domains with large inter-image style variance.
Unfortunately, this assumption cannot always hold.
In robust object detection, there is usually only one source domain due to the expensive annotation cost, which means that only relatively small inter-image style variance can be present.
In this situation, Mixstyle and DSU cannot produce sufficient noises to perturb the feature map.
Table 2 shows the feature statistic variance of different datasets, i .
e ., Cityscapes, Sim10k and PACS (Li et al., 2017).
Both Cityscapes and Sim10k 2 datasets have much smaller feature statistic variance than
PACS.
DSU achieves SOTA on the classification DG benchmark PACS with Mixstyle performing comparably.
This is because the large style variance of multiple source domains enables these methods to produce decent feature noise for sufficient domain style synthesis.
By contrast, DSU and Mixstyle perform much worse than the robust object detection SOTA method CycConf (Wang et al., 2021).
Here, smaller feature statistic variance results in larger performance gap between DSU/Mixstyle and the SOTA CycConf method.
This performance gap indicates that DSU and Mixstyle do not have sufficient diversity in style synthesis when the inter-image style variance is too small.
Large Context Diversity Restricts Image-level Domain Synthesis.
The image-level domain synthesis methods (Jackson et al., 2019; Geirhos et al., 2018) leverage large-scale style image dataset to generate diverse stylized training images through style transfer techniques (Huang & Belongie, 2017).
Unfortunately, such image generation procedure may destroy the content structures underlying the original images.
Figure 4 (second column) shows significant lost of meaningful object details in the stylized training images under large context diversity.
The trained model substantially sacrifices the source domain performance where its generalization ability improvement is also limited.
The above two analyses show that existing classification DG methods cannot be directly applied for robust object detection, especially for the domain synthesis approach, which motivate us to propose a novel domain synthesis method to address the two issues, namely, small domain style variance and large context diversity, in robust object detection.
M ETHOD AND A NALYSIS
It has been known that feature channel statistics, e .
g ., mean and standard deviation, are closely related to image styles, where changing feature channel statistics can be regarded as implicitly changing the input image styles.
The Adaptive Instance Normalization (AdaIN) (Huang & Belongie, 2017) achieves arbitrary style transfer through the feature channel statistics normalization and transformation.
Given a mini-batch B of CNN features x ∈ R B × C × H × W with C channels and H × W spatial size from
[Figure]
OUT the content images, AdaIN can be formulated as:
[Equation]
where both { µ c , σ c } ∈ R B × C and { µ s , σ s } ∈ R B × C are feature channel statistics, estimated from the input content images and style images, respectively.
The normalized features y can be decoded into the stylized content images.
AdaIN provides a feasible and efficient way to implicitly change image styles in the feature space.
4. N ORMALIZATION P ERTURBATION M ETHOD
Our proposed method, Normalization Perturbation (NP), perturbs the feature channel statistics of training images by inserting random noises.
Formally, NP can be formulated as:
[Equation]
where { µ c , σ c } ∈ R B × C are the channel statistics, mean and standard deviation, estimated on the input features.
The { α, β } ∈ R B × C are random noises drawn from the Gaussian distribution.
This equation can be further simplified as:
[Equation]
As shown in Figure 2, NP is applied at shallow CNN layers (following the backbone stage 1 and stage 2).
NP is enabled with probability p only in the training stage.
Our NP method is fundamentally different from conventional normalization methods (Huang &
Belongie, 2017; Ioffe & Szegedy, 2015; Ulyanov et al., 2016; Wu & He, 2018), whose affine parameters { µ s , σ s } are learned from the training set or estimated from extra input style images.
While NP affine parameters { µ ⋆s , σ ⋆s } are obtained by perturbing the input feature channel statistics, they are obtained without relying on extra style inputs and are optimization-free.
The perturbed affine parameters can be regarded as the channel statistics corresponding to diverse latent domain styles, enabling models to learn domain-invariant representations and preventing them from style overfitting.
In NP, all channel statistics are randomly perturbed with the same noise distribution.
We further propose Normalization Perturbation Plus (NP+) to adaptively control the noise magnitude in different channels, based on the feature statistic variance across different images.
Such adaptive perturbation is motivated by the observation that some channels significantly vary as the domain style changes.
We thus apply more noise on these style-sensitive channels.
Specifically, we use the mini-batch of
B feature channel statistics µ c = { µ 1 c , ..., µ bc , ..., µ Bc } to compute the statistic variance ∆ ∈ R 1 × C : ∆ = 1 B (cid:80)
Bb =1 ( µ bc − ¯ µ c ) 2 , where ¯ µ c = 1 B (cid:80)
Bb =1 ( µ bc ) .
Then we use the normalized statistic variance δ = ∆ / max (∆) ∈ R 1 × C to control the injected noise magnitude for each channel:
[Equation]
where max is the maximum operation.
When applying NP+, we use the photometric data augmentation (Color Jittering, GrayScale, Gaussian Blur and Solarize, only for NP+ by default) to generate pseudo domain styles to facilitate the exploration on style-sensitive channels.
4. A DDRESSING R OBUST O BJECT D ETECTION
Our Normalization Perturbation can be implemented as a plug-and-play component in modern CNN models to effectively solve the domain style overfitting problem, where the domain style has low variance while large context diversity exists typical of robust object detection.
[Figure]
Effective Domain Blending.
Our NP can effectively blend feature channel statistics of different domains, corresponding to learning better domain-invariant representations.
Figure 3 shows that the Ours model trained with NP can effectively reduce the learned distribution distance between source and target domains, especially on deep CNN layers.
Compared to other methods, NP results in smaller cross-domain distribution distance and better generalization performance on target domains.
Diverse Latent Styles.
Our NP can effectively diversify latent styles.
Table 1 shows NP’s σ 2 ( ε x ) does not rely on inter-image statistic variance, and thus it provides more diversified style diversity during synthesis even under the small domain style variance typical of robust object detection.
On the other hand, although our NP+ method is also related to inter-image statistic variance, its σ 2 ( ε x ) = x 2 σ 2 α + [ (cid:80) µ ( x )] 2 ( σ 2 α + σ 2 β ) µ 21 has the x 2 σ 2 α term to ensure its scale is sufficiently large.
Our NP+ only partially uses inter-image style variance while achieving the best performance.
This indicates that proper usage of inter-image style variance alongside with our NP method is important for DG tasks, especially for robust object detection.
Sufficiently large perturbation variance σ 2 ( ε x ) is the key to synthesizing higher style diversity for achieving better domain generalization ability.
Table 4 shows the ablation study on noise hyperparameters of our NP method, reflecting different noise variance σ 2 ( ε x ) scale.
Our NP is also insensitive to the noise types and hyperparameters, performing well with Beta, Uniform and Gaussian noises with different α and β .
For better understanding, we map the NP perturbed feature channel statistics back into the image space using the feature inverting technique (Zeiler & Fergus, 2014).
Figure 4 shows that the generated latent styles effectively diversify the style scopes, covering various potential unseen domain styles in real-world environments, e .
g ., dawn, dusk, night times and foggy, rainy, snowy weathers.
Figure 4 also shows the stylized images of the image-level synthesis method (Geirhos et al., 2018), which transfers diverse styles into content images for model training.
But the generated stylized images fail to preserve the content structure details inherent in the original images.
In contrast, our
NP generated latent styles can often cover the potential synthesized domain styles generated by style randomization, while simultaneously maintaining high content fidelity to original images. [Table] [Table]
High Content Fidelity.
Our NP processes feature channel statistics while faithfully preserving image and feature spatial structures.
Note that image-level domain synthesis methods may destroy the content structures of the original images in the image generation procedure.
Besides, NP trains deep models with numerous content-style combinations in the high-dimensional feature space, which is much more efficient and effective than the image-level methods, whose styles are deterministic, limited and that their style augmentation is only performed on the low-dimensional image space.
Table 3(a) shows the comparisons between image- and feature-level domain synthesis methods.
Image-level style randomization (Geirhos et al., 2018) sacrifices the source domain performance due to its potential damage on image contents, although this method effectively improves the detection performance on unseen target domains.
Our NP method can be regarded as a variant of AdaIN, which has been widely shown (Huang & Belongie, 2017; An et al., 2021) being able to preserve well the image content information by keeping the normalized feature maps unchanged.
Benefiting Other Methods.
Our NP can be regarded as better generalization of existing SOTA methods.
Table 3(b) shows that NP can boost the performance of other classification DG methods on robust object detection, by perturbing the feature statistics to synthesize sufficient diverse domain styles.
Specifically, when combining our NP method with other DG methods, the source domain performance on Cityscapes (C) is almost unchanged while always better than the performance of baseline.
The generalization performance on target domains Foggy Cityscapes/BDD100k (F/B) is significantly boosted with 16.9 ↑ /5.
↑ , 14.6 ↑ /4.1 ↑ , 10.8 ↑ /3.
↑ AP50 improvements respectively for pAdaIN, Mixstyle and DSU, achieving better performance than the baseline model with NP method. Discussion on Data Augmentation.
The image-level data augmentation can be regarded as a special case of feature-level augmentation, if we treat the input image as a feature map with the low-dimensional RGB color channels.
Thus such data augmentation is less effective than our featurelevel perturbation, which is performed in the high-dimensional feature channels.
Table 3(b) shows that the data augmentation (DA) significantly improves the generalization performance on different target domains. Our NP method boosts more performance improvement than data augmentation in most settings. Our NP method also benefits from data augmentation and the resulting NP+ method achieves the best generalization performance of 46.3/32.8 AP50 on Foggy Cityscapes/BDD100k.  C OMPARISON E XPERIMENTS
We compare our method to object detection DG/UDA methods.
In the appendix, we perform extensive comparisons on semantic segmentation DG and classification DG tasks.
The experiment setting details are also included in the appendix.
5. R OBUST O BJECT D ETECTION
We follow CycConf (Wang et al., 2021) to train and evaluate models on the robust object detection benchmark.
Specifically, there are two settings: Domain Shift by Time of Day, where the model is [Table] trained on BDD100k (Yu et al., 2020) daytime/night train set and evaluated on the night/daytime val set, and Cross-Camera Domain Shift, where the model is trained on Waymo (Sun et al., 2020)
Front
Left/Right train set and evaluated on BDD100k night val set.
Table 5 shows our NP outperforms previous SOTA CycConf (Wang et al., 2021) on all domain shift settings, especially on the Waymo
Front Left/Right to BDD100k Night setting, where our NP improves the SOTA performance from 12.27/9.99 to 17.85/14.97 AP.
Our NP+ further boosts the performance to a new SOTA.
Note that our method is general without any assumption on the inputs, while CycConf is designed to operate on videos, requiring consecutive video frame pairs to form a time cycle.
5. U NSUPERVISED D OMAIN A DAPTIVE D ETECTION
performance on ResNet-50 backbone except † which are ResNet-101.
[Table]
Unsupervised domain adaptive object detection models are trained on labeled source domain and unlabeled target domain.
We consider two popular adaptation settings:
Sim10k (Johnson-Roberson et al., 2017) to Cityscapes (S → C) and Cityscapes (Cordts et al., 2016) to Foggy
Cityscapes (Sakaridis et al., 2018)
(C → F) adaptations.
Table 6 shows that our method significantly outperforms the UDA methods by a large margin even without accessing the target domain data , where the UDA methods (DA-Faster (Chen et al., 2018b),SC-DA (Zhu et al., 2019),ViSGA (Rezaeianaran et al., 2021), EPM (Hsu et al., 2020), and CycConf (Wang et al., 2021) with selfsupervised rotation)
all require unlabeled target domain images for model training.
We also apply closely related classification DG methods
(SFA (Li et al., 2021), pAdaIN (Nuriel et al., 2021), Mixstyle (Zhou et al., 2020c), and DSU
(Li et al., 2022b))
on our baseline for a fair comparison.
Our method substantially outperforms other DG methods on UDA object detection, especially on the C →
F setting, where Cityscapes (C) source domain has smaller domain style variance.
Our method also substantially outperforms BIN (Nam & Kim, 2018) and
IBN (Pan et al., 2018), whose normalization layer integration manner is fundamentally different from our normalization perturbation manner.
C ONCLUSION
SOTA classification DG methods perform unsatisfactorily on robust object detection, since they are limited by the small domain style variance and large context diversity in the object detection
DG task.
We propose Normalization Perturbation (NP) to perturb the channel statistics of source domain features to synthesize various latent styles.
NP is not limited by the small source domain style variance in the input images and does not destroy image structures.
The trained deep model can perceive diverse potential domains and thus generalizes well on unseen domains thanks to the learned domain-invariant representations.
Our NP method only relies on a single source domain to generalize on diverse real-world domains, which is effective and easy to implement, while boosting the performance of other classification DG methods on robust object detection.
Extensive analysis and experiments verify the effectiveness of our Normalization Perturbation.
A M ORE A NALYSIS AND D ISCUSSION A. M ORE I MPLEMENTATION D ETAILS
To compute [ (cid:80) µ ( x )] 2 and [ (cid:80) σ ( x )] 2 for the dataset shown in Table 2 of the main paper, we first shuffle the train set and then sample a batch of 64 images to extract their ResNet stage1 features, and finally we compute the variance of their feature channel statistics, i .
e ., mean and standard deviation, respectively.
We traverse all image batches of the train set and average all the computed variance values as the output.
A. M ORE P ROBLEM A NALYSIS
We perform further problem analysis for robust object detection.
We observe that the domain style overfitting problem is mainly caused by the biased distribution of low-level features learned in shallow
CNN layers.
Table 7 shows four Faster R-CNN models used for analysis and their performance on three datasets.
[Table]
The Baseline trained with our Normalization Perturbation.
[Table]
Biased Model Impedes Domain Generalization.
With training data under the same domain style, the learned model performs well in testing for data under the same distribution as the training data, [Table] Seg, Det with ability of grouping in-domain features together.
But the learned model tends to separate distinct domains and thus hardly generalizes from the source to target domain.
Figure 1 shows image feature channel statistics of the same domain are grouped together, while different domains are separated.
Figure 5 and Table 7 show that the biased distribution in the Baseline and Overfit models causes large domain feature statistic discrepancy which impedes model generalization to unseen domains.
Shallow CNN Features Matter for Generalization.
Figure 1 and Figure 5 show that shallow CNN layers exhibit larger domain feature statistic discrepancy.
Such discrepancy is propagated from the shallow to deep layers and finally results in the poor target domain performance.
The shallow CNN layers suffer more from severe biased distribution when they are further trained on the source domain.
Note in particular Figure 5 shows that the Overfit model has larger domain feature gaps on all layers.
Table 7 further shows quantitatively that this overfitting model generalizes worse on unseen target domains, while capable of producing better source domain performance.
Thus shallow CNN layers do matter for generalizing model to different domain styles, because they preserve more style information through encoding local structures, such as corner, edge, color and texture, which are closely relevant to styles (Zeiler & Fergus, 2014).
While the deep CNN layers encode more semantic information which are more insensitive to the style effect, if the model is trained on the biased shallow
CNN features, the deep layers cannot effectively calibrate the style-biased semantic information and thus the entire model overfits to the source domain.
Reducing Domain Style Overfitting.
Diverse training domains would help deep models to learn domain-invariant representations and thus reduce the domain style overfitting.
Our NP efficiently synthesizes diverse latent domain styles and effectively reduces any inherent domain style overfitting.
Figure 1 and Figure 5 show our NP significantly reduces the domain feature gap, especially in the shallow and deep CNN layers.
Table 7 shows that Ours model with NP generalizes well on unseen target domains while simultaneously keeping the source domain performance.
The imagelevel domain style synthesis method StyeRD also reduces domain style gaps and improves domain generalization.
However, as we show in the main paper, this method is not as desirable as ours.
A. M ORE M ETHOD C OMPARISONS
Table 8 summarizes the fundamental differences among relevant feature perturbation methods:
• SFA (Li et al., 2021) performs activation-wise feature perturbation by adding activation-wise noises
to features • pAdaIN (Nuriel et al., 2021) performs random feature statistic permutation by exchanging feature statistics of multiple images.
• Mixstyle
(Zhou et al., 2020c) performs random linear interpolation of feature statistics of two images.
• Our NP method performs random feature statistic extrapolation along random direction by perturbing single image feature statistics.
The activation-wise feature perturbation may destroy meaningful image contents, and thus our feature statistic perturbation method performs better than SFA because of the high content fidelity.
Both pAdaIN and Mixstyle may suffer from insufficient style diversity in synthesis when the interimage feature statistic variance is small.
In contrast, our NP method provides more diversified style diversity during synthesis because it does not rely on the inter-image style variance. [Figure] i [Figure]
t y l e R D
O u r
[Figure] [Figure] a s e li n [Figure]
O
[Figure]
e r f i t S t y l e
[Figure]
B a s e li n e O v e r f i t S t y l e R D
O u r
[Figure]
B a s e li n e O v e r f i t S t y l e R D
O u r
[Figure]
B a s e li n e O v e r f i
[Figure]
y l e
[Figure]
D O u
[Figure]
sc
[Figure]
[Figure]
[Figure]
[Figure] In most real-world scenarios [Figure] domain discrepancy is mainly caused by different styles, which is the fundamental assumption of our method and other feature statistic perturbation methods.
However, discrepancies can occur in object and background contents, e .
g ., automobiles in the 1950s and today are very different, even from the same car manufacturer.
Background content can vary across wide domains, such as forest, desert, countryside and city.
All the feature statistic perturbation based DG methods including ours cannot handle well such content discrepancy.
This is a significant future DG research direction to address the domain content discrepancy, especially in the age of globalization.
B M ORE EXPERIMENT RESULTS B. M ORE N ORMALIZATION P ERTURBATION A BLATION S TUDIES
NP probability.
Figure 6 shows how NP probability p affects the model performance.
When the
NP probability is higher, the generalization performance on target domains tends to be better, but the source domain performance will suffer when the probability is too large.
Interestingly, when
NP probability is lower than 0.5, NP helps model to perform better on the source domain, probably because NP provides desirable regularization and feature augmentation effect.
We set the probability p as 0.5 in our experiments to achieve the best balance, improving model generalization performance on unseen target domains and simultaneously keeping the source domain performance.
Perturbation types.
NP perturbs feature channel statistics, which is much better than the spatiallevel and activation-level (Li et al., 2021) perturbation as shown in Table 9(a).
This is because our channel-level perturbation fittingly solves the style overfitting problem, without affecting spatial content structures. [Table] [Table] [Table]
[Table]
Comparison to related methods.
As shown in Table 9(b) [Table] our method performs better than other methods.
While photometric data augmentation (DA) improves model generalization, its performance depends on the augmentation and dataset matching degree.
Other feature-level domain synthesis methods SN & CN (Tang et al., 2021) and Mixstyle (Zhou et al., 2020c) also work [Table] but are inferior to our method thanks to our diverse latent styles generated by the perturbation operation.
NP+.
NP+ adaptively applies heavier perturbation on style-sensitive channels containing larger channel statistic variance.
This is motivated by the observation that style information is mainly preserved at the style-sensitive channels.
As shown in Figure 7, we choose the style-sensitive/insensitive channels to perform style transfer, based on the channel statistic variance between the content and style features.
The style can be effectively transferred into the content image with only the 20% highest style-sensitive channels, while the remaining 80% channels with low channel statistic variance are style-insensitive and are hardly used to transfer styles.
Table 9(c) shows the NP+ performs slightly worse than the NP method without data augmentation because the in-domain images have minor style difference.
But when equipped with photometric data augmentation, the pseudo domain images enable NP+ to effectively find style-relevant channels and thus performs better than NP. NP Position Ablation.
Table 10 shows our NP performs best when applied in shallow CNN layers.
This is because shallow CNN layers are more style sensitive, which fundamentally affect the entire model training as discussed before.
NP Effectiveness on Advanced Techniques. Table 11 shows the results of applying InstaBoost (Fang et al., 2019) on Faster R-CNN and applying our NP method on YOLOv6 (Li et al., 2022a). Instaboost can consistently improve the detection performance on all datasets thanks to its object-level augmentation, but still heavily suffers from the domain shifts problem. Instaboost introduces random jittering to objects, which is orthogonal to our method focusing on perturbing feature channel statistics.
Thus our NP method can be fittingly combined with Instaboost to further boost the performance on all testing domains.
Moreover, our method is also orthogonal to other popular data augmentation methods, i .
e ., Mosaic (Bochkovskiy et al., 2020) and Mixup (Zhang et al., 2018).
We further conduct experiments on YOLOv6-M.
The powerful YOLOv6 model significantly improves the detection performance on all datasets.
But the performance improvement is mainly derived from the stronger backbone, better data augmentation (Mosaic and Mixup), and advanced detection techniques, which are all specifically designed for the in-domain object detection.
YOLOv6 still heavily suffers from the real-world domain shifts problem.
Our NP method can be effectively combined to YOLOv6 to further boost the performance on all datasets.
[Table]
[Table]
In summary
[Table]
our NP method can be fittingly combined with most advanced detection techniques to further boost the detection performance, e .
g ., popular data augmentation techniques and SOTA detectors.
NP Class-wise mAP.
Table 12 shows the class-wise mAP to evaluate our NP effectiveness on each class.
The model is trained on Cityscapes dataset and evaluated on Foggy Cityscaapes and BDD100k.
Our method consistently improves the generalization performance on all classes.
B. C LASSIFICATION D OMAIN G ENERALIZATION
We compare our method to other DG techniques on the classification domain generalization (DG) task, i .
e ., MMD-AAE (Li et al., 2018), CCSA (Motiian et al., 2017), JiGen (Carlucci et al., 2019a),
CrossGrad (Shankar et al., 2018), Epi-FCR (Li et al., 2019), Metareg (Balaji et al., 2018), L2AOT (Zhou et al., 2020b), Manifold Mixup (Verma et al., 2019), Cutout (DeVries & Taylor, 2017),
CutMix (Yun et al., 2019), Mixup (Zhang et al., 2018), DropBlock (Ghiasi et al., 2018), and DSU (Li et al., 2022b).
We follow MixStyle (Zhou et al., 2020c) to implement our method on the popular
PACS (Li et al., 2017) dataset.
Specifically, we use the MixStyle public codebase (MixStyle) to train and evaluate our method by directly replacing the mixstyle module with our NP/NP+ module, keeping all other settings unchanged.
We remove the default photometric data augmentation of NP+ for a fair comparison.
The model is trained on three domains and evaluated on the leave-out domain.
Table 13 shows our NP substantially improves the classification DG performance and our NP+ further boosts the performance to 84.0.
Note that the PACS domain shifts in classification DG are distinct from real-world domain shifts in dense preidiction tasks.
Although not specifically designed for classification DG, our method still performs better or comparable to previous classification DG methods thanks to our diverse latent styles generated by the perturbation operation.
B. S EMANTIC S EGMENTATION D OMAIN G ENERALIZATION
We follow the previous semantic segmentation domain generalization SOTA method RobustNet (Choi et al., 2021b) to train and evaluate our method.
The model is trained on GTAV/Cityscapes datasets, and evaluated on various datasets, i .
e ., GTAV (G) (Richter et al., 2016), Cityscapes (C) (Cordts et al., 2016), BDD100k (B) (Yu et al., 2020), Mapillary Vistas (M) (Neuhold et al., 2017), and Synthia (S)
(Ros et al., 2016).
We compare our method to UDA segmentation methods, i .
e .,
SW (Pan et al., 2019), IBN-Net (Pan et al., 2018), IterNorm (Huang et al., 2019), and ISW (Choi et al., 2021b).
We also compare our method to classification DG methods by applying them on our baseline for a fair [Table] w/ [Table]
+ MixStyle [Table] domain label [Table]
+ DSU
[Table]
+ NP (Ours)
[Table]
+ NP
[Table]
[Table]
[Table]
[Table]
[Table]
comparison, i [Table] e ., SFA (Li et al., 2021), pAdaIN (Nuriel et al., 2021), Mixstyle (Zhou et al., 2020c), and DSU
(Li et al., 2022b).
Table 14 shows that our method performs the best [Table]
B. S EMANTIC S EGMENTATION D OMAIN G ENERALIZATION ON ACDC DATASET
We evaluate our method on the recently proposed ACDC (Sakaridis et al., 2021) dataset [Table] which contains four adverse weather types: fog, night, rain, and snow.
Specifically, we directly apply our
DeepLabv3+ (Chen et al., 2018a) model trained on Cityscapes (Cordts et al., 2016) dataset on ACDC [Table] [Table] val set.
Table 15 shows our method significantly improves the semantic segmentation generalization performance under adverse weather domain shifts [Table]
C E XPERIMENT S ETTING D ETAILS
Given the simplicity [Table] anyone can implement NP/NP + with ease.
Our experiments are all based on public codebases.
While we will make our codes and data public upon acceptance, a competent practitioner can easily reproduce our results.
The following gives all the necessary details.
C. O BJECT D ETECTION
In our main paper, all object detection experiments (Table 2–6 of the main paper) are based on the
CycConf (Wang et al., 2021) public codebase (Confusion) and mmdetection (Chen et al., 2019) public codebase (MMDetection) (UDA object detection strong baseline).
We use their default settings in our experiments, except that we adjust the training iterations for some datasets to avoid overfitting.
Table 16 shows the object detection experiment setting details.
C. S EMANTIC S EGMENTATION
In the appendix, all semantic segmentation experiments (Table 14 and Table 15) are based on the
RobustNet (Choi et al., 2021b) public codebase (RobustNet).
We use their default settings to train models on Cityscapes (Cordts et al., 2016) and GTAV (Richter et al., 2016) in our experiments without any other changes.
Table 17 shows the semantic segmentation experiment setting details.
Specifically, for semantic segmentation models, we stop the gradient of our NP/NP+ for better domain generalization.
Such implementation also applies to other models whose shallow CNN layers are trainable.
C. D ATASETS
We evaluate our method on multiple datasets, under diverse real-world and synthetic domain shifts.
Table 18 shows the datasets details used in our experiments, including Cityscapes (Cordts et al., 2016), Foggy Cityscapes (Sakaridis et al., 2018), Sim10k (Johnson-Roberson et al., 2017),
BDD100k (Yu et al., 2020), Waymo (Sun et al., 2020), GTAV (Richter et al., 2016), Mapillary [Table]
Vistas (Neuhold et al., 2017), Synthia (Ros et al., 2016) [Table] and ACDC (Sakaridis et al., 2021).
For
Foggy Cityscapes (Sakaridis et al., 2018) [Table] we evaluate models on the highest fog intensity images (with least visibility).
ACDC (Sakaridis et al [Table] , 2021) dataset is used in the additional semantic segmentation domain generalization experiments.
[Table]
D N ORMALIZATION P ERTURBATION I MPLEMENTATION
As shown in the below code block
[Table]
our Normalization Perturbation (NP) and Normalization Perturbation Plus (NP+) can be easily inserted into popular CNN backbones
[Table]
import torch import random
# ResNet backbone equipped with our Normalization_Perturbation.
Class ResNet_backbone(): def __init__ (self, ...): ... def forward (x): p = random.random() x
= self.stage_1(x) if p < 0.5 and self.training: # Can be replaced by Normalization_Perturbation_Plus x = Normalization_Perturbation(x) x = self.stage_2(x) if p < 0.5 and self.training:
# Can be replaced by Normalization_Perturbation_Plus x = Normalization_Perturbation(x) x = self.stage_3(x) x = self.stage_4(x) x = self.stage_5(x) return x def Normalization_Perturbation (feat): # feat: input features of size (B, C, H, W)
feat_mean = feat.mean(( 2 , 3 ), keepdim= True ) # size: B, C, 1,ones_mat = torch.ones_like(feat_mean) alpha = torch.normal(ones_mat, 0.75 * ones_mat) # size: B, C, 1, 1 (cid:44) → beta = torch.normal(ones_mat, 0.75 * ones_mat) # size: B, C, 1, 1 (cid:44) → output = alpha * feat - alpha * feat_mean + beta * feat_mean return output # size: B, C, H, W def Normalization_Perturbation_Plus (feat): feat_mean = feat.mean
(( 2 , 3 ), keepdim= True ) ones_mat = torch.ones_like(feat_mean) zeros_mat = torch.zeros_like(feat_mean) mean_diff = torch.std(feat_mean, 0 , keepdim= True ) mean_scale = mean_diff / mean_diff.max() * 1.
alpha = torch.normal(ones_mat, 0.75 * ones_mat)
beta = 1 + torch.normal(zeros_mat, 0.75 * ones_mat) * mean_scale (cid:44) → output = alpha * feat - alpha * feat_mean + beta * feat_mean return output
When applying NP+ in the backbone, we apply photometric data augmentation to generate pseudo domain images to facilitate the domain style synthesis by exploring style-insensitive channels.
Specifically, we adopt the MoCoV3 (Chen* et al., 2021) data augmentation implementation in our method, as shown in the below code block.
from PIL import ImageFilter, ImageOps import random import torchvision.transforms as transforms photometric_data_augmentation =
[ transforms.
RandomApply([ transforms.
ColorJitter( 0.4 , 0.4 , 0.2 , 0.1 ) ], p= 0.8 ), transforms.
RandomGrayscale(p= 0.3 ), transforms. RandomApply([GaussianBlur([ .1 , 2. ])], p= 0.3 ), transforms. RandomApply([Solarize()], p= 0.3 ), ] class GaussianBlur (object): "Gaussian blur augmentation from SimCLR" def __init__ (self, sigma=[ .1 , 2. ]): self.sigma =
sigma def __call__ (self, x): sigma = random.uniform(self.sigma[ 0 ], self.sigma[ 1 ]) x = x.filter(ImageFilter. GaussianBlur(radius=sigma)) return x class Solarize (object): "Solarize augmentation from BYOL" def __call__ (self, x): return ImageOps.solarize(x)