T EST - TIME B ATCH S TATISTICS C ALIBRATION FOR C OVARIATE S HIFT
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks have a clear degradation when applying to the unseen environment due to the covariate shift.
Conventional approaches like domain adaptation requires the pre-collected target data for iterative training, which is impractical in real-world applications.
In this paper, we propose to adapt the deep models to the novel environment during inference.
An previous solution is test time normalization, which substitutes the source statistics in BN layers with the target batch statistics.
However, we show that test time normalization may potentially deteriorate the discriminative structures due to the mismatch between target batch statistics and source parameters.
To this end, we present a general formulation α -BN to calibrate the batch statistics by mixing up the source and target statistics for both alleviating the domain shift and preserving the discriminative structures.
Based on α -BN, we further present a novel loss function to form a uniﬁed test time adaptation framework CORE, which performs the pairwise class correlation online optimization.
Extensive experiments show that our approaches achieve the state-of-the-art performance on total twelve datasets from three topics, including model robustness to corruptions, domain generalization on image classiﬁcation and semantic segmentation.
Particularly, our α -BN improves 28.4% to 43.9% on
GTA5 → Cityscapes without any training, even outperforms the latest source-free domain adaptation method.
I NTRODUCTION
Deep neural networks (DNNs) achieve impressive success across various applications, but heavily rely on the independent and identical distribution (i.i.d.) assumption.
However, in real-world applications, the model is prone to encounter the novel instances.
For examples, an automatic pilot should have robust performance under different weather conditions.
Unfortunately, when applying DNNs to novel environment, the performance has a clear degradation due to the covariate shift (Ben-David et al., 2010), i.e., the test data distribution differs from the training distribution.
Domain adaptation (DA) is a promising alternative, which transfers the knowledge learned on labeled source domain to unlabeled target domain, where the data distribution is distinct (Long et al., 2015).
However, domain adaptation needs the pre-collected target data, which is not applicable.
Unlike DA, Domain generalization (DG) aims at training a general model from multiple source domains and generalizing to the unseen target domain.
DG is more challenging since the target domain is totally unseen during training.
Recently, another practical scenario named test-time adaptation (TTA) is proposed.
In TTA, the target data are not pre-collected for iterative training, but used for adapting the source-trained model during inference.
We show the comparison between different settings in
Table 1.
To make the comparison clear, we introduce two indicators.
Iterative training means the model is trained on the unlabeled target data iteratively.
Online training means the model parameters are updated during inference.
Noticing that a recent DG work (Pandey et al., 2021) adapts the model during inference, while most previous DG methods did not.
We call it as optimization-free TTA.
In this paper, we focus on the practical scenarios: DG and TTA.
One of the main approaches to adapt during inference is test-time normalization (T-BN)
(Nado et al., 2020).
T-BN re
-calculates the target batch statistics to replace the source statistics in BN layers during inference.
Motivated by T-BN, Wang et al.
(2021) proposed to perform test time adaptation by re-calculating the target batch statistics and updating the afﬁne parameters in BN layers with
[Table]
entropy minimization.
However, this paradigm has critical restrictions.
For representation learning, the discriminative representations are crucial for recognition task.
Substituting the source statistics with target statistics in BN layers will inevitably lead to a mismatch with the source-trained model parameters.
This mismatch will probably perturb the original discriminative structures.
Another limitation is the estimation error on target batch statistics.
The source statistics in BN layers are updated in a moving average manner during training time, while the target statistics are calculated in each batch during test time.
The statistics estimated in a batch introduces more errors in reﬂecting the domain characteristics.
A preliminary empirical investigation of the mentioned restrictions are shown in Fig.
1 and Table 2.
Motivated by the hidden restrictions of T-BN, we propose a more general method named α -BN to calibrate the batch statistics during inference.
Speciﬁcally, we mix up the source and target statistics in BN layers to both alleviate the domain shift and preserve the discriminative structures.
Equipped with α -BN, common DG models can be further improved without any training.
Based on α -BN, we further propose an uniﬁed test-time adaptation framework named CORE with an online optimization, which exploits the pairwise C lass c or r e lation to facilitate robust and accurate test time adaptation.
To sum up, we have following contributions:
We investigate two practical yet challenging transfer learning scenarios: domain generalization and test time adaptation, which release the requirement of pre-collected target domain data.
Motivated by the hidden restrictions of test time normalization, we present a general formulation α -BN for both alleviating domain shifts and preserving discriminative informations .
It can be seamlessly incorporated into mainstream deep neural networks to improve the generalization on unseen domains without any training.
Based on α -BN, we propose a uniﬁed framework CORE for test-time adaptation.
CORE optimizes the pairwise class correlation in an unsupervised online learning manner.
We conduct numerous experiments on total twelve datasets from three topics: robustness to corruptions, DG on image classiﬁcation and DG on semantic segmentations.
The empirical results show that both α -BN and CORE achieve the state-of-the-art (SoTA) performance in their respective communities.
The result on GTA5 → Cityscapes, for instance, is improved
from 28.4% to 43.9% without any training, which even outperforms the SoTA source-free
DA method.
R ELATED W ORKS
Domain Adaptation and Generalization Domain adaptation (DA) enables transferring the knowledge from source domain to target domain by jointly optimizing both labeled source data and unlabeled target data.
This paradigm has gained a lot of attention in the last decade and various methods are proposed, which can be roughly divided into three categories: metric learning (Long et al., 2015;
Sun & Saenko, 2016; Kang et al., 2019; Li et al., 2020), adversarial training (Ganin et al., 2016;
Long et al., 2018;
Tsai et al., 2018) and self-training Zou et al.
; Liang et al.
Ge et al.
However, training with the large amounts source data is inefﬁcient and impractical in many real-world applications.
To address this issue, source-free DA is proposed, which adapts to the target domain with target data and the model pre-trained on source domain.
Chidlovskii et al.
(2016) are the ﬁrst to investigate source-free DA and proposed a denoising auto-encoder for adaptation.
SHOT (Liang et al., 2020) is another representative work, which proposed information maximization loss and clustering-based pseudo-labeling.
However, despite getting rid of the source domain data, source-free DA also requires the pre-collected target domain data for iterative training, limiting its application scenarios.
Therefore, domain generalization (DG) and fully test-time adaptation (TTA) are proposed (Let us elaborate TTA in the next paragraph).
Domain generalization aims at generalizing the model trained on a (multiple) source domain (s) to the unseen target domain directly.
Recently, various DG methods are proposed including domain-invariant representation learning (Zhao et al., 2020; Matsuura &
Harada, 2020), proxy tasks (Carlucci et al., 2019; Huang et al., 2020), augmentations (Volpi et al., 2018; Zhou et al., 2021), meta-learning (Li et al., 2018; Balaji et al., 2018) and so on.
However, Gulrajani & Lopez-Paz (2020) provided a DG benchmark named DomainBed for fair comparison and found a well-implemented empirical risk minimization (ERM) model outperforms most DG methods.
Recently, Pandey et al.
(2021) proposed label-preserving target projections during inference time for DG.
This work differs from most previous works, which focus on learning from source domains, while it performs optimization-free TTA during inference.
Our proposed α -BN also belongs to it, but the methods differ.
Fully Test-time Adaptation Fully test-time adaptation is proposed by Wang et al.
(2021), which adapts the model to target domain by online training.
TTA can be seen as a compromise between source-free DA and DG.
Different from source-free DA, TTA does not require the pre-collected target domain data but trains on the target data in an online manner.
Also, different from DG, TTA allows optimization during test, which introduces additional test time cost but usually guarantees better performance.
Therefore, TTA is a really practical scenario since it gets rid of iterative training and yields better performance compared to generalizing to the new environment directly.
Wang et al.
(2021) proposed TENT to achieve TTA by feature modulation.
Feature modulation contains two steps: test-time normalization (we will elaborate it in the next paragraph.)
and afﬁne parameters optimization by entropy minimization, which is a widely-used regularization term on DA and semisupervised learning (Grandvalet et al., 2005).
Another similar scenario is test-time training, which optimizes the networks before making a prediction during inference Sun et al.
Since this setting is weaker than TTA, we mainly talk about TTA in this paper.
Normalization and Adaptation Batch normalization (BN) is widely-used in DNNs nowadays for stable training and fast converge.
BN is originally proposed to alleviate the internal covariate shift during training a very deep neural networks Ioffe & Szegedy (2015).
Recently, Schneider et al.
(2020) and Nado et al.
(2020) discovered that updating the batch statistics during testing improves the robustness to common corruptions.
In this paper, we call it as test-time normalization.
Similar to their works, Wang et al. (2021) proposed feature modulation, which also updates the batch statistics rather than freezes them.
The key insight for these methods is that batch statistics are closely related to the domain characteristics (Li et al., 2016; Pan et al., 2018).
Based on this ﬁnding, Jeon et al.
(2021) and Zhou et al.
(2021) proposed to synthesise the novel domain styles to facilitate generalization.
Another kind of work focuses on instance normalization (Ulyanov et al., 2016).
Similarly,
Huang & Belongie (2017) found the instance-wise statistics are related to instance characteristics (i.e., image styles).
Motivated by this ﬁnding, Zhou et al.
(2021) proposed to generate novel
in- stances by mixing instance-level feature statistics to enhance out-of-distribution generalization.
It is worth noticing that the aforementioned DG method can be summarized as an augmentation-based technique.
Different from them, we propose a post-processing method to calibrate the batch statistics on target domain during test time.
U NDERSTANDING T EST - TIME N ORMALIZATION
[Table]
Test-time normalization ( T-BN ) re-calculates the batch statistics on target domain during inference.
Since the batch statistics are closely related to the domain characteristics (Li et al., 2016; Pan et al., 2018), T-BN adapts the model to target domain explicitly.
However, every coin has two sides, and
T-BN is not a free lunch.
During training on the source domain, the model parameters are associated with the source statistics.
Therefore, substituting the source statistics with the target ones inevitably results in a mismatch with the model parameters, which leads to the degradation of discriminative structures.
In short, T-BN alleviates the negative effects caused by domain shift, but perturbs the discriminative structures.
We report the averaged accuracy of “Source” and “T-BN” on three DG classiﬁcation benchmarks in Table 2.
We observe that the accuracy of “T-BN” is consistently lower than “Source”, revealing that substituting the source statistics by the estimated target batch statistics directly is not effective in generalizing to the new environment.
To further understand T-BN, we begin with the following two perspectives.
Error of ideal target hypothesis.
Based on domain adaptation theory (Ben-David et al., 2010), the domain shift can be reﬂected by the error of the ideal target hypothesis based on the target representations learned by source model.
“target representation” means the representations are obtained on the target domain data.
To obtain the ideal target hypothesis, we train a new classiﬁer over the target representations with corresponding labels.
Two methods are compared: “Source” and “T-BN”.
“Source” obtains the target representations by the source model directly, while “T-BN” performs test-time normalization.
The error of the ideal target hypothesis is shown in Fig.
1 (b).
As expected, the error of the ideal target hypothesis in “T-BN” is lower over all tasks.
It is worth noticing that the only difference between them is that T-BN normalizes the BN layer inputs by the target statistics rather than the source statistics, and others remain consistent (e.g., the same network architecture and the same network parameters).
Therefore, we reasonably postulate that T-BN alleviates the domain shift, which results in the lower error of the ideal target hypothesis.
Representation visualization.
Discriminative representation learning is essential for recognition task.
The discriminative representation satisﬁes two basic principles: intra-class tightness and interclass separation.
To qualitatively verify that how T-BN affects the learned representations, we visualize the target representations in Fig.
1 (c).
The variance of each category cluster in “T-BN” is signiﬁcantly larger compared to “Source”, which indicates the discriminative structures are injured due to the mismatch between target statistics and source model parameters.
T EST - TIME B ATCH S TATISTICS C ALIBRATION
Let { µ ( i ) s , σ ( i ) s } be the source statistics in i -th BN layers.
After training on the source domain data, the BN statistics are always ﬁxed for stable inference.
However, when encountering the data with distinct distribution, the model performance usually has a clear degradation due to the covariate shift.
T-BN re
-calculates the target statistics { µ ( i ) t , σ ( i ) t } to replace the ﬁxed BN statistics, which alleviates domain shift but perturbs the discriminative structures.
In this paper, we present a general formulation α -BN to generalize to new domains while preserve the discriminative structures.
α -BN considers both source and target statistics.
More speciﬁcally, α -BN mixes the source and target statistics during test time.
During test time, given an target input batch x t , α -BN re-calculates the target batch statistics { µ ( i ) t , σ ( i ) t } before forwarding i -th BN layer.
Then, α -BN calibrates the batch statistics as:
[Equation]
where α is a hyper-parameter to balance the source and target statistics.
Similar formulation is also proposed by Schneider et al.
(2020) for alleviating the estimated error caused by small batch size.
However, we discover that even with a large batch size (e.g., 200), T-BN also yields inferior performance on the large distribution shift (e.g., Ofﬁce-Home).
In practice, we set α to 0.9 for classiﬁcation tasks, and 0.7 for segmentation tasks on domain generalization benchmarks.
Noticing that α -BN is a post-processing method, and can be easily incorporated into mainstream neural networks to enhance the generalization performance on unseen domains.
Based on α -BN, we adopt the class correlation optimization (Jin et al., 2020) for robust and accurate test-time adaptation:
[Equation]
where p .j is the averaged softmax output, which indicates the probabilities that the samples in the mini-batch belongs to j -th class.
p (cid:62)· j p · j (cid:48) depicts the correlation between j -th class and j (cid:48)
-th class.
Minimizing the class correlation leads to a more conﬁdent and accurate predictions like other regularization terms (e.g., entropy minimization).
However, we theoretically show that CORE loss prevents the easy samples dominate the learning procedure in Appendix C, thus providing a more effective optimization-based test-time adaptation framework.
In short, we propose a uniﬁed framework named CORE for test-time adaptation based on α -BN and class correlation.
Firstly, we calibrate the batch statistics with proposed α -BN.
Then, we only optimize the afﬁne parameters in BN layers with the loss function presented in Eq.
Noticing that CORE is fully unsupervised, and only updates the parameters once in each test batch.
The test data will not be replayed just like the assembly line products.
E XPERIMENTS 5. D ATASETS AND T ASK D ESIGNS
We evaluate our proposed method on 12 datasets, including three categories: robustness to corruptions, DG on image classiﬁcation and DG on semantic segmentation.
Topic 1: Robustness to corruptions.
We evaluate the robustness on CIFAR10-C, CIFAR100-C and
ImageNet-C. For each dataset, 15 types of algorithmically generated corruptions with ﬁve levels of severity are included.
For CIFAR10/100-C, we evaluate models on the highest severity corruptions.
For ImageNet-C, the results are averaged on ﬁve levels of severity, i.e., 75 distinct corruptions.
Topic 2: DG on image classiﬁcation.
We evaluate our method on four datasets: VLCS, PACS,
Ofﬁce-Home and DomainNet as suggested by DomainBed (Gulrajani & Lopez-Paz, 2020).
Topic 3: DG on semantic segmentation.
We adopts two synthetic datasets (i.e., GTA5 and SYNTHIA) and three real-world datasets (i.e., Cityscapes, BDD-100K, Mapillary).
More details about these datasets are provided in Appendix.
A. Based on these datasets, we conduct numerous tasks with various base models:
ResNet (He et al., 2016), WideResNet (Zagoruyko &
Komodakis, 2016) with AugMix (Hendrycks* et al., 2020) and DeepLabV3 (Chen et al., 2017).
The task design is shown in Table 3.
5. I MPLEMENTATION D ETAILS AND B ASELINES
To guarantee the reproducibility of our results, we implement the proposed methods on the widelyused benchmarks.
We use RobustBench (Croce et al., 2020) for Topic 1, DomainBed (Gulrajani &
Lopez-Paz, 2020) for Topic 2 and RobustNet (Choi et al., 2021) for Topic 3.
Since the proposed αBN is training-free, we illustrate the implementation details of test-time adaptation method CORE.
[Table]
[Figure]
[Table]
[Figure]
[Table]
We use SGD optimizer for ImageNet-C, and Adam optimizer (Kingma & Ba, 2015) for remaining tasks.
We set batch size (BS) as 200, learning rate (LR) as 0.001 for CIFAR10/100, which is consistent with TENT’s settings 1 .
For ImageNet-C, we set BS=64, LR=2.5e-4 for fair comparison with other methods.
For DG on image classiﬁcation, we set BS=64, LR=1e-4 for all tasks except
DomainNet, on which we reduce the LR to 1e-5.
To verify the effectiveness of proposed two methods: α -BN and CORE, we compare them with the state-of-the-art methods on each research community.
For Topic 1, test-time normalization (TBN) (Nado et al., 2020) and test-time adaptation by entropy minimization (TENT) (Wang et al., 2021) are noteworthy approaches.
For Topic 2, we incorporate α -BN with two baselines: Empirical
Risk Minimization (ERM) and CORrelation ALignment (CORAL) (Sun & Saenko, 2016).
For
Topic 3, we apply α -BN to the simplest method ERM to enhance the generalization on unseen domains.
5. R OBUSTNESS TO C ORRUPTIONS
[Table]
The results of CIFAR10-C and CIFAR100-C are shown in Table 5.
Our proposed loss consistently outperforms all existing approaches under TTA setting, which demonstrates the effectiveness of our method.
To verify the performance on large-scale dataset, we evaluate our method on ImageNet-C, whose results are shown in Table 4.
In the largest dataset ImageNet-C, we achieve a new state-of-theart: 42.5% mean error over 75 tasks, which proves the superiority of our method on improving the robustness to corruption by test-time adaptation.
It is worth noticing that we optimize the afﬁne parameters in BN layers by only one iteration in an online manner.
Therefore, the improvement is signiﬁcant and the proposed framework CORE shows clear advantages over previous state-of-the-arts like TENT.
[Table]
DG ON I MAGE C LASSIFICATION
In this topic, there are two proposed methods: ‘ERM+ α -BN’ and ‘ERM+CORE’ with different evaluation protocols.
Firstly, the comparison between ‘ERM+ α -BN’ and other DG methods is fair since α -BN belongs to optimization-free TTA that only calibrates the batch statistics during inference without any training.
Since ‘ERM+CORE’ belongs to optimization-based TTA, we compare it with previous state-of-the-art work TENT.
Noticing that TENT is based on T-BN but T-BN has inferior performance on large distribution shift (see Table 2), we equip TENT with the proposed α -BN for better comparison with our CORE.
To emphasize the proposed α -BN and CORE are modelagnostic, we also use CORAL (Sun & Saenko, 2016) as another baseline.
The results are shown in
Table 6.
α -BN improves the performance on unseen target domain.
Domain generalization is a really challenging task and many recent proposed methods have few improvements or even degradations compared to ERM.
Equipped with our proposed α -BN, the performance on unseen target domain is consistently improved.
For instance, ERM+ α -BN reaches 68.8% and CORAL+ α -BN reaches 69.4% averaged accuracy over four DG benchmarks.
CORE outperforms TENT with robust performance.
Equipped with α -BN, we compare the proposed method CORE with TENT, the state-of-the-art algorithm in test-time adaptation.
CORE consistently outperforms TENT over four benchmarks, which veriﬁes the effectiveness of our method.
Speciﬁcally, CORE surpass TENT by considerable margins of 2.4% and 1.7% averaged accuracy on
ERM and CORAL baselines.
Noticing that test-time adaptation only updates the parameters once, thus the improvement is signiﬁcant.
Another interesting ﬁnding is that CORE surpasses TENT by 8.4% and 5.6% accuracy on the most challenging dataset DomainNet.
5. DG ON SEMANTIC SEGMENTATION
To evaluate the universality of proposed methods, we conduct DG experiments on semantic segmentation with ﬁve large-scale datasets: GTA5, SYNTHIA, Cityscapes, BDD-100K, Mapillary.
The results are shown in Table 7, and the segmentation visualization is presented in Fig.
α -BN is embarrassingly simple but achieves state-of-the-art performance.
Previous state-ofthe-art method ISW proposed instance selective whitening to remove the domain-speciﬁc styles, and reaches 40.0% mIoU over all tasks.
The popular IBN-Net achieves 38.2% mIoU by utilizing both batch normalization and instance normalization.
Different from previous works, we propose a post-processing method named α -BN, which adapts the batch statistics during test time.
Equipped with α -BN, the baseline ERM reaches 40.7% mIoU over total nine tasks, yielding a new state-ofthe-art on domain generalization.
[Table]
T-BN 27.
39.1 α -BN(ours) 28.
Image
ERM (Source only)
ERM
[Figure]
BN
Ground truth α -BN even outperforms the source-free DA method without any training.
Source-free
DA (Liang et al., 2020) becomes a popular topic in DA community recently.
For semantic segmentation, SFDA (Liu et al., 2021) shares similar backbone as ours, and reaches 43.2% mIoU with the 34.1% mIoU ERM baseline on GTA5 → Cityscapes.
With weaker ERM baseline (28.3% mIoU), the proposed α -BN even achieves better performance (43.9% mIoU).
A NALYSIS
α -BN introduce little additional inference time.
To evaluate the efﬁciency of proposed α -BN, we calculate the wall-clock time on GTA5 → Cityscapes with the same running environments for ﬁve times.
The averaged wall-clock times for vanilla inference and α -BN are 72.84s and 80.94s, respectively.
With little additional inference time cost (i.e., additional 0.0158s on each image), the proposed α -BN brings 15.5% mIoU improvement on GTA5 → Cityscapes.
Analysis of batch size, parameter sensitivity and LogME score.
Test-time batch size.
Fig 3 (a) shows the results under different batch size on four DG segmentation tasks.
We observe that the performance is robust to test-time batch size (BS).
For classiﬁcation task, the observation is similar to (Schneider et al., 2020): the performance begins robust when BS ≥ 64.
The hyper-parameter α is robust to task.
There are only one hyper-parameter in our proposed α -BN.
To evaluate the parameter sensitivity, we illustrate the results on four segmentation tasks with different values of α in Fig.
3 (b).
We observe that α = 0 .
7 is a great choice for all DG segmentation tasks, and we also observe that α = 0 .
9 is robust for DG classiﬁcation tasks.
We also present the comparison with other methods in Fig.
3 (c).
The mIoU in Fig.
3 (c) is averaged on four tasks { GTA5 } → { SYNTHIA, Cityscapes, BDD-100K, Mapillary } .
We observe that α -BN ( α = 0.7) outperforms all alternatives.
It is worth noticing that α -BN is training-free, and only introduces little additional inference time based on the ERM mdoel, while other methods like ISW has additional loss functions and augmentations.
α -BN gains more transferable representations.
The LogME score is a practical assessment for the transferability of representations (You et al., 2021).
Fig.
3 (d) shows the LogME score on tasks { P,C,S } → { A } , { P,A,C } → { S } with the obtained representations of Source, T-BN and the proposed α -BN.
We observe that the LogME score on α -BN representations is larger than the LogME score on Source and T-BN representations.
The ﬁnding implies that α -BN gains more transferable representations.
α -BN reaches statistical signiﬁcance.
To verify the statistical signiﬁcance of α -BN, we conduct the McNemars Test (Dietterich, 1998) to show the signiﬁcant improvement brought by the proposed α -BN contrast to the baseline ERM.
The p -values for four tasks (the order of tasks is the same as Table B.4) on Ofﬁce-Home are 3.4 × 10 − 5 , 2.4 × 10 − 13 , 0.0012 and 0.0016, respectively.
As is widely acknowledged, we set the signiﬁcance threshold as 0.05.
Therefore, the signiﬁcance test indicates our α -BN brings signiﬁcant improvement to the common baselines like ERM.
We also conduct statistical signiﬁcance test on other benchmarks.
The result with ‘*’ indicates that it is statistical signiﬁcant.
D ISCUSSION
Why CORE ( α -BN+CORE loss) outperforms TENT (T-BN+Entropy loss)?
Firstly, T-BN leads to a mismatch between target statistics and source-learned parameters, while α -BN balances the covariate shift alleviation and discriminative structure preservation.
More essentially, α -BN provides a better initialization of BN parameters.
However, we show that a better initialization is important for test-time adaptation (see Table 2, 8 and 9).
Since the supervision signals come from the model predictions, a better initialization leads to a better supervision signals in the online unsupervised learning setting.
In Appendix C, we provide a theoretical insight on why CORE loss outperforms
Entropy loss.
Compare to Entropy loss, CORE loss has much smaller gradient on the easy samples, preventing easy samples dominate the learning procedure.
Extensive empirical results further demonstrate it (see Table 4, 5, 6, 8 and 9).
α -BN v.s. T-BN From Figure 3 (b,c), α -BN has limited improvement compared to T-BN, which is inconsistent with the ﬁnding provided in Table 2.
In fact, they are not contradictory.
Previous work TENT has shown that T-BN performs well on the synthetic-to-real adaptation (Figure 3 (b,c)).
However, in this paper, we ﬁrst ﬁnd that T-BN is not suitable for the nature distribution shift of real-world datasets (Table 2).
We think the latter one is more important since distribution shift is a common phenomenon in real world.
We have also provide the real-to-real adaptation results on semantic segmentation in Table 7.
We observe that α -BN outperforms T-BN by a margin (3. mIoU).
How to determine the value of α for a new dataset?
The value of α plays an essential role in α -BN and CORE.
In this paper, we perform grid search, and ﬁnd an empirical rule: α = 0 .
for classiﬁcation and α = 0 .
7 for semantic segmentation (see Figure 3 and 5).
Actually, this setting is not an optimal but an acceptable choice for each task, and always outperforms the original T-BN.
We think developing the learnable α -BN with an automatically adjusted α is an inspiring direction for future work.
C ONCLUSION
We present a general formulation named α -BN to generalize deep models to the novel environments.
By considering both source and target statistics, α -BN alleviates the covariate shift but preserves discriminative structures.
Based on α -BN, we further propose an uniﬁed test-time adaptation approach
CORE, which provides a robust optimization by investigating the pair-wise correlations.
Different from conventional domain adaptation approaches, our α -BN and CORE are online algorithms, which adapts to the target domain during inference.
Comprehensive experiments on twelve datasets from three research topics validate the effectiveness and efﬁciency of our methods.
A D ATASETS D ESCRIPTION
We evaluate our proposed method on 12 datasets from three communities: robustness to corruptions (CIFAR10/100-C, ImageNet-C), DG on image classiﬁcation (VLCS, PACS, Ofﬁce-Home, Terra
Incognita, DomainNet) and DG on semantic segmentation (GTA5, SYNTHIA, Cityscapes, BDD100K, Mapillary).
Some examples are illustrated in Fig.
Here are the detailed descriptions:
CIFAR10/100-C (Krizhevsky et al., 2009) has 50000 training samples and 10000 test samples with 10/100 classes.
The corruptions include gaussian noise, shot noise, impulse noise, defocus blur,
GTA
Cityscapes
Mapillary
BDD-100K
I m a g e N e tC
D o m a i n N e t
ImageNet-C (Russakovsky et al., 2015; Hendrycks & Dietterich, 2019) has 1.2 million training samples and 50000 validation samples with 1000 classes.
The corruption categories are the same as
CIFAR10/100-C.
VLCS (Fang et al., 2013) contains 10,729 examples from four domains: Caltech101 (C), LabelMe (L), SUN09 (S), VOC2007 (V), with 5 classes.
PACS (Li et al., 2017) contains 9,991 examples from four domains: Art (A), Cartoons (C), Photos (P), Sketches (S), with 7 classes.
Ofﬁce-Home (Venkateswara et al., 2017) contains 15,588 examples from four domains: Art (A),
Clipart (C), Product (P), Real (R), with 65 classes.
DomainNet (Peng et al., 2019) is the largest dataset on DG community, and contains 586,575 examples from six domains: clipart (C), infograph (I), painting (P), quickdraw (Q), real (R), sketch (S), with 345 classes.
GTA5 (Richter et al., 2016) is a large-scale synthetic dataset, which contains 24,966 high-resolution images collected from the game, Grand Theft Auto V (GTA5), and the corresponding ground-truth segmentation maps are generated by computer graphics.
It shares 19 classes with Cityscapes.
SYNTHIA (Ros et al., 2016) is also a synthetic dataset rendered from a virtual city and comes with pixel-level segmentation annotations.
We adopt the subset of it, which is called SYNTHIA-RANDCITYSCAPES.
This subset contains 9400 images and shares 16 common classes with Cityscapes dataset.
Cityscapes (Cordts et al., 2016) is the widely-used real-world dataset, which contains 3,450 highresolution (i.e., 2048 × 1024) urban driving scene images.
BDD-100K (Yu et al., 2020) is a recent real-world dataset containing 8000 urban scene images (7000 images for training and 1000 images for validation) with resolution of 1280 × 720.
Mapillary (Neuhold et al., 2017) is another recent real-world dataset containing 25,000 street-view images with a minimum resolution of 1920 × 1080.
B F ULL R ESULTS
We implement our methods with Pytorch on a single NVIDIA RTX 3090 GPU.
All results in this section are obtained by our implementations.
B. I MAGE N ET -C
All results are statistical signiﬁcant compared to baseline.
Corruptions
Level
Level Level
Level Level
Mean
Guass.
Shot 32.
Impulse 36.
Defocus 34.
Glass 33.
Motion 30.
Zoom 34.
Snow 34. Frost 34.
Fog 30.
Bright 26.
Contrast 28.
Elastic 30.
Pixel 28.
JPEG 30.
Mean 31.
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
CORAL+CORE 60.8* 21.3* 48.1* 17.1* 61.5* 53.0* 43.
B. DG ON SEMANTIC SEGMENTATION
[Table]
r o a d s i d e .
bu il d .
w a ll * f e n ce * po l e * li gh t s i gn v e g e .
t e rr .
s ky p e r s .
r
i d e r ca r t r u c k bu s t r a i n m o t
o r b
i k
e m I
o U
[Table]
i
[Table]
C T HEORETICAL I NSIGHT ON CORE L OSS
In this paper, we adopt the CORE loss with class correlation optimization rather than the Entropy loss suggested by TENT (Wang et al., 2021).
From Table 4, 5 and 6, we observe that CORE loss outperforms Entropy loss on a wide range of benchmarks.
To better understand CORE loss, we provide a theoretical insight on why CORE loss outperforms Entropy loss.
Firstly, the CORE loss in Eq.
(3) is equal to:
[Equation]
where p j denotes the probability of class j .
The entropy minimization loss is deﬁned as:
[Equation]
Consider a binary classiﬁcation problem, we can obtain the values and the gradients with respect to
Entropy loss and CORE loss:
[Equation]
From Eq.
(7) and Eq.
(9), the absolute value of Entropy loss gradient on the easy sample ( p > 0 .or
p < 0 . 1 ) is much larger than the absolute value of CORE loss gradient.
In other words, Entropy loss pays too much attention to the easy samples.
D M ORE E XPERIMENTAL R ESULT AND A NALYSIS D. A BLATION S TUDY
CORE contains two components: α -BN and Class Correlation Optimization (COO), while TENT also contains two components: T-BN and Entropy Minimization (EM).
In Table 6, we compare αBN + COO with α -BN + EM, and show that COO signiﬁcantly outperforms EM.
In this section, we further evaluate remaining two variants: T-BN + COO and T-BN + EM (Original TENT).
From
Table 8, we observe that T-BN works badly on the nature distribution shift of real-world datasets.
Equipped with T-BN, COO achieves similar performance with EM, and both of them are worse than the ERM baseline.
This ablation study indicates that α -BN is really important for adapting the model to a novel target domain.
[Table]
D. R ESULT ON GTA5 → C ITYSCAPES WITH A NOTHER B ACKBONE
ResNet101+DeepLabV2 is the popular backbone adopted by many domain adaptation methods.
To better evaluate our α -BN
[Table]
we report the performance with this backbone on the most popular transfer task: GTA5 → Cityscapes.
The mIoU of ERM baseline is 35.7%.
α -BN achieves 44.7% mIoU, which is also competitive state-of-the-art DG method FSDR (Huang et al., 2021) with a 44.8% mIoU.
[Table]
D. R ESULT OF O PTIMIZATION - BASED TTA METHODS ON GTA5 → C ITYSCAPES
We experiment the optimization-based TTA methods TENT, CORE and their variants on the semantic segmentation task GTA5 → Cityscapes.
We set batch size as 3 and learning rate as 1ewith Adam optimization.
The results are shown in Table 9.
We observe that COO outperforms EM with both T-BN and α -BN.
α -BN+COO achieves the best performance, demonstrating effectiveness of each component of CORE.
However, the single α -BN has already achieves 43.9% mIoU, COO appears to have limited improvement, and EM even leads to negative transfer.
Recent advances in domain adaptive semantic segmentation also revealed that the universal unsupervised loss is not suitable for semantic segmentation, since this task has many intrinsic challenges like class-imbalance, boundary confusion and so on.
Inspired by that self-training has become the dominant method in domain adaptive semantic segmentation, we think a well-designed self-training loss equipped with our α -BN is a hopeful direction for test-time adaptive semantic segmentation.
D. P ARAMETER S ENSITIVITY OF H YPER - PARAMETER α
The value of hyper-parameter α plays an essential role in α -BN and CORE.
In this paper, we determine the value of α through grid search: from 0 to 1 with an interval of 0.1.
The result of gird search on semantic segmentation is illustrated in Figure.
For image classiﬁcation, we ﬁnd the performance usually drops when α is decreasing from 0.9 to 0.
Actually, α = 0 .
9 is not the optimal choice for each task, but an acceptable choice.
We further provide the result on image classiﬁcation in Figure.
We think developing the learnable α -BN with an automatically adjusted α is an inspiring direction for future work.
[Figure]
G T
E R
M α - B
N
G T
E R
M α - B
N
G T
E R
M α - B
N