Robust Continual Test-time Adaptation: Instance-aware BN and Prediction-balanced Memory
s
AfﬁliationAddress email
Abstract
Test-time adaptation (TTA) is an emerging paradigm that addresses distributionalshifts between training and testing phases without additional data acquisitionor labeling cost; only unlabeled test data streams are used for continual modeladaptation.
Previous TTA schemes assume that the test samples are independentand identically distributed (i.i.d.), even though they are often temporally correlated(non-i.i.d.) in application scenarios, e.g., autonomous driving.
We discover thatmost existing TTA methods fail dramatically under such scenarios.
Motivatedby this, we present a new test-time adaptation scheme that is robust against noni.i.d.
test data streams.
Our novelty is mainly two-fold: (a) Instance-Aware Batch
Normalization (IABN) that corrects normalization for out-of-distribution samples,and (b) Prediction-balanced Reservoir Sampling (PBRS) that simulates i.i.d.
datastream from non-i.i.d.
stream in a class-balanced manner.
Our evaluation withvarious datasets, including real-world non-i.i.d.
streams, demonstrates that theproposed robust TTA not only outperforms state-of-the-art TTA algorithms in thenon-i.i.d.
setting, but also achieves comparable performance to those algorithmsunder the i.i.d. assumption.
Introduction
While deep neural networks (DNNs) have been successful in several applications, their performancedegrades under distributional shifts between the training data and test data [31].
This distributionalshift hinders DNNs from being widely deployed in many risk-sensitive applications, such as autonomous driving, medical imaging, and mobile health care, where new types of test data unseenduring training could result in undesirable disasters.
For instance, Tesla Autopilot has caused 12“deaths” until recently [2].
To address this problem, test-time adaptation (TTA) aims to adapt DNNsto the target/unseen domain with only unlabeled test data streams, without any additional data acquisition or labeling cost.
Recent studies reported that TTA is a promising, practical direction to mitigatedistributional shifts [28, 32, 40, 4, 43].
Prior TTA studies typically assume (implicitly or explicitly) that a target test sample x t at time t andthe corresponding ground-truth label y t (unknown to the learner) are independent and identicallydistributed (i.i.d.) following a target domain, i.e., ( x t , y t ) is drawn independently from a time-invariantdistribution P T ( x , y ) .
However, the distribution of online test samples often changes across the timeaxis, i.e., ( x t , y t ) ∼ P T ( x , y | t ) in many applications; for instance, AI-powered self-driving car’sobject encounter will be dominated by cars while driving on the highway, but less dominated bythem on downtown where other classes such as pedestrians and bikes are visible.
In human activityrecognition, some activities last for a short term (e.g., a fall down), whereas certain activities last longer(e.g., a sleep).
Figure 1 illustrates that some data distributions in the real world, such as autonomousdriving and human activity recognition, are often temporally correlated.
Considering that most
… … … 𝑡𝑖𝑚𝑒 …
Car
Van Pedestrian
Person (sitting)
Cyclist
Misc
[Figure]
Standing
[Figure]
KKF PQPKKF      % N C UU K H K E C V K QP  G TT Q T  2GTHQTOCPEG1XGTXKGY 5QWTEG$05VCVU10&#2.6'06%Q66#
NOTE existing TTA algorithms simply use an incoming batch of test samples for adaptation [28, 32, 40, 43],the model might be biased towards these imbalanced samples under the temporally correlated teststreams.
Figure 2 compares the performance of the state-of-the-art TTA algorithms under the i.i.d.
andnon-i.i.d.
1 conditions.
While the TTA methods perform well under the i.i.d.
assumption, their errorsincrease under the non-i.i.d.
case.
Adapting to temporally correlated test data results in overﬁtting totemporal distributions, which in turn harms the generalization of the model.
Motivated by this, we present a NOn-i.i.d.
TEst-time adaptation scheme, NOTE , that consists oftwo components: (a) Instance-Aware Batch Normalization (IABN) and (b) Prediction-Balanced
Reservoir Sampling (PBRS).
First, we propose a novel normalization layer, IABN, that eliminatesthe dependence on temporally correlated data for adaptation while being robust to distributionshifts.
IABN detects out-of-distribution instances sample by sample and corrects via instance-awarenormalization.
The key idea of IABN is synthesizing Batch Normalization (BN) [15] with Instance
Normalization (IN) [36] in a unique way; it calculates how different the learned knowledge (BN) isfrom the current observation (IN) and corrects the normalization by the deviation between IN and BN.
Second, we present PBRS that resolves the problem of overﬁtting to non-i.i.d.
samples by mimickingi.i.d.
samples from non-i.i.d.
streams.
By utilizing predicted labels of the model, PBRS aims forboth time-uniform sampling and class-uniform sampling from the non-i.i.d.
streams, and stores the‘simulated’ i.i.d.
samples in memory.
With the i.i.d.-like batch in the memory, PBRS enables themodel to adapt to the target domain without being biased to temporal distributions.
We evaluate NOTE with state-of-the-art TTA baselines [28, 32, 21, 26, 4, 43] on multiple datasets,including common TTA benchmarks (CIFAR10-C and CIFAR100-C [12]) and real-world non-i.i.d.datasets (KITTI [8], HARTH [24], and ExtraSensory [37]).
Our results suggest that NOTE not onlysigniﬁcantly outperforms the baselines under non-i.i.d.
test data, e.g., it achieves a 21.1% error rateon CIFAR10-C which is on average 15.1% lower than the state-of-the-art method [4], but also showscomparable performance even under the i.i.d.
assumption, e.g., 17.6% error on CIFAR10-C where thebest baseline [43] achieves 17.8% error.
Our ablative study demonstrates the individual effectivenessof IABN and PBRS and further highlights their synergy when jointly used.
Finally, we summarize the key characteristics of NOTE.
First, NOTE is a batch-free inferencealgorithm (requiring a single instance for inference), different from the state-of-the-art TTA algorithms [28, 32, 40, 4, 43] where a batch of test data is necessary for inference to estimate normalizationstatistics (mean and variance).
Second, while some recent TTA methods leverage augmentationsto improve performance at the cost of additional forwarding passes [34, 43], NOTE requires onlya single forwarding pass.
NOTE updates only the normalization statistics and afﬁne parameters in
IABN, which is, e.g., approximately 0.02% of the total trainable parameters in ResNet18 [11].
Third,
NOTE’s additional memory overhead is negligible.
It merely stores predicted labels of test data to run
PBRS.
These characteristics make NOTE easy to apply to any existing AI system and particularly,are beneﬁcial in latency-sensitive tasks such as autonomous driving and human health monitoring.
Background2.1 Problem setting: test-time adaptation with non-i.i.d. streams
Test-time adaptation.
Let D S
= {X S , Y} be the data from the source domain and D T =
{X T , Y}be the data from the target domain to adapt to.
Each data instance and the corresponding ground-truthlabel pair ( x i , y i ) ∈
X S × Y in the source domain follows a probability distribution P S ( x , y ) .
Similarly, each target test sample and the corresponding label at test time t , ( x t , y t )
∈
X T × Y ,follows a probability distribution P T ( x , y ) where y t is unknown for the learner.
The standard covariateshift assumption in domain adaptation is deﬁned as P S ( x ) (cid:54) =
P T ( x ) and P S ( y | x ) =
P T ( y | x ) [31].
Unlike traditional domain adaptation that uses D S and X T collected beforehand for adaptation,test-time adaptation (TTA) continually adapts a pre-trained model f θ ( · ) from D S by utilizing only x tobtained at test time t .
TTA on non-i.i.d.
streams.
Note that previous TTA mechanisms typically assume that each targetsample ( x t , y t )
∈
X T ×Y is independent and identically distributed (i.i.d.)
following a time-invariantdistribution P T ( x , y ) .
However, the data obtained at test time is non-i.i.d.
in many scenarios.
Bynon-i.i.d., we refer to distribution changes over time, i.e., ( x t , y t ) ∼ P T ( x , y | t ) , which is a practicalsetting in many real world applications [45].
2.2 Batch normalization in TTA
Batch Normalization (BN) [15] is a widely-used training technique in deep neural networks as itreduces the internal covariant shift problem.
Let f ∈ R B × C × L denote a batch of feature maps ingeneral, where B , C , and L denote the batch size, the number of channels, and the size of eachfeature map, respectively.
Given the statistics of the feature maps for normalization, say mean µ andvariance σ 2 , BN is channel-wise , i.e., µ , σ 2 ∈ R C and computes:
[Equation]
where γ and β are the afﬁne parameters followed by the normalization, and (cid:15) > 0 is a small constantfor numerical stability.
Although a conventional way of computing BN in test-time is to set µ and σ 2 as those estimatedfrom training (or source) data, say ¯ µ and ¯ σ 2 , the state-of-the-art TTA methods based on adapting
BN layers [28, 32, 40, 43] instead use the statistics computed directly from the recent test batch tode-bias distributional shifts at test-time, i.e.:
[Equation]
This practice is simple yet effective under distributional shifts and is thus adopted in many recent
TTA studies [28, 32, 40, 43].
Based on the test batch statistics, they often further adapt the afﬁneparameters via entropy minimization of the model outputs [40] or update the entire parameters withself-training [43].
Method
In the same vein as previous work [28, 32, 40], we focus on adapting BN layers in the given model toperform TTA, and this includes essentially two approaches: (a) re-calibrating (or adapting)
channelwise statistics for normalization (instead of using those learned from training), and (b) adapting theafﬁne parameters (namely, γ and β ) after the normalization with respect to a certain objective basedon test samples, e.g., the entropy minimization of model outputs [40].
Under scenarios where test data are temporally correlated, however, naïvely adapting to the incomingbatch of test samples [28, 32, 40, 43] could be problematic for both approaches: the batch is now morelikely to (a) remove instance-wise variations that are actually useful to predict y , i.e., the “contents”rather than “styles” through normalization, and (b) include a bias in p ( y ) rather than uniform, whichcan negatively affect the test-time adaptation objective such as entropy minimization.
Case1: Out of distribution
Case2:
In distribution 𝐈𝐈𝐈𝐈 𝐈𝐈𝐈𝐈 �𝝁𝝁 c , �𝝈𝝈 c �𝝁𝝁 b , c , �𝝈𝝈 b , c 𝐈𝐈𝐈𝐈 : learned stats 𝐈𝐈𝐈𝐈 : instance stats �𝝁𝝁 b , c , �𝝈𝝈 b , c �𝝁𝝁 c , �𝝈𝝈 c
Non-i.i.d.
(temporally correlated) 𝒙𝒙 𝑡𝑡 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 … (1) Normalize & Predict via 𝐈𝐈𝐈𝐈𝐈𝐈𝐈𝐈 (2) Manage Memory & Adapt
IABN via 𝐏𝐏𝐈𝐈𝐏𝐏𝐏𝐏 𝐈𝐈𝐈𝐈 𝐈𝐈𝐈𝐈𝐈𝐈𝐈𝐈 𝐈𝐈𝐈𝐈 � 𝝁𝝁 c , �𝝈𝝈 c �𝝁𝝁 b , c , �𝝈𝝈 b , c 𝐈𝐈𝐈𝐈 𝐈𝐈𝐈𝐈𝐈𝐈𝐈𝐈 𝐈𝐈𝐈𝐈 �𝝁𝝁 b , c , �𝝈𝝈 b , c �𝝁𝝁 c , �𝝈𝝈 c = Correct BN stats
Use BN stats 𝐈𝐈𝐈𝐈𝐈𝐈𝐈𝐈 = 𝐈𝐈𝐈𝐈 + 𝝍𝝍 ( 𝐈𝐈𝐈𝐈−𝐈𝐈𝐈𝐈 ) 𝝍𝝍 ∗ : soft-shrinkage 𝝁𝝁 𝑡𝑡 ←𝝁𝝁 𝑡𝑡−1 , �𝝁𝝁 𝑡𝑡 𝝈𝝈 𝑡𝑡 ←𝝈𝝈 𝑡𝑡−1 , �𝝈𝝈 𝑡𝑡 �𝝁𝝁 𝑡𝑡 , �𝝈𝝈 𝑡𝑡 Time-uniform ( 𝒙𝒙 𝑖𝑖 , �𝑦𝑦 𝑖𝑖 ) P r ed i c t i on - un i f
o r m
Old
New
Class 𝒃𝒃
Class 𝒂𝒂 Class 𝒄𝒄 ( 𝒙𝒙 𝑡𝑡 , �𝑦𝑦 𝑡𝑡 )
Adapt learned stats 𝒙𝒙 𝑡𝑡′ = 𝐈𝐈𝐈𝐈𝐈𝐈𝐈𝐈 ( 𝒙𝒙 𝑡𝑡 ) ( 𝝁𝝁 𝑡𝑡 , 𝝈𝝈 𝒕𝒕 ) ( 𝝁𝝁 𝑡𝑡−1 , 𝝈𝝈 𝑡𝑡−1 )
Normalized output We propose two approaches to tackle each of the failure modes of adapting BN under temporalcorrelation.
Our method consists of two components: (a) Instance-Aware Batch Normalization(IABN) (Section §3.1) to overcome the limitation of BN under distribution shift and (b) PredictionBalanced Reservoir Sampling (PBRS) (Section §3.2) to combat with the temporal correlation of testbatches.
Figure 3 illustrates the overall workﬂow of NOTE with IABN and PBRS.
3.1 Instance-Aware Batch Normalization
As described in Section §2.2, recent TTA algorithms rely solely on the test batch to re-calculate BNstatistics.
We argue that this common practice does not successfully capture the feature statistics tonormalize the feature map f ∈ R B × C × L under temporal correlation in the test batch B .
In principle,standardizing a given feature map f : ,c, : by the statistics ˆ µ c , ˆ σ 2 c computed across B and L is positedon premise that averaging information across B can marginalize out uninformative instance-wisevariations for predicting y .
Under temporal correlation in B , however, this assumption is no longervalid, and averaging across B may not successfully de-correlate useful information in f : ,c, : from µ cand σ 2 c .
In an attempt to bypass such an “over-whitening” effect of using ˆ µ c and ˆ σ 2 c in test-time under temporalcorrelation, we propose correcting normalization statistics on a per-sample basis : speciﬁcally, insteadof completely switching from the original statistics of (¯ µ , ¯ σ 2 ) into (ˆ µ c , ˆ σ 2 c ) , our proposed InstanceAware Batch Normalization (IABN) considers the instance-wise statistics ˜ µ , ˜ σ 2 ∈ R B,C of f similarlyto Instance Normalization (IN)
[36], namely: [Equation]
We assume that ˜ µ b,c and ˜ σ 2 b,c follow the sampling distribution of a sample size L in N (¯ µ , ¯ σ 2 ) as thepopulation.
Then the corresponding variances for the sample mean ˜ µ b,c and the sample variance ˜ σ 2 b,ccan be calculated as:
[Equation] [Equation]
IABN corrects (¯ µ , ¯ σ 2 ) only in cases when ˜ µ b,c
(and ˜ σ 2 b,c ) signiﬁcantly differ from ¯ µ c (and ¯ σ 2 c ).
Speciﬁcally, we propose to use the following statistics for TTA:
[Equation]
where ψ ( x ; λ ) =  
[Equation] − λ[-0 , otherwise is the soft-shrinkage function and k ≥ 
-]
[Equation] if x > λ x + λ, if x < − λ 0 , otherwise is the soft-shrinkage function. Input: target stream x t ∼ P T ( x | t ) , 1: for t ∈ { 1 ...K } do 2: if memory is not full then 3:
Add ( x t , ˆ y t ) 4: else 5:
M ← majority class(es) in memory 6: if ˆ y t / ∈ M then (cid:46) Prediction-Balanced 7:
Randomly discard one instance ( x i , ˆ y i ) in memory where ˆ y i ∈ M 8:
Add ( x t , ˆ y t ) 9: else (cid:46) Reservoir Sampling 10: m c ← number of instances in memory where ˆ y i
= ˆ y t 11: n c ← number of instances encountered so far where ˆ y i
= ˆ y t 12:
Sample p ∼ Uniform(0,1) 13: if p < m c /n
c then 14:
Discard a random instance ( x i , ˆ y i ) in memory where ˆ y i
= ˆ y t 15:
Add ( x t , ˆ y t ) 16: else 17:
Discard ( x t , ˆ y t ) α ≥ 0 is the hyperparameter of IABN that determines the conﬁdence level of the BN statistics.
Ahigh value of α relies more on the learned statistics (BN), while a low value of α is in favor of thecurrent statistics measured from the instance.
Finally, the output of IABN can be described as:
[Equation] Observe that IABN becomes IN and BN when α = 0 and α = ∞ , respectively. If one choosestoo small α ≥ 0 , IABN may remove useful features, e.g., styles, of input (as with IN), which candegrade the overall classiﬁcation (or regression) performance [29]. Hence, it is important to choosean appropriate α .
Nevertheless, we found that a good choice of α is not too sensitive across testedscenarios, where we chose α = 4 for all experiments.
This way, IABN can be robust to distributionalshifts without the risk of eliminating crucial information to predict y .
3.2 Adaptation via Prediction-Balanced Reservoir Sampling
Temporally correlated distributions lead to an undesirable bias in p ( y ) , and thus adaptation witha batch of consecutive test samples negatively impacts the adaptation objective, such as entropyminimization [40].
To combat this imbalance, we propose Prediction-Balanced Reservoir Sampling(PBRS) that mimics i.i.d.
samples from temporally correlated streams with the assistance of a small(e.g., a mini-batch size) memory.
PBRS combines time-uniform sampling and prediction-uniformsampling to simulate i.i.d.
samples from the non-i.i.d.
streams.
For time-uniform sampling, we adoptreservoir sampling (RS) [39], a proven random sampling algorithm to collect time-uniform data in asingle pass on a stream without the prior knowledge of the total length of data.
For prediction-uniformsampling, we ﬁrst use the predicted labels to compute the majority class(es) in the memory.
We thenreplace a random instance of the majority class(es) with a new sample.
We detail the algorithm of
PBRS as a pseudo-code in Algorithm 1.
We found that these two heuristics can effectively balancesamples among both time and class axes, which mitigates the bias in temporally correlated data.
With the stored samples in the memory, we update the normalization statistics and afﬁne parametersin the IABN layers.
Note that IABN assumes ˜ µ b,c and ˜ σ 2 b,c follow the sampling distribution of
N (¯ µ , ¯ σ 2 ) and corrects the normalization if ˜ µ b,c and ˜ σ 2 b, c are out of distribution. While IABN isresilient to distributional shifts to a certain extent, the assumption might not hold under severedistributional shifts.
Therefore, we aim to ﬁnd better estimates of ¯ µ , ¯ σ 2 in IABN under distributionalshifts via PBRS.
Speciﬁcally, we update the normalization statistics, namely the means µ andvariances σ 2 , via exponential moving average: (a) µ t =
(1 − m ) µ t − 1 + m NN − 1 ˆ µ t and (b) σ 2 t =(1 − m ) σ 2 t − 1 + m NN − 1 ˆ σ 2 t where m is a momentum and N is the size of the memory.
We further optimize the afﬁne parameters, scaling factor γ and bias term β via a single backward pass withentropy minimization, similar to a previous study [40].
These parameters account for only around0.02% of the total trainable parameters in ResNet18 [11].
The IABN layers are adapted with the
N samples in the memory
every N test samples.
We set the memory size N as 64 following thecommon batch size of existing TTA methods [32, 4, 40] to ensure a fair memory constraint.
3.3 Inference
NOTE infers each sample via a single forward pass with IABN layers. Note that NOTE requires onlya single instance for inference, different from the state-of-the-art TTA methods [28, 32, 40, 4, 43] thatrequire batches for every inference. Moreover, NOTE requires only one forwarding pass for inference,while multiple forward passes are required in other TTA methods that utilize augmentations [34, 43]. The batch-free single-forward inference of NOTE is beneﬁcial in latency-sensitive tasks such asautonomous driving and human health monitoring.
After inference, NOTE determines whether tostore the sample and predicted label to the memory via PBRS.
Experiments
We implemented NOTE and the baselines via the PyTorch framework [30].
We ran all experimentswith three random seeds and speciﬁed the standard deviations.
Additional experimental details are inthe supplementary material.
Baselines.
We consider the following baselines including the state-of-the-art test-time adaptationalgorithms: Source evaluates the model trained from the source data directly on the target data withoutadaptation.
Test-time normalization ( BN stats ) [28, 32] updates the BN statistics from a batch of testdata.
Online Domain Adaptation ( ONDA ) [26] adapts batch normalization statistics to target domainsvia a batch of target data with exponential moving average.
Pseudo-Label ( PL ) [21] optimizes thetrainable parameters in BN layers via hard pseudo labels.
We update the BN layers only in PL, asdone in previous studies [40, 43].
Test entropy minimization ( TENT )
[40] updates the BN parametersvia entropy minimization.
Laplacian Adjusted Maximum-likelihood Estimation ( LAME ) [4] takesa more conservative approach; it modiﬁes the classiﬁer’s output probability and not the internalparameters of the model itself.
By doing so, it prevents the model parameters from over-adaptingto the test batch.
Continual test-time adaptation ( CoTTA ) [43] reduces the error accumulation byusing weight-averaged and augmentation-averaged predictions.
It avoids catastrophic forgetting bystochastically restoring a part of the neurons to the source pre-trained weights.
Adaptation and hyperparameters.
We assume the model pre-trained with source data is availablefor TTA.
In NOTE, we replaced BN with IABN during training.
We set the test batch size as64 and the adaptation epoch as one for adaptation, which is the most common setting among thebaselines [32, 4, 40].
Similarly, we set the memory size N as 64 and adapt the model every 64 samplesin NOTE to ensure a fair memory constraint.
We conduct online adaptation and evaluation, where themodel is continually updated.
For the baselines, we adopt the best values for the hyperparametersreported in their papers or the ofﬁcial codes.
We followed the guideline to tune the hyperparameterswhen such a guideline was available [43].
We use ﬁxed values for the hyperparameters of NOTE,soft-shrinkage width α = 4 and exponential moving average momentum m = 0 . 01 , and update theafﬁne parameters via the Adam optimizer [17] with a learning rate l = 0 . 0001 for all experiments.
We detailed hyperparameter information of the baselines in the supplementary material.
4.1 Robustness under corruptions
Datasets.
CIFAR10-C/CIFAR100-C [12] are common TTA benchmarks for evaluating the robustness to corruptions [28, 32, 40, 43].
Both CIFAR10/CIFAR100 [18] have 50,000/10,000 training/testdata.
CIFAR10/CIFAR100 have 10/100 classes respectively.
CIFAR10-C/CIFAR100-C apply 15types of corruption to CIFAR10/CIFAR100 test data.
Similar to previous studies [28, 32, 40, 43],we use the most severe corruption level of 5.
We use ResNet18 [11] as the backbone network andpre-trained it on the clean training data.
Following prior studies [22, 14, 42, 41], we adopt Dirichletdistribution to generate synthetic non-i.i.d.
test streams from the originally i.i.d.
CIFAR10/100 data.
The details are provided in the supplementary material.
We vary the Dirichlet concentration parameter
[Table] 4 ± 0. ( 41.0 ± 0.2) 33. [Figure]
D i r i c
h l e
[Figure]
e t e r 𝛿 0.
[Figure]
[Figure]
δ to simulate diverse streams and visualize the resulting data in Figure 4.
We use δ = 0 .
1 as thedefault value unless speciﬁed.
Overall result.
Tables 1 shows the result under the temporally correlated (non-i.i.d.)
data and theuniform (i.i.d.) data, respectively.
We observe signiﬁcant performance degradation in the baselinesunder the temporally correlated setting.
For BN Stats, PL, TENT, and CoTTA, this degradationis particularly due to the dependence on the test batch for the re-calculation of the BN statistics.
Updating the batch statistics from test data via exponential moving average (ONDA) also suffersfrom the temporally correlated data.
This indicates relying on the test batch for re-calculating the
BN statistics indeed cancels out meaningful instance-wise variations under temporal correlation. Interestingly, LAME works better in the non-i.i.d.
setting than in the i.i.d.
setting, which is consistentwith previous reports [4].
The primary reason is, as stated by the authors, it “discourages deviationsfrom the predictions of the pre-trained model,” and thus it “does not noticeably help in i.i.d andclass-balanced scenarios.”
NOTE achieves on average 15.7% improvement over the best baseline(LAME) under the non-i.i.d.
setting.
With the i.i.d.
assumption, NOTE still achieves comparableperformance to the baselines.
When we know the target samples are i.i.d.
, we can simply use the testbatch without using PBRS.
We report the result of the variant version of NOTE in the parentheses,which achieves on average 3.9% improvement further when the i.i.d.
assumption is known.
Effect of the degree of temporal correlation.
We also investigate the effect of the degree oftemporal correlation for TTA algorithms.
Figure 5a shows the result.
The lower δ is, the severer
[Table] temporal correlation becomes. The error rates of most of the baselines deteriorate as δ decreases,which shows that the existing TTA baselines are susceptible to temporally correlated data.
NOTEshows consistent performance among all δ values, indicating its robustness under temporal correlation.
Effect of batch size.
While we experiment with a widely-used value of 64 as the batch size (ormemory size in NOTE), one might be curious about the impact of batch size under temporallycorrelated streams.
Figure 5b shows the result with six different batch sizes.
As shown, NOTE is notmuch affected by the batch size, while most of the baselines recovers performance degradation asthe batch size increases.
This is because a higher batch size has a better chance of adaptation withbalanced samples under temporally correlated streams.
Increasing the batch size, however, mitigatestemporal correlation at the expense of inference latency and adaptation speed.
2 Real
-distributions with domain shift Datasets.
We evaluate NOTE under three real-world distribution datasets: object detection inautonomous driving (KITTI [8]), human activity recognition (HARTH [24]), and user behavioralcontext recognition (ExtraSensory [37]).
KITTI is a well-known autonomous driving dataset thatprovides consecutive frames that contains natural temporal correlation in driving contexts.
We adaptfrom KITTI to KITTI-Rain [10] - a dataset that converted KITTI images to rainy images.
This contains7,481/7,800 train/test samples with nine classes.
We use ResNet50 [11] pre-trained on ImageNet [7]as the backbone network.
HARTH was collected from 22 users in free-living environments forseven days.
Each user was equipped with two three-axial Axivity AX3 accelerometers for recordinghuman activities.
We use 15 users collectively as the source domain and the remaining seven usersas each target domain, which entails natural domain shifts from source users to target users asdifferent physical conditions make domain shifts across users.
HARTH contains 82,544/39,377train/test samples with 12 classes.
We report the averaged error over all target domains.
We usefour one-dimensional convolutional layers followed by one fully-connected layer as the backbonenetwork for HARTH.
The Extrasensory dataset collected users’ own smartphone sensory data (motionsensors, audio, etc.)
in the wild for seven days, aiming to capture people’s authentic behaviors intheir regular activities. We use 16 users as the source domain and seven users as target domains.
ExtraSensory includes 17,777/4,862 train/test data with ﬁve classes.
For ExtraSensory, we use twoone-dimensional convolutional layers followed by one fully-connected layer as the backbone network.
For both HARTH and ExtraSensory models, a single BN layer follows each convolutional layer.
Additional details are in the supplementary material.
Result.
Table 2 shows the result for the real-world datasets.
The overall trend is similar to thetemporal correlation experiments with CIFAR10-C/CIFAR100-C datasets, which indicates thatthe real-world datasets are indeed temporally correlated.
NOTE consistently reduces errors afteradaptation under real-world distributions.
We believe this demonstrates NOTE is a promising methodto be utilized in various real-world ML applications with distributional shifts.
[Table] 4.3 Ablation study
[Table]
We conduct an ablative study to further investigate the effectiveness of the individual components.
Table 3 shows the result under both
i [Table] i.d. and non-i.i
[Table]
d. settings.
Using IABN alone signiﬁcantlyreduces error rates over Source, demonstrating the effectiveness of correcting normalization for outof-distribution samples.
Using PBRS with BN shows comparable improvement with the IABN-onlyresult.
Note that there is only a marginal gap (around 1%) between the non-i.i.d.
and i.i.d.
resultsin PBRS.
This indicates that PBRS could effectively simulate i.i.d.
samples from non-i.i.d.
streams.
The joint use of IABN and PBRS outperforms using either of them, meaning that PBRS provides
IABN with better estimates for the normalizing operation.
In addition, PBRS is better than Reservoir
Sampling (RS) that has been a strong baseline in continual learning [16, 5].
This shows storingprediction-balanced sampling in addition to time-uniform sampling leads to better adaptation in
TTA.
We also investigated the joint use of IN and PBRS with the combination of IABN and PBRSon CIFAR100-C, and the result shows that IABN+PBRS (47.0%) achieves lower error rate than
IN+PBRS (52.5%) on CIFAR100-C under temporal correlation.
Discussion and conclusion
This paper highlights that real-world distributions often change across the time axis, and existing testtime adaptation algorithms mostly suffer from the non-i.i.d.
test data streams.
To address this problem,we present a NOn-i.i.d.
TEst-time adaptation algorithm, NOTE.
Our experiments evaluated robustnessunder corruptions and domain adaptation on real-world distributions.
The results demonstrate that
NOTE not only outperforms the baselines under the non-i.i.d./real distribution settings, but it alsoshows comparable performance under the i.i.d.
assumption.
We believe that the insights and ﬁndingsfrom this study are a meaningful step toward the practical impact of the test-time adaptation paradigm.
Limitations.
NOTE and most state-of-the-art TTA algorithms [28, 21, 26, 32, 40, 43] assumethat the backbone networks are equipped with BN (or IABN) layers.
While BN is a widely-usedcomponent in deep learning, several architectures, such as LSTMs [13] and Transformers [38], donot embed BN layers.
A recent study uncovered that BN is advantageous in Vision Transformers [44],showing potential room to apply our idea to architectures without BN layers.
However, more in-depthstudies are necessary to identify the actual applicability of BN (or IABN) to those architectures.
While LAME [4] is applicable to models without BN, its limitation is the performance drop in i.i.d.scenarios, as shown in both its paper and our evaluation.
While NOTE shows its effectiveness in bothnon-i.i.d and i.i.d.
scenarios, a remaining challenge is to design an algorithm that generalizes to anyarchitecture.
We believe the ﬁndings and contributions of our work could give valuable insights tofuture endeavors on this end.
Potential negative societal impacts.
As TTA relies on unlabeled test samples and changes themodel accordingly, the model is exposed to potential data-driven biases after adaptation, such asfairness issues [3] and adversarial attacks [35].
In some sense, the utility of TTA comes at theexpense of exposure to threats.
This vulnerability is another crucial problem that both ML researchersand practitioners need to take into consideration.
In addition, TTA entails additional computationsfor adaptation with test data, which may have negative impacts on environments, e.g., increasingelectricity consumption and carbon emissions [33].
Nevertheless, we believe NOTE would notexacerbate this issue as it is computationally efﬁcient as mentioned in Section §1.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section §5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection §5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section §4 and the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)? [Yes] We ran the entire experiments with three different random
seems (0,1,2) and reported the standard deviations.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]