Original: T EST - TIME TRAINING FOR OUT - OF - DISTRIBUTION GENERALIZATION
Revised: T EST -T IME T RAINING FOR G ENERALIZATION UNDER D ISTRIBUTION S HIFTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce a general approach, called test-time training, for improving the performance of predictive models whentest and training  data come from different distributions.
Revised: We introduce a general approach, called test-time training, for improving the performance of predictive models when training and test data come from different distributions.

Original: Test-time training turns a single unlabeled test instance into a selfsupervised learning problem, on which we update the model parameters before making a prediction on the test sample.
Revised: Test-time training turns a single unlabeled test instance into a selfsupervised learning problem, on which we update the model parameters before making a prediction on this instance.

Original: We show that this simple idea leads to surprising improvements on diverse image classiﬁcation benchmarks aimed at evaluating robustness to distribution shifts.
Revised: We show that this simple idea leads to surprising improvements on diverse image classiﬁcation benchmarks aimed at evaluating robustness to distribution shifts.

Original: Theoretical investigations on a convex model reveal helpful intuitions for when we can expect our approach to help.
Revised: Theoretical investigations on a convex model reveal helpful intuitions for when we can expect our approach to help.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Supervised learning remains notoriously weak at out-of-distribution generalization.
Revised: Supervised learning remains notoriously weak at generalization under distribution shifts.

Original: Unless trainingdata and test data are drawn from the same distribution,predictive models tend to fail easily, and even seemingly minor natural variations in distribution turn out to defeat state-of-the-art models.
Revised: Unless training and test data are drawn from the same distribution, even seemingly minor differences turn out to defeat state-of-the-art models.

Original: Adversarial robustness, transfer learning and domain adaptation  are but a few existing paradigms that anticipate differences of sorts between  training and test distributions.
Revised: Transfer learning, domain adaptation and adversarial robustness are but a few existing paradigms that anticipate differences of sorts between the training and test distribution.

Original: In this work, we explore a new take onthe problem of out-of-distribution generalization, without any mathematical structure or data available at training-time about the distributional differences.
Revised: In this work, we explore a new take on generalization, without any mathematical structure or data available at training-time to anticipate the distribution shifts.

Original: We start from a simple observation.
Revised: We start from a simple observation.

Original: When presented with an unlabeled test instance x , this instance itself gives us a hint about the distribution from which it was drawn.
Revised: The unlabeled test instance x presented at test-time gives us a hint about the distribution from which it was drawn.

Original: Our approach, called test-time training, allows the model parameters θ to depend on the test case x , but not its unknown label y .
Revised: Our approach, called test-time training, allows the model parameters θ to depend on the test instance x , but not its unknown label y .

Original: While such a sample-speciﬁc model θ ( x ) is powerful in principle, seizing this information-theoretic advantage in practice raises new challenges  that we  begin to address in this work.
Revised: This variable decision boundary θ ( x ) is powerful in theory (see Appendix A), and raises new challenges in practice that we only begin to address here.

Original: The route we take is to create a self-supervised learning problem basedonly on this single test case x , which we use to update θ at test-time before we then make a prediciton on x .
Revised: Our algorithm creates a self-supervised learning problem based on this single test instance x , updating θ at test-time before making a prediction.

Original: Self-supervised learning uses an auxiliary task that automatically creates labels from unlabeled data.
Revised: Self-supervised learning uses an auxiliary task that automatically creates labels from unlabeled inputs.

Original: For the visual data we work with, the task rotates an image x by a multiple of 90 degrees, and assigns the angle as the label (Gidaris et al., 2018).
Revised: For the visual data we work with, the task rotates an image x by a multiple of 90 degrees, and assigns the angle as the label (Gidaris et al., 2018).

Original: Our approach can also be easily modiﬁed to work outside the standard supervised learning setting.
Revised: Our approach can also be easily modiﬁed to work outside the standard supervised learning setting.

Original: If several test instances arrive in a batch, we can use the entire batch for test-time training.
Revised: If several test instances arrive in a batch, we can use the entire batch for test-time training.

Original: If test instances arrive online sequentially, we obtain further improvements by keeping the state of the parameters.
Revised: If test instances arrive online sequentially, we obtain further improvements by keeping the state of the parameters.

Original: After all, prediction is rarely a single event.
Revised: After all, prediction is rarely a single event.

Original: The online setting can be the natural mode of deployment in practice, and shows the strongest improvements.
Revised: The online setting can be the natural mode of deployment in practice, and shows the strongest improvements.

Original: We experiment with test-time training forout-of-distribution generalization  in the context of object recognition on several benchmarks.
Revised: We experiment with test-time training for generalization under distribution shifts in the context of object recognition on several benchmarks.

Original: These include images with diverse types of corruption at various levels (Hendrycks & Dietterich, 2019), video frames of moving objects (Shankar et al., 2019), and a new test set of unknowndistribution shifts collected by Recht et al.
Revised: These include images with diverse types of corruption at various levels (Hendrycks & Dietterich, 2019), video frames of moving objects (Shankar et al., 2019), and a new test set of unknown shifts collected by Recht et al.

Original: Our algorithm does not hurt on the original distribution, but makes substantial improvements under distribution shifts.
Revised: Our algorithm does not hurt on the original distribution, but makes substantial improvements under distribution shifts.

Original: In all our experiments, we compare with a strong baseline (labeled joint training) that uses both supervised and self-supervised learning at training-time, but keeps the model ﬁxed at test-time.
Revised: In all our experiments, we compare with a strong baseline (labeled joint training) that uses both supervised and self-supervised learning at training-time, but keeps the model ﬁxed at test-time.

Original: Very recent work shows that additional training-time self-supervision improves out-of-distribution generalization Hendrycks et al.
Revised: Very recent work shows that additional training-time self-supervision improves robustness Hendrycks et al.

Original: (2019b).
Revised: (2019b).

Original: The joint training baseline we use corresponds to an improved implementation of their work.
Revised: The joint training baseline we use corresponds to an improved implementation of their work.

Revised: We are also inspired by Sun et al.

Revised: (2019), which performs joint training with a selfsupervised task in the context of unsupervised domain adaptation, simultaneously on the source and target domain.

Original: A comprehensive review of related work follows in section 5.
Revised: A comprehensive review of related work follows in section 5.

Original: We complement the empirical results with theoretical investigations in Section 4, of when test-time training is expected to help on a convex model, and establish an intuitive sufﬁcient condition, which, roughly speaking, is to have correlated gradients between the loss functions of the main task and self-supervised task.
Revised: We complement the empirical results with theoretical investigations in section 4, of when test-time training is expected to help on a convex model, and establish an intuitive sufﬁcient condition, which, roughly speaking, is to have correlated gradients between the loss functions of the two tasks.

Original:  M ETHOD
Revised:  M ETHOD

Original: Next we describe the algorithmic details of our method.
Revised: Next we describe the algorithmic details of our method.

Original: To setup notations, consider a standard Klayer neural network with parameters θ k for layer k .
Revised: To setup notations, consider a standard Klayer neural network with parameters θ k for layer k .

Original: The stacked parameter vector θ =
Revised: The stacked parameter vector θ =

Original: ( θ 1 , . .
Revised: ( θ 1 , . .

Original: . , θ K ) speciﬁes the entire model for a classiﬁcation task with loss function l m ( x, y ; θ ) on the test instance ( x, y ) .
Revised: . , θ K ) speciﬁes the entire model for a classiﬁcation task with loss function l m ( x, y ; θ ) on the test instance ( x, y ) .

Original: We call this the main task , as indicated by the subscript of the loss function.
Revised: We call this the main task , as indicated by the subscript of the loss function.

Original: We assume to have training data ( x 1 , y 1 ) , . . .
Revised: We assume to have training data ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) drawn i.i.d.
Revised: , ( x n , y n ) drawn i.i.d.

Original: from a distribution P .
Revised: from a distribution P .

Original: Standard empirical risk minimization corresponds to solving the optimization problem:
Revised: Standard empirical risk minimization corresponds to solving the optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Our method requires a self-supervised auxiliary task with loss function l s ( x ) .
Revised: Our method requires a self-supervised auxiliary task with loss function l s ( x ) .

Original: In this paper, we choose the rotation prediction task (Gidaris et al., 2018), which has been demonstrated to be simple and effective at feature learning for neural networks.
Revised: In this paper, we choose the rotation prediction task (Gidaris et al., 2018), which has been demonstrated to be simple and effective at feature learning for neural networks.

Original: The task simply rotates x on the image plane by one of 0, 90, 180 and 270 degrees and have the model predict the angle of rotation as a four-way classiﬁcation problem.
Revised: The task simply rotates x on the image plane by one of 0, 90, 180 and 270 degrees and have the model predict the angle of rotation as a four-way classiﬁcation problem.

Original: Other self-supervised tasks in section 5 might also be used for our method.
Revised: Other self-supervised tasks in section 5 might also be used for our method.

Original: The auxiliary task shares some of the model parameters θ e =
Revised: The auxiliary task shares some of the model parameters θ e =

Original: ( θ 1 , . .
Revised: ( θ 1 , . .

Original: , θ κ ) up to a certain κ ∈ { 1 , . . .
Revised: , θ κ ) up to a certain κ ∈ { 1 , . . .

Original: , K } .
Revised: , K } .

Original: We think of those κ layers as a shared feature extractor .
Revised: We think of those κ layers as a shared feature extractor .

Original: The auxiliary task uses its own task-speciﬁc parameters θ s = ( θ (cid:48) κ +1 , . . .
Revised: The auxiliary task uses its own task-speciﬁc parameters θ s = ( θ (cid:48) κ +1 , . . .

Original: , θ (cid:48) K ) .
Revised: , θ (cid:48) K ) .

Original: We call the unshared parameters θ s the selfsupervised task branch , and θ m = ( θ κ +1 , . .
Revised: We call the unshared parameters θ s the selfsupervised task branch , and θ m = ( θ κ +1 , . .

Original: , θ K ) the main task branch .
Revised: , θ K ) the main task branch .

Original: Pictorially, the joint architecture is a Y -structure with a shared bottom and two branches.
Revised: Pictorially, the joint architecture is a Y -structure with a shared bottom and two branches.

Original: For our experiments, the selfsupervised task branch has the exact same architecture as the main branch, except for the output dimensionality of the last layer due to the different number of classes in the two tasks.
Revised: For our experiments, the selfsupervised task branch has the exact same architecture as the main branch, except for the output dimensionality of the last layer due to the different number of classes in the two tasks.

Original: Training is done in the fashion of multi-task learning (Caruana, 1997); the model is trained on both tasks on the same data drawn from P .
Revised: Training is done in the fashion of multi-task learning (Caruana, 1997); the model is trained on both tasks on the same data drawn from P .

Original: Losses for both tasks are added together, and gradients are taken for the collection of all parameters.
Revised: Losses for both tasks are added together, and gradients are taken for the collection of all parameters.

Original: The joint training problem is therefore
Revised: The joint training problem is therefore

Original: [Equation]
Revised: [Equation]

Original: Now we describe the standard version of test-time training on a single test input x .
Revised: Now we describe the standard version of test-time training on a single test instance x .

Original: Simply put, test-time training ﬁnetunes the shared feature extractor θ e by minimizing the auxiliary task loss on x .
Revised: Simply put, test-time training ﬁnetunes the shared feature extractor θ e by minimizing the auxiliary task loss on x .

Original: This can be formulated as
Revised: This can be formulated as

Original: [Equation]
Revised: [Equation]

Original: Denote θ ∗ e the (approximate) minimizer of Equation 3.
Revised: Denote θ ∗ e the (approximate) minimizer of Equation 3.

Original: The model then makes a prediction using the updated parameters θ ( x ) =
Revised: The model then makes a prediction using the updated parameters θ ( x ) =

Original: ( θ ∗ e , θ m ) .
Revised: ( θ ∗ e , θ m ) .

Original: Empirically, the difference is negligible between minimizing
Revised: Empirically, the difference is negligible between minimizing

Original: Equation 3 over θ e versus over both θ e and θ s .
Revised: Equation 3 over θ e versus over both θ e and θ s .

Original: Theoretically, there exists a difference only when optimization is done with more than one step of gradient descent.
Revised: Theoretically, there exists a difference only when optimization is done with more than one step of gradient descent.

Original: In the standard version of our method, the optimization problem in Equation 3 is always initialized with parameters θ = ( θ e , θ s ) obtained by minimizing Equation 2 on data from P .
Revised: In the standard version of our method, the optimization problem in Equation 3 is always initialized with parameters θ = ( θ e , θ s ) obtained by minimizing Equation 2 on data from P .

Original: After making a prediction on x , θ ∗ e is discarded.
Revised: After making a prediction on x , θ ∗ e is discarded.

Original: Outside of the standard supervised learning setting, when the test instances arrive online sequentially, the online version of test-time training solves the same optimization problem as in Equation 3 to update the shared feature extractor θ e .
Revised: Outside of the standard supervised learning setting, when the test instances arrive online sequentially, the online version of test-time training solves the same optimization problem as in Equation 3 to update the shared feature extractor θ e .

Original: However, on test input x t , θ is instead initialized with θ ( x t − 1 ) updated on the previous test input x t − 1 .
Revised: However, on test instance x t , θ is instead initialized with θ ( x t − 1 ) updated on the previous instance x t − 1 .

Original: This allows θ ( x t ) to take advantage of the distributional information available in x 1 , . . .
Revised: This allows θ ( x t ) to take advantage of the distributional information available in x 1 , . . .

Original: , x t − 1 as well as x t .
Revised: , x t − 1 as well as x t .

Original: Test-time training naturally beneﬁts from standard data augmentation techniques.
Revised: Test-time training naturally beneﬁts from standard data augmentation techniques.

Original: On each test input x , we perform the exact same set of random transformations as used for data augmentation during training to form a batch  for test-time training.
Revised: On each test instance x , we perform the exact same set of random transformations as used for data augmentation during training to form a batch only containing these augmented copies of x for test-time training.

Original:  g a u ss s h
Revised: o r i g i n a l g a u ss s h

Original: o t
Revised: o t

Original: i m p
Revised: i m p

Original: u
Revised: u

Original: l s e d e f o c u s g l a ss m o t i o n z oo m s n o w f r o s t f o g b r
Revised: l s e d e f o c u s g l a ss m o t i o n z oo m s n o w f r o s t f o g b r

Original: i g h t c
Revised: i g h t c

Original: o n t r a s t e l a s t
Revised: o n t r a s t e l a s t

Original: i c p
Revised: i c p

Original: i x e l a t e j p e g
Revised: i x e l a t e j p e g

Original: [Figure]
Revised: [Figure]

Original:  E MPIRICAL R ESULTS
Revised:  E MPIRICAL R ESULTS

Original: We experiment with both versions of our method (standard and online) on three kinds of out-ofdistribution benchmarks, presented here in the order of visually low to high-level, which is roughly also the order of easy to hard.
Revised: We experiment with both versions of our method (standard and online) on three kinds of benchmarks for distribution shifts, presented here in the order of visually low to high-level, which is roughly also the order of easy to hard.

Revised: Our code will be released once the paper is accepted.

Original: Network details.
Revised: Network details.

Original: Our architecture and hyper-parameters are consistent across all experiments.
Revised: Our architecture and hyper-parameters are consistent across all experiments.

Original: We use Residual Networks (ResNets)
Revised: We use Residual Networks (ResNets)

Original: (He et al., 2016b), which are constructed differently for CIFAR10 1 (26-layer) and ImageNet 2 (18-layer).
Revised: (He et al., 2016b), which are constructed differently for CIFAR10 1 (26-layer) and ImageNet 2 (18-layer).

Original: ResNets on ImageNet have four groups, each containing convolutional layers with the same number of channels and size of feature maps; our splitting point is the end of the third group.
Revised: ResNets on ImageNet have four groups, each containing convolutional layers with the same number of channels and size of feature maps; our splitting point is the end of the third group.

Original: ResNets on CIFAR-10 have three groups; our splitting point is the end of the second group.
Revised: ResNets on CIFAR-10 have three groups; our splitting point is the end of the second group.

Original: In addition, Batch Normalization (BN), a popular module in deep networks, is ineffective when training on small batches, for which the estimated batch statistics are less accurate (Ioffe & Szegedy, 2015).
Revised: In addition, Batch Normalization (BN), a popular module in deep networks, is ineffective when training on small batches, for which the estimated batch statistics are less accurate (Ioffe & Szegedy, 2015).

Original: This technicality hurts test-time training since each batch only contains (augmented) copies of a single image.
Revised: This technicality hurts test-time training since each batch only contains (augmented) copies of a single image.

Original: Therefore our networks instead use Group Normalization (GN) (Wu & He, 2018), which achieves similar performance as BN on large batches without hurting on small ones.
Revised: Therefore our networks instead use Group Normalization (GN) (Wu & He, 2018), which achieves similar performance as BN on large batches without hurting on small ones.

Original: Results with BN are shown in Appendix E for completeness.
Revised: Results with BN are shown in Appendix E for completeness.

Original: Optimization details.
Revised: Optimization details.

Original: For Equation 2, optimization hyper-parameters are set to the default 3 in standard practice (Huang et al., 2016;
Revised: For Equation 2, optimization hyper-parameters are set to the default 3 in standard practice (Huang et al., 2016;

Original: He et al., 2016a).
Revised: He et al., 2016a).

Original: For Equation 3, we use stochastic gradient descent (SGD) with the learning rate set to that of the last epoch during training, which is 0.001 in all our experiments.
Revised: For Equation 3, we use stochastic gradient descent (SGD) with the learning rate set to that of the last epoch during training, which is 0.001 in all our experiments.

Original: Following standard practice (He et al., 2018; Liu et al., 2018) known to improve performance when ﬁnetuning, we do not use weight decay or momentum.
Revised: Following standard practice (He et al., 2018; Liu et al., 2018) known to improve performance when ﬁnetuning, we do not use weight decay or momentum.

Original: For the standard version, we take ten gradient steps, using batches independently generated by the same image.
Revised: For the standard version, we take ten gradient steps, using batches independently generated by the same image.

Original: For online we take only one step.
Revised: For online we take only one step.

Original: The computational aspects of our method are discussed in Appendix C.
Revised: The computational aspects of our method are discussed in Appendix C.

Original: Following standard practice, the transformations used for data augmentation are random crop with padding and random horizontal ﬂip for CIFAR-10 (Guo et al., 2017a;
Revised: Following standard practice, the transformations used for data augmentation are random crop with padding and random horizontal ﬂip for CIFAR-10 (Guo et al., 2017a;

Original: Huang et al., 2016), and random resized crop and random horizontal ﬂip for ImageNet (Ioffe & Szegedy, 2015; He et al., 2016a).
Revised: Huang et al., 2016), and random resized crop and random horizontal ﬂip for ImageNet (Ioffe & Szegedy, 2015; He et al., 2016a).

Original: Speciﬁcally, these transformations do not contain information about the distribution shifts.
Revised: Speciﬁcally, these transformations do not contain information about the distribution shifts.

Original: In all the tables and ﬁgures, baseline refers to the plain ResNet model (using GN, unless otherwise speciﬁed); joint training refers to the model jointly trained on both the main task and the selfsupervised task, ﬁxed at test-time as in Hendrycks et al.
Revised: In all the tables and ﬁgures, baseline refers to the plain ResNet model (using GN, unless otherwise speciﬁed); joint training refers to the model jointly trained on both the main task and the selfsupervised task, ﬁxed at test-time as in Hendrycks et al.

Original: (2019b)
Revised: (2019b)

Original: ; test-time training refers to the standard version described section 2; and test-time training online refers to the online version that does not discard θ ( x t ) for x t arriving sequentially from the same distribution.
Revised: ; test-time training refers to the standard version described section 2; and test-time training online refers to the online version that does not discard θ ( x t ) for x t arriving sequentially from the same distribution.

Original: Performance for test-time training online is calculated, just like the others, as the average over the entire test set; we always shufﬂe the test set before test-time training online to avoid ordering artifacts.
Revised: Performance for test-time training online is calculated, just like the others, as the average over the entire test set; we always shufﬂe the test set before test-time training online to avoid ordering artifacts.

Original: [Figure]
Revised: [Figure]

Original: h
Revised: h

Original: [Figure]
Revised: [Figure]

Original: i m p u l
Revised: i m p u l

Original: [Figure]
Revised: [Figure]

Original: e d e
Revised: e d e

Original: [Figure]
Revised: [Figure]

Original: c u
Revised: c u

Original: [Figure]
Revised: [Figure]

Original: m o
Revised: m o

Original: [Figure]
Revised: [Figure]

Original: o n
Revised: o n

Original: [Figure]
Revised: [Figure]

Original: s n o w f r o
Revised: s n o w f r o

Original: [Figure]
Revised: [Figure]

Original: f o g
Revised: f o g

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: g
Revised: g

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: l
Revised: l

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: 3. C OMMON C ORRUPTIONS
Revised: 3. C OMMON C ORRUPTIONS

Original: Hendrycks & Dietterich (2019) propose to benchmark robustness of neural networks on 15 types of corruptions from four broad categories: noise, blur, weather and digital.
Revised: Hendrycks & Dietterich (2019) propose to benchmark robustness of neural networks on 15 types of corruptions from four broad categories: noise, blur, weather and digital.

Original: Each corruption type comes in ﬁve levels of severity, with level 5 the most severe (details and sample images in Appendix D).
Revised: Each corruption type comes in ﬁve levels of severity, with level 5 the most severe (details and sample images in Appendix D).

Original: The corruptions are algorithmically simulated to mimic real-world corruptions as much as possible on copies of the test set for both CIFAR-10 and ImageNet.
Revised: The corruptions are algorithmically simulated to mimic real-world corruptions as much as possible on copies of the test set for both CIFAR-10 and ImageNet.

Original: According to the authors, training should be done on the original training set, and the diversity of corruption types should make it difﬁcult for any method to work well across the board if it relies too much on corruption speciﬁc knowledge.
Revised: According to the authors, training should be done on the original training set, and the diversity of corruption types should make it difﬁcult for any method to work well across the board if it relies too much on corruption speciﬁc knowledge.

Original: CIFAR-10-C.
Revised: CIFAR-10-C.

Original: Our results on the level 5 corruptions (most severe) are shown in Figure 9.
Revised: Our results on the level 5 corruptions (most severe) are shown in Figure 9.

Original: Due to space constraints, our results on levels 1-4 are shown in
Revised: Due to space constraints, our results on levels 1-4 are shown in

Original: Appendix E. Across all ﬁve levels and 15 corruption types, both versions of test-time training always improve over the baseline by a large margin.
Revised: Appendix E. Across all ﬁve levels and 15 corruption types, both versions of test-time training always improve over the baseline by a large margin.

Original: The standard version of test-time training always improves over joint training, and the online version often improves very signiﬁcantly ( > 10%) over joint training and never hurts by more than 0.2%.
Revised: The standard version of test-time training always improves over joint training, and the online version often improves very signiﬁcantly ( > 10%) over joint training and never hurts by more than 0.2%.

Original: Speciﬁcally, test-time training online contributes > 24% on the three noise types and 38% on pixelation.
Revised: Speciﬁcally, test-time training online contributes > 24% on the three noise types and 38% on pixelation.

Original: For the seemly unstable setup of a learning problem that abuses a single image, this kind of consistency is rather surprising.
Revised: For the seemingly unstable setup of a learning problem that abuses a single image, this kind of consistency is rather surprising.

Original: The baseline ResNet-26 has error 8.9% on the original test set of CIFAR-10.
Revised: The baseline ResNet-26 has error 8.9% on the original test set of CIFAR-10.

Original: The joint training baseline actually improves performance on the original to 8.1%.
Revised: The joint training baseline actually improves performance on the original to 8.1%.

Original: Most surprisingly, unlike many other methods that tradeoff original performance with robustness, test-time training further improves on the original test set by 0.2% consistently over many independent trials.
Revised: Most surprisingly, unlike many other methods that tradeoff original performance with robustness, test-time training further improves on the original test set by 0.2% consistently over many independent trials.

Original: This indicates that our method does not choose between speciﬁcity and generality.
Revised: This indicates that our method does not choose between speciﬁcity and generality.

Original: Separate from our method, it is interesting to note that joint training consistently improves over the baseline, as discovered by Hendrycks et al. (2019b).
Revised: Separate from our method, it is interesting to note that joint training consistently improves over the baseline, as discovered by Hendrycks et al. (2019b).

Original: Hendrycks & Dietterich (2019) have also experimented with various other training methods on this benchmark, and point to Adversarial Logit
Revised: Hendrycks & Dietterich (2019) have also experimented with various other training methods on this benchmark, and point to Adversarial Logit

Original: Pairing (ALP) (Kannan et al., 2018) as the most effective.
Revised: Pairing (ALP) (Kannan et al., 2018) as the most effective.

Original: Results of this additional baseline on
Revised: Results of this additional baseline on

Original: [Table]
Revised: [Table]

Original: We report accuracy for each class and the average over all samples.
Revised: We report accuracy for each class and the average over all samples.

Original: See subsection 3.2 for details.
Revised: See subsection 3.2 for details.

Original: all levels of CIFAR-10-C are shown in Appendix E, along with its implementation details.
Revised: all levels of CIFAR-10-C are shown in Appendix E, along with its implementation details.

Original: While surprisingly robust under some of the most severe corruptions (especially the three noise types), ALP incurs a much larger error (by a factor of two) on the original distribution and some corruptions (e.g. all levels of contrast and fog), and hurts performance signiﬁcantly when the corruptions are not as severe (especially on levels 1-3); this kind of tradeoff is to be expected for methods based on adversarial training, but not test-time training.
Revised: While surprisingly robust under some of the most severe corruptions (especially the three noise types), ALP incurs a much larger error (by a factor of two) on the original distribution and some corruptions (e.g. all levels of contrast and fog), and hurts performance signiﬁcantly when the corruptions are not as severe (especially on levels 1-3); this kind of tradeoff is to be expected for methods based on adversarial training, but not test-time training.

Original: ImageNet-C. Our results on the level 5 corruptions (most severe) are shown in Figure 2.
Revised: ImageNet-C. Our results on the level 5 corruptions (most severe) are shown in Figure 2.

Original: We use accuracy instead of error for this dataset because the baseline severely underperforms with most corruptions.
Revised: We use accuracy instead of error for this dataset because the baseline severely underperforms with most corruptions.

Original: The general trend is roughly the same as on CIFAR-10-C. Test-time training (standard version) always improves over the baseline and joint training, while the online version only hurts on the original by 0.1% over the baseline, but dramatically improves (by a factor of more than three) on many of the corruption types.
Revised: The general trend is roughly the same as on CIFAR-10-C. Test-time training (standard version) always improves over the baseline and joint training, while the online version only hurts on the original by 0.1% over the baseline, but dramatically improves (by a factor of more than three) on many of the corruption types.

Original: In the lower panel of Figure 2, we visualize how the accuracy (averaged over a sliding window) of the online version changes as more images are tested on.
Revised: In the lower panel of Figure 2, we visualize how the accuracy (averaged over a sliding window) of the online version changes as more images are tested on.

Original: Due to space constraints, we show this plot on the original test set, as well as every third corruption type, following the same order as in the original paper.
Revised: Due to space constraints, we show this plot on the original test set, as well as every third corruption type, following the same order as in the original paper.

Original: On the original test set, there is no visible change in performance after updating on the 50,000 samples.
Revised: On the original test set, there is no visible change in performance after updating on the 50,000 samples.

Original: With corruptions, accuracy has already risen signiﬁcantly after 10,000 samples, but is still rising towards the end of the 50,000 samples, indicating room for additional improvements if more samples were available.
Revised: With corruptions, accuracy has already risen signiﬁcantly after 10,000 samples, but is still rising towards the end of the 50,000 samples, indicating room for additional improvements if more samples were available.

Original: Without looking at a single label, test-time training online behaves as if we were training on the test set from the appearance of these plots.
Revised: Without looking at a single label, test-time training online behaves as if we were training on the test set from the appearance of these plots.

Original: 3. V IDEO C LASSIFICATION
Revised: 3. V IDEO C LASSIFICATION

Original: [Table]
Revised: [Table]

Original: The ImageNet Video Classiﬁcation (VID) dataset was developed by Shankar et al.
Revised: The ImageNet Video Classiﬁcation (VID) dataset was developed by Shankar et al.

Original: from the Video Detection dataset of ILSVRC2015 (Russakovsky et al., 2015), to demonstrate how deep learning models for object recognition trained on ImageNet (still images) fail to adapt well to video frames 4 .
Revised: from the Video Detection dataset of ILSVRC2015 (Russakovsky et al., 2015), to demonstrate how deep learning models for object recognition trained on ImageNet (still images) fail to adapt well to video frames 4 .

Original: Without any modiﬁcation for videos, we apply our method to VID on top of the same ImageNet model as in the previous subsection.
Revised: Without any modiﬁcation for videos, we apply our method to VID on top of the same ImageNet model as in the previous subsection.

Original: Our results are shown in Table 2.
Revised: Our results are shown in Table 2.

Original: Again, we use accuracy instead of error because the baseline performance is poor.
Revised: Again, we use accuracy instead of error because the baseline performance is poor.

Original: In addition, we take the seven classes in VID that overlap with CIFAR-10, and rescale those video frames to the size of CIFAR-10 images, as a new test set for the model trained on CIFAR-10 in the previous subsection.
Revised: In addition, we take the seven classes in VID that overlap with CIFAR-10, and rescale those video frames to the size of CIFAR-10 images, as a new test set for the model trained on CIFAR-10 in the previous subsection.

Original: Again, we apply our method to this dataset without any modiﬁcation.
Revised: Again, we apply our method to this dataset without any modiﬁcation.

Original: Our results are shown in Table 1, with a breakdown for each class.
Revised: Our results are shown in Table 1, with a breakdown for each class.

Original: Noticing that test-time training does not improve on the airplane class, we inspect some airplane samples, and observe that most of them have black margins on the sides, which provide a trivial hint for the rotation prediction task.
Revised: Noticing that test-time training does not improve on the airplane class, we inspect some airplane samples, and observe that most of them have black margins on the sides, which provide a trivial hint for the rotation prediction task.

Original: In addition, for airplanes captued in the sky, it is often impossible even for humans to tell if an image is rotated.
Revised: In addition, for airplanes captued in the sky, it is often impossible even for humans to tell if an image is rotated.

Original: This shows that our method requires the self-supervised task to be both well deﬁned and non-trivial on the new domain.
Revised: This shows that our method requires the self-supervised task to be both well deﬁned and non-trivial on the new domain.

Original: 3. CIFAR-10.1: A N EW T EST S ET W ITH U NKNOWN D ISTRIBUTION S HIFTS
Revised: 3. CIFAR-10.1: A N EW T EST S ET W ITH U NKNOWN D ISTRIBUTION S HIFTS

Original: CIFAR-10.1 (Recht et al., 2018) is a new test set of size 2000 modeled after CIFAR-10, with the exact same classes and image dimensionality, following the dataset creation process documented by the original CIFAR-10 paper as closely as possible.
Revised: CIFAR-10.1 (Recht et al., 2018) is a new test set of size 2000 modeled after CIFAR-10, with the exact same classes and image dimensionality, following the dataset creation process documented by the original CIFAR-10 paper as closely as possible.

Original: The purpose is to investigate the distribution shifts present between the two test sets, and the effect on object recognition.
Revised: The purpose is to investigate the distribution shifts present between the two test sets, and the effect on object recognition.

Original: All models tested by the authors suffer a large performance drop on CIFAR-10.1 comparing to CIFAR-10, even though there is no human noticable difference, and both have the same human accuracy.
Revised: All models tested by the authors suffer a large performance drop on CIFAR-10.1 comparing to CIFAR-10, even though there is no human noticable difference, and both have the same human accuracy.

Original: This demonstrates how insidious and ubiquitous distribution shifts are, even when researchers strive to minimize them.
Revised: This demonstrates how insidious and ubiquitous distribution shifts are, even when researchers strive to minimize them.

Original: [Table]
Revised: [Table]

Original: The distribution shifts from CIFAR-10 to CIFAR-10.1 pose an extremely difﬁcult problem, and nobody has made a successful attempt to improve the performance of an existing model on this new test set, probably because 1) Researchers cannot even identify the distribution shifts, let alone describe them with mathematics.
Revised: The distribution shifts from CIFAR-10 to CIFAR-10.1 pose an extremely difﬁcult problem, and nobody has made a successful attempt to improve the performance of an existing model on this new test set, probably because 1) Researchers cannot even identify the distribution shifts, let alone describe them with mathematics.

Original: The samples in CIFAR-10.1 are only revealed at test-time; and even if revealed during training, the distribution shifts are too subtle, and the sample size is too small, for domain adaptation algorithms (Recht et al., 2018).
Revised: The samples in CIFAR-10.1 are only revealed at test-time; and even if revealed during training, the distribution shifts are too subtle, and the sample size is too small, for domain adaptation algorithms (Recht et al., 2018).

Original: On the original CIFAR-10 test set, our baseline has error 8.9%, and with joint training 8.1%; comparing to the ﬁrst two rows of Table 3, both suffer the typical performance drop (by a factor of two).
Revised: On the original CIFAR-10 test set, our baseline has error 8.9%, and with joint training 8.1%; comparing to the ﬁrst two rows of Table 3, both suffer the typical performance drop (by a factor of two).

Original: Test-time training yields an improvement of 0.8% (relative improvement of 4.8%) over joint training.
Revised: Test-time training yields an improvement of 0.8% (relative improvement of 4.8%) over joint training.

Original: We recognize that this improvement is still small comparing to the performance drop, but see it as an encouraging ﬁrst step for this very difﬁcult problem.
Revised: We recognize that this improvement is still small comparing to the performance drop, but see it as an encouraging ﬁrst step for this very difﬁcult problem.

Original:  T OWARDS U NDERSTANDING T EST - TIME T RAINING
Revised:  T OWARDS U NDERSTANDING T EST - TIME T RAINING

Original: This section contains our preliminary study of when and why test-time training is expected to work.
Revised: This section contains our preliminary study of when and why test-time training is expected to work.

Original: For convex models, we prove that positive gradient correlation between the loss functions leads to better performance on the main task after test-time training.
Revised: For convex models, we prove that positive gradient correlation between the loss functions leads to better performance on the main task after test-time training.

Original: Equipped with this insight, we then empirically demonstrate that gradient correlation governs the success of test-time training on the deep learning model discussed in Section 3.
Revised: Equipped with this insight, we then empirically demonstrate that gradient correlation governs the success of test-time training on the deep learning model discussed in Section 3.

Original: Before stating our main theoretical result, we ﬁrst illustrate the general intuition with a toy model.
Revised: Before stating our main theoretical result, we ﬁrst illustrate the general intuition with a toy model.

Original: Consider a regression problem where x ∈ R d denotes the input, y 1 ∈ R denotes the label, and the objective is the square loss (ˆ y − y 1 ) 2 / 2 for a prediction ˆ y .
Revised: Consider a regression problem where x ∈ R d denotes the input, y 1 ∈ R denotes the label, and the objective is the square loss (ˆ y − y 1 ) 2 / 2 for a prediction ˆ y .

Original: Consider a two layer linear network parametrized by A ∈ R h × d and v ∈ R h (where h stands for the hidden dimension).
Revised: Consider a two layer linear network parametrized by A ∈ R h × d and v ∈ R h (where h stands for the hidden dimension).

Original: The prediction according to this model is ˆ y = v (cid:62) A x , and the main task loss is
Revised: The prediction according to this model is ˆ y = v (cid:62) A x , and the main task loss is

Original: [Equation]
Revised: [Equation]

Original: In addition, consider a self-supervised regression task that also uses the square loss and automatically generates a label y s for x .
Revised: In addition, consider a self-supervised regression task that also uses the square loss and automatically generates a label y s for x .

Original: Let the self-supervised head be parameterized by w ∈ R h .
Revised: Let the self-supervised head be parameterized by w ∈ R h .

Original: Then the selfsupervised task loss is
Revised: Then the selfsupervised task loss is

Original: [Equation]
Revised: [Equation]

Original: Now we apply test-time training to update the shared feature extractor
Revised: Now we apply test-time training to update the shared feature extractor

Original: A by one step of gradient descent on l s , which we can compute with y 2 known.
Revised: A by one step of gradient descent on l s , which we can compute with y 2 known.

Original: This gives us
Revised: This gives us

Original: [Equation]
Revised: [Equation]

Original: (6) where A (cid:48) is the updated matrix and η is the learning rate.
Revised: (6) where A (cid:48) is the updated matrix and η is the learning rate.

Original: If we set η = η ∗ where
Revised: If we set η = η ∗ where

Original: [Equation]
Revised: [Equation]

Original: then with some simple algebra, it is easy to see that the main task loss
Revised: then with some simple algebra, it is easy to see that the main task loss

Original: l m ( x, y 1 ; A (cid:48) , v ) = 0 .
Revised: l m ( x, y 1 ; A (cid:48) , v ) = 0 .

Original: Concretely, test-time training drives the main task loss down to zero with a single gradient step for a carefully chosen learning rate.
Revised: Concretely, test-time training drives the main task loss down to zero with a single gradient step for a carefully chosen learning rate.

Original: In practice, this learning rate is unknown since it depends on the unknown y 1 .
Revised: In practice, this learning rate is unknown since it depends on the unknown y 1 .

Original: However, since our model is convex, as long as η ∗ is positive, it sufﬁces to set η to
Revised: However, since our model is convex, as long as η ∗ is positive, it sufﬁces to set η to

Original: [Figure]
Revised: [Figure]

Original: be a small positive constant (see Lemma 1).
Revised: be a small positive constant (see Lemma 1).

Original: If x (cid:54) = 0 , one sufﬁcient condition for η ∗ to be positive (when neither loss is zero) is to have
Revised: If x (cid:54) = 0 , one sufﬁcient condition for η ∗ to be positive (when neither loss is zero) is to have

Original: [Equation]
Revised: [Equation]

Original: and v (cid:62)

Original: [Equation]
Revised: [Equation]

Original: For our toy model, both parts in Equation 8 have an intuition interpretation.
Revised: For our toy model, both parts in Equation 8 have an intuition interpretation.

Original: The ﬁrst part says that the mistakes should be correlated, in the sense that predictions from both tasks are mistaken in the same direction.
Revised: The ﬁrst part says that the mistakes should be correlated, in the sense that predictions from both tasks are mistaken in the same direction.

Original: The second part, v (cid:62) w > 0 , says that the decision boundaries on the feature space should be correlated.
Revised: The second part, v (cid:62) w > 0 , says that the decision boundaries on the feature space should be correlated.

Original: In fact, these two parts hold iff (cid:104)∇ [Equation] A [Equation]
Revised: In fact, these two parts hold iff (cid:104)∇ l m ( A ) , ∇ l s ( A ) (cid:105) > 0 (see Lemma 2).

Original: , ∇

Original: [Equation]

Original: A [Equation] 0 (see Lemma 2). To summarize, if the gradients have positive correlation, test-time training is guaranteed to reduce the main task loss.
Revised: To summarize, if the gradients have positive correlation, test-time training is guaranteed to reduce the main task loss.

Original: Our main theoretical result extends this to general smooth and convex loss functions.
Revised: Our main theoretical result extends this to general smooth and convex loss functions.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let l m ( x, y ; θ ) denote the main task loss on test instance x, y with parameters θ , and l s ( x ; θ ) the self-supervised task loss that only depends on x 5 .
Revised: Let l m ( x, y ; θ ) denote the main task loss on test instance x, y with parameters θ , and l s ( x ; θ ) the self-supervised task loss that only depends on x 5 .

Original: Assume that for all x, y , l m ( x, y ; θ ) is differentiable, convex and β -smooth in θ , and both (cid:107)∇ l m ( x, y ; θ ) (cid:107) , (cid:107)∇ l s ( x, θ ) (cid:107) ≤ G for all θ .
Revised: Assume that for all x, y , l m ( x, y ; θ ) is differentiable, convex and β -smooth in θ , and both (cid:107)∇ l m ( x, y ; θ ) (cid:107) , (cid:107)∇ l s ( x, θ ) (cid:107) ≤ G for all θ .

Original: With a ﬁxed learning rate η = (cid:15)βG 2 , for every x, y such that
Revised: With a ﬁxed learning rate η = (cid:15)βG 2 , for every x, y such that

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where θ ( x ) = θ − η ∇ l s ( x ; θ ) i.e. test-time training with one step of gradient descent.
Revised: where θ ( x ) = θ − η ∇ l s ( x ; θ ) i.e. test-time training with one step of gradient descent.

Original: The proof is in Appendix B.2.
Revised: The proof is in Appendix B.2.

Original: Theorem 1 reveals gradient correlation as a determining factor of the success of test-time training in the smooth and convex case.
Revised: Theorem 1 reveals gradient correlation as a determining factor of the success of test-time training in the smooth and convex case.

Original: In Figure 3, we empirically show that our insight also holds for non-convex loss functions, on the deep learning model and across the diverse set of corruptions considered in Section 3; stronger gradient correlation clearly indicates higher performance improvements over the baseline.
Revised: In Figure 3, we empirically show that our insight also holds for non-convex loss functions, on the deep learning model and across the diverse set of corruptions considered in section 3; stronger gradient correlation clearly indicates higher performance improvements over the baseline.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work has been inﬂuenced by the successes and limitations of many related ﬁelds 6 .
Revised: Our work has been inﬂuenced by the successes and limitations of many related ﬁelds.

Revised: Each of these ﬁelds contains maybe hundreds of interesting works, which we unfortunately do not have enough time and space to acknowledge in this draft.

Revised: We apologize for the omissions and are happy to include additional citations upon request.

Original: Learning on test instances.
Revised: Learning on test instances.

Original: Our idea is inspired by Glasner et al. 
Revised: Concurrent work Bau et al. (2019) improves photo manipulation with a generative adversarial network, by adapting its image prior to the statistics of the input image.

Original: ; Shocher et al.
Revised: Shocher et al.

Original: (2018), which learns to perform super-resolutiononly on a single image, by learning to recover the original image itself from its downsampled version.
Revised: (2018) performs super-resolution on an image, by learning to recover the original image itself from its downsampled version.

Original: In addition, Jain & Learned-Miller (2011) improves viola-jones for face detection, by using instances of faceswith high conﬁdence in an image to bootstrap the more difﬁcult ones  in the same image.
Revised: An older work, Jain & Learned-Miller (2011) improves Viola-Jones (Viola et al., 2001) for face detection, by bootstrapping the more difﬁcult faces in an image from the easier ones with high conﬁdence in the same image.

Original: The online version of our algorithm is inspired by Mullapudi et al.
Revised: The online version of our algorithm is inspired by Mullapudi et al.

Original: (2018), which makes video segmentation more efﬁcient by using a student model that learns online from a teacher model.
Revised: (2018), which makes video segmentation more efﬁcient by using a student model that learns online from a teacher model.

Original: The idea of online updates has also been used in Kalal et al.
Revised: The idea of online updates has also been used in Kalal et al.

Original: (2011) for tracking and detection.
Revised: (2011) for tracking and detection.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2019), a concurrent work in echocardiography, improves the deep learning model that tracks myocardial motion and cardiac blood ﬂow with sequential updates.
Revised: (2019), a concurrent work in echocardiography, improves the deep learning model that tracks myocardial motion and cardiac blood ﬂow with sequential updates.

Original: Lastly, we share the philosophy of transductive learning (Vapnik, 2013; Gammerman et al., 1998), but have little in common with their classical algorithms e.g. nearest neighbors.
Revised: Lastly, we share the philosophy of transductive learning (Vapnik, 2013; Gammerman et al., 1998), but have little in common with their classical algorithms; concurrent work Tripuraneni & Mackey (2019) theoretically explores this for linear prediction.

Original: Adversarial robustness studies the robust risk: R P, ∆ ( θ ) =
Revised: Adversarial robustness studies the robust risk: R P, ∆ ( θ ) =

Original: E x,y [Equation] P max δ ∈ ∆ l ( x + δ, y ; θ ) where l is some loss function, and ∆ is the set of perturbations;one popular choice of ∆ is  the L p ball, for p ∈ { 1 , 2 , ∞} .
Revised: E x,y ∼ P max δ ∈ ∆ l ( x + δ, y ; θ ) where l is some loss function, and ∆ is the set of perturbations; ∆ is often chosen as the L p ball, for p ∈ { 1 , 2 , ∞} .

Original: Works in this area rely heavily on the mathematical structure of ∆ , which might not be realistic models of perturbations in the real world. The most popular algorithms use either robust optimization (Sinha et al., 2017; Raghunathan et al., 2018; Wong & Kolter, 2017; 
Revised: Many popular algorithms formulate and solve this as a robust optimization problem (Goodfellow et al., 2014; Madry et al., 2017; Sinha et al., 2017; Raghunathan et al., 2018; Wong & Kolter, 2017; Croce et al., 2018), and the most well known technique is adversarial training. Another line of work is based on randomized smoothing (Cohen et al., 2019;

Original: Croce et al., 2018), or input transformations (Guo et al., 2017b; Song et al., 2017), both can be seen as smoothing the decision boundary.
Revised: Salman et al., 2019), while some other approaches such as input transformations (Guo et al., 2017b; Song et al., 2017) are shown to be less effective (Athalye et al., 2018). There are two main problems in this ﬁeld. First, all the approaches mentioned above can be seen as smoothing the decision boundary.

Original: This establishes a theoretical tradeoff between accuracy and robustness (Zhang et al., 2019); intuitively, the more diverse ∆ is, the less effective this one-boundary-ﬁts-all approach can be for a particular subset of ∆ . 
Revised: This establishes a theoretical tradeoff between accuracy and robustness (Tsipras et al., 2018; Zhang et al., 2019), which we also observe empirically with our adversarial training baseline in Section 3. Intuitively, the more diverse ∆ is, the less effective this one-boundary-ﬁts-all approach can be for a particular element of ∆ . Second, adversarial methods rely heavily on the mathematical structure of ∆ , which might not accurately model perturbations in the real world. Therefore, generalization remains hard outside of the ∆ we know in advance or can mathematically model, especially for non-adversarial distribution shifts.

Original:  Kang et al.
Revised: Empirically, Kang et al.

Original: (2019) show thatempirically, robustness for one ∆ might not transfer to another, and training on the L ∞ ball actually hurts robustness on the L 1 ball.
Revised: (2019) shows that robustness for one ∆ might not transfer to another, and training on the L ∞ ball actually hurts robustness on the L 1 ball.

Original: Non-adversarial robustness studies the effect of corruptions, perturbations, out-of-distribution examples, and real-world distribution shifts (Hendrycks et al., 2019a;b; 2018; Hendrycks & Gimpel, 2016).
Revised: Non-adversarial robustness studies the effect of corruptions, perturbations, out-of-distribution examples, and real-world distribution shifts (Hendrycks et al., 2019a;b; 2018; Hendrycks & Gimpel, 2016).

Original: Geirhos et al.
Revised: Geirhos et al.

Original: (2018) show that training on images corrupted by Gaussian noise makes deep learning models recover super-human performance on this particular noise type, but cannot improve performance on another those corrupted by another noise type e.g. salt-and-pepper noise.
Revised: (2018) show that training on images corrupted by Gaussian noise makes deep learning models recover super-human performance on this particular noise type, but cannot improve performance on another those corrupted by another noise type e.g. salt-and-pepper noise.

Original: Unsupervised domain adaptation (a.k.a. transfer learning) studies the problem of distribution shift (from P to Q ), when unlabled data from Q is available at training-time (Tzeng et al., 2017;
Revised: Unsupervised domain adaptation (a.k.a. transfer learning) studies the problem of distribution shift (from P to Q ), when unlabled data from Q is available at training-time (Tzeng et al., 2017;

Original: Ganin et al., 2016;
Revised: Ganin et al., 2016;

Original: Gong et al., 2012; Long et al., 2016; Chen et al., 2018; 2011; Hoffman et al., 2017; Csurka, 2017; Long et al., 2015).
Revised: Gong et al., 2012; Long et al., 2016; Chen et al., 2018; 2011; Hoffman et al., 2017; Csurka, 2017; Long et al., 2015).

Original: We are inspired by this very active and successful community, especially (Sun et al., 2019), and believe that progress in this community can motivate new algorithms in the framework of test-time learning.
Revised: We are inspired by this very active and successful community, especially (Sun et al., 2019), and believe that progress in this community can motivate new algorithms in the framework of test-time learning.

Original: Our update rule can be viewed as performing one-sample unsupervised domain adaptation on the ﬂy 7 .
Revised: Our update rule can be viewed as performing one-sample unsupervised domain adaptation on the ﬂy 6 .

Original: On the other hand, test-time learning comes from realizing the limitations of the unsupervised domain adaptation setting, that outside of the speciﬁc target distribution where data is available for training, generalization is still elusive.
Revised: On the other hand, test-time learning comes from realizing the limitations of the unsupervised domain adaptation setting, that outside of the speciﬁc target distribution where data is available for training, generalization is still elusive.

Original: Previous works make the source and target distributions broader with multiple and evolving sources and targets without fundamentally address this problem (Hoffman et al., 2018; 2012; 2014).
Revised: Previous works make the source and target distributions broader with multiple and evolving sources and targets without fundamentally address this problem (Hoffman et al., 2018; 2012; 2014).

Original: Self-supervised learning studies how to create labels from the data, by designing ingenious tasks that contain semantic information without human annotations, such as context prediction (Doersch et al., 2015), solving jigsaw puzzles (Noroozi & Favaro, 2016), colorization (Larsson et al., 2017;
Revised: Self-supervised learning studies how to create labels from the data, by designing ingenious tasks that contain semantic information without human annotations, such as context prediction (Doersch et al., 2015), solving jigsaw puzzles (Noroozi & Favaro, 2016), colorization (Larsson et al., 2017;

Original: Zhang et al., 2016), noise prediction (Bojanowski & Joulin, 2017), and feature clustering (Caron et al., 2018).
Revised: Zhang et al., 2016), noise prediction (Bojanowski & Joulin, 2017), and feature clustering (Caron et al., 2018).

Revised: Self-supervision has also been used on videos (Wang & Gupta, 2015; Wang et al., 2019).

Revised: Particularly relevant to our work, Asano et al.

Revised: (2019) shows that self-supervised learning on only a single image, surprisingly, can produce low-level features that generalize well.

Revised: In addition,

Original: Hendrycks et al.
Revised: Hendrycks et al.

Original: (2019b) proposes that jointly training a main task and a self-supervised task (our joint training baseline in section 3) can improve robustness of the main task.
Revised: (2019b) proposes that jointly training a main task and a self-supervised task (our joint training baseline in section 3) can improve robustness of the main task.

Original: The same idea is used in few-shot learning (Su et al., 2019), and domain generalization (Carlucci et al., 2019).
Revised: The same idea is used in few-shot learning (Su et al., 2019), and domain generalization (Carlucci et al., 2019).

Original: Domain generalization studies when a meta distribution generates multiple environment distributions, some of which are available during training (source), while others are used for testing (target) (Li et al., 2018; Shankar et al., 2018; Muandet et al., 2013;
Revised: Domain generalization studies when a meta distribution generates multiple environment distributions, some of which are available during training (source), while others are used for testing (target) (Li et al., 2018; Shankar et al., 2018; Muandet et al., 2013;

Original: Balaji et al., 2018; Ghifary et al., 2015;
Revised: Balaji et al., 2018; Ghifary et al., 2015;

Original: Motiian et al., 2017; Li et al., 2017a; Gan et al., 2016).
Revised: Motiian et al., 2017; Li et al., 2017a; Gan et al., 2016).

Original: With only a few environments, information on the meta distribution is often too scarce to be helpful, and with many environments, we are back to the i.i.d.
Revised: With only a few environments, information on the meta distribution is often too scarce to be helpful, and with many environments, we are back to the i.i.d.

Original: setting where each environment can be seen as a sample, and a strong baseline is to simply train on all the environments (Li et al., 2019).
Revised: setting where each environment can be seen as a sample, and a strong baseline is to simply train on all the environments (Li et al., 2019).

Original: The setting of domain generalization is limited by the inherent tradeoff between speciﬁcity and generality of a ﬁxed decision boundary, and the fact that generalization is again elusive outside of the meta distribution i.e. the actual P .
Revised: The setting of domain generalization is limited by the inherent tradeoff between speciﬁcity and generality of a ﬁxed decision boundary, and the fact that generalization is again elusive outside of the meta distribution i.e. the actual P .

Original: Continual learning (a.k.a.
Revised: Continual learning (a.k.a.

Original: learning without forgetting) studies when a model is made to learn a sequence of tasks, and not forget about the task at the beginning (Li & Hoiem, 2017; Lopez-Paz & Ranzato, 2017; Kirkpatrick et al., 2017; Santoro et al., 2016).
Revised: learning without forgetting) studies when a model is made to learn a sequence of tasks, and not forget about the task at the beginning (Li & Hoiem, 2017; Lopez-Paz & Ranzato, 2017; Kirkpatrick et al., 2017; Santoro et al., 2016).

Original: In comparison, test-time learning does not careat all about forgetting (and might even encourage it).
Revised: Test-time training does not care about forgetting the past test instances since they have already been evaluated on; and if a past instance comes up by any chance, it would go through test-time training again. In addition, the impact of forgetting the training set is minimal, because both tasks have been jointly trained. This is in contrast to continual learning, when the tasks are trained one-by-one from scratch.

Original: Few (one)-shot learning studies extremely small training sets (maybe for some categories)
Revised: Few (one)-shot learning studies extremely small training sets (maybe for some categories)

Original: (Snell et al., 2017; Vinyals et al., 2016; Fei-Fei et al., 2006; Ravi & Larochelle, 2016; Li et al., 2017b;
Revised: (Snell et al., 2017; Vinyals et al., 2016; Fei-Fei et al., 2006; Ravi & Larochelle, 2016; Li et al., 2017b;

Original: Finn et al., 2017; Gidaris & Komodakis, 2018).
Revised: Finn et al., 2017; Gidaris & Komodakis, 2018).

Original: Our update rule can be viewed as performing one-shot self-supervised learning and can potentially be improved by progress in few-shot learning.
Revised: Our update rule can be viewed as performing one-shot self-supervised learning and can potentially be improved by progress in few-shot learning.

Original: Online learning (a.k.a.
Revised: Online learning (a.k.a.

Original: online optimization) is a well-studied area of learning theory (ShalevShwartz et al., 2012; Hazan et al., 2016).
Revised: online optimization) is a well-studied area of learning theory (ShalevShwartz et al., 2012; Hazan et al., 2016).

Original: The basic setting repeats the following: receive x t , predict ˆ y t , receive y t from a worst-case oracle and learn.
Revised: The basic setting repeats the following: receive x t , predict ˆ y t , receive y t from a worst-case oracle and learn.

Original: Final performance is evaluated using the regret, colloquially how much worse than the best ﬁxed model in hindsight.
Revised: Final performance is evaluated using the regret, colloquially how much worse than the best ﬁxed model in hindsight.

Original: It is easy to see how our setting differs, even for the online version.
Revised: It is easy to see how our setting differs, even for the online version.

Original: We learn before predicting ˆ y t , but never receive any y t that is evaluated on, thus do not need to consider the worst-case orcale or the regret.  
Revised: We learn before predicting ˆ y t , but never receive any y t that is evaluated on, thus do not need to consider the worst-case oracle or the regret.  C ONCLUSION

Revised: The idea of test-time training also makes sense for other tasks such as segmentation and detection, and in other ﬁelds such as speech recognition and natural language processing.

Revised: More generally, we hope this paper can encourage researchers to abandon the self-imposed constraint of a ﬁxed decision boundary at test-time, or even altogether, the artiﬁcial divison between training and testing.

Original: A A T HEORETICAL D ISCUSSION ON O UR V ARIABLE D ECISION B OUNDARY
Revised: A I NFORMAL D ISCUSSION ON O UR V ARIABLE D ECISION B OUNDARY

Original: In ?? , we claim that in traditional supervised learning θ gives a ﬁxed decision boundary, while our θ gives a variable decision boundary.
Revised: In section 1, we claim that in traditional supervised learning θ gives a ﬁxed decision boundary, while our θ gives a variable decision boundary.

Original: Here we explain this claim from a theoretical perspective.
Revised: Here we informally discuss this claim.

Original: Denote the input space X and output space Y .
Revised: Denote the input space X and output space Y .

Original: A decision boundary is simply a mapping f : X → Y .
Revised: A decision boundary is simply a mapping f : X → Y .

Original: Let Θ be a model class e.g R d .
Revised: Let Θ be a model class e.g R d .

Original: Now consider a family of parametrized functions g θ : X → Y , where θ ∈ Θ .
Revised: Now consider a family of parametrized functions g θ : X → Y , where θ ∈ Θ .

Original: In the context of deep learning, g is the neural network architecture and θ contains the parameters.
Revised: In the context of deep learning, g is the neural network architecture and θ contains the parameters.

Original: We say that f is a ﬁxed decision boundary w.r.t.
Revised: We say that f is a ﬁxed decision boundary w.r.t.

Original: g and Θ if there exists θ ∈ Θ s.t. f
Revised: g and Θ if there exists θ ∈ Θ s.t. f

Original: ( x ) =
Revised: ( x ) =

Original: g θ ( x ) for every x ∈ X , and a variable decision boundary if for every x ∈ X , there exists θ ∈ Θ s.t. f
Revised: g θ ( x ) for every x ∈ X , and a variable decision boundary if for every x ∈ X , there exists θ ∈ Θ s.t. f

Original: ( x ) =
Revised: ( x ) =

Original: g θ ( x ) .
Revised: g θ ( x ) .

Original: Note how selection of θ can depend on x for a variable decision boundary, and cannot for a ﬁxed one.
Revised: Note how selection of θ can depend on x for a variable decision boundary, and cannot for a ﬁxed one.

Original: It is then trivial to verify that our claim is true under those deﬁnitions.
Revised: It is then trivial to verify that our claim is true under those deﬁnitions.

Original: A critical reader might say that with an arbitrarily large model class, can’t every decision boundary be ﬁxed?
Revised: A critical reader might say that with an arbitrarily large model class, can’t every decision boundary be ﬁxed?

Original: Yes, but this is not the end of the story.
Revised: Yes, but this is not the end of the story.

Original: Let d = dim( X ) × dim( Y ) , and consider the enormous model class Θ (cid:48)
Revised: Let d = dim( X ) × dim( Y ) , and consider the enormous model class Θ (cid:48)

Original: =
Revised: =

Original: R d which is capable of representing all possible mappings between X and Y .
Revised: R d which is capable of representing all possible mappings between X and Y .

Original: Let g (cid:48) θ (cid:48) simply be the mapping represented by θ (cid:48) ∈ Θ (cid:48) .
Revised: Let g (cid:48) θ (cid:48) simply be the mapping represented by θ (cid:48) ∈ Θ (cid:48) .

Original: A variable decision boundary w.r.t.
Revised: A variable decision boundary w.r.t.

Original: g and Θ then indeed must be a ﬁxed decision boundary w.r.t.
Revised: g and Θ then indeed must be a ﬁxed decision boundary w.r.t.

Original: g (cid:48) and Θ (cid:48) , but we would like to note two things.
Revised: g (cid:48) and Θ (cid:48) , but we would like to note two things.

Original: First, without any prior knowledge, generalization in Θ (cid:48) is impossible with any ﬁnite amount of training data; reasoning about g (cid:48) and Θ (cid:48) is most likely not productive from an algorithmic point of view, and the concept of a variable decision boundary is to avoid such reasoning.
Revised: First, without any prior knowledge, generalization in Θ (cid:48) is impossible with any ﬁnite amount of training data; reasoning about g (cid:48) and Θ (cid:48) is most likely not productive from an algorithmic point of view, and the concept of a variable decision boundary is to avoid such reasoning.

Original: Second,selecting θ based on x for a variable decision boundary can be thought of as “training” on all points x ∈ R d ; however, “training” only happens when necessary, for the x that it actually encounters.
Revised: Second, selecting θ based on x for a variable decision boundary can be thought of as “training” on all points x ∈ R d ; however, “training” only happens when necessary, for the x that it actually encounters.

Original: Altogether, the concept of a variable decison boundary is different from what can be described by traditional learning theory.
Revised: Altogether, the concept of a variable decison boundary is different from what can be described by traditional learning theory.

Original: A more in depth discussion is beyond the scope of this paper.
Revised: A formal discussion is beyond the scope of this paper.

Original: B P ROOFS
Revised: B P ROOFS

Original: Here we prove the theoretical results covered in Section 4.
Revised: Here we prove the theoretical results covered in section 4.

Original: B. P ROOFS FOR THE TOY PROBLEM
Revised: B. P ROOFS FOR THE TOY PROBLEM

Original: The following setting applies to the two lemmas; this is simply the setting of our toy problem, reproduced here for ease of reference.
Revised: The following setting applies to the two lemmas; this is simply the setting of our toy problem, reproduced here for ease of reference.

Original: Consider a two layer linear network parametrized by A ∈ R h × d (shared) and v , w ∈ R h (ﬁxed) for the two heads, respectively.
Revised: Consider a two layer linear network parametrized by A ∈ R h × d (shared) and v , w ∈ R h (ﬁxed) for the two heads, respectively.

Original: Denote x ∈ R d the input and y 1 , y 2 ∈ R the labels for the two tasks, respectively.
Revised: Denote x ∈ R d the input and y 1 , y 2 ∈ R the labels for the two tasks, respectively.

Original: For the main task loss
Revised: For the main task loss

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) and the self-supervised task loss
Revised: ) and the self-supervised task loss

Original: [Equation]
Revised: [Equation]

Original: (12) test-time learning yields an updated matrix
Revised: (12) test-time learning yields an updated matrix

Original: [Equation]
Revised: [Equation]

Original: (13) where η is the learning rate.
Revised: (13) where η is the learning rate.

Original: Lemma 1.
Revised: Lemma 1.

Original: Following the exposition of section 4, denote
Revised: Following the exposition of section 4, denote

Original: [Equation]
Revised: [Equation]

Original: Assume η ∗ ∈ [ (cid:15), ∞ ) for some (cid:15) > 0 .
Revised: Assume η ∗ ∈ [ (cid:15), ∞ ) for some (cid:15) > 0 .

Original: Then for any η ∈ (0 , (cid:15) ] , we are guaranteed an improvement on the main loss i.e. l m ( A (cid:48) ) <
Revised: Then for any η ∈ (0 , (cid:15) ] , we are guaranteed an improvement on the main loss i.e. l m ( A (cid:48) ) <

Original: l m ( A ) .
Revised: l m ( A ) .

Original: Proof.
Revised: Proof.

Original: From the exposition of section 4, we know that
Revised: From the exposition of section 4, we know that

Original: [Equation]
Revised: [Equation]

Original: which can also be derived from simple algebra.
Revised: which can also be derived from simple algebra.

Original: Then by convexity, we have
Revised: Then by convexity, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses the assumption that l m ( A ) >
Revised: where the last inequality uses the assumption that l m ( A ) >

Original: [Equation]
Revised: [Equation]

Original: which holds because η ∗ > 0 .
Revised: which holds because η ∗ > 0 .

Original: Lemma 2.
Revised: Lemma 2.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. the Frobenious inner product, then
Revised: e. the Frobenious inner product, then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By simple algebra,
Revised: By simple algebra,

Original: [Equation]
Revised: [Equation]

Original: 22) which has the same sign as η ∗ .
Revised: 22) which has the same sign as η ∗ .

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: For any η , by smoothness and convexity,
Revised: For any η , by smoothness and convexity,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: Then Equation 23 becomes
Revised: Then Equation 23 becomes

Original: [Equation]
Revised: [Equation]

Original: And by our assumptions on the gradient norm and gradient inner product,
Revised: And by our assumptions on the gradient norm and gradient inner product,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Because we cannot observe η ∗ in practice, we instead use a ﬁxed learning rate η = (cid:15) βG 2 , as stated in
Revised: Because we cannot observe η ∗ in practice, we instead use a ﬁxed learning rate η = (cid:15) βG 2 , as stated in

Original: Theorem 1.
Revised: Theorem 1.

Original: Now we argue that this ﬁxed learning rate still improves performance on the main task.
Revised: Now we argue that this ﬁxed learning rate still improves performance on the main task.

Original: By our assumptions, η ∗ ≥ (cid:15)βG 2 , so η ∈ (0 , η ∗ ] .
Revised: By our assumptions, η ∗ ≥ (cid:15)βG 2 , so η ∈ (0 , η ∗ ] .

Original: Denote g = ∇ l s ( x ; θ ) , then by convexity of l m , l m ( x, y ; θ
Revised: Denote g = ∇ l s ( x ; θ ) , then by convexity of l m , l m ( x, y ; θ

Original: [Equation]
Revised: [Equation]

Original: x ))
Revised: x ))

Original: =
Revised: =

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: ( x,
Revised: ( x,

Original: [Equation]
Revised: [Equation]

Original: Combining with Equation
Revised: Combining with Equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Since η/η ∗ >
Revised: Since η/η ∗ >

Original: [Equation]
Revised: [Equation]

Original: we have shown that
Revised: we have shown that

Original: [Equation]
Revised: [Equation]

Original: C C OMPUTATIONAL A SPECTS OF O UR M ETHOD
Revised: C C OMPUTATIONAL A SPECTS OF O UR M ETHOD

Original: At test-time, our method is 2 × batch size × number of iterations times slower than regular testing, which only performs a single forward pass for each sample.
Revised: At test-time, our method is 2 × batch size × number of iterations times slower than regular testing, which only performs a single forward pass for each sample.

Original: As the ﬁrst work on test-time learning, this paper is not as concerned about computational efﬁciency as improving robustness, but here we provide two potential solutions that might be useful, but have not been thoroughly veriﬁed.
Revised: As the ﬁrst work on test-time learning, this paper is not as concerned about computational efﬁciency as improving robustness, but here we provide two potential solutions that might be useful, but have not been thoroughly veriﬁed.

Original: The ﬁrst is to use the thresholding trick on l s , introduced as a solution for the small batches problem in section 2.
Revised: The ﬁrst is to use the thresholding trick on l s , introduced as a solution for the small batches problem in section 2.

Original: For the models considered in our experiments, roughly 80% of the test instances fall below the threshold, so test-time learning can only be performed on the other 20% without much effect on performance, because those 20% contain most of the samples with wrong predictions.
Revised: For the models considered in our experiments, roughly 80% of the test instances fall below the threshold, so test-time learning can only be performed on the other 20% without much effect on performance, because those 20% contain most of the samples with wrong predictions.

Original: The second is to reduce the number of iterations of test-time updates.
Revised: The second is to reduce the number of iterations of test-time updates.

Original: For the online version, the number of iterations is already 1, so there is nothing to do.
Revised: For the online version, the number of iterations is already 1, so there is nothing to do.

Original: For the standard version, we have done some preliminary experiments setting number of iterations to 1 (instead of 10) and learning rate to 0.01 (instead of 0.001), and observing results almost as good as the standard hyper-parameter setting.
Revised: For the standard version, we have done some preliminary experiments setting number of iterations to 1 (instead of 10) and learning rate to 0.01 (instead of 0.001), and observing results almost as good as the standard hyperparameter setting.

Original: A more in depth discussion on efﬁciency is left for future works, which might, during training, explicitly make the model amenable to fast updates.
Revised: A more in depth discussion on efﬁciency is left for future works, which might, during training, explicitly make the model amenable to fast updates.

Original: D S AMPLE I MAGES FROM THE C OMMON C ORRUPTIONS B ENCHMARK
Revised: D S AMPLE I MAGES FROM THE C OMMON C ORRUPTIONS B ENCHMARK

Original: [Figure]
Revised: [Figure]

Original: E A DDITIONAL R ESULTS ON THE C OMMON C ORRUPTIONS D ATASET
Revised: E A DDITIONAL R ESULTS ON THE C OMMON C ORRUPTIONS D ATASET

Original: For table aethetics, we use the following abbreviations: B for baseline, JT for joint training, TTT for test-time training standard version, and TTTO for test-time training online version.
Revised: For table aethetics, we use the following abbreviations: B for baseline, JT for joint training, TTT for test-time training standard version, and TTTO for test-time training online version.

Original: We have also abbreviated the names of the corruptions.
Revised: We have also abbreviated the names of the corruptions.

Original: The full names are, in order: original test set,
Revised: The full names are, in order: original test set,

Original: Gaussian noise, shot noise, impulse noise, defocus blur, glass blue, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic transformation, pixelation, and JPEG compression.
Revised: Gaussian noise, shot noise, impulse noise, defocus blur, glass blue, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic transformation, pixelation, and JPEG compression.

Original: E. R ESULTS USING B ATCH N ORMALIZATION
Revised: E. R ESULTS USING B ATCH N ORMALIZATION

Original: As discussed in section 3, Batch Normalization (BN) is ineffective for small batches, which are the inputs for test-time training (both standard and online version) since there is only one sample available when forming each batch; therefore, our main results are based on a ResNet using Group
Revised: As discussed in section 3, Batch Normalization (BN) is ineffective for small batches, which are the inputs for test-time training (both standard and online version) since there is only one sample available when forming each batch; therefore, our main results are based on a ResNet using Group

Original: Normalization (GN).
Revised: Normalization (GN).

Original: Here we provide results of our method on CIFAR-10-C level 5, with a ResNet using Batch Normalization (BN).
Revised: Here we provide results of our method on CIFAR-10-C level 5, with a ResNet using Batch Normalization (BN).

Original: These results are meant to be merely a point of reference for the curious readers, instead of our technical contributions.
Revised: These results are meant to be merely a point of reference for the curious readers, instead of our technical contributions.

Original: In the early stage of this project
Revised: In the early stage of this project

Original: [Table]
Revised: [Table]

Original: we have experimented with two potential solutions to the small batches problem with BN.
Revised: we have experimented with two potential solutions to the small batches problem with BN.

Original: The naive solution is to ﬁx the BN layers during test-time training.
Revised: The naive solution is to ﬁx the BN layers during test-time training.

Original: but this diminishes the performance gains since there are fewer shared parameters.
Revised: but this diminishes the performance gains since there are fewer shared parameters.

Original: The better solution, adopted for the results below, is hard example mining: instead of updating on all inputs, we only update on inputs that incur large self-supervised task loss l s , where the large improvements might counter the negative effects of inaccurate statistics.
Revised: The better solution, adopted for the results below, is hard example mining: instead of updating on all inputs, we only update on inputs that incur large self-supervised task loss l s , where the large improvements might counter the negative effects of inaccurate statistics.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Test-time training (standard version) is still very effective with BN.
Revised: Test-time training (standard version) is still very effective with BN.

Original: In fact, some of the improvements are quite dramatic, such as on contrast (34%), defocus blue (18%) and Gaussian noise (22% comparing to joint-training, and 16% comparing to the baseline).
Revised: In fact, some of the improvements are quite dramatic, such as on contrast (34%), defocus blue (18%) and Gaussian noise (22% comparing to joint-training, and 16% comparing to the baseline).

Original: Performance on the original distribution is still almost the same, and the original error with BN is in fact slightly lower than with
Revised: Performance on the original distribution is still almost the same, and the original error with BN is in fact slightly lower than with

Original: GN, and takes half as many epochs to converge.
Revised: GN, and takes half as many epochs to converge.

Original: We did not further experiment with BN because of two reasons: 1)
Revised: We did not further experiment with BN because of two reasons: 1)

Original: The online version does not work with BN, because the problem with inaccurate batch statistics is exacerbated when training online for many (e.g. 10000) steps. 2)
Revised: The online version does not work with BN, because the problem with inaccurate batch statistics is exacerbated when training online for many (e.g. 10000) steps. 2)

Original: The baseline error for almost every corruption type is signiﬁcantly higher with BN than with GN.
Revised: The baseline error for almost every corruption type is signiﬁcantly higher with BN than with GN.

Original: Although unrelated to the main idea of our paper, we make the interesting note that GN signiﬁcantly improves model robustness .
Revised: Although unrelated to the main idea of our paper, we make the interesting note that GN signiﬁcantly improves model robustness .

Original: E. A DDITIONAL B ASELINE : A DVERSARIAL L OGIT P AIRING
Revised: E. A DDITIONAL B ASELINE : A DVERSARIAL L OGIT P AIRING

Original: As discussed in subsection 3.1, Hendrycks & Dietterich (2019) point to Adversarial Logit Pairing (ALP) (Kannan et al., 2018) as an effective method for improving model robustness to corruptions and perturbations, even though it was designed to defend against adversarial attacks.
Revised: As discussed in subsection 3.1, Hendrycks & Dietterich (2019) point to Adversarial Logit Pairing (ALP) (Kannan et al., 2018) as an effective method for improving model robustness to corruptions and perturbations, even though it was designed to defend against adversarial attacks.

Original: We take ALP as an additional baseline on all benchmarks based on CIFAR-10 (using GN), following the training procedure in Kannan et al.
Revised: We take ALP as an additional baseline on all benchmarks based on CIFAR-10 (using GN), following the training procedure in Kannan et al.

Original: and their recommended hyper-parameters.
Revised: and their recommended hyper-parameters.

Original: The implementation of the adversarial attack comes from the codebase of Ding et al.
Revised: The implementation of the adversarial attack comes from the codebase of Ding et al.

Original: We did not run ALP on
Revised: We did not run ALP on

Original: ImageNet because the two papers we reference for this method, Kannan et al.
Revised: ImageNet because the two papers we reference for this method, Kannan et al.

Original: and Hendrycks & Dietterich (2019), did not run on ImageNet or make any claim or recommendation.
Revised: and Hendrycks & Dietterich (2019), did not run on ImageNet or make any claim or recommendation.

Original: E. R ESULTS ON CIFAR-10-C AND I MAGE N ET -C, LEVEL
Revised: E. R ESULTS ON CIFAR-10-C AND I MAGE N ET -C, LEVEL

Original: The following two tables correspond to the bar plots in section 3.
Revised: The following two tables correspond to the bar plots in section 3.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: TTTO 68.8 26.3 28.6 26.
Revised: TTTO 68.8 26.3 28.6 26.

Original: E. R ESULTS ON CIFAR-10-C, LEVELS 1- 
Revised: E. R ESULTS ON CIFAR-10-C, LEVELS 1-

Original: The following bar plots and tables are on levels 1-4 of CIFAR-10-C.
Revised: The following bar plots and tables are on levels 1-4 of CIFAR-10-C.

Original: The original distribution is the same for all levels, so are our results on the original distribution.
Revised: The original distribution is the same for all levels, so are our results on the original distribution.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: B
Revised: [Figure] [Table]

Original: [Table]
Revised: [Table]

Original: JT
Revised: JT

Original: [Table] [Table]
Revised: [Table] [Table]

Original: TTT
Revised: TTT

Original: [Table]
Revised: [Table]

Original: TTTO 8.2 22.9 20.0 23.
Revised: TTTO 8.2 22.9 20.0 23.

Original: ALP 16.5 21.3 20.5 24.
Revised: ALP 16.5 21.3 20.5 24.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: B 8.9 31.7 22.6 24.

Original: JT 8.1 31.0 22.6 23.
Revised: JT 8.1 31.0 22.6 23.

Original: TTT 7.9 28.8 20.7 23.
Revised: TTT 7.9 28.8 20.7 23.

Original: TTTO 8.2 16.8 13.8 15.
Revised: TTTO 8.2 16.8 13.8 15.

Original: ALP 16.5 18.0 17.2 19.
Revised: ALP 16.5 18.0 17.2 19.

Original: [Figure]

Original: [Figure]

Original: rr

Original: [Figure]

Original: r

Original: [Figure]
Revised: [Figure]

Original: orig gauss shot impul defoc glass motn zoom snow frost fog brit contr elast pixel jpeg
Revised: orig gauss shot impul defoc glass motn zoom snow frost fog brit contr elast pixel jpeg

Original: B 8.9 21.7 17.1 17.
Revised: B 8.9 21.7 17.1 17.

Original: [Table]
Revised: [Table]

Original: [Figure]